A ProgrammableWeb plugin for SoapUI

The ProgrammableWeb is one of the major players in the API-directory space - wouldn't it be neat if you could browse their directory from inside SoapUI and import an API directly into your SoapUI project from there? Of course it would! Let's get right to it and create a simple plugin for SoapUI that lets you do just that.

Prerequisites

The API directory at programmableweb.com holds at the time of writing over 8000 APIs in its database, and although the majority is based on REST principles, over 1800 are actually still using SOAP. Since SoapUI can easily import the WSDL for a SOAP based API to allow you to do some ad-hoc explorations of the APIs operations, let's add a dialog to SoapUI that allows you to browse those SOAP-based APIs and import them into SoapUI. Of course SoapUI could import REST APIs as well if they had WADL or swagger definitions, but since the ProgrammableWeb directory doesn't expose links for these metadata formats, we'll have to stick to SOAP/WSDL for now.

My initial thought was to use the ProgrammableWeb Atom-based API for browsing and extracting all SOAP APIs in their directory, but unfortunately the API currently has no easy way to search for these APIs, I would basically have had to go through the entire directory, page by page, (remember, over 8000 APIs) and manually extract those of interest. Instead, I opted to do something I never thought I would do, "screen-scrape" the web interface of programmableweb.com - which is of course extremely fragile since any change to the underlying HTML will make the plugin useless, but on the other hand - the URL http://www.programmableweb.com/apis/directory/1?protocol=SOAP returns a nicely formatted list of all SOAP APIs in the directory - making things so much easier. As an added advantage, this approach does not require me (or anyone else using the plugin) to have a ProgrammableWeb API key, as we are just extracting data and not making modifications.

Enough talk - let's look at how this baby actually works:

In Action

Before we dive into the code, here comes a screenshot of the actual dialog invoked from a new "Add API from ProgrammableWeb" action on the project popup menu:

As you can see you select one of the API categories on top and the list under will contain all SOAP APIs in that category. The WSDL endpoint for the API is shown at the bottom - pressing OK will import the WSDL directly into SoapUI for further exploration:

I've posted a short video of this in action further down - let's just have a quick look at the code first.

The plugin code

The plug-in code is straight-forward - just like the soapui-swagger-plugin I published a couple of weeks ago. I'm not going to go through all lines of code (you can do that on GitHub, see below), just highlight a few things.

First, the project structure:

As you can see this is a maven project using both groovy and java. The main reason for using groovy was (once again) its superiority when it comes to parsing; the GPathResult returned by the HTTPBuilder for the ProgrammableWeb HTML pages is easily searched for the relevant information. For example the following code is used to load the SOAP API listing and extract all the APIs into a Map of categories to collections.

And the code for extracting the WSDL from an APIs detail page is equally simple:

The remaining code in the AddWsdlFromProgrammableWebAction class is mainly boilerplate code for setting up the form and initializing its data. Since the form is created only once SoapUI will have to be restarted for new APIs to show up in these lists.

A very short video

Ok, here comes a short video showing you how this actually works;

Nice - isn't it!?

Download and installation

Installing the plugin is straight-forward, download it from here and drop it in the soapui\bin\plugins folder (you will need at least SoapUI 4.5 - download from http://www.soapui.org). Also you will need to add the http-builder-0.6.jar and xml-resolver.jar files to the soapui\bin\ext folder - which are needed for loading and parsing the HTML pages. If your're on a Mac you can find or add these folders "inside" the application bundle, presumably at /Applications/SmartBear/soapUI-4.5.1.1-SNAPSHOT.app/Contents/Resources/app/bin.

SoapUI will load these during startup, as seen in the SoapUI log:

Wrapping up

That's it - check out the code at GitHub (https://github.com/olensmar/soapui-programmableweb-plugin) - and have fun with the plugin - and please don't hesitate to comment, suggest or complain :-)

Thanks for your time,

/Ole