Accessing external APIs from eagle.io

Let's get the bad news out of the way first; when faced with the question "Can I use eagle.io to access an external API which has the data I want", then in general the answer is no. Specifically, eagle.io doesn't have the capabilities (as of writing this article) to act as a fully-featured API client such that it can connect to and authenticate with an API, and then download data.

But that's not the end of the story...

In some cases, the data you want may be accessible as a web service by using a URL, without requiring additional authentication or headers. In these cases, if the URL returns delimited text that can be parsed by eagle.io, then you will be able to download and ingest this data using the Download via HTTP transport. How do you know if the URL requires additional authentication or headers? As a general rule, if you can download the data by putting the URL in your web browser, then you will also be able to access it via an eagle.io data source.

Let's use USGS earthquake data as an example. This USGS website describes the various recent earthquake data that is available in "Spreadsheet Format" (which is their way of saying CSV files). It includes a link to download all significant earthquake data for the last week; the URL is:

https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.csv

If you click that link in your web browser, it will either download or display the CSV file (depending on your browser settings). A snippet of the file looks like this in my text editor:

This certainly looks like the sort of time-series data that eagle.io was designed to ingest. And it's accessible simply via a URL (which we proved with a web browser). So it's time to set up a data source.

Create a new data source and choose File, Delimited Text:

Choose a transport type of Download via HTTP:

Enter the URL (it's not all visible in the screenshot below):

Configure the parser to suit the data:

And now we have ingested the earthquake time-series data. Below is a fragment of the eagle.io Parameters view:

What about web service data that is protected via authentication? It's still possible to access the data if the type of authentication used by the web service is HTTP Basic authentication. In this case, the username and password can be included in the URL in the following format:

http://:@example.com

For example, assuming the USGS URL from the previous example required HTTP Basic authentication (which it doesn't) with a username of foo and a password of bar, then the full URL would now be:

https://foo:bar@earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.csv

The same guideline still applies from the previous example; if you could use this URL in your web browser to access the data, you should also be able to use it in eagle.io with the Download via HTTP transport.

When an external API or web service has more complex authentication, such as using tokens or headers, then eagle.io is no longer able to access the data directly. And even in cases where you can access the data, if it is not available as delimited text (such as CSV) then it cannot be parsed by eagle.io.

In these situations, third-party software would be required. The software would first need to access the external API or web service, negotiate any authentication that is required, and download the data. If necessary, the software would then need to translate the data into a format that eagle.io can ingest (such as delimited text or JTS JSON). Finally, it would use an API key to access the eagle.io API and send the translated data to a data source.

This software must be written, hosted and scheduled externally to eagle.io. To avoid encountering API rate and size limits, it should be able to keep track of what data has already been sent in previous transmissions, so that only new data is sent in each subsequent transmission.

In future, eagle.io plans to include the ability to add HTTP headers within the Download via HTTP transport, which will increase the flexibility of this option. The text parser will also be enhanced to allow JSON data to be parsed. These features combined should make a greater range of external APIs and web services accessible by eagle.io data sources.