Integrations are Hard Part II: API Resources, Search and Pagination

By Brian Rothhaar in GET/technical Posted Jun 3, 2015

Many of our customers have use cases involving integrations with multiple different services within the same category. These services provide APIs with similar, but not necessarily identical resources. Beyond obvious differences in the way resources are organized within these APIs, behavior and features of individual APIs can vary greatly.

 

10 Step Guide to API Integrations

Integrations-are-hard-part-2-1API RESOURCES

Often, one needs to integrate with multiple services performing similar functions. For example, in a business that must deal with data in multiple CRM systems, tracking corresponding data between the two systems can be a chore.

Different CRM systems may have customers, organizations, contacts, activities, leads and opportunities. These resources may mean the same thing, something totally different, or anywhere in between depending on the service. In Help Desk systems, an ‘incident’ in one help desk system may be an ‘issue’ in another and a ‘ticket’ in a third. One service’s ‘user’ may be another’s ‘account’.

We take a different approach. When possible and where it makes sense, we normalize resources and APIs between cloud services. This provides developers a more uniform approach to integrating with API categories. See our Hub API documentation for details.

In some cases, we normalize or modify payloads, as well. When we do this, we either filter or enrich (or both) the payload returned by the service. We provide some user-level tools to customize this behavior. Transformations, for example, allow a developer to customize the format of the returned payload. Upcoming features of our workflow framework will allow user-customized payload enrichment and filtering, as well.

SEARCH

Searching APIsIt seems like there are as many approaches to search APIs as there are cloud services. Some services provide a full-featured, SQL-like language for querying every available resource field. Salesforce.com, for example, provides SOQL for this purpose. Some services, Instagram for example, have a much narrower set of fields that users can filter by. Other services, like Pipedrive, handle search functions by providing JSON-defined filters that can be managed via APIs and attached to search calls to filter the returned data.

To address these differences, we have developed our own SQL-like query language called CEQL. Regardless of the search capabilities provided by the service, a developer can write queries in a familiar language to refine search results. The CEQL query is transformed and rendered into the service’s particular search capabilities and the call is made from the Cloud Elements platform.

This approach is not without it’s limitations, of course. For example, when an endpoint doesn’t allow searching on a given field, we don’t support it. (Or do we? Call our sales team for details. Just kidding! Or am I? I’m not. Though I wish I was.)

PAGINATION

Result pagination, where large result sets are returned across multiple API calls, is another area where you may see significantly different approaches. There are three main pagination types that we commonly see: limit/offset, page/page size and cursor-based.

  • The limit/offset and page/page size cases are similar. The limit/offset approach uses a limit (the max number of results returned) and an offset (the position of the first result) to determine the result set.
  • The page/page size approach is should be self explanatory to anyone who has ever had to move past the first page of Google results (shudder). The page size indicates how many results should be returned, while page tells which set of results to return. A little basic arithmetic can convert a limit/offset scheme to page/page size and vice versa.
  • Cursor-based paging returns a token value or URL that is provided with the following call to retrieve the next set of results. The cursor token is usually combined with a value to determine the maximum results to return.

It turns out that all three pagination types can be emulated with a cursor-based scheme, which is precisely what we do in the Cloud Elements platform. Ultimately, we return a token that is an encoded version of the limit and offset of the next set of results. This token can then be returned with the next call. Alternatively, we support indicating limit and offset directly, as well.

CONCLUSION

In my experience integrating with cloud services, I’ve found that the devil is in the details. Differences and nuances in APIs generally take an inordinate amount of the development effort.

Read more: