Integrations are Hard Part 1: Resources, APIs, and Payloads, oh my!

By Brian Rothhaar in GET/technical Posted Apr 16, 2015

API integrations are hard, especially when you need to integrate with multiple endpoints in the same category (e.g. Documents, CRM, Finance). In this blog series, I’ll discuss some of the difficulties we’ve faced at Cloud Elements, and some of the solutions we’ve come up with. Let’s start with an overview of the series. Tune into following posts for more detail.

API Integrations are hard

DIFFICULTY 1: DISSIMILAR (OR BROKE-ASS) APIS

If you’ve ever tried to integrate with more than one API (even APIs for services that do the same thing, e.g. cloud storage), you’ve probably noticed differences in resources, pagination and search features. For example, an “incident” resource in one Help Desk system may be an “issue” in another. And don’t even get me started on CRM systems and their users/people/accounts/customers/organizations/contacts/activities/leads/opportunities.

Beyond the obvious differences in managing API resources, once you get to the details of search and pagination, more difficulties arise. Some services provide a page size and offset mechanism for paginating search results, others use next page tokens.

These problems are present when integrating with sane APIs. Things get even worse when you need to integrate with a less-sane API. I won’t name names, but we’ve seen some doozies.

We address many of these difficulties by normalizing resources and APIs when we can, and where it makes sense.

DIFFICULTY 2: MISSING FEATURES (UGH)

Have you ever been working with a service API and came across a feature that you really needed, but the API didn’t offer? Me too. A good example is event notifications.

Some services (such as Shopify and Dropbox) provide webhook callbacks to notify a user (or customer application) that something has happened. Other services (like Etsy and SharePoint) have no such feature. The Cloud Elements platform helps in this case by emulating the event-driven webhook callback through a polling mechanism, effectively adding this feature to existing services. Events handled through the platform will behave the same, regardless of whether the endpoint supports them.

DIFFICULTY 3: PAYLOADS

What do you do when request or response payloads don’t jive with your interpretation of the data they represent? Why is an account in one system a user in another? Why did Salesforce label my custom field Dumb_Custom_Field_Name__c when I wanted it to be called dumbCustomFieldName?

We’ve dealt with this by developing a transformations framework that is capable of transforming the request and response payloads going to and from a service. This allows you to define the payload format in any way you choose for easy integration with your application.

DIFFICULTY 4: SYNCHRONIZATION

So, what if you need to synchronize resources between two endpoints? What if you’d like to update comments in Zendesk when a related ticket is updated in JIRA? We provide workflows to assist in cases like this. A workflow can be triggered by an event or an API call and can perform actions to synchronize data, notify other users, or anything else you can think of (within reason).

In later posts in this series, I’ll go into more detail on these problems and some of the solutions we’ve come up with.

10 Step Guide to API Integrations