Webhooks vs. Polling: Sync your Apps

By Mark Geene in How-To, Cooperative Apps Posted Oct 8, 2015

The ultimate magic of integration happens once you have your objects and resources mapped, your data transformed, and now apps are staying in sync with events on a regular cadence.

Welcome back to the Cooperative App series, a series designed to take you step-by-step through the process of designing a fully orchestrated integration between your application and other cloud services. As we have learned, it’s more than writing to the API. It’s about authentication, discovering custom objects, mapping and transforming data. Today I’d like to dive further into the series discussing how you can leverage webhooks and events to trigger actions, execute workflows and synchronize data.

Types of Event Management

The first step is to determine how you’ll consume events from the endpoint(s) you're integrating with.  Many endpoints support webhooks that can be used to track create, update and delete actions as they occur to objects notifying your application. Webhooks keep your app up-to-date with changes (events) at the endpoint.Cloud Documents Webhooks Workflow

If the endpoint does not support webhooks, then your application will need to poll the endpoint to identify changes as they occur. With polling, you’re querying the timestamps on objects at the endpoint to see if actions (create, update or delete) have occurred. Your app becomes responsible for reaching out on a periodic basis based on a frequency that you set such as every minute, hourly or daily.

CRUD Actions Create Events

Events are critical to giving you the “trigger” to sync data between multiple endpoints. RESTful methods initiate Create, Retrieve, Update and Delete actions to resources at endpoints using GET, POST, PUT, PATCH and DELETE methods. Since the goal is to synchronize data when it’s changed, you will need to consume events that Create, Update and Delete data.  Consuming events will enable your application to execute corresponding methods to Create, Update or Delete when notified that an event has occurred at the endpoint; thereby synchronizing actions that change data.

A Source of Truth?Marketing App Webhook Example

In many use cases, consider which endpoint will be the source of truth (“master”) for the data object and which app will be the consumer of the data. As an example, let’s say we’re developing a new “MyCoolNewMarketingApp.” The “MyCoolNewMarketingApp” needs to consume the the source of truth for Company data, which in this case is the Salesforce CRM Account Object.

MyCoolNewMarketingApp” will consume Company (aka “Account”) data from Salesforce. Since Salesforce is the Master, we will retrieve Company data from Salesforce in “MyCoolNewMarketingApp” to see if it already exists. If it does then we’ll populate Company data from Salesforce, and then synchronize any changes we make between the endpoints. If a Company does not exist in Salesforce, we will create a new Account in Salesforce and then synchronize.

In this case, we let the Salesforce Company object be the master, or source of truth, for all Company data that “MyCoolNewMarketingApp” consumes.

As a Product Manager of “MyCoolNewMarketingApp,” questions I will be asking:

  • Where are my customers going to master and keep data?
  • How will my App consume this data e.g., will I copy it or just look it up from my App?
  • What do I need to synchronize on? CREATES, UPDATES, and/or DELETES?

EXAMPLES OF WEBHOOKS IN ACTION

Implementing webhooks varies greatly by endpoint. Many services have a UI to create your triggers or webhooks and enter your specific callback URL. Zendesk Webhooks APIFor example, with Zendesk you choose what type of event you want, and you provide the URL for a callback. Also with Zendesk, you can modify the data that you would post to that URL.

You can setup a webhook to listen for when a ticket is created or updated, and then specify which URL you want, and what specific data you would like returned in the body (e.g. ticket ID and status). An example of response from Zendesk is shown below for Standard Event Webhooks. Here ^ you will see a quick walkthrough of the Zendesk UI to create the event trigger. Learn more from our Developer Documentation on Zendesk Events.

STANDARD EVENT WEBHOOK

Here is an example of response Zendesk would return as the body of a POST for a webhook. With this, you can use the ticket ID to do a GET and get more information on the ticket that was updated:

Zendesk Webhook JSON

CUSTOM EVENT WEBHOOK

Here is an example of response from a Salesforce webhook. Keep in mind that Salesforce does not inherently support webhooks; they have internal language to create webhooks. The example below comes from an ApexTrigger class that we wrote to post our events URL when an object is created, updated, or deleted. To work with Salesforce, you have to write custom code that will send data to your specified endpoint.

Salesforce Webhooks JSON

**We are currently working on updating our event orchestration with Salesforce to change over all of our custom webhooks to a polling framework. Much easier to configure.**

POLLING FRAMEWORKS

When an endpoint does not support webhooks you will need to implement a polling framework in order to consume events. This is a more significant effort, as your application now needs to call out to the endpoint to check if anything has changed since the last API call.

HubSpot Polling Framework

By creating a polling framework we are getting a POST call from an endpoint provider. We are making the GET requests to the provider at a certain time interval (every minute, every hour, every day), to retrieve any changes that have been made within that time interval. You can setup the framework by searching for the last modified date. (*This all depends on what the end provider supports, search our developer documentation for specifics on each of our integration endpoints limitations).

The final topic in the Cooperative App Series will discuss how to pull all of the integration design aspects, from authentication and transformation to bulk loading and polling, together into measurable data for logging and usage. Subscribe to our blog to stay up-to-date with the latest posts in the series. Until then, if you need help creating a killer cooperative app, get the eBook 'The Definitive Guide to API Integrations'.  

the-definitive-guide-to-API-Integrations-eBook