Step 9: Events and Synchronization
The ultimate magic of integration happens once you have your objects and resources mapped, your data transformed, and now apps are staying in sync with events on a regular cadence.
Welcome back to the Getting Started with API Integrations series, a series designed to take you step-by-step through the process of designing a fully orchestrated integration between your application and other cloud services.
As we have learned, it’s more than writing to the API. It’s about authentication, discovering custom objects, mapping and transforming data. Today I’d like to dive further into the series discussing how you can leverage webhooks and events to trigger actions, execute workflows and synchronize data.
Types of Event Management
The first step is to determine how you’ll consume events from the endpoint(s) you're integrating with. Many endpoints support webhooks which are used to track create, update and delete actions as they occur to objects. Webhooks keep your app up-to-date with changes (events) at the endpoint.
In the event your API endpoint does not support webhooks, then the application will need to poll the endpoint to identify changes as they occur. With polling, your app becomes responsible for reaching out on a periodic basis to see if actions (create, update or delete) have occurred.
CRUD ACTIONS CREATE API EVENTS
Events are critical to giving you the “trigger” to sync data between multiple endpoints. RESTful methods initiate Create, Retrieve, Update and Delete actions to resources at endpoints using GET, POST, PUT, PATCH and DELETE methods. Since the goal is to synchronize data when it’s changed, you will need to consume events that Create, Update and Delete data. Consuming events will enable your application to execute corresponding methods to Create, Update or Delete when notified that an event has occurred at the endpoint; thereby synchronizing actions that change data.
Endpoint: the Source of Truth?
In many use cases, consider which endpoint will be the source of truth (“master”) for the data object and which app will be the consumer of the data.
As an example, let’s say we’re developing a new “MyCoolNewMarketingApp.” The “MyCoolNewMarketingApp” needs to consume the the source of truth for Company data, which in this case is the Salesforce CRM Account Object.
“MyCoolNewMarketingApp” will consume Company (aka “Account”) data from Salesforce. Since Salesforce is the Master, we will retrieve Company data from Salesforce in “MyCoolNewMarketingApp” to see if it already exists. If it does then we’ll populate Company data from Salesforce, and then synchronize any changes we make between the endpoints. If a Company does not exist in Salesforce, we will create a new Account in Salesforce and then synchronize.
In this case, we let the Salesforce Company object be the master, or source of truth, for all Company data that “MyCoolNewMarketingApp” consumes.
Three questions A Product Manager will ask:
- Where are my customers going to master and keep data?
- How will my App consume this data e.g., will I copy it or just look it up from my App?
- What do I need to synchronize on? CREATES, UPDATES, and/or DELETES?
EXAMPLES OF WEBHOOKS IN ACTION
Implementing webhooks varies greatly by endpoint. Many services have a UI to create your triggers or webhooks and enter your specific callback URL.
For example, with Zendesk you choose what type of event you want, and you provide the URL for a callback. Also with Zendesk, you can modify the data that you would post to that URL.
You can setup a webhook to listen for when a ticket is created or updated, and then specify which URL you want, and what specific data you would like returned in the body (e.g. ticket ID and status).
An example of response from Zendesk is shown below for Standard Event Webhooks. Here ^ you will see a quick walkthrough of the Zendesk UI to create the event trigger. Learn more from our Developer Documentation on Zendesk Events.
STANDARD EVENT WEBHOOK
Here is an example of response Zendesk would return as the body of a POST for a webhook. With this, you can use the ticket ID to do a GET and get more information on the ticket that was updated:
CUSTOM EVENT WEBHOOK
Here is an example of response from a Salesforce webhook. Keep in mind that Salesforce does not inherently support webhooks; they have internal language to create webhooks. The example below comes from an ApexTrigger class that we wrote to post our events URL when an object is created, updated, or deleted. To work with Salesforce, you have to write custom code that will send data to your specified endpoint.
**We are currently working on updating our event orchestration with Salesforce to change over all of our custom webhooks to a polling framework. Much easier to configure.**
When an API endpoint does not support webhooks, you will need to implement a polling framework in order to consume events. This is a more significant effort, as your application now needs to call out to the endpoint to check if anything has changed since the last API call.
By creating a polling framework we are getting a POST call from an endpoint provider. We are making the GET requests to the provider at a certain time interval (every minute, every hour, every day), to retrieve any changes that have been made within that time interval.
You can setup the framework by searching for the last modified date. (*This all depends on what the end provider supports, search our developer documentation for specifics on each of our integration endpoints limitations).
The final topic in this series will discuss how to pull all of the integration design aspects, from authentication and transformation to bulk loading and polling, together into measurable data for logging and usage.
Need help creating a killer cooperative app? Dive into our resource: The Definitive guide to APi integrations.