Event Driven APIs, or webhooks, bridge the gap between inherently decoupled, disparate applications.
Most APIs can be used synchronously. However, in working toward the notion of cooperative applications, apps should provide event driven APIs. Doing so improves the user experience of applying APIs and is extremely useful for some integration use cases.
In order to perform asynchronous API operations, you have a couple of options. The first is to poll. Polling uses unnecessary compute cycles, and has a frequency that is application specific. And if done concurrently, polling becomes complicated and introduces instability. Your second option is to let the API provider notify your application. This is done by registering a callback API with event handler logic, triggering the action for the callback API, and responding immediately to your users. Ultimately, the registered event handler is invoked when the data is ready for consumption. What I’ve just described is basically a webhook.
Event driven APIs have a number of use cases. They are helpful when you’re dealing with integrations that need to consume large volumes of data, or with applications that can’t keep a synchronous connection open for long periods of time (e.g. 15 minutes). Event driven APIs also come in handy when you have long running, intensive operations such as complicated searches, or when you’re dealing with services that cannot provide a definitive response synchronously.
The following is an example of a webhook workflow:
- Your app sends an email using a cloud email service
- Your app wants to know when the email is processed, sent, and opened
- Register the event handler or callback API for the webhook
- Create an email message and send it via the cloud email service
- Persist the email message ID
- Respond to your user that the message has been queued.
- The cloud email service provider invokes the callback API for each event
- Your app receives the event and message ID via the Callback API request
- Your app handles each event and notifies the user if applicable
There are two primary webhook pitfalls to be aware of: Circular references and security.
A circular reference example is as follows: updating Field 1 invokes a callback with a specific event, your app processes the event and invokes the API to update Field 2, and the update of Field 2 invokes a callback, which in turn updates Field 1 again, on and on forever. The bottom line is to always double check webhooks and their associated actions to ensure no circular references are introduced.
The most important security consideration is that the public API endpoint is now exposed to the world. A couple of options to mitigate the exposure risk are to implement temporary tokens or consumable message ID’s. By providing a new, random token with a request, that same token can be provided on the webhook callback, ensuring that the request is legitimate. Depending on the type of callback, the token may be “consumed” so that it cannot be reused. The options for these sorts of security measures are partly up to the vendor, and what they provide.Options for Testing Webhooks in Development
You can use the following mechanisms to test webhooks while in development: API Capture, TCP Tunneling, OpenSSH Tunneling, Dynamic DNS, and Remote Server.
The OpenSSH Tunneling mechanism opens and keeps open the socket from localhost to a remote server. The remote server is configured to pass requests down the connection. To initiate the tunnel, setup OpenSSH reverse proxy to a public server.
ssh -N -g -R :8080:localhost:8080 user@remote_server &
This option requires a remote server with root access or system administrator access, and enabling of GatewayPorts and AllowTcpForwarding on the remote server’s SSH configuration. As well, the tunnel will need to be reset if the internet connection changes.
For an explanation on API Capture, TCP Tunneling, Dynamic DNS, or Remote Server, check out our blog post, 4 Options When Developing with Webhooks.
Hopefully this explanation has given you a better understanding of webhooks, what they are, and why you might want to leverage them, or create your own. The options for using webhooks in a development environment should give you a good start down the path of integrating them into your applications.
Travis originally presented this at our ‘All Things API’ Meetup. If you would like to attend our next meetup, please check out our schedule. If you can’t make it to one of our events, be sure to check out our Developer Blog and Developer Portal.