Experiments in Integration (Part 1): The Serverless Framework

By Rick Rothenberg in GET/technical Posted Oct 31, 2018

As a member of the Cloud Elements Delivery Team, I see a lot of integrations; both those we build ourselves (on behalf of customers), as well as those our customers build. And often, while thinking about those integrations, I’m also thinking about how we can make them better/faster/easier/reliable-er/etc.

integration-experiment-blogbanner

I’m particularly drawn to easier. To some extent, that’s because I’m kind of lazy (just ask my wife). 

One aspect of programming that can feel like it’s more work than it ought to be is deployment. There are nearly as many options out there for deployment, as there are programmers to argue about the best one. One approach to deployment that promises to make things easier is the serverless architecture or FaaS (Function-as-a-Service). Doing away with a dedicated server has a number of advantages - potential cost savings, scaling by default, reduced system administration and a simpler deployment process.

There are still a number of options to choose from such as AWS Lambdas, Google Cloud Functions, and Azure Functions. Cloud Elements even offers our own FaaS - a low code, graphical programming facility called Formulas which can be an excellent choice for creating an integration. I’ve personally used Formulas many times. But I’m an old school programmer who really likes code so I’m looking for ways to make it as easy to build and deploy code as it is to deploy Formulas.

One thing that has piqued my interest lately is the Serverless Framework. The Serverless Framework is an open-source CLI for building and deploying serverless applications. It supports a number of different serverless architectures and provides a mostly uniform way of deploying and testing serverless applications. It also allows extension via a plugin mechanism. So it was hard to resist building a Cloud Elements specific plugin for it to see how much it helped make my life easier. Thus, this experiment in integration was born.

Serverless Framework

The question then became “how do I want this to work”? Serverless Framework has a concept of resources that you define and it manages them for you. Examples of supported resources include (for AWS) DynamoDB tables, S3 buckets, and SNS topics. It seemed reasonable to leverage this capability by adding types for Elements (our idea of a connector for a specific vendor, such as Salesforce, Marketo, or DropBox). When you specify a resource in your serverless.yml configuration file the plugin will supply a Javascript SDK (using our SDKifier tool) that can be used by the code for accessing the Element.

Serverless Framework also has a concept of events which can be configured and then delivered to your code. Cloud Elements has a similar concept of events that can be delivered from an instance. The plugin will configure an API Gateway endpoint to receive the event and then wire the correct URL after deployment such that the instance invokes the code when events occur.

An example of a simple contact sync showing both the code and the configuration file is shown below.

If you’d like to experiment yourself with this plugin, you can download it using npm.

There are limitations to this approach, like there’s a maximum execution time of about 5 minutes with the FaaS products currently available. I have some thoughts about addressing that I’ll talk about in my next blog post, titled, not surprisingly, Experiments in Integration, Part 2.

image (2)-1So far, I feel like this is a promising direction, though YMMV. I’d love to hear your thoughts on what’s hard about doing integrations and what you think might make it easier.

So I can steal your ideas.

Because I’m lazy.