Don’t Get Stranded on API Data Islands

By David Honan in GET/technical, Product Updates Posted Nov 16, 2018

I’ve seen many API specifications, varying across the board from SOAP to REST, with random others in between. While you can typically sum up the components of an API spec into category types—such as web service, authentication, resources, methods, events, search, and so forth— they are never the same in terms of data payload definition. Every endpoint is different in how it expects to receive or return data from calls made via its API. It is its own API island, with its own language, so to speak.

Each API is and island of data


POINT-TO-POINT
So if you have a SaaS application or you are looking to sync data between endpoints, the traditional way many iPaaS platforms solve the issue of inconsistent data structures is to provide simple wrappers, or “connectors,” which convert the data into a format that can be read by another connector designed to work with it. This, in the world of API integration, is referred to as point-to-point connectivity. This works well and has been the model used to connect endpoint services together for years.

But does it scale? No.

Why? Because point-to-point is great when integrating to a small number of back office services, but it falls short when your use case requires integrations to multiple endpoint services or if reusability of a sync application across multiple endpoints is required.

With point-to-point based solutions, each new endpoint needs to be connected directly to every other endpoint in the ecosystem. Each connector is specialized, custom, and programmed to connect to only one other endpoint rather than map and transform data to a common object format that can be consumed by multiple other connectors.

That said, there are platforms that have the notion of a common object or format converter functions, which are more optimal than point-to-point, for sure. But typically these data mappings are one-size-fits-all and not generally adjustable or modifiable to your specific business needs. Thus, the point-to-point nature is still in play in this case, just lessened in severity.

DATA ISLANDS
Speaking of data mappings, many integration platforms ignore the fundamental value of integration: the data. By ignoring the data, it is impossible to create centralized or canonicalized data models that can be governed by your application’s data needs. As a result, you are limited to what a generic connector provides in terms of data interactions and thus, you still have to create endpoint specific data mapping customizations. Again, back to the point-to-point connection issue.

Another issue with point-to-point based connectors is that they typically do not expose a callable API resource that allows your SaaS application to directly interact with a common object definition, even if the common object is built into the given connector. In other words, even if the connector has the ability to map and transform data to a common format, it is not set up in a way that allows you to interact directly, via API, with that common data definition rather than the using the native API data definition provided by the endpoint. You are stuck trusting the connector to interact with endpoint service data on your behalf.

So what is the next generation of the above? What you need is a platform that provides, out of the box, the ability to:

  • Define what data your application needs to interact with

  • Bidirectionally transform that data definition as required to the endpoint’s data specifications

  • Interact with a callable API resource based on your specific data definition


THE VIRTUAL DATA RESOURCE (VDR)
Cloud Elements’ Virtual Data Resources put your data models at the center of your application ecosystem and enable you to manage the data you care about in the structure that is best for your application or business. VDRs provide a canonicalized view of your data objects while eliminating the need for point-to-point mapping of data to each and every new application.

 

I often use this analogy when I demo Cloud Elements’ VDR:

It is like you called up Salesforce’s CEO and said, “Hey Marc Benioff, listen, I need you to change your API spec to match my application’s specific data definitions—and I want that in the next 20 mins.”

Yeah, right. Marc is awesome, but it is not very likely he’d make that happen specifically for your app.

But guess what? That is what a VDR gets you—a callable, REST resource that is crafted specifically to your application’s data needs, across multiple endpoints - regardless of whether you have a SaaS app or are constructing a back office data sync.

If you are curious, even slightly technical, and have eight minutes to spare, here is a demo of VDRs in action!


WATCH THE QUICK VDR DEMO HERE



HOW IT'S DONE
You can create your own VDRs that match your specific data needs easily in the Cloud Elements platform, either via our UI or via our REST APIs.

For the UI, you just click on “Build a new VDR”:

Virtual Data Resource | Cloud Elements


And enter the data fields that you need, mapping them to the corresponding endpoint’s data fields. Note that you can also enter JavaScript as needed if transformations are required beyond just field mapping.

Data Transformations | Cloud Elements


Virtual Data Resource templates
are also coming soon to the Cloud Elements community. We’ve created and will soon publish the first set of templates for the more common resources used by our customers and pre-mapped them to multiple endpoints for you, complete with example JavaScript transformations. They are just that, a template, providing a starting point and saving you time as you tailor them to meet your specific use case and data structure needs.

You'll be able to view a catalog of VDR templates in the Cloud Elements UI, clone them, and make them yours! There will be VDR templates that provide minimalist fields (e.g. /basic-contacts) for simple use cases, as well as full-blown, super-detailed, every-field-included (e.g. /contacts) VDR templates. Simply clone, modify, and use!


Virtual Data Resources Catalog | Cloud Elements


THE NEXT LEVEL
I’ll leave you with this thought: at the end of the day, accessing and sharing data is the fundamental reason for integrating into endpoint services. The Virtual Data Resource provides the next level way to do that. We purposefully designed the VDR to provide a way to interact with the data islands across endpoints as defined by you and your application’s data needs, wrapped up in callable REST resource. Your application can now easily interact with endpoint data islands as if each spoke a common API language that was defined by you!

And you don’t even need to ask the CEO of each endpoint company to modify their API specification—no need to trouble Marc Benioff, as he's busy enough already.

If you're curious about how you can leverage VDRs to access, share, and sync your data across endpoints, try out our API integration platform for free for 30 days by clicking below. Enjoy, aloha!

Try Cloud Elements for 30 Days FREE