Combining Various Object Data in to a Virtual Data Resource (VDR)

By Jamie Thorson in Posted Jul 31, 2018

High speed data connections have allowed us to progress in the world with significantly large amounts of information about any number of resources. For every customer or lead you have, it is quite possible that you have hundreds or thousands of lines of data about that customer. From personal and business data to specific orders and customer orders line items, we can access it all. Access to this data has opened up doors of possibility for new integrations and business models. As businesses need access to more information, endpoints have began offering more data. As they offer more data, more businesses open, ready to use this data.

However with so much data it can be overwhelming to get the information you actually want. Every business model revolves around specific use cases and the data you need to be successful will differ from other businesses. Therefore it is crucial when beginning the integration journey that you have an easy way to separate out the data you need from the data that you don't.

Many of you may be familiar with Cloud Elements and our Virtual Data Resources. Did you know you can combine data from various objects into a single resource? In other words, instead of having to make a call to a /customers endpoint and then having to make a call to /orders endpoint, you make a call to one endpoint and get the specified data you care about. Quite possibly you don't want this for one endpoint or just for yourself. Maybe you want this for all of your payment/order integrations, and for all of your customers to have access to the same data on their accounts.

So how is this possible if the endpoint does not offer exactly what you want? I would like to say “it's easy” but the truth is, no custom integration is easy. But it’s made easier thanks to Cloud Elements and we have the tools to help you become more successful, faster. Below is how it works, but before I give you all the details it is important that if you are not a Cloud Elements customer and want to try this you go to our websites homepage and hit the “try it out” button and sign up for a free trial.

I will walk through the steps with you. As a basic overview, what we are about to walk through is creating two Virtual Data Resources with the information we want from each of the unique endpoints and then having one of them use a small amount of Javascript (within the transformation) to invoke the other one and add it to the transformed data. Then we will add that to our api docs to complete the full flow. If you would like to follow along with these steps and have a basic understanding of the Cloud Elements platform please visit https://github.com/cloud-elements/ce-examples/tree/master/vdrs/combining-objects to get some open source reusable code snippets to help you get started.

*Note: for my example I am using Shopify. I will be making a call to a customer endpoint and the result I desire it to get some customer information and some of that customers order information from my one call. For this case, in my customer Virtual Data Resource I will make a call to my orders Virtual Data Resource. If this is not your exact use case don't worry, the same basic Javascript can be used for any elements and endpoints you are trying to combine if you alter the steps ever so slightly. 

1. In your cloud-elements account make sure you have a Shopify instance and then on the left hand side click the Virtual Data Resource button and create a new resource. I am naming my resource MyOrders and mapping it to Shopify's native /order endpoint. I also made my an Organization level field so all Shopify instances in my Cloud Elements account will have access to this resource which means that any one of my users that authenticate to their Shopify account will be able to see the same manipulated data that I am seeing in my account in their account.

 

2. Next I make a Virtual Data Resource on the customer object with the customer information that I need. For this example I will call this myCustomers and it will be on the Shopify native /customer object.

 
3. Now I need to write some Javascript to call the orders Virtual Data Resource I made called MyOrders within the myCustomers work
 
 
4. In this part, depending on how I want these objects to combine a little bit of manipulation to the Javascript might be required. However for this case if I get a customer by id (or email) I want to then use that id (or in the case of email the id in the returned object of the email) to make a call to get that customers orders and bring them back to this Virtual Data Resource.
 
The first step of this is to note that I only want to make this extra call when I am getting customer data so for this use case I am not doing the same get customer orders call for a POST, PUT, or PATCH. In other words I only need to do this extra call when the information is coming from the vendor (not going to the vendor) so I begin my Javascript with an if statement

if(fromVendor){ … }

  1. Then we will construct the http request to our other Virtual Data Resource 

It is important to note:

  • port can be configured if you are testing locally but it not required if you are using this in our platform.
  • You will need to include the id in the myCustomers VDR in order to make sure that the way the vendor sets id is standard for all elements and won't need future tweaking and incase the id is not returned as a number you need to make sure to encode the query
  • All of the other fields are similar to what you would use in a typical call to Cloud Elements and should look familiar

`/elements/api-v2/myOrders?where=${encodeURIComponent(`CustomerID='${customerId}'`)}`

The above code constructs the basic request but you will still need to handle the response. How you handle the response data is ultimately your choice but this is an example of one way to handle the data 

In this example we are handling response code that are not equal to 200 by just sending back https call failed but you can handle these differently or make specific else if statements for various status codes to be handled in unique ways (although if set up correctly you should not be hitting these errors often). This is the last step to getting your data. You can build a function and pass in parsedData to do more manipulating with the response data or you do something as simple as adding it to your transformed object (shows above).

Finally in our myCustomer Virtual Data Resource we can hit the transformation setting and click the boolean to make sure Add to API Docs is set to true. Now if we look at our api docs we will see that our Virtual Data Resource was added but the cool thing is it is not added just to our authenticated instance, this will be included in any instance that is authenticated in this Cloud Elements account.

Now when I call GET myCustomerOrder with a where statement and include a customer id I will get a response object that includes the combined customer and order data in just the way I want it and I have a Virtual Data Resource that can mostly be reused (I will need to add the Javascript for every element I attached to the Virtual Data Resource and make sure that I update the path as needed) but with just minor changes I now have a Virtual Data Resource that has the information I want for all of my payment/order and it is available to all of my customers authenticated accounts.

Thanks for following along with me on this. As I mentioned before, no custom integration is easy. It takes work to get the data the way you want it but now imagine spending time and work on this first set up and then being able to reuse this for so many endpoints including endpoints like Magento 1.9 Soap api's

You could create a second one-off solution to connect to Magento and then handle the soap response and manipulate the customer and order data to try and yield results similar to Shopify but that sounds like a lot of extra work, whereas now you can just create an instance of Magento, tie it to your Virtual Data Resources, edit the Javascript slightly to call the correct query for Magento orders and get a nice JSON response with your specified data.

As data content and data centric integrations expand the need for multi-connector solutions and creating custom but uniform data is going to become increasingly important. The ability to easily add new connection and reuse work specific to your business needs, is going to speed up your ability to expand and onboard/integrate your customers. Setting up the ability to combine various objects data in to a single Virtual Data Resource in a way that is reusable and accessible allows you to enable and push your business and your customers on their journey to success.