Back to the 90s: Moving on from Flat Files to Automated Integration

By David Honan in GET/technical Posted Jun 29, 2018

The saying "I still think 1990 was 10 years ago," so perfectly reflects not only my current view of time, but also how many cloud applications, even modern ones, handle integrations. The use of flat file transfers as a supported method of syncing data - which was the most common way of doing so in the 1990s (and even in the early 2000s) - while proven, often requires quite a bit of manual intervention to get the data in the correct format before it can be loaded into the receiving service. Preparing flat files is also error prone - mistakes happen and when they do, they are often hard to undo.


There are plenty of reasons (code for excuses) as to why flat files are still used:

  1. There are so many services out there that it can seem impossible to create a reliable, integration to each one - it simply costs too much and takes too long to integrate directly via APIs into each endpoint.

  2. Providing a flat file export and import capability in your product is a good, MVP way to get integrations going and get to market. It is zero or very low cost for your customers as they they already understand how to make flat files work, and can use interns to support the process.
  3. Flat files make moving and handling data in bulk easier than “singular” API calls as many endpoints don’t support bulk data transfers natively via their API.

import files csv

For example, let's say your business handles the actual payments for your customers. Likely they are using something like Quickbooks, NetSuite, Freshbooks, Plaid or some other financial software to run their business. What they buy your service for is to facilitate making payments out to their vendors, thus they need to let your service know who and when to pay out.

The reverse is true too. For example, once payments are made within your service, your customers will need to extract the data from your service, and load it into their finance system. The fastest way to make this happen is to use a flat file transfer import / export mechanism. Even modern, prominent, successful payment services use flat files for transfers. In fact, there are several YouTube videos showing how to do this, mainly for importing payment status data into Quickbooks.

For the second use case where your customers would export payment status data from your service and load it into Quickbooks Online, there are preset templates that QBO provides to do such. It is quite a process to prepare the data and move it from an exported flat file to a format that QBO can ingest, but it works.

What if I were able to show you a better way? Read on >>

Let's start with the user experience that your customers expect. Regardless of whether you are syncing payments, invoices, employee records, contacts, opportunities - any object really - the following user experience is what your customers are looking for:

  1. Click to authenticate into external service, e.g. Quickbooks.

  2. If needed, a simple UI to adjust default data mappings to accommodate for custom data in your customer’s specific endpoint payment data configuration.

  3. Set the start date for the data if manually retrieving data from endpoint (e.g. give me all the payments that are authorized in Quickbooks in the last 2 days).

  4. Activate transfer.

Thats it, nice and automated. No need for YouTube videos to show how to pull and prep the data. You’ve successfully moved on from 1990’s technology.

Even better, you really should provide an option to automate the pull of payments at a frequency of your customer’s choosing - set it and forget it. Bam, welcome to 2018! This is what your modern customers will expect of your application - to already have this type of automation built in and ready to go.

Wait. Hold on. Right now you are thinking, “That is a fine user experience, but how can I overcome the barriers listed above and make it so that, as an MVP, my customers can sync data directly from within my application?” Answer:The Cloud Elements Platform.

Lets knock down the barriers one at a time:

First up, we support over 150 integrations, called Elements, which are super smart, ready to use connectors that have a uniform REST API interface. We make each endpoint look nearly the same from an integration perspective. We’ve done the hard work of learning the nuances of each endpoint and normalizing to a common, uniform REST API spec. You do the work once and then leverage it across many endpoints.


Not so scary now, is it?

Second, because of the smarts that are built into our Elements, much of the functionality needed to get to an MVP is already done, ready to use. For example, our Virtual Data Resources to map and transform data - easy to set up for each each endpoint allowing your app to interact with only the data that it requires. Uniform REST APIs for retrieving and posting data in real-time. Let our platform’s API resources shape the data for your customers automatically rather than manually doing so within flat files. 

Third, regarding moving data in bulk. Once again, our Elements come to the rescue. Every Element in our catalog provides uniform bulk API resources - even if the endpoint doesn’t provide them natively (like QBO). Poof, there goes the last objection on the list, moving data in quantity via APIs. 

Oh, and for those of you who really want to hit out of the park, let's automate! The Cloud Elements platform provides Formulas, which are configurable workflows that can be set to run automatically at a given frequency to sync your customers data, automatically and in near real time, with your application. Now your talking! Welcome to the next generation, welcome to 2018.

Anyone recall the hit song by Killing Joke? I think it was titled, “Living in the 80s” or something like that. Ahh, the 1980s, I think they were only like 15 years ago, right? 

Need to get started? Get your free trial today!

Enjoy. Aloha.