When Should I Consider Using an Integration Platform for System Integrations?

The way by which data is transferred from one source to another has changed substantially over the last decade. In the past, migrating data was more about moving information from one on-premise database to another. Connectivity was rarely a concern and databases had mature tools, techniques and plenty of engineers and database analysts who had the prerequisite skills to do the work.

Today, there are more challenges and more technical approaches to integrations than ever before.

Modern enterprise systems are often comprised of a combination of on-site legacy applications, self-hosted applications managed within a virtual private cloud, and applications that are vendor-hosted under a Software-as-a-Service agreement. The need for these systems to share data remains the same, but the techniques for doing so have evolved along with the shift towards cloud architectures.

In the cloud era, data integrations take a different form and are often best solved through the implementation of an integration platform.

What are Integration Platforms and How Do They Help?

An integration platform is an application or service designed specifically to help share data and commands between independent technology applications or systems. The types of problems they solve range from one-time data migrations to real-time, tightly coupled system integrations that share data and commands on a  frequent batch or synchronous basis. Following the trend of so many applications in the cloud era, many integration platforms are now delivered as a service and are categorized under the label of Integration Platform as a Service (iPaaS).  Many technologists run across integration platforms for the first time when they are tasked with moving data between two separate technology applications.  For example, you may be asked to migrate a large volume of data from a legacy application to a new cloud solution such as a cloud CRM like Salesforce, or an ERP system like Microsoft Dynamics.

Experienced technologists are likely requested to move data between systems when they have access to the application databases. The tricky part usually comes into play when integrations require the use of APIs as opposed to having access to the source database, which quickly removes the ability to script out your data migration with Structured Query Language (SQL) scripts. Modern-day system integrations often require a combination of techniques including database-to-database, API-to-API and API-to-database integrations. Unless the integration needs are small, or you are prepared to write a significant volume of custom code, it is common to seek out an integration platform to help with these situations.

Why Integration Platforms Are Used

Most of the integration platforms that are available are designed for rapid integration development, so that integration can be accomplished quickly. Many use drag and drop tools along with other out-of-box options that make them easy to use, particularly compared to starting from scratch manually. 

• When you only need to swap data from point A to point B for a very small database, you may prefer to write a one-time SQL script. If you are using a small app, it makes sense to develop a one-off code interface. 
• If you need ongoing integration, an integration platform will often be the most efficient solution. For example, when you want data pulled from the sales team’s CRM app and push it into the ERP system that drives order fulfillment, ongoing integration will be needed and that’s going to also require system logging and debugging capabilities when something inevitably goes wrong.
• If you require high volume migration of data and/or transformations, integration platforms will save time and avoid potential errors as they are excellent at logging and tracking automatically. You will be able to trace issues and rerun jobs more easily when using a robust platform that would otherwise take significant coding efforts.

Available Integration Platforms

Of course, there are several possible integration platforms from which to choose. Some of the most used features of integration platforms include:

• Pre-built system connectors for popular cloud software so you don’t have to build one-off integrations
• Rapid integration builders that map data sources to data destinations whether they are databases or API’s on each end
• Data transformations – in the event that one data source is structured differently than the data’s destination, then you can use either formula based transformation (for standard changes) or write more sophisticated custom transformation code as needed.  
• Run migrations in the cloud (typically used for cloud-to-cloud integrations) or you can install the applications on-site behind the firewall so the solution can access internal systems as-needed for local-to-cloud integrations.
• Logging and transaction management to monitor integrations and recover when issues arise

Below are a few integration platforms that come up in our day-to-day work:

Talend Open Studio: One of the only open-source integration platforms that can be used without paying a fee, although there is a paid upgrade available.  Powerful and easy to use, Talend Open Studio is ready to download and is quite powerful out-of-the-box if you take the time to learn the system features.  Talend is quite functional in its free capacity but can only be used for manually executed integrations.  For automated integrations, you will need to upgrade to a paid version.

Jitterbit: A classic integration platform with a significant number of available system connectors and known for high-touch customer service. Jitterbit also has a pricing model that focuses on the size of the integration, in terms of integration endpoints so it is well-suited to starting small and growing into the platform.

Mulesoft: This hybrid integration platform connects apps, data, and devices on-premises or in the cloud. It offers pre-built templates, real-time analytics, and creates a virtual private cloud. Mulesoft was acquired by Salesforce in 2018 and often comes up when considering integration to Salesforce but it is a full-features platform on its own tailored to large and complex integrations.

SQL Server Integration Services (SSIS): Integration product provided by Microsoft for data migrations and system integration, which is often included with SQL Server licenses.  SSIS has been used heavily over the years for moving data between databases but has also been evolving in recent years to include connectors to API-based systems as well. SSIS is a very capable tool that just might be free to use if you have SQL Server Standard, Enterprise, or Business Intelligence licenses.

Zapier: The vendor platform is known for providing a high-volume of connectors between popular cloud software platforms. Zapier integrations are often based on individual app-to-app integrations and are typically low-cost but not as robust as some of the other solutions in the list. It is an excellent first choice for low-budget and light integrations.

There are also industry-specific integration tools available, such as Bridge Connector for healthcare integration, Blendr.io for accounting integration, and HighQ for the legal industry.

In Summary

If you are just looking for a one-time migration from one database to another, it may be more efficient to write out SQL scripts that can move the data between systems.

If you have high volume or more complicated integration, you would likely benefit, particularly over time, to invest in an integration platform. Prebuilt connectors will be able to save upfront implementation time in many scenarios, and you will likely find the interface maintenance, debugging, and onboarding costs of new support resources over time to be lower than a custom interface, which helps offset the operational cost of licensing the tool. For larger enterprises with high volume and complexity integrations, a good integration platform is almost a necessity.

The bottom line is if you are trying to connect cloud-to-cloud or database-to-cloud or need a live integration that runs on an ongoing basis, you should definitely consider the use of an integration platform.

However, before buying an integration platform or developing an integration sequence start by making sure you know exactly what you need in advance. Check out our Guide to the Discovery Process – a downloadable eBook that may help you in your journey.

Thayer Tate

Thayer is the Chief Technology Officer at SOLTECH, bringing over 20 years of experience in technology and consulting to his role. Throughout his career, Thayer has focused on successfully implementing and delivering projects of all sizes. He began his journey in the technology industry with renowned consulting firms like PricewaterhouseCoopers and IBM, where he gained valuable insights into handling complex challenges faced by large enterprises and developed detailed implementation methodologies.

Thayer’s expertise expanded as he obtained his Project Management Professional (PMP) certification and joined SOLTECH, an Atlanta-based technology firm specializing in custom software development, Technology Consulting and IT staffing. During his tenure at SOLTECH, Thayer honed his skills by managing the design and development of numerous projects, eventually assuming executive responsibility for leading the technical direction of SOLTECH’s software solutions.

As a thought leader and industry expert, Thayer writes articles on technology strategy and planning, software development, project implementation, and technology integration. Thayer’s aim is to empower readers with practical insights and actionable advice based on his extensive experience.