Enterprise software and IT infrastructures have evolved significantly over the past several years. Cloud-based architectures for mission-critical applications are now the standard. And many companies use a best-of-breed approach rather than licensing all their software from one provider.

For instance, in the HR system domain, an enterprise might use a SaaS offering such as Workday as its core HR software but may opt for a recruiting platform like Greenhouse, a time-tracking package like Kronos, a third-party payroll system, and dozens of other solutions.

The Data Integration Challenge

It is not unusual for large companies to have dozens or hundreds of applications running in the cloud, each producing, consuming and transforming data from other systems thousands of times daily.

These applications aren’t owned by the enterprise or installed on-premises. Only the data belongs to the company, and it flows between the applications over the open Internet to create a virtual “nervous system.” This nervous system needs to be seamlessly and reliably integrated for business operations to function.

It’s not enough for each of these disparate systems to work flawlessly and securely unto themselves – they must also flawlessly and securely exchange data, potentially in various formats and often in real-time.

At one of our clients that uses Workday, there are 190 integration points with other third-party applications and services, and these integrations exchange data over 40,000 times daily. This is just in HR, with even more data flowing in finance, operations, sales, and other functions.

Managing integrations is a new and critical IT competence. SaaS vendors are responsible for their own applications and will typically provide ways to get data to the “edge” of their system through application program interfaces (APIs). They don’t take responsibility for the data once it leaves their application, and certainly don’t take responsibility to ensure data arrives at its destination or in the right format. Therefore, the responsibility for integration portfolio management must be owned by IT.

The Challenges with Integration Portfolio Management

This challenge of managing data integrations is akin to when you check baggage at the airport. You hope your bag gets on your plane, is handled correctly, isn’t picked up by someone else, things aren’t stolen or added, and arrives undamaged at the other end. What happens when your bags don’t arrive when you get to your destination? Your tag says it’s supposed to be there, but it isn’t. Now what? Why did the system fail? Is it the departure airport’s fault? The airline? The arrival airport? Where did the bag get lost? How will it be retrieved? How long before I get it back, if ever?

There are a number of things that can go wrong, any of which can cause significant business disruption.

• Poorly Designed  API’s
  Most software vendors offer Application Programming Interfaces (API’s). But not all API’s are created equal. Some are better at integrating with other applications than others, more feature-rich than others, well (or poorly) documented, unexpectedly deprecated or incomplete. In short, the quality of API’s is inconsistent, requires technical skills to interpret, and can impact integration quality.
• Loss of Data
  Poorly constructed integrations can cause them to fail intermittently and unpredictably. When integrations fail silently, business processes can be significantly impacted. What happens when data does not make it from one application to another? How do you know? Is there adequate monitoring to confirm that the right data arrived? If there is a problem, how do you determine the root cause? Is it an ongoing or intermittent problem?
• The Wrong Data
  In many systems (often where humans are involved in the integration process), it is easy to send or receive incorrect data without it being flagged before it’s too late. For example, the wrong payroll file accidentally uploaded to an integration system to be sent to a payroll processor can have unforeseen and adverse consequences.
• Compromised Security
  Businesses send their most confidential data through integrations. If security is not well designed, it can create significant risk. In particular, Personally Identifiable Information (PII) and Private Health Information (PHI) are data that have additional regulatory security requirements (such as GDPR in Europe, PIPEDA in Canada, and HIPAA in the U.S.A.). Sending PII or PHI through unsecured or unencrypted integration methods can result in breaches of trust and compromised goodwill, along with enormous fines and penalties.
• Data Conversion
  Data from one application must often be converted in type and format to integrate with other applications. Integrations typically don’t simply move data from one application to another – they translate, enrich, combine or transform it into other data structures. Understanding the data models of each application and how to create these translations requires specialized skills to design and test.
• Transient Problems
  Transient events can take down one or more applications and temporarily disrupt communications. If mission-critical data is in the middle of flowing between applications, being transmitted or received during the transient event, what happens? This creates the need for integrations to be fault-tolerant with guaranteed delivery and data integrity.
• Lack of Monitoring to Flag Problems
  Integration monitoring is required to flag problems immediately to minimize errors or disruption to crucial business operations.
• No Error Checking Mechanisms
  API’s might do basic error checking, but typically don’t include business-quality data validation checking mechanisms to ensure the data being received is valid and meets predefined criteria.
• Integration Lifecycle Management
  If a company has hundreds of cloud-based applications, you can bet that at any point in time, one or more of them are being upgraded, reconfigured, or replaced. These changes can affect downstream applications if something changes in the data type or format sent through integrations to other systems. Inadequate system-level regression testing and data quality assurance can cause unforeseen problems.

Modern Integration Platforms Can Help

Most enterprises now use Integration Platforms as a Service (iPaaS) systems to build, manage, and govern their integration portfolio. Modern platforms such as Workato have been game changers in enabling IT to take control of their integrations and ensure they are secure, reliable, and efficient. 

These platforms are rapidly becoming essential, especially as companies look to fully automate workflows, including leveraging Agentic AI approaches within their business processes. 

At Dispatch, we help companies leverage platforms like Workato to fully unlock the value of their app ecosystems. We act as a partner with our clients to design, build, and manage integration portfolios, and deliver advanced workflow automation solutions that orchestrate systems, data and people.  These solutions improve efficiency, improve scalability, and enable business process innovation that can be a competitive advantage.

To find out more about how we can help you take your integration strategy to the next level, Contact us today.