In a previous post, I talked about the post-deployment processes that we practice at Dispatch and observe with all of our projects.
As a quick refresher – the post-deployment phase of our projects is what we consider “Empowerment.” Everything we do in this phase is preparing our clients for “life in production.” This phase typically has four stages:
- Go-live (hours to days)
- Shakedown (30-day Hypercare)
- Stabilization (30-60 days)
- Monitor (60 days+)
Throughout these stages, Dispatch supports you through the Empowerment phase with Hypercare.
Hypercare
All professional services organizations offer go-live and post-go-live support, which is often referred to as “Hypercare.” In many cases, this ends up being reactive support. You (as a customer) see a problem and call the vendor for support. For some period of time (often 30 days), the vendor commits to respond (eventually) to solve the problem. In some cases, you file a ticket in a support desk. Sometimes the project manager will remain involved and help orchestrate a fix. And hopefully the developer who built your solution hasn’t been assigned to another project.
Because most of us at Dispatch came from the client-side, we have all lived through situations where we haven’t felt supported after project go-live. We understand how important the Empowerment phase of a project is, and our approach at Dispatch is entirely informed by these experiences. Rather than being passive and reactive to issues, our philosophy is to remain deeply involved alongside our clients during this phase. We strive to be a reliable partner who will respond quickly to address any issues that emerge in production. But more than just a fast responder, we also deploy solutions that help us proactively detect defects before they become problems. In many cases we can discover a potential issue and have a fix ready before our clients have noticed anything amiss.
Integration Failure
Integrations are notoriously difficult to manage when in production. The best ones are invisible – you don’t think about them, you don’t see them, they just work in the background making your life easier.
Integrations can also be incredibly frustrating when they fail us. They can fail silently, and we often find out about the failure after too much time has passed and damage has been done. Our recent post on Failure Modes and Effects Analysis (FMEA) discusses how we try to anticipate and mitigate potential failure modes as part of our solution design processes. Nevertheless, unanticipated errors that were never tested for can and do emerge in the messy real world of production. And these most often will occur immediately after go-live.
Sometimes failures are due to complex edge-cases that were never anticipated. Sometimes the root causes of failures are simpler (and more embarrassing) to explain…for instance if a new integration was inadvertently turned off.
Figure 1: True story 1…this chart shows a specific integration’s activity and data flow. It worked fine, until someone in production support deactivated it on July 6.
Sometimes an integration doesn’t deliver data correctly but doesn’t actually “fail”, and therefore no automated alert is sent to tell someone that something’s wrong.
Figure 2: True story 2…this chart shows an onboarding integration that was working (green bars), then failed (red bars). The failures weren’t noticed for days until the Director of HR called IT to complain that new employees weren’t being set up in Active Directory.
Sentinel by Dispatch
To help us support our customers during Hypercare, Dispatch developed Sentinel, a tool specifically designed for Workday integrations that shines a light onto the integration production lifecycle. Sentinel is fundamental in powering our Hypercare production support period and we are now proud to share this tool with the rest of the world.
You no longer need to wonder – did your job requisition gets created in the ATS, did the payroll bank file get successfully uploaded to the bank? Did the integration complete successfully, or were there errors that could impact the business? Sentinel monitors all integration activity and can inform you about all successful integration events, and more importantly can alert you when integrations fail.
Sentinel is an essential tool for our Workday integration Hypercare process that enables us to monitor the newly live integration continuously and automatically.
- We use the Sentinel daily digest to keep an eye on processing cycles. It’s delivered via email twice a day – once at 6am ET and once at 3pm ET. We check the digest at the start of the day to see if there were any problems overnight, and once again before the end of the day.
In addition to the daily digest, Sentinel alerts us immediately if there are critical failures – via email, Slack, or text. This ensures we are aware of a failure seconds after it occurs, so we can take immediate action.
- If we see a problem, we go directly to the Sentinel dashboard to get the context for the problem. In this case, we can see that it failed 8 times since 7pm UTC. Not great!
- Sentinel lets us drill down further to determine if this is a transient issue or if a deeper analysis is needed. In this case, we can see that there is an error after retrieving the candidate from the Applicant Tracking System and more review is needed.
- We can now drill down directly into the event where this problem was found. Without Sentinel, integration problem diagnosis using the built-in Workday Process Monitor or Integration Events screens would have required clicking into many different events to find the one we are interested in. Using Sentinel, the time from arrival of the digest to diagnosis – about 60 seconds.
In this real-life example, we were able to find the defect (a problem on a launch parameter), determine a solution, and inform the customer in less than an hour.
A great thing about Sentinel is that it can monitor any Workday integration – regardless of who built it or when it was built. This means it is great not just as a Hypercare tool, but as an ongoing integration production monitoring system.
Sentinel can be set up to monitor any Workday environment in just a few hours. Out of the box, Sentinel can send twice-daily integration digest emails to give you a full picture of all integration activity that occurred over the last twelve hours. You get access to a comprehensive dashboard that helps provide history and context for integration activity. Most importantly, Sentinel generates centrally managed alert emails if issues do occur so the right people can be alerted right away if there are potential problems.
Sentinel can also be connected to your existing helpdesk system to auto-generate tickets with the right level of urgency for each type of integration and failure mode.
Here is an example ticket in our helpdesk system:
We find Sentinel’s automated monitoring and alerting capabilities ensure we can provide proactive Hypercare support and that no integrations ever fail silently. Many of our clients elect to continue to use Sentinel themselves after Hypercare is over so they can have ongoing visibility of integration health for all their Workday integrations.
If you manage Workday integrations and you would like a better way to monitor and respond to integration errors and failures, please reach out to us and request a free demo of Sentinel today!
About Dispatch Integration:
Dispatch Integration is a software development and professional services firm that develops, delivers, and manages advanced data integration and workflow automation solutions. We exist to help organizations effectively deal with the complex and ever-changing need to integrate data and optimize end to end workflows between cloud-based, mission-critical applications.
Read More from Dispatch Integration:
The Role of Project Managers in Software Integration Projects
The Role of a Business Analyst in an Integration Project
Managing Complex, Security, and Reliability with Your Workday Integrations
Gavin Hay is the co-founder, President, and CTO of Dispatch Integration with experience leading high performing cross-functional teams. He has over 20 years of experience as a systems architect in the HR and Payroll industry and has a deep understanding of the full stack technology infrastructure required to deliver exceptional software integrations.