Context
A title and property valuation services company supporting lenders, for example confirming a property’s value before a bank funds a loan. Several lines of business each worked with their own vendor network, and each had its own operational needs. The work came early in the move to the cloud, before Kubernetes became the usual way to host services, so the services ran on Azure Service Fabric.
The problem
Requests arrived from many sources, and each vendor could use its own message contract. An earlier on-premises effort, built on web services and BizTalk Server, wired each connection specifically to its endpoint and captured far more data than the work needed, duplicating what other systems already held. Overnight messages took hours of manual work each morning to push through, and every new vendor or contract meant another bespoke path.
Constraints
- Accepting a message could never make a sender wait on anything downstream, and nothing a vendor sent could be lost.
- Messages could arrive more than once, through redelivery or a resend, and had to take effect only once.
- Each vendor could send its own message contract.
- The rules for each line of business belonged to the systems that line of business owned.
- Requesters needed to know their information had arrived, and every later message had to correlate back to it.
Approach
A hub and spoke. Accept each message immediately, write it unchanged to blob storage, and do everything else asynchronously over Azure Service Bus. A first step classifies the message, from the endpoint it arrived on or metadata on the HTTP request, without touching the stored original. A second step routes a pointer to that message to the line of business it belongs to. There, an adapter translates it into the format the receiving system expects, sends receipts back to the requester on that system’s behalf, and wires in correlation: an alias system maps each party’s identifiers, so a receiving system can trust the pipeline’s ID and attach its own for every later message. Each internal system was treated as an integration in its own right, with what it required from a message written as a specification inside its adapter rather than in the shared path. The pipeline kept only what it needed to route and correlate, not a copy of each system’s data.
Key engineering decisions
Accept immediately, store first
Accepting a message only stores it and returns, so senders never wait on anything downstream and a failing step later can never lose what they sent. Classification and routing then work from a pointer to the stored original, which stays unchanged until it reaches the adapter that understands it.
Idempotent at every step
Service Bus can deliver a message more than once, and a vendor can resend one. Every step was safe to repeat, so a redelivered or duplicate message never did its work twice.
A loose envelope, strict consumers
Messages and their metadata travel in an abstract shape with few constraints. Each consuming system asserts the values it requires and, when some are missing, says which, so the information can be requested.
Pause delivery, never intake
Outgoing delivery could be switched off while messages kept arriving and being stored. A fix was deployed, delivery switched back on, and the held messages flowed through, so code could be patched during the working day without losing anything.
Result
A handful of services replaced bespoke point-to-point integration, and the mornings spent pushing overnight messages through by hand. Processing became consistent enough for daily code pushes, with fixes deployed during the day while messages kept arriving. Supporting another vendor contract or line of business meant adding an adapter, not reworking the pipeline.