Have you cursed when it takes ages to get a welcome e-mail or a password reset? Have you wondered why it doesn’t happen right after you click “Ok”? There’s a lesson in patience there (maybe) but likely the reason is related to the process not being event-based OR part of the process not being event-based (typical of banking). And, chances are, your ‘welcome’ (or ‘cash’ transfer) is held in a queue, subject to certain checks – i.e. there’s more going on than you can see, even when what passes-through sometimes apparently immediately, is ‘delayed’ at other times.
In an ‘Event’-based integration—Integration being subject of this series of articles, something (when you click “Ok”) triggers a request to be sent ‘downstream’ (you might think of your browser sending something up to the server – but we’re framing this illustration as sending downstream). An event can be sent at the end of a process (e.g. click “Ok” to ask for your password to be reset) but can also happen part-way through a process (to start another process e.g. check an address is valid in the postal system). What’s sent downstream: all your details, some details, only the detail that changed, is the purpose of change data capture (“CDC”) – as a design principle or software implementation.
Imagine someone fills-in a user (web form. CRM is interested in the person’s name and e-mail, while other systems are interested in other parts of the user record. Name and e-mail, collected first, are used to create (and part-fill) the record. Details are sent to the CRM; hopefully the CRM is event-based, and welcome e-mail gets sent immediately. All sorts of things can crop-up that prevent this from happening right away, and we’ll look at some of these things shortly.
For CDC the user record has gone from not existing to existing. CDC triggers an event downstream ‘saying’ “this is a new record, do with it what you will!” CRM has what it needs (for the welcome e-mail) but other processes wait (have to wait) for more information to be collected, which might get entered later, or might never get entered at all.
Perhaps that someone fills a shopping basket (anonymously) and the basket should be tied to the user, somehow. It could be a reference from the basket to the user, or from the user to the basket, or, why not, both? Often the relationship isn’t bi-directional because it adds effort to de-couple the user and the basket when a downstream system requires one but not both. Let’s tie the basket to the user right away, because we don’t want an abandoned basket—because it gives us more to clean up later, although modern basket software tracks partially-complete processes and sends reminder e-mails encouraging the customer to complete their purchase. And why not? Since the software already checks for abandoned baskets on a timer (e.g. every minute, every hour, etc.) to do the clean-up, so it may as well take the opportunity to send a reminder while it’s as it.
As the user fills their basket, other events are triggered; possibly by CDC software too – the basket ‘changed’ i.e. an item was added (or removed). Adding an item requires inventory level checks; and let’s hope this happens in parallel and as a synchronous request-reply (so we know right away whether there’s stock or not) – because not knowing isn’t good customer experience – when the basket only gets checked at checkout (and the system finds that some items aren’t in stock). Some systems are designed to check stock at checkout, because the designer thought “I have to check at checkout so it’s pointless checking as we go along”, but when you put yourself in the customer’s shoes, knowing an item isn’t in stock AS YOU GO, is far more stress-free than finding out at the checkout, and abandoning the basket. Plus, in a fully-‘integrated’ event-based system, out-of-stock ‘events’ can be used to trigger the inventory system to replenish stock. There’s an increase in complexity, but there’s also an increase in customer satisfaction, and “If the customer could have ordered the item, would they have?” This proves demand and shapes supply (managing stock levels).
The customer goes to check-out. They provide their address (for shipping) and the details are added to their user record. CDC sends the address only (because this is the only change to the user record to fulfillment – shipping charges (based on shipping options) are calculated, credit card details are captured and added to the record, or shared with a third party and payment reference added to the record.
There are many processes (and side-processes) running in parallel, but limited resources available to the whole system (CRM, Inventory, Fulfillment, and Payment system), and even when we design integrations as ‘event’-based there can still be delays getting the e-mail sent, stock level checked, and payment processed.
I’ve been involved with dozens of these designs and involved in lots of discussions. Here’s where one of them landed, followed by thoughts for improvement –
“Let’s use CDC to detect change and Pub/Sub to send downstream!” For each system and record type upstream we’ll have a publisher (CDC will trigger the publisher to request details from where the change was recorded, or CDC will pass details of the change to the publisher) and for downstream systems (interested in part-or-all of the record, we’ll have a subscriber. We’ll send properties with the details being published, the pub/sub system can use to route the (whole) user record to whichever subscribers are interested. The subscribers will ‘copy-and-paste’ the specific detail/s downstream needs, in a shape-and-form the downstream system ‘understands’ (because almost every system we’ve ever built expects different data structure). We’ll go into more depth of pub/sub rationale in the next article.
It sounds convoluted but we’ve successfully ‘decoupled’ upstream from downstream, meaning we can change one system without taking the whole system down for maintenance, and (assuming we have enough compute resource) we can scale publishers and subscribers independently so changes aren’t held-up upstream from being sent downstream e.g. stock levels are being checked more frequently than payments being taken, or stock level is more resource intensive process. ‘Event’-based pub/sub integration is often (not always) used to make sure all processes are run in an efficient manner, and nothing’s kept waiting at the hand-off point between upstream and downstream. It can run like a well-oiled machine.
You may recognize what’s described here as a service bus, or service mesh (yes, there’s differences between the two) which here we’ll simply call our “broker”. We’ve centralized management and operations; and instead of hard-wiring systems (generally not good practice) to each other (in what resembles a spider web) we wire systems through a “broker” (in what resembles a bicycle wheel). And we pretend the complexity of the spider web has vanished (when we draw it on a slide) but truth be told, we just moved it inside the broker. We built a team, we built processes, and we built infrastructure, so we could deploy business support system / operational support system integration logic inside the broker, instead of deploying our broker inside our support systems (excluding, for a moment, our broker as a support system).
Deploying (embedding) a broker inside a business/operational support system, is no small task. Except, if the broker’s deployed inside our system, and the broker breaks down, it could take the whole support system with it – although this risk can be mitigated with robust coding and appropriate consideration to isolating the impact.
With Auxillery™ you don’t have to build infrastructure external to your support systems. We believe (with sufficient What-If systems thinking) the need for any system to speak every ‘languages’ of every other system it integrates with (which is the core precept of a broker) is entirely unnecessary. Except, with so many systems out there and so many different ‘languages’ (internal data representations) and so many ways to connect them (e.g. web, file, queue, database), it’s a problem that’s more easily said than solved, if it could indeed be solved at all. As it happens, we think it’s worth solving, and we haven’t just started thinking about it, we’ve already solved the big part of the problem, already.
Deploy Auxillery alongside your support system – embedded module written in the same programming language as your system, or on the same infrastructure (directly and securely connected within the same VPN) – and all your systems can communicate with one API (our Rest-In-Peace API) and one abstracted uniform SDK (to share data between systems and kick-off processes across systems).
