A SALE LIVES IN ONE SYSTEM, THE TRANSACTION IN ANOTHER
Orders arrive from the tills and from order management. Payments arrive from the payment providers. Gift cards arrive from a third party. It is hard to put the three next to each other and see the gaps.
Finance wants two questions answered. Was everything we sold actually paid? Was everything that was ordered actually delivered? And when an auditor asks what happened on a given day, HEMA wants to answer that day.
This project delivers the system that answers all three. HEMA is moving from a file sharing platform and point-to-point APIs to a data mesh, where every source publishes to a shared catalog.
So the first work is not the system. It is the sources. And the hardest of them produces several thousand events a minute, every minute the shops are open.
Build the sources first, in the order they are needed
Turnover is a finance system with a queue in front of it. We work at the front of that queue: the datasets, the ingestion, and the model turnover will read.
Sit with the other teams
The biggest problem in this project is not technical. Turnover needs data that belongs to other teams, so we joined those teams and built the datasets with them rather than filing requests.
Accept everything, validate later
A till cannot wait. API Gateway takes the request and does no validation, so the intake never builds a backlog that takes the system down with it. Validation happens further along.
Draw one write boundary
The till is only told the event was accepted once Kinesis confirms it. Past that line the event is ours, it cannot be lost, and it can be replayed.
Buffer the fast against the slow
Tills fire continuously, analytics reads in batches. Kinesis sits between the two so each runs at its own pace, and leaves room to hang a real-time consumer off the same stream later.
Publish tables, not files
Firehose lands the raw events in S3. Lambdas run Athena queries over them and produce Glue tables. Those tables are the POS dataset, and any team in the mesh can subscribe to it.
Never scan twice
Athena bills for data scanned, so the layout was designed to pinpoint only the newest data. A run touches around 15 MB instead of the whole history.
Four satellites, one catalog
Each of these used to hand its data over as files or through a bespoke API. Each now becomes a published dataset that turnover, and anyone else, can read.
POS
Sales at the till. By far the most complex of the four: several thousand events a minute, arriving whether or not anything downstream is ready for them.
OMS
Orders from the online channels. The other half of the sold side, and the source that tells finance whether an order was actually delivered.
Gift cards
Issued and redeemed value from the gift card provider. Money that moves on a different timeline from the sale it belongs to.
Payments
The payment provider, and the card schemes behind it. The paid side of the ledger, and where the anomalies turnover is meant to catch will show up.
The pipelines turnover will be built on
A POS ingestion system that does not flinch
Serverless intake sized well past what the estate produces today, with a persistence boundary the tills can trust and replay when something downstream needs rebuilding.
Published datasets in the mesh
POS, order management and gift cards as structured Glue tables in the shared catalog, available to any team that subscribes, not only to turnover.
A transformation layer built for the bill
Lambdas that run Athena over the newest partitions only. Cost was a design input from the first day, not a surprise on the first invoice.
Room for real time, unused
Analytics reads in batches, so the pipeline is near real time by choice. A real-time consumer can be attached to the same stream without touching what is there.
The turnover data model, in progress
Modelling and analysis of what finance needs to reconcile sold against paid.
Engineers who work across the teams, not around them
A finance system that reads from four other departments is an organisational problem before it is a technical one. We build in each of those teams, which is why the datasets exist.
Stress test before you promise
Ten million records through the pipeline is not the load HEMA has. It is the load we wanted proof of before saying the system scales, and it cleared in five minutes.
Run cost as an architecture decision
Athena charges by data scanned, so the partitioning was designed around that. The difference between a considered layout and a naive one is thousands of euro a year.
Say what is not finished
Turnover is not done. The foundations under it have, and the business value lands when finance starts reconciling on them.
More of our work
All cases →
Can your finance team answer what an auditor asks?
Tell us which systems hold your sales, your payments and your deliveries today. We will tell you what it takes to read them as one.