by Rosario Cannavò & Tommaso Nocchi, Big Data Engineers, Onetag
Real-time bidding (RTB) is one of the most demanding environments in technology. Every time a web page loads with an empty ad slot, a sealed-bid auction fires and closes in under 250 milliseconds. Dozens of advertisers value the request, predict the outcome, set a price, and the winning creative appears before the page has finished rendering.
You never notice it. That’s the point.
At Onetag, this invisible auction is our entire business. Running it at global scale means living with numbers that leave little room for error: more than two million requests per second to value and respond to, around fifteen million events per second collected to optimise bidding, and roughly 170 billion auctions processed every single day, across publishers and buyers in North America, LATAM, EMEA, and APAC.
At this scale and speed, the infrastructure has one job: absorb unpredictable spikes, process enormous volumes of data instantly, and never miss a single auction. Over time, we’ve learned that meeting that bar isn’t about a single clever component. It comes from getting one principle right and building everything around it. Or, as we like to say:
“Independent at the edge. Centralised in the data.”
A quick word on scope.What we want to focus on here is narrower and, in our view, the hard part: how the bidding nodes themselves interact with our centralised datastores. At two million requests per second, those interactions are where inconsistency, latency, and contention would normally creep in. This is the layer that has to make them disappear.
The central tension: decentralised bidding, a single source of truth
Our bidding runs on five clusters across three regions. Proximity to the source cuts network latency, and every millisecond we save goes straight back into the bidding logic, giving us more time to evaluate signals, apply our models, and make smarter decisions before the auction closes.
Bidding has to stay decentralised at the edge. But the data those clusters generate, and the models trained on it, cannot live in isolation. Writes, aggregations, and model training all need a single, consistent source of truth. Reconciling those two opposing forces is the real architectural challenge.
Our answer is an ingestion layer that sits between the clusters and that source of truth, funnelling events from every cluster inbound to a single point for storage and computation, then fanning the resulting data and models back out to each point of presence.
For a long time, that layer was a single monolith: streaming ingestion and batch ingestion under one roof. It was hard to evolve, offered no isolation, and shared failure modes across responsibilities that had nothing to do with each other. So we split it by latency profile, by responsibility, and by failure mode, into two specialised layers.
The streaming layer: bid and forget
The streaming layer is the decoupling buffer between bidding and our datastores. Bidding nodes must write fire-and-forget, with no synchronous round-trips during an auction. Downstream faults must stay invisible to bidding. And sudden traffic bursts have to be absorbed without propagating backpressure upstream.
Amazon Kinesis is the backbone here. Bidding nodes write into a sharded, ordered stream and move on; consumers lease and checkpoint their progress, and apply the writes to the cache and database at their own pace. If a consumer goes down, the data sits safely in the stream, replicated across availability zones, and we recover without losing anything and without bidding ever noticing.
We manage consumption state explicitly with the Kinesis Client Library backed by DynamoDB. We know where we are per shard, restarts are safe, scaling happens automatically through lease balancing, and our aggregation, deduplication, and enrichment logic lives inside the application itself.
The batch layer: from opaque monolith to event-driven pipeline
Not all data moves the same way. Streaming data is continuous and incremental, while batch data, including statistical models, ETL output, and configuration maps, tends to arrive as a complete snapshot. As the business grew, our earlier batch flow began to show three limitations that matter more in an RTB context than elsewhere.
The first was resilience: without a buffer, backpressure, or structured retry, a pipeline error could leave downstream consumers waiting, showing up as less accurate budgets and spending. The second was data variety: each new dataset tended to bring its own parsing code, gradually tightening the coupling between ingestion and business logic. The third was visibility: ingestion ran on a time-based schedule, with limited awareness of whether the upstream job had actually finished.
We rebuilt the layer as an event-driven pipeline. Managed Airflow orchestrates everything as DAGs with explicit dependencies, per-step retry, and full visibility in the UI. SQS triggers ingestion the instant the upstream stage writes its output, with one message per cluster letting clusters process in parallel. And the piece that quietly changed the most for us is the engine underneath it: DuckDB.
DuckDB is an in-process analytical database. Think of it as SQLite for analytics. It lives inside the ingestion application as a library, reads CSV, JSON, and Parquet natively from S3 through plain SQL, and works schema-on-read, so there’s no upfront DDL and it copes with whatever actually arrives. We spin up an isolated instance per run, with no shared cache and no residual state between executions, so one job can never contaminate the next. Because it runs in-process, there’s no network hop between the query engine and the application. The performance comes for free.
The practical result: adding support for a new input format is zero lines of code. Just SQL.
Observability closes the loop. Each run writes a status file to S3, and if the last successful run is older than a threshold, the system recognises the anomalous state and runs a recovery pipeline automatically, with no human intervention.
The outcomes
Pulling one monolith apart into two purpose-built layers changed how the whole system behaves. It became more reactive, because data is ingested the moment it’s available rather than on an artificial, time-based cadence. It became more transparent, because every failure is immediately visible in the orchestrator. It went from a monolith to modules: two independent layers forming one coherent system, each able to evolve without breaking the other. And it became resilient under burst, with spikes absorbed by the streaming and queueing layers instead of cascading back into bidding.
That last point matters most, because the entire reason for this work is growth: more publishers, more buyers, more regions, more auctions every day. An architecture that absorbs that growth, rather than cracking under it, is the difference between scaling and stalling.
Final thoughts
Operational excellence in RTB is rarely the result of a single breakthrough. It comes from getting your core principle right, then having the discipline to split responsibilities cleanly, choose the right tool for each flow, and make the system honest about its own state.
If you’re building or operating distributed systems at scale, the most useful thing we can pass on is this: separate your concerns by failure mode before you’re forced to, and let events, not timers, drive your pipelines. The auctions stay invisible either way. The infrastructure behind them just gets a lot harder to break.
Originally Published on: LinkedIn
