Design an Ad Click Aggregator — System Design Walkthrough

"Design an ad click aggregator" is the canonical streaming-analytics interview question. On the surface it is just counting: a user clicks an ad, we increment a number. What makes it a senior-level problem is that the number has two jobs at once — it powers a dashboard an advertiser refreshes every few seconds, and it is the basis for an invoice. Fast and approximate serves the first; slow and exact serves the second, and reconciling the two is the real design. Below is a walkthrough you can mirror in a real system design interview. Keep the system design interview cheat sheet open for the shared building blocks.

1. Clarify the requirements

Functional requirements

• Ingest click events at very high volume from browsers, apps, and ad servers worldwide.
• Aggregate counts by ad, campaign, advertiser, and dimensions like country and device, over time windows (per minute, per hour, per day).
• Serve near-real-time dashboards: "clicks on campaign X in the last 5 minutes."
• Produce accurate, auditable totals for billing.
• Support arbitrary slice-and-dice queries over a retention window.

Non-functional requirements

• Write-heavy and bursty: ingestion must never block on aggregation.
• No data loss: a dropped click is lost revenue, so the ingestion path must be durable.
• Low dashboard latency: seconds, not minutes.
• Correctness for billing: the number on the invoice must be defensible and reproducible.

Naming that last split — two consumers with different tolerances for approximation — is the move that frames the whole answer.

2. The ingestion path

Clicks arrive at edge collectors: stateless services deployed close to users. Their only job is to validate minimally, stamp the event, and write it to a durable stream. No aggregation, lookups, or fraud scoring inline — anything slow at the edge becomes lost clicks during a spike.

The event itself is small and should carry an id generated by the client at click time:

{ click_id, ad_id, campaign_id, user_or_device_id,
  event_ts,          // when the click happened (event time)
  ingest_ts,         // when we received it (processing time)
  country, device, referrer, ip_hash }

Collectors append to a durable partitioned log — the Kafka-style abstraction, with a replicated, append-only, replayable commit log partitioned by key. Three properties matter here. It decouples ingestion rate from processing rate, absorbing bursts as backlog instead of errors. It is replayable, so a bug in the processor can be fixed and history reprocessed. And it partitions, which is what lets aggregation scale out. Partition by campaign_id so all events for a campaign land on one consumer and can be counted with local state.

3. Windowed stream processing

A stream processor (Flink, Spark Structured Streaming, or similar) consumes partitions and maintains running counts per key per window.

• Tumbling windows are fixed and non-overlapping — 12:00:00–12:00:59, then 12:01:00–12:01:59. Every event belongs to exactly one window. This is the right shape for billing buckets and for the minute-level rollups the OLAP store stores.
• Sliding (hopping) windows overlap — a 5-minute window advancing every 1 minute. An event lands in several windows at once. This is what powers a smooth "last 5 minutes" dashboard line that updates every minute.

A practical pattern: compute minute-level tumbling aggregates once, then derive hourly, daily, and sliding views by summing minute buckets rather than maintaining every granularity independently.

4. Event time vs processing time, and watermarks

This is where the question separates candidates.

Processing time is when the pipeline handles a record. Event time is when the click actually happened. They diverge constantly: mobile clients flush buffers later, a region's network hiccups, a consumer falls behind and catches up in a rush. Bucketing by processing time is trivial but produces numbers that change with how fast your own pipeline happened to be running — unacceptable for billing. So aggregate by event time.

That raises the completeness problem: if events can be late, when is the 12:00 window done? You cannot wait forever. The answer is a watermark — the processor's assertion that it no longer expects events with an event time earlier than some point. When the watermark passes the end of a window, the window fires.

The watermark is a tunable trade-off, and saying so out loud is the point:

Events arriving after the watermark go to a late-data path: either discarded (with a metric so you know how much), or routed to a side output that emits a correction to the published window. Corrections after billing has run are painful, which is another reason the authoritative number comes from batch.

5. Lambda vs kappa: the fast path and the source of truth

Because dashboards and invoices have different requirements, the standard answer runs both.

• Speed layer (streaming). Windowed counts within seconds, published to the serving store. Approximate: it may miss very late events, and may double-count in a rare replay. Perfect for dashboards, budget pacing, and anomaly alerts.
• Batch layer. On a schedule, a batch job reprocesses the raw event log from durable storage for a closed period — deduplicating exhaustively, applying the full fraud rules, and including every straggler. Its output overwrites the streaming estimate for that period and is what billing reads.

That is the classic lambda shape, and its known cost is maintaining two implementations of the same logic that can drift apart. Kappa is the alternative: keep only the streaming path and get "batch" by replaying the log through the same code with a longer watermark. Kappa removes the duplicate-logic problem; lambda gives you a genuinely independent recomputation, worth something when the output is an invoice.

Either way, the discipline that matters is reconciliation: continuously compare streaming and authoritative totals for closed windows and alert when the gap exceeds a threshold. That gap is your best early detector of a broken processor or a partition that silently stopped consuming. When money is the output, a monitored delta between two paths is worth more than either number alone.

6. High-level architecture

  Browsers / apps / ad servers
            |  click events (client-generated click_id)
            v
    [ Edge collectors ]  stateless, global, validate + stamp only
            |
            v
    [ Durable partitioned log ]  replicated, replayable
        partitioned by campaign_id
            |
     +------+---------------------------+
     |                                  |
     v  SPEED LAYER                     v  BATCH LAYER
 [ Stream processor ]              [ Raw event archive ]
  dedup (windowed)                  object storage
  fraud filter (fast rules)               |
  event-time tumbling windows             v
  watermarks + late-data path       [ Batch recompute ]
     |                               full dedup + fraud
     |  approximate, seconds          authoritative
     +--------------+------------------------+
                    v
             [ OLAP store ]  columnar, pre-rollups
                    |          batch overwrites speed
        +-----------+-----------+
        v                       v
   Dashboards            Billing / invoices
   (near real time)      (source of truth)

   Reconciliation job: |speed - batch| per closed window -> alert

7. Deduplication and idempotency

Clients retry on flaky networks, collectors retry into the log, and consumers may reprocess after a restart. Without care, one click becomes three and an advertiser is billed for phantom traffic.

The fix starts at the source: the client generates click_id at click time and reuses it on every retry. An id assigned at ingestion is useless — each retry would get a fresh one. The streaming processor keeps a set of recently seen ids, scoped to its partition and bounded by a time window so state does not grow without limit, and drops repeats. That bound makes streaming dedup best-effort for very delayed duplicates; the batch job, unconstrained by latency, deduplicates across the entire raw log — the same at-least-once-plus-idempotency reasoning that governs a distributed job scheduler.

8. Fraud filtering, skew, and the serving store

Click fraud filtering. Not every click is a human with intent. Cheap deterministic rules run in the stream — known bot user agents and data-center IP ranges, implausible click rates from one device, clicks arriving impossibly fast after impression — while heavier behavioural and ML-based scoring runs offline in the batch layer with full session context. Filtered clicks should be recorded rather than deleted, so totals can be explained to advertisers.

Hot partitions. Partitioning by campaign_id means one enormous campaign can send far more traffic to a single partition than it can absorb while others idle. Detect it with per-key volume metrics, then apply key salting: append a small random suffix so the hot campaign spreads across several partitions, aggregate the sub-keys in parallel, and sum the partials in a second stage. Salt only the keys identified as hot. Smoothing pathological sources with a rate limiter at the edge is a complementary lever.

The OLAP store. Aggregates land in a columnar analytical store (Druid, ClickHouse and Pinot are common examples) rather than a row-oriented transactional database. Analytical queries scan a few columns across enormous numbers of rows — exactly what columnar storage, compression, and time-partitioned segments are built for. Store pre-rolled minute buckets, let the store aggregate upward at query time, and keep raw events in cheap object storage for replay and audit.

Observability. Track consumer lag per partition, watermark lag, late-event rate, filtered-as-fraud rate, and the reconciliation delta. Consumer lag tells you the pipeline is behind; the reconciliation delta tells you it is wrong, which is worse.

FAQ

What is the difference between event time and processing time?

Event time is the timestamp at which the click actually happened on the user's device. Processing time is the moment the pipeline happens to handle that record. The two differ because of network delay, retries, backlogs, and offline mobile clients that buffer events. Aggregating by processing time is easy but produces numbers that shift depending on how fast the pipeline is running, so correct billing and reporting aggregate by event time and accept that a window may need to wait for stragglers.

What is a watermark in stream processing?

A watermark is the stream processor's assertion that it does not expect to see any more events with an event time earlier than a given point. It is how the system decides a time window is complete enough to emit a result, because otherwise a window could stay open forever waiting for stragglers. Watermarks trade completeness against latency: an aggressive watermark emits results sooner but drops more late data, while a conservative one waits longer and captures more. Events arriving after the watermark are either discarded or routed to a late-data path that corrects the earlier result.

Why have both a real-time path and a batch path for ad clicks?

Dashboards and pacing decisions need numbers within seconds and can tolerate small inaccuracies, while billing has to be exact and auditable. A fast streaming path produces approximate counts for near-real-time views, and a slower batch job periodically recomputes the same windows from the raw event log to produce the authoritative figures that invoices are generated from. Because money is involved, the batch result is treated as the source of truth and reconciliation differences between the two paths are monitored as a signal that something in the streaming path is broken.

How do you deduplicate ad clicks?

Clients retry on flaky networks, so the same click can arrive more than once. The fix is to have the client generate a unique click id at the moment of the click and send that same id on every retry, rather than assigning an id at ingestion. The stream processor then keeps a set of recently seen ids, keyed by partition and bounded by a time window so the state does not grow without limit, and drops repeats. The batch path can deduplicate exhaustively over the whole raw log because it is not latency constrained.

How do you handle a hot campaign that overwhelms one partition?

Partitioning by campaign id keeps each campaign's counts on one processor, but a single very large campaign can then send far more traffic to one partition than the others can absorb. The usual fix is key salting: append a small random suffix to the key so the campaign spreads across several partitions, aggregate each sub-key separately, then sum the partial counts in a second stage. Detecting skew with per-key volume metrics comes first, and the salt factor can be applied only to keys identified as hot.