Batch or streaming: pick by the freshness you actually need

Ask about staleness first

When someone asks for real-time data, I try to ask one question before opening an editor: how old can this number be before someone makes a bad decision with it? The answer is almost never zero. A finance team closing the month is fine with figures that were correct at midnight. An engineer watching error rates wants numbers that are a minute old at most. A check that blocks a card payment needs an answer in well under a second.

Those three consumers need three different pipelines, and the cost difference between them is large. Write the staleness requirement down with a number attached. Ask what the consumer does when the figure is wrong, and what they do when it arrives late. If the honest answer is "we look at it tomorrow morning," a nightly batch job is the right tool.

Micro-batch as the middle ground

Between nightly batches and per-event processing sits a wide range, usually five to fifteen minutes, where a scheduler runs an incremental job over whatever arrived since the last run. I call this micro-batch. It keeps the habits of batch: runs are idempotent, failures are retried, and each run is a clear unit of work that either finished or did not. The lag drops from hours to minutes.

For most analytics layers this is enough. The job is dull, and that is useful. You can rerun it, read its logs, and reason about its state without learning how a stream processor manages its internal state.

Cost and operations

Continuous streaming costs more than the first estimate suggests. A consumer that must stay up all day needs lag monitoring, alerting, and someone who answers when it breaks at night. Schema changes hit a live pipeline, so you cannot simply pause it while you fix a column type. Checkpoints and replay are features you must understand well enough to debug under pressure.

Compute is the smaller part of the bill. The larger part is the hours your team spends keeping the pipeline healthy. A failed batch run is easy to explain: you fix the cause and run it again. A failed stream may have partly processed its input, and working out exactly what landed takes time.

Late data and watermarks

Streaming forces a question that batch hides. An event stamped 14:02 may reach the system at 14:09 because a phone was offline. If you aggregate by event time and close the window at 14:05, that event arrives late. A watermark is the processor's estimate of how far event time has advanced, and it decides when a window counts as complete. Set it too tight and you drop real events. Set it too loose and results stay open longer than anyone wants to wait. A batch job avoids the problem by waiting until its input is closed, which is one reason it is easier to get right.

One aggregation, two ways

Take a table page_events with event_time, ingested_at, user_id, and page. The goal is daily unique users per page. The batch version runs once a day over a closed day:

SELECT
    CAST(event_time AS DATE) AS event_date,
    page,
    COUNT(DISTINCT user_id) AS unique_users
FROM page_events
WHERE event_time >= DATE '2026-09-13'
  AND event_time <  DATE '2026-09-14'
GROUP BY 1, 2;

The incremental version runs every ten minutes. Distinct counts cannot be added across partial results, so the job does not sum yesterday's counts with new ones. It finds the dates touched since the last run and recomputes those days in full:

BEGIN;

DELETE FROM daily_page_users
WHERE event_date IN (
    SELECT DISTINCT CAST(event_time AS DATE)
    FROM page_events
    WHERE ingested_at >= :last_run_started_at
);

INSERT INTO daily_page_users (event_date, page, unique_users)
SELECT CAST(event_time AS DATE), page, COUNT(DISTINCT user_id)
FROM page_events
WHERE CAST(event_time AS DATE) IN (
    SELECT DISTINCT CAST(event_time AS DATE)
    FROM page_events
    WHERE ingested_at >= :last_run_started_at
)
GROUP BY 1, 2;

COMMIT;

The ingested_at filter is what makes a late event work. It may belong to a day that was already published, so that day gets recomputed. The cost is that a busy day is reread on every run. For this table that is cheap. For a much larger one, you would narrow the recompute or accept a coarser window.