At 8:40 in the morning, a dispatcher wants to know which vans are likely to miss their delivery windows. The organization already stores months of location events. Its dashboard has maps, filters, and an impressive total-record count. Yet the dispatcher is still calling drivers because the latest positions are old and several vehicles appear twice. The company has accumulated data. It has not made the particular information needed for this decision reliably available.

That is a better starting point for understanding big data than a forecast about how many zettabytes the world will create. Scale matters when it changes what you must do to answer a question on time, at an acceptable cost, with evidence you can trust. A large archive and a difficult workload are related, but they are not interchangeable. You can have a huge dataset that is rarely queried and a small stream whose latency makes it operationally demanding.

Consider a hypothetical delivery network with 1,200 vans. During a ten-hour operating day, each van emits ten telemetry events per minute. Each event occupies 600 bytes in its raw representation. The business needs two products: a dispatch view that is fresh enough to support intervention, and a weekly analysis of route performance. We will use this same network throughout. The numbers are explicit planning assumptions, not a report about a real fleet or an infrastructure benchmark.

Turn “large” into a workload

The fleet produces 7.2 million events a day. At 600 bytes per event, that is 4.32 GB daily, or 129.6 GB over 30 operating days, using decimal units. During the stated operating window, the average arrival rate is 200 events per second. Those figures describe different pressures. Storage must accommodate retained history; ingestion must cope with arrivals; queries must finish before their answers lose value. A single total cannot tell you which part is difficult.

Average arrival rate is only a starting point. Drivers may lose connectivity and upload a backlog when they return to a depot. A software update might cause events to arrive in bursts. A busy morning can concentrate demand on the dispatch view while weekly jobs are still running. You should measure peak rates and recovery behavior, rather than multiplying a daily average by a vague “safety factor” and assuming the capacity problem is solved.

NIST’s Big Data Interoperability Framework makes the useful distinction: the need for scalable architecture depends on the interaction of application requirements and constraints. Its definitions discuss volume, velocity, variety, and variability. That is a reminder that the familiar V lists are descriptive aids, not a certification scheme. There is no universal row count beyond which a responsible engineer must buy a distributed platform.[1]

The popular additions, veracity and value, ask questions that should be present at every size. Can you trust what an event means? Who uses its result? For the dispatch view, a location without a trustworthy vehicle identifier is not “almost usable.” It might lead someone to call the wrong driver. For the weekly report, a change in event frequency can look like a change in activity. More observations do not automatically correct a changing measurement process.

Define the service before choosing its machinery

Specify what “fresh enough” means with the dispatcher. In this example, take a proposed target of seeing accepted telemetry within two minutes of its event time during normal connectivity. That is a service objective to evaluate, not a guarantee this architecture has demonstrated. The weekly report has a different deadline and can wait for reconciliation. Forcing both uses into one pipeline often creates an uncomfortable compromise: too much machinery for reporting and too little clarity for operations.

A useful workload description includes data size, arrival pattern, retention, query shape, concurrency, freshness, and recovery expectations. It should also say what happens when the service falls behind. The dispatcher might see a clear stale-data warning and revert to manual checks. The analyst might wait for a reconciled partition. Those responses are part of the design, because the organization’s need for an answer does not disappear when a stream processor is unavailable.

You can often reduce the work before adding machines. The weekly route review needs a defined period, a subset of fields, and a particular aggregation. It does not need every byte from every telemetry event. Start by making that question cheap to express and execute. Keep event dates and vehicle identifiers explicit. Separate the analytical representation from the original payload when doing so makes repeated queries easier to understand and cheaper to run.

Apache Parquet is a column-oriented file format designed for efficient storage and retrieval, with compression and encoding support. Columnar storage lets engines organize data by field rather than treating every record as an indivisible object. For the fleet, an analysis of event time and route identifier need not conceptually require the full diagnostic payload. The exact bytes read still depend on the engine, file layout, metadata, encodings, and the query.[2]

Partitioning retained data by a useful date key can help restrict a weekly query to the relevant days. It can also create trouble if you partition too finely and generate thousands of tiny files. Partitioning by every vehicle and every minute sounds precise, but it may increase listing, metadata, and scheduling work. The right layout is the one that serves actual access patterns. Precision in folder names is not the same thing as efficient execution.

Count physical copies, not just logical records

Suppose, purely for a capacity exercise, that the retained payload compresses by 3:1 and you keep two physical copies of that compressed payload. The 30-day raw size of 129.6 GB becomes 43.2 GB per compressed copy and 86.4 GB across the two copies. This is not a predicted compression ratio for telemetry. It is a transparent scenario that reveals why raw bytes, compressed bytes, and replicated bytes should never appear under one ambiguous “data size” label.

Even that physical-copy estimate leaves things out. Indexes, metadata, temporary shuffle files, logs, backups, and separately retained raw data can all contribute to the footprint. If you keep the raw archive alongside the two compressed copies, you must add its size rather than pretending the compressed representation replaced it. A capacity estimate should name what it includes. Otherwise two teams can report different numbers while both are correctly measuring different parts of the system.

The calculator changes the fleet size while holding event frequency, operating hours, event bytes, and retention fixed. Its outputs are arithmetic consequences of those assumptions. They are not estimates of cloud cost or proof that one machine will be sufficient. Use the exercise to identify which inputs you need to measure. An unexpectedly large payload may matter more than a small increase in vehicle count, and a photo attachment would create an entirely different size distribution.

Benchmark the question you actually ask

Run a representative route query against a representative period and record the result, elapsed time, bytes scanned when available, memory behavior, and concurrent load. Include a cold run and a repeated run if caching matters to the service. A fast result obtained from a warmed cache can be perfectly useful; it becomes misleading only when you report it as if it described first-time access or a recovery after restart.

The working set matters as much as retained history. A system might hold a large archive while a typical report touches a small slice. Another query might join nearly everything and produce a large intermediate result. Input size alone will not predict the difference. Inspect the query plan and the sizes of intermediate operations. A join that multiplies records because its key is not unique can exhaust resources without teaching you anything about genuine business growth.

The practical choice is therefore conditional. Start with the simplest arrangement that meets the measured workload, and record what would trigger a change: missed freshness objectives, sustained memory pressure, concurrency limits, recovery time, or an unacceptable cost curve. That is a stronger architecture argument than either “everything should be distributed” or “everything fits on a laptop.” Both slogans replace a workload with an identity.

The fleet’s most interesting problem arrives when a van reconnects. It uploads an event stamped 08:32 at 08:41, after a newer event has already reached the platform. Which position should the dispatch view display? If the system blindly uses arrival order, the van may appear to move backward. The issue is not solved by storing more rows. You need to distinguish when an event happened from when your system learned about it.

Keep event time and ingestion time as separate fields. Decide how late data changes each product. The operational view might retain the latest valid position by event time and show a freshness indicator. The weekly report can include reconciled late events before its cutoff. If very late data arrives after a report is finalized, record whether you revise the report or publish a correction. “Real time” is not a complete policy for information that arrives out of order.

The diagram separates an operational projection from a reconciled analytical table. Both originate from the fleet’s accepted events, but they have different rules about freshness and completeness. The event record remains available for replay within the retention policy. Replay is valuable when transformation logic changes or a downstream job fails. It also means that consumers must be designed for records they may encounter again. Recovery is a normal path through the system, not an exceptional footnote.

A retry is also a data event

Imagine the client sends a telemetry event, the server accepts it, and the acknowledgement is lost. The client retries. Without a stable identity and a deduplication rule, one physical occurrence can become two analytical records. Define an event identifier or a suitable compound key at the source, then state what uniqueness means. A timestamp by itself may not be unique, and an identifier generated afresh on every retry defeats the purpose.

Apache Kafka’s design documentation distinguishes at-most-once, at-least-once, and exactly-once processing guarantees and explains the boundaries of its transactional support. In particular, writing to an external destination requires cooperation with that destination to coordinate progress and output. A broker feature does not automatically prevent a downstream application from sending two alerts or performing the same business action twice. Read the guarantee at the boundary where your actual effect occurs.[3]

For the dispatch product, make alert handling idempotent where the action permits it: processing the same event again should not create another identical task. Store a stable alert key and an action state. Be careful when replacing a record, however. A late correction is not always a duplicate, and simply discarding repeated keys can erase legitimate updates. The business semantics determine whether you append, replace, reconcile, or reject an incoming record.

Distribution makes failure more varied

MapReduce helped establish a practical way to express large computations through map and reduce operations while a runtime handled partitioning, scheduling, and failures across machines. The important contribution was an execution model, not a promise that distribution removes engineering trade-offs. Dean and Ghemawat’s original paper is still useful reading because it explains the machinery that sits behind an apparently simple parallel computation.[4]

On a fleet workload, a distributed aggregation may need to move records between workers so that matching route keys can be combined. Uneven keys can concentrate work on one worker. Slow storage can make additional compute irrelevant. Retries consume resources, and partial failures complicate diagnosis. These are reasons to measure and design carefully, not reasons to reject distributed systems. Parallel execution earns its complexity when it solves a demonstrated constraint.

Make recovery measurable. Ask how long it takes to rebuild the dispatch projection, replay a day’s events, or restore a reconciled table. If replay shares capacity with live traffic, test the combined demand. A platform that keeps up only when nothing goes wrong has not met an operational requirement. Your architecture should explain how it catches up, how it signals degraded freshness, and who decides which workloads receive priority during recovery.