Characteristics of Application Data

Part of the chapter on Application Data Modeling

This latest draft section on application data focuses on the characteristics of application data. Why focus on this instead of jumping into data modeling? Part of me feels data modeling is too much of an ivory tower, especially with apps. Proclaim the model to the developers and engineers, and it will appear. This is delusional. Developers and engineers are busy and move fast because of “agile” (that’s a joke). From my observation, they don’t often think about data, let alone data modeling. Perhaps this book changes this dilemma. Who knows?

This section¹ contains a lot of information. Part of me feels like it’s too crushed together, and the other part says go with it. At 8+ pages, the application chapter will be 50 pages. Probably much longer. As always, please give me your feedback in the comments.

Thanks,

Joe

---

Like every use case we’ll look at in this book, the way data is used has particular nuances and quirks. As much as data models should be standardized against conceptual and logical models, the reality is often very different. Data models are often very physical, which is where they matter the most. There are also user requirements we need to consider. I sometimes find data modelers ignore the realities of user expectations and instead stick with orthodoxy. On the flip side, software developers occasionally ignore data modeling and cram data into a database or streaming system. The result is often an incoherent mess of data that, although it delivers the user requirements on paper (like speed), fails to be understood by the business or downstream consumers. When we model data, we need to be aware of the characteristics of how the data will be used, lest we shoehorn data models or design data systems in a vacuum that’s ill-suited to the expectations and characteristics of the data and its end users.

Let’s look at some of the characteristics of application data.

Speed, Performance, and Access

Users today are impatient. They expect apps to be lightning-fast, with responses that feel practically instantaneous. This means minimizing latency. A few seconds' delay is not a big deal. But in the aggregate, that slowness means many frustrated users. Remember that old story about Amazon, how back in the day they figured every 100ms of lag cost them a whole 1% of profit? Who knows what that number is now, but it highlights how crucial speed has always been. These days, speed is not just important. It's mandatory. And data, whether we're talking about how quickly it's read or written or how efficiently it's queried, is central to making applications meet user needs.

But every application has different data needs. Some are write-heavy and constantly bombarded by events that must be quickly written into the database. Others lean towards being read-intensive, serving lots of requests. Some applications might be both. Most of the time, these reads and writes involve small pieces of data. Think kilobytes, maybe a few megabytes at most. However, today's data-intensive applications might be processing millions (or much more) of these small pieces of data every second.