"Good Enough" Data Models
Data modeling along the spectrum of perfect vs "good enough"
The other day Sol Rashidi1 and I were having lunch when I mentioned that I feel like European data modeling practices were “good” compared with those in America. I base this on my personal experiences working and chatting with European data modelers (often working at massive private European conglomerates), who are generally more thorough, strict, and detailed in their data modeling approaches than their American peers. By contrast, American data modeling practices seem typically far less rigid and formal. Though a bit of a generalization, I’ve seen enough companies and spoken with enough practitioners to say this is a pretty accurate depiction of data modeling practices on both sides of the Atlantic.
Sol seemed taken aback by my use of the word “good,” which is very subjective. She felt it unfair to compare European and American companies, where the former often move more slowly and deliberately than the latter. It's not always necessary for a data model to be flawless, as long as it facilitates decisions and actions helping move the business in the right direction. There's more opportunity to develop a cohesive and precise enterprise data model in a slower or static environment because this type of work requires more time and patience. In faster-moving business environments, business models and mandates change much more quickly, and it’s often the case that the data model iterates more quickly, often at the expense of robustness or precision.
Data modeling aims to represent a business's rules, vocabulary, and workflows as precisely as possible, and there are many strict (almost dogmatic) data modeling methodologies and approaches. However, this aim is challenging when the underlying business model constantly evolves. Sol emphasized that even if American data models might not meet European standards of perfection, they can still be considered "good" if they are fit for their purpose and satisfy the needs of the business. Precision is traded off for “good enough” and practicality for the situation. Flexibility and agility are crucial in such scenarios. Practitioners often get caught up in aiming for technical perfection and precision. While perfection can benefit a technician’s sense of craftsmanship and pride, her comment highlights that executives frequently prioritize practicality over perfection. And since executives are in charge of the direction and budget of data initiatives, practitioners should read the room and understand what’s expected of them on the spectrum of perfection or practicality.
I’m not implying that data modelers shouldn’t strive for the best data models possible. Data models should be the best for the situation at hand, focusing on outcomes over rigid processes. On one extreme, sloppy data models providing wrong or misleading data leading to detrimental decisions and actions would be professional malpractice. On another extreme, I’ve heard people (especially in Europe) strive for things like 6th normal form and the perfect Data Vault or Kimball model, to the point of claiming anything less is garbage. An ideal goal might be a data model that anticipates every question that might be asked and the data precisely conforms to the contours of the business. For the craftsman data modeler, there’s always an ideal of the perfect data model. Of course the ideal data model would be fantastic to have, and if you have a lot of time on your hands and the ability to create this ideal data model, go for it.
In a rapidly evolving business landscape, achieving perfection in data models and practices can be challenging. Having the luxury of working in a static environment doesn’t exist these days. The world moves too fast. Executives often prioritize practicality over perfection, as the business environment demands quick adaptations. While striving for an ideal state is important, it's essential to recognize that "good" is often context-dependent. Effective data models should address the immediate situation while anticipating potential future changes.
Note: The audiences between Practical Data Modeling and my personal Substack don’t align, so reposting here for visibility and commentary.
I know this is not a popular opinion, but I don't understand why no one else is talking about Data Modeling from the MVC.
Software Engineers are supposed to understand the business to better understand how the user will interact with the controller, and the view, when the Software Engineers rarely seem to have a basic understanding of Data Modeling in their Software Model, and the Data Professionals are left to clean up their mess when we could be actually focused on actual Data Science and LLMs with a much smaller requirement for Data Modeling for Data Quality.
Also strange that Software Engineers can tend to be paid more than many Data Professionals when sometimes it feels like I should be constantly failing their Peer Reviews.
The trick about "good enough" is that you have to know what to focus on. Take Data Vault, for example. Many modelers spend lots of time figuring out multiactive status-tracking satellites or something, when they haven't even picked the right hubs to begin with! Too much attention is paid to the technical finesses of the chosen method instead of just understanding the business first.