The organizational aspects of data modeling are often overlooked in books and articles I’ve read on data modeling. Sadly, most data modeling efforts stall or fail because the org aspects are ignored. Data modeling is a full contact sport.
Here’s Part 1 of the two draft chapters on how organizations shape your data model. This first part covers Conway’s Law, some common organization and data team structures, and more. Part 2 will cover the politics of data, selling your data modeling initiative, change management, and more.
Writing this chapter made me very uncomfortable. There’s so much to cover, and I can’t say that I’ve covered everything in sufficient detail or usefulness. Countless books have been written on organizational dynamics and their problems. On top of that, despite great effort and endless money and attention, nobody has solved organizational problems. Please leave a comment if you feel I’ve missed something important. And if there are grammatical or similar errors, please leave a note here.
Thanks,
Joe
“The major problems of our work are not so much technological as sociological in nature”- Tom DeMarco and Timothy Lister (Peopleware)
Not much has changed since the 1980s when Peopleware was first published. The story of data modeling cannot be told without understanding how people interact within organizations. Work is very much sociological in nature, and for some reason, it is very underserved as a topic.
In my own experience, I’ve seen countless tech and data initiatives succeed or fail based not on the technical acumen of the practitioners but on the organizational dynamics surrounding them. The classic trap many practitioners fall into is believing their technical expertise will exempt them from interacting with “the business” or shield them from the whims and decisions of the broader organization. I’ve had it happen to me many times, thinking my smarts and skills were enough. While there are enough resources up to a point, I find that many data initiatives, particularly data modeling, which can be very time-consuming, are at risk because the teams lack the situational awareness and organizational structure needed to accomplish their goals. I believe this chapter is one of the most crucial in the book, as ignoring it will likely lead to failure in your data modeling journey.
A discussion of data modeling is incomplete without discussing its impact on people, organizations, and architecture. In the spectrum of people, process, and technology, people are paramount. You can be as technical as you wish, but understanding how to navigate organizations and models alongside your organization's architecture is crucial to success. In my experience, people and organizational issues sink more IT projects, data modeling or otherwise, than anything else.
A classic case study of organizational problems disguised as technical ones is the spectacular dumpster fire of Target Canada in the early 2010s. Historically, Target operated conservatively and methodically. In the case of Target Canada, they chose a big‑bang market entry, opening well over a hundred stores in a single year on brand new systems, with new teams, new suppliers, and no local operating history. The reason? In 2011, Target purchased the leasehold rights of defunct Canadian retailer Zellers for $2 billion, and the lease deal set demanding, compressed timelines to open three new distribution centers and convert the stores into Target Canada stores, all in less than 2 years. Never mind that one distribution center often takes around two or more years to construct.
That aggressive, real estate‑driven timeline became the de facto project manager. New people were severely undertrained and worked under immense time pressures. In the rush, the item master, the backbone of retail data, was hastily and inaccurately populated, with inconsistent units, packaging hierarchies, and identifiers that didn’t match what arrived on pallets. It only got worse. Shoppers received advertisements for products that were not in stock in the stores, despite the distribution centers being overstocked with assortments that didn’t fit local demand.
Underneath those symptoms sat a dysfunctional tangle of organizational dynamics. Supplier onboarding was often handled by junior staff who lacked the authority to push back on bad data. Validations that should have blocked flawed SKUs were treated as speed bumps rather than gates. Replenishment forecasting tools went live without the historical data they require to behave sensibly. Because it was Target, the forecast was literally as naive as “Whatever Zellers sold, just double it.” Different groups are optimized for various goals: merchants for launch count, supply chain for flow, and stores for in-stock metrics. Predictably, people found ways to bypass controls to make their numbers look better. None of this can be blamed on data or software failure. It was misaligned ownership, incentives, and sequencing that produced systemic data defects.
For data modelers, the lesson is not “pick a better tool.” Instead, it’s “design the social contract around the data model.” In Target Canada’s case, hindsight suggests treating the item master as a governed interface, not a disposable spreadsheet, where canonical units, packaging levels (each/inner/case/pallet), price zones and currencies, and required fields are enforced at write‑time. Instead of using lease dates as the go-live date, ensure your data is ready for production by setting explicit quality bars for your highest-velocity SKUs and conducting end-to-end scans that reconcile physical labels to digital records. Give named business owners real accountability for data quality, modeling, and the power to reject non‑conforming supplier inputs. And if your planning systems need history, start small so you can build it. All of these best practices in retail were thwarted, with the result being Target Canada’s closure, lost jobs, and billions of dollars wasted.
A single bad decision or a single bad table didn’t cause Target Canada’s downfall. It emerged from the interaction of structure, incentives, and speed with fragile data foundations. That’s the core theme of this chapter. Models live inside organizations, and when the organization is misaligned, the data will tell on you first in your dashboards, and then in reality. Although Target Canada is a widely studied example of a massive corporate boondoggle, organization-led failure is common.
The old industry trope is that 80% of data initiatives fail. This has been a commonly cited number for decades. The other trope is that this is not generally due to the technology, but organizational problems that arise while this initiative is underway. And now with AI getting a lot of attention and funding, the same projects are running into the same roadblock that I’ve encountered in other IT initiatives over the decades. Very rarely is technology the solution on its own. It’s often a matter of understanding the boundaries of what the technology can offer improvements, and where the organization needs to upskill or participate in these advancements.
Before we dive in, let’s look at how data modeling has been treated historically. I found that data modeling is often a theoretical exercise that is taught in a vacuum, ignoring the realities of organizations and the real world.
Theory vs the Real World
Data modeling theory paints a picture of a sterile, top-down exercise in a neat and orderly organization. Ideas and concepts are clearly understood, articulated, and always in sync. Data modeling is as simple as bringing people together for a series of workshops. If only things were that easy.
While theory provides a good starting mental framework for understanding how things should be, it's essential to recognize how theory intersects with the real world. And the real world is a tough and unpredictable place. No matter how hard you try, applying theory to the real world will be hampered and challenged in ways that will leave you frustrated.
Your job will be a mix of practitioner, sales, and servant. You’ll need to build or maintain a data model. But before you do that, there are often some roadblocks. People need to understand why they should invest in data modeling, why they should collaborate with you on this initiative, and what makes it matter to them. Especially in today’s organizations, people are pressed for time and budget. Most people are overworked and have little patience. Unless you can show them why they need to pay attention to data modeling, you’ll have little hope of building it.
Let’s briefly look at some places where data modeling theory often gets a reality check.
Ivory Tower Modeling. Data modeling is prescribed as a top-down exercise of gathering requirements from eager stakeholders, understanding every domain, and designing a pristine data model. What usually happens is you’re reverse-engineering some arcane, undocumented system, trying to decipher poor-quality data, and modeling under deadline pressure with incomplete information. Business rules are fuzzy and often undocumented. Stakeholders contradict each other. Requirements shift halfway through the project. The data may not align with the domain language (e.g., the model refers to “customer” but also includes prospects, partners, and employees). Approach every situation not with an eye on perfection, but with an eye on what can go wrong and when this might happen.
Data is political. Data modeling doesn’t happen in a vacuum. Every data model is a political artifact. It reflects who had influence, what got prioritized, and what got ignored. These political dynamics are influenced by factors such as team communication, company strategy, technical debt, internal politics, and even personal relationships or vendettas. For instance, a department might resist a new customer definition because, while it is more accurate for the business, it makes their legacy metrics appear worse. Always be aware that some people may support your data model, while others may attempt to sabotage it.
Needs and expectations vary. All too often, data modelers want to use their pet approach for everything, regardless of whether it's a good fit. For example, I’ve heard people say the relational data model is universally applicable. This is a huge mistake. Instead of viewing a data model as a single overarching thing, understand how different models serve diverse needs within an organization. This typically involves one data model feeding into multiple data models, each with its own use case and purpose. Decision-makers require readily digestible data to inform their choices and actions. Analysts seek easily explorable data for their analyses. Engineers prioritize cost-efficiency and dependable performance. Finally, ML/AI models rely on clean and reliable features to build models for classification, prediction, or content generation. You wouldn’t hand a CEO a raw ML model output, and you wouldn't give a software engineer a star schema. Both are unfit for their specific purpose without translation and context. It all comes back to the purpose of the data model and what people expect from it.
Compromises happen. Every situation is different and has its constraints. You might choose a perfect textbook data modeling approach that, if ideally implemented, will take several years. Then, you realize that due to time, budget, or resource constraints, you’ll need to make compromises and take shortcuts to deliver something in a far shorter time. What matters is quickly delivering value for the situation at hand, rather than adhering to the ideal textbook data model.
Focusing only on tools and technology. When working within an organization, a primary goal of data modeling is to achieve a shared understanding of the data. Especially for non-technical individuals, discussions about technology can be a significant distraction. Also, stay focused on discussions and avoid bombarding people with overly technical diagrams. Engaging and thoughtful conversations are more critical than diagrams. Diagrams are the result of these conversations, but not the reason to have them.
As you can see, data modeling relies on more than just possessing a range of technical approaches. You’ll also need situational and social awareness. We’ll revisit some of these examples throughout this chapter.
Keep reading with a 7-day free trial
Subscribe to Practical Data Modeling to keep reading this post and get 7 days of free access to the full post archives.