I just watched / listened to the conversation during my walk. I am gutted to have missed it. This is brilliant. Hoping to participate in one of such sessions soon.
Finally had time to catch up! Some intersting thoughts!
Firstly, +1 for BEAM*
I agree with the idea that starting with specific questions can be too narrow, and trying to answer all questions is too wide.
I take an approach that focusses on one event. I then build a conceptual model by asking the 7Ws about that event (who, what, where, when, why, how and how many) - this covers a wider scope than JUST the questions people want answered, but also helps plug some of the gaps (forcing stakeholders to think about the event under the 7W lens often makes them think about their data in ways they hadn't considered)
I then take the set of user questions and use them to validate the conceptual model we've now built. this gives the stakeholders the confidence that they have what they need to answer the questions of the day, whilst giving the development team the confidence that the wider scope covered in the 7Ws can answer the questions of tomorrow. Sometimes our model can't answer a question, which helps us identify gaps and improve the design.
i just started my data engineering learning and i learnt a lot here about what it takes in building a data model which is the ground level for developing a clear and concise data set that is required by the upstream consumer, the key point here is all about collaboration and communication with both the upstream consumer who needs the data set for analytics and the software engineer who craft the data sources where the data model is being derived from
Good discussion. One observation though is a lot of the heartache seems to be around organizational issues, not issues specific to data modeling. If your source system owners don't want to talk to you then where is leadership to build those connections? If the business wants immediate answers to their questions but doesn't want to think about them up front, then how do you get product and marketing to change their behavior so you have the requirements when building? It's something that maybe enterprise architecture has a role in helping as they define the flows end to end, but you also have to have the right people there. Leadership is really the key though. If your company views its data as valuable asset, then they need to enable the data team to get the maximum value out of that asset. Management generally understands how teams need to collaborate to bring products to market and it's the same concept to bring high quality data to the business.
I just watched / listened to the conversation during my walk. I am gutted to have missed it. This is brilliant. Hoping to participate in one of such sessions soon.
Next time!
Just watched the recording - great discussion- loved the MDM piece which I sent to a work colleague (laughing all the way to the MDM vendor bank)
Awesome discussion, very insightful. Makes me realize I'm not alone with these data modeling challenges I face
Excellent discussion. Sorry to have missed it live.
Is there a date for the next discussion?
Not yet. Will send a survey out soon to figure out a more regular cadence
Thanks for posting the video, I was sad to miss it this time! Looking forward to the next one, though!
See ya soon
Finally had time to catch up! Some intersting thoughts!
Firstly, +1 for BEAM*
I agree with the idea that starting with specific questions can be too narrow, and trying to answer all questions is too wide.
I take an approach that focusses on one event. I then build a conceptual model by asking the 7Ws about that event (who, what, where, when, why, how and how many) - this covers a wider scope than JUST the questions people want answered, but also helps plug some of the gaps (forcing stakeholders to think about the event under the 7W lens often makes them think about their data in ways they hadn't considered)
I then take the set of user questions and use them to validate the conceptual model we've now built. this gives the stakeholders the confidence that they have what they need to answer the questions of the day, whilst giving the development team the confidence that the wider scope covered in the 7Ws can answer the questions of tomorrow. Sometimes our model can't answer a question, which helps us identify gaps and improve the design.
i just started my data engineering learning and i learnt a lot here about what it takes in building a data model which is the ground level for developing a clear and concise data set that is required by the upstream consumer, the key point here is all about collaboration and communication with both the upstream consumer who needs the data set for analytics and the software engineer who craft the data sources where the data model is being derived from
Sorry correct me if I'm wrong. I think you meant downstream instead of upstream consumers?
How do I get into the cool kids club?
Good discussion. One observation though is a lot of the heartache seems to be around organizational issues, not issues specific to data modeling. If your source system owners don't want to talk to you then where is leadership to build those connections? If the business wants immediate answers to their questions but doesn't want to think about them up front, then how do you get product and marketing to change their behavior so you have the requirements when building? It's something that maybe enterprise architecture has a role in helping as they define the flows end to end, but you also have to have the right people there. Leadership is really the key though. If your company views its data as valuable asset, then they need to enable the data team to get the maximum value out of that asset. Management generally understands how teams need to collaborate to bring products to market and it's the same concept to bring high quality data to the business.
It is an organizational issue a lot of the time huh, Will. Something about theory works great in theory but not so much in practice. :D
Finally watched the recording, very rich discussion indeed. I published this piece last week where I discuss some of the challenges growth teams face while collaborating with Eng and Data, would love any thoughts: https://newsletters.databeats.community/i/143259287/software-engineers-and-data-engineers-dont-speak-the-same-language