Hi, I would like to ask if you plan to adress the technical debt in modeling. Like, what happens if legacy tables were modeled using a certain method and you want to change it, or used no modeling at all, how to deal with this kind of changes. Thanks!
I've been following your work on data engineering and data modeling with great interest and am excited about the upcoming book. My career has been centered on finance, operations, and data and analytics layers in middle-market, private equity-backed businesses, a segment I believe is underserved by data and analytics and data science currently.
In my role, I frequently encounter the challenge of activating for analytics and data science multiple versions of disparate systems - from PoS to CRMs and ERPs, etc. - especially in businesses undergoing acquisitions. The analytics layer integration is crucial for operational management and demonstrating value to potential buyers, ideally in lieu of costly and time consuming source system integrations (especially if nothing is broken where hands hit the keyboard and data is produced).
Given these complexities, I hypothesize MDM and Data Modeling to be even more critical in my context than perhaps it may be for others.
Therefore, the practical insights your substack intends to highlight I suspect will not be just helpful, but essential for navigating these challenges.
"Sure enough, application developers and software engineers also ignore data modeling."
Would love to understand what you mean by this. If the application is backed by a relational db, some amount of modeling is unavoidably necessary (though not typically anything we'd directly use for analytics).
I was watching a data modeling course and your comment brought one of its concepts to mind. One thing that was mentioned is that data modeling and database design use similar techniques. So while you're describing is more logical modeling, I suspect what he's referring to that there needs to be more discussion of conceptual modeling(one level above logical) before we consider more logical modeling/DB design. Hope that helps!
Any update into your data engineering spec release?
What are you referring to?
In this article you mentioned Andrew Ng asking you to spearhead a data eng specialization over at deep learning.ai
I would be first to sign up whenever/if that comes out!
I enjoyed the recorded session on YouTube.
Throughout the session, Joe recommended many book including:
The data model resource book Vol. 1, Vol2, and Vol3.
Hi, I would like to ask if you plan to adress the technical debt in modeling. Like, what happens if legacy tables were modeled using a certain method and you want to change it, or used no modeling at all, how to deal with this kind of changes. Thanks!
I've been following your work on data engineering and data modeling with great interest and am excited about the upcoming book. My career has been centered on finance, operations, and data and analytics layers in middle-market, private equity-backed businesses, a segment I believe is underserved by data and analytics and data science currently.
In my role, I frequently encounter the challenge of activating for analytics and data science multiple versions of disparate systems - from PoS to CRMs and ERPs, etc. - especially in businesses undergoing acquisitions. The analytics layer integration is crucial for operational management and demonstrating value to potential buyers, ideally in lieu of costly and time consuming source system integrations (especially if nothing is broken where hands hit the keyboard and data is produced).
Given these complexities, I hypothesize MDM and Data Modeling to be even more critical in my context than perhaps it may be for others.
Therefore, the practical insights your substack intends to highlight I suspect will not be just helpful, but essential for navigating these challenges.
Looking forward to your guidance, Joe!
"Sure enough, application developers and software engineers also ignore data modeling."
Would love to understand what you mean by this. If the application is backed by a relational db, some amount of modeling is unavoidably necessary (though not typically anything we'd directly use for analytics).
I was watching a data modeling course and your comment brought one of its concepts to mind. One thing that was mentioned is that data modeling and database design use similar techniques. So while you're describing is more logical modeling, I suspect what he's referring to that there needs to be more discussion of conceptual modeling(one level above logical) before we consider more logical modeling/DB design. Hope that helps!
Can’t wait!
Looking forward to this!
Very excited!