To be clear, this article shouldn’t need to be written. I feel bad even having to write something like this. But there’s so much bad advice around this topic that I feel I need to weigh in and add facts. It’s like telling people that water is wet. Wow, no shit…
Refreshingly astute Joe. I just wish Databricks hadn't called it an 'architecture'. It is an easy-to-understand approach, and perhaps an overly simplistic way to organize data flow and data classification. So it is what it is, nothing more, nothing less.
Us techy folks need to embrace the reality that Medallion only presents an overylay to the required 'real' architecture, methodology, and an appropriate data model underneath.
Not sure what to think. Obviously your the g when it comes to data modeling. But I guess I must also be conflating medallion with modeling.
I just think I do it by assumption. Like for instance I assume the data in the gold layer will be cleaner or more ready to serve than that in bronze layer therefore it has gone through some sort of modeling. All of which I sort of roll up into whichever layer. So I guess sometimes when I say the gold layer I am talking about the architecture (were it’s landed) and the modeling it had to go through to get there.
Sue me 😀, at the time I was explaining alot of this (to clients) I was half suit half lab coat and it was easier to roll everything into layers then explain them separately when I know they’d get confused.
Hope all is well Joe, I’ve been off the radar for some time and that’s on purpose . But I still do enjoy reading this when they come across my email
Thanks for clarification. Additional why/how all business stakeholders (especially those with budget authority) must have data + tech (such as AI) context.
To be clear, this article shouldn’t need to be written. I feel bad even having to write something like this. But there’s so much bad advice around this topic that I feel I need to weigh in and add facts. It’s like telling people that water is wet. Wow, no shit…
Refreshingly astute Joe. I just wish Databricks hadn't called it an 'architecture'. It is an easy-to-understand approach, and perhaps an overly simplistic way to organize data flow and data classification. So it is what it is, nothing more, nothing less.
Us techy folks need to embrace the reality that Medallion only presents an overylay to the required 'real' architecture, methodology, and an appropriate data model underneath.
Exactly
Definitely agree with you : medallion architecture and modeling are two totally different things.
the real issue (or debate), is whether medallion is a data architecture design pattern or just a kind of data classical processing stages.
Like I wrote, that’s a separate debate
Love the metaphor about the car and the parking lot.
Not sure what to think. Obviously your the g when it comes to data modeling. But I guess I must also be conflating medallion with modeling.
I just think I do it by assumption. Like for instance I assume the data in the gold layer will be cleaner or more ready to serve than that in bronze layer therefore it has gone through some sort of modeling. All of which I sort of roll up into whichever layer. So I guess sometimes when I say the gold layer I am talking about the architecture (were it’s landed) and the modeling it had to go through to get there.
Sue me 😀, at the time I was explaining alot of this (to clients) I was half suit half lab coat and it was easier to roll everything into layers then explain them separately when I know they’d get confused.
Hope all is well Joe, I’ve been off the radar for some time and that’s on purpose . But I still do enjoy reading this when they come across my email
Thanks for clarification. Additional why/how all business stakeholders (especially those with budget authority) must have data + tech (such as AI) context.
?
To hire the right people.