The Automated Modern Data Stack. We believe it was our partner Fivetran that used a TON of “marketing dollars” to come up with this entire “data concept” of the “Modern Data Stack”. [If anyone feels differently, then comment or directly reach out to me]. Once this term / concept was taking off in the data mindsphere, it seems like every data vendor under the sun with any type of solution tries to market that they are part of this concept.
Since 2018 when we came across this pay as you go, separated compute from storage simple cloud database of Snowflake, we immediately recognized this was a huge step forward for data professionals and what we have termed as “Data to Value”. Snowflake’s architecture COMPLETELY leap frogged all the competitors at that time. As we became one of the top consulting partners (we sold that line of business back in 2018 – We FOCUS now only on Snowflake and Automated Modern Data Stack Thought Leadership, Education, and Saas based services like Snoptimizer™), we became more and more cognizant of what was truly differentiating Snowflake within the entire set of various data processing use cases within OLAP. We view it as the massive removal of friction around the workloads. This is where we see all of our OTHER main data related partners doing the same thing as well. That is how they became part of our Automated Modern Data Stack.
Automation within the “Automated” Modern Data Stack is what really matters!
[okay. Automation and the continuous data security, data governance, and checking/assurance of DATA QUALITY.] While we make strong positions for the automation of data, the reality is data is USELESS and even detrimental without some form of guaranteed or “checked” quality. Without further adieu, here is our 2022 ITS Automated Modern Data Stack.
ITS Snowflake Solutions – Automated Modern Data Stack
Base Layer 0 – Snowflake.
Hey, the website does have the domain Snowflake Solutions and it is our ITS Snowflake Solutions Community where we try to answer any question related to Snowflake is a completely transparent and validated approach. Unlike tons of other Q&A areas around Snowflake, we work to have Snowflake Experts and Snowflake Data Superheroes validate ALL answers here. I see so much outdated answers in other locations and it annoyed the hell out of all of us here. 🙂
Layer 1– Extract – Load. Fivetran
ITS has been a long time partner with Fivetran, Matillion, and Stitch (now Talend). We realized though for most of our work which ALWAYS involves the top tools related to AUTOMATION, we still see Fivetran as the clear winner here within our data s. Love or hate Fivetran and their consumption based pricing (:)). They are growing fast and from our view they dominate the “AUTOMATED” EL space.
Layer 2– Transform – Document – Be Column Aware. Coalesce.
When we came across the full demonstration of Coalesce, we were blown away. We view coalesce automation tool as one of the largest game changers of 2022. We have not seen a transformation product like this which is so easy to use and does not have the complexity that all the other ETL, ELT, and T tools have that make them much more difficult to work with. Our only reservation around Coalesce is that it ONLY works on Snowflake and the reality is that there are many customers that are more diversified in their Layer 0 of a cloud database or data cloud…. If you are Snowflake Centric then get a demo of Coalesce from someone as soon as possible. Signup here if you want to get on our calendar for a Data Superhero overview.
Layer 3– Reverse Extract-Load (or I guess some people call it Reverse-ETL as well). Hightouch.
We have reviewed this Reverse EL/ELT space for quite some time. It has really grown a ton over the last couple years. We found Hightouch to be the easiest to use and partner with.
Layer 4– Data Analytics WITHIN THE DATA CLOUD. Sigma Computing.
We have followed Sigma Computing’s rise in the analytics space for the last few years. We originally met with Rob Woolen back in 2018 I believe. At first the tool was pretty limited but its coming on strong and the reason why we see it as a MASSIVE game changer eventually is because it goes where Tableau, Microstrategy, Domo, blah blah. blah, etc. never were designed from the ground up for. It operates as a calculation engine on top of Snowflake and the BEST PART if done corectly is that Sigma NEVER … EVER EVER creates extracts. While this was necessary in pre-history cloud times…. It is one of our MAJOR MAJOR data analytics anti-patterns. Copying the same data even once can create havoc throughout your organization. Listen, I’ve been and I”ve done that but we shouldn’t be doing it anymore.
Other layers we do not have partners with:
Layer x – Data Governance.
At this time we have not chosen a vendor for this part. There is tons of fragmentation of competitors. At some of our work we definitely see Alation and Collibra as the two dominant players but we have not partnered with either yet. There are free
Layer y – Data Quality. Same here.
Layer z – Data Platform Optimization.
We are biased – We recommend our Saas Service Snoptimizer™ for this when using Snowflake as Layer 0.
Are we missing any of the most necessary automated data layers? Is there any data automation technology you think we need to add? We purposely have kept this list very short but definitely comment below or reach out directly to us.
***Also, we do use sqlDBM as one of our key partners as well. We just do not have this heavily automated.
This page is the current main AUTOMATED MODERN DATA STACK we use 95% of the time which we recommend to our customers who are predominantly moving to Snowflake. Also, one thing that DEFINITELY concerns us is how Snowflake has really become the “GREAT” Co-opetitor. We see it moving every year to take on more and more of the workloads adjacent to its HUGE Snowflake Data Cloud. What assurances will Fivetran, Hightouch, Sigma, etc. ever have that Snowflake will not use its HUGE war-chest to enter their areas more and more. [okay. with the big man Michael
I hope you found this useful for thinking about what your truly Automated Modern Data Stack should be.