Matillion ETL

When we started using Snowflake in 2018, we chose Matillion ETL as our tool of choice for data integration and orchestration of data pipelines. We are currently in the process of migration. We have been using Matillion as our main data integration and data transformation tool for our Snowflake data warehouse to enable data-driven initiatives in B2C sales and marketing. We have been processing mostly batch data. As our team consists of people with strong programming and data backgrounds as well as less tech-savvy users, Matillion facilitates easy collaboration between the two. The representation of a data pipeline as a flow chart of components with customizable properties on a canvas is easily readable even without a technical background. Some complex SQL usage patterns have no equivalent component in Matillion ELT but can be implemented using an SQL component. Sampling at any step of the data pipeline makes ongoing validation of the data transformation and debugging very easy. Matillion ETL offers connectors to integrate with most data sources relevant to our use cases. Seamless integration of Python in the orchestration jobs, offers the ability for API calls should a native connector to any source system be missing. Rollout on both Azure and AWS was well documented and easily done. In general, Matillion ETL integrates well into our workflow, however the GIT integration is lacking some features to make it truly elegant, especially for users who are less used to it. We onboarded other teams on Snowflake and Matillion ELT who were quickly able to work independently after an early short period of enablement and support. In total, our Matillion ETL instance hosted 15-20 data engineers across several teams. As we did not have a data warehouse beforehand, we cannot supply comparative metrics. Snowflake and Matillion outright enabled us for the first time to conduct truly data-driven use cases with state of the art technology solutions.