Azure Data Factory

As a Microsoft Azure-based analytics organization, I appreciate the ease at which data flows can be created within a code-free interface. As someone who is rather new to building data pipelines, that helps me reduce manual errors on my part and improves the ability for other team members to quickly understand how we use the platform.

It simplifies a lot the interconnection of different services of Azure like reading secrets from a key-vault or ingesting data from an external source into a storage account. It also allows you to connect to a repository from multiple sources (not only Azure DevOps) such as Github or GitLab. The application is intuitively designed and the box system simplifies pipeline construction.

1. Works effortlessly with Azue SQL, Synapse, and Key Vault. 2. Scales workloads efficiently and supports parametrization and CI/CD through ARM templates and Git integration. 3. Natively supports hundreds of on-prem and cloud connectors, minimizing custom connector development.

If anything, there is a bit of a learning curve around the UI. I found myself taking several weeks to remember how to navigate simple functions or make alterations to previous jobs. It is not as intuitive as one would expect.

There is a limit on the number of boxes you can use in a pipeline. The test connection for some actions does not work, even if the implementation works. If you have a repo connected and you are using a publish branch to use it as a productive branch, sometimes the changes in global variables are not tracked by the version control and you need to force manually the change by adding a dummy change

1. Error messages are often vague; troubleshooting large pipelines is tedious. 2. The web interface can lag with large dataflows or when switching between tabs. 3. Pricing can be tricky - escpecially with data movement and compute activity changes that add up fast.