Databricks AI/BI

Very suited for data intensive workloads combined with AI, but to costly for lightweight operational AI use cases. Databricks seems a strong long term candidate in the new AI age, but it needs to be complemented with other technologies that are more cost-effective for light-weight use cases.

It is a powerful, unified platform for large scale data engineering, analytics and AI with excellent scalability and Lakehouse architecture. It also offers collaborative workspaces and seamless integration with major cloud providers.

With the time I have spent so far with Databricks Genie it has turned tasks that would ordinarily take hours into tasks that take seconds or minutes. However, as part of the rollout there have been some regional deployment issues that have prevented us from taking full advantage of the tool.

The best part about working with this platform is our business users can interact with the datasets using NLP. They don't need to learn or hunt down someone with SQL expertise. The auto generated summaries in Lakeview are also great for executive facing reports. You drop in a chart and it writes takeaways in plain English. And all of these can be done through Unity catalog integration without maintaining a separate access layer. The thing we have struggled with is Genie. It can be hit or miss for complex questions where sometimes uses wrong window function logic. Also, the dashboards are limited in functionality compared to BI only tools. For example, you can't click on a bar in a chart and have it filter rest of the dashboard. It also sometimes has performance issues when dealing with large datasets.

I use databricks to run several machine learning jobs on a schedule. One of the most important jobs, is the integration runtime agent which copies data from our production db to the data warehouse. The reliability has been very good, but at times it has been a struggle to keep pacakges updated with the various containers that are used. It seems every few months an update causes one of our db scripts to become flaky and we need to patch it.