Exploring the Future of Data and AI: Insights from the Data + AI Summit 2024 by Databricks

From June 10 to 13, Databricks took over the Moscone Center in San Francisco from Snowflake to present all its new features. Even though we were not physically present, our Specialist, Helle Kogsbøll Leerberg, was ready online for the live-streamed keynotes! 

In cooperation with our other Intellishorians, she has gathered some of her favorite highlights, and you can read and learn more about the most exciting ones! Read more below.


Serverless compute

at the 2024 Data + AI Summit, Databricks unveiled its serverless compute feature, designed to simplify data operations by removing the need for manual cluster setup and optimization. This innovation allows users to concentrate on their data tasks while Databricks handles infrastructure concerns, improving efficiency, reducing costs, and enhancing reliability by only charging for actual usage.

With serverless compute, Databricks enhances security and performance through automatic data layout optimization and machine learning-driven improvements. This feature aligns with Databricks’ mission to make data and AI more accessible and user-friendly. Additionally, Databricks signaled that future features might be exclusively available on the serverless platform, encouraging current users to transition to this new model.

For further details, you can explore the Databricks serverless documentation.

Databricks LakeFlow

consisting of Connect, Pipelines and Jobs specifically for ingestion, transformation and orchestration. While Pipelines build on the existing Delta Live Tables and Jobs build on the existing Workflow Jobs, the ingestion element “LakeFlow Connect” is a completely new addition. LakeFlow Connect is currently in Private Preview but will include connectors for databases including MySQL, Postgres, SQL Server and Oracle and enterprise applications like Salesforce, Microsoft Dynamics, NetSuite, Workday, ServiceNow and Google Analytics.

Read more here: Introducing Databricks LakeFlow: A unified, intelligent solution for data engineering | Databricks Blog.


Federated Lakehouse

will soon be going GA. The Federation Lakehouse enables companies to fetch data from external data sources, which means that in companies where they might have multiple Data Platforms, they are able to query data from all sources using the Federated Lakehouse. Today, supported data sources are MySQL, PostgreSQL, Amazon Redshift, Snowflake, Microsoft SQL Server, Azure Synapse (SQL Data Warehouse), Google BigQuery, and Databricks. 

Read more here: What is Lakehouse Federation | Databricks on AWS.

Data Clean Rooms

now works across clouds. Data Clean Rooms enables companies to do cross company data sharing for collaborative projects. This means that two departments in different companies are able to share and collaborate on selected data in a safe environment, now independently of whether they are working on different cloud setups.

Read more here: What Is a Data Clean Room? | Databricks.


Delta Lake 4.0

will include many new things, some interesting highlights for us were:

Unity Catalog

was published as an Open Source GitHub repo live at the conference.

See the repo here GitHub – unitycatalog/unitycatalog: Open, Multi-modal Catalog for Data & AI or read more here: Open sourcing Unity Catalog, creating the industry’s only universal catalog for data and AI | Databricks Blog.


Unity Catalog Metrics

were announced, which will enable defining metrics directly in Databricks. Third-party metrics are supported from e.g. dbt, and metrics can be accessed both internally from Databricks, but also through third-party BI tools like Power BI. Read more here: Hubert Dudek Post | LinkedIn

Excited for the year to come

Excited for the year to come

Joining the Data + AI Summit always brings lots of excitement for the year to come. Databricks has, for a long time, continuously evolved, and the platform we see today is remarkably different from the one we worked with just last year. The platform today supports a wide range of use cases, but I am really excited to see the changes to come, especially the ones that will relieve some concrete pain points in regard to my current work.

Above, you will find a selection of my personal highlights from this year’s Data + AI Summit. I hope you will be just as intrigued as I am.”

Helle Kogsbøll Leerberg, Specialist at Intellishore

Get in touch
Send us your details, and we will get back to you with more inspiration on how you can unleash more value to your organization by utilizing Databricks.

We look forward to hearing from you.

Need help getting started with Databricks?

We are proud of our accomplishments and are always ready to share experiences. Feel free to reach out to one of our consultants if you need help with how to start using Databricks in your organization.
Helle Kogsbøll Leerberg
Martin Kjær Dideriksen
Principal Consultant
Next Up
Brick by brick - Why Databricks kicks