Exploring the Future of Data and AI: Insights from the Data + AI Summit 2024 by Databricks

From June 10 to 13, Databricks took over the Moscone Center in San Francisco from Snowflake to present all its new features. Even though we were not physically present, our Specialist, Helle Kogsbøll Leerberg, was ready online for the live-streamed keynotes! 

In cooperation with our other Intellishorians, she has gathered some of her favorite highlights, and you can read and learn more about the most exciting ones! Read more below.

Serverless compute

at the 2024 Data + AI Summit, Databricks unveiled its serverless compute feature, designed to simplify data operations by removing the need for manual cluster setup and optimization. This innovation allows users to concentrate on their data tasks while Databricks handles infrastructure concerns, improving efficiency, reducing costs, and enhancing reliability by only charging for actual usage.

With serverless compute, Databricks enhances security and performance through automatic data layout optimization and machine learning-driven improvements. This feature aligns with Databricks’ mission to make data and AI more accessible and user-friendly. Additionally, Databricks signaled that future features might be exclusively available on the serverless platform, encouraging current users to transition to this new model.

For further details, you can explore the Databricks serverless documentation.

Databricks LakeFlow

consisting of Connect, Pipelines and Jobs specifically for ingestion, transformation and orchestration. While Pipelines build on the existing Delta Live Tables and Jobs build on the existing Workflow Jobs, the ingestion element “LakeFlow Connect” is a completely new addition. LakeFlow Connect is currently in Private Preview but will include connectors for databases including MySQL, Postgres, SQL Server and Oracle and enterprise applications like Salesforce, Microsoft Dynamics, NetSuite, Workday, ServiceNow and Google Analytics.

Read more here: Introducing Databricks LakeFlow: A unified, intelligent solution for data engineering | Databricks Blog.

Federated Lakehouse

will soon be going GA. The Federation Lakehouse enables companies to fetch data from external data sources, which means that in companies where they might have multiple Data Platforms, they are able to query data from all sources using the Federated Lakehouse. Today, supported data sources are MySQL, PostgreSQL, Amazon Redshift, Snowflake, Microsoft SQL Server, Azure Synapse (SQL Data Warehouse), Google BigQuery, and Databricks. 

Read more here: What is Lakehouse Federation | Databricks on AWS.

Data Clean Rooms

now works across clouds. Data Clean Rooms enables companies to do cross company data sharing for collaborative projects. This means that two departments in different companies are able to share and collaborate on selected data in a safe environment, now independently of whether they are working on different cloud setups.

Read more here: What Is a Data Clean Room? | Databricks.

Delta Lake 4.0

will include many new things, some interesting highlights for us were:

• Delta UniForm going GA. Today, Apache Hudi, Delta Lake, and Apache Iceberg all use parquet files but with each their own metadata layer. Delta UniForm can work with all three metadata layers and thus ensures interoperability. Read more here: Delta UniForm: a universal format for lakehouse interoperability | Databricks Blog.
• Open Variant Data Type has been announced. The Variant data type is created for semi-structured data such as json and enables faster processing of json compared to if it was stored as a string. Read more here: Introducing the Open Variant Data Type in Delta Lake and Apache Spark | Databricks Blog.

Unity Catalog

was published as an Open Source GitHub repo live at the conference.

See the repo here GitHub – unitycatalog/unitycatalog: Open, Multi-modal Catalog for Data & AI or read more here: Open sourcing Unity Catalog, creating the industry’s only universal catalog for data and AI | Databricks Blog.

Unity Catalog Metrics

were announced, which will enable defining metrics directly in Databricks. Third-party metrics are supported from e.g. dbt, and metrics can be accessed both internally from Databricks, but also through third-party BI tools like Power BI. Read more here: Hubert Dudek Post | LinkedIn