April 2025 Release Highlights…
April 2025 Release Highlights
Salesforce and Workday connectors are GA
Google Analytics raw data connector is now public preview
Salesforce ingestion and Workday reports connectors are GA
Google Analytics raw data connector is now public preview
Access UDF context information using TaskContext
The TaskContext Pyspark API allows to retrieve context information like identity and cluster tags while running BATCH UNITY CATALOG PYTHON UDF or PYSPARK UDF. Documentation
Batch Unity Catalog Python UDFs
Unity Catalog batch python UDFs extend the capabilities of Unity Catalog UDFs by allowing you to write python code to operate on batches of data. Batch python UDFs support service credentials to access external cloud services. Documentation
Run a subset of tasks within a job
You can run a subset of tasks when manually triggering the job.
Multi file editor for ELT pipeline development in DLT
You can now use the new multi-file editor in DLT to develop and debug ETL pipelines. The multi-file editor shows a pipeline as a set of files in the pipeline assets browser. You can edit the files and control the configuration of the pipeline and which files to include in one location. Documentation
Delta Live Tables enhancements
All streaming tables and materialized views are moving to use predictive optimization for maintenance tasks.
New streaming tables and materialized views will follow the workspace settings for deletion vectors
Share STs and MVs to all recipients using Delta sharing
You can now use Delta Sharing to share streaming tables and materialized views to all recipients.
Enforcement of RLS/CLS in Delta Sharing
Delta Sharing now consistently enforces row-level security and column masking policies applied to tables a shared data asset is dependent on, of whether those policies were applied before or after the data asset was shared. Recipients may experience differences in query behavior when accessing shared data that is depending on tables with row-level security or column masking policies. This ensures that data access aligns with the provider's intended security controls at all times.
Browse privilege is GA
The BROWSE privilege in Unity Catalog is crucial for enhancing user data discovery capabilities. This privilege enables users to view metadata without having direct access to the underlying data, thus facilitating the exploration of data lineage and object relationships without compromising data security.
Compute System tables are GA
Compute system tables: Clusters, node_types and node_timeline are GA.
Databricks MFA is now available
Account Admins can enable MFA in Databricks. This feature is only available for accounts where SSO is not enabled. Documentation
Databricks assistant improvements
In answer to prompts and queries, assistant now provides suggestions for next steps and citations from documentation. The assistant can also run any code
Reference SQL output in downstream tasks of a job
You can use dynamic values to reference the output of a SQL task in downstream tasks in the same job. For each tasks can iterate over the rows of data in the output. Documentation
Use vector search indexes to prototype tool-calling generative AI agents in AI Playground
🔗Watch the Demo video
Rather than creating custom Unity Catalog functions with a vector search index, you can now add the vector search index to your gen AI agents directly in AI Playground. The agent will be able to use the vector search index as a tool to help it respond to queries. See Prototype tool-calling agents in AI Playground.
AI Gateway now supports custom model serving endpoints
You can now enable the following governance and monitoring features on your custom model serving endpoints:
Permission and rate limiting to control who has access and how much access.
Payload logging to monitor and audit data being sent to model APIs using inference tables.
Usage tracking to monitor operational usage on endpoints and associated costs using system tables.
Llama 4 Maverick is now available as a Databricks-hosted foundation model.
You can use it for batch inference endpoints optimized for AI Functions.
AI Builder: Model specialization for custom gen AI text agents is in Beta
🔗Watch the Demo video
AI Builder provides a simple, no-code approach to build and optimize domain-specific, high-quality AI agent systems for common AI use cases. In Beta, AI Builder supports model specialization for custom text-based tasks, such as summarization, classification, text transformation, and content generation.
See Use AI Builder: Model specialization to create a gen AI agent for text.
Build GenAI Focused Front End Applications on Databricks with Ivan Trusov
In this session, we cover the fundamentals of Databricks Apps, deployment best practices and supported frameworks. The highlight is a live demo of a real-world Retrieval-Augmented Generation (RAG) chatbot with advanced PDF highlighting capabilities.
End of support for legacy dashboards
Legacy dashboards can no longer be created or cloned via the UI or API. Databricks will continue to address critical security issues and outages, but the recommendation will be to use AI/BI dashboards for new development. Documentation.
Source control for dashboards
Dashboards can now be integrated into Git folders. Documentation.
Streamlined publish flow
Publishing a dashboard now embeds the user’s warehouse permissions, removing the previous requirement of end users requiring individual access to the selected warehouse. Credentials to the data will still need to be specified.
Sankey Visualisations are now available on AI/BI dashboards. Documentation.
Genie Conversations on AI/BI dashboards now resumes the last conversation when opened until a new one is started.
Custom Dimensions
Authors can now create custom dimensions which allows a new set of functions including CASE WHEN statements, CONCAT, COALESCE to be supported. Documentation.
The Genie Conversations API is now in available in public preview. Documentation.
Example and follow up questions
Genie now uses table metadata to offer example and follow up questions
Improved data access transparency
Users can now view and remove missing tables in a Genie space. The Data tab now shows visual indicators to mark tables that cannot be accessed. You can use the UI to remove those tables.
Revamped feedback flow
Genie feedback workflow is now more interactive and responsive. When a response is incorrect, you can explain why it's wrong by choosing from a selection of common issues or entering your own explanation. Documentation.
Write procedural SQL scripts based on ANSI SQL/PSM (Public Preview):
You can now use scripting capabilities based on ANSI SQL/PSM to write procedural logic with SQL, including conditional statements, loops, local variables, and exception handling. Documentation.
New H3 functions have been added:
Browse every issue of What’s New in Databricks, get source-backed Databricks Answers on the latest features, or watch the video guides — all curated by Youssef Mrini, Senior Developer Advocate at Databricks.
Prefer it in your inbox? Subscribe to the newsletter.