By Youssef Mrini · Senior Developer Advocate at Databricks6 min readNextGenLakehouse
November 2025 Release Highlights…
November 2025 Release Highlights
You can run Scala or Java JAR tasks on serverless compute as long as you build a Databricks‑compatible JAR using the right JDK/Scala versions and Databricks Connect.
The new MCP Servers tab centralizes management of managed and external MCP servers and lets you install curated servers from Databricks Marketplace.
Genie’s Research Agent adds multi‑step reasoning over Genie spaces to answer complex business questions.
Attribute‑based access control (ABAC) in Unity Catalog is in public preview and ABAC‑secured assets can be shared like standard tables.
The Databricks SQL Alerts UI has been revamped to streamline how you define queries, conditions, schedules, and notifications.
The TRUTH About Product Management & AI’s Future With David Meyer Databricks SVP
David Meyer is the Senior Vice President (SVP) of Products at Databricks, where he leads product strategy and execution for the company. In this role, Meyer oversees the development and direction of Databricks’ core offerings, including its Data Intelligence Platform, the Lakehouse architecture, and AI-driven features.
✨Tips and Tricks
Did you know you can use the inline Databricks Assistant with Cmd+I to iterate on just the code below your cursor (or a highlighted block) without rewriting the whole cell? This speeds up micro-edits and refactors.
For better results, @-mention tables and keep Unity Catalog descriptions and column comments up to date; the Assistant uses metadata context to generate more accurate SQL and code examples.
The Assistant has a “Find tables related …” action that puts it in table search mode; selecting “Query in natural language” explicitly binds the next query to your chosen table for precise answers
🛠️Data Engineering
You can run JAR Tasks on Serverless Compute
Use JAR task to deploy Scala or Java code compiled into JAR. Make sure to create a Databricks compiled JAR. 📖 Documentation
Lakeflow updates
Lakeflow Declarative Pipelines has been renamed to Lakeflow Spark Declarative Pipelines.
SFTP Connector extends autoloader features to ingest files from SFTP servers.
You can schedule when full refresh snapshots occur and enable automatic full refresh to recover from unsupported schema changes. This applies to managed database connectors like SQL Server in Lakeflow Connect. 📖 Documentation
Convert Foreign tables to managed tables
You can convert foreign tables from external catalogs (such as Hive Metastore or AWS Glue) to Unity Catalog managed while retaining table history and configurations. Use ALTER TABLE SET MANAGED to convert to managed tables with an optional DRY RUN validation. 📖 Documentation
🖥️Platform
You can disable DBFS root and mounts
Once your file-based workflows migration to Unity Catalog volumes is completed you can prevent users from uploading modifying or accessing data in DBFS root and DBFS mount. 📖 Documentation
Audit Logs for request-for access events are emitted
Improved experience with tag search
You can search by tag and value => tag:<tag_key>:<tag_value>
JDBC Unity Catalog connection
You can use a JDBC Unity Catalog connection to read and write to a data source with the Spark Data Source API or Databricks Remote Query SQL API. 📖 Documentation
Why should I use it ?
Governed access to data sources using Unity Catalog connections
Create the connection one time and reuse it across any Unity Catalog compute type: serverless, standard clusters, dedicated clusters, and Databricks SQL
Stable for Spark and compute upgrades
Connection credentials are hidden from the querying user
New jobs triggers available
Table update triggers for jobs support views as a source. 📖 Documentation
Automatic email reminders for expiring personal access tokens
Databricks sends automatic email notifications to workspace users approximately 7 days before their personal access tokens expire. 📖 Documentation
You have more flexibility to size Databricks Apps
You can configure the compute size to control CPU, memory and cost based on your workloads requirements. 📖 Documentation
Git CLI Commands in Git folders
You can run standard Git commands directly from the Databricks web terminal in Git folders. 📖 Documentation
Improved collaboration in notebook cells, files and SQL Editor
Real-time collaboration is available in notebook cells, files, and the new SQL editor. Multiple users can edit the same cell simultaneously and can view each other’s edits. 📖 Documentation
🤖GenAI & ML
Function calling and structured output is available on Qwen Next instruct Model
Improvements on the MCP Server
The new MCP Servers tab in the workspace shows your managed and external MCP servers, and a curated list of MCP servers available to install from Databricks Marketplace. 📖 Documentation
The Marketplace includes listing MCP Servers
New available Models on Databricks
OpenAI GPT-5.1 and Google Gemini Pro 3 are available as a hosted models allowing you to access them through:
Foundation Model APIs in pay‑per‑token mode
Query reasoning model endpoints
Query vision model endpoints
Batch inference workloads using AI Functions
📝AIBI Genie
You can connect Genie Spaces to Microsoft Copilot. 📖 Documentation
Increased SQL snippets and JOIN relationship limit: The limit for SQL snippets and JOIN relationships in a Genie space has been increased to 200.
Research Agent starter questions: Research Agent displays starter questions to help users begin their data exploration.
Auto-suggested benchmarks: Genie automatically suggests benchmarks based on the space context to help authors build evaluation sets.
Genie Create and Update APIs: Genie Create and Update APIs have been released. Get API has been updated to allow users to retrieve a serialized definition of the Genie space. 📖 Documentation
Research Agent in Beta: Research Agent extends Genie’s capabilities to help you uncover deeper insights and tackle complex business questions using multi-step reasoning and hypothesis testing. 📖 Documentation
Improved column selection: Genie optimizes column selection to provide more columns with higher relevance while efficiently managing context consumption.
Text explanations for empty results: Genie includes text explanations of query results when query results are empty.
Improved auto-generated charts: Visualizations use query result metadata for higher-quality auto-generated charts.
Improved date formatting: The process for formatting date results in visualizations has been improved.
📊AIBI Dashboard
Databricks AIBI Caching explained
For more information read the medium article by Andrea Picasso.
Sort by hidden measure fields: Users can now sort charts by a different measure field that does not visually appear on the bar chart. 📖 Documentation
Counter visualization improvements: You can use counters to track and compare metrics over time and show sparklines. To create a counter with a sparkline, select a main value and a date column. 📖 Documentation
Configure filter application behavior: Dashboards support applying multiple filters at the same time alongside the default instant application. 📖 Documentation
Draft dashboards no longer displayed: Only published dashboards are visible in Databricks One.
Enhanced search functionality: Users can search over dashboard names, page names, widget titles and descriptions, dataset names, and their queries.
Ask Genie button for embedded dashboards: When viewers access dashboards configured with basic embedding, they can access the Ask Genie button for dashboards published with an associated Genie space.
Per-widget filter enhancements: Per-widget filters support date ranges, relative date ranges, and number filters.
Metric view metadata support: AI/BI dashboards consume Unity Catalog metadata from metric views, including formatting and display name.
Horizontal chart scrollbars on hover: Scrollbars for horizontal charts now show on hover.
🛡️Governance and data sharing
Attribute-based access control is now public preview
SAP BDC Connector for Databricks supports Cross cloud sharing
You can share data between different cloud environments using the SAP Business Data Cloud (BDC) Connector for Databricks.
ABAC secured assets can be shared
You can share a table or schema secured by ABAC policies like standard tables.
Delta Sharing views permitted in Serverless network policies
Delta Sharing views are permitted when using serverless network policies, regardless of storage location alignment between shared tables and view dependencies. 📖 Documentation
🔍 Data Warehousing
The DBSQL Alerts experience has been revamped. 📖 Documentation
