By Youssef Mrini · Senior Developer Advocate at Databricks6 min readNextGenLakehouse
Your long awaited Data+AI recap - part 2…
Missed the DA+AI Summit? Don’t worry, we've got you covered! Here is what you do not want to miss to stay up to date with the latest Data + AI! This is PART 2, make sure you read Part 1 for all the Data Engineering, Lakebase, UC, and more!
GenAI with Mosaic AI
Databricks has just unveiled a wave of innovation with Mosaic AI, redefining how organizations build, deploy, and govern production-grade AI and BI at scale. Let’s break down the highlights—matching your enterprise-forward, solution-focused style.
Agent Bricks: Auto-Optimized Agents for Production
Agent Bricks is a game-changer for enterprise agent development. Instead of painstakingly scripting logic or wrangling pipelines, just describe the agent’s purpose and connect your datasets in a few clicks!
Applications range from information extraction, custom knowledge assistants to multi-agent orchestration and robust enterprise copilots. Agent Bricks is already making it possible for companies to go from prototype to production in days—not weeks.
MLflow 3.0: Unified AI Observability and Governance
MLflow 3.0 reimagines agent and AI application lifecycle management for the GenAI era:
Universal tracing: Monitor, debug, and benchmark any model or agent—on Databricks or across clouds and on-prem.
End-to-end assessments: Built-in and customizable LLM judges, plus human-in-the-loop feedback, keep your agents compliant and continually improving.
Prompt registry: Version, test, and deploy prompt chains side-by-side with model artifacts.
MLOps for Agents: Complete integration with Unity Catalog for data lineage, governance, and audit trails. MLflow 3.0 transforms GenAI evaluation from guesswork into measurable business quality.
AI Functions in SQL: Lightning-Fast, Multi-Modal ai_parse_document
Mosaic AI’s SQL-native AI Functions now deliver:
Up to 3x faster and 4x lower cost data transformation at scale.
Multi-modal capabilities (text, images, documents) with functions like ai_parse_document, unlocking value from complex enterprise content instantly—not just structured tables.
Storage-Optimized Vector Search
Vector Search infrastructure now separates compute and storage for massive scalability and economics:
Easily scale up to billions of vectors for RAG, semantic retrieval, and more.
Achieve up to 7x lower cost versus previous architectures, removing the bottleneck for advanced search and copilot scenarios.
Serverless GPU Compute and High-Scale Model Serving
GPU-powered AI is now available in serverless: launch jobs or notebooks on A10G and soon H100s, with zero infrastructure burden.
Model Serving: Ultra-low latency and throughput exceeding 250,000 QPS, powered by an in-house LLM inference engine that outpaces open-source alternatives. Production LLM API on demand, globally.
Agent Ecosystem: MCP Support and AI Gateway
Model Context Protocol (MCP) is now native: Build and host MCP servers within Databricks Apps, and directly connect agents to Genie, Vector Search, and more—all tested in the Mosaic Playground.
AI Gateway: The one-stop entry for cross-service governance, safety, logging, and seamless fallback between providers.
AI/BI, Genie and Databricks One
Genie
Next-Generation Knowledge Store for Genie
Authors can now enrich data assets with semantic context—like column synonyms, joins, and certified metrics—directly in Unity Catalog. Genie also suggests new knowledge based on lineage and query history, helping teams build a living semantic model.
Automatic Knowledge Extraction
Genie extracts insights from every chat, proposing reusable knowledge snippets for user approval and storage.
Deep Research Mode
For complex “why” and “how” questions, Genie now creates research plans, tests multiple hypotheses in parallel, and summarizes results—enabling deeper, automated analysis.
Databricks One
One of the most exciting announcements! A new experience for non-technical users to access dashboards & Genie as well as Databricks Apps, without having to navigate the technical Databricks workspace.
More Value Dictionaries: Genie now supports up to 60 value dictionaries, improving accuracy through value sampling.
Consumer Entitlement GA: Admins can grant consumer access to users, service principals, and groups—enabling SQL warehouse queries, dashboard views, and limited workspace UI access.
Editable Filters: Users can now refine responses by editing filter values. For string columns with dictionaries, Genie intelligently offers editable filters when they improve precision.
New Genie API Endpoints (Public Preview): Programmatically list or delete conversations and spaces using the new Genie API.
AI/BI Dashboard
File upload to Unity Catalog from dashboards: Authors can now upload files to Unity Catalog directly from the dashboard data tab. See Create and manage dashboard datasets.
Customizable dashboard theme: Authors can now customize the dashboard theme by setting custom font, background, and widget colors. See Dashboard settings.
Improved diff review for dashboard files: When committing dashboard changes to Git, queries and text box content now appear across multiple lines in the diff, making changes easier to review.
Multi-statement queries in Genie on dashboards: Genie on dashboards now supports multistatement SQL queries.
Data Sharing
Databricks is making it easier to share assets securely and work across platforms.
Delta Sharing now gets full iceberg interoperability.
Cross-platform sharing to any iceberg-compatible engine, allowing data providers to reach their customers across platforms like Snowflake.
Data providers can now use Delta Sharing to share Iceberg Tables just like any other UC asset, both within Databricks and beyond
The new Delta Sharing network gateway removes the burden. Providers can share with recipients with minimal manual network configuration.
Delta Sharing’s Attribute-Based Access Control support allows providers to securely apply fine-grained data access policies when sharing data, ensuring sensitive information is protected while enabling broad data collaboration
Data providers can securely share data with non-Databricks users on any computing platform who prefer to authenticate using a custom Identity Provider (IdP), such as Azure Entra ID or Okta with OIDC Token Federation for Enhanced Delta Sharing Security
Cleanrooms
Databricks introduced an advancement in identity resolution. It does empower partners and markets to confidently and securely match and enrich datasets using common identifiers
Cleanrooms now support multiple collaborators in a single room
Apps
No more struggling with the “last mile” to turn AI and analytics into real-world business value. Databricks Apps are now GA, letting teams build, deploy, and scale secure, interactive, data-powered apps—all directly inside the Databricks Data Intelligence Platform.
Instant, Secure Production: Apps run in a fully managed, serverless environment, eliminating manual setup for authentication, containers, and governance. Security, access, and observability are built in—so you focus on building, not babysitting infrastructure.
Faster Innovation: Deploy LLM copilots, business dashboards, data quality monitors, or custom ops tools within the workspace, with governance and identity handled automatically. No more data duplication or risky exports.
Zero Platform Drag: Start building right now in your language of choice (Python, JS, and more). Push live apps to business teams in days, not months. Over 20,000 apps have already launched across 2,500+ organizations—proof that users want to bring their AI and analytics to life, without infrastructure headaches.
Unified with Your Data: Your apps run alongside all your governed, secure data—enabling richer, safer user experiences and blazing-fast analytics.
Marketplace: Your Open Platform for Data, Analytics, and AI
The Marketplace now goes beyond data—offering ML models, dashboards, notebooks, and applications, all instantly usable across Databricks or any Delta Sharing-compatible platform.
Built on Open Standards: Powered by Delta Sharing, the Marketplace enables secure, cross-cloud collaboration with no vendor lock-in or ETL overhead.
Faster Adoption with Prebuilt Solutions: Prebuilt notebooks and sample data help users quickly evaluate and implement data products with confidence.
Expanded Sharing Capabilities: Now with full Iceberg support and a new Delta Sharing network gateway, providers can securely share assets across any analytics engine. Attribute-Based Access Control ensures secure, fine-grained sharing—even outside Databricks.
That’s a wrap! Thanks for reading all the latest Data+AI announcements!
