Learn Databricks — video guides, tutorials & interviews

A searchable library of answer-first write-ups for every NextGenLakehouse Databricks video, organized by topic: Unity Catalog governance, Lakeflow data engineering, Delta Lake, Databricks SQL and AI/BI, Genie, generative AI and agents, Lakebase, Databricks Apps, and the wider Databricks Data and AI Platform. Each guide has a timestamped transcript and links to the official Databricks documentation.

169 guides across 11 topics · curated by Youssef Mrini, Senior Developer Advocate at Databricks · updated October 2026

Data Engineering & Lakeflow

22 guides

Structured Streaming for Real-Time Data EngineeringLearn how to transition from batch pipelines to production real-time systems using Apache Spark Structured Streaming.
Build Data Pipelines Without Code—Lakeflow Designer DemoBuild production-ready data pipelines without code. Lakeflow Designer combines visual simplicity with AI intelligence and native Databricks governance.
Build Pipelines with Spark Declarative Pipelines (Open-Source)Spark Declarative Pipelines (SDP) is now open-source in Apache Spark 4.1 and powers Databricks' pipeline platform, Lakebase sync tables, Materialized…
Parse FHIR Healthcare Data with VARIANT & Lakeflow in DatabricksHealthcare organizations need to integrate FHIR (Fast Healthcare Interoperability Resources) JSON data from medical records, but traditional tools like…
Chuck Data: Automate Data Engineering with Natural LanguageChuck Data from Amperity: AI-powered data engineering agent that automates complex workflows in Databricks using natural language — no SQL required.
Enterprise Data Ingestion & Orchestration with LakeflowLakeflow is a unified data engineering platform for ingestion, transformation, and orchestration.
No-Code Data Pipelines with AI: Lakeflow DesignerLakeflow Designer is an AI-powered, no-code visual ETL builder for data engineers and business analysts.
Delta Live Tables 2026: Multi-Catalog, Kafka Sinks & Cost OptimizationDelta Live Tables (DLT) ships five major updates in 2026: multi-catalog writes, Kafka sink support, dramatically faster builds, enhanced error…
Databricks Workflows: Table Triggers, for-Each & ServerlessExplore the latest features in Databricks Workflows, the enterprise orchestration engine for ETL, analytics, and ML pipelines.
Databricks Data Engineer: Career Path & Best PracticesLearn how to become a Databricks data engineer: career insights from Daniel Sahal, a 10-year platform veteran who transitioned from Excel and on-prem…
DQX: Lightweight Data Quality Framework for Spark PipelinesExplore DQX, a lightweight open-source data quality framework from Databricks Labs, designed for Spark pipelines and streaming workloads.
Laktory: Terraform + dbt + Asset Bundles in One FrameworkLaktory is an open-source DataOps framework that consolidates Terraform, Databricks Asset Bundles, and dbt into a single, unified YAML-based platform…
Rivery ELT: 200+ No-Code Connectors for DatabricksRivery is a modern ELT (Extract, Load, Transform) platform with 200+ no-code connectors for SaaS ingestion.
Lakeflow: Unified Data Engineering PlatformLakeflow is Databricks' unified data engineering platform that simplifies ingestion, transformation, and orchestration through three integrated…
Future of Data Engineering: Declarative Streaming + AIMichael Armbrust (Databricks Distinguished Engineer) explores the evolution of data engineering: from Hadoop's manual code to Spark SQL's declarative…
Lakeflow & Unity Catalog: DAIS 2024 AnnouncementsLakeflow brings declarative ETL with SQL generation, Unity Catalog goes open source, Delta Lake 4.0 adds Liquid Clustering, Genie enables…
Databricks Data Engineering: Practitioner Perspective at DAIS 2024Databricks data engineering execution beats competitors, but product cohesion matters as the platform scales.
Ingest Salesforce & SAP to Databricks in 15 minutesWatch a live demo of data ingestion at scale. In 15 minutes, Kelly Connors from Fivetran walks through setting up a Salesforce connector and an SAP ERP…
Monitor and Automate Databricks Pipelines with OrchestraLearn how Orchestra orchestrates Databricks data pipelines end-to-end. In this demo, see how to define DAGs (directed acyclic graphs) with task…
Optimize Spark Workloads: Shuffling, Spilling & Skew FixesMaster Apache Spark performance optimization with techniques for shuffling, spilling, and data skew.
Rivery & Databricks: No-Code Data Integration for TeamsLearn how Rivery, a Databricks Partner Connect data integration platform, brings 50+ pre-built source connectors to Databricks: no-code ELT with SQL…
Debug Spark Locally on Databricks: Connect V2Write and debug Spark code in your local IDE while executing on Databricks clusters using Databricks Connect V2.

Delta Lake & Storage

13 guides

Stop Manual Tuning: Predictive Optimization ExplainedPredictive Optimization automatically tunes your Databricks data tables. Instead of manually running OPTIMIZE, VACUUM, LIQUID CLUSTERING, and ANALYZE…
Delta Lake 4.0: Connect, Clustering, Variant Type, Type WideningDelta Lake 4.0 introduces six major features that transform schema management, data organization, and client connectivity for the lakehouse.
SET MANAGED: Convert External Tables with Predictive OptimizationSeamlessly convert Unity Catalog external tables to managed tables with SET MANAGED.
Mastering Delta Lake: Liquid Clustering, Time Travel & VacuumingLearn 14 hands-on Delta Lake features: shallow & deep clones for instant data snapshots, deletion vectors for merge-on-read performance, liquid…
Liquid Clustering & Delta Lake Optimization: Author's Best PracticesExplore Delta Lake's evolution with Scott Haines, Databricks MVP and author of the free Delta Lake Definitive Guide. Learn when to use liquid clustering vs.
Apache XTable: Write Once, Read in Delta, Iceberg, or HoodieApache XTable translates table format metadata across Delta Lake, Apache Iceberg, and Apache Hoodie without data duplication—write once in your chosen…
Delta Lake 4.0: Liquid Clustering & Cross-Format EvolutionTathagata Das, Staff Software Engineer at Databricks and core Delta Lake developer, discusses the evolution of Delta Lake, its unprecedented scale…
Delta Lake 4.0, Metrics Layer, and DAIS 2024 AnnouncementsFranco Patano, Databricks field engineer, recaps DAIS 2024 announcements: Delta Lake 4.0 GA, Unity Catalog metrics layer launching later in 2026, and…
Databricks Lakehouse: Delta Lake & SQL Warehouse GuideDatabricks SQL is a serverless data warehouse on the Lakehouse platform. Learn to set up warehouses, query Delta Lake data, govern with Unity Catalog,…
Delta Lake Optimization: Z-Ordering, MERGE Tuning & VACUUMMaster 7 Delta Lake optimization techniques to improve query performance: file layout (Z-ordering, bin-packing), partitioning, statistics collection,…
Hands-Off Table Maintenance: Databricks Predictive Optimization SetupEnable Predictive Optimization to automatically VACUUM and OPTIMIZE your Databricks Delta tables without manual intervention.
Replace Partitioning with Delta Lake Liquid ClusteringDelta Lake Liquid Clustering simplifies data layout optimization by replacing traditional partitioning and ZORDER.
Recover Deleted Tables: Databricks UndropRecover accidentally deleted Databricks managed tables using the UNDROP command. Learn permissions requirements, how to list dropped tables, and…

Unity Catalog & Governance

30 guides

Databricks SQL & AI/BI

23 guides

Atomic Transactions in Databricks: BEGIN ATOMIC & Multi-Table CommitsMaster multi-table transactions in Databricks with atomic commits. Learn how BEGIN ATOMIC and BEGIN TRANSACTION ensure data consistency across multiple…
How to Use Temporary Tables in Databricks SQLTemporary tables in Databricks SQL are session-scoped, materialized result sets that optimize complex calculations reused multiple times.
Databricks SQL: ETL Migration Strategy with Franco Patano (Part 2)Learn 3 ETL migration strategies for Databricks: lift-and-shift, minimal refactoring, or declarative pipelines.
AI/BI Dashboard Caching: 100K Rows, Filters vs. ParametersAI/BI dashboards with slow query performance destroy user trust and adoption. This tutorial unpacks the two-level caching architecture that determines…
Collations: Case-Insensitive SQL & Legacy Data Warehouse MigrationLearn how to use Collations in Databricks SQL to sort, compare, and optimize text data—with support for case sensitivity, accents, language-specific…
Spatial SQL in Databricks: ST_Point, ST_Contains, ST_DistanceMaster geospatial data analysis using Databricks SQL's native Spatial SQL functions.
Databricks SQL Performance: 25% Gain, Free Edition, Warehouse MigrationFranco Patano (Principal Solutions Architect, Databricks) recaps DAIS 2026 and the convergence of Databricks One (unified front-door), AI/BI (95%…
Getting started with recursive CTE in Databricks SQLMaster Recursive CTEs in Databricks SQL — a powerful feature that makes SQL Turing-complete, enabling you to solve complex algorithms directly in SQL.
Getting started with Stored procedures in Databricks SQLMaster Stored Procedures in Databricks SQL — the latest procedural SQL feature now in public preview.
Define Business Metrics Once, Use Everywhere with UC MetricsUC Metrics brings semantic layer capabilities to Unity Catalog, allowing you to define business KPIs once and use them consistently across SQL queries,…
Migrate PL/SQL to Databricks: SQL Scripting Procedural Language GuideDatabricks SQL Scripting lets SQL developers write procedural logic without learning Python or Spark.
Databricks SQL Cost Attribution: Track Costs per QueryLearn to track Databricks SQL costs at query-level granularity using a Databricks-provided dashboard template.
DBSQL Warehouse Advisor: SLA-Based Optimization & Cost TuningLearn to optimize Databricks SQL warehouse performance and cost using the Warehouse Advisor Dashboard.
Databricks SQL: From $0 to $400M in 2.5 YearsInside Databricks SQL: How Reynold Xin and the lakehouse platform redefined data warehousing from $0 to $400M ARR in 2.5 years—and what's next.
VARIANT Data Type: Query Semi-Structured JSON in DatabricksLearn how to query semi-structured JSON data with Databricks VARIANT data type—no schema definition required.
Power BI Performance Tuning for Databricks SQLMaster three Power BI performance optimization techniques for Databricks SQL.
Data Warehouse Migration: 4-Step Databricks StrategyMigrate your data warehouse to Databricks: a 4-step strategy from legacy to lakehouse.
Simplify Databricks Analytics with AtScale Semantic LayerLearn how AtScale creates a semantic layer on Databricks Lakehouse to simplify BI and self-service analytics.
Real-Time Data Warehousing: Streaming Tables in Databricks SQLDiscover how Databricks SQL evolved into a real-time data warehouse with 16x performance gains.
Lakeflow View Dashboards: Self-Contained Models & Secure SharingLearn how Lakeflow View Dashboards evolved Databricks' native BI platform: self-contained data models instead of query references, secure…
ThoughtSpot Natural Language Search on Databricks DataLearn how to connect ThoughtSpot to your Databricks data and use natural language to ask questions without writing SQL.
Build Interactive Dashboards with AI/BI Dashboards: No-Code BIBuild interactive dashboards connected to Unity Catalog data—no SQL or code required.
Master Lakehouse Federation: Step-by-Step TutorialLearn the essentials of Lakehouse Federation in this 08:18 tutorial. Discover how to get started, configure the feature, and apply it to real use cases.

Genie & Conversational Analytics

12 guides

Genie ZeroOps: Find & Fix Data Pipeline Issues in MinutesGenie ZeroOps is an autonomous background agent that detects, diagnoses, and proposes fixes for data pipeline and ML model failures—in minutes, not hours.
Master the Genie Agent: Setup, Optimization & Best PracticesLearn how to master Genie Agent in Databricks—the conversational AI interface that empowers business users to ask natural-language questions of their…
Building Intelligent Event Discovery Agents with Databricks GenieWatch a real multi-agent workflow on Databricks: an event-discovery assistant for marketing that pairs Databricks Genie for governed, natural-language…
Enterprise Dashboards with Metric Views & Genie AnalyticsBuild data-driven dashboards with AI assistance, reusable metrics, and conversational analytics.
Genie Deep Research: Open-Ended Business AnalysisGenie Deep Research transforms open-ended business questions into actionable insights without SQL.
AI/BI Genie Goes GA: Deep Research, Knowledge Store & MoreLearn explore GA release and new features including Deep Research Mode and Knowledge Store.
Chat Your Way to Cost Visibility with Genie AgentsMaster Genie + System Tables for cost analytics. Numan Ali (Solutions Architect, Databricks London) demonstrates how to build a natural language cost…
Genie Agents Setup: Metadata, Sampling & Accuracy TuningConfigure Genie Agents to answer business questions accurately: master metadata optimization, data sampling, and iterative refinement workflows.
Upload CSV to Genie: Explore Data with Natural LanguageLearn how to upload CSV and Excel files directly to Genie Agents, configure a data room, and ask natural language questions about your data without…
AI/BI Dashboards & Genie: Databricks AI-Powered AnalyticsMaster Databricks AI/BI Dashboards: a complete guide to low-code, AI-powered analytics with built-in Genie conversational interface.
Genie Text-to-SQL: Compound AI for Data ExplorationGenie uses a compound AI architecture to convert natural language questions into accurate SQL queries, making data exploration accessible to business…
Genie & AI/BI: Conversational Analytics ExplainedGenie is Databricks' conversational AI that answers data questions in natural language. AI/BI Dashboards provide embedded analytics.

Machine Learning & MLflow

3 guides

Gaming Anti-Cheat: Real-Time ML with Carly Taylor & Joe ReisLearn how Databricks powers real-time ML for gaming anti-cheat detection, centralized governance, and unified data pipelines.
Databricks + Salesforce: Unified CDP with ML Churn PredictionWatch how Databricks + Salesforce create a unified customer data platform: train ML models in Databricks, predict churn in real-time, and empower sales…
Monitor Data Quality, Drift & Model Performance UnifiedMonitor your entire data pipeline—data quality, features, and ML models—without additional tools.

Generative AI & Agents

15 guides

Omnigent: Multi-Agent Orchestration & Real-Time CollaborationOmnigent is an open-source meta-harness for AI coding agents, providing control, composition, and real-time collaboration capabilities missing from…
Extract and Structure Enterprise Documents: AI Functions on DatabricksExtract and structure messy enterprise documents in production pipelines using AI Functions on Databricks.
Kasal: Databricks Labs Low-Code Agent Framework with Genie IntegrationKasal is a Databricks Labs field engineering project enabling non-technical users to build multi-step AI agents in natural language.
Agent Bricks & AI Platform: Patrick Wendell on Databricks GenAIPatrick Wendell shares how Databricks is unifying GenAI, classic ML, and online serving through Agent Bricks, Lakeflow Designer, and Genie.
Bauplan Labs: Serverless Python Data Pipelines & AI AgentsBauplan Labs is a serverless data platform that replaces complex Apache Spark deployments with Python functions as a service.
Databricks Data Science Agent: Auto-Generate & Execute AnalyticsDatabricks Assistant just got smarter. The new Data Science Agent autonomously generates code, executes SQL, and builds visualizations based on natural…
Agent Bricks Demo: Building and Optimizing AI AgentsLearn pre-built AI agent components for Knowledge Assistant and Supervisor patterns.
Databricks MVP: Data Engineering, GenAI & Governance Best PracticesLearn Databricks best practices from Rajaniesh Kaushikk, a 25-year veteran data professional, Databricks MVP, and head of Databricks practice at…
Semantic Layers: Why LLMs Need Them for Accurate QueriesDavid Mariani (CTO, AtScale) explains semantic layers and why they're critical for LLM accuracy.
SAS to Databricks: Real Migration Costs & LLM Hybrid ApproachThe true cost of enterprise legacy system migration is often $3–10M, proportional to the system's complexity and build history.
Building AI Agents with Unity Catalog Tools in DatabricksBuild AI agents that reason over your enterprise data using Databricks Agents, Unity Catalog tool definitions, and MLflow tracing.
How to Train Custom LLMs: From Prompting to Pre-TrainingLearn how to train custom LLMs: follow the maturity ladder from prompting to fine-tuning to pre-training.
SQL AI Functions: Sentiment, Entity Extraction, ClassificationLearn how to use Databricks AI functions to add natural language processing to SQL queries without writing Python or ML code.
SQL LLMs with AI_QUERY: Sentiment Analysis & ExtractionBring LLMs directly into SQL queries using the AIQUERY function in Databricks SQL.
Vector Search and RAG Chatbots with Databricks at Microsoft IgniteDatabricks CEO Ali Ghodsi demonstrates Vector Search, federated governance, and RAG chatbots at Microsoft Ignite 2023.

Lakebase (Operational Database)

4 guides

What Is LTAP? Lakebase + Genie, Explained by Databricks CTO Matei ZahariaLTAP is a Databricks architecture that runs your operational database (Lakebase, serverless Postgres) and your analytics engine (Spark) on a single…
Databricks DAIS: Top 3 Announcements—Lakebase, Agent Bricks, OneDatabricks MVP Josue Bogran ranks the top product announcements from DAIS 2026 and their real-world impact: Databricks One (business user front-end),…
DAIS 2025 Recap: Lakebase, Genie & Free EditionCommunity highlights from Data+AI Summit 2025. Explore Lakebase’s operational breakthrough, Genie integration with Slack for natural language queries,…
Lakebase: 19,000 Transactions Per Second, 4.56ms LatencyLakebase is a serverless PostgreSQL OLTP database engine that powers real-time Databricks Apps at enterprise scale.

Databricks Apps