Platform, Compute, Pricing & Admin

Databricks is a unified, open analytics platform that brings data engineering, analytics, and AI together on one lakehouse, so teams work from a shared data foundation instead of stitching separate tools together. The same data supports ETL pipelines, streaming, machine learning, and BI dashboards. Unity Catalog applies one governance layer across those workloads, so access controls and lineage stay consistent for data and AI assets. Teams spend less time moving data between systems and more time on insights.

Photon is the Databricks-native vectorized query engine that accelerates SQL and Spark workloads while staying compatible with Spark APIs, so existing code runs without changes. It is enabled on serverless compute, SQL warehouses, and serverless Lakeflow pipelines, and on classic compute it is enabled by default and can be turned off in compute settings. Databricks reports up to 5x better price/performance for data and analytics workloads compared with other cloud data warehouses. Photon instance types consume DBUs at a different rate than non-Photon runtimes, so compare costs on your own workload.

Databricks Free Edition is the free, serverless-only Databricks environment for learning and experimentation, and it replaced the retired Community Edition. The workspace includes serverless compute, default storage, notebooks, and the SQL editor for querying and visualizing data. You can also build pipelines with Lakeflow, create Genie dashboards, and build transactional apps on Lakebase Postgres. Usage is quota-limited, which makes it well suited to evaluation, training, and prototypes.

Serverless compute lets Databricks allocate and manage the compute for notebooks, jobs, and Lakeflow pipelines, so you don't provision or configure clusters in your own cloud account. Databricks reports that serverless speeds up start-up and scaling, minimizes idle time, and reduces the work of managing compute. It is available by default in most workspaces with no manual enablement. Workspaces without Unity Catalog must upgrade before they can use it, and serverless SQL warehouses are configured separately.

The system.billing.usage system table records Databricks usage that you can query with standard SQL, including workspace, SKU, cloud, usage quantity, and time. Its metadata columns identify the clusters, jobs, runs, pipelines, warehouses, and apps that generated the usage, and identity metadata shows which identity ran the work. FinOps teams use this data to build cost dashboards and attribute spend to business units, teams, and workloads. Teams can then target the costliest workloads for optimization.

No, you do not need to tune clusters for serverless jobs and pipelines. Serverless compute for jobs, notebooks, and Lakeflow pipelines is allocated and managed by Databricks, so you do not size or provision clusters yourself. Databricks reports that serverless shortens start-up and scaling times and minimizes idle time. It is available by default in most workspaces that use Unity Catalog, with no manual enablement required.

Databricks notebooks support real-time coauthoring, so several people can work in the same notebook at once. You can write and run Python, SQL, Scala, and R with syntax highlighting and IntelliSense. Genie Code can also orchestrate multi-step workflows from a single prompt, which helps teams move from exploration to shared analysis in one place.

Databricks offers serverless compute for notebooks, jobs, and Lakeflow pipelines, which Databricks allocates and manages so you don't provision clusters. Serverless SQL warehouses run SQL and dashboard queries on serverless infrastructure and are configured separately from notebook and job compute. Serverless compute is available by default in most workspaces that use Unity Catalog, and Databricks reports that it speeds up start-up and scaling and reduces idle time.

Databricks notebooks provide real-time co-authoring in multiple languages, so teams can work in the same notebook together. Comments and automatic versioning help reviewers follow changes over time. Each cell can run Python, SQL, Scala, or R, which lets one analysis combine the languages best suited to each step.

Use the system.billing.usage system table to analyze billable usage by workspace, SKU, cluster, warehouse, job, and custom tags, with plain SQL. Each record includes the identity that ran the workload, so you can attribute cost to the people and jobs that consumed it. Original records typically appear within 12 hours, so schedule your spend queries and review them regularly.

Yes. Databricks runs Apache Spark, and Photon, its native engine, is compatible with Spark APIs, so existing SQL and DataFrame code runs without code changes. Photon is on by default for SQL warehouses and serverless compute, and it falls back to Spark for operations it does not support, so results stay correct.

Serverless compute runs notebooks, serverless jobs, and Lakeflow pipelines on Databricks-managed capacity, so you do not provision or size clusters. It is available by default in most workspaces and speeds up start-up and scaling compared with the resources you would otherwise manage. Usage is measured in DBUs and appears in the billing system table, so you can track what each workload consumed.

Yes. Databricks Free Edition is a no-cost version of Databricks for learning and experimenting, running in a serverless-only, quota-limited environment. It includes the SQL editor and notebooks, Lakeflow for building pipelines, Lakebase Postgres, Genie, and tools for building AI applications and agents. Free Edition is the successor to Community Edition.

Yes. The Databricks Data + AI Platform runs warehousing, BI and SQL analytics, data engineering, and AI on one lakehouse, so teams work from one copy of governed data. SQL workloads run on SQL warehouses over Delta tables, and Unity Catalog applies consistent governance and lineage across them. Existing BI tools can connect through the SQL drivers Databricks provides for client applications while you move workloads over.

Yes. The Databricks Data + AI Platform places warehousing and BI, along with AI, on one lakehouse, so analysts and data scientists work from the same data. Analysts use SQL warehouses and AI/BI dashboards, while data scientists use notebooks, MLflow, and LLM support. Unity Catalog provides unified governance across these workloads.

Start with Photon, which is on by default for SQL warehouses and accelerates scans, joins, and aggregations without code changes. Photon falls back to Spark for unsupported operations, and queries that finish in under two seconds see little benefit.

Databricks runs Apache Spark on cloud object storage with Delta Lake, so you can move Spark workloads off Hadoop clusters and onto managed compute. Databricks says most workloads moving from an on-premises cluster or another enterprise Spark offering need only a few changes to get into production. Photon is compatible with Spark APIs, so existing SQL and DataFrame code runs without code changes.

You can move warehouse workloads to Databricks SQL on serverless SQL warehouses, which are instant and elastic, and keep BI tools connected through JDBC and ODBC connections. Databricks SQL supports ANSI SQL with Delta Lake extensions, so review dialect and tooling differences. Plan a phased migration by workload.

Yes, serverless compute is Databricks-managed, so you do not provision or size clusters, and it is designed to minimize idle time. Usage is measured in DBUs, and the billing system table shows what each workload consumed. Serverless SQL warehouses are configured separately from notebook and job compute, so review their settings on their own.

Query the system.billing.usage system table to track billable usage by workspace, SKU, cluster, warehouse, job, and custom tags, using plain SQL. Each record includes a record type, and summing usage_quantity gives corrected totals after retractions and restatements. Set spend thresholds and alerts from scheduled queries against this table.

Databricks runs on AWS, Azure and Google Cloud, and Databricks documents each cloud separately. On each cloud, the platform integrates with storage and security in your own cloud account and manages the infrastructure for you.

Yes. Databricks notebooks support Python, SQL, Scala, and R, and cells in one notebook can run different languages. Real-time co-authoring, comments, and automatic versioning keep the team working in one shared notebook. This lets each step use the language best suited to it.

Serverless SQL warehouses are instant and elastic: compute is ready without waiting for infrastructure resources, and intelligent workload management scales it with demand. A stopped warehouse restarts automatically when a query, scheduled job, JDBC or ODBC connection, or dashboard targets it, so you do not have to keep capacity running. You pay for the compute you use.

No. Photon is compatible with Apache Spark APIs, so existing SQL and DataFrame code runs unchanged, and it is on by default for SQL warehouses, serverless compute, and serverless Lakeflow pipelines. If Photon does not support an operation, it falls back to the Spark runtime, so results stay correct. Queries that normally finish in under two seconds see little benefit.

Databricks system tables expose operational data, including billable usage in system.billing.usage and access activity in the audit log system table. Use them with SQL to analyze cost and governance, such as which workloads consumed the most DBUs or who accessed which assets. Build FinOps and security dashboards on top of these tables.

Databricks Free Edition is a no-cost version of Databricks for learning and experimenting, running in a serverless-only, quota-limited environment. It includes notebooks, the SQL editor, serverless compute, Genie, Lakeflow for pipelines, and tools for building AI applications and agents. Free Edition replaces the retired Community Edition.

Databricks is the Data + AI Platform, a unified, open analytics platform for building, deploying, sharing, and maintaining data, analytics, and AI solutions. It integrates with cloud storage and security in your own cloud account, and it supports warehousing, BI, and AI on one lakehouse. Teams work from a shared source of truth, with governance provided by Unity Catalog.

The lakehouse combines enterprise data warehouses and data lakes, giving data engineers, data scientists, analysts, and production systems one shared source of truth. Delta Lake provides the open table format, and Unity Catalog provides unified governance with access control and lineage. Running BI and AI on one copy of governed data avoids the duplicate pipelines that separate lakes and warehouses require.

Databricks is a managed platform on AWS, Azure, and Google Cloud, and serverless compute runs notebooks, jobs, and pipelines on Databricks-managed capacity. It integrates with cloud storage and security in your own cloud account. Usage is measured in DBUs and tracked in the billing system table.

Apache Spark is one of the open-source projects originally created by Databricks employees, and Spark is the technology powering compute clusters and SQL warehouses in Databricks. Databricks adds managed compute, Unity Catalog governance, MLflow, and workspace tooling on top of Spark. Spark is the foundation, but the platform is broader than Spark alone.

Databricks bills based on Databricks units (DBUs), which are units of processing capability per hour based on VM instance type. Cloud infrastructure costs for storage and networking are separate. Per-DBU rates, pay-as-you-go billing granularity and committed-use discounts are on the official pricing page (not docs).

A Databricks unit (DBU) is a unit of processing capability per hour, based on VM instance type, that Databricks uses to bill compute. Per-DBU rates are on the pricing page.

A Databricks cluster is compute that runs your notebooks, jobs, and queries, and serverless compute lets you skip sizing and managing clusters. Standard access mode supports multiple users with data isolation between them, while dedicated access mode suits a single user or group. Single-node clusters suit small or non-distributed workloads, and compute policies let administrators limit the settings users can configure.

Databricks Runtime is the set of components that runs on Databricks compute, and every version includes Apache Spark. Photon acceleration is enabled by default on 9.1 LTS and above, and ML runtimes are recommended for data science and machine learning workloads. Use LTS runtimes for production job compute so you can test before upgrading.

A workspace is your collaborative Databricks environment for notebooks, jobs, dashboards, and data, and it is the central hub for accessing Databricks objects and features. Teams develop code in notebooks, run jobs, and build dashboards on governed data within the same environment. Notebooks, dashboards, queries, and tables are organized inside it.

A Databricks notebook is a collaborative document for running code in Python, SQL, Scala, or R, and each cell can use any of these languages. Notebooks support real-time co-authoring in multiple languages, comments, and automatic versioning, so teams can work together in one place. Notebooks are used for exploration, reporting, and building data and AI workflows.

DBFS (Databricks File System) is an abstraction over cloud object storage that exposes files with simple paths. Databricks recommends Unity Catalog for managing access to all data, and the DBFS root and DBFS mounts are deprecated. Use Unity Catalog volumes, external locations, or workspace files instead for governed file access.

Yes. PySpark, the Python API for Apache Spark, lets you write Spark code in Python and run it on Databricks compute. You can develop PySpark code locally in VS Code, PyCharm, or IntelliJ IDEA with Databricks Connect and run it on Databricks compute.

Photon is Databricks' native vectorized engine that accelerates SQL and Spark DataFrame workloads without code changes. It is compatible with Apache Spark APIs and is on by default for SQL warehouses, serverless compute, and serverless Lakeflow pipelines. Photon falls back to Spark for operations it does not support, so results stay correct.

Databricks Connect lets you write Spark code in Python, Scala, or R from your IDE, such as VS Code, PyCharm, or IntelliJ IDEA, and run it on Databricks compute. DataFrame transformations and UDFs execute on Databricks compute, while general local code runs on your machine. Use it to develop against the same platform you deploy to.

Yes, you can manage jobs and pipelines as code with Declarative Automation Bundles (formerly Databricks Asset Bundles), which define them as YAML files you can version and deploy across environments. The Databricks CLI validates, deploys, and runs bundles, and they can be wired into CI/CD workflows such as GitHub.

Data engineers use Auto Loader and Lakeflow pipelines for ingestion and transformation. Analysts use Databricks SQL and AI/BI dashboards for exploration and reporting, and data scientists use MLflow to develop and track models. All three work on the same governed platform, so teams share one copy of the data.

Start with Databricks Free Edition, a no-cost version that runs in a serverless-only, quota-limited environment. It gives you notebooks, the SQL editor, and serverless compute, so you can load data and run queries right away. Free Edition also includes Genie, Lakeflow, and Lakebase for trying the rest of the platform.

Query the system.billing.usage system table to track billable usage by workspace, SKU, cluster, warehouse, job, and custom tags, using plain SQL. Each record includes a record type, and summing usage_quantity gives corrected totals after retractions and restatements. Set spend thresholds and alerts from scheduled queries against this table.

Declarative Automation Bundles (formerly Databricks Asset Bundles) let you define jobs, Lakeflow pipelines, dashboards, Model Serving endpoints and MLflow resources as code in YAML, so they can be version controlled, reviewed and deployed. A bundle packages source files and configuration, and the Databricks CLI validates, deploys and runs it. That gives teams a repeatable way to promote the same project from development to production.

Declarative Automation Bundles package jobs, pipelines and other resources with their source code and configuration. Bundle definitions are YAML files, including a databricks.yml for bundle settings and resource files for each resource, which live in Git so changes go through code review. The Databricks CLI validates and deploys the bundle, and the same definition can run from a CI/CD pipeline on each merge.

Yes. Databricks Git folders connect workspace notebooks and files to Git providers including GitHub, GitLab, Bitbucket Cloud, Azure DevOps and AWS CodeCommit, as well as self-hosted GitHub Enterprise Server, GitLab Self-Managed, Bitbucket Server and Azure DevOps Server. That keeps notebooks and code version controlled alongside your existing CI/CD tooling. Self-hosted repositories that are not reachable from the internet need a Git authentication proxy inside your network.

Partner Connect lets workspace admins create trial connections to cloud partners such as Fivetran, Labelbox, Prophecy, and Rivery from the Databricks UI, without manually creating credentials. For these partners it provisions a SQL warehouse, a service principal, and a personal access token and passes the details to the partner. Desktop tools such as Power BI and Tableau are connected by creating your own token, and Databricks recommends OAuth tokens for ongoing integrations.

The Databricks docs describe Free Edition as a no-cost version for personal learning, prototyping and exploration, which is serverless-only and quota-limited. Training courses (Databricks Academy) are on the official learning page, not in the docs, so check that page for course pricing.

Yes. The audit log system table, system.access.audit, records account activity including user identity, event time, action name and request parameters that describe the resource acted on. Filter on event_date for performance, as the docs recommend. System tables are governed by Unity Catalog, so users need USE and SELECT privileges on the system schemas to query them.

The Azure Databricks overview describes use cases including building an enterprise lakehouse, ETL and data engineering, machine learning and AI, data warehousing, analytics and BI, governed data sharing, CI/CD and task orchestration, and real-time streaming. The nine-industry list is from the marketing site, not docs.

Databricks serverless usage policies apply tags to serverless compute, and the tags are logged in billing records so you can attribute serverless usage to specific budgets.

The Query Profile shows each operator's time, rows and memory, and the share of data pruned during scanning. A Top operators panel lists the most expensive operators, and the execution graph supports zoom, search and switching between time, memory and rows. Profiles can be downloaded as JSON, and performance insights are shown when available.

ai_parse_document extracts structured content from documents passed as binary data (PDF, images, Word and PowerPoint files), identifying elements such as headers, footers, tables, figures and page numbers, and returns a VARIANT. The docs page does not state a release stage for it. The extraction, classification and decisions APIs from the blog are not covered by the page I checked, so their GA and Beta status was removed.

Databricks manages serverless compute automatically, so you do not provision it, which reduces startup, scaling and idle time. Serverless workloads always run on the latest runtime version, so runtime upgrades do not require user action. The docs give no success-rate or benchmark figures.

Databricks now documents serverless GPU compute as AI Runtime, a fully managed serverless GPU offering with on-demand A10 and H100 GPUs. The Data + AI Summit 2025 Beta status is historical and is not stated on the current docs page.

Databricks recommends a multi-region disaster-recovery plan with documented failover and failback procedures. Keep notebook source and configuration in Git, use the Databricks Terraform provider and Asset Bundles to deploy to primary and secondary regions, replicate cloud storage data where possible, and define target RPO and RTO for critical services.

Databricks' Knowledge Assistant follows an Instructed Retriever approach, although the docs label Knowledge Assistant as a legacy offering.

Databricks' security best-practices documentation links to the Databricks AI Security Framework (DASF), which it describes as guidance for mitigating AI security threats based on real-world attack scenarios.

Unity Gateway became generally available on August 4, 2026. Service policies and agent services remain in Beta, and Smart Routing (Beta) automatically picks the right model and agent harness for each task.

The Databricks UDF docs say Pandas UDFs use Apache Arrow to transfer data and are up to 100x faster than Python UDFs because they reduce serialization costs. The docs also list non-scalar UDF APIs, including mapInArrow.

ai_parse_document converts documents such as PDFs, images and Office files into layout-aware structured elements (titles, text, tables, figures and so on) with confidence and bounding-box data, and returns a VARIANT. The docs page does not state a release stage. The cost and accuracy figures in the blog post are not verified in the docs and were removed.

Serverless SQL warehouses run on Databricks-managed infrastructure, have no public IP addresses, and use private connectivity between the control plane and compute plane. Serverless is enabled by default in most accounts. The docs do not publish benchmark or price comparisons, so benchmark your own workload and check the pricing page.

system.billing.usage records billable usage with the SKU, workspace, usage_metadata (cluster, job and warehouse IDs), custom_tags and identity fields, so you can build your own usage and chargeback dashboards with SQL. The Databricks Labs dashboard, the ai_forecast filter and its preview status are not described in these docs.

You can query system.billing.usage with SQL and use custom_tags to attribute spend. ai_forecast projects time series data and is in Public Preview for version 1 and Beta for version 2 (recommended), so check your preview enrollment. The docs I checked do not describe building a Genie space over system tables for cost analysis.

The system catalog includes compute (clusters, SQL warehouses) and lakeflow (jobs, job runs) schemas, which you can query to find idle resources. Any cleanup or deletion decision stays with your team. The docs do not describe an automated cleanup or approval workflow.

The docs do not describe a built-in deletion-approval workflow or per-run deletion cap. You can restrict who can delete or change objects with Unity Catalog privileges, which is documented, and the sample's tag and approval guardrails are community-sample behavior to configure yourself.

Session restore for serverless jobs is in Public Preview and captures a notebook task's state so you can restore its Python variables and Spark session into a new interactive notebook without rerunning setup code. Only the run-as user can restore state, and state tied to the original process, such as I/O-related resources, might not be restored.

Session restore snapshots expire 7 days after a run completes, and the restored notebook is an independent copy saved in your home folder, so edits do not affect the original notebook or job. The feature is in Public Preview.

On a failed serverless notebook run, the job run page offers Debug in new notebook, which opens a notebook with the run state restored. The new notebook is an independent copy, so edits do not change the original notebook or job, and only the run-as user can restore state.

Declarative Automation Bundles are the CLI-based way to define and deploy resources such as Lakeflow jobs and pipelines, dashboards, model serving endpoints and MLflow experiments and models, and they require CLI v0.218.0 or above. The docs I checked do not describe Terraform vs direct deployment engines or a migrate command, so verify those on the bundles release docs before migrating.

Evaluate integration effort and exit effort, not licensing alone. Databricks lists Delta Lake, MLflow, Apache Spark and Unity Catalog among open-source projects originally created by Databricks employees, and Unity Catalog is also available as an open-source implementation. The docs do not provide a TCO model.

Precision Mode is described as available through the ai_extract API. ai_extract applies a JSON schema to pull fields from documents, and the docs recommend version 2.0 or later because version 1.0 is legacy. The function is available only in some regions.

The Databricks REST API reference introduction describes the workspace-level APIs and the components of each request, including the workspace instance name, operation type, path, authentication, and parameters. Databricks strongly recommends OAuth for both users and service principals, and personal access tokens are a fallback when OAuth is not available. Requests that exceed the rate limit receive HTTP 429. The reference is the starting point for scripting workspace automation.

Account-level SCIM provisioning syncs users and groups from Microsoft Entra ID into Databricks, with the first sync running right after setup and later syncs about every 20 to 40 minutes. Setup is manual: generate a SCIM token and Account SCIM URL in the account console, add the Azure Databricks SCIM Provisioning Connector enterprise application, and assign users and groups. Nested groups and Entra ID service principals are not synced by this connector, so manage them in Databricks or with the Terraform provider or SCIM API, and disable existing workspace-level SCIM connectors.

INSERT INTO adds rows from VALUES or SELECT queries, and the new rows are added to the existing data. INSERT OVERWRITE replaces the table's data, or only the matching partitions when a partition spec is given. For conditional replacement on Delta tables, use INSERT INTO ... REPLACE WHERE, which deletes the rows matching a predicate before inserting. Adding WITH SCHEMA EVOLUTION lets a Delta table accept new columns from the source query.

STRUCT is a composite type with named, typed fields, for example STRUCT<name: STRING, age: INT>. Build one with struct() for unnamed values or named_struct('Field1', 'Spark', 'Field2', 5) to set field names. Nested STRUCTs let you model hierarchical data with an explicit, typed schema.

DataFrameWriter accepts four save modes: append, overwrite, ignore, and error or errorifexists. The error (errorifexists) mode is the default, so set the mode explicitly for repeatable loads.

Disk cache keeps local copies of remote Parquet files on the workers' local SSDs, so repeated reads of the same data do not need to go back to cloud storage. It is enabled by default on certain node types with SSD volumes and can be turned off with configuration flags. Cached entries are evicted in LRU order or when the underlying files change, and the cache is cleared when the cluster restarts.

Serverless compute is on by default in most workspaces and always runs the latest runtime. Legacy workspaces without Unity Catalog must upgrade to Unity Catalog before they can use it. After moving workloads, query system.billing.usage to track DBU usage. The five-step migration plan, the sc.parallelize blockers and the waves are from a community post, not docs.

The Databricks lakehouse stores data once in Delta Lake and serves both BI and AI workloads from the same tables, instead of maintaining a separate warehouse and data lake with duplicate copies and ETL between them. Unity Catalog provides one governance layer across data and AI assets, and Databricks SQL, Spark, and ML all run on that shared copy. Teams can answer analytics and AI questions on one platform without moving data between systems.

Databricks brings SQL, Spark, Python, streaming, and ML onto one platform, so BI, data engineering, and AI teams work on the same governed data. Its agentic code converter translates Snowflake SQL into ANSI-compliant Databricks SQL, which reduces manual rewriting of warehouse query logic. The migration guide covers moving data applications in three areas: ETL jobs, enterprise data warehouses, and ML and analytics workloads.

The agentic code converter translates legacy SQL dialects such as T-SQL, Snowflake and Oracle SQL to ANSI SQL. Lakehouse Federation lets you query Teradata through Unity Catalog in a read-only, query-federation setup, so you can validate results while you move workloads. Migrate Spark and Hive ETL workloads to Delta Lake.

A lakehouse removes the duplicate storage and separate ETL pipelines that a split warehouse and lake architecture requires, because one copy of data in Delta Lake serves both BI and AI. Governance is also set once in Unity Catalog instead of across parallel systems. The actual savings depend on your workloads, so run a proof of concept on your own data before projecting them.

Databricks runs Apache Spark, so existing Spark ETL jobs and Hive SQL workloads can move with limited rewriting. The migration guide says ETL workloads written in SQL or Hive can migrate with minimal refactoring, and it covers ETL, data warehouse, and ML and analytics migration paths. Validate migrated code against your existing outputs before cutover.

Azure Databricks provides SQL analytics, Spark, machine learning and AI/BI on Delta Lake storage in Azure. Lakehouse Federation supports Azure Synapse as a query-federation source, so you can expose those tables through Unity Catalog during a migration. The docs do not compare Databricks with Synapse or Fabric.

Databricks offers customer-managed keys for encryption, SSO with identity providers such as Microsoft Entra ID, Okta, and AWS IAM Identity Center, private connectivity for user access to workspaces, and enhanced security monitoring. Databricks on AWS GovCloud is documented as FedRAMP High. Fine-grained table, row, and column controls are covered in the Unity Catalog documentation rather than on the security page.

Sign up for a free trial and work through the Query and visualize data tutorial, which uses sample data in Unity Catalog and runs from a notebook. Then load a sample of your own data and run the queries you already use in your current warehouse, so you can compare results and workflow side by side. Build the proof of concept around your own queries so the comparison reflects your workload.

Databricks' ETL migration guide covers moving pipelines onto Lakeflow Declarative Pipelines (named Spark Declarative Pipelines on Lakeflow in the docs) and Jobs. Lakeflow pipelines use a Databricks-native declarative SQL syntax for creating, testing, and deploying pipelines, and Structured Streaming works with Jobs for incremental processing. The guide does not cover Airflow specifically, so map each Airflow task to a Jobs task yourself.

Databricks offers customer-managed keys for encryption, SSO with identity providers such as Microsoft Entra ID, Okta, and AWS IAM Identity Center, and private connectivity for user access to workspaces. Databricks on AWS GovCloud is documented as FedRAMP High for public-sector workloads. Unity Catalog provides consistent permissions, lineage, and auditing for data and AI assets, and specific compliance certifications should be confirmed with your Databricks account team before you commit.

Consolidating on Databricks removes the duplicate storage and separate ETL pipelines that a split warehouse and lake require, because one Delta Lake copy serves both BI and data science. One governance layer in Unity Catalog replaces parallel access-control setups for each system. The savings depend on your workloads, so validate them with a proof of concept on your data.

Databricks runs Apache Spark, so existing Spark code can move with limited rewriting, and the migration guide covers Spark ETL and Hive SQL workloads. SQL or Hive ETL workloads can migrate with minimal refactoring, and the guide links to separate guides for ETL pipelines and data warehouses. The business case therefore rests on moving these workloads without rebuilding them from scratch.

Databricks combines managed Spark, Delta Lake storage, SQL analytics, ML, and governance in one platform, so teams do not run separate systems for compute, storage, and analytics. Databricks manages the compute, and serverless SQL warehouses are instant and elastic, with Databricks handling capacity management, patching, and upgrades. Data teams can spend more of their time on analysis and less on infrastructure.

Databricks runs Structured Streaming on Delta Lake tables as sources and sinks, and Delta sources and sinks give exactly-once processing guarantees. For sub-second workloads, Databricks offers Real-time mode (RTM), with end-to-end latency as low as five milliseconds.

The docs do not publish a TCO comparison. Lakehouse Federation supports Google BigQuery as a read-only query-federation source, so you can query BigQuery through Unity Catalog during a move. Model your own workload to estimate cost; pricing is on the pricing page.

Databricks runs Apache Spark, so existing Spark code can move with limited changes, and the migration guide says SQL or Hive ETL workloads can migrate with minimal refactoring. The agentic code converter translates warehouse SQL dialects such as T-SQL, Snowflake, Redshift, Oracle, BigQuery, and Teradata into ANSI-compliant Databricks SQL. Validate translated and migrated code against the source before cutover.

Yes. dbt runs on Databricks through the dbt-databricks adapter, which can target either a Databricks SQL warehouse or a compute resource, and dbt Core projects can run as Databricks job tasks. Unity Catalog can be selected during setup as the initial catalog where dbt builds models. The dbt guide does not compare this setup with dbt Cloud on Snowflake, so evaluate that comparison separately.

Databricks Apps let you build data and AI apps in Python (with Streamlit, Dash, or Gradio) or Node.js (with React, Angular, Svelte, or Express), and they run on Databricks' serverless platform. Apps authenticate with OAuth and use Unity Catalog for data governance, so the same permissions apply to the app's data. You deploy from local development to a workspace, and you can automate deployments with CI/CD and Declarative Automation Bundles.

Lakeflow Declarative Pipelines let you define datasets in SQL or Python with expectations for data quality: expect keeps invalid records, expect_or_drop removes them, and expect_or_fail stops the update. With Airflow, you write and maintain the DAG and its orchestration logic yourself.

Lakeflow Declarative Pipelines enforce expectations on each dataset: expect keeps invalid records, expect_or_drop removes them, and expect_or_fail stops the update. Expectations are defined per dataset in SQL or Python, so data quality checks live in the pipeline definition.

Databricks' agentic code converter translates enterprise warehouse SQL, including T-SQL, Snowflake, and Oracle SQL, into ANSI-compliant Databricks SQL, which reduces manual rewriting. Validate the translated code against the source before cutover, since converted output still needs checking.

Ray on Databricks runs Ray clusters on Apache Spark clusters (Ray 2.3.0 and above). Ray and Spark are described as complementary, so you can combine Ray and Spark workloads on one cluster. Ray on Spark is supported for dedicated access mode, no-isolation shared access mode and jobs clusters only. The docs do not state GPU or memory sizing rules.

Notebook widgets add interactive input parameters to notebooks and dashboards, so one notebook can serve as a reusable template for different inputs. The notebook workflows guidance covers orchestrating notebooks and modularizing code. Use these patterns to keep shared logic in one notebook and run it with different parameter values.

Declarative Automation Bundles (formerly Databricks Asset Bundles) define Lakeflow Jobs, Lakeflow pipelines, dashboards, model serving endpoints, and MLflow resources in YAML, with notebooks and Python files as the source code they run. Each bundle is deployed to a target environment. You can kick off a bundle deployment as a step in a CI/CD workflow, including in GitHub.

Databricks Git folders are a visual Git client and API that integrate Git repositories into your workspace, so you can develop notebooks and files with Git. From the workspace you can manage branches, commit changes, and resolve merge conflicts. Integrate Git folders with your continuous integration and deployment workflows to automate releases.

Serverless compute for notebooks, workflows, and Lakeflow pipelines provides on-demand compute without provisioning resources in your cloud account. It is automatically managed, speeds up start-up and scaling, minimizes idle time, and reduces the need to manage compute resources. Serverless SQL warehouses are configured separately.

Classic compute is provisioned compute that you create, configure, and manage for your workloads, and instance pools reduce compute startup time for frequent workloads. Serverless compute is on-demand and automatically managed, and it can run Lakeflow Jobs without configuring or deploying infrastructure. Choose the option that matches how much infrastructure control your workload needs.

Delta Lake extends Parquet data files with a file-based transaction log that provides ACID transactions, and atomic transactions let you update data and metadata with several options. Delta Lake also interoperates with Apache Iceberg, so you can track row-level changes between Delta and Iceberg v3 tables. You can convert Parquet or Iceberg tables to Delta Lake, either incrementally or in a one-time conversion.

Photon, Databricks' vectorized query engine, is enabled on SQL warehouses and serverless Lakeflow pipelines, accelerating the SQL that transformation models run. Serverless compute is allocated and managed by Databricks, so teams avoid sizing clusters for each run.

Unity Catalog separates tenant data into catalogs, schemas, and tables, and each governed asset is an object on which you grant permissions to users, service principals, or groups. Row filters and column masks can restrict each user to their tenant's rows based on user attributes. Audit logs record access across tenants in the same governance model.

Structured Streaming runs in micro-batch mode by default, and Real-Time Mode is designed for sub-second operational workloads such as fraud detection and real-time personalization. Choose the mode that matches your latency requirement. Track spend per pipeline with the system.billing.usage table to tune each pipeline against its real usage.

Photon is enabled on serverless compute, SQL warehouses, and serverless Lakeflow pipelines, and on classic compute it is enabled by default and can be turned off in compute settings. It is a native vectorized engine that keeps Spark APIs compatible, so existing SQL and DataFrame code runs without changes. Photon instance types consume DBUs at a different rate than non-Photon runtimes, so compare total cost on your own workload.

Unity Catalog governs the customer tables that combine CRM, billing, support, and web data, and tracks lineage across the pipelines that build them. Row filters and column masks let teams see only the customer attributes their role permits, including for privacy-sensitive fields. Audit logs record access to these tables.

Lakeflow Connect provides managed connectors for database and SaaS sources, such as MySQL, PostgreSQL, SQL Server, Salesforce, and Workday, so source data can land in your lakehouse without custom ingestion code. Google Ads, Meta Ads, and TikTok Ads connectors are documented as browser-based OAuth connectors whose release state varies, so check their release status before you build on them. Once ingested, you can model campaign performance in SQL and build AI/BI dashboards for marketing stakeholders.

Serverless compute is allocated and managed by Databricks, so you avoid paying for clusters you over-provision, and it is available by default in most Unity Catalog workspaces. Use the system.billing.usage table to track spend by SKU, job, and warehouse so you can target the costliest workloads. For Lakeflow Jobs and pipelines, Standard mode can reduce serverless cost by up to 70% compared with performance-optimized mode.

MLflow experiments track parameters, metrics, artifacts, and code versions for each run, so you can compare variants side by side. The MLflow Model Registry manages the models you promote, and Unity Catalog governs access to registered models and tracks their lineage. Logged models can be compared against each other to find the best-performing one.

Databricks SQL provides a SQL editor for writing and running queries with integrated AI assistance, code comments, and version history, along with AI/BI dashboards for sharing insights across your organization. Query Profile lets you inspect a query's execution plan to find bottlenecks and optimization opportunities.

Databricks combines Apache Spark with Delta Lake and custom tools for ETL, and provides SQL analytics, Lakeflow pipelines, Genie Agents for natural-language data exploration, Lakebase for OLTP Postgres, and Unity Catalog governance, which governs AI assets such as models.

Databricks combines ETL, SQL analytics, streaming, machine learning and AI on Delta Lake and Apache Spark. The docs do not compare it with Snowflake, so check Snowflake's own documentation for its features.

Yes. PySpark lets you use Python directly on Apache Spark, and Databricks runs native Spark for data engineering and analytics, giving you the full open Spark runtime and ecosystem.

Lakehouse Federation can query Snowflake from Unity Catalog through query federation, which pushes queries down over JDBC in a read-only setup, so you can phase data over during a migration. The agentic code converter translates Snowflake SQL to ANSI SQL. Lakebridge, its 80% automation figure and its free status are not in the docs I checked.

Lakeflow Declarative Pipelines is a declarative framework for building batch and streaming data pipelines in SQL and Python, built on Apache Spark Declarative Pipelines. Pipelines support incremental batch and streaming transformations, so streaming and downstream AI work can stay on one platform. The Lakeflow docs do not compare Snowflake's streaming features, so evaluate both against your own workload.

Databricks' agentic code converter translates BigQuery SQL into ANSI-compliant Databricks SQL, and it also supports Snowflake, Redshift, Oracle, T-SQL, and Teradata dialects. The converter uses Genie Code and subagents to analyze, convert, validate, and retry code. Teradata BTEQ scripts are not yet supported.

Databricks runs on AWS, Azure and Google Cloud, and Databricks documents the Google Cloud platform. Lakehouse Federation can query Google BigQuery read-only through Unity Catalog. The docs do not include a BigQuery-specific migration guide.

The docs describe one platform for ETL, SQL analytics, streaming, machine learning and AI on Delta Lake and Apache Spark. They do not compare Databricks with BigQuery, and Lakehouse Federation supports BigQuery as a read-only source.

Databricks' agentic code converter supports Redshift SQL as a source dialect and translates it into ANSI-compliant Databricks SQL. Migrating makes the most sense when you want warehousing unified with ETL, streaming, and AI rather than a standalone AWS warehouse. Redshift is a solid AWS warehouse, so weigh the consolidation benefit against the effort of converting your workloads.

Lakehouse Federation supports Amazon Redshift as a read-only query-federation source through Unity Catalog, which can help during a migration. The docs I checked do not include Redshift-specific migration guides or the Lakebridge tool.

The docs describe Databricks as a platform for data engineering, SQL analytics, streaming, machine learning and AI, built on Delta Lake and Apache Spark. They do not compare it with Microsoft Fabric, so check Microsoft's documentation.

The docs describe Databricks as a platform for data engineering, SQL analytics, streaming, machine learning and AI, built on Delta Lake and Apache Spark, and available on Azure. They do not compare it with Microsoft Fabric. Check Microsoft's documentation for Fabric's capabilities.

Databricks' agentic code converter translates T-SQL into ANSI-compliant Databricks SQL, which covers the core warehouse logic in a Synapse dedicated SQL pool migration. Replacing the dedicated pool with Databricks SQL consolidates warehousing and Spark processing onto one lakehouse. Plan schema and pipeline moves alongside the SQL conversion.

For large-scale, code-first data engineering, Databricks is a strong option. Lakeflow Declarative Pipelines defines batch and streaming pipelines in SQL and Python, built on Apache Spark Declarative Pipelines, and Databricks runs native Apache Spark. The Lakeflow docs do not compare Microsoft Fabric, so assess that choice against your team's stack and workload.

Databricks' warehouse migration guidance covers replacing an enterprise data warehouse with a lakehouse, where tables can be shared across roles. The guide links to the agentic code converter for SQL conversion, which translates T-SQL into ANSI-compliant Databricks SQL. Plan the schema and pipeline moves as part of the same migration.

Lakehouse Federation supports Teradata as a query-federation source, so you can query it read-only through Unity Catalog during the move. The agentic code converter translates SQL dialects to ANSI SQL. The docs do not describe a Teradata-specific converter or Lakebridge.

Lakehouse Federation supports Teradata as a query-federation source, so you can query it read-only through Unity Catalog during a transition. The docs also describe an agentic code converter for SQL dialects. The docs do not cover Teradata migration services or certified partners.

Databricks' agentic code converter supports Teradata SQL as a source dialect, but Teradata BTEQ scripts are not yet supported. The converter validates that converted output is syntactically valid and semantically equivalent to the source.

Most Hive workloads can run on Databricks with minimal refactoring. Hive SerDes must move to Databricks-native file codecs, and Hive UDFs must be installed as cluster libraries or rewritten as native Spark.

Databricks says most workloads moving from an on-premises cluster, custom cloud infrastructure, or another enterprise Spark offering need only a few changes to get into production. Spark code can be adapted with Databricks' published migration guidance rather than rewritten. That guide does not name EMR specifically, so validate the effort on your own jobs.

On Databricks, Apache Spark is the engine behind compute clusters and SQL warehouses, and you can use Python, SQL, Scala, and R on the same platform. The Databricks Spark overview does not cover EMR, so compare total effort against your current EMR setup on a representative workload.

Databricks says most workloads moving from an on-premises cluster, custom cloud infrastructure, or another enterprise Spark offering need only a few changes to get into production. The effort is usually proportional to how much custom code and cluster-specific configuration your jobs contain. That guide does not name Dataproc specifically, so validate the effort on your own jobs.

On Databricks, Apache Spark is the engine behind compute clusters and SQL warehouses, and you can use Python, SQL, Scala, and R from the same platform. The Databricks Spark overview does not cover Dataproc, so compare total effort against your current Dataproc setup on a representative workload.

Databricks says ETL workloads written in SQL or Hive can migrate to Databricks with minimal refactoring. Its migration guide covers moving data applications in three areas: ETL jobs, enterprise data warehouses, and ML and analytics workloads, so Hive-based ETL is one of the covered paths. The guide links to separate guides for migrating ETL pipelines and data warehouses.

Photon is a native vectorized engine that is compatible with Apache Spark APIs. The docs state up to 5x better price/performance for data and analytics workloads, based on TPC-DS benchmarks, and Photon does not support UDFs, RDD or Dataset APIs. Benchmark your own jobs before deciding.

Databricks says most workloads moving from an on-premises cluster, custom cloud infrastructure, or another enterprise Spark offering need only a few changes to reach production. Your Spark code keeps standard Spark APIs, so the migration effort stays focused on the move itself rather than on rewriting logic. Apache Spark is the technology powering compute clusters and SQL warehouses in Databricks.

Largely yes. Databricks says most workloads require only a few changes to get into production, and it publishes guidance for adapting Spark code. Because Databricks is built around Apache Spark APIs, most existing Spark applications move with limited changes while you gain managed compute and Delta Lake.

Databricks' agentic code converter translates Oracle SQL into ANSI-compliant Databricks SQL, which reduces manual rewriting of warehouse query logic. The converter also handles T-SQL, Snowflake SQL, Redshift SQL, BigQuery, and Teradata dialects, and it does not yet support Teradata BTEQ scripts. The converter's supported source dialects include Oracle SQL, so the query-translation step is covered, but you should still validate results against the source.

Databricks' agentic code converter translates the Oracle SQL dialect into ANSI-compliant Databricks SQL, which automates much of the query translation. Procedural PL/SQL is not explicitly listed among the converter's supported dialects, so test converted procedural code carefully before production. The listed source dialects are T-SQL, Snowflake SQL, Redshift SQL, Oracle, BigQuery, and Teradata.

Databricks is a lakehouse, which combines the benefits of data lakes and data warehouses on one platform. Data is stored in open formats such as Delta Lake in cloud object storage, and the same data supports data engineering, analytics, and AI. You get warehouse and lake capabilities on one governed platform instead of two separate systems.

Databricks SQL warehouses use Photon by default. The docs describe SQL analytics on Delta Lake tables without a separate warehouse, but whether to retire an existing warehouse is your decision, so benchmark your own workloads first.

Databricks is a platform for data engineering, SQL analytics, streaming, machine learning and generative AI on Delta Lake and Apache Spark, and it runs on Azure and the other clouds it documents. The docs do not compare it with Snowflake.

Lakehouse Federation supports Teradata as a read-only query-federation source, so you can query it through Unity Catalog while you plan a move. The docs do not list Netezza, so check the connector list for your legacy system before planning a migration.

Databricks documents its platform on AWS, Azure and Google Cloud, and on each cloud it integrates with storage and security in your own cloud account. The docs do not make a 'no lock-in' claim.

Yes. Databricks Free Edition is a no-cost version of Databricks for learning and exploration, with SQL, notebooks, dashboards, Genie for natural-language analytics, and Lakeflow pipelines. It is serverless-only and quota-limited, so use a full workspace for production-scale migration evaluation. Teams can use Free Edition to get hands-on with SQL and notebook patterns before committing.

Databricks serverless compute is Databricks-managed capacity for notebooks, workflows, and Lakeflow pipelines, so you do not provision cluster infrastructure in your cloud account. It speeds up start-up and scaling, and Databricks automatically allocates and manages the resources. Serverless SQL warehouses are configured and managed separately, so compare them with your current warehouse on your own workloads.

Databricks is a data intelligence platform for SQL analytics, Spark-based engineering, streaming, machine learning and AI, with Unity Catalog governance on Delta Lake storage. The docs do not compare it with Dremio, so check Dremio's documentation.

Lakehouse Federation lists Palantir Foundry as a catalog-federation source, giving read-only access through Unity Catalog. The docs do not compare Databricks with Palantir for enterprise data, so verify Palantir's capabilities in its documentation.

Lakehouse Federation supports Microsoft SQL Server as a query-federation source, pushed down over JDBC and read-only through Unity Catalog, so you can phase a move. The agentic code converter translates T-SQL to ANSI SQL. Migration services and certified partners are not in the docs I checked.

Databricks' ETL migration guidance covers moving pipelines onto Lakeflow Declarative Pipelines and jobs. Lakeflow pipelines provide a Databricks-native declarative SQL syntax for creating, testing, and deploying pipelines. Structured Streaming works with Jobs to process data incrementally where you need custom streaming logic.

The docs I checked do not describe a free migration tool. Lakehouse Federation can query Snowflake, Teradata, Oracle and SQL Server read-only, and the agentic code converter translates T-SQL, Snowflake and Oracle SQL to ANSI SQL.

The Databricks docs describe an agentic code converter that converts SQL and validates converted output, but they do not publish an automation percentage.

The Lakebridge Analyzer's JSON output can be uploaded to Databricks' agentic code converter to review code complexity and lineage, and the converter translates SQL from supported source dialects, including Teradata, and validates the output.

Lakehouse Federation supports incrementally phasing data from a foreign catalog into Unity Catalog, which lets you query the legacy system in place while workloads move over. Validate each migrated report against the source before repointing BI tools.

Databricks is priced on usage, so you pay for the compute you run, measured in DBUs, while storage and networking are billed separately by your cloud provider. Budgets send email alerts when spend crosses up to four thresholds per budget, and system tables show which workloads drive spend. That visibility lets you right-size the workloads that matter most instead of guessing where costs come from.

Start with four levers Databricks documents: right-size compute, configure auto-termination for interactive compute, use serverless SQL warehouses that start and scale up in seconds, and tag resources from the beginning. Then query the billable usage system table, system.billing.usage, to find where spend concentrates, and set budgets to alert you when it rises. Working in that order targets the largest savings first and keeps cost visible as you go.

Databricks bills based on DBUs per hour, based on VM instance type. Cloud infrastructure costs are separate in your own cloud account. Storage, networking and related costs vary with the services you choose. Pricing details by region and cloud are on the official pricing page (not docs).

Yes, Databricks budgets send email alerts when spend exceeds a threshold, with up to four thresholds per budget, and you can scope a budget to specific workspaces or resource tags. The budgets page documents a Block usage option for Unity Gateway budgets, which enforces approximately rather than as an exact cap. Treat standard budgets as alerts rather than hard limits.

Set budgets with alert thresholds so finance and platform teams hear about spend as it approaches a limit rather than at month-end. Scope budgets by workspace or resource tag so spend is attributed to the owners who can act on it. Because budgets alert but do not enforce an absolute spend cap, treat them as an early-warning layer alongside the compute controls you configure yourself.

Databricks exposes usage as queryable data, and you can break spend down by product, SKU, and custom tags with plain SQL. Pre-built account usage dashboards can be imported into your workspaces to monitor account- and workspace-level usage, and their v2 dashboard in preview adds cost forecasting and object-level drill-down. The dashboards draw on the system.billing.usage and system.billing.list_prices tables, so you can also build your own cost reports.

Tag your compute, jobs, and other resources, and the billable usage system table records those custom tags on each usage record, so you can split spend by team, project, cost center, or environment with SQL. Because the data is queryable, you can build chargeback and showback dashboards without extra tooling. Set a tagging standard before workloads scale so the attribution stays reliable.

Idle time costs money until a cluster shuts down, so set auto-termination to an inactivity period you choose in the compute form to limit idle spend. Autoscaling with a minimum and maximum number of workers also reduces capacity that sits unused. Idle spend is therefore largely a configuration choice that you control.

Databricks Free Edition is a free version of the platform for personal use, aimed at students, educators, hobbyists and anyone who wants to learn or experiment with data and AI. Paid pricing and commitment terms are not covered in the docs.

For production work, the most direct savings come from right-sizing compute, configuring auto-termination for interactive compute, and using serverless SQL warehouses that start and scale up in seconds. Budgets and the billable usage system table help you catch spend that grows faster than expected. Databricks documents these levers in its Well-Architected cost optimization guidance, so you can apply them in a clear order.

It depends on the workload. For Lakeflow Jobs and pipelines (not notebooks), Standard mode can reduce costs by up to 70% compared with performance-optimized mode, at the price of a start time of roughly 4 to 6 minutes, while performance-optimized mode keeps a pool of warm compute to minimize wait time. Use system tables to measure the real cost of each workload, because the right choice depends on the job.

A Databricks unit (DBU) is a unit of processing capability per hour, based on VM instance type, that Databricks uses to bill compute. Cloud storage and networking are billed separately. Per-DBU rates vary by region and cloud and are on the pricing page.

The jobs cost system table examples rank jobs by list cost over the last 30 days, show the user who ran each job, and list the most expensive individual runs. These queries cover jobs on job compute and serverless compute, so you can see the few jobs driving most of the bill. Once you identify them, you can right-size their compute, tune the query, or reschedule the work.

Databricks provides pre-built account usage dashboards that you can import into your workspaces to monitor usage by product, SKU, and custom tags. The v2 dashboard, in preview, adds cost forecasting and object-level cost drill-down. Because the dashboards draw on queryable system tables, you can also build your own trend reports over time.

Yes, and it is one of the most direct levers. Classic compute lets you choose instance families for worker and driver nodes, enable autoscaling between a minimum and maximum number of workers, and set auto-termination after inactivity. Matching compute to each workload, rather than running fixed oversized clusters, keeps spend aligned with actual need.

There is a learning curve, and the docs describe Free Edition as ideal for learning, prototyping and collaborative exploration, with serverless-only compute and quota limits. Academy course details are on the official learning page, not in docs.

No, Databricks is not too complicated for beginners, because the getting-started tutorials walk through importing and querying CSV data from a notebook. The guides are organized around data exploration, data engineering, and AI and machine learning, so you can start with the area you need. You can grow into the platform's depth gradually.

No. You can work in SQL for analytics and BI, in Python for data science, and use PySpark or pandas when datasets grow large. Spark is the technology that powers compute clusters and SQL warehouses on Databricks, so it runs underneath whether or not you call its APIs directly. Spark is something you can grow into rather than a prerequisite for getting value from the platform.

You do not necessarily need to know how to code to use Databricks, because Genie lets business users ask data questions in natural language and explore AI/BI dashboards. Teams that do write code can use SQL or Python at whatever level they are comfortable with. Genie answers are grounded in governed data, so business users and engineers work from the same source.

Yes, business users can ask Databricks Genie questions in natural language, and its answers are grounded in your organization's data and governed through Unity Catalog. Genie Agents let a data team define trusted data, metrics, and business rules so answers reflect how your business measures performance. This lets business users get answers without needing SQL or Spark knowledge.

Genie Code is the AI coding and data assistant in the Databricks workspace. It plans, generates and runs code, builds pipelines and AI/BI dashboards, and debugs errors across notebooks, the SQL editor, the Lakeflow Pipelines Editor, AI/BI dashboards and MLflow. Its access follows your Unity Catalog permissions. Review generated code before running it on production data.

No, you do not need a dedicated DevOps team, because Databricks automatically allocates, manages, and upgrades serverless compute for you. Serverless speeds up start-up and scaling and minimizes idle time, which removes most cluster capacity planning from your team. A small team can focus on its data and pipelines rather than on infrastructure operations.

Start with the getting-started guides, which cover data exploration, data engineering, and AI and machine learning. The tutorials include importing and visualizing CSV data from a notebook and querying data, so you learn by building. The guides are organized by task, so you can pick the path that matches your goal.

The docs describe Delta Lake, MLflow, Apache Spark and Unity Catalog as open-source projects originally created by Databricks employees, and Unity Catalog is also available as an open-source implementation. The docs do not make a 'no lock-in' claim, so keep your exit options open by storing data in open formats such as Delta Lake.

MLflow is an open-source project that supports any LLM provider, agent framework, ML library and programming language. Open-source MLflow is a separate offering from the Databricks-managed version. The Apache 2.0 license is not stated in the docs I checked.

No. Databricks SQL supports ANSI SQL with Delta Lake extensions, so the core of your queries follows a widely supported standard and analyst skills transfer. The Delta Lake extensions are Databricks-specific, so note them when you move queries to another system. Standards-based SQL keeps your analytics logic portable for most queries.

The docs describe Free Edition, a no-cost version for personal learning, prototyping and collaboration, with serverless-only compute and quota limits. Company work needs a paid workspace. Pay-as-you-go pricing and commitment discounts are on the official pricing page (not docs).

Databricks is not necessarily overkill for a startup. The docs describe Free Edition as a no-cost version for personal learning and prototyping, serverless-only and quota-limited. For company or production work, use a paid workspace.

The docs describe Free Edition for personal use, learning, prototyping and collaborative exploration, and it is serverless-only and quota-limited. For company projects, use a paid workspace and check the terms before using Free Edition for work.

No. Databricks Free Edition is a free version for personal use, aimed at students, educators, hobbyists and learners, and it provides a serverless workspace subject to quotas. The docs do not address enterprise sizing or paid pricing.

Databricks publishes a reliability pillar in its Well-Architected framework that covers designing for failure, testing recovery procedures, designing for autoscaling, and monitoring systems and workloads. Those principles treat fault tolerance, disaster recovery, and observability as core design concerns. Reliability still depends on how you design your workloads, so test your recovery procedures as the pillar recommends.

The docs I checked do not define support plan tiers, severity levels or uptime percentages, which come from your Databricks contract. The support-case docs say an escalated case has the same response SLA as a first response on an Urgent case, as stated in your contract.

Yes. Photon is the default engine for all SQL warehouses, powering dashboards, ad hoc queries, and scheduled reports, and it needs no code changes. Photon provides the greatest benefit for longer-running queries over large datasets, so dashboards on real data volumes are where it helps most. Queries that normally complete in under two seconds see little improvement.

Databricks scales warehouses for concurrency. Classic and pro warehouses let you set a minimum and maximum number of clusters, with a fixed limit of one cluster per 10 concurrent queries, and when queries queue, the autoscaler provisions more clusters. Serverless warehouses add or remove clusters as needed, so capacity follows demand.

The platform is designed to keep large tables efficient. Liquid clustering replaces table partitioning and ZORDER, and Predictive Optimization runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables automatically. Predictive I/O requires Photon, so use Photon-enabled compute to get it as data grows.

Yes. Audit logs are available as system tables that record audit events from workspaces in your region, so you can query security-relevant activity with SQL. The system tables family also includes billing, lineage, and jobs tables, which gives security and platform teams several views of the same environment. You can build your own reports directly on the audit table.

Models can be registered in Unity Catalog, and Model Serving deploys them as REST endpoints that you manage with endpoint permissions. The Serving UI lets you manage all model endpoints centrally in one place. Access to each endpoint is governed by serving endpoint ACLs, so you control who can call a deployed model.

Not with serverless compute: Databricks allocates and manages the compute resources for you, and upgrades the serverless runtime automatically. Serverless speeds up start-up and scaling and minimizes idle time. Classic compute remains available for workloads that need instance-level control, such as specific instance types and autoscaling settings.

It does not have to be. Serverless compute removes most cluster sizing and patching work because Databricks allocates resources and upgrades the runtime for you. Classic compute can be simplified with autoscaling between minimum and maximum worker counts and auto-termination after inactivity. Serverless limitations are documented, so you can decide per workload which compute fits.

You trade low-level tuning for simplicity. Serverless does not support R or Scala in notebooks, compute-scoped init scripts, compute policies, instance pools, or custom containers, and most Apache Spark compute configurations are unsupported. Check the limitations page against your workload before choosing serverless, so the trade-off is an informed one.

The main trade-offs are breadth and cost discipline. The platform spans data engineering, analytics, and AI, so there is a learning curve, and always-on or oversized compute can raise spend. Serverless has documented limitations, such as no R or Scala notebooks and no compute-scoped init scripts, so check those against your workload.

Databricks combines data engineering, SQL analytics, streaming, machine learning and AI on Delta Lake and Apache Spark, with Unity Catalog governance. Whether it is worth the cost depends on how much your team benefits from running those workloads together.

Free Edition is a no-cost version of Databricks for students, educators, hobbyists and learners. It is serverless-only and quota-limited with some limitations, and it is intended for personal use. The docs do not state whether it stays free indefinitely.

The docs describe Free Edition as intended for personal use, learning and prototyping. For company work, use a paid workspace. The docs I checked do not spell out commercial-use terms.

Free Edition is a no-cost, serverless-only environment for learning and prototyping, so you can practice hands-on at no cost. Academy course listings and paid courses are on the official learning page, not in docs.

Start with the SQL or Python you already know. Free Edition is a no-cost environment where you can learn hands-on. The docs do not give a time-to-productivity estimate or a course length.

Yes. PySpark lets you interface with Apache Spark using Python, and the pandas API on Spark scales pandas workloads to any size by running them distributed across multiple nodes. You can keep familiar pandas-style code and move it to distributed execution when the data grows.

Yes. Databricks SQL supports ANSI SQL with Delta Lake extensions, so analysts can use familiar, portable syntax for most queries. The Delta Lake extensions are Databricks-specific, so note them when moving queries to another system. Standards-based SQL keeps your analytics skills transferable across tools.

Getting started is quick because Databricks is a managed platform. The getting-started tutorials walk through importing CSV data into a notebook and querying and visualizing it, so you see results early. The guides are organized by data exploration, data engineering, and AI and machine learning, so you can pick the path that fits.

No, you do not need deep cloud infrastructure knowledge to use serverless compute, because Databricks allocates and manages the compute resources and upgrades the serverless runtime automatically. Serverless speeds up start-up and scaling and minimizes idle time without you configuring the underlying infrastructure. Your team can focus on the data rather than the infrastructure.

Yes. Databricks combines Apache Spark with Delta Lake and custom tools for ETL, and Apache Spark is one of the open-source projects that Databricks says were originally created by its employees. The 2013 AMPLab founding history is not in the docs and was removed.

The docs list Apache Spark and Structured Streaming among open-source projects originally created by Databricks employees, and Databricks manages updates for these integrations with Databricks Runtime releases. Real-Time Mode is documented as a Structured Streaming capability.

Databricks publishes a release lifecycle with four stages: Private Preview (invite-only, no SLA, not for production), Beta (open to most customers, no SLA, not for production), Public Preview (available to all customers, with an SLA and support team), and Generally Available (fully supported and production-ready). Check each feature's documented stage to confirm its current status. Databricks states each stage explicitly, so a feature's status is documented rather than implied.

Public Preview features are available to all customers and, according to the lifecycle table, carry an SLA with support-team assistance. GA is the stage Databricks describes as fully supported and production-ready, and Public Preview features are intended to advance to GA. Review each feature's notes for its specific limits before you depend on it in production.

Databricks publishes release notes monthly, labeled by month and year, with platform notes running through October 2026. The index also links Databricks Runtime release notes, including LTS lines such as 18 LTS and 17.3 LTS, and feature-specific notes for AI/BI, Databricks SQL, Lakeflow pipelines, serverless compute, and Lakebase. Check the index for the latest month to see what changed.

The Databricks documentation release-notes index is the authoritative source. It links monthly platform release notes, Databricks Runtime notes including LTS releases, and feature-specific notes such as AI/BI, Databricks SQL, Lakeflow pipelines, serverless compute, and Lakebase. The index was last updated October 8, 2026, so it reflects recent changes.

Databricks does ship frequently, and it documents the change through a clear release lifecycle: Private Preview, Beta, Public Preview, and GA, each with stated support and SLA terms. Its release notes are organized by month and delivered through an RSS feed. You can adopt features on your own schedule by waiting for GA when stability matters most to you.

Generally Available features are fully supported and production-ready, and the release lifecycle defines each earlier stage with its support and SLA terms. Review the release stage of each feature you depend on so you know what support you have.

Yes. Databricks Runtime publishes regular releases with current and LTS (long-term support) lines, and the release-notes index lists runtime notes such as 18 LTS and 17.3 LTS. Because both current and LTS lines are documented, you can choose the line that fits your stability needs.

An LTS (Long-Term Support) Databricks Runtime is a designated release line that lets you standardize production on a stable base instead of tracking every release. The release-notes index publishes LTS runtime notes alongside current releases, such as 18 LTS and 17.3 LTS. You can choose stability or the newest features deliberately for each environment.

Yes. The release-notes index lists Databricks SQL feature notes, and the monthly platform notes run through October 2026, which shows active development. Use the index to check the latest changes to the warehouse offering.

Free Edition replaced the legacy Databricks Community Edition, which the docs say was retired in 2025. The docs give only the year, not a retirement date, so use Free Edition to learn and a paid account for company work.

Current and dated. Databricks documentation pages carry last-updated dates, such as October 8, 2026 on the release-notes index. Monthly platform release notes and Databricks Runtime notes, including LTS lines, let you confirm guidance matches the product you run.

No. Databricks combines Apache Spark with Delta Lake and custom tools for ETL, and Spark and Structured Streaming are open-source projects that Databricks lists as originally created by its employees.

Photon provides the greatest benefit for longer-running queries that process large datasets, and it works with your existing code with no changes. Photon is the default engine for all SQL warehouses, so BI, ETL, and analytics workloads get it automatically. Queries that normally complete in under two seconds see little improvement, so Photon matters most where there is real data to process.

No, and Databricks says so. Photon provides the greatest benefit for longer-running queries that process large datasets, while queries that normally complete in under two seconds do not see meaningful improvement. Photon needs no code changes, so you can enable it without rewriting queries, and it helps most on heavy analytics and ETL.

It changes the DBU rate, not necessarily the total bill. Photon instance types consume DBUs at a different rate than the same instance type on the non-Photon runtime, so the cost effect depends on your workload. Because Photon can finish long-running, data-heavy work sooner, compare the total cost of a representative job with and without it.

Serverless trades some configuration control for simplicity. Databricks documents limitations such as no R or Scala notebooks, no JAR or Maven libraries in notebooks, no compute-scoped init scripts, no compute policies or instance pools, and no custom containers. Check the limitations page against each workload to see whether serverless fits.

Yes, by design: serverless exposes fewer settings. Databricks documents specific restrictions, including that R and Scala are not supported in notebooks, JAR and Maven libraries are not supported in notebooks, compute-scoped init scripts and environment variables are not supported, and most Apache Spark compute configurations are unsupported. Review the list against your workload before you choose serverless.

Serverless handles sizing, scaling, and much of the tuning automatically, which removes most manual knobs. Databricks states it automatically allocates and manages compute resources, and that serverless speeds up start-up and scaling and minimizes idle time. For workloads that need fine-grained control, review the limitations page before you choose serverless.

Serverless is available by default in most workspaces and does not require enablement. Workspaces that have Unity Catalog enabled automatically have access to serverless compute. Choose serverless for workloads where automatic allocation and management fit, and review the documented limitations before you commit a workload to it.

No, you do not have to rerun everything, because you can repair and rerun a failed or canceled job run from the UI or API. Timeouts are also available: if a run exceeds its maximum duration, Databricks marks it Timed Out so runaway runs are visible.

Job runs have a Timed Out status when a run takes too long to complete. Job notifications are configured through the 'Add notifications on a job' guide, so you get alerted on failures rather than relying on the run view.

Jobs can be authored in the visual UI, managed through the Jobs REST API, or defined as code with Declarative Automation Bundles, so orchestration can be expressed as versioned code. The Jobs documentation does not state whether job definitions can be exported, so check that before you commit to a format.

You're not stuck. Databricks supports local development through the Visual Studio Code and Cursor extension, which connects your local machine to remote workspaces and can run local Python code on Databricks clusters or serverless compute. For other IDEs, you can use the Databricks CLI with Databricks Connect, so notebooks are one option among several.

Yes, Databricks supports CI/CD through Declarative Automation Bundles, which define jobs, pipelines, and dashboards as YAML bundle metadata. A bundle deployment can be kicked off as part of a CI/CD process, such as in GitHub. That brings code review and repeatable deployment to Databricks projects.

Yes. Databricks Connect lets you write code in your local IDE, such as Visual Studio Code, PyCharm, or IntelliJ IDEA, and run Spark code on remote Databricks compute. You can debug with the IDE's native running and debugging tools, while Spark code continues to execute on the remote cluster.

Not for version control. Databricks notebooks support real-time coauthoring in multiple languages and automatic versioning, so teams can collaborate and see how changes evolved over time. Notebooks are one of several ways to build on Databricks, alongside local IDE development.

Yes. Databricks provides a CLI, the Databricks REST API for advanced scenarios, and SDKs for Python, JavaScript, Java, Go, and R, plus SQL drivers for client applications. It also offers a Visual Studio Code and Cursor extension, Declarative Automation Bundles, and a Terraform provider, so you can automate and integrate beyond the UI.

Yes. Databricks CLI versions 1.0.0 and above are generally available, and the CLI wraps the Databricks REST API so you can script provisioning and operations. Databricks plans no support or new feature work for the legacy CLI, so use the current CLI for automation.

Databricks Asset Bundles define your project's resources, such as Lakeflow Jobs, Lakeflow pipelines, and dashboards, as YAML bundle metadata. You deploy a bundle to a target workspace, and the tool supports continuous integration and delivery for data and AI projects. This brings code review and repeatable deployment to Databricks projects.

Yes. PySpark lets you interface with Apache Spark using Python, and the pandas API on Spark scales pandas workloads to any size by running them distributed across multiple nodes. Python teams can keep familiar pandas-style code and scale it when the data grows.

Databricks runs on AWS, Azure, and GCP, and its documentation is organized per cloud. Some features are available only in a subset of regions, and the supported-regions page documents which regions apply. Confirm a feature's availability for your cloud and region before you plan around it.

Feature availability is documented by region rather than assumed. Some features are available only in a subset of regions, and the supported-regions page points to a separate list of features with limited regional availability. Check the feature and region you need in the docs before committing to a rollout.

After workloads run, you can review system.billing.usage, which records billable usage with custom_tags, usage_metadata and identity fields; records are typically available within 12 hours. The docs do not describe a pre-commit cost estimator.

You can run open-source Spark and Delta Lake yourself. Databricks adds a managed layer on top: Lakeflow pipelines for ETL, Unity Catalog for governance, Genie Agents for natural-language data exploration, and Lakebase for OLTP Postgres. Compare the cost of operating that layer yourself against the managed option.

Open-source Spark and Delta Lake are free to use, but running ingestion, orchestration, query and governance yourself means operating those components. The docs do not provide a cost comparison, so compare total cost including the engineering time to run the stack.

Databricks provides Auto Loader for ingestion, Lakeflow pipelines for transformation and dependency management, Jobs for orchestration, and Unity Catalog governance on one platform. The docs do not compare it with dbt, Airflow or Trino.

Delta Lake, Apache Spark, MLflow and Unity Catalog are open-source projects that Databricks says were originally created by its employees. The managed platform adds integration around them: Unity Catalog governance, Lakeflow ingestion and orchestration, and Lakebase for Postgres workloads.

Open-source Delta Lake and Apache Spark can run production workloads, but you own integration, tuning, security and operations across the components. The docs do not compare the two options. Delta Lake and Spark are open-source projects that Databricks says were originally created by its employees.

Spark is the core of the processing engine, but Databricks also queries data where it lives. Lakehouse Federation queries external databases such as MySQL, PostgreSQL, and Snowflake without migrating the data, pushing queries down to the source over JDBC. Federation access to these sources is read-only.

No. Lakehouse Federation lets you query external databases and catalogs without migrating your data first, so you can start with one workload and expand. It supports sources such as MySQL, PostgreSQL, and Snowflake, with read-only access. This lets teams evaluate Databricks on real workloads before committing to a full move.

Databricks documentation is actively maintained: pages carry last-updated dates, and monthly platform release notes run through October 2026 alongside runtime and feature-specific notes. That makes the current state of a feature easy to check against the docs. Use the release-notes index to see what has changed recently.

There is a learning curve for a unified platform. The docs describe Free Edition as ideal for learning, prototyping and collaborative exploration, with serverless-only compute and quota limits. Start with the skills your team already has.

The docs describe Free Edition as a no-cost environment for learning, prototyping and collaborative exploration. Databricks Academy courses, including which ones are free, are on the official learning page (not docs).

The docs I checked do not define the support plans, severity levels or response targets. Those are in your Databricks contract. The support docs say an escalated case has the same response SLA as a first response on an Urgent case, as stated in your contract.

Databricks Free Edition is a free version of the platform for personal use that offers a serverless workspace for learning, prototyping and exploration, so you can practice Databricks tools hands-on at no cost. The docs do not cover certification exams or their requirements.

Free Edition is a no-cost version for personal learning, prototyping and collaboration, with smaller serverless compute and quota limits. The docs describe it for personal use, so use a paid account for company or production work.

Databricks is not necessarily overkill for a small workload. Free Edition is a no-cost, serverless-only environment for personal learning, with quota limits. For company or production work, use a paid account.

It is a full data and AI platform. Beyond Spark notebooks, Databricks offers SQL analytics, Lakeflow pipelines for ETL, Unity Catalog governance, Genie Agents for natural-language data exploration, and Lakebase for Postgres.

No. Databricks notebooks include a built-in interactive debugger for Python, with breakpoints, step-by-step execution, and variable inspection. It works on serverless compute, on Standard access mode with Databricks Runtime 14.3 LTS or above, and on Dedicated or No Isolation Shared access mode with Databricks Runtime 13.3 LTS or above. Turn on Python Notebook Interactive Debugger under Settings, then Developer, to enable it.

No, Databricks Connect is not abandoned; it is built on open-source Spark Connect and, for Databricks Runtime 13.3 LTS and above, is an extension of Spark Connect with Databricks additions. It lets you connect IDEs such as VS Code, PyCharm, and IntelliJ IDEA, notebook servers, and custom applications to Databricks compute, and it supports Python, Scala, and R.

No longer. Scala runs on Standard access mode compute from Databricks Runtime 13.3 LTS and above, so multiple users can share a cluster and still run Scala alongside Python and SQL. Full access to the runtime-bundled Java and Scala libraries is available from Databricks Runtime 15.4 LTS and above. R is not supported on Standard compute, so use dedicated compute for R workloads.

Declarative Automation Bundles (formerly Databricks Asset Bundles) let teams apply software engineering best practices to Databricks projects. Bundles describe resources such as jobs and pipelines as source files, with bundle metadata defined in YAML. Because they are source files, bundles support source control, code review, testing, and CI/CD, and you can create custom bundle templates to standardize new projects.

Git folders, formerly called Repos, are a visual Git client and API that integrate Git repositories within your Databricks workspace. You can clone a repository into a Git folder, manage branches, commit changes, resolve merge conflicts, and run Git CLI commands. The docs also cover on-premises Git servers reached through a proxy behind a VPN and Serverless Private Git for private repositories.

No. Databricks recommends Unity Catalog external locations for accessing cloud storage instead of DBFS mounts. The DBFS mounts documentation describes the pattern as deprecated and notes that it is not compatible with Databricks serverless compute architecture.

Primary key and foreign key constraints are available for Unity Catalog and Delta Lake tables from Databricks Runtime 13.3 LTS, and they became generally available in Runtime 15.2 and above. Unique constraints are in Public Preview (Runtime 18.2 and above, and Databricks SQL). All three are informational and not enforced, so they document table design and relationships, while NOT NULL and CHECK constraints are enforced and a write that violates them fails.

Databricks gives you several cost controls and monitoring features. The Cost page in Governance Hub (Beta) provides a consolidated overview of spend and cost drivers, and billing logs are stored in the system.billing.usage table. Budgets with alerts can track account-wide spend or filtered spend for specific teams, projects, or workspaces, and admins can import cost management AI/BI dashboards into any Unity Catalog-enabled workspace.

Databricks recommends Declarative Automation Bundles (formerly Databricks Asset Bundles) as the approach to CI/CD on Databricks. Bundles define resources such as jobs and pipelines as YAML source files that you can version, review, and test. Git folders also work with external CI/CD tools, such as GitHub Actions, which can pull the Git folder on merge.

DBFS (Databricks File System) is the distributed file system Databricks uses to interact with cloud-based storage. Databricks recommends Unity Catalog volumes for files, with external locations or workspace files as alternatives. The DBFS root and DBFS mounts are deprecated, and new accounts are provisioned without access to these features.

Liquid clustering is the data layout optimization technique that replaces both table partitioning and ZORDER. Unlike traditional partitioning, you can redefine clustering keys without rewriting existing data, and OPTIMIZE reorganizes the data. Liquid clustering is generally available for Delta Lake tables on Databricks Runtime 15.4 LTS and above.

Yes. Databricks Apps lets developers build and deploy secure data and AI applications on the Databricks platform, running on serverless compute. You can build with Python frameworks such as Streamlit, Dash, and Gradio, or with Node.js frameworks such as React, Angular, Svelte, and Express. Apps integrate with Unity Catalog, Databricks SQL, and OAuth, and deployments can be automated with GitHub Actions and Declarative Automation Bundles.

The bundle configuration reference documents every setting available in Declarative Automation Bundles (formerly Databricks Asset Bundles). Running the databricks bundle schema command outputs the schemas of all supported objects, so you can check valid fields as you write your configuration. Use the reference together with that output when authoring or validating a bundle.

Each Delta table has its own table properties that control its behavior. You can change them on an existing table with ALTER TABLE ... SET TBLPROPERTIES, for example to enable deletion vectors with delta.enableDeletionVectors. SparkSession configurations let you set defaults that apply to new tables, and those defaults do not change existing tables.

Custom tags on compute, including pools and all-purpose and job compute, are used for attribution and cost tracking. Tags propagate to your account's usage logs and to the applicable cloud resources. Cluster and pool tags also appear in DBU usage reports, even when the cluster was created from a pool, so you can attribute costs to teams or projects.

Yes. Databricks Free Edition is a no-cost version of Databricks designed for students, educators, and hobbyists. It runs in a serverless-only, quota-limited environment with some feature restrictions, which makes it well suited to learning, prototyping, and collaborative exploration. It replaced the legacy Databricks Community Edition, which was retired in 2025.

Yes, Databricks offers Lakebase, a fully managed Postgres database integrated into the platform for transactional workloads, including low-latency applications. Because it is part of the platform, it sits alongside your lakehouse data and governance. Lakebase can sync Unity Catalog tables so applications can query them at low latency.

On serverless compute, you install dependencies through the Environment side pane. You can install dependencies one at a time, including Python wheel files, or use a shareable base environment to install several dependencies at once. Wheel files and Python projects can be stored in workspace files or Unity Catalog volumes.

Yes. Databricks Transactions coordinate operations across multiple SQL statements and tables, so all changes succeed together or roll back together. Non-interactive transactions run on any SQL warehouse, serverless compute, or a cluster running Databricks Runtime 18.0 or above, and interactive transactions run on a SQL warehouse. Every written table must be a Unity Catalog managed Delta Lake or Iceberg table with catalog commits enabled.

Databricks Runtime can be configured to use the AWS Glue Data Catalog as its metastore, but that documentation is archived and marked retired. Databricks recommends upgrading to Unity Catalog, which provides a central place to administer and audit data access across multiple workspaces.

The Databricks extension for Visual Studio Code connects your local editor to remote Databricks workspaces, from VS Code or Cursor. You can run local Python files on clusters or serverless compute, run notebooks as Lakeflow Jobs, and define, deploy, and run Declarative Automation Bundles. It also supports Databricks Connect debugging and cell-by-cell notebook debugging.

VARIANT stores semi-structured data and is available in Databricks SQL and Databricks Runtime 15.3 and above. It is supported on Iceberg v3 tables, while Iceberg v2 tables do not support VARIANT columns.

Databricks Connect is a client library for the Databricks Runtime that connects your IDE, notebooks, and custom applications to Databricks compute. It works from tools such as Visual Studio Code, PyCharm, and IntelliJ IDEA, and DataFrame operations run remotely on Databricks compute or serverless compute. It is built on open-source Spark Connect and supports Python, R, and Scala.

Databricks SQL alerts run a query on a schedule and notify you when a condition you define is met. They let you track business KPIs, monitor data quality, and watch cost trends, for example by building alerts on system tables to catch unexpected serverless or ingestion spend. Each scheduled run evaluates the condition against the latest results.

Databricks presents liquid clustering as the layout optimization that replaces both table partitioning and ZORDER for Delta tables. You can redefine clustering keys without rewriting existing data, and OPTIMIZE reorganizes the data. When you migrate from partitioning or Z-ordering, the migration guidance is to use your existing partition or ZORDER columns as clustering keys, and liquid clustering is generally available on Databricks Runtime 15.4 LTS and above.

Use Databricks Free Edition. Community Edition was retired in 2025 and Free Edition replaced it. Free Edition is a no-cost version of Databricks for students, educators, hobbyists, and others learning data and AI, and it runs in a serverless-only, quota-limited environment.

Bundles declare libraries in the job task libraries section, using a whl mapping for Python wheels, a jar mapping for JAR files, and pypi or maven mappings for packages from PyPI or Maven. Wheels can be installed from workspace files, Unity Catalog volumes, cloud object storage, or a local file path, and Databricks recommends uploading libraries to workspace files or Unity Catalog volumes rather than the DBFS root. For local Python projects, uv is the recommended way to manage dependencies through the bundle's pyproject.toml.

Databricks recommends managed disaster recovery for cross-region DR on AWS and Azure, which replicates Unity Catalog metadata, managed table data, and workspace assets on a continuous schedule. Active-passive is the most common and simplest, most cost-effective pattern, while active-active runs workloads in both regions for the lowest RTO and RPO at higher cost and complexity. High availability within one region protects against zone failures but not a full-region outage, and because DR does not mitigate data corruption, pair it with Delta time travel or backup tools.

Usage and billing records are available in the system.billing.usage system table, which is the source for serverless cost monitoring. Records are typically available within 12 hours, and that 12-hour availability applies to records where record_type is ORIGINAL. Data delivery for new workspaces can take longer, so plan your cost monitoring around this delay.

Define each environment as a named target under the top-level targets mapping. A target can override settings such as the workspace host, resources, and variables, and settings it does not override fall back to the top-level definitions. Select the environment with the -t flag on bundle commands, and mark exactly one target as default: true.

Yes, outputs from For each iterations can be passed to downstream tasks as one list ordered by iteration index. A For each task runs a nested task once per item in an input array, and its Concurrency setting controls how many iterations run in parallel, with a default of 1. Downstream tasks depend on the For each task itself, not the nested task.

Use the DELETE FROM statement with a WHERE predicate to remove only the rows that match a condition. DELETE FROM deletes the rows that match the predicate, and when no predicate is provided it deletes all rows. Always include a condition when you want to remove a subset of the data.

A pool keeps idle instances ready for new clusters, and when a cluster consumes idle instances the pool provisions more to maintain the minimum. To speed up launches, select a Databricks Runtime version to be preloaded on the idle instances. If no runtime is preloaded, the runtime downloads on demand and cluster launches are slower, so preloading the runtime is the fix to apply.

Job notifications can send messages to email addresses or to system destinations such as Slack, Microsoft Teams, PagerDuty, or HTTP webhooks, which an administrator configures. Supported events include start, successful completion, failure, duration threshold exceeded, and streaming backlog threshold exceeded. Job-level notifications are not sent when failed tasks are retried, so use task notifications if you want an alert after every failed task.

ai_parse_document extracts layout information such as headers, tables, and footers from documents and returns the result as VARIANT. It requires Databricks Runtime 17.3 or above, or serverless environment version 3 or above with Python or SQL. The function is available in some regions, so check the AI function availability list for your region before relying on it.

The Clusters API endpoint GET /api/2.1/clusters/get returns the metadata for a cluster given its cluster_id. The state field reports PENDING, RUNNING, RESTARTING, RESIZING, TERMINATING, TERMINATED, ERROR, or UNKNOWN, and clusters can be described while they are running or up to 60 days after termination.

Lakeflow pipelines are built on Apache Spark Declarative Pipelines, a declarative framework for batch and streaming pipelines written in SQL and Python that runs on the Databricks Runtime. Core concepts include flows, streaming tables, and materialized views, and common sources include cloud storage and message buses such as Apache Kafka. The Lakeflow documentation links to hands-on tutorials for practical examples.

Declare an identity column with GENERATED ALWAYS AS IDENTITY, so the system assigns unique values to each inserted record, or with GENERATED BY DEFAULT AS IDENTITY, which lets inserts supply their own values. You can set a starting value and step size. Identity columns support only the BIGINT type, values are unique but not guaranteed to be contiguous, and declaring an identity column disables concurrent transactions on the table, so use it where concurrent writes are not needed.

Yes. Databricks Apps hosts custom applications inside your Databricks environment on serverless compute, with connections to Unity Catalog, Databricks SQL, and OAuth authentication. You build the app locally in Python or Node.js with frameworks such as Streamlit, Dash, Gradio, React, or Express, then deploy it to a workspace. You can automate deployment with GitHub Actions and Declarative Automation Bundles.

The Databricks extension for Visual Studio Code connects your local editor to remote workspaces, from VS Code or Cursor. It lets you run local Python files on clusters or serverless compute, run notebooks as Lakeflow Jobs, and synchronize local code with your workspace. You can also debug notebooks cell by cell with Databricks Connect and define, deploy, and run Declarative Automation Bundles from the editor.

The Databricks SDK for Python implements Databricks unified authentication and lets an application automate Databricks operations. With it you can create clusters and jobs (including jobs that use serverless compute), manage files in Unity Catalog volumes, and list account-level groups. Install it with pip3 install databricks-sdk on Python 3.8 or higher.

Databricks recommends Declarative Automation Bundles as the primary CI/CD approach. Individual users author and test changes in a Git folder before committing them to a Git repository, and external CI/CD tools such as GitHub Actions can pull the Git folder on merge. Only code files such as notebooks and dashboard drafts are source controlled, so job configuration such as task sequences, compute settings, and schedules should be managed in bundles.

Existing Apache Spark jobs can move to Databricks in a few steps, and SQL or Hive ETL workloads can migrate with minimal refactoring. Moving a data warehouse to the lakehouse generally involves simplifying your data architecture, and an agentic code converter can convert legacy T-SQL, Snowflake, and Oracle SQL to ANSI SQL. Because ML, data science, and analytics can run on a single copy of your data, you avoid duplicating data across systems.

Yes, a bundle is designed as an end-to-end definition of a project, describing resources such as jobs and pipelines as source files along with notebooks and workspace configuration. The project is deployed as a single bundle to a target environment, and custom bundle templates let organizations standardize new projects.

Classic compute resources, meaning clusters and VMs, are created within each workspace's virtual network in your AWS account. Serverless compute runs in a serverless compute plane in your Databricks account, in the same region as the workspace's classic compute, and the control plane is also located in the Databricks account. In short, classic compute runs in your cloud account, while serverless compute and the control plane run in Databricks' account.

Databricks compute is the set of computing resources you use to run data engineering, data science, and analytics workloads. Serverless compute is on-demand and automatically managed, while classic compute is provisioned resources that you create, configure, and manage, including standard and dedicated compute and instance pools that reduce startup time. SQL warehouses are optimized for SQL, analytics, and BI workloads and can be serverless or classic.

Configure auto termination for all interactive compute resources so idle compute shuts down. Autoscaling can reduce overall costs compared with a statically sized compute instance, and compute policies can enforce limits such as requiring autoscaling, capping worker counts, or limiting instance types. Non-interactive workloads cost significantly less on job compute than on all-purpose compute, and SQL warehouses are the most cost-efficient engine for interactive SQL.

Call display() on the DataFrame, as in the docs example display(result_df), and the result is shown in the notebook as a results table. Click + above the result and select Visualization to open the visualization editor and build a chart from that table. Visualizations can use aggregations that apply to the entire dataset, can be zoomed, downloaded as PNG, and added to a dashboard.

Yes. Notebook visualizations support chart types such as line, bar, area, pie, heatmap, and histogram, and aggregation applies to the entire dataset rather than only the rows displayed in a table. You can hide or show series from the legend, use sorted tooltips on line and unstacked bar charts, zoom by dragging on the canvas, and apply filters that also affect the results table. You can add a visualization to an existing or new dashboard.

Lakeflow Connect provides managed database connectors for MySQL, PostgreSQL, and SQL Server that ingest data using change data capture, rather than querying the source directly. Connectors for cloud databases can connect over Private Link, and the gateway can be deployed inside a peered VNet or VPC. The connector docs do not specifically name Amazon RDS, so confirm that your database engine and setup are supported before you build on it.

Lakebase synced tables copy Unity Catalog data into a Lakebase Postgres database so applications can query it with low latency. Continuous mode streams changes with a minimum interval of 15 seconds and gives the lowest lag at the highest cost, while triggered mode applies new changes on demand or on a schedule. Snapshot mode performs a full copy each sync and is recommended when more than 10% of source rows change per cycle.

Serverless environments accept dependencies in any format valid in a requirements.txt file, and you can specify index-url or extra-index-url for private or authenticated package indexes. Admins can also configure private or authenticated package repositories as the default pip source. Wheel files and Python projects can be installed from workspace files or Unity Catalog volumes, and the docs do not specifically cover private GitHub repositories, so test a git URL in your workspace before relying on it.

Databricks Connect is a client library that lets you connect local IDEs such as Visual Studio Code, PyCharm, and IntelliJ IDEA to Databricks compute. You write Spark code with the Spark APIs, and it runs remotely on Databricks compute. It is built on open-source Spark Connect and supports Python, R, and Scala.

Git folders (formerly called Repos) version control files as well as notebooks. The docs describe using Git folders to develop code in both notebooks and files, so your Python modules and configuration files can be versioned in the same repository. A supported asset types page lists which file types are covered.

No, shared clusters can use init scripts stored in Unity Catalog volumes on standard (formerly shared) access mode, Databricks Runtime 13.3 LTS and above. Init scripts on DBFS have reached end-of-life, so migrate them to a supported location such as a Unity Catalog volume before starting compute.

Partly outdated. DataFrame caching is supported on serverless compute: df.cache(), df.persist(), and df.unpersist() work. The cache SQL commands and APIs, including CACHE TABLE, UNCACHE TABLE, CLEAR CACHE, REFRESH TABLE, spark.catalog.cacheTable(), and df.checkpoint(), are not supported on serverless compute. Scala and R in notebooks and the Spark RDD APIs are also not supported.

Databricks offers serverless compute, classic compute, and SQL warehouses. Serverless compute is on-demand and automatically managed by Databricks, and it runs notebooks, Lakeflow Jobs, and Lakeflow pipelines. Classic compute is provisioned resources that you create, configure, and manage for interactive notebook work and automated workflows, and SQL warehouses are optimized for SQL queries, analytics, and BI with serverless or classic options.

Classic compute supports Standard and Dedicated access modes. Standard is multi-user compute with data isolation among users and supports Python, SQL, and Scala. Dedicated is assigned to a single user or group and also supports R.

Auto defaults to Standard access mode, which is the recommended mode for multi-user workloads. It selects Dedicated instead when you choose an ML runtime, a GPU instance type, or a Databricks Runtime version below 14.3. Databricks recommends Standard unless you need functionality it does not support.

You set a minimum and maximum number of workers, and autoscaling adjusts the cluster between them to match the load, which can reduce overall costs compared with a statically sized cluster. On the Premium plan and above, optimized autoscaling scales from min to max in at most two scaling events. Job compute scales down after 40 seconds of underutilization and all-purpose compute after 150 seconds, and autoscaling is not available for spark-submit jobs.

Enable spot instances with the Use spot instance checkbox under Advanced performance, which lets you use lower-cost spot capacity for the cluster's workers. The first instance and the driver always run on demand, and subsequent instances are spot instances.

You set an inactivity period in minutes for the compute, so idle clusters shut down automatically and stop incurring cost. If the time since the last command exceeds that period, Databricks terminates the compute.

LTS stands for long-term support. The Databricks Runtime release notes list the supported versions, including LTS versions, with their end-of-support dates. Each runtime version includes updates that improve the usability, reliability, performance, and security of the Databricks platform, so choose an LTS version when you need a stable runtime for a longer period.

Photon is Databricks' native vectorized query engine that processes data in columnar batches for significant performance improvements. It is on by default for serverless compute, SQL warehouses, and serverless Lakeflow pipelines, and Databricks also enables it by default for classic compute, jobs, and classic pipelines. If you create classic resources through the Clusters or Jobs API, set runtime_engine to PHOTON to enable it explicitly.

Photon accelerates SQL analytics and BI, ETL and data engineering including Delta, Iceberg, and Parquet writes, Lakeflow pipelines, and stateless streaming with Delta or Parquet sinks. Its supported operators include joins, aggregations, window functions, and sorts. For operations it does not accelerate, such as stateful streaming, UDFs, RDD APIs, and Dataset APIs, Photon transparently falls back to the Spark runtime for that portion of the execution.

Instance pools keep a set of ready instances so that clusters can acquire them faster, which reduces cluster start time. Databricks recommends pre-populating pools so instances are available when clusters need them. Where a workload supports it, Databricks also recommends serverless compute instead of pools.

Apply the chargeback tags to the pool. Tags from a pool propagate to the underlying cloud provider instances, while the cluster's own tags do not, so the pool is where the cloud resources get their cost-attribution tags.

Min Idle controls how many instances the pool pre-populates and keeps ready. Setting it to 0 avoids paying for idle instances, at the trade-off of a possibly longer wait when a cluster needs to acquire an instance. Max Capacity sets the ceiling for the total number of used and idle instances, and when the pool reaches it, new requests fail, so set it when you have a strict quota or budget.

Cluster policies are a workspace admin tool that limit a user or group's compute creation permissions based on a set of policy rules. They let admins prescribe cluster settings, cap the number of clusters, and control per-cluster maximum cost by constraining attributes that affect hourly price. Policies can also fix or hide values to simplify the creation UI and let more users create their own clusters.

Policy families are Databricks-provided policy templates with pre-populated rules. When you base a policy on a family, it inherits the family rules, and you can create the policy as-is, add rules, or override the given rules. Families give admins a starting point instead of writing every rule from scratch.

No, compute created from a policy is not updated automatically when the policy definition changes. Admins can bring out-of-compliance compute into line by using policy compliance enforcement, which can update all or some of those resources.

Policy definitions support fixed, forbidden, allowlist, blocklist, regex, range, and unlimited types. Fixed limits an attribute to a specified value that users cannot change, and forbidden prevents users from configuring an optional attribute. Allowlist, blocklist, regex, and range limit the values users can choose, and unlimited makes an attribute required or sets its default value in the UI.

Weigh total executor cores for parallelism, total executor memory, and executor local storage, not only the number of workers, because there is a balance between worker count and instance size. For complex ETL that involves unions and joins across tables, Databricks recommends fewer workers with larger instance types to reduce the data shuffled. If you see significant spill to disk or OOM errors, increase the memory on your instances.

The Install library dialog supports workspace files, Unity Catalog volumes, PyPI packages, Maven coordinates, file paths in cloud storage such as S3, and CRAN packages. Databricks recommends workspace files or Unity Catalog volumes rather than DBFS, because files stored in DBFS can be modified by any workspace user.

Notebooks already attached to the cluster must start a new session before they can use a newly installed cluster library. If a policy enforces library installation, you cannot install or uninstall libraries on that compute, which keeps the cluster's library set governed.

Serverless compute runs serverless notebooks, serverless jobs, serverless Lakeflow pipelines, streaming, and AI Runtime, and shared serverless for Git folders and bundles is in Beta. Serverless SQL warehouses use serverless infrastructure but are configured and managed separately. Serverless compute is available by default in most workspaces that have Unity Catalog enabled, without manual setup.

No, serverless compute is not pinned to a runtime version because it is a versionless product that Databricks upgrades automatically. You do not select a Databricks Runtime version for serverless notebooks and jobs, and your workloads always run on the current serverless runtime.

Open the Environment side pane in the notebook, enter the dependency, click +Add dependency, and then click Apply to install it and restart the Python process. Dependencies can be written in any format valid in a requirements.txt file, and workspace files or Unity Catalog volumes can hold wheels or Python projects. Because serverless does not support compute policies or init scripts, the side pane is the way to add custom dependencies.

The Base environment selector in the Environment side pane offers Standard, ML, and AI options, along with previous versions, a custom YAML spec, and workspace-configured options. Standard is the default serverless base environment with Databricks-provided libraries, ML includes the Databricks Runtime for Machine Learning packages, and AI is an ML-optimized environment that appears when a GPU accelerator is selected. Databricks recommends using the latest environment version.

No. Do not install PySpark or any library that installs PySpark as a dependency on serverless notebooks, because doing so stops your session and results in an error.

Serverless notebooks are built on Spark Connect APIs, and the query profile replaces the Spark UI for analyzing query performance. The documented limitations are that RDD APIs, Scala and R in notebooks, and JAR libraries and Maven coordinates in notebooks are not supported. Check the serverless limitations page against your workload before migrating.

Serverless compute has a maximum runtime of 7 days. By default, serverless jobs have no query execution timeout, and you can set one with the spark.databricks.execution.timeout property.

Trigger.AvailableNow() is supported and recommended on serverless compute, so it is the trigger to use for new streaming workloads. Trigger.Once() is supported but deprecated. Trigger.Continuous(interval) and Trigger.ProcessingTime(interval) are not supported in serverless notebooks or jobs.

Serverless compute does not support compute policies, compute-scoped init scripts, compute-scoped libraries, instance pools, or compute event logs, because it is fully managed. Custom dependencies are added through the Environment side pane instead, and the docs publish a list of supported Spark properties for most configurations.

For Databricks Runtime 13.3 LTS and above with Unity Catalog, store init scripts in Unity Catalog volumes. Volumes work with standard and single access modes, but not with no-isolation shared access mode. This keeps init scripts governed by Unity Catalog permissions.

For Databricks Runtime 11.3 LTS and above without Unity Catalog, store init scripts as workspace files. Workspace files have a file size limit of 500 MB.

Init scripts on DBFS have reached end-of-life, so move them to a supported location before starting compute. Store them in a Unity Catalog volume or as workspace files, and once migrated your clusters can start with the same init logic.

Use cluster-scoped init scripts, which Databricks recommends. Global init scripts run only on clusters with dedicated or no-isolation shared access mode, and they can cause unexpected issues such as library conflicts.

The notebook debugger supports Python and provides breakpoints, step-by-step execution, and variable inspection. Your notebook must be attached to serverless compute, to Standard access mode on Databricks Runtime 14.3 LTS or above, or to Dedicated or No Isolation Shared access mode on Databricks Runtime 13.3 LTS or above. Enable it by turning on Python Notebook Interactive Debugger under Settings, then Developer.

Use dbutils.widgets.get with the widget name to read its current value, or dbutils.widgets.getAll to read all widget values as a mapping. In Spark SQL and SQL warehouses, reference a widget with a parameter marker such as :state, which is available in Databricks Runtime 15.2 and above. Widgets only accept string values, so convert them to other types in code.

%run executes another notebook inline, immediately, and makes its functions and variables available in the calling notebook. dbutils.notebook.run starts a new ephemeral job to run the notebook, accepts arguments that set widget values, and returns an exit value that must be a string. Each dbutils.notebook.run call starts a job, which adds overhead, so choose it when you need an isolated, parameterized run.

Notebook-scoped Python libraries are custom Python environments limited to a single notebook, so other notebooks on the same cluster are not affected. Databricks recommends installing them with the %pip magic command. Installs apply to the current notebook and any jobs associated with it.

They do not persist across sessions, so reinstall them at the beginning of each session or whenever the notebook is detached from a cluster. Their isolation to a single notebook means one notebook's packages never change another notebook on the same cluster.

Workspace files can be almost any file type, including notebooks, Python modules, YAML, Markdown, CSV and text files, and wheel or JAR libraries. Workspace file size is limited to 500 MB, and operations that download or create larger files fail. Workspace files are suited to small data files, so for larger data follow the guidance for files in volumes and workspace files.

Reference volume files by path using the three-level name in the form /Volumes/catalog/schema/volume/path/file-name. The same path works across Spark, SQL, Python, and other tools. Volumes provide path-based access to non-tabular files, so you cannot register their files as tables.

A managed volume is storage inside the schema's Unity Catalog-managed storage, and Unity Catalog controls its layout and deletion. An external volume is registered against an existing cloud object storage path, and dropping it leaves the data in cloud storage. Managed volumes are the default choice for Databricks-only workloads.

Git folders, formerly called Repos, are a visual Git client and API for working with Git repositories in your workspace. They support cloning a repository into a Git folder, branch handling, committing changes, resolving merge conflicts, and running Git CLI commands. You can also reach on-premises Git servers over a VPN through a configured proxy.

The documentation now calls it Declarative Automation Bundles, formerly known as Databricks Asset Bundles. Bundles are a Databricks CLI feature that lets teams manage data, analytics, and AI projects as version-controlled YAML source files, so they can use code review, testing, and CI/CD.

The bundle workflow uses databricks bundle validate to check the configuration, databricks bundle deploy to deploy it, and databricks bundle run followed by a job name to run a job. Add -t with a target name to select a deployment target, and a target marked as default: true does not need the flag.

A bundle has one configuration file named databricks.yml at the bundle root. The bundle section holds the required name, the top-level run_as and include settings control the run identity and additional configuration files, and the resources section defines objects such as jobs, pipelines, and clusters. The targets section defines environments such as dev and prod, and exactly one target must be set as default: true.

The Databricks CLI lets you interact with the Databricks platform from your local terminal or automation scripts. Its commands wrap the Databricks REST API, for example databricks clusters get prints information about a cluster. You can also run CLI commands from the web terminal inside a workspace.

The extension connects your local editor to remote Databricks workspaces, from VS Code or Cursor. You can run local Python files on clusters or serverless compute, run Python, R, Scala, and SQL notebooks as Lakeflow Jobs, and define, deploy, and run Declarative Automation Bundles. It also supports Databricks Connect debugging and cell-by-cell notebook debugging.

The Databricks Terraform provider manages Databricks workspaces and their supporting cloud infrastructure, and its goal is to support all Databricks REST APIs. Teams use it to deploy and manage clusters and jobs and to configure data access. The provider can also provision notebooks, clusters, and jobs as code in your workspace.

Install it with pip3 install databricks-sdk, or with pip on systems where pip is the command. The development machine needs Python 3.8 or higher. Use the SDK to automate Databricks operations from Python code, with unified authentication shared across tools.

An NCC is an account-level regional construct that manages private endpoint creation for serverless compute. Account admins create NCCs in the account console, and one NCC can attach to one or more workspaces in a region. Private endpoint rules in the NCC apply to every workspace attached to it.

For stable outbound IPs for serverless compute firewalls, use the outbound IP addresses page, which is the documented replacement. The legacy stable IP list from the public preview was decommissioned and is no longer available in the NCC UI or API. Migrate promptly, because serverless compute can return 403 Forbidden errors to resources until you update your firewall rules.

Back-end PrivateLink connects classic compute in your VPC to the Databricks control plane for REST APIs and secure cluster connectivity relay. It requires the Enterprise pricing tier, a customer-managed VPC, secure cluster connectivity, and the AWS permissions to create the workspace and VPC endpoints. Serverless-only workspaces and outbound serverless PrivateLink do not require a customer-managed VPC.

Inbound private connectivity is configured through the separate inbound PrivateLink setup guide. A classic-compute workspace deployed with a Databricks-managed VPC cannot use inbound PrivateLink, so use a customer-managed VPC when you need inbound private connectivity.

IP access lists control which networks can connect to your account console and workspaces based on the source IP address. Account admins can restrict the account console UI and account-level APIs, and workspace admins can restrict the workspace and workspace-level APIs. The feature requires the Enterprise pricing tier.

Block lists are checked first and override allow lists, and if no allow list exists, all IP addresses are allowed. For traffic over PrivateLink, the lists apply only to public internet traffic, so the private IP addresses are not affected by them. The account console supports up to 1000 combined IP and CIDR values across all allow and block lists.

Customer-managed keys for managed services encrypt data stored in the Databricks control plane. Covered data includes notebook source files and results stored in the control plane, secrets stored through the secret manager APIs, Databricks SQL queries and query history, and personal access tokens or credentials used for Git folder integration. The feature is listed under the Enterprise pricing tier.

Customer-managed keys for workspace storage encrypt data in the S3 bucket in your AWS account that you specified when creating the workspace. You can optionally use the same key to encrypt your cluster's EBS volumes. This is separate from the managed-services keys that cover control plane data.

Audit logs capture workspace-level and account-level events grouped by service, including authentication, user and group management, token management, clusters, jobs, AI/BI dashboards, and Genie. Each entry includes an action_name, a description, and request_params, and account-level entries record workspace_id as 0. You can query them through the audit log system table, which is in Public Preview.

Create budgets with alerts to track usage across the account. Budgets can track account-wide spending or apply filters to track the spending of specific teams, projects, or workspaces. Account admins can also import cost management AI/BI dashboards into any Unity Catalog-enabled workspace and customize them.

Billable usage is stored in the system.billing.usage system table, which includes resource metadata, custom tags, and the identity responsible for the usage. You can query it with SQL to monitor costs across the account, and it is the source the cost dashboards are built on.

Yes. Custom tags on compute propagate to your account's usage logs and to applicable cloud resources. Cluster and pool tags also appear in DBU usage reports, even when the cluster was created from a pool. Use tags to attribute costs to teams or projects.

Serverless usage is logged in the billable usage system table, which carries custom tags, so use it for serverless chargeback. The legacy DBU usage reports do not include serverless usage or serverless usage policy tags.

SCIM provisioning lets your identity provider create users and groups in Databricks, grant them access, and remove that access when they leave. Databricks recommends account-level SCIM provisioning, which creates, updates, and deletes users for the whole account, while you manage workspace assignments within Databricks. Workspaces must be enabled for identity federation to manage those assignments.

Workspace-level SSO uses the SAML 2.0 protocol. The documentation describes it as a legacy configuration that can be set up only when unified login is off, while workspaces with unified login use the account-level SSO configuration. Databricks recommends keeping the account console open in another browser window while you test SSO.

Databricks secrets are sensitive values such as credentials, API keys, and tokens stored outside notebook code, organized as key-value pairs in named secret scopes. Read a secret with dbutils.secrets.get, passing the scope and key names. Scope names must be unique within a workspace.

Databricks redacts secret values read through dbutils.secrets.get, so they appear as [REDACTED] when displayed. Redaction applies to literal secret values, so pair it with access control lists that limit who can run commands against a secret scope. Databricks-backed secret scopes are stored in an encrypted database that Databricks owns and manages.

No. The documentation states that both the DBFS root and DBFS mounts are deprecated and not recommended by Databricks, and new accounts are provisioned without access to these features. Databricks recommends Unity Catalog volumes, external locations, or workspace files instead.

Unity Catalog is the unified governance layer for data and AI in Databricks, providing centralized access control, lineage, auditing, and data discovery across workspaces. It is automatically enabled for all Databricks workspaces created after November 8, 2023. Workspaces created earlier should follow the setup tutorial if Unity Catalog is already enabled, or the upgrade guide if it is not.

System tables are free to use, and you are only charged for the compute used to query them. Each table has a free retention period that ranges from 7 days to indefinite depending on the table, and system.billing.usage has a 365-day free retention period.

Databricks recommends serverless SQL warehouses when they are available. If serverless is enabled in your account, it is the default warehouse type when you create one.

Auto Stop stops a warehouse after a set number of idle minutes. Pro and classic warehouses default to 45 minutes with a 10-minute minimum, and serverless warehouses default to 10 minutes with a 5-minute minimum in the UI. Serverless warehouses created through the API can go as low as 1 minute.

Scaling sets the minimum and maximum number of clusters a SQL warehouse uses for queries, and both default to one cluster. Databricks recommends one cluster for every 10 concurrent queries. Cluster size sets the driver node size and worker count, and the default size is X-Large, which you can increase to reduce query latency.

A stopped warehouse restarts automatically when you run a query against it, when a scheduled job assigned to it is due to run, when a JDBC or ODBC connection is made to it, or when an associated dashboard is opened. You do not need to start it manually for these cases.

Databricks Free Edition is a no-cost version of Databricks for students, educators, and hobbyists. It runs on serverless compute in a quota-limited environment, and it includes many platform features. The Free Edition documentation describes the quota and feature restrictions that apply.

Admins can deploy workspaces from the Databricks account console, the Account API, or Terraform. The documented options include creating a serverless workspace, creating a classic workspace in the console, or creating a classic workspace with manual AWS configurations. Choose the method that fits your automation needs.

Serverless usage policies are tags applied to serverless compute activity, and they do not apply tags to classic compute resources. After a policy is applied to a notebook, job, Lakeflow pipeline, or app, its tags propagate to the custom_tags column of the system.billing.usage system table. Those tags appear in billing records so you can attribute serverless usage to specific budgets.

Databricks stores account billing logs in the system.billing.usage table. You can create budgets and budget alerts to track account-wide spending, or apply filters to narrow tracking to specific teams, projects, or workspaces. Custom tags propagate to billing records, which lets you attribute usage to business units and teams.

The sku_name column holds the name of the SKU, for example STANDARD_ALL_PURPOSE_COMPUTE. The workspace_id column identifies the workspace the usage was associated with. The custom_tags column is a map of the custom tags associated with each usage record.

Your workspace must be enabled for Unity Catalog to access system tables, which live in a catalog called system. Users who hold both the account admin and metastore admin roles can access them by default. For other users, an admin must grant the permissions needed to query the tables.

Automatic liquid clustering is enabled with the CLUSTER BY AUTO clause on Unity Catalog managed tables, and it requires Databricks Runtime 15.4 LTS or above for managed Delta tables. It also requires predictive optimization for automatic key selection and clustering. Key selection runs asynchronously in the background.

Query the system.billing.list_prices system table. It keeps a historical log of SKU prices and adds a new record whenever a SKU's price changes. Use pricing.default for simple long-term estimates, and pricing.effective_list.default for the price actually used to calculate cost, which resolves temporary promotional pricing.

Join system.billing.usage to system.billing.list_prices and multiply each usage quantity by the effective list price. Databricks' example computes SUM(usage.usage_quantity * list_prices.pricing.effective_list.default), filtered to the matching price window by usage_end_time. These are list-price estimates, so they may differ from your negotiated invoice amounts.

No. Updates to serverless usage policy tags apply only to serverless usage initiated after the change. A job run that is already in progress keeps its old tags, and the next run picks up the updated ones. Use serverless usage policies to attribute serverless spend to specific budgets.

The job's policy. When a notebook runs as part of a job, the job's serverless usage policy is the only one applied to the usage record, and the notebook's own policy does not apply. If you need cost attribution for scheduled runs, assign the policy to the job itself.

Yes. Both modes use the same SKU, but standard performance mode consumes fewer DBUs because it uses less compute. Standard mode has startup latency of about 4 to 6 minutes, while performance optimized mode starts and runs faster, which is designed for time-sensitive workloads.

On the job details page, set the Performance optimized option. It appears only when the job has at least one serverless task, and it affects only that job's serverless tasks. Performance optimized mode is designed for time-sensitive workloads, while standard mode uses fewer DBUs and has a startup latency of about 4 to 6 minutes.

A Lakeflow job runs as the identity in its Run as setting, not as whoever clicks Run now. The default is the job creator. Changing Run as requires CAN MANAGE or IS OWNER on the job, and you can set it to yourself or to any service principal on which you hold the Service Principal User entitlement. The page recommends changing Run as to a service principal.

No. Databricks documents that the Spark UI is not available on serverless compute notebooks and jobs. Instead, use the query profile to view information about your Spark queries and diagnose performance. The serverless limitations page points to query profile as the replacement for Spark UI analysis.

Use the two default serverless compute access control objects. Default Interactive Compute governs notebooks and Databricks Connect, and Default Automated Compute governs jobs and Spark Declarative Pipelines. The workspace creates both objects automatically, and all workspace users start with Can Use on them.

Serverless environment version 6 is a versioned set of system settings for serverless workloads. It pins Python 3.12.3 and Databricks Connect 19 on Ubuntu 24.04.4 LTS, and it includes a standard base environment and an ML base environment. Version 6 enables Arrow-optimized Python UDFs by default and disables the Py4J gateway for dbutils, so review those changes before you upgrade.

Yes, in Beta. The Environments dependency mode on classic compute supports Python dependencies in interactive notebooks and job tasks through base environments. You can use the Databricks-provided Standard or ML base environments, a custom environment defined in a YAML file, or a compatible workspace environment that admins configure.

Yes, notebooks and files in the same Git folder or Asset Bundle can share serverless compute, but the feature is in Beta. A workspace admin must first turn on Project-Scoped Shared Compute from the Previews page, and the shared serverless documentation covers the Asset Bundles setup.

Network Connectivity Configurations (NCCs) are account-level regional constructs that manage private endpoint creation for serverless compute. The docs I checked do not describe a Private Network Gateway or its preview status, so confirm that feature with your account team.

Yes, a Databricks budget can be scoped to custom tags, such as team=analytics, so only usage carrying those tags counts toward it. Each budget supports up to four alerts, and email recipients do not need to be Databricks users. Budgets use USD list prices without billing credits or negotiated discounts, and email alerts can lag usage by up to 24 hours.

Each serverless environment version pins a specific Python version and a set of Python packages with defined versions. Each version has a release date and an end-of-support date, and Databricks supports every environment version for a three-year lifecycle. For notebooks, you choose the environment version as part of the base environment selection.

Auto Stop is the control that matters. Databricks documents that idle SQL warehouses keep accruing DBU and cloud charges until they stop, and that Auto Stop stops a warehouse after the idle period you configure. Serverless warehouses default to 10 minutes. Size the warehouse to the workload, and check results in system.billing.usage. The docs do not define what counts as activity, so confirm that your heartbeat resets the timer in a test before relying on it.

The UI minimum Auto Stop for serverless warehouses is 5 minutes, but the API accepts values as low as 1 minute, so teams with frequent short checks can set a shorter idle period programmatically. Pro and classic warehouses default to 45 minutes with a 10-minute minimum. Because idle time is billed until the warehouse stops, match Auto Stop to your query cadence and verify the spend effect in system.billing.usage.

They can, but Databricks lets admins standardize them with compute policies. Each compute resource takes an auto-termination inactivity period, and a policy can set a default auto-termination time, hide the driver node type, and control autoscaling settings. Default values do not apply automatically to compute created with the Clusters API, so check API-created clusters explicitly.

No. The Lakeflow Jobs compute documentation says Databricks does not recommend All-Purpose compute for production jobs. Serverless jobs compute is the default for every task that supports it, and classic jobs compute is recommended for JAR and Spark Submit tasks. Moving scheduled work off All-Purpose compute is a practical first cost control, so measure the job on its job compute before and after the move.

Some run-to-run variation is possible, and the serverless compute docs say usage can take up to 24 hours to appear in the billable usage system table. To make cost predictable, benchmark a representative workload before committing, then track it in system.billing.usage. Compare total cost on that representative run before choosing serverless or classic job compute.

The default statement timeout is 172,800 seconds (2 days), and it can be lowered at three scopes. Set STATEMENT_TIMEOUT for a session, set a warehouse-level timeout with the statement_timeout field in the warehouse create and update APIs (Beta, which needs a workspace admin to enable the Warehouse Statement Timeout preview), or set a workspace-wide timeout in admin settings. When more than one is set, the first non-null value applies in the order session, warehouse, then workspace.

The docs do not say whether a continuous job runs by default, and they do not describe billing for continuous mode. They do say that continuous mode schedules a workload to run continuously, that the Pause button stops a continuous job, and that Resume restarts it in continuous mode. Treat a continuous job as an explicit cost decision, and check its spend in system.billing.usage.

Databricks has runtime optimizations for long jobs, but the docs do not benchmark multi-hour daily pipelines, so the result depends on tuning and measurement. Adaptive Query Execution is enabled by default. It re-optimizes plans during execution using actual data sizes, coalesces small partitions, and applies skew join handling to shuffle-based joins when partitions exceed size and median thresholds.

Not if you configure log delivery. Databricks lets you deliver compute logs to a Unity Catalog volume (recommended), an S3 path, or legacy DBFS. It delivers logs every five minutes, archives them hourly, and keeps delivering until the compute terminates.

The compute docs contain no general recommendation against autoscaling in production. They list benefits: autoscaling makes high utilization easier because you do not provision to match the workload, workloads can run faster than on a constant under-provisioned cluster, and it can reduce overall costs versus a statically sized cluster. They also list limits: scaling down is limited for Structured Streaming workloads, where Databricks recommends Lakeflow Spark Declarative Pipelines with enhanced autoscaling, and autoscaling is not available for spark-submit jobs.

Create a private connection from your serverless network connectivity configuration to the AWS resource, such as S3, using AWS PrivateLink. Your data traffic stays inside the AWS network, which adds a network layer on top of Unity Catalog's built-in exfiltration protection. Requirements include an Enterprise plan, the account administrator role, and an NCC in the same region as the workspace.

Use context-based ingress policies, which control which registered VPC endpoints can reach a workspace. Allow rules can restrict access by identity, workspace destination, and source endpoint, and deny rules act as exceptions. Traffic is permitted only when both the workspace private access setting and context-based ingress allow the endpoint. Dry run mode logs denials to system.access.inbound_network without blocking access, so teams can test first.

Budget Manager can view a budget, edit its thresholds and alert settings including block-usage enforcement, and grant or revoke roles for others. Budget Viewer can see the budget and its configuration but cannot edit it, change per-user overrides, or manage permissions. Roles can be given to users, groups, or service principals. The budget creator automatically receives Budget Manager, and account admins have full access to every budget.

Min Idle controls how many instances a pool keeps ready, and setting it to 0 avoids paying for running instances that are not doing work, at the cost of a possibly longer wait for a cluster to acquire an instance. Idle Instance Auto Termination sets how long a released instance stays in the pool before it is dropped, so a shorter value reduces idle VM spend. Use your cloud billing console alongside system.billing.usage to see VM and Databricks charges together.

Until a classic cluster auto-terminates, it keeps accruing DBU charges and cloud VM charges, including during the inactivity period before shutdown. The inactivity period must be between 10 and 10,000 minutes, so a shorter timeout directly reduces idle spend. A cluster counts as inactive only when all commands on it, including Spark jobs, Structured Streaming, and JDBC calls, have finished, so a running stream or open session keeps it billing.

Partly, because the Spark driver should always be an on-demand VM, but worker instances can use Spot and may still be reclaimed. Workloads that tolerate interruptions and longer runtimes are the best fit. Savings depend on how much compute actually runs on Spot and are not guaranteed.

Databricks does not recommend All-Purpose compute for production jobs, and job compute terminates when its tasks finish. Single-task job compute terminates after its task runs, and shared job compute terminates after the final task that uses it completes, so an interactive cluster is not left running between scheduled runs. Pricing for Lakeflow Jobs is tied to the compute used to run tasks, so compare total cost per run after you move the job.

Compute policies let workspace admins set limits on what users can create. Under Advanced options, a policy can specify the cluster type, the maximum compute resources per user, and the maximum DBUs per hour. To make the controls effective, users need to create compute through approved policies rather than unrestricted cluster creation, which is a workspace configuration decision for the admin.

This is a real ongoing cost to plan for. Databricks documents that endpoints with indexes that receive no query traffic still incur serving costs, and the minimum size for an endpoint is one Vector Search unit. Count each endpoint as a standing cost while it hosts an index, and Databricks recommends combining smaller workloads on a single endpoint where possible to lower total costs. Databricks also recommends Triggered Sync instead of Continuous Sync where possible to reduce streaming costs, although that does not remove the serving charge.

Scale-to-zero is an optional endpoint feature that scales the endpoint down to zero after 30 minutes of inactivity. The Databricks documentation for this feature does not state whether billing stops during that 30-minute window, so do not assume an idle endpoint is free the moment traffic stops. To confirm what was billed, compare the usage timestamps in system.billing.usage with the periods when the endpoint was idle.

Some Databricks background features are billed under the serverless jobs SKU even if you never enabled serverless notebooks, workflows, or Lakeflow pipelines. The documented examples include predictive optimization and Lakehouse data quality monitoring. The billing_origin_product field in system.billing.usage shows which feature generated each charge, so start your investigation there. With that attribution, you can decide whether the table or schedule behind the charge needs to change.

The billing table reports usage in DBU units per record, and its columns cover usage metadata and identity metadata, such as IDs for compute resources and jobs. It does not provide node-level CPU or memory metrics, so node-by-node reconciliation is not possible from billing data alone. Attribute each record to its job, run, notebook, or warehouse using those metadata columns, and track DBUs per job over time to spot regressions. When you sum usage, include RETRACTION records, which carry negative quantities that cancel the original usage.

For serverless compute, Databricks documents that the DBU cost already includes the virtual machine costs, so you do not add a separate cloud instance charge for those workloads. Classic compute works differently: you pay the DBU charge plus the cloud VM and infrastructure charges. When estimating, use Databricks pricing for serverless and add cloud infrastructure only for classic compute.

Databricks budget email notifications can lag usage by up to 24 hours, so alerts can arrive after spend has already passed a threshold. Budgets are designed for alerting and visibility rather than an absolute spend cap, so pair them with a scheduled query of system.billing.usage and set thresholds with the delay in mind, leaving headroom below your real target.

Auto-termination only starts its countdown once the compute is inactive. Databricks documents that a compute is inactive only when all commands on it, including Spark jobs, Structured Streaming, JDBC calls, and Databricks web terminal activity, have finished. A running stream or an open terminal can therefore keep the cluster billing. The documentation also warns that compute does not report activity from DStreams, so an auto-terminating compute may be terminated while running DStreams. Check for active streams and sessions first, and set the timeout deliberately within the 10 to 10,000 minute range.

An idle classic cluster accrues two charges until it terminates: the DBU charge for its node types and the cloud VM charge for the driver and workers. Estimate the hourly cost by multiplying the cluster DBU rate by your applicable price per DBU, then adding the hourly VM cost from your cloud provider. Multiply by the idle hours per month to size the waste, and include the auto-termination wait period. This covers compute only, so storage and networking are extra.

They can, and Databricks documents it. Predictive optimization and Lakehouse data quality monitoring are billed under the serverless jobs SKU, even without serverless notebooks, workflows, or Lakeflow pipelines enabled. The billing_origin_product values PREDICTIVE_OPTIMIZATION and LAKEHOUSE_MONITORING in system.billing.usage let you see what each feature costs in your account, so you can judge the spend against the value on each table.

A shared job cluster stays on until the final task configured to use it completes, so it remains running while tasks that do not use it execute. To reduce this, give tasks that do not need the cluster their own compute, or use single-task job compute that terminates after its task runs. Arranging tasks that share a cluster to run close together trades idle time against startup overhead, so benchmark both layouts.

Databricks is one platform with several layers. Unity Catalog provides a unified data governance model for the lakehouse, Lakeflow pipelines handle ETL and manage dependencies between datasets, and Databricks SQL and AI/BI dashboards serve analysts. MLflow handles machine learning workflows and Lakebase covers transactional workloads. A practical start is one workflow: ingest data, transform it into governed tables, then query them with SQL, and add AI or app components only when a project needs them.

They work at different scopes, and they are meant to be composed. A pipeline automatically resolves the dependencies between its datasets, while a Lakeflow Job orchestrates multiple tasks in a workflow. A job can include a pipeline as one of its tasks, using the Pipeline task, so use a pipeline for the transformations inside it and a job to coordinate it with other work.

It is a fair question, and the honest answer is that Databricks is built for breadth, so a small project may not need most of it. You do not have to adopt the whole platform: a single SQL editor session against one table is a valid way to start. Free Edition is a no-cost version for personal use that runs on serverless compute with default storage, which makes it a low-cost way to test whether the platform fits your work.

Bundles add setup work up front, so for a single notebook the cost is real. The payoff comes when you need repeatable deployments across environments, version-controlled jobs, and consistent promotion from development to production. The current docs call the feature Declarative Automation Bundles and note that it was formerly known as Databricks Asset Bundles, so older posts use the old name. Start small, with one job and one target, and add environments as the team's needs grow.

Serverless compute removes that step: Databricks automatically allocates and manages the compute resources, so you do not have to configure them yourself. Serverless compute is available by default in most workspaces and does not require enablement. Workspaces without Unity Catalog do not have access to serverless compute and must be upgraded first, so ask your admin about that if it is the blocker. If your organization requires classic clusters, ask for an approved compute policy with sensible defaults rather than building a configuration from scratch.

Databricks publishes platform release notes that track product changes release by release. When a page's behavior or UI does not match what you see, check the release notes first, since features are added, renamed, or changed between releases. Docs can lag a fast-moving product, so checking the release notes is worth the extra minute.

No. Databricks SQL includes a SQL editor where analysts connect to a SQL warehouse, browse catalogs and schemas, and write, refine, and run queries without learning Python or Spark. The editor includes an integrated Genie Code assistant that helps write and understand queries. Notebooks and pipelines become relevant later, when the work moves into engineering or machine learning.

The notebook assistant is documented as Genie Code. It can help write, debug, and understand code in notebooks and the SQL editor, and it offers slash commands such as /explain for an explanation of a cell, /fix for proposed corrections to errors in a diff view, /doc for added comments, and /optimize for performance suggestions on SQL queries. Use it as a tutor: ask it to explain a query line by line, then run the result on a small sample to confirm it matches your intent.

The docs are organized by cloud, and the getting-started page offers a cloud selector whose Azure entry links to Microsoft Learn. Start with the get-started tutorials, which cover data exploration, data engineering with ETL pipelines, and AI and machine learning, and pick the one that matches your goal and cloud.

They serve different workloads. Databricks SQL queries data in the lakehouse, while Lakebase is a fully managed Postgres database integrated into the Databricks platform for real-time transactional applications that work alongside that lakehouse data. Lakebase can sync Unity Catalog tables so applications can query them at low latency. For application reads and writes that need a transactional database, Lakebase is the documented fit.

There is no universal ease-of-use winner; it depends on the workload. Databricks is a strong fit when work is Python-heavy, mixes engineering and data science, or uses notebooks that combine SQL and Python. Its SQL editor lets analysts query tables without learning Spark, so a SQL-first team can still use it for reporting. If your team is SQL-first, evaluate the SQL editor and a SQL warehouse on a representative workload before deciding, because the platform setup often matters as much as the product.

Free Edition is for non-commercial use, so it is not a production environment. It is serverless-only, so there are no classic clusters to configure, and R and Scala are not supported. Usage quotas apply: if you exceed your quota, the workspace's compute resources are shut down for the rest of the day. Outbound internet access is restricted to a limited set of trusted domains. It is a good place to learn the notebook, SQL, and Unity Catalog basics before deciding what a real environment needs.

Databricks Apps is the platform's way to build and deploy data and AI applications, and apps run on the serverless platform, so you do not provision hosting yourself. Apps integrate with Unity Catalog for data governance, Databricks SQL for querying data, and OAuth for authentication, so access control follows your existing governance rather than being rebuilt in each app. You still write the application code, which you can develop locally in Python or Node.js. If you only need a shared dashboard, check whether AI/BI dashboards cover it before adding an app.

No, that wait is compute provisioning rather than Spark itself. Databricks' job performance guide lists Waiting for resources as its own phase, separate from Running, and names slow VM launch time and quota or capacity limits as causes. For classic compute, the guide recommends instance pools with idle instance auto-termination to span the gap between scheduled jobs. Serverless compute is the other option, covered in the next question.

The 4 to 6 minute figure is the startup latency the docs give for standard performance mode, which the page describes as slightly higher startup latency that depends on compute availability and optimized scheduling. Performance optimized mode is for time-sensitive workloads, and the docs say that when it is enabled the job starts and runs faster, though the page gives no startup figure for it. You select it with the Performance optimized setting on the job details page, and it only affects the serverless tasks within the job. The docs also warn that jobs with large memory requirements or many tasks can see longer startup times.

The failing-jobs guide lists autoscaling removals, spot instance losses, and executors running out of memory as the most common reasons for executor removal. Check the compute's event log first to see which cause applies before you change memory settings. If memory is the cause, the guide links to its Spark memory issues page for next steps, so you avoid raising executor size blindly.

Genie Code has a Diagnose Error button in the results of a failed cell. Clicking it automatically runs /fix and shows the proposed change in a diff window. Select Accept to apply the change or Reject to keep the original code. Accepting does not run the cell, so you can review and run it yourself. Genie Code can also help identify and suggest fixes for errors from its pane, and /explain describes the code in a cell. Treat its suggestion as a fast first pass and verify it against the error.

The job run view splits duration into Queued, Waiting for resources, Library installation, and Running, so the missing time is in one of those phases. Queued means the run is waiting on concurrency limits, Waiting for resources means it is waiting for compute to become available, and Library installation means packages are installing before the run begins. Only the Running phase reflects your code. On serverless, the guide points you to query history, query profiles, and the job's Timeline view to find the slowest queries.

Not necessarily. Databricks' knowledge base describes driver unresponsiveness from memory pressure: the driver is up but not responsive, and the event log shows a Driver_Not_Responding event noting it is likely due to garbage collection. Common causes include an undersized driver, collect() or toPandas() on large data, and many notebooks or jobs running in parallel on one cluster. The article's easiest fix is to increase driver memory, and you should also limit large driver-side collection and separate workloads.

Databricks' knowledge base attributes this error to the cluster running out of memory, which can crash the Python kernel. Its suggested fixes are to run batch and non-interactive work on job clusters, right-size the cluster for the workload, reduce how much data is processed at once, and avoid large collect() or toPandas() calls. The article recommends monitoring with Ganglia metrics to see what the cluster is actually using.

Write the reusable project logic as ordinary Python code in an IDE, where it can be versioned, reviewed, and imported. Databricks Connect is a client library for IDEs such as Visual Studio Code, PyCharm, and IntelliJ IDEA. With it you write code using Spark APIs and run them remotely on Databricks compute instead of in a local Spark session. Notebooks remain useful for exploration, while the IDE-based project structure suits maintainable code.

Databricks Connect runs Spark operations on remote Databricks compute, so ordinary Python runs locally while Spark operations execute remotely. Some notebook-only features are not available through Connect: the Databricks Utilities credentials, library, notebook workflow, and widgets modules, plus RDDs and the Spark Context. The limitations page lists these explicitly. A good first step is to confirm a minimal remote session works, then isolate which operation hits a listed limitation.

Databricks recommends storing Python and R functions and their unit tests outside notebooks, so they work with standard test frameworks. The testing guidance also says it is generally a best practice not to run unit tests against functions that work with production data, especially functions that add, remove, or change data. Instead, it suggests fake data that is as close as possible to production, or running unit tests against views instead of tables.

Declarative Automation Bundles (formerly Databricks Asset Bundles) let you define jobs and pipelines as bundle configuration and use deployment targets to set environment-specific settings, such as workspace paths and access controls. The FAQ describes moving from manual deploys to Git-integrated automated workflows, where every deploy maps to a commit and bundle version. The rename is non-breaking: the bundle CLI command and existing configuration do not need to change.

Yes. The bundle init command creates a project from a template, which gives you a starting bundle to build on. If you already built jobs or pipelines in the UI, databricks bundle generate can generate configuration for them, and databricks bundle deployment bind associates that configuration with the existing workspace resource. The CLI can then validate, deploy, and run the bundle, and the overview links to a lifecycle page for each step.

Standard Git folders have documented limits: a working branch size of 1 GB, a recommendation to keep the total number of workspace assets and files under 20,000, and a 10 MB limit on files you can view in the Databricks UI. For larger repositories the docs recommend sparse checkout or Git CLI commands, and they advise against monorepos, suggesting you split the project into smaller repositories or use sparse checkout to limit which directories are cloned. Git folders with Git CLI access are another option, and the Git operations page covers what they support.

During a Git operation, some notebooks might be updated while others are not yet, which the limits page says causes unpredictable behavior. A job that points at a workspace folder path can therefore run a mix of versions. The page recommends configuring job tasks to use your Git provider as the source instead of a workspace path.

The Git folder troubleshooting guide covers the common causes: invalid credentials (you must enter both your Git provider username and token), a misconfigured linked account or provider under Settings > Linked accounts, missing repository permissions on your token, and SSO authorization that has not been completed. For SSL or connection errors, the guide says these typically indicate a network or TLS certificate problem. For clone timeouts, wait a few minutes, refresh the Git folder, and check whether the expected files are present. The guide also covers recovering from a detached HEAD by creating or selecting a branch.

Git folders with Git CLI access, which the Git operations page labels Public Preview, let you run any Git command, including git stash and interactive rebase (git rebase -i). They also support submodules, Git LFS, pre-commit hooks for linting and code scanning, and staging multiple commits locally before pushing. The page notes that UI Git operations such as pull, push, and commit require serverless compute. Check the page for current status and requirements before adopting CLI-enabled folders.

For user-defined functions, the local minor version of Python must match the minor version of Python on the Databricks Runtime of the cluster or serverless compute you connect to. Check the requirements page for the compatible combination of your local Python, Databricks Runtime, and Databricks Connect versions before you run UDFs remotely.

Yes. The Databricks extension for VS Code lets you debug the current Python file with Databricks Connect. Set breakpoints in the file, then click the Run on Databricks icon and choose Debug current file with Databricks Connect, or right-click the file and choose the same option. Output appears in the Debug Console pane. Spark operations run on remote compute while you step through the Python logic locally. Check the extension documentation for the setup supported for your cloud.

The docs I checked do not publish first-response targets by plan or severity. Those come from your Databricks contract. The support docs say an escalated case has the same response SLA as a first response on an Urgent case, as stated in your contract.

Submit the case as Urgent when it has critical, time-sensitive business impact. The first-response SLA for an Urgent case is stated in your Databricks contract. If an existing case becomes more urgent, use Escalate Case on the case details page, which alerts the on-duty manager that the case requires immediate attention. Keeping the case details and affected components in the first message speeds triage, and you should confirm your plan's coverage before an incident happens.

No. Solved cases close automatically after seven days without a response, which is a closure, not a deletion. To continue the case, open it from My Cases in the Help Center and click Reopen Case on the case details page. To find closed cases, use My Closed Cases under My Cases.

Authorized support contacts can open cases from inside the workspace, starting from the Genie Code icon in the upper-right corner or from Contact Support in the profile menu. If you cannot submit from the workspace, you can email help@databricks.com, but only if you are an authorized support contact. GovCloud and FedRAMP customers on AWS must email help@databricks.us instead. To see cases opened by colleagues, go to My Cases, choose Company Cases, and search.

Check the Databricks status page, which tracks service status per region and per component, so you can see whether the affected service is impaired in the region you use. Selecting an incident opens an incident page that shows its status, the affected components, and the affected locations. You can subscribe by email, webhook, RSS, or Slack to the components and regions you actually run, so you only hear about incidents that touch your workloads. Use it alongside your own job and workspace logs when you triage.

The docs I checked do not publish a platform-wide uptime percentage. Support and SLA terms come from your Databricks contract, so ask your account team which SLA applies to your order form.

Start with tagging and visibility, then add guardrails. Databricks documents tagging workspaces, clusters, SQL warehouses, and pools so usage can be attributed, and custom tags applied to workspaces and compute propagate to the system.billing.usage system table, where you can track spend. Serverless usage policies apply tags to serverless activity from users assigned to the policy. Set up budget email notifications for when a monthly budget is reached. Compute policies limit which instance types users can select. Budget notifications alert you; they are not a substitute for policy limits on runaway jobs.

Classic compute has an auto termination setting in the Performance section of the create compute form. You specify an inactivity period in minutes, and the compute terminates after that much idle time. Set it on interactive compute, and run scheduled work on job compute rather than all-purpose compute. The honest caveat is that a compute policy only limits the users who must use it, so remove unrestricted cluster-creation permission for teams that should be on approved policies.

Serverless SQL warehouses start in typically 2 to 6 seconds, compared with approximately 4 minutes for pro and classic warehouses, and they downscale quickly when demand is low. Databricks recommends serverless for query demand that varies greatly over time, so test whether a serverless warehouse fits your dashboard pattern. Compare spend before and after with system.billing.usage, and note that an audience needing warm capacity at all hours still pays for that uptime.

Partly, and Databricks documents it. For Azure Databricks, Standard tier has reached end of life: starting October 1, 2026, remaining Standard workspaces are automatically upgraded to Premium through a phased rollout. The Microsoft Learn page says you will not lose access to any capabilities, because higher tiers include the features available in Standard. Its feature table lists Unity Catalog as available on Premium and not on Standard. The page refers you to the Databricks pricing page to assess how the upgrade affects your bill, so check your agreement's numbers there rather than relying on a generic estimate. To manage the change, use system.billing.usage to measure impact per workload, move scheduled work to job compute, and evaluate serverless where it fits.

The honest tradeoff is overhead. For one analyst doing ad hoc work on a laptop-sized dataset, a local engine has less setup, and Databricks is not the right tool for every query. Databricks narrows that gap with single-node compute for small jobs, serverless compute that needs no cluster sizing, and Databricks Connect for the VS Code extension, which lets you run and debug individual Python files from VS Code against your selected compute. Where Databricks is strongest is when medium data must be shared, governed, scheduled, and combined with larger data later. Use a local engine for personal exploration and Databricks where the work needs governance, scheduling, or growth.

Databricks Git folders integrate Git repositories within your workspace, so you can clone, push, and pull a remote repository, create and manage branches, merge or rebase, and resolve merge conflicts from inside Databricks. Changes can be visually compared upon commit. Before rolling out, read the Git folder documentation for the operations and limits that apply to your workspace, since some Git operations can affect notebook state.

Start with the Azure Databricks networking overview, which groups the design into three connection points: users and applications to the workspace, the control plane and the classic compute plane, and the serverless compute plane and your resources. For classic compute, VNet injection deploys clusters into your own virtual network, and secure cluster connectivity enables outbound-only connections from clusters to the control plane with no open inbound ports. Private Link covers private connectivity for users and for the control plane, and serverless egress controls govern what serverless compute can reach. Treat each connection point as its own checklist, and note that Azure Databricks charges networking costs when serverless workloads connect to customer resources.

Free Edition is available at no cost to all qualified users, and Databricks states its accounts are meant for non-commercial use only. The limits are explicit: serverless compute only, a usage quota that shuts compute down for the rest of the day if exceeded (data and settings are not deleted), one workspace and one metastore per account, and no guaranteed reliability, support, or SLA. Use it to learn the platform, and use a paid workspace for production work.