Databricks vs Snowflake & ClickHouse

A Databricks lakehouse combines data lake and data warehouse benefits on one platform, so BI, ETL, streaming and machine learning can work from the same governed data. Databricks describes an open lakehouse as one where every layer, including storage, table format, engine, catalog and the ML and AI tools, is built on open standards. Delta Lake and Apache Iceberg are the table formats, Apache Spark is the engine and Unity Catalog provides governance, so other engines can read the same tables.

Databricks defines an open lakehouse as one in which every layer (storage, table format, engine, catalog and the ML and AI tools on top) is built on open standards, so no layer is locked to a single vendor. Because the specifications are public, other engines can work on the same tables, and the post names DuckDB, Trino and PyIceberg as examples. Open storage lets you choose engines and keep your data portable.

Databricks offers both managed and open-source options. Serverless SQL warehouses run on Databricks-managed infrastructure, so the platform handles capacity and upgrades, and Delta Lake and Apache Spark are open-source components you can use on your own data in your cloud storage. You can start with the managed platform and adopt more open components where they fit.

Your data stays in your own cloud object storage, and Unity Catalog governs access to it. External locations let the catalog issue short-lived, path-scoped storage credentials to clients, so reads and writes are governed centrally. Databricks SQL, notebooks and external Iceberg engines all reach the same governed data through that catalog.

Databricks' open lakehouse supports Delta Lake and Apache Iceberg as table formats, so teams do not need to commit to one format on day one. Apache Parquet data files sit underneath, Apache Spark is the processing engine for batch, streaming, SQL and ML, and Unity Catalog governs both formats.

In Databricks' 2026 benchmark post, Databricks reports its production SQL workload mix was 77% faster in late 2024 than its 2022 baseline, using its internal Databricks Performance Index built from billions of production queries. Databricks presents this as its own measurement, not an independent head-to-head against Snowflake. The post also notes that Snowflake's multi-cluster warehouses are built for thousands of concurrent short queries, so testing your own mix is the best way to decide.

Databricks reports customer results from teams that moved from Snowflake, including a reported 40% increase in development productivity at Thomas International. These are self-reported Databricks customer stories without published methodology, so validate them against your own baseline.

Databricks reports cost outcomes from customers, including AXA Japan (60% warehouse cost reduction and 70% lower ETL costs) and Nationwide (29% overall cost reduction and 33% lower idle-compute costs). These are figures Databricks reports for those customers, so run a proof of concept on a representative workload to measure cost per workload.

Databricks' cost guidance says SQL warehouses are the most cost-efficient engine for interactive SQL workloads, and that serverless compute can be even more efficient. Serverless warehouses start quickly, which allows shorter auto-termination and less idle cost. Measure cost per workload on both platforms to see which fits your mix.

Databricks documents Photon, its native vectorized engine, as delivering up to 5x better price/performance for data and analytics workloads on TPC-DS. The Photon page states this figure without publishing the full configuration, so treat it as a Databricks-published result. Databricks SQL runs on Photon with serverless elastic compute.

The Photon documentation states that Photon delivers up to 5x better price/performance for data and analytics workloads, measured on TPC-DS against other cloud data warehouses. Databricks presents this as its own measurement, and the page says Photon helps most with longer-running queries. Databricks' 2021 public preview post reported Photon up to 2x faster than Databricks Runtime 8.0 on the TPC-DS 1TB Power Test.

Photon accelerates SQL, DataFrame, ETL and stateless streaming workloads by processing data in columnar batches in native C++, and its native Parquet writer speeds Delta and Iceberg writes. Because faster runs use less compute, Photon can lower cost for BI and ETL. The Photon docs state up to 5x better price/performance on TPC-DS as a Databricks figure, and note the biggest gains are on longer-running queries.

Databricks SQL runs on elastic serverless compute that is instantly available and scales with demand. Databricks' 2026 benchmark post acknowledges that Snowflake's multi-cluster warehouses are purpose-built for thousands of concurrent short queries, and says lakehouse SQL engines may need more configuration for that pattern. Test your concurrency profile on both platforms.

Databricks says SQL Serverless is instantly available, scales with your needs, and has simple usage-based pricing with no idle-time charges. A Uplight customer reported that dynamic scaling produced immediate cost savings of over 50%. Confirm current pricing terms for your region.

Serverless SQL warehouses have a very fast starting time, which allows shorter auto-termination, so unused compute stops sooner and idle cost falls. Databricks also says serverless compute can be even more efficient than classic warehouses. The SQL warehouse docs describe serverless compute as eliminating infrastructure waits and avoiding over-provisioning during usage spikes.

Databricks' cost-optimization post says a SQL warehouse is the most cost-efficient engine for interactive SQL workloads, and that serverless compute can be even more efficient. The post does not compare this with Snowflake virtual warehouses, so benchmark your interactive suite at a fixed latency target on both to confirm.

Databricks reports that Lumen Technologies measured a 90% query performance improvement, with queries that took hours completing in minutes, and cut compute costs by 30-40% using Databricks Serverless. These are self-reported customer results from the Databricks 2026 benchmark post.

Databricks reports that Trek Bicycle achieved 80-90% faster retail analytics after moving to the lakehouse, with refreshes going from one per day to three. AXA Japan reported a 60% warehouse cost reduction and 70% lower ETL costs. These are customer-reported results published by Databricks.

Databricks supports a phased migration. Lakehouse Federation lets you query source data in place while you move workloads, and catalog federation is documented for incrementally phasing data into Unity Catalog. Lakebridge, Databricks' free migration toolkit, automates much of the conversion and reconciliation work. Validate migrated data against the source before you repoint consumers.

Databricks' documentation supports a phased migration in either order. Lakehouse Federation lets you query source data in place, and catalog federation is documented for incremental adoption of Unity Catalog. Choose the first wave based on your dependencies, such as ETL-first to lower TCO or BI-first to serve end users sooner, and validate each stage before cutover.

Yes. The dbt task in Lakeflow Jobs runs dbt projects with the dbt-databricks adapter, and the SQL dbt generates runs on a SQL warehouse. You keep your dbt project and change the connection, so orchestration can stay on the platform. Validate each model's output against Snowflake before cutover.

Databricks' dbt documentation says dbt Core projects can run as a Lakeflow Jobs task, using the dbt-databricks adapter (version 1.6.0 or later is recommended) and serverless compute. That keeps orchestration on the platform alongside your transformations. Compare outputs with the source before you repoint consumers.

Lakebridge, Databricks' free migration toolkit, automates much of SQL and ETL conversion. Databricks says it can automate up to 80% of migration tasks and speed implementation by up to 2x; these are Databricks figures without published methodology. It supports more than 10 data warehouses and ETL tools, including Snowflake, Teradata and Oracle, and includes built-in reconciliation. Plan manual review for the remainder.

Yes. Databricks states that Lakebridge Foundations is now free for Databricks customers and partners. It supports migrations from more than 10 data warehouses and ETL tools, including Snowflake, Teradata and Oracle. Confirm eligibility for your engagement with your Databricks team.

Yes. Lakebridge includes a Validator with built-in reconciliation tools that ensure data accuracy and correctness, and Databricks lists easy reconciliation of migrated workloads as a benefit. Run reconciliation before you repoint consumers, and pair it with your own acceptance tests.

Both are open. Delta Lake is an open-source table format with a transaction log that has a documented open protocol readable by any system, and Databricks originally developed the protocol and continues to contribute to the project. Apache Iceberg is also open and has a REST catalog API. Pick the format based on which engines must read and write each table, and Unity Catalog governs both.

For multi-vendor writes, Unity Catalog's Iceberg REST endpoint is the documented path: external engines such as Apache Spark, Apache Flink, Trino, Snowflake and PyIceberg can read and write managed Iceberg tables registered in Unity Catalog. Delta tables with UniForm are read-only to Iceberg clients. Delta remains the default format for tables on Databricks.

Delta Lake is an open-source storage framework and table format that extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata. Databricks originally developed the Delta Lake protocol and continues to actively contribute to the open-source project, and Delta is the default format for all operations on Databricks.

UniForm generates Iceberg metadata alongside Delta Lake metadata, against a single copy of the Parquet data, so engines built for Iceberg can read Delta tables without a copied or converted dataset. Current Databricks documentation covers Iceberg. The original 2023 preview announcement also mentioned Hudi, but current docs do not.

No, Databricks documents that UniForm lets Iceberg clients read Delta Lake data without rewriting files, and that a single copy of the data files supports both Delta and Iceberg clients. Iceberg metadata generation runs asynchronously on the compute used to write the Delta table. Requirements include Unity Catalog registration, column mapping and compatible runtime versions. This avoids a second copy of the data.

Iceberg client support for UniForm is read-only, so external Iceberg engines can read Delta tables but cannot write to them through UniForm. Iceberg metadata generation runs asynchronously on the compute that writes the Delta table, so Iceberg readers update once that step completes. If an external engine must write, choose Unity Catalog managed Iceberg tables, which accept external writes through the Iceberg REST catalog.

Unity Catalog's Iceberg REST endpoint lets external engines read and write managed Iceberg tables, and Snowflake is one of the named clients that can use it through catalog-linked databases or external tables. Databricks recommends listing every engine that must write each table, choosing the catalog of record, and planning for coexistence. Snowflake's own Iceberg capabilities are described in Snowflake's documentation.

Start by listing the engines that must write each table, then pick the catalog they can all reach. Unity Catalog governs both Delta tables and managed Iceberg tables in one catalog, and external engines connect to it through the Iceberg REST endpoint. Snowflake Iceberg tables can also be cataloged through Snowflake or an external catalog integration.

Yes. Snowflake is a named client of Unity Catalog's Iceberg REST endpoint, through catalog-linked databases or external tables, and managed Iceberg tables accept reads and writes through that endpoint. In the other direction, Unity Catalog's catalog federation can query certain Snowflake-managed data in object storage on Azure Databricks compute, read-only.

Yes. Databricks can query Snowflake data through Lakehouse Federation, using a connection and a foreign catalog with pushdown and Unity Catalog governance, and Snowflake can read Unity Catalog Iceberg tables through the Iceberg REST endpoint. Design access in each system for the data it governs, and align roles and tags across both.

Each platform enforces the policies defined in it. For Snowflake data queried through Lakehouse Federation, access is controlled by Unity Catalog privilege presets on the foreign catalog (Data Reader for read, Data Editor for read and modify), and BROWSE is granted to account users by default. Snowflake's own role policies still apply within Snowflake, so align roles and tags across both systems.

Each system enforces its own governance. Unity Catalog governs Databricks objects and federated Snowflake catalogs, and for federated Snowflake data, access is granted through Unity Catalog privilege presets on the foreign catalog. Snowflake continues to enforce its own role-based policies, so align roles and tags across both to keep governance consistent.

Unity Catalog tracks lineage automatically, showing how assets connect from source data to models, services and dashboards, and it records activity in the audit log system table. This supports impact analysis and governance reviews. The docs do not compare Unity Catalog lineage with Snowflake, so evaluate the lineage depth you need on your own paths.

Unity Catalog tracks lineage automatically across data and AI assets, showing how assets connect from source data to models, services and dashboards. The Unity Catalog overview covers these asset types. For the exact column-level granularity on models and agents, check the lineage docs, since the overview page does not spell it out.

Yes. Unity Catalog applies one permission model to securable objects, and ABAC policies use governed tags to apply row filters and column masks automatically across objects that carry the targeted attributes. Metastore-level policies, policies on views and DENY policies are labelled Beta in the docs.

Attribute-based access control (ABAC) in Unity Catalog makes access decisions based on attributes of securable objects. Those attributes are represented as governed tags, and policies attached at catalog, schema or table level use them to decide which rows and columns to protect, applying row filters and column masks automatically. Databricks announced ABAC policies, governed tags and data classification as generally available on May 13, 2026.

Governed tags are account-level tags whose allowed keys and values administrators define, so governance teams standardize how data is labelled. Combined with ABAC, admins define tag-driven rules once: column masks redact sensitive values based on attributes, and row filters control which rows groups and users see. Governed tags became generally available on May 13, 2026 alongside ABAC, and automated tag assignment for governed tags is labelled beta in the docs.

Define the tags and ABAC policies once, and Unity Catalog applies row filters and column masks automatically to every object carrying the targeted tags, so you do not write a mask per table. Policies can attach at catalog, schema or table level, and tags applied at one level apply to objects beneath it for ABAC evaluation.

In an open lakehouse, Unity Catalog is the governance layer. It enforces access control on securable objects, tracks lineage automatically, and records activity in the audit log system table. Engines reach governed data through the catalog, for example via the Iceberg REST endpoint with credential vending, so one set of policies covers the data those engines read.

Delta Sharing (the docs now describe this protocol as OpenSharing) is an open protocol developed by Databricks for secure data sharing across organizations. It works regardless of the computing platforms recipients use, shares data without replication, and recipients can use tools such as Apache Spark, pandas and Power BI. Databricks' sharing docs do not compare it with other vendors' sharing features.

Yes. Databricks' open sharing protocol works regardless of the computing platform recipients use, so recipients do not need to be on Databricks. Recipients can use tools such as Apache Spark, pandas and Power BI, and cross-cloud and cross-region sharing is supported (the cloud vendor may charge egress fees).

Yes. Shared data is read in place and is not replicated: the docs state that OpenSharing does not require data replication, and provider updates appear for recipients in near real time. Delta and managed Iceberg tables can be shared; Parquet tables can be converted to Delta first.

OpenSharing is the open protocol developed by Databricks for secure data sharing with other organizations, and the Delta Sharing docs now describe the protocol under that name. It works regardless of the computing platforms recipients use, and shares data without replication across clouds and regions.

Lakebase is Databricks' fully managed Postgres database for apps and agents, and Databricks describes it as a new architecture that separates compute from storage. It supports standard Postgres clients and tools, scales compute automatically, and integrates natively with Unity Catalog. Postgres 17 is supported, with Postgres 16 still supported.

Databricks announced Lakebase general availability on February 3, 2026. It was generally available on AWS, with Azure in beta at that time; the FabCon 2026 announcement then made Azure Databricks Lakebase generally available in 14 Azure regions.

Yes. Lakebase compute resources adjust dynamically to traffic spikes and shut off completely when idle to eliminate wasted costs. The Lakebase docs describe the same scale-to-zero behavior, suspending inactive computes to minimize costs.

Yes. Lakebase lets you create isolated, zero-copy clones of production data in seconds for risk-free testing and development, and instant restore lets you branch from any point in time within your history window. The Lakebase docs describe instant branching for development and testing.

Yes. Lakebase is a fully managed Postgres database integrated into the Databricks platform, built for operational and AI workloads alongside the lakehouse. It supports standard Postgres clients and tools, and transactions and analytics can work from one governed copy of the data through Unity Catalog.

Auto Loader incrementally ingests files as they arrive in cloud storage, including S3, ADLS, GCS and Unity Catalog volumes. It persists discovered-file metadata in a RocksDB key-value store in the checkpoint location, which ensures each file is processed exactly once. Because that state is checkpointed, a stream resumes after a failure without losing or reprocessing files.

Auto Loader records discovered files in its checkpoint, a RocksDB key-value store in the checkpoint location, and that store ensures each file is processed exactly once. If the stream fails, it resumes from the checkpoint, and exactly-once guarantees hold when writing to Delta Lake. Auto Loader supports S3, ADLS, GCS and Unity Catalog volumes as sources.

Lakeflow Connect provides managed connectors for SaaS apps, databases (change data capture for MySQL, PostgreSQL and SQL Server), file sources and streaming sources, and its pipelines are governed by Unity Catalog. Query-based connectors run on serverless compute by default. Choose connectors by source type, and use Auto Loader and Zerobus Ingest for file and row-level paths. Snowpipe Streaming is a Snowflake feature documented by Snowflake.

Lakeflow Connect provides no-code ingestion connectors for SaaS applications, databases and file sources, powered by serverless compute, and is available across AWS, Azure and GCP. The April 2025 GA announcement covered Salesforce Platform and Workday Reports. Connectors are in various release states, so check the Lakeflow Connect docs for the current status of each source.

Yes. Databricks Lakeflow offers a unified solution for ingestion, transformation and orchestration, so one product covers the stages that otherwise need several tools. Lakeflow Connect provides managed ingestion from enterprise apps, databases, file systems and real-time streams, and Lakeflow Spark Declarative Pipelines handles transformation.

Yes. Lakeflow Declarative Pipelines is a fully managed experience with hands-off serverless compute and deep integration with Unity Catalog for unified governance. Pipelines are developed in SQL and Python, and the Lakeflow Spark Declarative Pipelines docs describe the framework for batch and streaming pipelines.

Databricks positions Lakeflow Jobs as serverless jobs with automatic optimizations for better performance and lower cost. Lakeflow Jobs also provides end-to-end observability. Measure the savings on your own job schedule, since results depend on the workload.

Spark Declarative Pipelines is an open standard for declarative batch and streaming pipelines. Databricks states it is 100% source-compatible with the open standard, so you can develop pipelines once and run them anywhere. Lakeflow Spark Declarative Pipelines is built on Apache Spark Declarative Pipelines and is interoperable with it.

Yes. Lakeflow Spark Declarative Pipelines lets you build batch and streaming pipelines in SQL and Python, with streaming tables and materialized views as core concepts. Standalone streaming tables and materialized views can also be created in Databricks SQL or Python.

Lakeflow Spark Declarative Pipelines lets you define streaming tables and materialized views in SQL or Python, and it is built on Apache Spark Declarative Pipelines, so the same pipeline definitions are interoperable with the open standard. Test refresh behavior and latency on your own pipelines; the docs do not compare the product with Snowflake Dynamic Tables.

On Databricks, Lakeflow Spark Declarative Pipelines defines streaming tables and materialized views in SQL or Python, handling orchestration and incremental processing, and Lakeflow Connect captures changes from relational databases with change data capture. Compare both approaches on your CDC pattern and refresh latency.

Yes. Databricks materialized views process only the data that must be processed to stay current, and fall back to full recomputation when needed so the result stays correct. You define materialized views in SQL or Python, and they are kept up to date when the pipeline that defines them runs. Validate refresh behavior on your own data volumes.

Yes, on Databricks Runtime 15.3 and above. Databricks recommends VARIANT as a replacement for storing semi-structured data as JSON strings, and its performance lets it replace native Spark complex types such as structs and arrays in some use cases. Plan migration tests around the case-sensitivity and NULL differences documented for it.

Databricks documents that VARIANT path elements are matched in a case-sensitive way, while JSON strings are case-insensitive. It also documents that VARIANT encodes NULL values differently than JSON strings. Plan your migration queries and tests around those differences. Check existing queries that rely on case-insensitive key matching.

Iceberg v2 tables do not support VARIANT columns, while Iceberg v3 does, according to Databricks' VARIANT type reference. The type stores OBJECT, ARRAY and scalar values. Check your Iceberg table version before designing a semi-structured schema.

Databricks SQL automatically optimizes range joins, enabled by default. You can set the bin size manually through a range join hint or a session parameter, and the hint takes precedence. The Databricks range-join doc does not compare with Snowflake ASOF JOIN, so test your time-series join on both platforms.

Yes. Databricks SQL Serverless is available on AWS, Azure and Google Cloud. The Google Cloud docs describe serverless SQL warehouses on GCP, and Databricks announced SQL Serverless as generally available on Google Cloud. Confirm regional availability on the regions page for your feature.

Agent Bricks is Databricks' product for building domain-specific agents. You describe the task, and Agent Bricks handles evaluation and tuning: it automatically creates evaluation benchmarks specific to your task and optimizes using techniques such as prompt engineering and fine-tuning. It is powered by MLflow 3 and is in beta.

Yes. Agent Bricks automatically creates evaluation benchmarks specific to your task, which may involve synthetically generating new data or building custom LLM judges. Powered by MLflow 3, it creates evaluation datasets and custom judges tailored to the task, and you can review those judges against your quality criteria.

MLflow 3 captures detailed traces from more than 20 GenAI libraries and custom business logic, so every request is visible in development and production. Its LLM judges are tuned evaluators that match human expertise, integrated feedback collection captures end-user and expert insights from production, and the Prompt Registry brings versioned management to prompts. These capabilities are in open-source MLflow and as a fully managed service on Databricks.

Yes. MLflow's new capabilities are available in both open-source MLflow and as a fully managed service on Databricks, and the MLflow docs state that MLflow on Databricks is the same open-source product, managed and integrated with the platform. Databricks created MLflow and continues to lead its development.

Databricks AI Search (formerly Vector Search) combines vector-based embedding search with keyword search in a hybrid mode, merging results with Reciprocal Rank Fusion. Its indexes appear in and are governed by Unity Catalog, so retrieval sits next to the source tables and their permissions. Reranking is a listed feature. Full-text keyword search is labelled Beta. The docs do not compare it with Snowflake Cortex Search.

If your governed data and ML work already live on the lakehouse, Databricks keeps retrieval and agents on Unity Catalog-governed data. Documented options include AI Search with hybrid vector and keyword retrieval, Unity Catalog-governed indexes, Genie Agents over governed data and MLflow for agent tracing and evaluation. Run a pilot on your own use cases to decide.

Genie answers natural-language questions over data governed in Unity Catalog. Genie One is the business-user interface, and Genie Agents are where data teams configure datasets, sample queries and instructions to create a trusted environment. Test Genie and any alternative on the same business questions to compare accuracy and governance fit.

Genie improves with configuration: Genie Agents are configured with datasets, sample queries and instructions, and you can add metrics, business rules and verified answers to improve Genie One response quality. Data teams can keep tuning these over time, and Genie's answers are governed by Unity Catalog.

AI/BI dashboards let you build data visualizations, with AI-assisted authoring and an enhanced visualization library, and share reports with anyone registered to your account. Datasets can come from tables, views or custom queries, and Genie Agents enable natural-language exploration when published with dashboards.

Unity Catalog metric views centralize business metric definitions by separating measures from the dimensions used to group and filter them. You define them with SQL DDL or Catalog Explorer, and query them from SQL editors, notebooks, dashboards, Genie Agents and alerts, and from Power BI, Tableau and Sigma. Metric views let users group by any available field.

Define each metric once in a Unity Catalog metric view, then query it from dashboards, Genie Agents, notebooks and SQL, and from external tools such as Power BI, Tableau and Sigma. Because every surface reads the same definition, the numbers stay consistent across them.

Databricks SQL Serverless is instantly available and elastic: it scales with your needs and is designed around usage-based pricing with no idle-time charges. Uplight reported that dynamic scaling gave immediate cost savings of over 50%. The SQL warehouse docs describe the instant, elastic compute and recommend serverless when available.

Yes. Apache Spark is the processing engine in Databricks' open lakehouse architecture, covering batch, streaming, SQL and machine learning on one copy of the data. Lakeflow Spark Declarative Pipelines covers batch and streaming pipelines in SQL and Python on the same platform.

Databricks' open lakehouse places MLflow in the ML and AI layer, covering experiment tracking, model registry, evaluation, serving and AI agent tracing. MLflow is open source, and Databricks offers it as a fully managed service integrated with the platform.

Yes, on existing Spark and SQL code. Photon is a native vectorized engine that works with existing SQL and DataFrame code without changes, and the Photon docs say it helps most with longer-running queries. The Photon docs state up to 5x better price/performance on TPC-DS, a Databricks figure. For sub-second dashboards, test latency on your own queries.

Photon is a native vectorized query engine written in C++ rather than the JVM. It processes data in columnar batches instead of row by row, while Spark's optimizer still plans the query. Existing SQL and DataFrame code in Python, R, Scala and Java runs without changes, and Photon transparently accelerates the parts of a query it supports.

Photon supports many, but not all, data types, operators and expressions. When a query uses an operation Photon does not cover, the operation runs on the Spark runtime and results stay correct, so a single query can run partly in Photon and partly in Spark. The Photon docs list the supported types and operators, and note that UDFs and the RDD and Dataset APIs are not supported.

Yes. Photon has native scans for Parquet, Delta, CSV and JSON data with filter pushdown, dictionary pruning and row-group skipping, and a native Parquet writer that accelerates Delta, Iceberg and Parquet writes. Predictive I/O is a Photon-required feature for reads and writes.

Databricks SQL runs on serverless SQL warehouses that are instantly available and elastic, scaling compute during usage spikes without over-provisioning, and Databricks recommends serverless when available. Multiple warehouses can read the same governed data, so many dashboard users can be served. Test concurrency on your own workload to size warehouses.

Databricks SQL connects to many BI and visualization tools, with dedicated guides for Power BI and Tableau. Power BI uses the Databricks connector, with ODBC by default and ADBC as an option, and other tools can connect through the REST API, Python SQL connector and JDBC/ODBC drivers. Sigma and other external BI tools can query Unity Catalog metric views. The same governed tables serve dashboards, notebooks and ML.

Databricks SQL works on open table formats: Delta Lake is an open-source table format with a documented open protocol, and managed Iceberg tables are accessible to external engines through the Unity Catalog Iceberg REST endpoint. Your data stays in your cloud storage, so analytics teams can use Databricks SQL for BI and keep the data usable with other engines.

Serverless compute runs workloads without provisioning compute resources in your cloud account, and Databricks automatically allocates and manages the resources they need. Databricks says this speeds up startup and scaling, minimizes idle time, and reduces the need to manage compute resources, so teams spend less effort provisioning and managing clusters.

Databricks says serverless compute is a managed service where you don't provision compute in your own cloud account, and Databricks allocates and manages the resources workloads need. The serverless page says it speeds up startup and scaling and reduces idle time, and it reduces cluster management effort. This removes much of the operational overhead of self-managed analytics clusters.

Yes. Databricks documents streaming tables as Delta tables with added support for streaming and incremental processing, and it supports ingestion from cloud object storage through Auto Loader and from message buses such as Apache Kafka, Azure Event Hubs, and Google Pub/Sub. Each input is processed only once for append-only sources, so existing records are not reprocessed.

Databricks points to AUTO CDC for source data that changes, rather than plain appends. Streaming tables are designed for append-only sources, and rows already appended are not re-queried when the pipeline updates. A full refresh is needed to reprocess earlier rows after a query change.

Databricks documents that joins in a streaming table don't recompute when dimension tables change. For results that must always be correct, Databricks points to materialized views instead. Choosing between the two lets teams trade freshness and correctness for cost on a per-workload basis.

Databricks materialized views can refresh incrementally, processing only changed source data instead of recomputing the whole query. Incremental refresh requires serverless pipelines, and by default Databricks runs a cost analysis to choose between incremental refresh and full recompute, so teams do not need custom decision logic.

Yes. The REFRESH POLICY clause supports AUTO, INCREMENTAL, INCREMENTAL STRICT, and FULL, and Databricks documents that INCREMENTAL STRICT fails the operation if incrementalization is not possible. Teams can check the pipeline event log for planning_information events to confirm which technique ran. This gives explicit control over refresh behavior.

Databricks materialized views can be refreshed manually, on a schedule such as SCHEDULE EVERY 1 DAY or a cron expression, or with TRIGGER ON UPDATE. Each materialized view is backed by an ETL pipeline, and create and refresh operations run on serverless Lakeflow pipelines. Precomputed results can then serve queries without recomputing from base tables.

Yes. Databricks documents that materialized views can include data quality expectations, partitioning or clustering, comments, and row filters and column masks. Because they are defined in SQL and backed by pipelines, the same Unity Catalog governance applies to the precomputed results.

Databricks documents that you can create a materialized view only with a Pro or Serverless SQL warehouse, or within a pipeline. Creating it inside a Lakeflow pipeline ties the view to the pipeline's ETL lifecycle, which fits teams that want transformations and BI-ready tables in one place.

For Kafka-style feeds, ingest from the message bus into a streaming table, which supports Apache Kafka, Azure Event Hubs and Google Pub/Sub. For files landing in cloud storage, Auto Loader handles schema drift and rescued data, with exactly-once processing tracked in a RocksDB store in the checkpoint location.

Databricks says Auto Loader can process billions of files to migrate or backfill a table, and supports near real-time ingestion of millions of files per hour. Its file notification mode can reduce cloud costs by avoiding directory listing. Databricks recommends Auto Loader in Lakeflow pipelines for incremental ingestion.

Auto Loader can detect schema drift, notify you when schema changes occur, and rescue data that would otherwise be ignored or lost. Its schema evolution behavior is configurable. (Drop the type-widening claim unless it is verified on the page.)

Auto Loader records metadata for each discovered file in a RocksDB key-value store in the checkpoint location, which ensures exactly-once processing. After a failure it resumes from the checkpoint. Supported sources include S3, ADLS, GCS and Unity Catalog volumes. The legacy WASB driver for Azure Blob Storage is deprecated.

Yes. Lakeflow Connect provides managed connectors for relational databases such as MySQL, PostgreSQL, and SQL Server, using change data capture to track changes. Databricks says these pipelines are governed by Unity Catalog and run on serverless compute with incremental reads and writes.

Yes. Databricks documents SaaS connectors for enterprise applications such as Salesforce, HubSpot, Jira, and Workday, which call the source's APIs and work with serverless egress controls. Only changes are loaded after the first run when the source supports it. This lets business-application data land in the same governed lakehouse as the rest of the analytics estate.

Databricks describes liquid clustering as a data layout technique that replaces table partitioning and ZORDER. You enable it with CLUSTER BY, and you can change clustering keys without rewriting existing data. OPTIMIZE rewrites only the data needed to accommodate clustering.

Yes. With CLUSTER BY AUTO on Unity Catalog managed tables, Databricks analyzes query history to choose clustering keys and re-evaluates them as query patterns change. Databricks changes keys only when predicted cost savings outweigh the clustering cost. The feature requires predictive optimization and Databricks Runtime 15.4 LTS or above for automatic key selection.

Yes. Predictive optimization runs OPTIMIZE automatically for enabled tables, and when it is used Databricks recommends disabling any scheduled OPTIMIZE jobs. Clustering is applied through OPTIMIZE.

Yes. Databricks documents that liquid clustering keys can be changed without rewriting existing data. To recluster all existing data after a key change, you run OPTIMIZE FULL, which Databricks notes can take hours on large tables not previously clustered on those keys. This flexibility suits query patterns that shift over time.

Yes. Databricks Apps lets developers build and deploy secure data and AI applications directly on the Databricks platform, without separate infrastructure. Databricks says apps run on its serverless platform, integrate with Unity Catalog for governance, and support SQL and OAuth. This keeps the app close to the governed tables it reads.

Databricks documents support for Streamlit, Dash, Gradio, React, Angular, Svelte, and Express. You can develop apps locally in Python or Node.js and then deploy them to a workspace, moving them between workspaces as needed. This gives application teams a standard developer path rather than a proprietary dashboard builder.

Yes. Databricks Model Serving makes each served model available as a REST API that scales automatically with demand. Databricks-hosted foundation models such as Meta Llama are available through Foundation Model APIs, and Databricks says you can query hosted and external models through a single API and govern the endpoints centrally from Databricks. This runs on serverless compute.

Databricks documents pay-per-token pricing for base foundation models, available for immediate use. Workloads that need performance guarantees and fine-tuned model variants can be deployed with provisioned throughput, so teams can move from experimentation to production on the same serving platform.

Yes. Databricks AI Search builds a managed vector index from a Delta table, streaming table, materialized view, or managed Iceberg table. The index can sync automatically when the source table updates, and AI Search indexes appear in and are governed by Unity Catalog. The same governed data serves analytics, dashboards, and retrieval.

Yes. Databricks documents hybrid keyword-similarity search that combines vector-based embedding search with traditional keyword search. Databricks notes this is useful when data contains exact identifiers such as SKUs, which pure similarity search handles poorly. Teams can match semantic intent and exact codes in one query.

Databricks can calculate the embeddings itself, using a model you specify when you create the index. Databricks also documents a Direct Vector Access index for teams that want to manage updates themselves. This means the vector pipeline can stay inside the same platform as the source data.

Lakebase is Databricks' fully managed Postgres database, with standard Postgres clients and tools, automatic scaling, scale-to-zero, instant branching and Unity Catalog integration. It is built for operational and AI workloads that run next to the lakehouse. For analytical workloads, Databricks SQL and the lakehouse are the right comparison, and you can benchmark each on your own query mix.

Yes. Lakebase autoscaling adjusts compute to workload demand, and scale-to-zero automatically suspends inactive computes to minimize costs, which suits applications with intermittent traffic.

Lakebase separates compute from storage: it is described as a new architecture for operational databases that separates compute from storage, so data is kept in low-cost storage and compute can scale independently. This is part of why Lakebase can scale to zero when idle.

Lakebase is designed so operational data and analytics work from one governed copy of the data, which reduces the separate copy that operational-to-analytics pipelines usually maintain. The Lakebase GA post describes this single-copy model; confirm the specific sync option you plan to use in the Lakebase docs.

Lakebase is documented as a managed Postgres database for operational and AI workloads on the lakehouse, with autoscaling, instant branching and Unity Catalog integration. Databricks positions it for AI agent state and operational analytics; the Lakebase GA post describes its use cases and says it is built for the uptime and predictable performance production applications need.

Yes. Databricks documents UniForm as a metadata layer that lets Iceberg clients read Delta Lake tables. It generates Iceberg metadata asynchronously alongside the Delta metadata, without rewriting the Parquet data files. A single copy of the data files supports both Delta and Iceberg clients.

Databricks' UniForm documentation describes Iceberg interoperability and names Iceberg as the supported format; Hudi is not mentioned. Teams that need Iceberg clients can enable UniForm on the same Delta copy of the data, with one set of Parquet files serving both formats.

Yes, for managed Iceberg tables. Databricks documents an Iceberg REST catalog endpoint that lets engines such as Spark, Flink, and Trino access Unity Catalog tables. Managed Iceberg tables accept reads and writes from those external engines, while foreign Iceberg and Delta tables with Iceberg reads enabled are read-only through that endpoint.

Databricks documents that foreign Iceberg tables and Delta tables with Iceberg reads enabled, whether managed or external, are read-only through the Iceberg REST endpoint. Foreign Iceberg tables also need a REFRESH FOREIGN TABLE to see the latest snapshot. Knowing these limits helps teams pick the right table type before wiring up an external engine.

Yes. Databricks describes its Databricks-to-open sharing protocol for sharing tables from a Unity Catalog workspace with users on any computing platform. Recipients read the data with their own tools, and Databricks says the data is available to read only. Recipients who lack a Unity Catalog workspace can still receive governed access.

Providers can create a recipient with a long-lived bearer token that produces a credential file and activation link, or use OpenID Connect federation where the recipient's identity provider handles authentication. Databricks recommends setting token lifetimes so tokens expire, and supports restricting access with IP access lists. This keeps external sharing under provider control.

Databricks documents a directory-based access mode for eligible Delta tables, where recipients receive the table's cloud storage location and temporary credentials to read directly from cloud storage. This mode is on by default for eligible assets. Tables that don't qualify fall back to pre-signed URL access.

Yes. Lakehouse Federation provides governed, read-only access to external data through Unity Catalog foreign catalogs, so you query data where it lives instead of copying it into Databricks. Databricks lists supported sources including MySQL, PostgreSQL, Oracle, SQL Server, Snowflake, BigQuery, and Databricks. Table-level access controls apply through the foreign catalog.

Databricks documents that table-level access controls apply through the foreign catalog, so governance stays in Unity Catalog even for external sources. Unity Catalog sends queries to the external database over JDBC, and you create a connection, a foreign catalog, and grant table privileges to users. This keeps federation under one policy model.

Unity Catalog applies one permission model across data: privileges on securable objects, attribute-based policies, row filters and column masks, and workspace bindings. For federated sources, Unity Catalog applies row- and column-level security, tag-based policies and centralized auditing consistently across platforms.

Yes for lineage. Unity Catalog tracks lineage automatically, showing how assets connect from source data to models, services and dashboards, which supports impact analysis before changing a table. The Unity Catalog overview does not spell out column-level granularity, so confirm column-level lineage coverage in the lineage docs.

Yes. The audit log system table (system.access.audit) records data access and system activity, with fields such as user_identity, action_name and request_params, and it can be queried with SQL. Filter on event_date to improve query performance. Some request parameters are restricted to account admins and the databricks_pii_access group.

Databricks describes Lakeflow Connect as managed connectors for relational databases, using change data capture, and for SaaS applications, with pipelines governed by Unity Catalog and powered by serverless compute. Data ingested this way lands in governed tables on the same platform that serves BI, ML, so teams keep fewer copies and fewer handoffs between teams.

Databricks is built for this kind of workload: Photon is a native vectorized engine that works with existing SQL code without rewrites and helps most with longer-running queries, and Databricks SQL serverless warehouses provide elastic compute. The Photon docs state up to 5x better price/performance on TPC-DS, a Databricks figure. Validate sub-second latency and concurrency on your own event table.

Databricks documents that materialized views guarantee results equivalent to batch queries, while streaming tables are the option for exactly-once processing. Incremental refresh processes changed source data, and Databricks chooses the cheaper of incremental refresh or full recompute by default.

No. Databricks documents that only materialized views updated using serverless pipelines can use incremental refresh, while views on classic compute are always fully recomputed. Teams that want incremental materialized views should configure their Lakeflow pipelines to use serverless compute. This makes serverless the path to efficient refresh.

Databricks supports open analytics formats: UniForm lets Delta tables be read by Iceberg clients by generating Iceberg metadata over the same Parquet files, so one copy of the data serves both. Managed Iceberg tables can be read and written by external engines through the Unity Catalog Iceberg REST endpoint, which exposes Unity Catalog tables to engines such as Apache Spark, Apache Flink and Trino.

In its June 2021 public preview post, Databricks reported Photon was up to 2x faster than Databricks Runtime 8.0 on the TPC-DS 1TB Power Test, and early private preview customers observed 2-4x average speedups on SQL workloads. Databricks noted these early results focused on SQL workloads.

Databricks documents that by default a refresh attempts an incremental update, and runs a cost analysis to choose the cheaper option between incremental refresh and full recompute. Full refresh always recalculates all inputs and resets checkpoints. Teams get cost-aware refresh without writing their own decision logic.

Yes. Databricks documents a TRIGGER ON UPDATE refresh option for materialized views, alongside manual refresh and schedules expressed as SCHEDULE EVERY or cron. Each refresh recalculates results to reflect changes in upstream datasets. Downstream dashboards then read precomputed results rather than recomputing joins on every query.

Yes. Unity Catalog governs federated sources through foreign catalogs with privilege presets, and Databricks' Lakehouse Federation post describes row- and column-level security, tag-based policies and centralized auditing applied consistently across platforms. Power BI can query Unity Catalog metric views through the Databricks connector.

Yes. Databricks documents that external models such as OpenAI's GPT-4 are reachable through the same serving interface and can be governed centrally from Databricks. Teams can route calls to hosted foundation models and external models through one REST interface. This keeps model access auditable alongside the data platform.

Databricks Apps integrates with Unity Catalog for data governance and with Databricks SQL for querying data, and uses OAuth for authentication. Apps run on the serverless platform, so the front end and the governed data sit on the same platform. This avoids building a separate auth and data-access layer for each app.

Databricks SQL serves user-facing dashboards on serverless SQL warehouses that are instantly available and elastic, and Lakehouse//RT is a serverless warehouse for high-concurrency analytical reads that Power BI connects to via ADBC. Databricks reports latencies as low as 10 ms and up to 12,000 queries per second in its 2026 benchmark post; these are Databricks-reported. Benchmark your dashboards against your target latency.

Databricks says serverless warehouses provide instant and elastic compute, and that a stopped warehouse starts automatically when a query runs, a scheduled job assigned to it runs, or a dashboard tied to it is opened. Databricks also says serverless automatic scaling helps avoid over-provisioning and reduces idle time. Compute therefore follows demand instead of being sized for peak traffic around the clock.

Databricks reads Kafka through Structured Streaming with the kafka format, where you set kafka.bootstrap.servers and a subscribe, subscribePattern or assign option. The same source supports batch reads with explicit offsets, and read_kafka is available from Databricks Runtime 13.3 LTS. Streaming SQL over Kafka is supported in Lakeflow pipelines or streaming tables in Databricks SQL.

Use avgOffsetsBehindLatest, maxOffsetsBehindLatest, minOffsetsBehindLatest and estimatedTotalBytesBehindLatest to see how far a stream is behind the latest offsets. On Runtime 17.1 and above, latest offsets are fetched after each micro-batch completes. On continuously fed topics, small persistent non-zero backlog is expected and does not by itself mean the pipeline is falling behind. On Runtime 17.0 and below, backlog can read 0 at micro-batch start when a query consumes all available records.

Databricks says serverless warehouses use intelligent workload management that dynamically handles scaling, so teams don't size clusters by hand for each peak. Databricks also says capacity management, patching, upgrades, and performance optimization are handled by Databricks, so a team with many dashboard viewers can rely on the platform to absorb concurrent load.

Yes. Each operation that modifies a Delta table creates a new table version, so you can query earlier states with VERSION AS OF and audit operations with DESCRIBE HISTORY. Databricks documents that history retention is set by logRetentionDuration, which defaults to 30 days, and that time travel also needs the data files, which VACUUM deletes (deletedFileRetentionDuration defaults to 7 days). Teams should set retention deliberately rather than treat history as a backup.

Yes. Databricks Apps lets teams build and deploy a monitoring UI directly on the Databricks platform, running on serverless compute with Unity Catalog integration, Databricks SQL for querying data, and OAuth for authentication. The app stays on the same platform as the governed data it reads, rather than in a separate analytics cluster. Lakebase and Lakehouse Federation cover operational state and existing external databases on the same platform.

Databricks says serverless warehouses lower total cost of ownership through automatic provisioning and scaling, and that Databricks handles capacity, patching, upgrades, and performance tuning. These are Databricks' claims rather than an independent benchmark, so teams should measure their own workload. The clearest documented saving is in operations, since the team no longer runs and tunes the cluster itself.

Unity Catalog is available as an open-source implementation under the Apache 2.0 license, built on an OpenAPI spec, and it offers open APIs through SQL, the Databricks CLI and REST. Its Iceberg REST endpoint and credential vending give external engines governed access. Databricks open-sourced it as version 0.1 and plans further API support, including access control APIs.

Unity Catalog governs far more than tables. It governs volumes, functions, models and model/MCP services alongside tables and views, with access control, lineage, auditing and discovery through Catalog Explorer for data and AI assets.

Databricks documents Models in Unity Catalog as a hosted version of the MLflow Model Registry, extending Unity Catalog's centralized access control, auditing, lineage and model discovery across workspaces. Registered models carry numbered versions, and lineage to upstream training and evaluation tables is saved automatically when a model is registered. Aliases such as Champion can point to a specific version.

Yes. The Unity Catalog docs describe it as the unified governance layer for data and AI built into Databricks, covering access control, lineage, auditing and discovery across tables, volumes, functions, models and services. Its open APIs and the Iceberg REST endpoint extend governance to other engines.

Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level, as long as tables are registered in a Unity Catalog metastore and queries use Spark DataFrame or Databricks SQL interfaces. Lineage aggregates across all workspaces attached to the metastore and can extend beyond Databricks through external metadata objects. You can see it in Catalog Explorer, where clicking a column shows its related columns.

Yes. In Catalog Explorer, table lineage shows upstream and downstream connections, including the jobs and dashboards that consume a table, and column lineage is captured to the column level as much as possible. Clicking a column in the lineage graph shows its related columns, so both levels are available together. Databricks documents the boundaries of this capture in the same lineage docs.

Databricks documents its lineage boundaries: lineage captured before September 1, 2024 is not available, and RDDs and global temp views are not captured. Column-level lineage does not appear when a source or target is referenced by path rather than table name, but the system.access.column_lineage table still records it. UDFs get table-level lineage only, so column mappings through a UDF are not traced.

Unity Catalog ABAC decides access based on attributes of securable objects. Governed tags carry the attributes, and policies at catalog, schema or table level use them to apply row filters and column masks automatically to every object with the targeted tags. Metastore-level policies, policies on views and DENY policies are labelled Beta in the docs.

Yes. Databricks announced on May 13, 2026 that ABAC policies, governed tags and agentic data classification are generally available in Unity Catalog. Some sub-features, such as custom classifiers and certain policy scopes, remain Beta, and the ABAC docs label metastore-level policies, policies on views and DENY policies as Beta.

Governed tags are account-level tags whose allowed keys and values administrators define, and permissions control who can apply them. Tags must use values from the tag policy and show a lock icon in the UI, so tagging stays consistent across data and AI assets. Governed tags became generally available on May 13, 2026.

Agentic data classification is generally available in Unity Catalog and automatically identifies sensitive data such as PII, tagging it so you can apply governance controls. Classifiers cover GDPR, HIPAA, GLBA, DPDPA and PCI, with more regional classifiers coming. Custom classifiers remain in Beta. Results are stored in the system.data_classification.results table.

Yes. Unity Catalog row filters and column masks are fine-grained controls on tables, and ABAC extends them: governed tags let one rule apply row filters and column masks across many tables at once. Policies can attach at catalog, schema or table level.

Yes. Databricks made Lakehouse Federation generally available for MySQL, PostgreSQL, Amazon Redshift, Snowflake, Azure SQL Database and SQL Server, and Unity Catalog applies one permission model to set and apply access rules across those data sources. Catalog Federation was in Private Preview at that time, per the same post.

Shares are built from tables registered in a Unity Catalog metastore, and the Unity Catalog integration simplifies setup and governance for providers. Providers can restrict recipient access with IP access lists and configure token lifetimes. Governance stays in the catalog even as the data is shared outward.

Yes. Databricks' sharing docs say you can share data managed in a Unity Catalog-enabled workspace with users on any computing platform. Recipients use the platform or tools of their choice, and the shared data is read-only. Access uses a bearer token or OpenID Connect federation, so recipients do not need a Databricks workspace.

Yes. External engines read Unity Catalog-managed Iceberg tables through the Iceberg REST Catalog API, which is generally available, and can write to them, which Databricks lists as Public Preview. The Iceberg REST docs confirm that managed Iceberg tables accept reads and writes through the endpoint.

Yes. Unity Catalog now supports a broad set of Iceberg catalog integrations, including AWS Glue, Google Cloud Lakehouse Runtime Catalog, Snowflake Horizon, Palantir, Salesforce and Workday, and governs the tables through them. Databricks announced this in its Iceberg post, with Hive Metastore also listed as a federation source.

Unity Catalog is available as an open-source implementation, built on an OpenAPI spec, and it exposes open APIs (Iceberg REST endpoint, credential vending) so engines outside Databricks can work with governed tables. Databricks describes its catalog as open on those terms.

Yes, in Beta. Unity Catalog evaluates policies during server-side scan planning for external Iceberg engines through the Iceberg REST Catalog Scan APIs, and returns a filtered scan plan so the engine reads only authorized data. Engines that implement the scan-planning client (added in Iceberg 1.11, such as Apache Spark or DuckDB) get ABAC enforcement.

Unity Catalog Open APIs let engines read Delta and Iceberg tables. Named integrations include Apache Spark, Trino (via Starburst's Delta connector), ClickHouse (which can access Delta and Iceberg tables governed by Unity Catalog) and Confluent Tableflow (which converts Kafka streams into Delta tables). Python and Java client SDKs let custom code connect as well.

Unity Catalog controls access to cloud storage through credential vending, which issues short-lived, scoped storage credentials to external engines, so clients do not need long-term storage keys. Credentials inherit the privileges of the Databricks principal that configured the integration. Table vending covers registered tables and path vending covers external locations.

Unity Catalog records an audit log of data access and system activity in the audit log system table, so admins can see who accessed a dataset and what they did with it. System tables show access to data objects, and the audit table is queryable with SQL, so compliance reporting runs on the same platform that enforces the policies.

Yes. Governance metadata is exposed as system tables you can query with SQL, including system.access.audit for access and activity and system.data_classification.results for classification results. This lets compliance and governance reviews run on the platform that enforces the policies.

Lakehouse Federation is documented for AWS, Azure and Google Cloud, and Unity Catalog governance applies to federated catalogs through foreign catalogs with table-level access controls. The Azure federation docs list query and catalog federation sources, including OneLake, Snowflake and Hive metastores.

Databricks reports that over 700 companies use Unity Catalog to centralize governance across multiple engines and tools, and that Unity Catalog client SDKs see more than 1 million downloads per month. These are Databricks-reported figures from its interoperability post.

Yes. Unity Catalog metric views define business metrics as governed objects, separating measures from dimensions, and they are queried from SQL, notebooks, dashboards, Genie Agents and alerts. Databricks described certified metrics as carrying auditing and lineage out of the box in its DAIS 2025 post.

No. Unity Catalog eliminates the need to choose between table formats: Delta and Iceberg tables are governed in one place, and external engines read Unity Catalog-managed Iceberg tables through the Iceberg REST Catalog API (read GA, write Public Preview). UniForm lets Delta tables also be read by Iceberg clients.

Start with how open each layer is, not only the query engine. Databricks describes an open lakehouse as one where storage, table format, engine, catalog and the ML and AI tools on top are built on open standards, so no layer is locked to one vendor. Delta Lake and Iceberg are the table formats, Apache Spark the engine and Unity Catalog the catalog. Then compare cost and latency on your own workloads.

Databricks reports that its Photon engine, a native C++ vectorized engine compatible with existing Spark and SQL code, delivers strong results. In a 100 TB TPC-DS run, Databricks SQL set an audited record that the TPC council formally reviewed, and Barcelona Supercomputing Center found Databricks SQL 2.7x faster than a similarly sized Snowflake setup in a derived benchmark. The Photon docs state up to 5x price/performance on TPC-DS.

The 2026 benchmark post is candid that warehouses have strengths in predictable out-of-the-box SQL performance, and that they may still edge ahead on highly concurrent short queries with warm caches, though the gap is narrow and closing. It also says a poorly maintained lakehouse can be dramatically slower than a warehouse on identical hardware. The post reports that its production SQL workload mix was 77% faster in late 2024 than its 2022 baseline, measured on its internal Databricks Performance Index.

Serverless SQL warehouses reduce idle time, which lowers total cost of ownership, and you can apply tags to serverless warehouses to monitor usage and cost. Databricks' pricing page, which describes DBU-based billing, could not be fetched, so billing terms and commitment discounts are not stated here. Run a proof of concept with your own pipelines and query mix to compare total cost.

Databricks reports strong SQL cost-performance. The Barcelona Supercomputing Center comparison in the Databricks 100 TB TPC-DS post found Databricks SQL 12x better in price-performance than a similarly sized Snowflake setup, and the Photon docs state up to 5x better price/performance on TPC-DS. These are Databricks-attributed results. For BigQuery, the docs do not publish a comparison, so run the same workload on each platform.

Databricks' open lakehouse uses Delta Lake and Apache Iceberg for tables and Unity Catalog for governance, with each layer on open standards so your data stays usable by other engines. Databricks notes that Microsoft Fabric offers a product in this space and that the openness of each layer differs. Azure Databricks also reads OneLake data read-only through catalog federation, so the two can work together.

Databricks is the stronger fit when the same data also needs streaming, ML and open-format access across engines, which is the lakehouse model. The 2026 benchmark post acknowledges that warehouses can be simpler for SQL-only reporting, so a team doing only SQL dashboards on warehouse-shaped data should evaluate both.

Yes. Power BI connects to Databricks directly with the Databricks connector, which works with SQL warehouses and clusters and can use ODBC or the ADBC driver. Authenticate with a service principal for M2M OAuth or with Microsoft Entra ID SSO. Unity Catalog metric views can be queried from Power BI with MEASURE() in DirectQuery.

Lakehouse Federation lets Databricks query Snowflake data in place as an extension of the lakehouse without migrating it, through Unity Catalog foreign catalogs. Access is governed read-only with query pushdown and table-level access controls, so the data stays where it is while Databricks applies its own governance. This suits teams that keep an existing Snowflake estate and want to join it with Databricks workloads. Writes and the Snowflake workloads themselves remain on Snowflake.

Lakehouse Federation's query federation supports Snowflake, Google BigQuery, Amazon Redshift, Azure Synapse, PostgreSQL and other sources, and catalog federation also supports Snowflake. Databricks describes this as querying and governing external data without migrating it. Access is read-only and goes through Unity Catalog foreign catalogs, so each source keeps its own storage and compute. Choose federation for ad hoc or cross-system access and copy data only where performance or workload isolation demands it.

Yes. Unity Catalog exposes its Iceberg tables through an Iceberg REST catalog endpoint that engines such as Spark, Flink and Trino can use. Managed Iceberg tables accept reads and writes from those engines, while foreign Iceberg tables and Delta tables with Iceberg reads enabled are read-only through that endpoint. Decide which engine owns writes for each table, and Unity Catalog governs the access either way.

Yes. Databricks' open lakehouse post says engines such as DuckDB, Trino and PyIceberg can work on the same tables without a second copy, with Unity Catalog as the shared catalog. The Iceberg REST endpoint lets Apache Spark, Apache Flink, Trino, Snowflake and PyIceberg work against Unity Catalog-registered Iceberg tables, and credential vending gives each engine governed access.

Yes. Serverless SQL warehouses have a very fast starting time, which allows shorter auto-termination, so compute is available on demand and stops quickly when idle. The SQL warehouse docs describe serverless compute as eliminating waits for infrastructure, and warehouses restart automatically when a query, job or dashboard needs them.

Yes for many sources. Lakeflow Connect provides managed connectors for SaaS apps, databases (change data capture for MySQL, PostgreSQL and SQL Server), query-based sources, file sources and streaming sources, with pipelines governed by Unity Catalog and serverless compute by default for query-based connectors. Lakeflow Spark Declarative Pipelines handles transformations in SQL and Python.

Yes. Lakebase is a fully managed Postgres database integrated into the Databricks platform. It supports standard Postgres clients and tools, scales compute automatically, scales to zero when idle, supports instant branching, and integrates natively with Unity Catalog.

Lakebridge is Databricks' free migration toolkit. It supports migrations from more than 10 data warehouses and ETL tools, including Snowflake, Teradata and Oracle, and includes built-in reconciliation tools for data accuracy. Lakehouse Federation lets you query Snowflake data in place through a foreign catalog during the move.

Databricks' Redshift field guide describes Redshift's fixed clusters as coupling storage and compute. Databricks uses a lakehouse architecture that separates storage and compute, stores data in open formats such as Delta or Iceberg, and lets multiple SQL warehouses read the same data without copies.

Yes. Databricks' Redshift field guide recommends avoiding a 'big bang' approach and migrating low-risk workloads first to validate migration patterns before expanding, with the database objects, dependencies and ETL workflows inventoried first.

The Redshift field guide notes that handling spikes on Redshift means resizing the cluster or enabling concurrency scaling, both of which add cost. On Databricks, data can be read by multiple SQL warehouses without copies, and serverless SQL warehouses are instant and elastic, so teams can scale for peaks without over-provisioning.

Lakebridge, Databricks' free migration toolkit, supports migrations from more than 10 data warehouses and ETL tools and includes assessment, SQL conversion and reconciliation steps. Redshift is a Lakehouse Federation source, so you can query Redshift data in place while converted SQL is validated.

Databricks' Redshift field guide quotes an HP data engineering team lead who reported 30-40% cost savings from better performance and reduced data duplication. Databricks presents this as a customer-reported result, so validate it with a pilot on your own workloads.

Databricks' query federation documentation lists Amazon Redshift among the sources that Lakehouse Federation supports. The page is the reference for which external systems Databricks can query through federation while a migration is planned. The same list also includes Snowflake, SQL Server, PostgreSQL and Google BigQuery.

Databricks' UniForm documentation says engines that don't support Hive-style Parquet files, such as Athena or Redshift, need UniForm enabled with REORG, which rewrites underlying data files. For general use, UniForm generates Iceberg metadata asynchronously alongside Delta metadata without rewriting Parquet files, so one copy of the data files serves both Delta and Iceberg readers. REORG also turns off and purges deletion vectors while enabling Iceberg reads.

Yes. Delta Sharing (documented as OpenSharing) is an open protocol that works regardless of the computing platform recipients use, so data can be shared across organizations and platforms without copying it. Recipients can use tools such as Apache Spark, pandas and Power BI.

Yes. Lakeflow Connect provides database mirroring for nine of the most used databases, including SQL Server, Oracle and Teradata, writing to open storage governed by Unity Catalog. Query-based and CDC connectors for MySQL, PostgreSQL and SQL Server are documented in the Lakeflow Connect docs.

No. Databricks SQL Serverless is generally available on AWS, Azure and Google Cloud. The Google Cloud post says it is available in 7 GCP regions and 40+ regions across the three major clouds, and Databricks' AWS docs say serverless SQL warehouses are supported in all AWS regions. Check the regions page for the current list.

Databricks' 2026 benchmark post describes Lakehouse//RT as a serverless SQL warehouse for high-concurrency analytical reads, with latencies as low as 10 ms and up to 12,000 queries per second, as reported by Databricks. The Power BI docs note that Lakehouse//RT requires the ADBC driver. Validate these figures on your own concurrency profile.

Yes. The lakehouse covers SQL analytics alongside AI, streaming, unstructured data and open-format portability on one governed platform, and Databricks' 2026 benchmark post says its production SQL workload mix ran 77% faster in late 2024 than its 2022 baseline, measured on its internal Databricks Performance Index. The post also lists where warehouses still lead, which is useful when choosing.

Unity Catalog metric views define measures and dimensions once, and dashboards, Genie Agents and external BI tools query the same definition, so BI teams don't need a separate aggregate table per dashboard. Predictive Optimization also keeps managed tables compacted and clustered with OPTIMIZE and VACUUM for faster BI queries.

Yes. BigQuery now offers integrated Delta Lake support through its Delta connector, which includes deletion vectors, column mapping and liquid clustering, and BigQuery can also write Delta. Delta is used by over 10,000 companies, including 70% of the Fortune 500, and is a fully open-source Linux Foundation project.

Yes. Databricks states customers can access the same copy of data from either Unity Catalog or BigQuery without duplication, built on the Apache Iceberg REST catalog APIs. Google's catalog federation for BigQuery was in preview, and Google Cloud's Lakehouse federation for Unity Catalog reads was announced in Private Preview.

Use Lakehouse Federation for BigQuery. You create a connection and a foreign catalog that mirrors your BigQuery database in Unity Catalog, then query it without migrating the data. Reads use the BigQuery Storage API, Unity Catalog privileges control access, and materialization is optional and incurs BigQuery compute charges.

Databricks' UniForm documentation notes that some Iceberg clients, such as BigQuery, require a path to versioned metadata files to register an external Iceberg table. The docs describe reading UniForm tables by pointing the client at the Iceberg metadata JSON path. Because UniForm generates Iceberg metadata without rewriting Parquet data files, the same data files serve Delta and Iceberg readers.

Yes. Unity Catalog governs external data through foreign catalogs, with query federation for sources such as BigQuery, Snowflake, Redshift, PostgreSQL, SQL Server and Databricks, and catalog federation for sources such as AWS Glue, Hive metastores, Snowflake, Palantir Foundry and Workday Data Connect. Federated access is read-only and governed by Unity Catalog privileges.

Databricks' Delta Lake documentation describes Delta Lake as open source software that adds a transaction log for ACID transactions to Parquet files. It states Delta is developed for tight integration with Structured Streaming, so teams can use a single copy of data for both batch and streaming operations. Delta Lake is also fully compatible with Apache Spark APIs.

Databricks' documentation describes Apache Spark Declarative Pipelines as a declarative framework for building batch and streaming data pipelines in SQL and Python. Lakeflow pipelines are built on that framework, extend it, run on the performance-optimized Databricks Runtime, and are interoperable with Apache Spark Declarative Pipelines.

Databricks' AI functions documentation describes ai_query as the general-purpose function that takes a prompt and lets you choose any supported Foundation Model API. The docs also list task-specific AI functions for routine work such as entity extraction, translation and classification. Databricks recommends the task-specific functions for getting started on routine transformations such as entity extraction, translation and classification, because they need no customization.

Databricks' Agent Bricks documentation describes Agent Bricks as the Databricks agent developer platform for building and deploying agents. Agent Runtime hosts agents built with any framework or harness, and AgentKit adds durable execution so runs survive restarts and crashes. Agents run the code they write in an isolated environment.

Databricks' predictive optimization documentation says the feature automatically runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables, covering both Delta Lake and Iceberg. The docs describe this as eliminating manual maintenance and the time spent tracking performance issues. Databricks notes the feature is enabled by default for accounts created on or after November 11, 2024.

Databricks reports that in one customer example, selective queries ran 20x faster, large table scans improved by an average of 68%, and Predictive Optimization reduced storage costs by 2x on that customer's Unity Catalog tables. These are customer examples from Databricks' post, and your gains will depend on your tables.

Yes. Serverless SQL warehouses run on Databricks-managed infrastructure and provide instant, elastic compute that avoids over-provisioning during usage spikes. They are enabled by default in most accounts and reduce idle time, lowering total cost of ownership.

Databricks' Synapse migration post explains that Synapse customers often stitch together Dedicated SQL, Serverless SQL, Spark pools and Azure Data Factory, which duplicates governance and adds operational overhead. Unity Catalog gives consistent governance across data, notebooks and AI/ML assets on one lakehouse. Databricks cites customers such as Casey's, which cut data delivery times from eight hours to four, and Italgas, which reported a 73% reduction in workload costs.

Lakehouse Federation supports Azure Synapse (SQL Data Warehouse) as a read-only, pushdown-enabled source, so you can query Synapse data in place while you assess it. Lakebridge's Synapse profiling capability is described in Databricks' migration post, which was not fetched here, so it is not stated as doc-backed.

Databricks' query federation documentation lists Azure Synapse (SQL Data Warehouse) among the sources that Lakehouse Federation supports. The page is the reference for which external systems Databricks can query through federation while a Synapse migration is planned. The same source list includes Google BigQuery and Microsoft SQL Server.

Yes. Databricks announced that Mirrored Azure Databricks Catalog is generally available, which lets Microsoft Fabric access tables governed in Unity Catalog, with interoperability through Unity Catalog Open APIs.

Yes, read-only. OneLake catalog federation lets Unity Catalog queries run directly against OneLake storage for Fabric Lakehouse and Warehouse items without copying the data. It needs a Databricks Access Connector or service principal, the Fabric tenant settings, Databricks Runtime 18.0 or above, and SQL warehouses 2025.40 or above.

Azure Databricks integrates with OneLake today through read-only federation, which queries Fabric data without copying it. Storing managed Delta tables natively in OneLake is a Databricks announcement that I did not verify in the fetched pages, so check the Azure Databricks release notes for its current availability.

Azure Databricks has been a first-party Azure service since 2017, deeply integrated with Microsoft services. It integrates natively with Power BI, Excel, Teams and Azure OpenAI, and Lakehouse Federation supports Azure Synapse and OneLake (Fabric) sources.

Yes. Each workspace includes 100 free DBUs per day for Lakeflow Connect, which Databricks estimates covers roughly 100 million records per workspace per day at no charge before standard pricing applies. The free tier includes Unity Catalog governance.

Yes. Azure Databricks Lakebase is now generally available as the database for AI agents, available in 14 Azure regions, alongside Lakeflow Connect and Genie updates in the same FabCon 2026 release. The Lakebase docs describe it as a fully managed Postgres database.

Databricks' Iceberg documentation says Iceberg tables created in Unity Catalog are managed Iceberg tables, which can be created with Databricks Runtime or Databricks SQL. External engines that support the Iceberg REST Catalog API, such as Apache Spark, Flink, Trino or Kafka, can access them. Unity Catalog also manages lifecycle tasks such as snapshot expiration and file compaction.

Delta Sharing (documented as OpenSharing) lets you share tabular data with recipients outside Databricks: shared Delta or managed Iceberg tables can be read by Spark, pandas and Power BI. Recipients can also read Iceberg-enabled shared tables through the Iceberg REST Catalog API, which the UniForm docs label Public Preview.

Unity Catalog's governance scope goes beyond tables. It is the unified governance layer for data, models, agents and apps, and it applies attributes, tags and automated classification to enforce granular access at scale.

The Photon docs state up to 5x better price/performance for data and analytics workloads, measured against other cloud data warehouses on TPC-DS, and note that Photon helps most with longer-running queries. This is a Databricks-published figure; the page does not publish the full configuration.

No. Databricks SQL works on open table formats: Delta Lake is open source with a documented open transaction-log protocol, and managed Iceberg tables are readable and writable by external engines through the Iceberg REST endpoint. Your data stays in your cloud storage, and credential vending gives other engines governed access to it.

Yes. Databricks states that Lakebridge Foundations is free for Databricks customers and partners. Lakebridge covers profiling, assessment, SQL conversion, validation and reconciliation for migrations from more than 10 data warehouses and ETL tools. Databricks says it can automate up to 80% of migration tasks and speed implementation by up to 2x; these are Databricks figures without published methodology.

Yes. Delta Sharing (OpenSharing) works regardless of the computing platform recipients use, so recipients outside Snowflake, including Spark, pandas and Power BI users, can read shared data. Cross-cloud and cross-region sharing is supported, with possible egress fees from the cloud vendor.

Databricks' published TPC-DS work reports that the Barcelona Supercomputing Center found Databricks SQL 2.7x faster and 12x better in price-performance than a similarly sized Snowflake setup. These are Databricks-attributed results from a derived benchmark, not the audited TPC result. The Photon docs also state up to 5x better price/performance on TPC-DS, as a Databricks figure.

Yes. Mirrored Azure Databricks Catalog is generally available and lets Microsoft Fabric access tables governed in Unity Catalog through Unity Catalog Open APIs. Credential vending for those open APIs gives each engine short-lived storage access under governed privileges.

Yes. UniForm lets Iceberg clients query Delta tables by generating Iceberg metadata over the same Parquet data files, so a single copy of the data files supports both Delta and Iceberg readers. Metadata generation runs asynchronously, and the table must be registered in Unity Catalog with column mapping enabled.

Lakeflow Jobs, Databricks' orchestration service, goes beyond linear schedules. Per the docs, tasks can depend on other tasks, jobs support custom control flow such as if/else branching and for-each loops in a visual authoring UI, and triggers start runs based on specific conditions or events. Know the documented limits before designing very large jobs: a job can contain up to 1,000 tasks, and a workspace is limited to 2,000 concurrent task runs. Check your specific orchestration pattern against the docs before deciding.

Not on serverless SQL warehouses, which Databricks describes as instant and elastic compute that eliminates waiting for infrastructure resources. The docs give no fixed startup time, so measure time to first query on your own warehouse. A stopped warehouse also restarts automatically when a query runs against it or a JDBC or ODBC connection is established. The long waits people describe are typical of classic cluster setups, and serverless is the option Databricks positions for removing that wait.

Treat it as ClickHouse's own claim, not a general verdict. One benchmark cannot represent every SQL workload. Databricks SQL runs on Photon, a native C++ vectorized engine that Databricks documents as the default engine for all SQL warehouses. Reproduce any join benchmark with your own data volumes, join shapes, concurrency and freshness requirements before drawing conclusions from it.

That story describes one team's Spark setup. For incremental ingestion, Databricks says Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage, and that it scales to support near real-time ingestion of millions of files per hour. Freshness depends on pipeline design, so measure end-to-end latency on your own data, and check whether your serving requirement needs a different tool.

For very small data, a single machine can be enough, and Databricks supports that case: single-node compute is intended for jobs that use small amounts of data and cannot be converted to multi-node compute. Serverless compute minimizes idle time and reduces the need to manage compute resources. The Databricks docs do not mention DuckDB, so benchmark your own query mix, including concurrency, before deciding.

Not necessarily. Serverless compute reduces the need to manage compute resources and minimizes idle time, which addresses much of the cluster-overhead argument. Attribute cost per workload with the system.billing.usage system table, noting that usage can take up to 24 hours to appear. Keep ML and streaming where they run best, and base any move on measured cost per job rather than a generic cluster assumption.

It can be, depending on your skills and data. Databricks on Google Cloud documents a BigQuery connection through Unity Catalog, so BigQuery data can be queried and managed from Databricks. Setup needs a service-account JSON key, which Databricks recommends storing as a secret, and compute must use Databricks Runtime 16.1 or above, with SQL warehouses that are pro or serverless. It also requires metastore admin or the CREATE CONNECTION privilege. Check those requirements against your team before choosing one platform over the other.

Compute is billed on DBUs plus cloud infrastructure, and the best way to decide is to measure your own BI workload end to end. Serverless SQL warehouses reduce idle time, which lowers total cost of ownership, and tags let you monitor usage and cost. Compare the full cost of your BI workload, including cloud charges, before deciding.

The 70-80% share is not in the Databricks docs, so treat it as unverified. Databricks does give direct controls for idle capacity: auto-termination is set per compute by an inactivity period, after which the compute resource terminates. Serverless compute minimizes idle time and reduces the need to manage compute resources, per the serverless compute docs.

Much of the manual tuning is automated. Predictive optimization runs OPTIMIZE, VACUUM and ANALYZE automatically on Unity Catalog managed tables, and its compute is billed on a serverless jobs SKU. Photon is the default engine for all SQL warehouses, per the Photon docs. Teams can still tune by hand, so measure the net effect on your workloads.

Managed DR replicates Unity Catalog metadata, managed table data and workspace assets on a continuous schedule, for the in-scope catalogs and assets of your failover group. The docs give no numeric RPO or RTO and no replication-time figure. The page names catalogs, workspace assets and managed table data, and does not name materialized views, streaming tables or ML models, so confirm those objects for your workspace. Test with a planned failover from the account console before relying on it.

Liquid clustering tables use Delta writer version 7 and reader version 3, and Databricks states that Delta clients that do not support these protocols cannot read those tables. Writes require a client that supports all the table features liquid clustering uses. This is a documented protocol requirement, not a proprietary format. Choose clients that support it, or leave a table unclustered where broad interoperability matters.

Databricks handles semi-structured data natively with the VARIANT type, which stores OBJECT, ARRAY and scalar values. Know the limits: STRUCT and MAP values must be wrapped with to_variant_object to be stored, Iceberg v2 tables do not support VARIANT columns, and the type applies from Databricks Runtime 15.3. Test your own JSON payloads and query patterns against your Snowflake workload before judging fit.

A single test cannot settle load speed. Compare loads on the same cloud, region and file layout before drawing a conclusion. On Databricks, Auto Loader incrementally processes new files as they arrive in cloud storage, and Databricks says it scales to support near real-time ingestion of millions of files per hour. Measure end-to-end load time on your own file layout.

A 16-query subset is not a full benchmark. Databricks' audited 100 TB TPC-DS result covers the full TPC-DS suite, and it was formally reviewed by the TPC council. For a decision, run your full workload on both platforms with the same data and configuration.

Often it adds a copy and an extra hop, and open table formats can reduce that. UniForm generates Iceberg metadata without rewriting Parquet data files, so one copy of the data can serve Delta and Iceberg readers. The limit is that Iceberg clients are read-only on UniForm tables. Benchmark your own pipeline before choosing an architecture, since speedup figures from any vendor cover one workload.

Serverless compute addresses much of the infrastructure work behind those exits. Databricks says serverless speeds up start-up and scaling, minimizes idle time, and reduces the need to manage compute resources. Stories about moves to Airflow, EMR or Looker describe each team's platform and choices at the time. Assess your own operating burden against the current serverless options.

Case studies from competitors describe one customer's choice and are not neutral comparisons; their savings figures are the vendor's own reported results and can combine operating-model changes with platform changes. To measure your own economics, use the system.billing.usage table, which can take up to 24 hours to show workload usage, and run a proof of concept on your pipelines.

Serverless SQL warehouses reduce operational overhead: they run on Databricks-managed infrastructure with minimal management overhead, and they reduce idle time, which lowers total cost of ownership. Price a small workload on the SKU it actually uses and compare the cost with your current setup.

Databricks supports single-node work directly. Single-node compute is intended for jobs that use small amounts of data, and it cannot be converted to a multi-node compute resource, so choose it when the workload stays small. For investigation work that does not need a distributed cluster, single-node compute fits, and serverless compute avoids cluster sizing.

Measure batch job cost on the SKU each job actually runs on rather than assuming a premium. On portability, UniForm lets Iceberg clients read Delta tables without rewriting Parquet files, so the table data is not held in a Databricks-only format, though Iceberg clients are read-only. The docs I checked do not address Spark code portability, so test that separately for your code.

The disk cache is documented. Databricks stores a copy of remote Parquet files on the node's local storage, enables it automatically on worker instance types that support it, and spark.databricks.io.cache.enabled turns it off without dropping data already stored locally. For fair comparisons, state the cache setting and run cold and warm tests.

The lakehouse is a defined architecture: Databricks describes an open lakehouse as one where every layer, storage, table format, engine, catalog and ML/AI tools, is built on open standards. Delta Lake and Iceberg add reliable table management on open files, Apache Spark runs batch, streaming, SQL and ML, and Unity Catalog governs them.

Lakebase itself is generally available. Databricks' release notes say Lakebase is now generally available, dated January 22, 2026. Some related features are still Beta, including LTAP Direct Writes, which speeds up how synced tables load into Lakebase, and Lakebase Insights. The release notes do not define LTAP, so measure write latency and point-read performance on your own workload, and check the release notes for each feature's status.

Cost outcomes depend heavily on configuration, so an untuned migration can cost more than expected. Databricks recommends configuring auto-termination for all interactive compute resources, and documents that non-interactive workloads cost significantly less on job compute than on all-purpose compute. Serverless SQL warehouses scale down earlier than non-serverless warehouses, which lowers cost when they are idle. Compare cost per completed workload rather than DBU rate alone, since serverless and classic compute have different cost structures.

A standalone open source Iceberg REST catalog is a valid option when engine neutrality and catalog control are the top priorities. Unity Catalog also implements the Iceberg REST catalog specification and exposes it at /api/2.1/unity-catalog/iceberg-rest, so engines such as Spark, Flink, and Trino connect to it in the same way. Compare the governance and management features you need, and test your engines against either option before choosing.

Databricks' strongest published evidence is its November 2021 100 TB TPC-DS result, which was formally audited and reviewed by the TPC council. The same post cites a Barcelona Supercomputing Center comparison that found Databricks SQL 2.7x faster than a similarly sized Snowflake setup and 12x better in price-performance, a derived benchmark. The Photon docs also support raw-file scans: Photon supports Parquet, Delta, CSV and JSON scans. Test your own workload on the same data to settle it.

Teams that stay on Databricks usually value the one-platform model: Unity Catalog is one catalog for data and AI assets, gives access to data across platforms without moving it through open APIs and federation, and keeps governance consistent. The lakehouse also avoids separate systems for ML and BI. Weigh these against the reasons you moved, using your own requirements.

Market-share trends are outside what official Databricks documentation can settle, so this answer does not claim one. What the docs establish is what Databricks offers: a lakehouse that avoids isolated systems for ML and BI workloads, built on open storage formats, with Delta Lake tables and Unity Catalog governance for data and AI assets. Both platforms serve analytics well, and the right choice depends on workload, team skills, and cloud. A fair way to decide is to test your highest-value workloads on each and compare cost, latency, governance, and operational effort.