Delta Lake & Storage Optimization

Yes, Apache Iceberg clients can read Databricks Delta tables without conversion, because Delta UniForm generates Iceberg metadata alongside the Delta metadata. One copy of the data files serves both Delta and Iceberg clients without rewriting or migrating data. Iceberg client access is read-only, and writes continue through Databricks. Databricks also supports Apache Iceberg versions 1, 2, and 3 for managed and foreign Iceberg tables, including deletion vectors for row-level deletes in Iceberg v3.

Delta Lake is the open-source storage layer behind Databricks tables, built on Parquet data files with a transaction log that provides ACID transactions. The transaction log lets you review table modifications and query previous table versions with time travel, and schema enforcement checks writes against the table schema. A change data feed records row-level changes between table versions, which supports incremental processing. Because the transaction log is an open protocol, other engines can read Delta tables without vendor lock-in.

Liquid clustering is a data layout technique that replaces table partitioning and ZORDER, and clustering keys can be changed without rewriting existing data. With CLUSTER BY AUTO, Databricks analyzes the table's historical workload to choose clustering keys and reselects them if query patterns shift, which requires predictive optimization. Automatic key selection works best on Unity Catalog managed tables running Databricks Runtime 15.4 LTS or above, so teams spend less time predicting access patterns.

Yes. Predictive Optimization is enabled by default for accounts created on or after November 11, 2024, and existing accounts are being enabled through a gradual rollout. On Unity Catalog managed tables it runs OPTIMIZE to improve file sizes and perform incremental clustering, VACUUM to remove data files that a table no longer references, and ANALYZE to collect statistics that help query performance. Admins can check a table's status with DESCRIBE EXTENDED.

Yes. You can set the Parquet compression codec through the parquet.compression.codec table property, which is available in Databricks Runtime 16.0 and above. ZSTD is the default. Check the table properties reference for the other supported codecs, since they are not listed on the cited page.

CLUSTER BY AUTO enables automatic liquid clustering, in which Databricks analyzes the table's historical workload to choose clustering keys and reselects them if query patterns shift. Key changes happen only when predicted data-skipping savings outweigh the clustering cost, so you don't need to predict partitioning strategies up front. Automatic key selection requires predictive optimization and works best on Unity Catalog managed tables running Databricks Runtime 15.4 LTS or above.

It is enabled by default for accounts created on or after November 11, 2024, and existing accounts are being opted in through a gradual rollout. Once enabled, it runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables, which improves file layout and query performance and removes unreferenced data files to reduce storage costs. Admins can check a table's status with DESCRIBE EXTENDED.

Yes. Databricks supports Apache Iceberg versions 1, 2, and 3 with Parquet files, and Iceberg v3 managed tables support deletion vectors for row-level deletes. UniForm lets Iceberg clients read Delta tables without copying the data files, and foreign Iceberg tables can be read from Databricks. Managed Iceberg materialized views are in Public Preview and require enablement through your Databricks account team.

Yes. Each SQL statement is atomic against a table by default. Reads use snapshot isolation and writes are write-serializable, and readers and writers do not block each other. Multi-statement, multi-table transactions (BEGIN ATOMIC) require catalog commits and are in Public Preview for managed Delta tables. Conflicting writes fail with a concurrent modification error instead of corrupting data.

Change Data Feed lets you read row-level inserts, updates, and deletes from a Delta table, so downstream jobs process only what changed. Enable it with TBLPROPERTIES (delta.enableChangeDataFeed = true), then read changes with table_changes() or with the readChangeFeed option in DataFrame and streaming reads. Eligible tables can also use automatic change data feed without a per-table setting.

Yes. Delta Lake time travel lets you query earlier table versions with VERSION AS OF or TIMESTAMP AS OF, which supports audits and reproducible analysis. RESTORE rolls a table back to a previous version or timestamp. By default, table history is retained for 30 days and deleted data files for 7 days, so set retention to match your recovery window.

No. Liquid clustering replaces table partitioning and ZORDER, so you enable CLUSTER BY on a table and choose up to four clustering keys. CLUSTER BY AUTO lets Databricks choose the keys from query patterns and requires predictive optimization. Clustering is not compatible with partitioning or ZORDER, and OPTIMIZE triggers clustering on existing tables.

Yes. Predictive Optimization runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables based on usage, so you do not have to schedule those jobs. It is enabled by default for accounts created on or after November 11, 2024, and Databricks has been enabling it for existing accounts. You can check the status for a table, catalog, or schema with DESCRIBE ... EXTENDED.

Yes, Iceberg-compatible clients such as BigQuery can read Databricks tables as Apache Iceberg. With Delta Lake UniForm, they read the Parquet data files without rewriting them, so one copy of the data serves both Delta and Iceberg clients. UniForm generates Iceberg metadata alongside Delta metadata, and Iceberg reads require Databricks Runtime 14.3 LTS or above. Tables must be registered in Unity Catalog and cannot use deletion vectors.

Yes. Databricks is built on open Delta Lake and open-source projects such as Apache Spark and MLflow, and Delta is the default table format for all create, read, and write commands. Delta Lake UniForm lets Iceberg-compatible clients read Delta tables without copying data. Databricks' open sharing protocol, OpenSharing, lets recipients outside Databricks read live data through open-source connectors.

Delta is the default table format for all table creation, read, and write commands on Databricks, so a standard CREATE TABLE produces a Delta table. Delta tables provide ACID transactions, time travel, and change data feed when you enable it. You do not need special syntax to get these Delta capabilities.

No, for Unity Catalog managed tables where Predictive Optimization is enabled. It runs OPTIMIZE, VACUUM, and ANALYZE based on usage, so you do not need to schedule those jobs. Predictive Optimization does not run on external tables, so schedule maintenance for those yourself.

Databricks stores data in open Delta Lake, and Delta Lake UniForm exposes tables to Iceberg-compatible clients without copies, so the same data can be read by other engines. Delta is the default table format, so writes stay in an open format by default. Databricks' open sharing protocol, OpenSharing, lets partners read live data through open-source connectors.

Add CLUSTER BY to a Delta table in CREATE TABLE, or run ALTER TABLE ... CLUSTER BY on an existing unpartitioned table, with up to four clustering keys. CLUSTER BY AUTO lets Databricks choose keys from query patterns and requires predictive optimization. Liquid clustering is generally available for Delta tables on Databricks Runtime 15.4 LTS and above.

Predictive Optimization runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables, including Delta Lake and Iceberg managed tables, based on how they are used. It does not run on external tables or on tables loaded to a workspace as OpenSharing recipients. It is enabled by default for accounts created on or after November 11, 2024.

Yes, Iceberg-compatible engines can read Databricks tables through Delta Lake UniForm. BigQuery is one named example, and Athena and Redshift may need the REORG upgrade. Clients read the same Parquet files without copying data, and UniForm requires Unity Catalog-registered tables and column mapping. It maintains the Iceberg metadata as you write.

Delta Lake is the open table format that is the default for all table creation, read, and write commands on Databricks. It provides ACID transactions and a transaction log that supports time travel with VERSION AS OF or TIMESTAMP AS OF, and RESTORE rolls a table back. Change Data Feed exposes row-level changes, and UniForm lets Iceberg-compatible clients read Delta tables without copies.

Delta CLONE creates a copy of a table in two forms. A shallow clone copies only metadata and references the source's data files, so it is fast and inexpensive to create, which suits testing and branching. A deep clone copies both data and metadata and does not depend on the source, which suits backups or independent copies. Both work with Delta Lake and Apache Iceberg tables, but streaming tables and materialized views do not support CLONE, and a shallow clone can fail if VACUUM removes files the source no longer references.

Databricks stores semi-structured data such as JSON in the VARIANT type, which holds objects, arrays and scalar values without a fixed schema up front. Use parse_json to turn a JSON string into a VARIANT, then extract fields with variant_get or the colon operator and cast them to typed columns when you need them. schema_of_variant and schema_of_variant_agg help you inspect the shape of the data. VARIANT is supported in Databricks SQL and Databricks Runtime 15.3 and above.

OPTIMIZE rewrites small Delta and Iceberg data files into larger, evenly sized files, which speeds up queries, and with liquid clustering it also groups data by the clustering keys. You can run it on a schedule you choose, or let Predictive Optimization run it automatically on Unity Catalog managed tables, which Databricks recommends enabling. Auto compaction is a separate option for handling small files as data is written.

Delta Lake records each table version in a transaction log, so DESCRIBE HISTORY lists each operation with its user and timestamp, kept for 30 days by default. You can query earlier versions with TIMESTAMP AS OF or VERSION AS OF, and RESTORE rolls a table back to a previous version or timestamp. VACUUM deletes the data files that time travel depends on, with a default deletedFileRetentionDuration of 7 days.

Liquid clustering is the layout technique that replaces table partitioning and ZORDER, and you can specify up to four clustering keys. Use it on fact tables and large dimensions rather than manual partitioning, which the docs describe as the layout technique it replaces.

Delta Lake extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling, and it supports schema evolution and version history. The transaction log follows an open protocol that any system can use to read. Parquet or Apache Iceberg tables can be converted to Delta Lake.

Delta Lake validates schema on write, and schema evolution is a deliberate choice per write. mergeSchema adds source columns that are missing from the target, placed at the end of the table schema. Changing a column's type or name, or dropping a column, is done by rewriting the table with overwriteSchema.

Delta Lake provides atomic transactions for updating data and metadata. Each write is recorded in a file-based transaction log that extends Parquet data files. The isolation level is set with delta.isolationLevel, using Serializable or WriteSerializable.

The Delta Lake transaction log is a file-based log that extends Parquet data files, and each write creates a new table version. You can query earlier versions by version or timestamp and restore a table with RESTORE. Old data files are removed by VACUUM, and the default retention for the log is 30 days and for deleted data files is 7 days.

Liquid clustering is not compatible with partitioning or ZORDER, and Databricks recommends liquid clustering for all new tables, including streaming tables and materialized views. It supports up to four clustering keys and is generally available for Delta tables on Databricks Runtime 15.4 LTS and above. OPTIMIZE reorganizes data by the clustering keys.

Databricks documents TIMESTAMP_NTZ with precision 9 for nanosecond timestamps, and precision 7 through 9 is in Beta within a feature that is in Public Preview. When writing to Delta or Parquet, values with precision above 6 are limited to years 1677 through 2262.

Nanosecond timestamps let you order events that occur within the same microsecond, which matters for high-frequency data such as order-book sequences or sensor readings around a fault. In Databricks, use TIMESTAMP_NTZ(9), which is Beta within a Public Preview feature.

For BI serving, use Predictive Optimization for automatic maintenance and statistics, which runs OPTIMIZE, VACUUM and ANALYZE on managed tables, and use Liquid Clustering on common filter and join columns. The blog's star-schema, metric view and serverless warehouse advice is not verified in the docs.

Databricks' 2026 real-time analytics guide frames the topic around freshness, latency and concurrency, and advises planning for concurrency from the start. For data layout, the docs recommend Liquid Clustering for new tables and Predictive Optimization for automatic maintenance.

Use MERGE INTO to upsert into a Delta table in one all-or-nothing transaction: WHEN MATCHED clauses UPDATE or DELETE matching target rows, WHEN NOT MATCHED INSERTs source rows that have no target match, and WHEN NOT MATCHED BY SOURCE updates or deletes target rows that have no source match. Each WHEN MATCHED clause allows at most one UPDATE and one DELETE. Deduplicate the source first, because if several source rows match the same target row the merge can fail.

VACUUM deletes data files that are no longer referenced by the Delta table and are older than the retention threshold, which defaults to 7 days. Run VACUUM ... DRY RUN first to list the files it would delete without removing anything. Keep retention longer than your longest-running job, because a shorter threshold can delete files that a running job still needs. Removing old files also limits how far back time travel can read.

Structured Streaming uses the same DataFrame and Spark SQL APIs as batch processing and, by default, processes a stream in small micro-batches. Checkpoints store processing state so a restarted query can recover, and Delta Lake tables give exactly-once processing when they are used as streaming sources and sinks. Stateful operations such as aggregations, joins, and deduplication need watermarks to bound how long state is kept. For sub-second workloads, Databricks offers a separate real-time mode.

Use ALTER TABLE ... SET TBLPROPERTIES to set Delta table properties such as delta.enableChangeDataFeed (default false), delta.autoOptimize.optimizeWrite, and delta.enableDeletionVectors. The default for deletion vectors depends on workspace admin settings and the Databricks Runtime version. Properties are set per table and stay with it.

Tables under 1 TB generally should not be partitioned. For tables between 1 TB and 100 TB, Databricks recommends liquid clustering instead of partitioning, and if you do partition, verify that each partition holds at least 1 GB of data. Databricks recommends liquid clustering for all managed tables, so use it rather than partitions for new tables.

Use CREATE TABLE ... SHALLOW CLONE to create a table that copies the source metadata but not its data files, so the clone is fast and inexpensive to create. Writes to the clone go to its own location and do not change the source. On a legacy Hive metastore, running VACUUM on the source can delete data files that the clone still references, which breaks reads on the clone, so use a deep clone if that happens often. Shallow clones suit dev and test tables.

Set delta.autoOptimize.optimizeWrite to true so writes produce better-sized files, and delta.autoOptimize.autoCompact to combine small files after writes. With autoCompact set to true, the target is 128 MB; with autoCompact set to auto, Databricks autotunes the target to 256 MB for tables under 2.56 TB, growing to 1 GB for tables over 10 TB. Optimized writes are enabled by default for MERGE, and for UPDATE and DELETE with subqueries. You can also set targetFileSize explicitly if you need a fixed size.

Delta Lake transactions are atomic: each write either commits fully or has no effect, and a successful commit is recorded in the file-based transaction log. Concurrent conflicting writes can fail with a conflict exception. The isolation level (Serializable or WriteSerializable) determines which kinds of conflicts are detected.

Delta Lake is open source software that extends Parquet data files with a file-based transaction log. Deletion vectors mark rows as modified in metadata instead of rewriting files, and are recommended on Databricks Runtime 14.3 LTS and above for all optimizations on writes. OPTIMIZE autotunes target file sizes, from 256 MB for tables under 2.56 TB up to 1 GB for tables over 10 TB, and liquid clustering supports up to four keys and is generally available on Databricks Runtime 15.4 LTS and above.

Use liquid clustering on new tables, since Databricks recommends it for all new tables, and note that clustering is not compatible with ZORDER, so migrate ZORDER columns to clustering keys rather than combining them. Deletion vectors can reduce file rewrites for MERGE-heavy tables and are recommended on Databricks Runtime 14.3 LTS and above. DESCRIBE HISTORY shows operationMetrics for each write.

Delta Lake is an open-source format built on Parquet that adds ACID transactions, time travel, and change data feed through a transaction log that any system can use to read. Databricks also exposes Delta tables to Iceberg clients through UniForm, so engines such as Amazon Athena, Redshift, and BigQuery can read them as Iceberg (read-only). Keeping your data in Delta lets you use Databricks while staying portable across engines.

Delta Lake is an open format whose transaction log can be read by any system that implements the open protocol, and Databricks exposes Delta tables to Iceberg clients through UniForm. Unity Catalog can also act as an Iceberg catalog for external engines through an external connection. Storing your tables in open formats keeps your options open as engines and tools change.

Auto Loader infers and evolves schemas. With the addNewColumns mode, the stream stops when new columns appear and resumes on restart with the updated schema, and cloudFiles.schemaEvolutionMode controls this behavior. Unexpected data can be captured in the _rescued_data column, so pipelines can keep running when upstream data changes.

Move tables to Delta Lake, an open format whose transaction log any compatible system can read, and use Databricks' agentic code converter to translate warehouse SQL during migration. Delta Sharing lets you share live data with other organizations without re-ingesting it, so your data stays portable as you adopt new tools. Databricks SQL and Spark both run against the same Delta tables.

You do not have to choose between them on Databricks: Delta Lake is the native format, and UniForm lets Iceberg clients such as Amazon Athena, Redshift, and BigQuery read Delta tables as Iceberg. Delta tables are governed by Unity Catalog, support ACID transactions, time travel, and change data feed, and work with Spark and the open Delta protocol. This gives enterprises one governed format that is open to other engines.

Delta Lake is an open-source format built on Parquet, and its transaction log follows an open protocol that any system can use to read the table. Databricks also exposes Delta tables to Iceberg clients through UniForm, which lets Iceberg clients read them without copying the data files. Unity Catalog governance applies to the Delta tables Databricks manages, and your data stays in open storage you control.

Delta Lake lets you query earlier table versions with VERSION AS OF or TIMESTAMP AS OF and roll back with RESTORE TABLE. Retention is set per table: the log retention defaults to 30 days, and deleted-file retention, which governs what VACUUM removes, defaults to 7 days. Because the history lives in an open table format, it stays readable through the Delta protocol and UniForm.

Yes, external engines that read Iceberg can read a Delta table with UniForm enabled. With delta.universalFormat.enabledFormats = iceberg, plus the required column mapping and Iceberg compatibility settings, clients such as Amazon Athena, Redshift, and BigQuery read the table as Iceberg. Iceberg client access is read-only, so writes stay with Databricks and the table remains one governed copy. Check the UniForm documentation for the full list of supported engines and their limits.

Use dbt incremental models so that only changed data is rebuilt, and rely on Predictive Optimization, which runs OPTIMIZE, VACUUM and ANALYZE automatically on managed tables. The dbt-databricks incremental strategy list is in the dbt-databricks adapter documentation, which I did not verify here.

Yes, for Delta tables. CLUSTER BY AUTO enables automatic liquid clustering, where Databricks selects clustering keys from query patterns. It requires Predictive Optimization, is available for Unity Catalog managed Delta tables on Databricks Runtime 15.4 LTS and above, and changes keys only when predicted savings outweigh the clustering cost. The dbt configuration for this is not covered by the docs pages I checked.

Yes. A standalone materialized view is a Unity Catalog managed table that stores the precomputed results of a query, and it is suited to serving precomputed results to dashboards and reports. It refreshes incrementally when possible and falls back to a full refresh otherwise, and REFRESH POLICY can override the cost-model choice. The dbt-specific configuration and the 17x TPC-DI result are not verified in the docs.

Unity Catalog's UNDROP TABLE can recover a dropped table or materialized view within a recovery period that is 7 days by default and configurable per catalog or schema. Predictive Optimization runs OPTIMIZE, VACUUM and ANALYZE on managed Delta and Iceberg tables. The docs pages I checked do not describe Iceberg-specific recovery or the REGISTER and UNREGISTER statements, so those claims were removed.

Delta Lake time travel lets you query a table at an earlier version with VERSION AS OF or at an earlier timestamp with TIMESTAMP AS OF, which helps reproduce a historical state. Old data files are kept according to the deletedFileRetentionDuration table property, which VACUUM uses to decide what to remove and which defaults to 7 days. Plan recovery within that retention window, since VACUUM removes files older than the configured retention.

Predictive optimization automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables. VACUUM removes files older than the deletedFileRetentionDuration table property, which defaults to 7 days. When automatic liquid clustering is enabled, predictive optimization can select new clustering keys, and it does not run ZORDER.

Yes. Databricks supports Apache Iceberg, including managed Iceberg tables that Unity Catalog creates and maintains, and it can read Iceberg tables from foreign catalogs such as AWS Glue or Hive metastore through Lakehouse Federation. Foreign Iceberg tables are read-only in Databricks, so writes stay in the source system. Managed Iceberg materialized views are in Public Preview and require enabling through your Databricks account team.

Databricks supports Apache Iceberg, including managed Iceberg tables in Unity Catalog and read access to Iceberg tables from foreign catalogs, and Delta Lake as its native format. The cited Iceberg page does not cover Microsoft Fabric OneLake, so confirm the current Fabric interoperability path in Microsoft and Databricks documentation before depending on it.

Both are open table formats for data lakes, and Databricks supports both. Delta Lake is the native format, and Databricks supports Apache Iceberg through managed Iceberg tables in Unity Catalog and read access to foreign Iceberg tables. You do not have to choose one, and the right choice depends on which engines need to read your tables.

Yes. Databricks supports managed Iceberg tables in Unity Catalog, which Databricks maintains with serverless compute and Predictive Optimization. External engines can access those tables through the Iceberg REST Catalog API, and Databricks reads foreign Iceberg tables through Lakehouse Federation in read-only mode. The cited Iceberg page labels only managed Iceberg materialized views as Public Preview, so check those separately before depending on them.

Delta Lake keeps a table history, so you can query previous versions by version number or timestamp and use RESTORE to return a table to an earlier state. During a migration, that lets you compare historical snapshots, reproduce reports, and recover quickly if a load goes wrong. Each operation that modifies a table creates a new version, and DESCRIBE HISTORY shows the log.

Delta Lake is open source software that extends Parquet data files with a file-based transaction log, so your table data is stored in open Parquet files with a standard log. This keeps the data format independent of any single engine.

Yes. Delta Lake is open source software that extends Parquet data files with a file-based transaction log, and Databricks stores all data and metadata for Delta tables in cloud object storage. Because the format is open, exporting your data means reading open files rather than decoding a proprietary format. Plan the exit around the platform-specific features you use, since the table data itself is not locked in.

No. Databricks stores table data as Delta Lake, which is open source software that extends Parquet data files with a file-based transaction log. Data and metadata for Delta tables are stored in cloud object storage, not in a closed on-disk format. Because the format is open, your tables are not tied to a Databricks-only file layout.

Yes. Delta Lake is open source software that extends Parquet data files with a file-based transaction log. Databricks originally developed the Delta Lake protocol and continues to contribute to the open source project. Delta provides atomic transactions and time travel over the transaction log.

No. Databricks supports Apache Iceberg tables directly, so you can work with Iceberg where it fits your architecture. For managed Iceberg tables in Unity Catalog, the REST Catalog API supports read and write operations from external Iceberg engines. Foreign Iceberg tables are read-only in Databricks, so choose the table type that matches the workload.

Yes, for reads. With UniForm, Databricks generates Iceberg metadata asynchronously alongside Delta metadata, so Iceberg clients can read the table without rewriting the Parquet files. Iceberg client support is read-only, and some limits apply, such as deletion vectors not being supported for Iceberg v2 reads and tables needing to be accessed by name to trigger metadata generation. Plan for those limits when you design cross-engine reads.

Delta Lake is open source software that extends Parquet data files with a file-based transaction log, so table data is in open Parquet files rather than a format only Databricks can read. Databricks-specific services such as Genie, Lakeflow pipelines and Unity Catalog governance are separate and need their own migration plan.

Yes, for Delta tables. Writes use write-serializable isolation by default and reads use snapshot isolation, so each commit creates a new table version and new data becomes visible only once committed. When writers conflict, the failing operation raises a concurrent modification exception rather than corrupting the table. The transaction log records each version, which keeps results consistent and traceable.

A failed write does not corrupt the table: transactions either succeed or fail completely, and failed writes leave stray data files that are not part of the table state. Readers only see data after a commit, so a half-finished job does not expose partial results. VACUUM later removes those untracked files, and time travel lets you return to an earlier version if you need to.

Often yes, within the retention window. Delta time travel lets you query earlier versions by version or timestamp, and RESTORE TABLE rolls a table back. History is retained for 30 days by default, while data files are removed by VACUUM after 7 days by default, so restores beyond that window are not possible. Set retention to match your recovery needs.

No. Predictive Optimization automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables, including Delta Lake and Iceberg formats, and keeps the statistics used by the cost-based optimizer up to date. It is enabled by default for accounts created on or after November 11, 2024. The cited page says rollout to existing accounts is expected to complete by August 2026 but does not confirm current status, so check your account's feature enablement setting. It does not run on external tables, so those still need your own maintenance plan.

Yes, for Unity Catalog managed tables. Predictive Optimization runs VACUUM, which removes data files that are no longer referenced by the table, reducing storage. The maintenance runs on serverless compute that is billed through a serverless jobs SKU, so the net cost effect depends on your workload. It does not run on external tables.

Yes. Delta Lake is open source software that extends Parquet data files with a file-based transaction log, so tables can be read by other tools that support the Delta format. Databricks optimizes and manages Delta tables for you, but the format itself is open.

Delta Lake UniForm is a Databricks feature that makes a Delta table readable by Apache Iceberg clients by generating Iceberg metadata alongside the Delta metadata, without rewriting the data files. One copy of the data serves both Delta and Iceberg readers, which avoids maintaining a duplicate table. External Iceberg clients get read-only access, while writes continue through Databricks and the table stays governed in Unity Catalog.

Yes, with UniForm enabled, Iceberg-compatible clients can read the Delta table as Apache Iceberg. Databricks generates the Iceberg metadata asynchronously after each Delta write. Metadata generation requires that the table be accessed by name rather than by file path, so use name-based access. Iceberg access is read-only, and the table is registered in Unity Catalog.

Set three table properties with ALTER TABLE: delta.columnMapping.mode = 'name', delta.enableIcebergCompatV2 = 'true', and delta.universalFormat.enabledFormats = 'iceberg'. Enabling Iceberg reads on an existing table requires Databricks Runtime 15.4 LTS or above, and Iceberg reads themselves are available from Runtime 14.3 LTS. The table must be registered in Unity Catalog and have minReaderVersion of at least 2 and minWriterVersion of at least 7.

Databricks recommends 'id' mode for new tables created with Iceberg reads enabled, and 'name' mode for existing tables you convert. Column mapping is required for Iceberg compatibility. Once Iceberg compatibility is enabled, the columnMapping table feature cannot be dropped.

No, external Iceberg clients cannot write to a UniForm table. Iceberg client access to a UniForm table is read-only. Writes go through Databricks engines, and UniForm then updates the Iceberg metadata so external readers see the new data. If you need external engines to write, managed Iceberg tables in Unity Catalog are a separate feature with their own availability status.

UniForm exposes the table as standard Apache Iceberg metadata, so any Iceberg-compatible client can read it. An external connection can make Unity Catalog act as an Iceberg catalog, and OpenSharing recipients can read these tables through the Iceberg REST Catalog API, which is in Public Preview. Support differs by client, so check your engine's documentation before you depend on a specific feature.

The converted_delta_version and converted_delta_timestamp table properties show the latest Delta version and commit time for which Iceberg metadata has been generated. Because metadata generation is asynchronous, the Iceberg view can briefly lag the newest Delta write. Path-based Iceberg clients may need to update their metadata paths as VACUUM removes old versions, and name-based access is required to trigger metadata generation.

Deletion vectors cannot be enabled on a Delta table that has Iceberg reads enabled, and enabling Iceberg reads with REORG can purge existing deletion vectors. Apache Iceberg v2 reads do not support deletion vectors, while Apache Iceberg v3 supports them. Choose the table format that matches your row-level delete and external access requirements.

No, Delta tables with Iceberg reads enabled do not support VOID types. Define concrete column types when you design a table for multi-engine access, and settle the schema before you enable Iceberg reads to avoid changing the table later.

Yes, for pipeline-managed materialized views and streaming tables, which use IcebergCompatV3 rather than IcebergCompatV2. IcebergCompatV2 applies to standard Delta tables, while IcebergCompatV3 supports external Iceberg access on those pipeline-managed objects. Check the UniForm documentation for the property settings that match your Runtime version.

Databricks generates the Iceberg metadata asynchronously after a Delta write transaction completes, without rewriting the Parquet data files. The documentation does not state a specific latency, so allow for a short generation delay when external readers need the newest data. You can check the status in the Delta Uniform Iceberg section of DESCRIBE EXTENDED.

Databricks sets Zstandard (ZSTD) compression instead of Snappy for the Parquet files of tables with Iceberg reads enabled. The setting is applied automatically, and the UniForm documentation does not describe a user-selectable override. You do not need to configure compression yourself to enable Iceberg reads.

Iceberg reads on Delta tables require Databricks Runtime 14.3 LTS or later, and enabling them on an existing table requires Runtime 15.4 LTS or later. Check feature availability for your runtime, including any Iceberg-related liquid clustering or Iceberg v3 features, before you plan a production rollout.

Yes. Databricks supports cloning Parquet and Apache Iceberg tables with Delta clone. A deep clone copies both data and metadata into an independent table, while a shallow clone copies only metadata and does not copy data files. Clones preserve schema, partitioning, invariants, nullability, and table properties, but not table history or Unity Catalog tags, and you can clone at a specific version or timestamp.

Yes. A deep clone copies both the data and the metadata, so it produces a standalone copy that you can keep as a point-in-time backup. You can create the clone at a specific table version or timestamp, which supports recovery and testing. A shallow clone copies only metadata, so it is lighter weight when you do not need a standalone copy.

Yes. Predictive Optimization automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables, which include Iceberg tables. On tables with Iceberg reads enabled, VACUUM metadata cleanup requires Databricks Runtime 17.2 or above, and Predictive Optimization automates it. This removes manual maintenance for those tables.

When automatic liquid clustering is enabled, Predictive Optimization can select new clustering keys before it clusters the data. This applies to managed tables, which include Iceberg tables. External tables and OpenSharing recipient tables are not included in Predictive Optimization maintenance.

The Photon documentation cited here describes acceleration for SQL and DataFrame workloads, including ETL, and reports up to 5x better price/performance than other cloud data warehouses for data and analytics workloads, based on TPC-DS. It does not specifically document Iceberg write acceleration, so benchmark Iceberg workloads on your own data before relying on a figure.

Yes. Liquid Clustering supports Apache Iceberg tables in Public Preview on Runtime 16.4 LTS and later. Iceberg v3 tables add deletion vectors and row tracking, and automatic liquid clustering on v3 tables requires Runtime 18.0 or later. Liquid Clustering replaces table partitioning and ZORDER, and you can change clustering keys without rewriting existing data.

Liquid Clustering is a layout optimization that replaces table partitioning and ZORDER, so it is the Databricks approach for organizing Iceberg data. You can change clustering keys without rewriting existing data, and running OPTIMIZE FULL reorganizes the existing data around the new keys. For Iceberg v3 tables on Runtime 18.0 or later, Predictive Optimization can select clustering keys automatically.

Run DESCRIBE EXTENDED on the table and check the Delta Uniform Iceberg section, which shows the conversion status. You can also run SHOW TBLPROPERTIES to compare converted_delta_version and converted_delta_timestamp with your latest Delta write. Catalog Explorer shows the same status for the table.

Tables registered in Unity Catalog get access control, lineage, and audit logging for the people and tools that query them through Databricks. Lineage extends from source data through to models, services, and dashboards. Unity Catalog is also available as an open-source implementation.

Managed Iceberg tables are created and managed directly in Unity Catalog, so they are registered by definition. Iceberg tables in foreign catalogs such as AWS Glue or a Hive metastore are read through Lakehouse Federation as read-only foreign tables, and that registration brings the Unity Catalog access controls that govern who can query them. A Delta table must be registered in Unity Catalog to use UniForm Iceberg reads.

Managed Iceberg tables in Unity Catalog can be created with Databricks Runtime or Databricks SQL, or from external engines through the REST Catalog API, and Predictive Optimization is required for them. The Iceberg docs state that managed Iceberg materialized views are in Public Preview, and do not state a release stage for managed Iceberg tables in general.

Yes. The Unity Catalog REST Catalog API supports read and write operations from external Iceberg engines for managed Iceberg tables, with limits: external engines cannot read Unity Catalog views, materialized views are readable but not writable, and Iceberg v2 position and equality deletes are unsupported. Databricks recommends Iceberg client 1.9.2 or later. By contrast, UniForm exposes Delta tables to Iceberg clients as read-only.

Yes. Unity Catalog creates and manages Iceberg tables, and Databricks handles maintenance such as snapshot expiration and file compaction. Managed Iceberg tables require serverless compute and Predictive Optimization. The cited Iceberg page labels only managed Iceberg materialized views as Public Preview, so check current availability for managed tables in the release notes.

Yes, as a Delta table sink. Structured Streaming supports Delta Lake tables as streaming sinks with exactly-once processing guarantees, and checkpoints store the processing state for fault tolerance. UniForm-enabled tables are Delta tables, so a stream writes to them like any other Delta table; UniForm itself, and the Iceberg reader behavior, are not covered on the cited page. Iceberg clients cannot write to that table.

VACUUM removes old Delta versions and the Parquet files they reference, which can break path-based Iceberg clients that point at older metadata files. Name-based access resolves the current metadata through the catalog and is what triggers Iceberg metadata generation. If a path-based client breaks, update its metadata path to the current metadata file.

Delta Change Data Feed works for Delta clients that read a UniForm table, but Iceberg clients cannot access Change Data Feed information. If external Iceberg readers need row-level changes, build that feed on the Delta side and expose the result to them. Delta clients can use Change Data Feed on a UniForm table without any change to the Iceberg side.

No, Databricks supports PRIMARY KEY and FOREIGN KEY constraints on Unity Catalog and Delta Lake tables from Databricks Runtime 13.3 LTS, but they are informational and not enforced. They document table design and relationships. NOT NULL and CHECK constraints are enforced, and a transaction that violates them fails with an error.

Delta Lake solves the same problem with liquid clustering. You can redefine a table's clustering keys without rewriting existing data, and OPTIMIZE reorganizes the data over time as query patterns change. Liquid clustering replaces both manual partitioning and ZORDER, so you manage one clustering strategy instead of juggling partition schemes.

Databricks enforces NOT NULL and CHECK constraints on Delta tables, and a write that violates either one fails the transaction with an error. Primary key, foreign key, and unique constraints are informational only and are not enforced, but they document table relationships and can help query optimization. To keep keys unique in practice, deduplicate incoming data and use MERGE to upsert on the business key, since MERGE requires that only a single source row matches each target row. From Databricks Runtime 16.0, MERGE also evaluates WHEN MATCHED conditions, in addition to ON, when it detects duplicate matches.

Tables with liquid clustering support row-level concurrency, which reduces write conflicts when concurrent writers modify the same data files. Row-level concurrency depends on deletion vectors and row tracking, so keep both enabled, because turning off either one also turns off row-level concurrency and makes conflicts more likely. Liquid clustering itself is a data layout technique that replaces table partitioning and ZORDER, so the same table gets the layout benefits as well.

Liquid clustering is a data layout optimization that replaces table partitioning and ZORDER. Unlike partitioning, you can redefine clustering keys with ALTER TABLE ... CLUSTER BY without rewriting existing data, and OPTIMIZE then clusters data incrementally. Use OPTIMIZE FULL to recluster all existing data after you enable clustering or change its keys. Liquid clustering also avoids partitioning's fixed boundaries and small-file issues, and it works for both low- and high-cardinality columns, and tables with liquid clustering support row-level concurrency.

Databricks recommends liquid clustering over partitioning for managed tables, and the partitioning guidance says tables under 1 TB generally should not be partitioned. Partitioning works well only for low or known-cardinality fields, while liquid clustering works for both low- and high-cardinality columns and avoids fixed partition boundaries and small-file issues. For tables from 1 TB to 100 TB, use liquid clustering instead of partitioning. Because clustering keys can be changed without rewriting existing data, liquid clustering is also easier to adapt over time, and existing partitioned tables can be converted with ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY (Databricks Runtime 18.1 and above).

For Unity Catalog managed tables, predictive optimization runs OPTIMIZE, VACUUM, and ANALYZE for you, so you usually don't need to schedule these yourself. Databricks recommends enabling predictive optimization for all Unity Catalog managed tables. VACUUM deletes data files that the table no longer references, using a default retention window of 7 days. Predictive optimization also collects statistics when data is written to a managed table.

VACUUM and the transaction log follow separate retention settings, which is why they can appear to be cleaned up inconsistently. Log files are removed automatically after checkpointing, and table history is kept for delta.logRetentionDuration, which defaults to 30 days. Data files are removed only when VACUUM runs, and only after they are older than the deleted file retention threshold, which defaults to 7 days. Time travel needs both the log and the data files for a version, so raise both retention settings if you need to query older versions.

Change data feed records row-level changes between table versions, with _change_type values of insert, update_preimage, update_postimage, and delete, plus _commit_version and _commit_timestamp metadata. The recommended pattern is Structured Streaming, which tracks versions for you: set readChangeFeed to true on spark.readStream and use a checkpoint location. Automatic change data feed needs no per-table setting, computes changes at query time, and requires Databricks Runtime 19 or above on a Unity Catalog table. For a permanent audit history, write the changes to another table, because change feed records expire after a retention window, and use the AUTO CDC APIs for SCD Type 1 or Type 2 processing.

Auto compaction and optimized writes both improve file sizes, at different points in the write. Auto compaction accepts auto (recommended), legacy, true (a 128 MB target), or false. Optimized writes default to on for MERGE and for UPDATE and DELETE with subqueries, and they also apply to CTAS and INSERT on SQL warehouses.

Auto Loader's default schema evolution mode, addNewColumns, is used when you don't provide a schema. In that mode the stream stops once with an UnknownFieldException after merging new columns into the stored schema, and restarting the stream resumes processing. Databricks recommends running streams with Lakeflow Jobs so they restart automatically after a schema change. If you want different behavior, choose another cloudFiles.schemaEvolutionMode: rescue never fails and sends new columns to the rescued data column, and addNewColumnsWithTypeWidening also widens compatible types (Public Preview on Databricks Runtime 16.4 and above).

Use ALTER TABLE to change the schema in place: ADD COLUMNS (with optional COMMENT and FIRST or AFTER positioning), ALTER COLUMN for comments or position, and REPLACE COLUMNS to redefine the full column list. Renaming and dropping columns require Delta Lake column mapping; with column mapping enabled, RENAME COLUMN and DROP COLUMN are metadata-only operations that don't rewrite data files, though dropped data stays in the files until REORG TABLE and VACUUM remove it. Type widening is also a supported schema change. For other column type changes, rewrite the table with mode('overwrite') and overwriteSchema set to true. For schema changes that arrive with the data, use WITH SCHEMA EVOLUTION on INSERT or MERGE, or the mergeSchema option on batch and streaming writes.

Delta Lake time travel lets you query earlier versions of a table, which is useful for recreating past outputs, fixing data mistakes, and providing snapshot isolation. Every write creates a new table version, and DESCRIBE HISTORY lists the operation, user, and timestamp for each one. A version stays queryable only while both its log and data files are retained: table history defaults to 30 days through delta.logRetentionDuration, and data files become eligible for VACUUM after delta.deletedFileRetentionDuration, which defaults to 7 days. Table history is not a long-term backup or archive, and raising retention settings increases storage cost.

Yes. Databricks supports Apache Iceberg in two ways. You can create managed Apache Iceberg tables in Unity Catalog, and OPTIMIZE applies to both Delta Lake and Apache Iceberg tables. Delta tables can also expose Iceberg metadata through UniForm (Universal Format), so Iceberg clients can read the same Parquet files, although Iceberg client support is read-only. UniForm requires column mapping and Delta writes on Databricks Runtime 14.3 LTS or above, and deletion vectors cannot be enabled on a table with Iceberg reads enabled.

Delta Lake provides ACID transactions, and multiple writers across multiple clusters can modify the same table partition at once, with writes ordered serially so that each writer sees a consistent snapshot. Readers keep the consistent snapshot their job started with, even if the table changes during the job. Writes that conflict, such as a ConcurrentAppendException, fail with a conflict exception. Isolation levels, WriteSerializable and Serializable, control how concurrent operations interact and are set with the delta.isolationLevel table property, and row-level conflict detection reduces conflicts on the same data files.

Delta Lake collects data skipping statistics for each file, including minimum and maximum values, null counts, and total record counts, automatically when it writes to a Delta Lake or managed Apache Iceberg table. At query time, Databricks uses these statistics to skip files that cannot match the query predicates, so less data is read. By default, statistics are collected on the first 32 columns of a Unity Catalog external table, and for Unity Catalog managed tables predictive optimization can also collect them on frequently filtered columns beyond the first 32. You can choose the statistics columns with delta.dataSkippingStatsColumns, and Databricks recommends liquid clustering for the table layout that file skipping depends on.

Liquid clustering is the recommended layout for large tables. Define clustering keys on the columns your queries filter by, and OPTIMIZE rewrites data files to group rows by those keys. Clustering keys can be changed later without rewriting existing data, and OPTIMIZE FULL reclusters existing data after you enable clustering or change keys. For Unity Catalog managed tables, CLUSTER BY AUTO lets Databricks choose the keys based on workload when predictive optimization is enabled, and this layout is what lets queries skip irrelevant files.

Use MERGE INTO to upsert the source into the target: WHEN MATCHED updates changed rows, WHEN NOT MATCHED inserts new keys, and WHEN NOT MATCHED BY SOURCE can delete or update target rows that are absent from the source (available on Databricks Runtime 12.2 LTS and above). Deduplicate the source first, because only a single source row can match a given target row. For a full SCD Type 1 or Type 2 pattern in a Lakeflow pipeline, AUTO CDC natively supports both types, so you don't need to hand-write the close-out and insert logic. Add conditions to the MERGE clauses so that only changed rows are rewritten, since unconditional clauses can fully rewrite the target table.

Databricks Transactions let you update multiple tables atomically. They coordinate multiple SQL statements and tables so that all changes succeed together or roll back together. Non-interactive transactions run on any SQL warehouse, serverless compute, or a cluster running Databricks Runtime 18.0 or above, and interactive transactions run on a SQL warehouse. Every written table must be a Unity Catalog managed Delta Lake or Iceberg table with catalog commits enabled.

Delta Lake generated columns compute and store values from a user-defined expression, and they can be used as partition columns. For example, eventDate defined as CAST(eventTime AS DATE) can appear in PARTITIONED BY (eventType, eventDate), and Delta derives partition filters automatically when you filter on the base column. The docs mark this feature as Public Preview.

Not with column mapping. When delta.columnMapping.mode is set to name, Delta Lake column mapping allows metadata-only renames and drops of columns without rewriting data files, and it also allows special characters in column names. It requires Delta reader version 2 or higher and writer version 5 or higher, and reading such tables requires Databricks Runtime 10.4 LTS or above.

Not with deletion vectors. Without them, changing a single row requires rewriting the entire Parquet file that contains it. With deletion vectors enabled, rows are marked as modified or removed in metadata instead. Reads require Databricks Runtime 12.2 LTS or above, and writes using all optimizations require Runtime 14.3 LTS or later.

Yes. On Databricks SQL and Databricks Runtime 12.2 LTS and above, the WHEN NOT MATCHED BY SOURCE clause can UPDATE or DELETE target rows that have no corresponding row in the source. Add an optional condition to the clause so that it does not fully rewrite the target table. Source columns cannot be referenced inside this clause, so use literals or target columns.

A WHEN MATCHED clause runs when a source row matches a target row on the ON condition and the clause's optional condition is true. A MERGE can include at most one update and one delete action in WHEN MATCHED, and multiple WHEN MATCHED clauses run in order, with every clause except the last requiring a condition. The allowed actions are DELETE and UPDATE SET, including UPDATE SET *. If no WHEN MATCHED condition is true, the target row is left unchanged.

MERGE fails with the DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE error when more than one source row matches the same target row, because the update would be ambiguous. Deduplicate the source before merging so that each target row has at most one match. Unconditional DELETE matches are the documented exception, since Databricks allows multiple matches when they are unconditionally deleted. From Databricks Runtime 16.0, MERGE also evaluates WHEN MATCHED conditions when detecting duplicate matches.

Add WITH SCHEMA EVOLUTION after MERGE, for example MERGE WITH SCHEMA EVOLUTION INTO target USING source ..., to enable automatic schema evolution for that single operation (Databricks Runtime 15.2 and above). When it is enabled, the target Delta table's schema is updated to match the source, so new source columns are added. Databricks recommends enabling evolution per write operation rather than through the session-wide spark.databricks.delta.schema.autoMerge.enabled setting.

With schema evolution enabled, source columns that are missing from the target are added to the target, preserving their case, and existing rows get NULL for the new columns. Target columns that are missing from the source are left unchanged: UPDATE SET * leaves them as they are, and INSERT * sets them to NULL. Without evolution, referencing a column that is missing from the source or target produces an error, while evolution changes the schema and keeps the merge running.

For batch writes and Auto Loader streaming writes, pass .option('mergeSchema', 'true') so that source columns are added to the table during the write. On Databricks Runtime 18.1 and above, INSERT WITH SCHEMA EVOLUTION does the same for INSERT INTO, INSERT OVERWRITE, and INSERT INTO ... REPLACE on Delta Lake or Apache Iceberg tables. Databricks recommends enabling evolution per write operation rather than through the session-wide spark.databricks.delta.schema.autoMerge.enabled setting, because the session setting can cause unintended schema changes across operations.

By default, overwriting table data does not overwrite the table schema. To replace both the schema and the partitioning, set the overwriteSchema option to true, which cannot be combined with dynamic partition overwrite. Use that rewrite for a column type change that type widening does not cover; to rename a column, use ALTER TABLE ... RENAME COLUMN with column mapping enabled.

Schema enforcement rejects writes that do not match the table schema on Delta tables, so mismatched data does not land silently. An INSERT must reference only columns that exist in the target, and column types must match, with Databricks casting safely where possible, such as an integer written into a BIGINT column. With INSERT * or UPDATE SET * in a MERGE, the source must contain every target column. Enforcement applies to Delta tables, not to CSV or JSON files in cloud storage, and it checks writes against the existing schema, whereas schema evolution changes the schema.

Set the table property delta.enableDeletionVectors to true, either in CREATE TABLE ... TBLPROPERTIES or with ALTER TABLE ... SET TBLPROPERTIES. Deletion vectors record row-level changes as soft deletes, so Parquet files are not rewritten until a physical operation such as OPTIMIZE or REORG TABLE ... APPLY (PURGE) runs. The cited page does not state that materialized views or streaming tables must be created with deletion vectors; check their documentation before relying on that.

Yes. Enabling deletion vectors upgrades the table protocol, so clients without deletion vector support cannot read the table. Databricks Runtime 12.2 LTS and above can read these tables, as can open source Delta Lake 2.3.0 and above. If needed, you can drop the feature in Databricks Runtime 14.1 and above, and plan client compatibility before enabling it.

Reads of deletion vector tables require Databricks Runtime 12.2 LTS or above, and writes that use all optimizations require Databricks Runtime 14.3 LTS or later. Row-level concurrency for these tables is available from Databricks Runtime 14.2. Open source Delta Lake supports reads from 2.3.0 and above, DELETE from 2.4.0, and UPDATE from 3.0.0. The per-operation, Photon and non-Photon version split is not stated on the cited page, so confirm it against the Databricks Runtime release notes before relying on it.

Deletion vector changes stay soft until a physical operation rewrites the affected data files. OPTIMIZE rewrites affected files, REORG TABLE ... APPLY (PURGE) rewrites all data files that contain deletion vector changes, and an auto-compacting write can rewrite a file that has a deletion vector. After the purge, run VACUUM with the retention threshold set to the purge completion timestamp to delete the old files. For large tables, setting spark.databricks.delta.reorg.purgeMode to rows can speed up purges, whose default is all.

To use legacy change data feed, set delta.enableChangeDataFeed to true in CREATE TABLE ... TBLPROPERTIES or with ALTER TABLE ... SET TBLPROPERTIES. Automatic change data feed needs no per-table setting, computes changes at query time, and requires Databricks Runtime 19 or above on a Unity Catalog table with row tracking. Change data feed records row-level inserts, updates, and deletes so downstream consumers can read only what changed.

The _change_type column takes four values: insert, update_preimage, update_postimage, and delete. update_preimage holds the row as it was before an update, and update_postimage holds it after the update. Each change also carries _commit_version and _commit_timestamp metadata columns.

Use the table_changes function, for example SELECT * FROM table_changes('<table_name>', 0), which reads from version 0 to the latest version. Start and end versions are inclusive, and you can omit the end version to read through the latest. In Python and Scala batch reads, use readChangeFeed with startingVersion, endingVersion, startingTimestamp, and endingTimestamp options, where timestamps use the yyyy-MM-dd[ HH:mm:ss[.SSS]] format. Requesting a version from before change data feed was enabled raises an error.

No, legacy and automatic change data feed cannot be used together on the same table. To migrate a table from legacy to automatic change data feed, run ALTER TABLE <table_name> UNSET TBLPROPERTIES ('delta.enableChangeDataFeed'). Automatic change data feed works on a managed Delta Lake table with row tracking, a managed Iceberg v3 table, or an external Delta Lake table with row tracking, all registered in Unity Catalog.

DESCRIBE HISTORY returns information for each write to the table, including the operation, the user, and the timestamp, listed in reverse chronological order. History is kept for delta.logRetentionDuration, which defaults to 30 days, and that retention determines how far back the history goes. Use DESCRIBE HISTORY to audit writes or to find the version you want to query with time travel.

By default, time travel covers the past 7 days, because the data file retention threshold (delta.deletedFileRetentionDuration) defaults to 7 days, and time travel needs both the log and the data files for a version. Table history is kept for 30 days by default through delta.logRetentionDuration. To go further back, increase both retention settings, which raises storage cost. On Databricks Runtime 18.0 and above, time travel to versions older than deletedFileRetentionDuration is blocked.

VACUUM removes data files that a Delta table no longer references and that are older than the retention threshold, which reduces storage cost. Because it permanently deletes those older files, versions that depend on them can no longer be queried with time travel after VACUUM runs. Log files are deleted automatically and asynchronously after checkpoint operations and are not governed by VACUUM.

The default retention threshold for data files removed by VACUUM is 7 days, set by the delta.deletedFileRetentionDuration table property. Databricks strongly recommends keeping at least 7 days, because long-running jobs may write files that are not yet committed. Setting a shorter interval risks deleting files that a running job still needs.

Run VACUUM table_name DRY RUN to list the files that would be deleted without removing anything. Review that list before you run VACUUM for real.

Yes. Delta Lake has a safety check that prevents a dangerous VACUUM command, such as one with a retention period shorter than the table's operations need. You can turn it off by setting spark.databricks.delta.retentionDurationCheck.enabled to false, but only if no operation on the table runs longer than the retention interval you specify. Databricks strongly recommends a retention interval of at least 7 days.

Retention is governed by the delta.deletedFileRetentionDuration table property, which you can raise with ALTER TABLE ... SET TBLPROPERTIES. On Databricks Runtime 18.0 and above, this table property is the setting that controls VACUUM retention. Keep it at 7 days or longer, since a shorter period can remove files that long-running jobs still need.

Predictive optimization runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables, and with automatic liquid clustering enabled it may select new clustering keys. It identifies tables that would benefit from maintenance and queues those operations, and it collects statistics when data is written to a managed table. These operations run on serverless compute, which is billed under a serverless jobs SKU, so there is no manual scheduling to maintain.

Predictive optimization supports only Unity Catalog managed tables, which cover Delta Lake and Iceberg. External tables and tables loaded to a workspace as OpenSharing recipients are not supported. Databricks recommends enabling predictive optimization for all Unity Catalog managed tables, and it requires a Premium plan or above in a supported region.

An account admin can enable predictive optimization for all metastores in the account from the accounts console, under Settings and then Feature enablement. Catalogs, schemas, and tables inherit the account value by default, and you can override it at any level with ALTER ... ENABLE, DISABLE, or INHERIT PREDICTIVE OPTIMIZATION. Accounts created on or after November 11, 2024 have it enabled by default.

Liquid clustering is a data layout optimization that replaces table partitioning and ZORDER. Unlike partitioning, its clustering keys can be changed later without rewriting existing data. Databricks recommends liquid clustering over partitions and ZORDER for all new tables and recommends it for managed tables generally.

Add a CLUSTER BY clause to CREATE TABLE, for example CREATE TABLE table1 (col0 INT, col1 STRING) CLUSTER BY (col0). On an existing unpartitioned Delta table, use ALTER TABLE <table_name> CLUSTER BY (<clustering_columns>). Clustering keys can be up to four columns and cannot be complex types such as struct, map, or array, and after enabling clustering, OPTIMIZE clusters the existing data incrementally.

Run ALTER TABLE table_name CLUSTER BY (new_column1, new_column2) to change the keys without rewriting existing data, and run ALTER TABLE table_name CLUSTER BY NONE to remove them. Removing keys does not rewrite clustered data, but it stops future OPTIMIZE operations from using keys. After you change keys, run OPTIMIZE table_name FULL to recluster existing data, which the clustering guide cites as available on Databricks Runtime 16.4 LTS and above.

No. Liquid clustering is not compatible with partitioning or ZORDER, and you cannot use ZORDER BY on a table that uses liquid clustering. When migrating from ZORDER, use the former ZORDER BY columns as the clustering keys. Liquid clustering keeps the layout benefits of the older approaches while letting you change keys over time.

CLUSTER BY AUTO has Databricks intelligently choose clustering keys to optimize query performance, instead of you picking them. Enable it with ALTER TABLE table1 CLUSTER BY AUTO on Unity Catalog managed tables (Databricks Runtime 15.4 LTS and above), and it requires predictive optimization for automatic key selection and clustering operations. You can set starting hints with CLUSTER BY (c1, c2) before enabling AUTO. It is not available for managed Apache Iceberg v2 tables.

OPTIMIZE rewrites data files to improve layout, using bin-packing, which is idempotent, so running it again on the same data has no further effect. On tables with liquid clustering it groups data by the clustering keys, and on partitioned tables it compacts files within each partition. OPTIMIZE changes only file layout and no data, so queries return the same results before and after. Databricks recommends starting with OPTIMIZE once a day.

No, OPTIMIZE does not change query results or interrupt concurrent readers. OPTIMIZE makes no data changes, so a read before and after returns the same results. Readers of Delta Lake tables use snapshot isolation, so running readers are not interrupted when OPTIMIZE removes files from the transaction log. Running OPTIMIZE on a streaming source also does not affect current or future streams that read the table, so it can run alongside normal reads.

Add a WHERE predicate, for example OPTIMIZE table_name WHERE date >= '2022-11-18', to optimize only a subset of a large table. The predicate supports only filters on partition or clustering key attributes. To force reclustering of a liquid clustered table, run OPTIMIZE table_name FULL, which requires Databricks Runtime 16.0 or above.

OPTIMIZE FULL rewrites all data files in the table, which optimizes previously clustered data for tables that use liquid clustering. It is also how you recompress existing files after you change the delta.parquet.compression.codec property. The OPTIMIZE reference lists Databricks Runtime 16.0 and above as the requirement for FULL.

No. The partitioning guidance says that with less than 1 TB of data you should not partition, and with 1 TB to 100 TB of data you should use liquid clustering instead of partitioning. Databricks recommends liquid clustering for all managed tables, and unpartitioned tables automatically use ingestion time clustering, which gives performance similar to datetime-based partitioning without manual tuning.

Declare a generated column with GENERATED ALWAYS AS, for example dateOfBirth DATE GENERATED ALWAYS AS (CAST(birthDate AS DATE)). In the Python builder, pass the expression as generatedAlwaysAs. If a write omits the generated column, Delta Lake computes its value automatically. The feature is in Public Preview, and enabling it upgrades the writer protocol, which might affect compatibility with external Delta Lake clients.

A generation expression can use any deterministic SQL function that returns the same result for the same inputs, such as arithmetic like base_price * 1.1, string functions like SUBSTRING, and date functions like YEAR(eventTime). User-defined functions, aggregate functions, window functions, and functions that return multiple rows are not allowed. A generated column also cannot reference a column that has a column mask, because doing so raises an error.

If a write supplies a value for a generated column, the value must satisfy (value <=> generation expression) IS TRUE, or the write fails. If a write omits the column, Delta Lake computes the value automatically. This check means the stored value always matches the expression.

Setting delta.columnMapping.mode to name allows metadata-only renames and drops of columns without rewriting data files, and it allows special characters such as spaces in column names. You can set the mode on new or existing tables with ALTER TABLE ... SET TBLPROPERTIES. Databricks recommends id mode for most use cases, and name mode is set automatically if you enable Iceberg compatibility features such as Iceberg reads.

No. The id column mapping mode must be set when the table is created and cannot be set on an existing table. Databricks recommends id mode for most use cases, and the name mode can be set on existing tables instead. Either mode, once enabled, lets you rename and drop columns without rewriting data.

Yes. With column mapping enabled in name or id mode, ALTER TABLE ... RENAME COLUMN and ALTER TABLE ... DROP COLUMN are metadata-only changes that do not rewrite data files. Dropped data remains in the files until you run REORG TABLE and then VACUUM to remove it. Column mapping requires reader version 2 or above and writer version 5 or above, and reading such tables requires Databricks Runtime 10.4 LTS or above.

Databricks enforces NOT NULL and CHECK constraints. NOT NULL blocks null values in the specified columns, and CHECK requires a boolean expression to be true for every row. When a constraint is violated, the transaction fails with an error, and all constraints on Databricks require Delta Lake.

No, primary key, foreign key, and unique constraints are not enforced on Databricks tables. The constraints documentation says they are informational only; they describe relationships between table fields and can still improve performance through query optimizations. Primary key and foreign key support is generally available from Databricks Runtime 15.2 and above.

The table must be registered in Unity Catalog and must have column mapping enabled, with minReaderVersion of at least 2 and minWriterVersion of at least 7. Writes to the table must use Databricks Runtime 14.3 LTS or above, and both managed and external tables are supported. You cannot enable deletion vectors on a table with Iceberg reads enabled.

Set delta.columnMapping.mode, delta.enableIcebergCompatV2, and delta.universalFormat.enabledFormats (set to iceberg) in CREATE TABLE ... TBLPROPERTIES for a new table. On an existing table, set the same properties with ALTER TABLE ... SET TBLPROPERTIES, which requires Databricks Runtime 15.4 LTS or above and column mapping mode name. For tables with deletion vectors, tables on the older IcebergCompatV1, or tables read by engines such as Athena or Redshift, run REORG TABLE table_name APPLY (UPGRADE UNIFORM(ICEBERG_COMPAT_VERSION=2)), which rewrites data files.

A shallow clone copies only the metadata, not the data files, so it is cheaper to create because it uses less compute and storage. It copies the schema, partitioning, invariants, nullability, and TBLPROPERTIES, and its readers need access to the source table's data files. On legacy Hive metastore tables, running VACUUM on the source can break shallow clones that still reference its files.

A deep clone copies both the data and the metadata, including stream metadata, so it does not depend on the source table's data files afterward. A stream that writes to the source can be stopped and resumed on the clone. Only deep clones also carry over COPY INTO metadata, but creating a deep clone costs more compute and storage than a shallow clone.

Neither clone type copies the table description, user-defined commit metadata, the Delta history, or Unity Catalog properties such as tags, because a cloned table has its own independent history. Schema, partitioning, invariants, nullability, and TBLPROPERTIES are copied. If you need tags or a description on the clone, reapply them after cloning.

Row tracking assigns stable row IDs and row commit versions to each row, enabling row-level lineage tracking. It exposes the metadata field _metadata.row_id, which stays the same when a row is modified by MERGE or UPDATE, and _metadata.row_commit_version, which records the table version at which the row was last inserted or updated. Row tracking is available on Databricks Runtime 14.0 and above, and all Apache Iceberg v3 tables include it.

Set delta.enableRowTracking to true in CREATE TABLE ... TBLPROPERTIES, or run ALTER TABLE ... SET TBLPROPERTIES on an existing table. Enabling it on an existing table assigns row IDs to all current rows, which can create multiple table versions and take significant time, so pause continuous writes first because the ALTER causes concurrent writes to fail. Row tracking is available on Databricks Runtime 14.0 and above.

Run UNDROP TABLE with the table name to restore the most recently dropped table with that name that you own, or use UNDROP TABLE WITH ID when several dropped tables share a name (find the ID with SHOW TABLES DROPPED). The parent schema and catalog must still exist, and if a table already uses the name, rename it first. Privileges, column specs, and properties are restored, but primary and foreign key constraints must be recreated with ALTER TABLE ... ADD CONSTRAINT. UNDROP applies on Databricks Runtime 12.2 LTS and above.

Dropped tables (managed or external Unity Catalog tables) can be restored within a default 7-day recovery period. Catalog or schema owners can change this window with ALTER CATALOG or ALTER SCHEMA, and the schema-level setting takes precedence when both are set. Temporary tables cannot be undropped.

REORG TABLE ... APPLY (PURGE) rewrites only the files that contain soft-deleted data, such as column data removed by ALTER TABLE ... DROP COLUMN. An optional WHERE predicate limits the purge to matching partitions, and only partition key filters are supported. REORG TABLE is idempotent, so running the purge twice has no further effect.

After APPLY (PURGE), the soft-deleted data may still exist in the old files, so run VACUUM to physically delete them. Once VACUUM has removed the old files, the soft-deleted data is no longer recoverable.

Each Delta table protocol has minReaderVersion, the minimum read capability a client needs (values 1 to 3), and minWriterVersion, the minimum write capability (values 2 to 7). Protocol versions are cumulative, and table features replace these integers with per-feature flags at minReaderVersion 3 and minWriterVersion 7. Databricks recommends that you never alter these two table properties directly.

No. Turning off a table feature does not downgrade the protocol. To fully remove a feature from the protocol, use ALTER TABLE ... DROP FEATURE, which works only for the features Databricks supports dropping. Other table features cannot be dropped.

ALTER TABLE <table> DROP FEATURE <feature> can drop catalogManaged, checkConstraints, collations-preview, columnMapping, deletionVectors, typeWidening, v2Checkpoint, and checkpointProtection, one feature per command. Other Delta Lake table features cannot be dropped. You need MODIFY privilege on the table, and Databricks recommends Databricks Runtime 16.3 or above for all DROP FEATURE commands.

For a full downgrade so that legacy external Delta clients can keep working, use TRUNCATE HISTORY. It runs in two steps at least 24 hours apart: the first turns off the feature, resets its properties, and rewrites files, and the second truncates history to the retention threshold, confirms no transactions use the feature, drops it, and downgrades the protocol. The command removes transaction log data older than 24 hours, so after a full downgrade you can no longer access table history or use time travel.

Use ALTER TABLE ... SET TBLPROPERTIES to change properties on an existing table, and Delta Lake properties use the delta. prefix. To set defaults for new tables in a session, use spark.databricks.delta.properties.defaults.<conf>, which does not change existing tables. Setting table properties conflicts with concurrent writes and can fail, so change them when no concurrent writes are running.

delta.logRetentionDuration controls how long table history is kept, with a default of 30 days, and VACUUM operations override this threshold. delta.deletedFileRetentionDuration is the shortest time that logically deleted data files are kept before physical deletion, with a default of one week. Databricks recommends keeping the default or a longer value, because shorter periods can make long-running jobs lose uncommitted files.

A Delta streaming source only accepts append inputs, so an UPDATE, DELETE, MERGE INTO, or OVERWRITE on the source makes the stream fail with an exception. Setting skipChangeCommits makes the stream ignore transactions that change existing records and process only appends. ignoreChanges is the older option, and skipChangeCommits replaced it.

skipChangeCommits replaced ignoreChanges in Databricks Runtime 12.2 LTS and above, and ignoreChanges is supported only up to Databricks Runtime 11.3 LTS. ignoreChanges re-emits rewritten files, so downstream consumers can see duplicate rows, and deletes are not propagated. The legacy ignoreDeletes option only handles full partition drops, so skipChangeCommits is the recommended option.

The default maxFilesPerTrigger for a Delta streaming source is 1000 files per micro-batch. You can lower or raise it to control how much new data each micro-batch processes. Set it with your file sizes and latency targets in mind.

The query must run at least once within the source table's retention window. Default retention is 7 days for VACUUM-removed data files and 30 days for the transaction log. A query that falls behind fails with DELTA_FILE_NOT_FOUND_DETAILED and must be reset with a full refresh. Setting spark.sql.files.ignoreMissingFiles to true to get past the error is discouraged, because it silently produces incorrect results.

For Unity Catalog external Delta tables, data skipping statistics are collected on the first 32 columns of the table schema by default. For Unity Catalog managed tables, predictive optimization can also collect statistics on columns beyond the first 32 that are frequently used in query filters. Statistics are collected automatically when you write to a Delta Lake or managed Apache Iceberg table.

Set delta.dataSkippingStatsColumns to a list of column names on Databricks Runtime 13.3 LTS and above, which supersedes the count-based delta.dataSkippingNumIndexedCols setting. Set these properties at table creation or with ALTER TABLE ... SET TBLPROPERTIES. Changing them affects only future writes, and statistics are not used for columns outside the current statistics column list. Long strings are truncated during collection, so you may choose to exclude long string columns.

On Databricks Runtime 14.3 LTS and above, run ANALYZE TABLE table_name COMPUTE DELTA STATISTICS to recompute statistics for existing data. Changing the statistics properties does not recompute statistics for data that is already written. Predictive optimization runs ANALYZE automatically on Unity Catalog managed tables.

Delta Lake is the open-source storage layer that extends Parquet data files with a file-based transaction log, providing ACID transactions and scalable metadata handling. It also adds schema validation on write, table versioning and time travel, and change data feed on top of the Parquet files. On Databricks, Delta Lake is the default format for all operations, so tables are Delta Lake tables unless you specify otherwise. The transaction log has an open protocol that any system can use to read it.

The sharing platform is documented as OpenSharing, an open protocol developed by Databricks for secure data sharing that works regardless of the computing platforms recipients use. Recipients access shared data assets in a read-only format and cannot write to them. Open sharing lets you share with any user, including people who do not use Databricks, and Databricks-to-Databricks sharing requires the recipient to have a workspace that is enabled for Unity Catalog.

Predictive optimization automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables. Its OPTIMIZE operations do not run ZORDER, so use liquid clustering for data layout on those tables. Set delta.deletedFileRetentionDuration before enabling predictive optimization if you need longer retention for time travel, since the default is seven days.

VACUUM removes data files that are no longer referenced by the table and are older than the retention threshold. The default retention threshold is seven days.

Yes, CLUSTER BY AUTO lets Databricks choose the clustering keys, relying on predictive optimization and the table's query history. You enable liquid clustering by adding the CLUSTER BY clause to a table creation statement, and the same clause can be applied to existing unpartitioned tables with ALTER TABLE. Automatic selection can decline to pick keys for small tables or tables that already have an effective layout.

Do not combine liquid clustering with partitioning or ZORDER on the same table, since Databricks recommends liquid clustering in place of those layouts. To force reclustering of an existing liquid clustered table, run OPTIMIZE table_name FULL, which requires Databricks Runtime 16.0 or above.

Deletion vectors are soft deletes that mark rows as changed in existing Parquet files, so a single-row change does not require rewriting the whole file, and reads apply them at query time. Enable them with the delta.enableDeletionVectors table property, for example in TBLPROPERTIES. Running OPTIMIZE or REORG TABLE ... APPLY (PURGE) rewrites the affected files to apply the marked changes, and VACUUM is still needed to physically remove old data after a purge.

delta.logRetentionDuration defaults to 30 days, and delta.deletedFileRetentionDuration defaults to one week. Changing these values changes how far back you can query. Databricks recommends the default of 7 days or higher for deletedFileRetentionDuration.

Use the RESTORE command to return a table to a previous version or timestamp, for example with TO VERSION AS OF or TO TIMESTAMP AS OF. Time travel queries use VERSION AS OF or TIMESTAMP AS OF after the table name, and DESCRIBE HISTORY lists the operation, user, and timestamp of each write. Restoring or querying an old version only works while the log and data files for that version are still retained.

Yes, type widening changes a column to a wider type without rewriting data files, for example INT to BIGINT. Enable it by setting the delta.enableTypeWidening table property to true, either at table creation or with ALTER TABLE ... SET TBLPROPERTIES. The documentation describes it for Delta Lake tables in Databricks Runtime 15.4 LTS and above, with integer widening among the supported conversions.

Enable legacy change data feed by setting the table property delta.enableChangeDataFeed to true, in CREATE TABLE or with ALTER TABLE; automatic change data feed needs no per-table setting. Read the changes by setting the readChangeFeed option to true for streaming reads, and each row carries a _change_type value of insert, update_preimage, update_postimage, or delete. Commit metadata columns such as _commit_version and _commit_timestamp are included as well.

MERGE INTO upserts a source table, view, or DataFrame into a Delta table, and the WHEN NOT MATCHED BY SOURCE clause handles target rows that have no matching source row, with delete or update actions. A merge can fail when multiple source rows match the same target row, so preprocess the source to remove duplicate matches. Multiple WHEN MATCHED clauses are evaluated in the order they are specified, and all but the last must have a condition.

Yes, OPTIMIZE accepts an optional WHERE partition predicate, so you can rewrite only a subset such as one date range. Bin-packing is idempotent, so running it again on the same data has no further effect. Auto compaction handles small files separately from the OPTIMIZE command, which you run explicitly to trigger compaction.

Define a generated column with GENERATED ALWAYS AS (expression), and Delta Lake computes its value automatically when you write without supplying it. If you do supply a value, it must match the generation expression or the write fails. The expression can use any deterministic SQL function, while user-defined functions, aggregate functions, window functions, and functions that return multiple rows are excluded.

No, PRIMARY KEY, FOREIGN KEY, and UNIQUE constraints are informational only and are not enforced. NOT NULL and CHECK constraints are enforced, and you add or drop CHECK constraints with ALTER TABLE ADD CONSTRAINT and ALTER TABLE DROP CONSTRAINT. When an enforced constraint is violated, the transaction fails with an error.

Unity Catalog tables are Delta Lake tables by default unless you specify USING iceberg. Unity Catalog manages the read, write, storage, and optimization responsibilities for these managed tables. Predictive optimization is enabled by default for accounts created on or after November 11, 2024, and other accounts are being enabled gradually.

Create a VARIANT column with the parse_json function, then extract fields with variant_get, or with the colon shorthand for top-level fields such as raw:owner. The documentation covers querying and transforming semi-structured data stored as VARIANT. This avoids declaring a fixed schema for every JSON payload you ingest.

CONVERT TO DELTA performs a one-time conversion of Parquet and Apache Iceberg tables to Delta Lake. Iceberg conversion requires Databricks Runtime 10.4 LTS or above. In Unity Catalog, CONVERT TO DELTA applies to external tables, while Hive metastore managed tables are converted with CTAS. For incremental conversion of Parquet or Iceberg tables, the docs point to a separate guide.

CHECK constraints are enforced on Delta tables. Each row must satisfy the boolean expression, and when a constraint is violated the transaction fails with an error. NOT NULL constraints are also enforced. Primary key and foreign key constraints are informational only and are not enforced.

When an INSERT, UPDATE, or MERGE insert does not specify a column that has a DEFAULT, Databricks applies the column's default expression. Defaults are supported in Databricks Runtime 11.3 LTS and above. The expression can use literals and built-in functions, but not aggregates, window functions, or subqueries. A nullable column without a default receives NULL.

Delta tables default to WriteSerializable, which Databricks recommends keeping for most workloads. Reads always use snapshot isolation under both levels. Under WriteSerializable, a DELETE, UPDATE, or MERGE can commit after a concurrent insert, so table contents may not match a replay of the history order. Serializable makes a later conflicting operation fail so it must be retried, at a cost to concurrent write throughput. Set it with ALTER TABLE <table-name> SET TBLPROPERTIES ('delta.isolationLevel' = 'Serializable'), and set it back to WriteSerializable to restore the default.

No. Databricks has deprecated Bloom filter indexes and recommends removing existing ones. The recommended replacements are Predictive I/O, which handles file skipping across every column on Photon-enabled compute in Databricks Runtime 12.2 or later and fully supersedes Bloom filters, and liquid clustering, available in Databricks Runtime 13.3 and later.

Run ALTER TABLE ... SET TBLPROPERTIES ('delta.appendOnly' = 'true'). When it is true, new records can be added but existing records cannot be updated or deleted. The property is a Boolean with a default of false, and setting it back to false removes the restriction.

Use TIMESTAMP_NTZ for wall-clock values, such as a local appointment time, because its operations apply no time zone. Declaring a TIMESTAMP_NTZ column on a new Delta table enables the feature automatically. TIMESTAMP_NTZ requires Databricks Runtime 13.3 LTS or above and is in Public Preview.

Run DESCRIBE HISTORY on the table and read the operationMetrics map for MERGE operations. Keys include numTargetRowsInserted, numTargetRowsUpdated, numTargetRowsDeleted, numSourceRows, and executionTimeMs. Metrics vary by operation, so check the metric table for the operation you are analyzing before you query the keys.

Databricks recommends REPLACE USING or REPLACE WHERE for most use cases. Legacy partitionOverwriteMode replaces all existing data in every partition the incoming write touches, so one misplaced row can overwrite an entire partition. REPLACE WHERE fails by default if any incoming row falls outside its predicate, which protects existing data.

Yes. Databricks documents RESTORE as a data-changing operation that might result in duplicate data for downstream workloads. Streaming jobs that read the table can see restored rows as new data changes. Also, VACUUM deletes data files, and you cannot restore a table to an older version that references files VACUUM removed.

Yes, Delta Lake type widening lets you change a column's data type without rewriting the table. It widens supported types, such as INT to BIGINT or DECIMAL, FLOAT to DOUBLE, or DATE to TIMESTAMP_NTZ, using ALTER TABLE ... ALTER COLUMN ... TYPE. The docs say a manual type change updates the table schema without rewriting the underlying data files. Enable it with the delta.enableTypeWidening table property set to true. Reading tables with type widening requires Databricks Runtime 15.4 LTS or above.

Delta type widening lets a column move to a wider compatible type, such as BYTE to SHORT, INT, BIGINT, DECIMAL or DOUBLE, INT to BIGINT, DECIMAL or DOUBLE, and DATE to TIMESTAMP_NTZ. It covers top-level columns and nested struct, map and array fields. It requires Databricks Runtime 15.4 LTS or above, and streaming support requires Runtime 16.4 LTS or above. Enable it with ALTER TABLE ... SET TBLPROPERTIES ('delta.enableTypeWidening' = 'true').

Use ALTER TABLE table_name ALTER COLUMN column_name SET DEFAULT expression. Inserts that omit the column, including MERGE ... INSERT actions, receive the default. Changing the default does not affect existing rows, and the default expression cannot contain subqueries or aggregate, analytic or ranking window functions. To add a column with a default, run ADD COLUMN first, then SET DEFAULT in a separate statement, since one ADD COLUMN cannot define it.

V2 checkpoints let a Delta table rebuild its state without replaying the full transaction log, which speeds up query planning and supports more concurrent writers with fewer write conflicts. Enable them with ALTER TABLE ... SET TBLPROPERTIES ('delta.checkpointPolicy' = 'v2'). Databricks Runtime 13.3 LTS and above can read and write these tables.

Yes, in Beta. Databricks supports reading a change data feed from materialized views created in Lakeflow pipelines or Databricks SQL. It requires Databricks Runtime 18 LTS or above, row tracking enabled on the materialized view, the PREVIEW channel on the materialized view or on the pipelines involved, and the external metadata flag. Only Databricks can query it. Read it with table_changes() or a streaming table using readChangeFeed, from a different pipeline than the one that creates the view, since same-pipeline reads are not supported.

Yes, in Beta. Time travel on Unity Catalog managed tables can enforce ABAC row filters and column masks. It requires the Time Travel on UC Managed Tables with Attribute-Based Access Controls preview, standard compute on Databricks Runtime 19 or above or serverless compute, and either a managed Delta table with catalog commits enabled or a managed Iceberg table. Databricks applies the ABAC policies currently defined on the table to the historical data.

No. Delta Lake is open source software that extends Parquet data files with a file-based transaction log, so it is not tied to Spark or to Databricks. Connector support varies by engine and version, so confirm that your connector supports the table features you use.

They can, and the Databricks docs say so. A client must support every table feature a Delta table uses, or it cannot read the table. Open-source Delta clients choose how to implement each feature, and some Databricks Runtime operations might not be available in the matching OSS Delta Lake version. The docs also say table protocol versions cannot be downgraded. Before enabling newer features, confirm that every tool reading the table supports them.

Databricks announced its acquisition of Tabular in June 2024, a company announcement described in its blog. Iceberg interoperability in Unity Catalog is documented: external engines can read and write managed Iceberg tables through the REST Catalog API, with the limits listed in the docs.

Performance depends on choosing clustering keys that match your filters. The docs say to choose clustering keys based on the columns most commonly used in query filters, allow up to four keys,. Liquid Clustering replaces table partitioning and ZORDER. The docs do not include a benchmark against Z-ordering, so test your own multi-column filters before drawing a conclusion.

No. Delta Lake is open source software that extends Parquet data files with a file-based transaction log, so your data is stored in standard Parquet files in storage you control.

The table data is Parquet files plus a transaction log, but a reader must support each table feature the table uses. Reader features require both read and write client support, so a basic reader may not open every table, for example one that uses deletion vectors, column mapping, or liquid clustering. Databricks recommends testing compatibility before enabling new features on production tables. Plan the exit around a representative production table, not just the format name.

UniForm lets Iceberg clients read a Delta table without rewriting its Parquet files, because Databricks generates Iceberg metadata alongside the Delta metadata and a single copy of the data files supports both. Iceberg client support is read-only, so writes still go through Delta. The table must be registered in Unity Catalog with column mapping enabled, and writes must run on Databricks Runtime 14.3 LTS or above. Iceberg metadata is generated asynchronously after a Delta write transaction completes.

Not for native Iceberg. Unity Catalog managed Iceberg tables support reads and writes, including from external engines through the Iceberg REST catalog. Foreign Iceberg tables are read-only in Databricks. Use managed Iceberg tables where external multi-engine writes are required, and confirm the workload on a test table first, since the documented limitations are subject to change.

Databricks documents its current Iceberg limits openly, and you should check them against your workload before moving. Iceberg branching and tagging are not supported, and Iceberg v2 position and equality deletes are not supported (v3 deletion vectors are used instead). Only Parquet is supported, expression-based partition transforms such as years() and bucket() are not supported on managed tables, and external Iceberg engines cannot read Unity Catalog views. Managed Iceberg tables require serverless compute to be enabled in the workspace and can only be created when predictive optimization is enabled.

Delta tables declare the reader and writer features they need, and enabling a feature upgrades the table protocol. Reader features affect both reads and writes, so an older client may not be able to open the table. The docs say most protocol version upgrades are irreversible and that not all table features can be dropped, and Databricks recommends never directly altering the minReaderVersion and minWriterVersion table properties. Test your engine's support for each feature, and test compatibility before enabling features on production tables.

Yes, table contents can be exported. INSERT OVERWRITE DIRECTORY writes query results to a directory using a specified Spark file format, including TEXT, CSV, JSON, PARQUET, and ORC. That moves the data but not the catalog entries, grants, or the pipelines that maintain the tables, which you must rebuild elsewhere. Pair the export with an inventory of tables, grants, and jobs so the handover is complete.

Open formats reduce lock-in but do not remove the choice of write path. With UniForm, Databricks generates Iceberg metadata for a Delta table over the same data files, so a single copy supports both Delta and Iceberg clients. Iceberg client support is read-only and writes are not supported, so writes stay on Delta. Metadata generation runs asynchronously after a Delta write transaction completes, and external Iceberg clients cannot query a change until it finishes, so plan for that lag and choose your write engine before you commit.

Automation removes routine tuning, and you keep control of what matters. Predictive optimization automatically runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables, Databricks recommends it for all Unity Catalog managed tables, and it is enabled by default for accounts created on or after November 11, 2024. Automatic liquid clustering can choose clustering keys from your data usage patterns. Retention is still a design decision: VACUUM's deleted-file retention defaults to 7 days, and shorter retention limits time travel, so set it deliberately before relying on automation.