Unity Catalog, Governance & Security

Unity Catalog is the unified governance layer for data and AI assets, applying access controls, row filters and column masks across the platform from one place. It tracks lineage automatically across data, models, services, and dashboards, and it can classify and tag sensitive data. Activity is recorded in the audit log system table, so compliance teams can review access with standard SQL. Unity Catalog documentation covers Azure and GCP as well as AWS, so the same governance model applies wherever your data lives.

Yes. Genie lets you ask natural-language questions over governed Unity Catalog data and returns the SQL and results behind each answer. Answers are grounded in your organization's data and governed through Unity Catalog. Data teams can improve answers by configuring trusted datasets, sample queries, metrics, and instructions.

Governed tags let you define the allowed tag keys and values for Unity Catalog objects and control which users and groups can assign them, so classifications such as sensitivity level or data owner stay consistent. Tag values must come from the associated tag policy. Automated tag assignment for governed tags is currently in beta.

Yes. Multi-factor authentication is available for accounts that do not use single sign-on, and account admins configure it in the account console under Security > Identity provider setup. The setting can be Recommended (the default), Required, or Disabled. Users can authenticate with a passkey or security key, which Databricks strongly recommends, or with an authenticator app that uses a 6-digit code.

Unity Catalog centralizes governance for data and AI assets, with role-based access control, attribute-based access control (ABAC) policies, row filters and column masks for fine-grained security, and sensitive-data classification. Lineage is tracked automatically across data, models, services, and dashboards, and activity is recorded in the audit log system table. Governed tags and Delta Sharing extend the same controls to metadata and external sharing. Some ABAC components, such as metastore-level policies, are still in beta.

Governed tags restrict tag keys and values to a tag policy, so classifications such as data owner or sensitivity stay consistent across the organization. Metric views add a governed layer of business metrics: measure definitions are kept separate from the dimensions used to query them, so each KPI is defined once and reused across dashboards, SQL, and Genie. Together these give teams a shared business vocabulary for data discovery and analysis.

Models and functions are governed through Unity Catalog, which applies access control and attribute-based access control (ABAC) policies and governed tags to AI assets. Unity Gateway adds guardrails, rate limits, and usage tracking for model access, MCP servers, and skills. Customers can also protect data at rest with their own encryption keys to maintain control over encryption.

Unity Gateway is Databricks' governance layer for AI model access, applying guardrails, rate limits, and usage tracking to foundation models, MCP servers, and skills. Admins use it to control access to these AI resources from one place, and Agent Bricks routes its model access through it.

Yes. Unity Catalog governs volumes, the securable objects that hold unstructured files, alongside tables, so access control and audit logging apply across structured and unstructured data. Volumes can be managed or external, and the same three-level namespace of catalog, schema, and object organizes both.

Yes. Through Delta Sharing's open protocol, recipients on any compute platform can read shared tables, and the provider controls what they see with dynamic views and token policies. Databricks-to-Databricks recipients authenticate through the platform without managing tokens. Sharing activity is recorded in audit logs and system tables, so providers can track external access.

Managed tables are tables where Unity Catalog handles both governance and the underlying storage lifecycle, so you do not manage the storage location yourself. When Predictive Optimization is enabled, Databricks runs OPTIMIZE, VACUUM, and ANALYZE on them automatically. Permissions are governed centrally in Unity Catalog, and row filters and column masks are enforced on supported engines and runtimes.

Unity Catalog row filters and column masks are SQL user-defined functions that apply policies at query time. A row filter excludes rows where it returns FALSE, and a column mask returns either the original value or a masked value of a compatible type. Table owners attach them to tables, or you can use attribute-based policies at catalog or schema scope that owners cannot override. Runtimes below 12.2 LTS fail securely without returning data, and Delta Lake APIs and the Iceberg REST catalog cannot read tables that have them.

Unity Catalog captures table-level and column-level lineage automatically for queries run on Databricks, including those from notebooks, jobs, and Lakeflow pipelines. Open a table in Catalog Explorer and use the Lineage tab to see upstream sources and downstream consumers, and click a column in the lineage graph to see its upstream and downstream links. Column-level capture is best-effort, and UDFs get table-level lineage only.

Unity Gateway is Databricks' governance layer for enterprise AI, and it registers AI assets as Unity Catalog securables so you grant and revoke model access with the same privileges used for tables. You can set rate limits on model services, route traffic with fallbacks, and set per-user budgets with hard caps. Requests, token usage, and cost attribution are tracked in system tables.

Unity Catalog data classification scans tables and tags sensitive data, including PII, through compliance-framework classifications. Once automatic tagging is enabled for a classification, all existing and future detections are tagged. You can then enforce policies with row filters and column masks, which apply at query time to hide or restrict sensitive values.

Unity Gateway lets you control which AI models and providers teams can use. AI assets are registered as Unity Catalog securables, so you grant or revoke access with the same privileges used for tables. You can set rate limits and per-user budgets with hard caps, and track requests and token usage in system tables.

Unity Catalog is the unified governance layer for data and AI on Databricks. It enforces access control, tracks lineage, and logs activity for securable objects organized in a three-level catalog.schema.object namespace. Permissions are granted once in Unity Catalog and apply to the objects they cover.

Managed tables are tables where Unity Catalog handles both governance and the underlying storage lifecycle. External tables get governance from Unity Catalog while you manage the storage location. Managed tables are the ones Predictive Optimization maintains automatically, so choose managed for simplicity and automatic maintenance, and external when your data must stay in storage you control.

Use Catalog Explorer and Unity Catalog search to find tables, models, and other governed assets across your organization. The Lineage tab shows upstream sources and downstream consumers, including jobs and dashboards, so you can judge whether a dataset fits your need. Lineage also shows how each asset is produced and used.

Yes. Unity Catalog data classification uses an agent to scan and tag tables for sensitive-data classifications such as PII, and you can enable automatic tagging per classification at the metastore or catalog level. New tables are typically scanned within 24 hours, and you can trigger a full scan of enabled catalogs. Row filters and column masks then enforce access restrictions on the tagged data.

Unity Catalog is the unified governance layer for data and AI on Databricks, enforcing access control, tracking lineage, and logging activity. Securable objects are organized in a catalog.schema.object namespace, and row filters and column masks enforce policies at query time. Attribute-based policies and automatic data classification help you tag and protect sensitive data.

Databricks provides layered security controls: customer-managed keys for encryption at rest, encryption of data in transit between cluster nodes, and Unity Catalog for governing access to tables and AI assets. Network controls such as IP access lists and private connectivity restrict how users and workloads reach your environment, and audit logs in system tables record activity for review. The security documentation also covers compliance security profiles and a FedRAMP High deployment option on AWS GovCloud.

Yes. Databricks supports customer-managed keys, so you can use a key in your cloud provider's key management service to encrypt the data Databricks stores in its control plane (such as notebook source, SQL query history and secrets) and in your workspace storage. The feature requires the Enterprise tier, and on AWS the page covers both managed-services data and the workspace storage bucket. Keys are maintained in your cloud's key management service, which is where you manage their lifecycle.

Yes. Databricks supports private connectivity, including front-end PrivateLink for user access, back-end PrivateLink and customer-managed VPCs for classic compute, and IP access lists at the account and workspace level. For serverless compute, network policies restrict outbound connections, and private connectivity to resources in your own VPC is available through private endpoint rules. These controls are documented for AWS, so check the cloud-specific networking pages for Azure or GCP equivalents.

Databricks' certifications and attestation reports are published on its Trust Center rather than in the product documentation. The security documentation describes an Enhanced Security and Compliance add-on that includes a compliance security profile and enhanced security monitoring. Confirm which certifications cover your cloud, region and services before you deploy regulated workloads.

Yes. Databricks supports single sign-on through SAML 2.0 with your identity provider. With unified login, one account-level SSO configuration covers the account and all of its workspaces, and Databricks recommends enabling unified login everywhere. Workspace-level SSO is a legacy option that can only be used when unified login is disabled.

Yes. Account-level SCIM provisioning syncs users and groups from your identity provider into Databricks, so access is granted when people join or join a group, and removed when they leave. Removing a user from the account-level SCIM connector deactivates them across the account and all of their workspaces. It requires the Premium plan and account admin rights, and supports up to 10,000 users and service principals and 5,000 groups per account.

Databricks manages users, groups and service principals at the account level and assigns them to workspaces. Account admins control account identities and workspace access, and identity federation is enabled by default for new workspaces. Identities can be synced from an identity provider such as Microsoft Entra ID or Okta through automatic identity management or SCIM, and service principals give automated jobs and integrations their own non-human identity.

Lakehouse Monitoring profiles tables and tracks how their data changes over time. It computes profile metrics (summary statistics) and drift metrics, and for ML inference tables it also tracks model performance across time windows. Metrics are written to Delta tables in a Unity Catalog schema you choose, Databricks creates a dashboard for each monitor, and you can build SQL-based alerts on the metric tables.

Account admins manage users, service principals and groups across the account, assign them to workspaces, and control account-level settings such as SSO and SCIM provisioning. Workspace admins can grant workspace access and add users within the account, but they cannot update or delete account users or service principals. Identity federation is on by default for new workspaces, which keeps identities consistent across every workspace in the account.

Automatic data classification in Unity Catalog uses an agent to find and tag sensitive data in tables, with classification types including PII, PCI DSS, GDPR, HIPAA, GLBA, DPDPA and PIPEDA. You turn it on per catalog, or at the metastore level with catalog settings overriding it. New tables and columns in a catalog are typically scanned within 24 hours of creation, and you can trigger a full scan manually. Scanning Unity Catalog views is in Beta.

Databricks serverless compute runs in the Databricks-managed serverless compute plane. Connectivity between the control plane and the serverless compute plane is always over the cloud network backbone, not the public internet. Serverless network connectivity is managed with network connectivity configurations (NCCs). Features using Databricks private connectivity are in private preview; contact your Databricks account team for access.

Unity Catalog managed tables are the default and recommended table type for Delta Lake and Apache Iceberg. Unity Catalog handles reads, writes, storage and optimization, and the tables remain accessible to external clients through open APIs. Path-based access to managed tables is not supported, so reads and writes use the catalog.schema.table name.

Enable Predictive Optimization, which runs OPTIMIZE, VACUUM and ANALYZE automatically on Unity Catalog managed tables, and use Liquid Clustering with CLUSTER BY AUTO instead of manual partitions. The merge versus replace_where, materialized view and query-tag advice in the source blog is not verified in the docs.

AI Runtime is Databricks' fully managed serverless GPU compute offering, with on-demand A10 and H100 GPUs for training and fine-tuning models. Data loaders can stream data from Unity Catalog volumes. The docs do not state a GA or Beta status, so none is asserted.

Predictive optimization automatically decides when to run OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables (Delta Lake or Iceberg); it does not run on external tables. It is enabled by default for accounts created on or after November 11, 2024, and existing accounts are being enabled through a gradual rollout. The docs do not publish performance or storage-reduction figures, so measure the effect on your own tables.

With automatic liquid clustering, predictive optimization may select new clustering keys for Unity Catalog managed tables before it clusters the data. You enable it with CLUSTER BY AUTO. For Delta Lake tables it requires Databricks Runtime 15.4 LTS or above, and for managed Iceberg v3 tables it requires Runtime 18.0 or above. Automatic clustering is not available for Iceberg v2 tables. The docs do not state a release stage for automatic clustering.

Liquid clustering is generally available for Delta Lake tables on Databricks Runtime 15.4 LTS and above. It is a data-layout technique that replaces table partitioning and ZORDER, and you can specify up to four clustering keys. The docs note that on smaller tables (under 10 TB) more keys can slow single-column filtering. The blog's benchmark multipliers and row-level concurrency claims are not in the docs.

Bronze holds raw, unvalidated data in its original format and retains history for reprocessing and auditing. Silver holds validated, cleaned and enriched data, where deduplication and normalization happen. Gold holds aggregated, filtered data modeled for reporting and analytics, often dimensionally and per department or project.

MLflow 3 on Databricks covers GenAI tracing, expert and developer feedback, and evaluation and monitoring in one platform. MLflow on Databricks is the same open source product, and traces and evaluations are stored in Unity Catalog.

Unity Catalog's access mode guidance recommends standard access mode for classic compute unless the workload depends on a listed limitation, in which case dedicated compute is the option. The fetched access-mode page does not describe row-filter or column-mask support, Auto access mode or MERGE INTO status, so those claims are unverified.

AI Gateway (Unity Gateway) service policies control the content of model and agent requests and responses based on the content and on the caller, with built-in, custom and partner policies supported. Access to AI assets is governed by Unity Catalog privileges, and The docs do not give attack-success rates or describe deny-by-default egress for serverless workloads in this context.

Unity Catalog managed tables are the default and recommended table type for Delta Lake and Apache Iceberg. Unity Catalog handles storage and optimization, and compared with external and foreign tables they cost less to store and query and maintain themselves. The docs do not publish the 50% cost-savings or 20x query figures.

For tables on Databricks, predictive optimization runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables in the background, and when automatic liquid clustering is enabled it can select clustering keys before clustering data.

Table update triggers in Lakeflow Jobs start a job run when monitored tables are updated, merged or deleted, so downstream work starts when inputs change rather than on a fixed schedule. Supported sources include Unity Catalog Delta and Iceberg managed tables, external Delta tables, materialized views and streaming tables. Advanced settings are a minimum time between triggers and a wait after the last change. The docs do not state a GA, Beta or Preview status.

Data profiling, formerly known as Lakehouse Monitoring, profiles tables and tracks integrity over time, such as the fraction of null or zero values and value distributions and drift. It can also monitor inference tables for model input and prediction shifts and model performance trends. Anomaly detection checks freshness and completeness. The docs page does not state a release status and does not name accuracy, F1, precision or recall metrics.

Lakeflow Jobs table update triggers start a job run when monitored Unity Catalog tables change, so you do not need a continuously running cluster to watch for updates. Triggers support Unity Catalog managed Delta and Iceberg tables, external Delta tables, materialized views and streaming tables, and they add no charges beyond cloud provider costs for listing and reading updates.

For point-in-time lookups on time-series feature tables with Photon enabled, the docs recommend use_spark_native_join=True. This requires databricks-feature-engineering 0.6.0 or above. Time-series feature tables need timeseries_columns (Unity Catalog) or timestamp_keys, and liquid clustering on these tables improves point-in-time lookup performance.

Attribute-based access control (ABAC) lets you define row filter and column mask policies on governed tags at the catalog or schema level, and the policies apply to current and future tables and views beneath that scope. Row filters and column masks are generally available as of May 2026, while metastore-level policies, view policies and DENY policies remain in Beta.

Unity Catalog tracks lineage automatically from source data through to models, services and dashboards, and it records activity in the audit log system table. Governed assets use a three-level catalog.schema.object namespace.

Discover is a curated browsing experience in Unity Catalog where users browse assets by business domain and subdomain. The Discover page and domains became generally available on October 8, 2026, per the Databricks October 2026 release notes.

Unity Catalog does not govern direct reads of cloud storage by external engines. The external-access docs say external systems need additional controls to stay governed: either short-lived credentials vended by Unity Catalog, which inherit the requesting principal's privileges, or cloud-native IAM policies, bucket policies or ACLs that match the Unity Catalog privileges. Cross-engine ABAC is not documented on the pages checked.

Iceberg clients, including Apache Spark, Apache Flink, Trino and Snowflake, can read, write and create Unity Catalog managed tables through the Iceberg REST catalog. Creating and writing managed tables from Delta clients is in Public Preview.

The Lakebridge Analyzer's JSON output can be uploaded to Databricks' agentic code converter to review code complexity and lineage.

Multi-statement, multi-table transactions coordinate operations across several SQL statements and tables. BEGIN ATOMIC ... END commits automatically on success and rolls back on error, and interactive sessions use BEGIN TRANSACTION, COMMIT and ROLLBACK. Write targets are Unity Catalog managed Delta or Iceberg tables with catalog commits enabled. The docs page does not state a release stage.

Unity Catalog supports managed Apache Iceberg tables. Iceberg clients including Apache Spark, Apache Flink, Trino and Snowflake can read, write and create managed tables through the Iceberg REST catalog. Creating and writing managed tables from Delta clients is in Public Preview.

UniForm lets Iceberg clients read Delta tables read-only, with Iceberg metadata generated alongside Delta metadata so one copy of the Parquet files serves both formats. Iceberg reads are available from Databricks Runtime 14.3 LTS. OpenSharing recipients can read these tables as Iceberg tables through the Iceberg REST Catalog API, which is in Public Preview. Cross-engine ABAC status is not stated on this page.

Anomaly detection in data quality monitoring checks table freshness and completeness using per-table learned patterns: freshness flags an unusually late commit relative to a predicted next-commit time, and completeness flags a row count below the predicted range for the last 24 hours. The docs page does not describe root-cause tracing to upstream jobs, lineage-based prioritization, percent-null or uniqueness checks, and it states no Public Preview label.

System tables include system.lakeflow.jobs, system.lakeflow.job_run_timeline and system.lakeflow.pipeline_update_timeline, which track jobs, job run start and end times, and pipeline update start and end times. Billing, query history and lineage system tables are also available. Audit logs are labeled Public Preview. The dashboard template and the specific SQL recipes in the blog are not on this page.

Unity Catalog lineage and the system tables can help trace upstream errors and downstream impact of pipeline issues. Lineage is captured for Lakeflow pipelines in most cases, with column lineage requiring Databricks Runtime 13.3 LTS or above, and pipelines that use PRIVATE tables have incomplete coverage.

Foundation Model APIs offer Databricks-hosted foundation models on pay-per-token pricing, on-demand provisioned throughput with no term commitment, and reserved provisioned throughput for a fixed term. External models are provider-hosted models served through Unity Gateway, and access is controlled with Unity Catalog privileges such as MANAGE and query permission.

Unity Gateway governs AI assets in three ways: permissions, through Unity Catalog privileges on model services, external model providers, MCP tools, custom tools and HTTP connections; consumption and cost controls, through rate limits, budgets with per-user thresholds and hard caps, and usage tracking via system tables; and guardrails, through service policies that apply to requests and responses. Request and response payloads can be logged to Unity Catalog Delta tables.

Unity Catalog does not govern direct cloud-storage access by external engines. External systems need additional controls: short-lived credentials vended by Unity Catalog that inherit the requesting principal's privileges, or cloud-native IAM policies and ACLs that match the Unity Catalog privileges. The docs do not describe an Iceberg REST scan-planning API that returns filtered scan plans.

Service policies for AI securables govern the content of interactions with AI services registered in Unity Catalog: input is evaluated before Databricks invokes the service (ON CALL) and output after the service responds (ON RESULT). Service policies are in Beta.

Yes. Databricks made transactions generally available on October 14, 2026 for Unity Catalog managed Delta Lake and Apache Iceberg tables, so you can run multiple SQL statements across tables as one atomic transaction using BEGIN ATOMIC ... END or BEGIN TRANSACTION ... COMMIT, with catalog commits enabled on each table.

Governance Hub is a Beta, account-level UI for monitoring governance across your Databricks account. It covers data (governed tags, classification, data quality monitoring, object access), AI usage and spend, and cost (spend trends and tagging coverage), and an account admin enables it from the Previews page in the account console.

Attribute-based access control (ABAC) policies for row filters and column masks are documented, but the ABAC docs do not label row filters or column masks with a release stage, so they should not be presented as GA on this basis. The docs label ABAC DENY policies, metastore-level ABAC policies and view policies as Beta.

The FILE type, which references files stored in Unity Catalog volumes from tables rather than storing binaries, is in Beta. Volumes govern non-tabular data through path-based access and are accessible under Unity Catalog governance.

External Delta clients can create and write Unity Catalog managed tables through the Unity REST API, which the docs label Public Preview. Named external Delta engines in the blog, such as Delta Flink or Starburst Trino, are not confirmed on the pages fetched.

Unity Gateway registers model services, external model providers, MCP tools, custom tools and HTTP connections as Unity Catalog securables, so access is granted and revoked with standard Unity Catalog privileges. Agent Bricks Document Intelligence GA status is not on any fetched docs page.

Apache Spark and DuckDB can create, read and write managed and external Delta tables through the Unity REST API, and Apache Flink is supported in Beta. Writes and creates on managed tables require catalog commits, and external data access must be enabled on the metastore.

Unity Gateway provides a governed route for model and agent requests, with permissions set through Unity Catalog privileges. Usage tracking writes requests, token usage and latency to system tables, and payload logging writes requests and responses to Unity Catalog Delta tables that can be queried with SQL.

Unity Gateway attributes Databricks cost to services, target models, principals and tags, and budgets set per-user thresholds and hard caps. Usage data is in system tables and payload logs in Unity Catalog Delta tables, so you can build your own breakdowns with SQL. The GatewayIQ companion app is not documented on the fetched pages.

Unity Gateway governs model services, external model providers, and Databricks-hosted models under Unity Catalog privileges, and it supports traffic splitting and fallbacks across them. The docs do not describe a single endpoint serving custom models plus external models in the exact form the community post states.

Row filters and column masks in Unity Catalog can restrict access to sensitive columns such as PHI and PII, and the docs label view policies and metastore-level ABAC policies as Beta. Cohort building and outreach programs are blog guidance and are not documented.

Unity Gateway service policies control the content of requests and responses based on content and on the caller. AI Gateway budgets set per-user thresholds and hard caps on spend. Domains, the business glossary and Governance Hub are not described on the fetched docs pages, so their status is unverified.

Unity Catalog governs AI assets and their access with Unity Catalog privileges, and Unity Gateway provides usage tracking through system tables, payload logging to Delta tables, and per-user budgets with thresholds and hard caps. The docs do not describe on-behalf-of token passing, masked-column behavior across agents, or LangGraph and CrewAI support.

Unity Catalog does not govern direct cloud-storage access by external engines, so external systems need either short-lived credentials vended by Unity Catalog, which inherit the requesting principal's privileges, or cloud-native IAM policies and ACLs that match the Unity Catalog privileges. The centralized-enforcement, delegated-read and catalog-label patterns from the blog are not documented on the pages fetched.

Unity Catalog managed tables are maintained by Unity Catalog, so you do not manage their files manually. Dropping a managed table can be recovered with UNDROP TABLE, with a default recovery period of 7 days. Path-based access to managed tables is not supported, so use catalog.schema.table names.

Unity Gateway's MCPs page lets you discover Databricks-provided MCP servers or register your own, and Databricks-provided and registered MCPs use Unity Catalog permissions to control access. The docs I checked do not describe audit, lineage or on-behalf-of authorization for MCPs, so those claims were dropped.

Unity Gateway governs external model providers with Unity Catalog privileges, applies guardrails and service policies per endpoint, and writes request logs with identity, token usage and latency to Delta tables. The docs do not state GPT-5.5 availability on Databricks, and automatic failover on rate limits is not confirmed on the page fetched.

Unity Catalog row filters and column masks restrict which rows and columns a user can see, and the docs label view policies and ABAC DENY policies as Beta. Random-UUID pseudonymization and the SQL view example are community guidance and are not documented as Databricks recommendations.

Unity Catalog privileges are granted per securable object, such as catalogs, schemas, tables, and functions, using GRANT and REVOKE SQL statements. Table privileges include SELECT and MODIFY, and EXECUTE applies to functions. A grant on a catalog or schema applies to the objects inside it, so granting MODIFY on a schema automatically covers all current and future tables in that schema. Inheritance from parent containers lets you manage access centrally across large catalogs.

You can create a catalog with Catalog Explorer, the SQL CREATE CATALOG command, the REST API, the Databricks CLI, or Terraform. Creating a catalog requires the CREATE CATALOG privilege on the metastore, and access to the catalog is controlled with grants such as USE CATALOG, CREATE SCHEMA, and BROWSE. Specifying a managed storage location is optional but strongly recommended, and it is required if the metastore has no metastore-level managed storage. Catalogs hold schemas and tables as a governed container.

Dynamic views apply row filtering and column masking per user by using CASE expressions with is_account_group_member(), which the page recommends for Unity Catalog data. A column such as email can show its real value only to members of an auditors group and a redacted placeholder to everyone else, and a CASE expression in the WHERE clause can limit rows for other groups. Databricks documents that this strategy has no negative impact on query performance. The page also advises avoiding is_member() with these views.

Lakeflow Jobs support scheduled triggers that run on a time-based schedule, table update triggers that run when source tables change, and file arrival triggers that run when new files land in a monitored Unity Catalog storage location. Continuous triggers keep a job running by starting a new run when one completes or fails, and manual runs can be started with Run now or from external orchestration tools. A model update trigger fires when a Unity Catalog model is created, a version is ready, or an alias is set, and the alias event is marked Beta. The Multiple Triggers setting, which allows up to five triggers per job, is also Beta.

Databricks external models let you register providers such as OpenAI (including Azure OpenAI), Anthropic, Cohere, Amazon Bedrock, and Google Cloud Vertex AI, or any OpenAI API-compatible endpoint through the custom provider option, and serve them behind a Model Serving endpoint. Applications call the endpoint with the OpenAI client, the REST API, or the MLflow Deployments SDK, and provider credentials are stored as Databricks secrets or encrypted values. AI Gateway can add rate limiting, usage tracking, and guardrails. Network connectivity configurations with AWS PrivateLink for external models are in Public Preview.

Databricks secret scopes hold sensitive values such as API keys, passwords, and tokens in an encrypted database managed by Databricks, and you read them in code with dbutils.secrets.get(scope, key). Databricks redacts literal secret values read through dbutils.secrets.get(), replacing them with [REDACTED] when displayed. The MANAGE permission on a scope lets users read, write, and grant permissions on it, and the Secret ACLs documentation describes the full set of permission levels. Scope names are readable by all workspace users, so keep the sensitive values in the secrets themselves.

The Databricks SDK for Python is installed by default on clusters running Databricks Runtime 13.3 LTS or above, so you can import it in notebooks without a separate install. It covers workspace, cluster, and job operations, along with Unity Catalog volume file operations, so scripts can automate those tasks from Python. On Databricks Runtime 12.2 LTS and below, install the SDK first. Verify that the authentication method you use has the permissions required by the operations you call.

Lakeflow pipelines record data lineage and data quality checks in the pipeline event log, which captures a comprehensive record of pipeline events. Streaming throughput is tracked as messages processed per micro-batch, and streaming metrics can be exported to external services for alerting or dashboarding. Reviewing lineage, quality checks, and throughput together gives you visibility into pipeline health from source to sink.

Workspace search finds objects such as notebooks, queries, and dashboards by name across everything you have access to. It also searches the text of notebook cells, though not non-notebook files, and results appear live as you type. In workspaces that use customer-managed keys for encryption, notebook and query contents are not included in search. Search helps teams locate reusable code and definitions without manual cataloging.

An external location pairs a cloud storage path, such as an s3:// URL, with a storage credential that grants access to it, and Unity Catalog uses it to govern access to that storage for external and managed tables. Creating an external location requires the CREATE EXTERNAL LOCATION privilege on the metastore and the storage credential, or the MANAGE privilege on the location. You can restrict an external location to specific workspaces with workspace bindings, and you can mark it read-only to block writes. Databricks recommends governing file access with volumes where possible.

You can associate a managed storage location with a metastore, catalog, or schema, and managed storage at a lower level overrides the one above it when managed tables or volumes are created. Data is placed in the schema location first, then the catalog location, then the metastore location. Standard catalogs and schemas must be contained within an external location. Databricks recommends catalog-level managed storage for logical data isolation.

Change Data Feed records row-level changes with a _change_type column (insert, update_preimage, update_postimage, delete) plus _commit_version and _commit_timestamp. Enable it on Delta tables with ALTER TABLE ... SET TBLPROPERTIES (delta.enableChangeDataFeed = true); on Databricks Runtime 19 and above, automatic CDF applies to Unity Catalog managed tables with row tracking and needs no property. Read changes with table_changes('<table>', <start_version>) in SQL, or with the readChangeFeed option in batch or streaming reads.

OPTIMIZE compacts small files into larger ones so queries read less data. Databricks recommends liquid clustering for data layout; ZORDER BY still works on tables that do not use liquid clustering, but it is optional and not the recommended approach. With Predictive Optimization enabled, Databricks runs OPTIMIZE automatically on Unity Catalog managed tables, so manual runs are usually unnecessary.

COPY INTO loads files from cloud object storage or Unity Catalog volumes into a Delta table, and it is idempotent: files it has already loaded are skipped on later runs, so retries do not duplicate data. It supports pattern and format options (for example CSV, JSON, Parquet, and Avro) plus copy options such as mergeSchema. For more scalable ingestion, Databricks recommends streaming tables.

In Unity Catalog you can declare one PRIMARY KEY per table, and primary key columns are implicitly NOT NULL. Databricks does not enforce primary key, foreign key, or unique constraints; the primary key is informational, and the optimizer uses it only when you specify RELY, which requires Photon-enabled compute. Primary keys are not supported in the hive_metastore catalog.

Run ANALYZE TABLE ... COMPUTE STATISTICS to collect table size and row count, and add FOR ALL COLUMNS or FOR COLUMNS col1, col2 to also collect column statistics for the optimizer. NOSCAN collects only the size in bytes without scanning the table. Predictive Optimization runs ANALYZE automatically on Unity Catalog managed tables, so manual runs are mainly needed for tables it does not cover.

Governed tags are account-level tags whose allowed values are defined by admins, so users can assign only values from that set (up to 500 allowed values per tag). Admins control who can create, edit, and assign governed tags, and system governed tags require the ASSIGN permission. Governed tags can be used as attributes in attribute-based access control policies, so access follows data classification.

Volumes give Unity Catalog governance over non-tabular files such as images, audio, video, PDFs, and JAR or Python wheel files. Create a managed volume, where Unity Catalog manages storage, or an external volume that registers an existing cloud storage path. Access files at /Volumes/<catalog>/<schema>/<volume>/<path> from Spark, SQL, and Python on Databricks Runtime 13.3 LTS or above or a SQL warehouse; external systems that read the cloud storage directly bypass Unity Catalog.

Azure Databricks guidance supports one account with a separate workspace per business unit or environment, one Unity Catalog metastore per region, and catalog-level segregation, and workspace-catalog binding restricts each catalog to approved workspaces.

system.billing.usage is centralized and routed to all regions, so one account-level query can compare usage across workspaces. Sum usage_quantity across records, including retraction and restatement records, and expect ORIGINAL records within about 12 hours.

Yes, external systems can access Unity Catalog volumes with short-lived credentials. Unity Catalog can vend temporary credentials that inherit the requesting principal's Unity Catalog privileges, so external tools reading volume files do not need a separate IAM setup for that access. Databricks-native tools should continue to use the volume path API.

Iceberg clients can read, write and create Unity Catalog managed tables through the Iceberg REST catalog, and the docs label writing and creating managed tables from Delta clients as Public Preview. The docs do not state that Unity Catalog serializes Delta commits for external writers, or the Beta status and per-engine write support described in the blog.

Lakehouse Federation's catalog federation provides read-only access to Hive metastore and AWS Glue tables from Unity Catalog, which suits incremental migration to Unity Catalog. Migrating a table to a managed Unity Catalog table is a separate step. The UCX workflow, SYNC and CLONE guidance from the community playbook are not on the pages fetched.

Unity Catalog governs tables, views, volumes, functions, and models in one layer, using a three-level catalog.schema.object namespace and per-object permissions. It supports row filters and column-level controls, and lineage and audit information is exposed through system tables. Because one catalog covers analytics and AI assets, teams manage access in one place instead of maintaining separate governance tools for each asset type.

Unity Catalog governs Delta and Iceberg tables used by Databricks workloads, and Delta tables can be read by any system that implements the open Delta protocol, so your data is not tied to one engine. UniForm lets Iceberg clients read Delta tables, and Lakehouse Federation brings external warehouse data under the same catalog. Engines that read the storage files directly bypass Unity Catalog, so route external readers through UC-governed access paths.

Unity Catalog enforces governance at query time for tables, volumes, functions, and models, and it is available on AWS, Azure, and GCP. It works with Databricks engines and with external engines that access tables through its Iceberg and open-format interfaces. Lineage and audit information is exposed through system tables, so governance and lineage live in one place.

Lakehouse Federation governs federated sources through Unity Catalog foreign catalogs, which apply table-level access controls, so users see only the tables they are authorized to query across Snowflake, BigQuery, Oracle, and other sources. Permissions are defined once in Unity Catalog instead of separately in each source. Row and column filters are available on Unity Catalog tables, and you should confirm which of them apply to a given federated source before you rely on them.

Unity Catalog provides one governance layer for data and AI assets, and the Governance Hub gives account-level visibility into governance health, AI usage, and cost. Permissions and policies are defined in Unity Catalog and apply to the objects in your metastores, so governance is managed from one control point. For sharing with other organizations, Delta Sharing works with recipients on any computing platform.

Databricks exposes audit logs and lineage information through system tables in the system catalog, so you can query lineage with SQL alongside the data it describes. Unity Catalog connects lineage to governance, so access policies and data assets share one metadata layer. Teams that want lineage without a separate catalog product can use these system tables directly.

Lakehouse Federation's catalog federation reads Hive metastore and AWS Glue tables from object storage, is read-only, and suits incremental migration. The ALTER TABLE SET MANAGED conversion and the 20x and 50% figures from the community post are not on the fetched pages.

Unity Catalog governs tables, views, volumes, functions, models and model services under a three-level catalog.schema.object namespace, tracks lineage automatically from source data through to models, services and dashboards, and records activity in the audit log system table. One catalog per domain is the community recommendation and is not documented as a Databricks requirement.

The Metrics tab for a compute resource in the Compute section shows Hardware metrics such as CPU utilization, container and JVM memory, and network bytes, plus Spark metrics such as task counts and shuffle bytes. The docs give no CPU percentage threshold, and serverless compute uses query insights instead.

Unity Catalog governs models as securables in a catalog.schema.model hierarchy, so they can be discovered in Catalog Explorer and managed with standard Unity Catalog privileges. Lineage is tracked automatically to models and services, and activity is recorded in the audit log.

Predictive optimization runs VACUUM on Unity Catalog managed tables, which deletes unreferenced data files to reduce storage costs. Predictive optimization applies to managed Delta and Iceberg tables. The docs do not state that files are removed from customer-owned buckets, nor the Zepto savings figure.

Yes. Dropped Unity Catalog managed tables can be restored with UNDROP TABLE. By default they are recoverable for 7 days after being dropped, and the recovery period can be set at the catalog or schema level, from 0 hours to 7 to 30 days, with the schema setting taking precedence. The configurable recovery period is in Public Preview.

Unity Catalog controls access with privileges, attribute-based policies, row and column filters, and workspace bindings, so you can apply least-privilege access across catalogs, schemas, and tables. Governed tags let you define and control the tags that classify and organize securable objects. Unity Catalog also automatically classifies and tags sensitive data in your catalog.

Unity Catalog data classification uses an agent to automatically classify and tag tables in your catalog, and it detects sensitive data such as PII once tagging is enabled. Compliance frameworks including PII, PCI DSS, GDPR, and HIPAA are covered by the supported governed tags. You can create an attribute-based access control policy that masks classified columns with a masking function registered in Unity Catalog. The User Access tab shows which users accessed columns with a given classification tag.

Databricks secret scopes hold credentials that you read in jobs, notebooks, and code with dbutils.secrets.get(). Databricks redacts literal secret values read through dbutils.secrets.get(), replacing them with [REDACTED] where possible, although workspace admins, secret creators, and permitted users can still view secret contents. Grant the MANAGE permission on a scope to let users read, write, and grant permissions on it, and see the Secret ACLs documentation for the full permission levels.

Databricks system tables are a Databricks-hosted store of operational account data in the system catalog. Query system.query.history to review queries run on SQL warehouses and serverless compute, and system.billing.usage to analyze billable usage across your account. Use system.access.table_lineage and system.access.column_lineage to see read and write events on Unity Catalog tables and columns, and system.access.audit for audit events.

Unity Catalog governance uses governed tags to define and control the tags that classify and organize securable objects, and automated classification can tag sensitive data such as PII across tables. Row filters and column masks restrict which rows and column values users can see. Audit logs track who accessed your data and what actions they took, which supports compliance and regulatory review.

Databricks supports private connectivity with AWS PrivateLink, which lets your corporate network reach Databricks workspaces privately. You register and manage VPC endpoints for private connectivity to Databricks services. Review the network security documentation to choose the private connectivity pattern that fits your architecture.

Manage users, groups, and service principals at the account and workspace level with SCIM provisioning, and configure workspace-level features, settings, and Databricks Previews for individual workspaces. The Governance Hub gives you a single account-level view of governance health, AI usage, and cost across your account.

Unity Catalog column masks are SQL user-defined functions that take a column value as input and return either the original value or a masked version. A mask can use other columns as inputs to vary its behavior based on multiple attributes. Row filters are also SQL UDFs, and they evaluate each row at query time, excluding rows where the function returns FALSE.

Unity Catalog records data access and system activity in the audit log system table, so you can query who accessed what with standard SQL. Lineage is tracked automatically across data, models, services, and dashboards, which shows how data flows from source systems through to reports. Sensitive-data classification tags PII, so access reviews can focus on the assets that carry the most risk.

Unity Catalog row filters and column masks restrict patient-level records and sensitive columns based on user attributes and role. Access activity is recorded in the audit log system table for review. Confirm HIPAA eligibility for your cloud and region in Databricks' compliance documentation before you process PHI.

Lakeflow Declarative Pipelines expectations are SQL Boolean constraints that validate each record as it flows through the pipeline. Each expectation can warn and keep violating rows, drop them before they are written, or fail the update, and metrics are tracked for every dataset. Expectations apply to streaming tables, materialized views, and temporary views.

Lakehouse Monitoring profiles tables over time, computing quality metrics across time windows and writing results to Delta metric tables for trend analysis. Pipeline expectations validate each record and track metrics for every dataset, so quality issues surface during processing. Teams can report these metrics against the quality targets they agree with stakeholders.

Lakebase supports Postgres row-level security: once RLS is enabled on a table, rows are inaccessible to users by default, and policies determine which rows each user can access. Lakebase Change Data Feed (Public Preview) captures Postgres changes into Unity Catalog Delta change-history tables.

Iceberg clients including Apache Spark, Apache Flink, Trino and Snowflake can read, write and create Unity Catalog managed tables through the Iceberg REST catalog. Comparing governance models and open APIs with Snowflake is left to the buyer.

Unity Catalog governs tables, views, volumes, functions, models, and model services and MCP services under a single three-level catalog.schema.object namespace, with lineage and audit logging for those assets.

Unity Catalog provides access control, lineage, and audit logging for governed tables, and its lineage extends to models, services, and dashboards. Unity Catalog is also available as an open-source implementation, which reduces dependence on one vendor's governance layer. Warehouse-specific governance is typically scoped to that vendor's platform, so consolidating permissions in one catalog simplifies auditing across data and AI.

Yes. Unity Catalog OSS is licensed under Apache 2.0 and includes an OpenAPI specification, a server, and clients. On Databricks, Unity Catalog is the managed governance layer, and the open project gives you an open standard for cataloging data and AI assets. Using the open-source version means your governance layer is not tied to a single closed product.

Databricks provides controls to encrypt data at rest and in transit, along with network controls that secure connections between Databricks and your environment, including private connectivity, customer-managed VPCs, and serverless egress control. The platform also supports a compliance security profile and links to a FedRAMP High guide for Databricks on AWS GovCloud. Security depends on how the environment is configured, so review the security best practices for your setup.

Yes. Managed services data in the Databricks control plane is encrypted at rest, and user queries and transformations are sent to clusters over an encrypted channel. Customer-managed keys on AWS can protect workspace storage, including the S3 bucket and optionally cluster EBS volumes, as well as managed services such as notebook sources, secrets, and SQL query history. One caveat: worker-to-worker traffic is not encrypted by default, and enabling it requires an init script that uses AES-128 over TLS 1.3.

Yes, on AWS. Databricks supports customer-managed keys, so you can use your own AWS KMS key to encrypt workspace storage such as the S3 bucket in your account and, optionally, cluster EBS volumes. The same keys can protect control-plane managed services, including notebook source files and results, secrets, Databricks SQL queries and query history, and personal access tokens or Git credentials.

Unity Catalog governs access with privileges on securable objects, attribute-based policies, row and column filters, and workspace bindings. Only identities you grant can access an object, and the audit log system table records activity so you can review who did what. Unity Catalog also provides automated column-level lineage for data and AI assets, which helps you see where sensitive data flows.

Yes, on AWS. Databricks offers Front-end PrivateLink so corporate networks reach workspaces privately, Back-end PrivateLink so classic compute can reach the control plane, and private endpoints for serverless compute reaching AWS resources in your VPC. You can also deploy classic compute in your own virtual network, use IP access lists, and apply serverless egress controls to restrict outbound connections.

Databricks provides encryption at rest and in transit, customer-managed keys on AWS, and network isolation through private connectivity and customer-managed VPCs. You can combine those with Unity Catalog privileges, row and column filters, and audit system tables to control who sees sensitive data. Review the certifications that apply to your cloud and region before you deploy regulated workloads.

Databricks' compliance certifications and reports are published on its Trust Center rather than in the product documentation, and the product docs describe an Enhanced Security and Compliance add-on with a compliance security profile. Ask your account team for the current SOC 2 report and due-diligence package as part of a security review.

Databricks' compliance certifications are published on its Trust Center rather than in the product documentation. The product docs describe an Enhanced Security and Compliance add-on, a compliance security profile, and a GovCloud deployment; confirm the certifications that apply to your cloud and region with your account team.

Use Unity Catalog privileges to grant access only to the catalogs, schemas, and tables each group needs. Add row and column filters and attribute-based policies for finer control, and use workspace bindings to control where catalog objects are accessible. The audit log system table records access and system activity, so you can verify that the controls work as intended.

It is manageable. Attribute-based policies take effect automatically on any securable object that has the targeted attributes, so you define a policy once rather than granting access table by table. Some ABAC features, including metastore-level policies and DENY policies, are in Beta, so check their release stage before depending on them in production.

Yes. Unity Catalog is the unified governance layer for data and AI built into Databricks, operating in your workspaces. It provides privileges, attribute-based policies, automated column-level lineage, and an audit log system table. That gives you one governance model rather than separate tools for data and AI assets.

Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level, and shows jobs and dashboards as downstream consumers. Lakeflow pipelines are captured in most cases. Be aware of the limits auditors may ask about: lineage before September 1, 2024 is not available, column lineage is not shown for path-referenced sources, and RDDs, global temp views, and column-level UDF lineage are not captured.

Unity Catalog is automatically enabled for all Databricks workspaces created after November 8, 2023, so newer workspaces need no separate setup to start. Workspaces created earlier must be upgraded. Once it is in place, Unity Catalog provides privileges, lineage, and audit logging from one governance layer.

Yes. Unity Catalog OSS is licensed under Apache 2.0 and includes an OpenAPI specification, a server, and clients. On Databricks it is the managed governance layer, and the open project gives you an open standard for cataloging data and AI assets.

No. Unity Catalog provides row-level security through row filters and fine-grained column protection through column masks. Each is a SQL function applied automatically at query time, so rows a user should not see are filtered out and sensitive column values are masked per user. You can attach them to individual tables with ALTER TABLE, or use attribute-based access control (ABAC) policies to apply one rule across a schema, catalog, or metastore.

Databricks supports attribute-based access control (ABAC) in Unity Catalog, where governed tags drive access decisions. A single ABAC policy can enforce row filters and column masks consistently across an entire metastore, catalog, or schema, so tagging data as PII can mask or restrict it everywhere. The docs mark some ABAC surfaces as Beta, including metastore-level policies, view policies, and DENY policies, so check their status before relying on them in production.

Unity Catalog captures data lineage automatically for queries run on Databricks, down to the column level, and shows connected notebooks, jobs, dashboards, and queries in a lineage graph. Once tables are registered in a Unity Catalog metastore and accessed through Spark DataFrame or Databricks SQL interfaces, lineage is captured without extra configuration. File paths also appear as lineage nodes, and column lineage for path-based access is recorded in the system.access.column_lineage system table.

Databricks recommends one Unity Catalog metastore per region, shared by all workspaces in that region, and using that single regional metastore for every development lifecycle scope, including dev, test, and prod. Catalogs are the primary unit of data isolation and can correspond to an environment, a team, a business unit, or a combination of these. When environments need strict separation, you can bind a catalog to specific workspaces so that only the assigned workspaces can access it.

Unity Catalog best practices treat catalogs as the primary unit of data isolation, and they can map to environments, teams, or business units. Schemas add a further layer of organization beneath catalogs, and a typical pattern is one schema per team, granting USE SCHEMA and CREATE TABLE only to that team along with USE CATALOG on the parent catalog. Because catalogs can be bound to specific workspaces, catalog boundaries can also align with workspace boundaries when isolation requirements match.

Unity Catalog best practices tie catalog boundaries to environment, team, or business-unit scope, so catalogs are usually organized around ownership and isolation needs. The medallion architecture describes Bronze (raw, unvalidated), Silver (validated, cleaned, and enriched), and Gold (refined, business-aligned) layers. Databricks' medallion example places those layers as schemas inside one catalog, such as ops.bronze, ops.silver, and ops.gold.

The medallion architecture organizes data into Bronze, Silver, and Gold layers that reflect increasing data quality. Bronze holds raw, unvalidated data, Silver holds validated, cleaned, and enriched data, and Gold holds refined, business-aligned datasets for analytics, dashboards, and ML. In Databricks' example, each layer is a separate schema in one catalog, such as ops.bronze, ops.silver, and ops.gold, and the page presents this as an example rather than a requirement.

You can create a Databricks-backed secret scope with the Databricks CLI (version 0.205 or later), the Secrets API, or the workspace UI. Scope names must be unique within a workspace, can contain alphanumeric characters, dashes, underscores, @, and periods, and cannot exceed 128 characters. Notebooks read secrets with dbutils.secrets.get, and secret values are displayed as [REDACTED].

Attribute-based access control (ABAC) decides access by evaluating attributes on securable objects, which are represented by governed tags. Policies use those tags in their conditions to identify the data they protect, and a policy takes effect automatically on any object that has the targeted attributes. ABAC supports row filter and column mask policies for row- and column-level security, along with GRANT policies. Metastore-level policies, policies on views, and DENY policies are labeled Beta in the documentation, and the page does not state an overall status for ABAC.

Volumes are Unity Catalog objects that govern non-tabular datasets, representing a logical volume of storage in a cloud object storage location. Managed volumes are the default choice for Databricks workloads, while external volumes suit non-tabular files that external systems also read or write. Files are reached through the same path in Spark, SQL, Python, and other tools: /Volumes/<catalog>/<schema>/<volume>/<path>/<file-name>.

A service principal is a specialized identity designed for automation and programmatic access, giving API-only access without relying on an individual user's credentials. Account admins can add service principals to the account and assign them admin roles, and workspace admins can add them to a workspace and assign the workspace admin role. The creator of a service principal automatically becomes its service principal manager, and users need the separate service principal user role to run jobs as it.

The hive_metastore catalog is the legacy Hive metastore, which appears as a top-level catalog in the three-level namespace and is a workspace-level object. Unity Catalog is account-level and provides built-in auditing, lineage, and access control, which Hive metastore tables do not get in full. Because Hive metastore permissions refer to workspace-local users and groups, Unity Catalog's account-level model is the better fit for governing data across workspaces.

Yes. Lakehouse Federation's catalog federation supports the AWS Glue metastore, alongside the external Hive metastore and the legacy Databricks Hive metastore. With catalog federation, Unity Catalog accesses the foreign tables directly in object storage, and the query runs only on Databricks compute. Access is governed through Unity Catalog.

Yes. Lakehouse Federation's catalog federation supports the external Hive metastore, and Unity Catalog accesses the foreign tables directly in object storage with the query running only on Databricks compute. Queries through query federation are read-only, and the exception is Hive metastore federation, where foreign tables are writeable. Access is governed through Unity Catalog.

External engines reach Unity Catalog tables through open APIs: Delta clients use the Unity REST API, and Iceberg clients use the Iceberg REST catalog, which supports read and write for managed Iceberg tables. Writing to managed tables from Delta clients through the Unity REST API is in Public Preview. Credential vending lets external clients inherit the privileges that apply to governed data, and the supported read and write modes depend on the table type.

A storage credential is an authentication and authorization mechanism for accessing data in your cloud tenant, such as an IAM role for S3 buckets. An external location combines a cloud storage path with a storage credential that authorizes access to that path. Privileges granted on the external location control which users and groups can reach the path, and privileges on the storage credential control who can create external locations.

Row filters and column masks are SQL user-defined functions attached to tables, and they are evaluated at query time: a row filter is added with ALTER TABLE ... SET ROW FILTER, and a column mask with ALTER TABLE ... ALTER COLUMN ... SET MASK. The function logic defines what each user sees. The documentation shows group-membership functions such as is_account_group_member() gating dynamic views, which are a separate mechanism that wraps base tables, and row filters and column masks are attached to tables rather than views.

Yes, Unity Catalog includes column masks as a built-in access control. A column mask is a SQL user-defined function that takes the column value as input and returns either the original value or a masked version, and each column can have one mask attached with ALTER TABLE ... ALTER COLUMN ... SET MASK. Row filters complement masks by restricting which rows a user can see, and both are evaluated at query time.

Call a Unity Catalog Python UDF by its three-part name, catalog.schema.function_name, and in PySpark wrap the call in expr(), for example df.withColumn('bmi', expr('my_catalog.my_schema.calculate_bmi(weight_kg, height_m)')). Python UDFs run on Databricks Runtime 13.3 LTS or above or on a serverless or pro SQL warehouse, and direct invocation is not supported on dedicated access mode for UDFs that use secrets.

Managed tables are the default and recommended table type, and they cost less to store and query than external tables while maintaining and optimizing themselves automatically. External tables keep their data in cloud storage you own, registered through an external location, and Unity Catalog manages only their metadata rather than the data's lifecycle, optimization, or layout. Dropping an external table removes the metadata but leaves the data files in place, which suits data that managed tables do not support, such as JSON or Avro, and direct access from non-Databricks clients.

The access mode limitations page says RDD APIs are not supported on standard access mode compute, and most Scala Spark Context functions and sqlContext are also unavailable there. The page does not describe an option to allowlist RDD usage on Unity Catalog compute. For workloads that depend on these features, it directs you to dedicated compute, and standard access mode remains the recommendation for classic compute otherwise.

A DEEP CLONE copies both the data and the metadata of the source table, including stream metadata, but not the Delta Lake table history. Deep and shallow clones are not supported on Unity Catalog tables that have row filters or column masks.

Databricks audit logs record activity across your account, including authentication, user and group management, clusters, jobs, and other services. You can access and query them with the audit log system table, system.access.audit, which is currently in Public Preview. Databricks highly discourages moving audit data outside the platform, so querying the system table in place is the recommended approach.

Yes. Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level. Column lineage is captured when sources and targets are referenced by table name, and Lakeflow pipeline column lineage requires Databricks Runtime 13.3 LTS or above. For UDFs, only table-level lineage is captured, and path-based references appear only in the system.access.column_lineage system table.

Managed tables are the default and recommended table type in Databricks for Delta Lake and Apache Iceberg. They cost less to store and query than external or foreign tables, and they maintain and optimize themselves automatically. Databricks recommends enabling predictive optimization for all managed tables to reduce storage and compute costs, and external clients can still reach them through open APIs.

Unity Catalog is the unified governance layer for data and AI in Databricks. It provides access control through privileges, attribute-based policies, row filters and column masks, and workspace bindings, along with lineage, auditing through the audit log system table, and automatic data classification. It also provides Catalog Explorer for discovering governed assets, data quality monitoring, and secure sharing across organizations and clouds through OpenSharing.

Databricks recommends granting privileges to groups rather than individual users, and avoiding direct grants to users whenever possible. Admin roles should be assigned to groups whenever possible, and ownership of most securable objects can be assigned to groups. Grants should follow least privilege, and direct MODIFY access to production tables is best reserved for service principals.

No. Shallow clones are supported for Unity Catalog managed tables on Databricks Runtime 13.3 LTS and above. Some constraints apply, such as CREATE OR REPLACE not being usable to overwrite an existing shallow clone in Unity Catalog, and streaming tables and materialized views not supporting CLONE. The clone page does not address migrating from DBFS specifically, so confirm the approach for your tables.

Every asset governed in Unity Catalog is a securable object, which means you can grant permissions on it to users, service principals, or groups. Tables, views, volumes, functions, and models sit in the three-level catalog.schema.object namespace. Objects such as storage credentials and shares sit directly under the metastore.

Yes. Databricks requires one Unity Catalog metastore for each region where your organization operates, and each regional metastore can be linked to any number of workspaces in that region. The metastore's region must match the region of your workspaces and of any storage bucket it uses.

The user who creates a metastore becomes its owner, known as the metastore admin, who can create top-level objects such as catalogs and manage access to objects. Databricks recommends transferring this role to a group. The page also links to instructions for assigning a metastore admin.

You must be a Databricks account admin, and your account must be on the Premium plan or above. Metastore-level root storage is optional, and if you want it you need the ability to create S3 buckets, IAM roles, IAM policies, and cross-account trust relationships in your AWS account. The metastore's region must match the region of your workspaces and storage bucket.

A standard catalog is the typical catalog and the primary unit for organizing data objects in Unity Catalog. A foreign catalog is used in Lakehouse Federation scenarios and mirrors a database in an external data system, so you can run read-only queries on that data from your Databricks workspace. Catalogs sit at the top of the three-level catalog.schema.table namespace.

Reading a table requires SELECT on the table, plus USE CATALOG on its parent catalog and USE SCHEMA on its parent schema. Privileges on a schema or catalog are inherited, so a SELECT granted on a schema also applies to all current and future tables and views in it. Both USE CATALOG and USE SCHEMA are required even when SELECT is granted on the table.

Yes. Because privileges are inherited, granting MODIFY on a schema automatically grants MODIFY on all current and future tables in that schema. The same behavior applies to catalog-level grants, so you can manage access at the container level rather than table by table.

On a table, SELECT (which also applies to views and materialized views) lets a user read from the object, and MODIFY lets a user insert, update, and delete data in it. Both privileges can be granted directly on the table or inherited from a grant on its schema or catalog. These table privileges use the same GRANT syntax as other securable objects.

Grant privileges with GRANT <privilege-type> ON <securable-type> <securable-name> TO <principal>, where the principal can be a user, service principal, or group. This standard SQL form works for any securable object, such as a table, schema, or volume. Specify the securable type, such as TABLE, SCHEMA, or VOLUME, to target the object.

The owner of an object, the owner of the catalog or schema that contains it, a user with the MANAGE privilege on the object, and a metastore admin can all grant privileges on it. Account admins can also grant privileges directly on a metastore. This gives you several delegated ways to manage access.

Use ALTER <securable-type> <securable-name> OWNER TO <principal>. The current owner, a metastore admin, the owner of the containing catalog or schema, or a user with MANAGE can transfer ownership. For views, functions, and models, only a metastore admin can transfer ownership to any principal, while other owners can transfer only to themselves or to a group they belong to.

Data files for managed tables are stored in the schema or catalog that contains the table. Managed tables are the default and recommended table type for Delta Lake and Apache Iceberg, and they maintain and optimize themselves automatically. Managed tables also offer automatic table optimization and faster query performance.

Dropped managed tables stay recoverable for 7 days by default, and UNDROP TABLE restores them within that window. After the window ends, an asynchronous purge deletes the underlying data files. The recovery period can be configured at the catalog or schema level, from 0 hours up to 30 days.

Databricks creates a Delta Lake table by default when you do not specify a format. To create an Apache Iceberg table instead, specify USING iceberg in the create statement. Managed tables are the default and recommended table type for Delta Lake and Apache Iceberg.

Dropping an external table removes its metadata from Unity Catalog, while the underlying data files remain in your cloud object storage. Because Unity Catalog does not manage the data lifecycle for external tables, you delete those files yourself if you want them removed. External tables store data in cloud storage you control, and Unity Catalog manages only their metadata.

The table's storage location must be an external location registered in Unity Catalog, and you need the CREATE EXTERNAL TABLE privilege on an external location that covers the table's path. You also need USE CATALOG on the parent catalog, and USE SCHEMA and CREATE TABLE on the parent schema. Automated setup creates both the storage credential and the external location for you.

An external location is a Unity Catalog securable object that combines a cloud storage path with a storage credential that authorizes access to that path. Privileges granted on the external location control which users and groups can access the path, so cloud storage access is managed through Unity Catalog grants. Privileges on the storage credential control who can create external locations.

A storage credential is an authentication and authorization mechanism for accessing data in your cloud tenant, and external locations use it to reach cloud storage paths. For AWS S3 buckets it is associated with an IAM role, and for Cloudflare R2 it is associated with an R2 API token. Granting or revoking access to a storage credential controls who can create external location objects.

Creating an external location requires the CREATE EXTERNAL LOCATION privilege on both the metastore and the referenced storage credential, or the MANAGE privilege on the external location. Metastore admins have CREATE EXTERNAL LOCATION on the metastore by default. Holding MANAGE on an existing external location is an alternative to those metastore-level privileges.

Writing files to an external location requires both WRITE FILES and READ FILES on that external location, so grant them together. A user with only WRITE FILES receives a PERMISSION_DENIED error when attempting to write. Granting WRITE FILES and READ FILES on the external location gives users the access they need to write files there.

Marking an external location as read-only blocks writes to files in that location, even where the IAM role or Unity Catalog grants would otherwise allow them. It also blocks inserting data into tables and creating managed tables, volumes, or catalogs from that location, which makes it a useful safeguard for a storage path. The change can take up to 60 minutes to apply across all compute.

Volumes are Unity Catalog objects that enable governance over non-tabular datasets, and they represent a logical volume of storage in a cloud object storage location. Volumes sit at the third level of the catalog.schema.volume namespace, and their files are accessed by path. Volumes can govern files in any format.

Files in a volume are addressed as /Volumes/<catalog>/<schema>/<volume>/<path>/<file-name>, and the same path works across Spark, SQL, Python, and other tools. Spark also accepts an optional dbfs:/ prefix. Access is controlled by Unity Catalog privileges on the volume, such as READ VOLUME.

Managed volumes are the default choice for Databricks workloads and the simplest option for Databricks-only workloads. External volumes suit mixed access, where external systems read or write the same non-tabular data files that Databricks uses. Both volume types are governed by Unity Catalog privileges.

Unity Catalog does not govern reads and writes made directly against cloud object storage by external systems, but the documentation describes two ways to extend control. Credential vending lets external engines request short-lived credentials that inherit their Unity Catalog privileges, and cloud-native IAM policies, bucket policies, or ACLs can be aligned with the volume's privileges. The volume comparison table notes that for external volumes, Unity Catalog governs access while external tools can use direct URIs.

Querying a view requires SELECT on the view, USE CATALOG on its parent catalog, and USE SCHEMA on its parent schema. On dedicated compute running Databricks Runtime 15.3 and below, Unity Catalog checks both the view owner's permissions and the view user's permissions on the underlying data. SELECT, USE CATALOG, and USE SCHEMA are required on all compute resources.

On SQL warehouses, standard compute, and dedicated compute on Databricks Runtime 15.4 LTS and above with fine-grained access control, Unity Catalog checks the view owner's permissions on the underlying data. Because the owner's permissions are the ones checked, view users do not need their own grants on the underlying tables to query the view. On dedicated compute with Runtime 15.3 and below, both the owner's and the user's permissions are checked instead.

Row filters and column masks are attached to tables rather than views. For view-level control, dynamic views are the separate mechanism that wraps base tables, and they are typically gated by group-membership functions such as is_account_group_member(). Define these controls on the base tables with ALTER TABLE statements.

A row filter is a SQL user-defined function that returns BOOLEAN and is evaluated for each row at query time, with rows returning FALSE excluded. Attach it to a table with ALTER TABLE ... SET ROW FILTER, for example ALTER TABLE employees SET ROW FILTER salary_filter ON (department). The page example defines salary_filter as a BOOLEAN function that takes a department value.

A column mask is a SQL user-defined function that takes the column value as input and returns either the original value or a masked version. Attach it with ALTER TABLE ... ALTER COLUMN ... SET MASK, and each column can have one mask. The page gives no complete mask example, so you write the masking logic in the UDF.

Tables with row filters or column masks keep their protection in query-time evaluation, and the documentation lists the operations that are not supported on them. Deep and shallow clones are not supported, time travel is not supported with table-level filters or masks, and the Iceberg and Unity REST APIs cannot access these tables. OpenSharing cannot share them directly, so share a dynamic view over the table instead, since views can be added to a share.

Dedicated compute can access these tables on Databricks Runtime 15.4 LTS and above, where fine-grained access runs on serverless compute, and writes are supported from Runtime 16.3. Runtimes below 12.2 LTS do not support these controls and fail securely, returning no data. Keep clusters on supported runtimes to use row filters and column masks.

Attribute-based access control (ABAC) decides access by evaluating attributes associated with securable objects, which are represented by governed tags. Policies take effect automatically on any object that carries the targeted attributes, so one policy can enforce consistent rules across many objects. Metastore-level ABAC policies are labeled Beta, and the page does not state an overall status for ABAC.

ABAC supports row filter and column mask policies for row- and column-level security on tables, along with GRANT policies. DENY policies, which deny the MANAGE ACCESS CONTROL privilege on supported securable types, are labeled Beta, and views support for ABAC is also labeled Beta. The ABAC page describes these policy types under governed tags.

Governed tags are the attributes that ABAC policy conditions use to identify which data a policy should protect. When a securable object has the attributes a policy targets, the policy takes effect on that object automatically. Tagging the data once lets one policy protect every object that carries the tag.

Tags are attributes made of keys and optional values, used to organize and categorize securable objects. They can be set on catalogs, schemas, tables, table columns, volumes, views, functions, registered models, model versions, secrets, and services, among others. Tags are assigned with ALTER statements, as shown in the next question.

On Databricks Runtime 13.3 and above, use ALTER TABLE ... ALTER COLUMN ... SET TAGS, for example ALTER TABLE abac.customers.profiles ALTER COLUMN SSN SET TAGS ('pii' = 'ssn'). Each column needs its own ALTER TABLE command, because one command cannot tag multiple columns. Tags can be set on any supported object type, including columns, views, and volumes.

You can tag an object if you own it, or if you have APPLY TAG on the object together with USE SCHEMA on the parent schema and USE CATALOG on the parent catalog. Adding a governed tag also requires the ASSIGN permission on that tag. A table owner can tag objects without needing APPLY TAG.

Data Classification uses an agent to automatically classify and tag tables in a catalog, so you can find sensitive data and apply governance controls to it. It scans incrementally, so new data is classified without manual setup. Parts of the feature, including view scanning and detection exclusions, are labeled Beta.

To enable Data Classification on a catalog, you must own the catalog or have the MANAGE privilege on it. Enabling automatic tagging on a catalog requires USE CATALOG and APPLY TAG on the catalog plus ASSIGN on the tag, and at the metastore level you must be a metastore admin with ASSIGN on the tag. Enablement is per catalog, with metastore-level automatic tagging as a separate option for metastore admins.

Unity Catalog captures table and column lineage automatically for queries run on Databricks, down to the column level. Lineage requires queries to use Spark DataFrame or Databricks SQL interfaces, such as notebooks or the SQL query editor. Users need BROWSE or SELECT on a table to explore its lineage.

Lineage shown in Catalog Explorer is retained indefinitely, and all lineage captured after September 1, 2024 is available there. The lineage system tables keep a rolling one-year window of data. Lineage events older than one year are removed from the system tables on a rolling basis.

Column lineage appears in Catalog Explorer when both the source and target are referenced by table name. When a source or target is referenced by path, column lineage is recorded in the system.access.column_lineage system table instead. Only table-level lineage is captured for UDFs, and RDDs and Spark SQL dataset checkpointing are not captured.

Unity Catalog captures lineage for streaming between Delta tables on Databricks Runtime 11.3 LTS or above. Column lineage for Lakeflow pipelines requires Databricks Runtime 13.3 LTS or above. Both requirements apply to lineage captured on Databricks.

System tables are a Databricks-hosted analytical store of your account's operational data, for historical observability. They live in the system catalog and are organized into schemas for areas such as access, billing, Lakeflow, and query history. They are governed by Unity Catalog.

Your workspace must be enabled for Unity Catalog, and the metastore must be on Privilege Model Version 1.0. Users who hold both the account admin and metastore admin roles can query system tables by default, and other users need an admin to grant USE CATALOG on the system catalog plus USE SCHEMA and SELECT on the system schemas. Access is granted through the standard Unity Catalog privilege model.

Audit events are available in the system.access.audit system table, which is in Public Preview. Most audit logs are available only in the region of the workspace where they were recorded, and account-level events record a workspace_id of 0. When a workspace is deleted, its audit events older than 14 days are removed from the table.

Use the audit log system table, which is in Public Preview, to access and query your account's audit logs. The audit log reference catalogs events for services such as authentication, user and group management, clusters, jobs, dashboards, Genie, and Lakebase. Databricks highly discourages moving this data outside the platform, so you can query it in place.

system.billing.usage records your account's billable usage, including SKU, cloud, usage time window, quantity and unit, and metadata about the workspace, compute, and identities involved. Usage data is centralized and routed to all regions, so you can view global usage from any region. The billing data is part of the system catalog's billing schema.

The lineage system tables record read and write events on Unity Catalog tables or paths. The table lineage table records table-level events, and the column lineage table records column-level sources and targets. Records are emitted only where lineage can be inferred, and the tables keep a rolling one-year window of data.

The compute system tables cover classic compute, including all-purpose, jobs, and Lakeflow pipeline compute, and the clusters table also covers pipeline maintenance compute. For SQL warehouse and serverless activity, use the query history system table, because the compute system tables do not contain records for serverless compute or SQL warehouses. The compute system tables page lists known limitations, so review them before relying on these tables for serverless reporting.

The Lakeflow system tables record job and pipeline activity in six tables, including jobs, job_tasks, job_run_timeline, job_task_run_timeline, pipelines, and pipeline_update_timeline. Each table has 365 days of free retention, and the latest record for each job, task, or pipeline is kept even when it is older than that. The jobs table is a slowly changing dimension, so the latest entry for each job is kept even when it is older than the retention window.

system.query.history records queries run on SQL warehouses, on serverless compute for notebooks and jobs, and on Lakeflow pipelines, across all workspaces in the account in the same region. By default only admins can read it, and statement text is redacted unless you are an account admin or a member of the databricks_pii_access group. Admins can share query history access by creating a dynamic view for each user or group, as the docs recommend.

OpenSharing is an open protocol developed by Databricks for secure data sharing with other organizations. It lets you share live data and AI assets across organizations and clouds, and shares are registered as securable objects in Unity Catalog. Recipients authenticate with a sharing identifier for Databricks-to-Databricks sharing, or with tokens for open sharing.

Shares are read-only: a share is a read-only collection of tables and table partitions, and recipients access shared data in read-only format. If a share is removed from the metastore, all of its recipients lose access to it. A share can include views, volumes, and AI models, not only tables.

Deleting a recipient removes that recipient's access to every share it could previously access. To cut off access to a share itself, remove the share from the metastore. Databricks-to-Databricks recipients access shared data through a secure connection managed by Databricks.

Create a Databricks-to-Databricks recipient in Catalog Explorer from OpenSharing, with the Databricks CLI, or with CREATE RECIPIENT <recipient-name> USING ID '<sharing-identifier>'. You need to be a metastore admin or have the CREATE RECIPIENT privilege, and the sharing identifier uses the format <cloud>:<region>:<uuid>. Use the entire sharing identifier string, which includes the cloud, region, and UUID.

No. Databricks-to-Databricks sharing uses a secure connection managed by Databricks, so the recipient does not need a token. Token-based authentication applies to open sharing recipients that do not have a Unity Catalog-enabled workspace.

A share can contain tables and table partitions, streaming tables, managed and foreign Iceberg tables, foreign Delta tables, views (including dynamic views), materialized views, metric views, volumes, Python UDFs, notebooks, AI models, and Genie Agents. Views, volumes, and AI models are all shareable alongside tables. Dynamic views are part of this list, which matters when you need to share filtered data.

Creating a share requires the CREATE SHARE privilege on the metastore where the data is registered, and you must use a workspace attached to that metastore. To add a table, you must be the share owner, have USE CATALOG and USE SCHEMA on its containing catalog and schema, and have SELECT on the table. Adding volumes and AI models has its own privilege requirements, covered in the next question.

To add a volume to a share, you must be the share owner, have USE CATALOG and USE SCHEMA on its containing catalog and schema, and hold READ VOLUME on the volume. To add an AI model, you need the same ownership and schema access, plus the EXECUTE privilege on the model. The share owner role is required for each of these additions.

Tables with table-level row filters or column masks cannot be shared directly through OpenSharing. Instead, share a dynamic view over the table, since dynamic views are among the objects that can be added to a share. Tables without table-level row filters or column masks can still be shared as regular tables.

Lakehouse Federation is the Databricks query federation platform, providing governed, read-only access to external data through Unity Catalog foreign catalogs. With query federation, queries are pushed down to the external source using JDBC. Lakehouse Federation supports both query federation and catalog federation.

Query federation pushes queries down to the foreign database over JDBC. Catalog federation instead lets Unity Catalog access foreign tables directly in object storage, and the query runs only on Databricks compute. Both approaches are part of Lakehouse Federation.

Query federation supports MySQL, PostgreSQL, Teradata, Oracle, Amazon Redshift, Salesforce Data 360, Snowflake, Microsoft SQL Server, Azure Synapse, Google BigQuery, and Databricks. Catalog federation supports the legacy Databricks Hive metastore, external Hive metastores, AWS Glue, Snowflake, Palantir Foundry, Salesforce Data 360, and Workday Data Connect. The lists differ by federation type, so check your source against the correct list.

Creating a connection requires being a metastore admin or having the CREATE CONNECTION privilege. Creating a foreign catalog requires CREATE CATALOG on the metastore, plus either ownership of the connection or the CREATE FOREIGN CATALOG privilege on it. In workspaces enabled for Unity Catalog automatically, workspace admins have CREATE CONNECTION by default.

Yes, queries through query federation foreign catalogs are read-only. The one exception is Hive metastore federation, where foreign tables are writeable. The documentation describes this as governed, read-only access through Unity Catalog foreign catalogs.

Databricks recommends storing connection credentials in Databricks secrets rather than in plaintext strings, and the documentation's examples reference a secret scope and key for the user and password. Creating a connection requires the CREATE CONNECTION privilege, as described in the previous question. Plaintext strings are discouraged for sensitive values like credentials.

A secret scope is a collection of secrets identified by a name. Notebooks read secrets with dbutils.secrets.get, and Databricks redacts the values so they display as [REDACTED]. Scope names are readable by all users in the workspace, so do not treat them as secret.

Use dbutils.secrets.get(scope = '<scope-name>', key = '<key-name>') in Python to read a secret. Databricks redacts values read this way, so they display as [REDACTED] in notebook output. The example uses the Python secrets utility.

Run databricks secrets create-scope <scope-name> with Databricks CLI version 0.205 or later, or create the scope through the Secrets API or the workspace UI at the createScope page. In the UI, you choose which principals get MANAGE permission on the scope. Verify the scope with the CLI list command after you create it.

Scope names must be unique within a workspace, can contain alphanumeric characters, dashes, underscores, @, and periods, and cannot exceed 128 characters. Names are case insensitive, and because scope names are non-sensitive, all users in the workspace can read them. Creating a scope in the UI asks you to choose which principals get MANAGE permission.

Databricks recommends standard access mode for classic compute, unless your workload depends on one of the documented limitations. Workloads that need features standard compute does not support should use dedicated compute instead. Dedicated compute is the alternative for workloads that need restricted features.

Standard compute supports most workloads, and the features it does not support include R, GPU-enabled compute, Databricks Runtime for ML, RDD APIs, and socket sources for streaming. For these features, use dedicated compute, and for Hive UDFs, use Unity Catalog UDFs, which the documentation recommends. Several limitations depend on the runtime, so check the runtime for your workload.

A service principal is a specialized identity designed for automation and programmatic access, which gives API-only access without relying on individual user credentials. You can grant it access to data in Unity Catalog in the same way you would grant access to a user. Service principal managers can manage roles on a service principal, and its creator becomes its manager.

Running jobs under a service principal's identity prevents jobs from failing if a user leaves the organization or a group is modified. Users need the service principal user role to run jobs as a service principal. Service principal users, not managers, can run jobs as the service principal.

Use account-level groups to grant access to Unity Catalog data. Workspace-local groups cannot be granted Unity Catalog data access or assigned to other workspaces, so Databricks recommends converting them to account groups. Unity Catalog best practices recommend using groups to assign ownership and access.

To create a SQL or Python UDF, users need USAGE and CREATE permission on the schema and USAGE permission on the catalog. To run a UDF, users need EXECUTE permission on the function, along with usage on its schema and catalog. Python UDF runtime requirements are covered in the next question.

Unity Catalog functions are schema-level objects named catalog.schema.function_name. Python UDFs run on a cluster with Databricks Runtime 13.3 LTS or above, or on a serverless or pro SQL warehouse. From PySpark, call the function with expr() and its qualified name.

When you pass a registered model to MLflow APIs, use the three-level name <catalog>.<schema>.<model>. To load a model for inference, you need EXECUTE on the registered model, plus USE SCHEMA and USE CATALOG on its enclosing schema and catalog. The model page uses an example such as prod.ml_team.iris_model.

Unity Catalog implements the Iceberg REST catalog API, so Iceberg engines can connect to it as a standard catalog through the endpoint /api/2.1/unity-catalog/iceberg-rest. External data access must be enabled on the metastore. Apache Spark, Apache Flink, Trino, and Snowflake are named as supported Iceberg clients.

The integration principal needs the EXTERNAL USE SCHEMA privilege on the schema that contains the tables. The page also requires that external data access be enabled on the metastore. The REST catalog endpoint is /api/2.1/unity-catalog/iceberg-rest.

The Unity Catalog REST catalog supports read and write for managed Iceberg tables. Foreign Iceberg tables, managed Delta tables with Iceberg reads enabled, and external Delta tables with Iceberg reads enabled are read-only through the API. Use managed Iceberg tables when external engines need to write, since they are the only read-write type listed.

Data shared with a Databricks-to-Databricks recipient can be accessed only on the metastore where the share is registered. Recipients without a Unity Catalog-enabled workspace use open sharing with token-based authentication instead. Databricks-to-Databricks sharing does not need a token for the recipient.

Binding a catalog to specific workspaces limits its access to the assigned workspaces, which is useful for strict environment separation. Any workspace not on the assigned list returns an error when users try to access the catalog, and this overrides even explicit privilege grants. Use bindings alongside catalogs organized by environment, team, or business unit, as the best practices describe.

In a workspace the catalog is not assigned to, the catalog appears grayed out for metastore admins and catalog owners, and its child objects are not visible or queryable there. Because binding overrides individual grants, a user with SELECT still cannot query the catalog from an unassigned workspace. For other users, Catalog Explorer and information schema queries show only catalogs assigned to the current workspace.

ABAC policies use governed tags and attach row filters and column masks at the metastore (Beta), catalog, schema, or table level of the Unity Catalog hierarchy. A single policy can enforce consistent access rules across an entire metastore, catalog, or schema. Per-table row filters and column masks are instead set directly on one table or column with ALTER TABLE.

Row filters and column masks are SQL user-defined functions that evaluate at query time, and they are attached with ALTER TABLE ... SET ROW FILTER or ALTER TABLE ... ALTER COLUMN ... SET MASK. They cannot be applied to views, and time travel and deep or shallow clones are not supported on tables that use them. Databricks Runtime versions below 12.2 LTS return no data rather than exposing it.

When ABAC policies are evaluated, tags applied at one level of the Unity Catalog object model apply automatically to all objects beneath it. Tags do not inherit down to the column level, and this inheritance applies only during ABAC evaluation. Tags are key-value pairs set with ALTER ... SET TAGS, and tag keys are case sensitive.

Querying a table requires SELECT on the table plus USE CATALOG on the parent catalog and USE SCHEMA on the parent schema, and without USE SCHEMA the user cannot access the table. Privileges granted on a catalog or schema are inherited by child objects, so you can grant SELECT on all current and future tables in a schema. Automatic inheritance requires Privilege Model version 1.0 on the metastore.

Volumes are Unity Catalog objects for governing non-tabular files. Managed volumes use Databricks-managed storage for the schema, while external volumes add governance to existing cloud object storage locations and keep their data in cloud storage if the volume is dropped. Files in either type are addressed with the path /Volumes/<catalog>/<schema>/<volume>/<path>, which works across Spark, SQL, and Python.

You need an external location, which combines a cloud storage path with a storage credential that authorizes access to that path. The path you reference must be included in the external location, either as its exact path or a subpath of it. Multiple external locations can share one storage credential, and permission to create and use external locations should go only to users who need to create external tables, external volumes, or managed storage locations.

Yes, Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level, and you can view it in the Lineage tab of a table in Catalog Explorer. The lineage system tables let you query lineage programmatically. Coverage has limits: pipeline capture happens in most cases, column lineage for pipelines requires Databricks Runtime 13.3 LTS or above, and UDF lineage is captured at the table level, not the column level.

System tables live in the system catalog and require a Unity Catalog-enabled workspace on a metastore using Privilege Model version 1.0. Users need USE CATALOG on system, plus USE SCHEMA and SELECT on the system schemas they query, and users who hold both the account admin and metastore admin roles have access by default. Tables such as system.access.audit, system.billing.usage, and system.access.table_lineage cover audit logs, billable usage, and lineage.

The catalog created for each new workspace is bound only to that workspace by default, so other workspaces attached to the same metastore cannot use it. If you unbind it or extend access, you must grant permissions manually, because workspace admins are a workspace-local group. Bindings can be set in Catalog Explorer or with the Databricks CLI.

Yes. Besides catalogs, Unity Catalog lets you bind external locations, storage credentials, and service credentials to specific workspaces. A catalog binding can also be set to read-only, which blocks writes from that workspace. To define bindings you need metastore admin rights, catalog ownership, or MANAGE on the catalog.

Governed tags are enforced at the account level and apply across all workspaces in the account. Each account can create up to 1,000 governed tags, and each governed tag can have up to 500 allowed values. Each governed tag has a tag policy that defines its allowed values, so only those values can be assigned.

Use the information schema. system.information_schema covers objects across all catalogs in the metastore, excluding hive_metastore, and each catalog also has its own information_schema. Results are filtered automatically to the objects you can access through Unity Catalog privileges, and the information schema does not require an explicit SELECT grant.

Quotas are set per scope. A metastore can hold 10,000 policies across all objects, with up to 100 attached directly at the metastore (Beta). Each catalog and each schema accepts up to 100 policies, and each table up to 50. A policy's TO and EXCEPT clauses can list up to 20 principals. One catalog-level policy can cover many tables instead of writing a filter for each one.

Yes. An ABAC policy's EXCEPT clause names principals that are exempt from the policy's filtering and masking. Exempt principals are not subject to the policy and see the full, unmodified data, while every other user stays governed by it. Name only the groups or service principals that need that access, so each exception stays explicit and easy to audit.

Databricks blocks access and returns an error instead of choosing a filter. Only one distinct row filter can resolve for a given table and user, and only one distinct column mask per column. Multiple policies are allowed when they resolve to the same filter or mask function with the same arguments. Reviewing policy scope and EXCEPT lists before rollout keeps access predictable.

Run SHOW POLICIES on the securable object. Adding EFFECTIVE also lists policies inherited from parent catalogs or schemas. DESCRIBE POLICY shows one policy's principals, match conditions, and function. Users who hold MANAGE on an object can query the ABAC_POLICY_DEFINITIONS information schema view. Policy create, delete, get, and list operations are also recorded in the audit log system table.

Yes, but it is in Beta. Applying ABAC policies to views requires a workspace admin to enable the ABAC on Views preview and Databricks Runtime 19 or above. Metric views are not supported, a query can layer at most 20 policies, and policies cannot be applied to a shared view on the recipient side. Check the requirements page for the current limits before relying on it in production.

Yes. Replacing a view with CREATE OR REPLACE clears its metadata, which removes its governed tags and directly attached policies, so they must be reapplied. ALTER VIEW preserves them. Build a reapplication step into view deployments so protection is not silently dropped after a rewrite.

Databricks implicitly casts the column to the function's parameter type. With ANSI mode disabled, values that fail to cast silently become NULL, which can produce incorrect filtering or masking. Enabling ANSI mode makes those failures visible as errors. A column mask's return type must match or be castable to the column's type, so align UDF parameter types with the column types.

Use simple SQL UDFs built from expressions that cannot throw errors, such as try_divide, and keep the number of distinct column masks low on large tables, because each distinct mask is evaluated during queries. Databricks notes that row filters and masks prevent users from seeing base values before filtering, so the secure choice can limit some optimizations. These guidelines reduce that performance cost.

Catalogs, schemas, tables, table columns, volumes, views, functions, registered models, model versions, secrets, external metadata objects, and services can carry tags. Tags on a catalog or schema do not generally pass to child objects. Implicit inheritance applies only when ABAC policies are evaluated. Tags are applied with ALTER statements using SET TAGS, and with SET TAG ON on Databricks Runtime 16.1 or above.

After automatic classification is enabled for a catalog, new tables and columns are typically scanned within 24 hours of creation. Scanning is incremental and runs as a background job. Tags are not backfilled immediately when automatic tagging is turned on, since they populate on the next scan. A manual full scan is available to re-evaluate existing tables when needed.

Not by default. Classification tags are not applied to view columns by default, and metric views are not scanned. Enabling view scanning, which is in Beta, increases cost because views are scanned along with tables. Views with unsafe definitions, such as those using reflect(), are skipped, so teams that need view-level tags should plan that scope explicitly.

By default, only account admins hold MANAGE and ASSIGN permissions on the data classification system governed tags, and account admins can grant them per governed tag to users, service principals, or groups. Sample values require SELECT on the results system table, which is accessible only to account admins by default, so grant that access deliberately to a small governance group.

Yes, but at the table level. UDF lineage is tracked to the table, not to individual columns, and it appears on the function's Lineage tab in Catalog Explorer. Column-level UDF lineage is not captured, and column lineage appears only when both source and target are referenced by table name. Teams can use UDF lineage to see which tables a function touches during impact analysis.

The column lineage table only records events that have a source column, so inserting literal values is not captured there. Lineage tables also represent a subset of all read and write events because lineage cannot always be inferred. Entities outside Lakeflow pipelines, notebooks, jobs, Databricks SQL, dashboards, Genie Agents, and alerts are not supported.

Retention varies by table. Many operational tables keep 365 days for free, data classification and data quality monitoring keep 13 months, and pricing, node types, and workspace metadata are kept indefinitely. Account admins can use a Beta setting to choose a retention period from 30 to 3,650 days, and when it is enabled the free period becomes 395 days.

For some events, SQL-definition keys in request_params are omitted unless you are an account admin or a member of the databricks_pii_access group. The affected keys are function_info, view_definition, definition_json, and managed_definition. Audit events are available in the audit log system table system.access.audit, which is in Public Preview, so governance teams can review access centrally.

Yes. Granting SELECT on a schema applies it to all current and future tables and views in that schema, and the same inheritance works at the catalog level. Databricks recommends granting CREATE TABLE at the schema level. Because privileges flow down the hierarchy, a single grant on a team's schema covers the team's new tables without per-table grants.

Run SHOW GRANTS on the object, or use Catalog Explorer. Users with MANAGE, the object owner, metastore admins, and users with READ METADATA can see all grants on an object, while other users see only their own. INFORMATION_SCHEMA currently shows only a user's own grants, so SHOW GRANTS gives the complete picture.

BROWSE on a catalog lets users discover and view metadata without USE CATALOG or USE SCHEMA. It supports data discovery without granting access to the data itself, so people can find assets and request access through the governed process that Unity Catalog provides.

Groups. Databricks advises assigning ownership of production catalogs and schemas to groups rather than individual users, and using groups to assign ownership of most securable objects. Provision principals at the account level, through automatic identity management or SCIM, and manage membership in your identity provider. Avoid direct grants to users wherever possible.

No. Files in volumes cannot be registered as Unity Catalog tables, because volumes support path-based access only. Tables cover tabular data, while volumes govern files in any format. Files are addressed with /Volumes/catalog/schema/volume/path, which works the same way across Spark, SQL, Python, and other tools.

Volumes require Unity Catalog-enabled compute on Databricks Runtime 13.3 LTS or later. The runtime requirement has an exception when you use Databricks UIs such as Catalog Explorer. The catalog, schema, and volume segments of a /Volumes path are read-only and managed by Unity Catalog, so filesystem commands cannot create or delete them.

Yes. For tables, views, materialized views, and metric views, the share owner needs USE CATALOG and USE SCHEMA on the containing objects plus SELECT on the asset. Databricks states that the SELECT privilege must be kept for the asset to continue being shared. Volumes need READ VOLUME, and Python UDFs and AI models need EXECUTE, so review grants before sharing.

Yes. Users with permission can read any path within an external location without direct access to the underlying storage credential. Grant READ FILES or WRITE FILES on the location to the principals who need them. The creator becomes the initial owner, and the owner or holders of MANAGE can change its name, URI, or storage credential.

No. Location names must be unique within the metastore, and one external location's path cannot sit inside another's. Plan external location paths as separate, non-overlapping prefixes in cloud storage, so each location's access rules stay clear and easy to audit.

Yes. An external location pairs a cloud storage path with a storage credential, and it backs external tables, external volumes, and managed storage locations. Multiple external locations can share one storage credential. Granting or revoking access to an external location determines whether users can reach the data at that path.

No. External clients connecting through the Iceberg REST catalog endpoint cannot run table maintenance such as expiring snapshots or removing orphan files on managed Iceberg tables. Managed Iceberg tables still support reads and writes from external engines through the endpoint, and Unity Catalog blocks commits that would add duplicate data files to a snapshot.

Foreign Iceberg tables are not automatically refreshed when read through the Iceberg REST Catalog API. To pick up the latest snapshot, run REFRESH FOREIGN TABLE. Foreign Iceberg tables are read-only for external engines, and credential vending is not supported for them, so add a refresh step to any pipeline that reads them from outside Databricks.

Databricks strongly recommends against registering a table as an external table in more than one metastore, because the registrations can drift apart. For cross-region or cross-cloud sharing, use Databricks-to-Databricks OpenSharing instead, since lineage and access control do not cross region or platform boundaries.

Yes, and it is the exception among metastore-level privileges. Granted on the metastore, READ METADATA inherits down to all objects in the metastore, which other metastore-level privileges do not do. Because it flows to every object, grant it deliberately to a small set of governance or discovery principals.

Yes, where the connector supports it. Databricks Runtime tries to push supported predicates down to the remote database, so less data crosses the network. A predicate that cannot be pushed down is evaluated in Databricks, and when predicates are combined with AND, the supported ones still push down. Run EXPLAIN FORMATTED to see the query sent to the remote database.

External clients request short-lived, scoped storage credentials from Unity Catalog through credential vending, and the requesting principal needs EXTERNAL USE SCHEMA plus the table, catalog and schema privileges. Table credentials include an access token and a cloud storage location URL, and path and volume credentials are limited to their storage location.

Reading a table through external access also requires the EXTERNAL USE SCHEMA privilege on the schema or its parent catalog. That privilege must be granted explicitly, because ALL PRIVILEGES does not include it. The principal also needs USE CATALOG, USE SCHEMA, and SELECT. A metastore admin must turn on External data access for the metastore, which is disabled by default, from the Catalog pane's Metastore settings.

Results are filtered by the querying user's Unity Catalog privileges, so each user sees the objects they can access. Unlike other system tables, the information schema does not require an explicit SELECT grant. Each catalog has its own information_schema, while system.information_schema covers all catalogs in the metastore and excludes hive_metastore objects. Query it with standard SQL to inspect metadata.

Yes. Hive metastore federation lets Unity Catalog govern tables stored in a Hive metastore. The tables are exposed through a foreign catalog with Unity Catalog access controls, auditing, and lineage. For the built-in legacy workspace metastore, reads and writes are supported, and teams can migrate to Unity Catalog incrementally without code adaptation.

Catalog federation and query federation are read-only, so foreign tables cannot be written through Unity Catalog. Move tables to managed Unity Catalog tables when you need write access.

The docs on external access say Unity Catalog does not govern direct reads of cloud storage by external engines, and that external systems need vended credentials or cloud IAM policies to stay governed. The docs do not document ABAC enforcement for external engines in the pages checked.

In Beta, yes. A workspace admin must first turn on the Tag propagation preview. For supported operations, Unity Catalog uses lineage to identify source objects, so governed tags follow the lineage at both the table and column level into derived tables and views. Classification travels downstream without tagging each object by hand.

Yes. Unity Catalog secrets are securable objects under a schema and use the three-level namespace catalog.schema.secret. Because they are securable objects, Unity Catalog privileges govern access to them, and the system.access.audit system table records events related to them, so secret access is audited in the same system as other Unity Catalog assets.

Yes, in Beta. A metastore-level ABAC policy, which can be a row filter, column mask, GRANT, or DENY, applies to every catalog, schema, and object supported by its policy type. Catalogs created after the policy is defined are covered automatically when they match its conditions, so administrators set governance once for the metastore.

Lakehouse Monitoring writes a profile metrics table named output_schema.table_name_profile_metrics and a drift metrics table named output_schema.table_name_drift_metrics. In the profile table, count is the number of non-null values in a column, not the total row count. The same table also reports avg, min, max, and percent_null for each column.

Yes. Data profiling supports three custom metric types: aggregate metrics computed from source columns, derived metrics computed from existing aggregates, and drift metrics that compare windows or a baseline. You write each one as a SQL column expression in a Jinja template with placeholders such as {{input_column}}. Aggregate and derived results go to the profile metrics table, and drift results go to the drift metrics table.

Yes. Anomaly detection, in Public Preview, checks tables in a Unity Catalog schema against learned commit history. The freshness check marks a table stale when a commit is unusually late against its predicted commit pattern, and the completeness check marks a table incomplete when rows committed in the last 24 hours fall below the predicted range. Health indicators appear in Catalog Explorer and in the system.data_quality_monitoring.table_results system table.

In Catalog Explorer, a user opens the object, selects Request permissions, and chooses the principals and privileges in a dialog that includes a message field. Requests route to one or more configured access request destinations, and if several are configured, all of them receive the request. Destinations can be email, Slack, Microsoft Teams, or webhooks, and approvers must hold permission to grant access.

No, not for foreign Iceberg tables. Unity Catalog's Iceberg REST endpoint gives external engines read-only access to foreign Iceberg tables. Writes are supported only for managed Iceberg tables, so if external engines must write to a table, store it as a managed Iceberg table. Note that external clients cannot run maintenance operations on managed Iceberg tables, such as expiring snapshots or removing orphan files.

Use multipart upload with the Files API. Single-request uploads are capped at 5000 MiB, so larger files are sent in parts. The Files API also supports conditional requests, so reads, writes, and deletes can depend on a file's current ETag or Last-Modified time.

Use query federation first for supported databases such as Oracle, MySQL, PostgreSQL, SQL Server and Snowflake, and use a JDBC connection for unsupported databases, specific drivers, Spark writes, or finer Spark-level control. A JDBC connection is referenced by name and needs the USE CONNECTION privilege.

Yes. External engines can query Unity Catalog views over Delta Lake tables through the Unity REST API, and the docs list this access as generally available. The engine runs the view's Spark SQL against the underlying tables, so the reader needs USE CATALOG, USE SCHEMA, and SELECT on the objects, plus EXTERNAL USE SCHEMA on the schemas that hold the tables. Creating or altering views from external compute is not supported.

Yes, for managed Delta tables with catalog commits enabled. Clients need Databricks Runtime 16.4 or above for those tables, Spark 4.0 or 4.1 with Delta Spark 4.3.0 or later, and the Unity Catalog Spark client 0.5.0 or later. Writes need MODIFY on the table, and the metastore needs external data access enabled. OPTIMIZE, VACUUM, and ANALYZE are not supported from external clients on managed tables.

Use OAuth custom identity claims. The application adds a custom_claim to the service principal's token request, and a dynamic view or ABAC row filter compares a column to current_oauth_custom_identity_claim(). This lets one service principal serve many end users without a Databricks user for each. It requires a SQL warehouse and Databricks Runtime 19 or above, and filters should return no rows when the claim is NULL. Claims are not encrypted, so never put secrets in them.

Most regional tags work only in workspaces deployed in the matching region. The US tags are the exception and are available in all regions. Regional tags cover countries and regions from Australia and Brazil to the United Kingdom, so teams can classify sensitive data with the tag set that fits where their workspace runs.

Yes. Create an external metadata object for the asset, such as a Tableau dashboard, then link it to Unity Catalog tables, models, or paths in the lineage graph. You need CREATE EXTERNAL METADATA on the metastore and MODIFY on the external object. Lakeflow Connect managed ingestion also records source-to-destination lineage automatically. External lineage is not recorded in the system.access lineage tables.

Yes. Schema-level HTTP connections are generally available. An HTTP connection stores the endpoint and credentials for an external REST service, and using one requires USE CONNECTION on the connection. Metastore-level HTTP connections remain in Public Preview, and HTTP connections are available only in regions where Model Serving is supported.

Not necessarily. For existing external Hive tables, SYNC registers the data in Unity Catalog as external tables without copying it, and can be rerun to update them when the source tables change. Hive-managed Delta tables must be cloned with DEEP CLONE, and CREATE TABLE AS SELECT copies data into a managed storage location. The upgrade guide's options differ depending on whether the data is external or managed. Pilot one table end to end, validate it, and keep a rollback plan before moving the rest.

No, and it is important to plan for it. The migration guide states that table history is not migrated when you run CREATE TABLE CLONE, so you cannot perform Delta Lake time travel or other operations that rely on pre-migration history on the cloned Unity Catalog table. DEEP CLONE does give you an independent managed copy, which makes a clean cutover point. Keep the original Hive table until the clone is validated, and archive any history you need before cutover.

The migration guide says SYNC cannot copy Hive managed tables stored in workspace storage, sometimes called DBFS root. For those tables the documented path is CREATE TABLE CLONE, which copies the data into Unity Catalog. SYNC remains the right tool for external tables. Check each table's location and type in your inventory first, because that determines which migration command applies.

UCX is a Databricks Labs project, and Databricks documents it as provided as-is and not formally supported with service-level agreements, so it should not be treated as a supported product. It covers assessment to plan your migration, migration of workspace-local groups to account-level groups, and migration of tables registered in the Hive metastore to Unity Catalog. Use it to inventory and plan at scale. Issues with UCX go to its GitHub repository as a GitHub issue, not to a support ticket.

Yes, using Hive metastore federation, which lets Unity Catalog govern tables stored in a Hive metastore through a foreign catalog that Unity Catalog crawls. The page describes this as enabling incremental migration without code adaptation. Federated internal Databricks Hive metastores support reads and writes, while federated external Hive metastores and AWS Glue catalogs are read-only. Known limits include that you cannot query federated tables whose files are stored outside the federated table's location, and performance can be marginally worse than on Unity Catalog or the Hive metastore. Check the limitations for your case before standardizing on it.

Yes. The recommended approach in the workspace upgrade guide is to first federate the Hive metastore as a foreign catalog, then upgrade the foreign tables in place. Tables can also be upgraded directly with SYNC or CREATE TABLE AS SELECT. The MOVE variant of SET MANAGED converts a table in place and disables access to the source table in the external catalog, so read the eligibility notes before running it.

Unity Catalog is also available as an open-source implementation, according to the Databricks documentation. Test the exact features you need before relying on self-hosting as an exit plan.

Yes. Unity Catalog implements the Iceberg REST catalog API specification, and the Apache Iceberg REST catalog lets clients such as Apache Spark, Apache Flink, and Trino read from and write to Unity Catalog-registered Iceberg tables. Configure access with the endpoint /api/2.1/unity-catalog/iceberg-rest. Managed Iceberg tables are readable and writable through this interface, while Delta tables with Iceberg reads enabled are read-only. Setup requires External data access enabled for the metastore, the EXTERNAL USE SCHEMA privilege on the schema, and authentication with a personal access token or OAuth.

With catalog-managed commits, Unity Catalog is the single source of truth for Delta Lake and Apache Iceberg table state, which is what enables multi-table transactions. That is a real dependency, so plan for it. The docs say you can turn the feature off for an existing managed table on Databricks Runtime 18.0 or above, and warn not to cancel an upgrade or downgrade operation while it is executing. Review that page before committing a design that depends on catalog-managed commits.

Yes, for Unity Catalog tables. External engines can read governed tables with fine-grained access controls enforced, and Databricks uses a specialized serverless compute layer to filter and return sanitized data. Delta reads require Delta-Spark 4.1 or above with the Unity Catalog Spark connector 0.4 or above, and Iceberg reads require Iceberg-Spark 1.11 or above with Apache Spark 4.0 or above. Limits include reads only when fine-grained access controls are enforced (to write, the writing principal must be exempted from the ABAC policy), no dynamic views, and no projection of VARIANT columns. Customers are charged for this filtering, and users with access to system.billing.usage can see it, so include it in cost planning.

Unity Catalog managed tables store their data files in the managed storage location of the schema or catalog that contains the table, and Unity Catalog handles reads, writes, storage and optimization. The docs do not state what share of managed tables sits in customer-owned storage, so check each table's storage location before planning a move.