Databricks SQL, Dashboards & Analytics

Yes. Lakehouse Federation lets you query external systems such as Snowflake, Redshift, BigQuery, PostgreSQL, MySQL, Oracle, and SQL Server in place, with automatic query pushdown. Access is governed through Unity Catalog foreign catalogs, with table-level access controls applied to the external data. Because the data stays where it is, teams avoid building separate ETL pipelines just to run analysis across systems.

Use INSERT INTO with REPLACE WHERE on a Delta Lake table to delete the rows matching a predicate and insert the rows returned by your query. By default Databricks checks that every returned row satisfies the predicate, and if one does not, the insert fails and the table is not changed. Databricks also offers REPLACE USING, which atomically deletes table rows that match rows in your query, for upsert-style refreshes.

Yes, with limits. Unity Catalog Python UDFs can use service credentials to access external cloud services without hardcoding secrets in function code. Default credential support is available only on Standard and Dedicated access mode clusters, not on SQL warehouses, and using a service credential in a Python UDF on classic compute requires Databricks Runtime 18.1 or above. On serverless SQL warehouses, Python UDFs cannot make outbound network requests by default, and such calls hang indefinitely.

Metric views centrally define business metrics in Unity Catalog, keeping measure definitions separate from the dimensions used to group, filter, and aggregate them, so each metric is defined once and queried consistently. They can be queried from dashboards, SQL editors, notebooks, alerts, and external BI tools such as Power BI and Tableau. Genie can also use metric views to answer natural-language questions, which keeps self-service analytics grounded in approved definitions.

AI/BI Dashboards let you create charts with AI-assisted authoring or manual configuration, with formatting options for each visualization. Global, page-level, and widget-level filters support cross-filtering and drill-through, and custom calculations add calculated fields without changing the underlying dataset. Dashboards can be version-controlled with Git and published with shared or individual data permissions.

Databricks AI/BI Dashboards combine visual analytics with AI-assisted authoring over governed data. Authors create charts with AI assistance or manual configuration, add global, page-level, and widget-level filters with cross-filtering and drill-through, and define custom calculations without changing the dataset. Dashboards can be version-controlled with Git, published with shared or individual data permissions, and built on governed Metric Views for consistent KPI definitions.

Yes, in two modes. Basic embedding places an iframe generated from the Share dialog in your site; viewers sign in with Databricks credentials, and workspace admins must define the allowed embedding surfaces. To let external users view dashboards without Databricks accounts, use embedding for external users: your application authenticates with a service principal and OAuth token (valid for one hour), and Ask Genie is not supported in that mode. Publish dashboards with individual data permissions so each viewer's access is enforced.

Yes. Serverless SQL warehouses provision compute for you and scale with workload demand through intelligent workload management, so you do not size clusters by hand. A stopped warehouse restarts automatically when a query, scheduled job, JDBC or ODBC connection, or associated dashboard targets it. Serverless limits on maximum cluster size are not stated on the warehouse overview page, so check the sizing docs for your workload.

Yes, SQL scripting is available in Databricks Runtime 16.3 and above. It supports BEGIN and END compound statements, variables, and control flow such as IF, WHILE, LOOP, and FOR, with error handling through SIGNAL and RESIGNAL. Cursor statements (OPEN, FETCH, CLOSE) require Databricks Runtime 18.1 or above. See the scripting reference for the statements available in your runtime.

Databricks provides ODBC and JDBC drivers so BI and desktop tools can connect to governed Databricks data. Tool-specific connection steps are covered by the technology partner integrations and each tool's own documentation. Check the partner documentation for the connection modes each tool supports.

Yes. Lakehouse Federation lets you query PostgreSQL, MySQL, SQL Server, Snowflake, Redshift, BigQuery, and other external databases in place, with queries pushed down to the source over JDBC. You can write SQL that joins those sources with lakehouse tables without copying the data first. Check the federation docs for the release status of each connector before you use it in production.

AI/BI dashboards let you build interactive dashboards on Databricks data, with datasets defined from tables, views, or custom queries. Global, page-level, and widget-level filters, plus cross-filtering and drill-through, let viewers explore the data. Published dashboards can refresh on a schedule and send email or Slack subscriptions, and AI-assisted authoring helps you build visualizations.

Yes. Databricks SQL runs BI and SQL analytics on the lakehouse through serverless SQL warehouses, and Photon is the default engine for SQL warehouses. Serverless warehouses provision compute for you and scale with demand through intelligent workload management. Dashboards let business users explore data with filters, cross-filtering, and drill-through, and BI tools connect over ODBC and JDBC drivers.

Yes. Lakehouse Federation lets you query external databases and warehouses in place, including PostgreSQL, Snowflake, and BigQuery, so you can start using Databricks before moving any data. Federated queries are pushed down to the source over JDBC, and you can join them with lakehouse tables. You can migrate historical data over time as each workload is ready.

Connect Power BI or Tableau through Databricks' ODBC and JDBC drivers, or through the technology partner integration for each tool. The partner documentation covers the step-by-step setup for each tool, and once connected the tool queries governed Unity Catalog data in Databricks. Check the partner documentation for the connection modes each tool supports.

Yes. A materialized view stores precomputed query results that queries can read, and it is backed by a serverless Lakeflow pipeline. You can refresh it manually with REFRESH, on a schedule with EVERY or CRON, or on upstream updates with TRIGGER ON UPDATE. Use them to speed up repeated queries that dashboards and reports run often.

Lakehouse Federation connects to PostgreSQL, MySQL, SQL Server, Oracle, Teradata, Snowflake, Redshift, Azure Synapse, BigQuery, and Databricks, and pushes queries down to the source over JDBC. Catalog federation also reads tables from Hive metastores and AWS Glue. Write unified SQL that joins these sources with lakehouse tables.

Databricks SQL is the platform's SQL analytics and BI capability, running on serverless SQL warehouses with Photon as the default engine. Dashboards let business users explore data with global, page-level, and widget-level filters, cross-filtering, and drill-through. Unity Catalog governs the data, and BI tools connect over ODBC and JDBC drivers.

Yes, you can schedule queries and set alerts in Databricks SQL. Alerts run a query on a schedule, evaluate a condition on the result, and move between OK, TRIGGERED and ERROR states, with an alert history for review. When a condition triggers, the alert sends notifications through configured notification destinations, such as webhooks, so an on-call system or chat tool can pick it up. This gives teams lightweight, governed monitoring on the same data they already query.

Yes. AI/BI dashboards support automated refresh schedules and email or Slack subscriptions for published dashboards. Dashboards can be shared at the workspace and account level and embedded in external websites and applications using iframes. Published dashboards can also include Genie Agents, so viewers can ask follow-up questions in natural language beyond the charts.

AI/BI dashboards support global, page-level, and widget-level filters, and the docs describe cross-filtering and drill-through for interactive exploration. The docs do not give a client-side row limit or list which chart types support cross-filtering, so check the dashboard and resource-limits docs for those details.

The Databricks SQL editor supports real-time collaboration, letting users edit queries together in real time. The docs I checked do not describe multiple result tabs, execution-history links to query profiles, split-screen editing, Assistant Quick Fix or Git version control for queries.

Photon is the native vectorized query engine. It is compatible with Apache Spark APIs, existing code works without changes, and it is the default engine for all SQL warehouses. The docs state up to 5x better price/performance for data and analytics workloads, based on TPC-DS benchmarks. Photon does not support UDFs, RDD or Dataset APIs.

Photon, Databricks' native vectorized query engine, is the default on SQL warehouses and serverless compute, and it accelerates SQL workloads.

Adaptive query execution re-optimizes a query during execution using actual statistics from each shuffle or broadcast exchange. The docs describe three runtime behaviors: coalescing small shuffle partitions, switching a sort-merge join to a broadcast join when one side is under a size threshold (30MB by default), and splitting skewed tasks. AQE is enabled by default, and the main switch is spark.databricks.optimizer.adaptive.enabled.

Account budgets track usage in USD at SKU list price and can cover the whole account or be narrowed by workspace, resource type, or custom tags. Each budget can have up to four thresholds, and each threshold emails its listed addresses when spend exceeds it. Alerts can lag usage by up to 24 hours, and standard budgets are monitoring tools rather than hard caps. Tag warehouses and jobs so spend can be attributed by team.

Databricks SQL includes five predictive AI functions: ai_forecast, ai_detect_anomalies, ai_top_drivers, ai_predict_value and ai_predict_class. They require the Predictive AI Functions preview, which a workspace admin enables on the Previews page. ai_predict_value is Beta; check each function's reference page for its current status before production use.

The Databricks Excel add-in lets you import data from Databricks tables, write SQL queries, create pivot tables, and use custom functions in Excel for the web and desktop on Windows and macOS. Scheduled automatic data refresh is in Public Preview as of October 1, 2026.

ai_decide is an AI Function listed as Beta in the AI Functions docs under text analysis. Check the function reference for its return types and parameters before relying on its output.

Databricks SQL includes a broad library of spatial ST_ functions that run directly in SQL, including ST_Within, ST_Distance, ST_Buffer, ST_Centroid and ST_Intersection, plus aggregates such as ST_Union_Agg. You can use them for location analytics such as delivery area checks, proximity searches and geofencing without exporting data to a separate GIS tool. The full list of signatures and behavior is in the built-in functions reference.

Databricks supports recursive common table expressions with WITH RECURSIVE, so you can traverse hierarchies such as org charts, roll up values across subtrees, and find multi-hop paths directly in SQL on Databricks Runtime 17.0 and later. MAX RECURSION LEVEL caps the number of iterations (default 100). For production queries, add depth caps and cycle checks to guard against runaway recursion.

Databricks SQL stored procedures are created with CREATE PROCEDURE ... LANGUAGE SQL, stored only in Unity Catalog, and invoked with CALL, and they require Databricks SQL or Runtime 17.0 or above. SQL SECURITY INVOKER runs the procedure body with the caller's privileges.

The http_request SQL function calls external HTTP APIs through a Unity Catalog HTTP connection, so credentials stay in the governed connection rather than in query text. It requires Databricks Runtime 16.2 or above, returns a struct with the status code and response text, and you can wrap the body in parse_json to get a VARIANT. It is rate limited and intended for interactive and agent-driven use, so it suits targeted lookups rather than high-volume batch pulls. For new code, Databricks recommends the Unity Catalog connections proxy endpoint with the provider's SDK.

The Power BI integration docs describe connection options (Power BI Desktop through Partner Connect or manual setup, and the Power BI service) and the ODBC and ADBC drivers. They do not give DirectQuery versus Dual storage guidance, so choose the storage mode from your model and validate it in Power BI.

system.query.history records SQL statements run on SQL warehouses, serverless compute for notebooks and jobs, and Lakeflow pipelines, and records are typically available within one hour. It is not a real-time monitor, so use it for trend analysis and SLA reporting rather than live alerting.

Lakehouse Federation lets you query external databases such as PostgreSQL, MySQL, SQL Server, Snowflake, BigQuery and Redshift from Databricks without ingesting the data. You create a Unity Catalog connection and a foreign catalog, then govern access to the external tables with Unity Catalog table privileges and the connection's USE CONNECTION privilege. Queries can join federated tables with Delta tables, and federation is read-only, so writes still go to native tables.

You can compute P95 query runtime per warehouse with SQL over system.query.history, which records SQL warehouse statements with results typically available within one hour. Because of that lag, use the result for SLA reporting rather than sub-hour alerting.

Query tags label SQL warehouse workloads: SET QUERY_TAGS sets session tags such as team or workload, and system.query.history exposes them in a query_tags map that you can filter and group by.

Set failOnError => false on ai_query so a row-level model error does not abort the whole batch. The default is true, which returns the parsed endpoint response, and the option requires Databricks Runtime 15.3 or later. With false, the function returns a STRUCT with response and errorMessage fields, and response is null when the endpoint fails. Other failure types still fail the whole query, so log rows that carry error messages and retry them.

In Lakeflow Spark Declarative Pipelines, a FLOW REPLACE WHERE flow deletes and recomputes all rows matching a predicate, such as a trailing seven-day date range, and leaves rows outside the predicate unchanged. Data outside that window still needs a one-time predicate override or a backfill.

Metric views define business measures centrally and separate the measure definitions from the fields used to group, filter, and aggregate them, so you can group by any available field. You define them with SQL DDL or the Catalog Explorer UI, which has built-in YAML validation. Materialization can pre-compute and incrementally refresh aggregations, and the query engine rewrites queries to use those materialized views when appropriate.

GENERATED ALWAYS AS IDENTITY is a Delta identity column option. Identity columns support only the BIGINT type and assign unique values to inserted rows, but the values are not guaranteed to be contiguous, so do not use them to count rows. GENERATED BY DEFAULT AS IDENTITY lets inserts specify their own values.

Predictive I/O is a set of Databricks optimizations that make data reads faster and reduce data rewritten during updates, and it runs automatically on the Photon engine. The docs state no release status for it.

AI/BI dashboard table columns support an HTML display type, which renders an HTML string after passing it through a sanitizer. For basic styling use the Basic display type, and for clickable cells use Actions > Link with the {{@}} placeholder.

AI/BI dashboards support custom visualizations that render a Vega-Lite JSON specification referencing dataset fields, which you can use to build a dumbbell chart of actual and target values per category. Custom visualizations are in Public Preview, so confirm availability in your workspace.

Identity columns are a Delta feature that generates BIGINT values for inserted rows. GENERATED ALWAYS AS IDENTITY has the table assign every value, while GENERATED BY DEFAULT AS IDENTITY assigns values only when the column is omitted, so explicit keys can be loaded. Values are unique but not guaranteed to be contiguous.

In Genie Code's Agent mode, run /importBI with a dashboard file or volume path, and Genie Code inspects the data model and proposes metric views and views that you approve before anything is built. The post recommends migrating the semantic layer first, then refining visuals one type of change at a time, with a manual checklist to validate metrics, filters, and charts. Calculated fields, data source connections, and dashboard filters migrate well, while chart types, layout, and drill-down behavior need review. Promote local metric views to Unity Catalog so dashboards, Genie Agents, and notebooks can reuse them.

Declarative Automation Bundles can define Lakeflow pipelines and jobs as code, so pipeline definitions can be version-controlled and deployed across environments. The bundles docs do not describe EXPLAIN MATERIALIZED VIEW, dry runs, or pipeline-level SQL parameters, so verify those separately before relying on them.

You can define metric views with SQL DDL or in the Catalog Explorer UI, which has built-in YAML validation. A YAML syntax reference is linked from the metric views docs.

AI/BI custom visualizations render a Vega-Lite JSON specification that references dataset fields, so a Pareto chart with bars, a cumulative percentage line and a threshold band can be built in one spec. Custom visualizations are in Public Preview, so confirm availability in your workspace.

Tableau can connect to Unity Catalog data through the Databricks ODBC driver, version 2.6.19 or later, and service principal OAuth requires version 2.8.2 or later. The Tableau docs advise reducing the records queried and aggregating where possible, since Tableau can struggle to lay out very large mark counts. Tableau Cloud Explore requires Unity Catalog compute.

Databricks SQL supports a GEOGRAPHY data type, which is in Public Preview. The docs I checked do not describe a GEOMETRY type, spatial SQL status or benchmark speedups.

Databricks dashboard dataset guidance says to avoid SELECT * and to explicitly select only the required columns, and to use WHERE clauses so only needed rows are included, because viewers can reach every field in a dataset.

Databricks SQL provides DATE for year, month, and day values without a time zone, TIMESTAMP for year-through-second values in the session local time zone, and TIMESTAMP_NTZ, whose operations ignore time zones. Use DATE for calendar dates such as order or birth dates, TIMESTAMP for events that should display in the user's session time zone, and TIMESTAMP_NTZ when values should not be adjusted for time zones.

Databricks SQL ships a large built-in function library covering aggregate and window functions, string, math, date and time, array and map, JSON and VARIANT, and spatial and sketch functions. It also includes AI functions such as ai_query for calling models from SQL. Use the function reference to check exact signatures and the Databricks Runtime version that supports each function.

Use substring(expr, pos, len) to extract part of a string; positions are 1-based, so substring('hello', 2, 3) returns 'ell'. Omit len to read to the end of the string, and use a negative pos to count from the end, so substring('hello', -2) returns 'lo'. substring is a synonym for substr.

Use get_json_object(json, path) to pull a value out of a JSON string with a JSONPath that starts at $, such as $.a.b or $.items[0]; it returns a string or NULL when the path is not found. For new code, Databricks recommends the VARIANT type: parse_json converts a JSON string to VARIANT, and the colon operator (for example v:name) reads fields from it. Keep get_json_object for simple one-off extraction from existing STRING columns.

Use datetime patterns such as yyyy, MM, dd, HH, mm, and ss with to_date(expr, fmt) and to_timestamp(expr, fmt) to parse strings into dates and timestamps. Use date_format(expr, fmt) to format a date or timestamp as a string. The pattern reference also covers time-zone letters and optional sections in square brackets.

Use collect_set(expr) to gather a group's values into an array of unique non-NULL values; duplicates are removed, NULLs are excluded, and the element order is not deterministic. collect_set also works as a window function with OVER and accepts a FILTER (WHERE ...) clause to limit which rows are aggregated. Use it to build deduplicated lists per group, such as the distinct product codes bought by each customer.

VARIANT stores semi-structured data such as JSON objects and arrays, and it is available in Databricks Runtime 15.3 and above and in Databricks SQL. Create a VARIANT with parse_json or CAST, and read fields with the colon operator or variant_get. Use VARIANT rather than a STRING holding JSON when you need to query fields directly.

Databricks SQL supports standard join syntax, including inner, outer, semi, anti, and cross joins. SEMI and ANTI joins filter left rows by whether they have a match on the right without duplicating rows. Stream-stream joins support inner, left outer, right outer, full outer, and left semi, and Databricks recommends watermarks on both inputs. Range join hints can speed up slow inequality joins.

Wrap a SELECT in parentheses with an alias to use it as a derived table in the FROM clause, for example SELECT * FROM (SELECT col1 FROM table1) AS t1 WHERE col1 > 10. Use a derived table for a one-off intermediate step. When a query needs the same intermediate result more than once, or needs named, readable steps, a WITH clause (common table expression) is usually clearer.

COALESCE returns its first non-NULL argument and evaluates arguments left to right, stopping at the first non-NULL value. For example, COALESCE(NULL, 1, NULL) returns 1 and COALESCE(NULL, 'hello') returns 'hello'. All arguments must share a compatible type, so choose a default that matches the column type, such as COALESCE(col, 0) for an integer column.

SELECT DISTINCT removes duplicate rows from the result, while ALL, the default, keeps them. Choose the column list you need and apply DISTINCT to that list. To count unique values in one column, use COUNT(DISTINCT col).

A WITH clause defines named result sets that the main query can reference, and later CTEs can refer to earlier ones. WITH RECURSIVE supports recursive queries over hierarchical data such as org charts, and it requires Databricks Runtime 17.0 or above. CTEs exist only for the duration of the query and create no persistent objects.

Add a Power BI task to a Databricks Workflows job and select a Power BI connection, which must be created in Unity Catalog first. The Power BI docs describe machine-to-machine OAuth with a service principal and role-based OAuth as connection options.

Materialized views refresh incrementally when Databricks can process only the data needed, and fall back to a full recomputation when correctness requires it. Not all computations can be incrementally computed, so check which views can refresh incrementally before deploying, and watch pipeline costs for full recomputes.

Databricks SQL serverless warehouses provide instant, elastic compute on Databricks-managed infrastructure, and the docs recommend serverless when it is available. Whether a lakehouse replaces a separate warehouse depends on your workload, so benchmark your own dashboard queries on serverless SQL warehouses before deciding.

Use serverless SQL warehouses, and configure enough max clusters for peak load while monitoring Peak Queued Queries. Predictive Optimization automatically runs OPTIMIZE (including incremental clustering for enabled tables), VACUUM, and ANALYZE on Unity Catalog managed tables, which helps with file layout and statistics.

The 5X-Large size is in Public Preview for Pro and serverless SQL warehouses in all regions, and workspace admins control access from the Previews page. In the sizing table for classic and pro warehouses, 5X-Large has 512 workers versus 256 for 4X-Large.

The Databricks add-in for Excel connects your workspace to Excel for the web and to desktop Excel on Windows and macOS. Users can browse and import Unity Catalog tables, pivot data, use custom functions to analyze data, write data from Excel to a Unity Catalog table, and ask Genie questions.

AUTO CDC in Lakeflow Spark Declarative Pipelines replaces hand-written MERGE logic. You create a streaming table and define a flow with CREATE FLOW ... AS AUTO CDC INTO, using KEYS and SEQUENCE BY, with STORED AS SCD TYPE 1 or SCD TYPE 2. SCD Type 2 targets include __START_AT and __END_AT columns, where a null __END_AT marks the active row. APPLY AS DELETE WHEN handles deletes, and the pipeline must use serverless or the Pro or Advanced edition.

Serverless SQL warehouses run on Databricks-managed infrastructure, and Lakehouse Federation lets Databricks query Google BigQuery, Amazon Redshift, Snowflake, and other listed sources through Unity Catalog foreign catalogs without copying the data. Databricks-reported performance comparisons are vendor claims, so benchmark your own workload.

Yes. Lakehouse Federation lets Databricks query Snowflake, Redshift, BigQuery, PostgreSQL, MySQL, Oracle, Teradata, SQL Server, and Azure Synapse without moving the data, with filters, joins, and aggregates pushed down to the source. Access is governed through Unity Catalog foreign catalogs with table-level access controls, so you can start analyzing external data now and migrate on your own schedule.

Yes. Lakehouse Federation lets one Databricks SQL query join data across sources such as Redshift, Azure Synapse, and BigQuery, and across on-premises or self-hosted systems such as Oracle, Teradata, and SQL Server, with filters, joins, and aggregates pushed down to each source. Access is governed through Unity Catalog foreign catalogs, so you manage permissions in one place. This supports hybrid and phased migrations without copying the data first.

Yes. Lakehouse Federation lets Databricks query Snowflake, Redshift, BigQuery, PostgreSQL, and other sources without migrating the data, so you can analyze external data first and move it gradually. Federated queries are governed through Unity Catalog foreign catalogs, which gives you one place to manage access while you decide what to migrate.

Power BI and Tableau can connect to Databricks. The Power BI docs cover Partner Connect and manual setup, and the Tableau docs cover the Databricks ODBC driver for Unity Catalog data. Lakehouse Federation lets Databricks query external sources such as Snowflake, Redshift, and BigQuery through foreign catalogs without first migrating them.

Databricks SQL uses serverless SQL warehouses that run on Databricks-managed infrastructure, which the docs describe as instant and elastic and recommend when available. Photon is the default engine for all SQL warehouses, and Predictive Optimization automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables.

Predictive Optimization automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables, so table maintenance does not depend on hand-run jobs. Lakehouse Federation lets you query Redshift tables from Databricks through a foreign catalog while you decide what to move.

Serverless SQL warehouses provision compute on demand rather than requiring reserved capacity, and the docs describe them as instant and elastic. Check the pricing docs for billing granularity before comparing with BigQuery slot pricing.

Yes. Lakehouse Federation connects to Snowflake over JDBC and pushes filters, joins, aggregates, limit and offset, and sorting down to Snowflake's engine, with window functions (Rank, RowNumber, DenseRank) also pushed down. Top-N queries (sort plus limit) are pushed down on Databricks Runtime 17.3 and above, and only result sets return to Databricks. This makes Databricks a practical query layer for hybrid Snowflake environments during a migration.

Yes. Lakehouse Federation lets a single Databricks query span BigQuery, Redshift, Snowflake, Oracle, Teradata, MySQL, PostgreSQL, SQL Server, and Azure Synapse, with each source's part of the query pushed down to it. Access is governed through Unity Catalog foreign catalogs with table-level access controls, so permissions are managed in one place rather than in each source.

Lakehouse Federation queries Redshift over JDBC and, through catalog federation, can read foreign tables in cloud object storage, so one Databricks query can span both. Redshift Spectrum is a Redshift feature, and the cited federation page does not describe its internals, so the engine comparison is not supported by that page. Redshift tables can be queried from Databricks without migration.

Use Lakehouse Federation to query Teradata directly while you migrate, so you can validate results before moving workloads. Databricks' agentic code converter helps translate enterprise warehouse SQL to Databricks SQL, and the migration guidance names T-SQL, Snowflake, and Oracle SQL dialects among its sources. Databricks SQL on Delta Lake then serves the migrated workloads with elastic compute.

Yes. Lakehouse Federation supports Oracle as a source, pushing query work down to Oracle, and one Databricks query can also span Snowflake, BigQuery, Redshift, Teradata, and other sources. Access is governed through Unity Catalog foreign catalogs, so permissions are managed in one place for every source.

Lakehouse Federation lets Databricks query BigQuery through a Unity Catalog foreign catalog, so you can move workloads gradually without an initial copy. The federation docs do not describe a BigQuery code-conversion or reconciliation tool, so evaluate those separately.

Photon is the default engine for all SQL warehouses. It is a native vectorized engine written in C++ that accelerates SQL and DataFrame API calls, and Databricks documents up to 5x better price/performance on TPC-DS against other cloud data warehouses. Predictive Optimization automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables.

Lakehouse Federation supports Azure Synapse (SQL Data Warehouse) as a query source, so you can query it from Databricks while you migrate. Databricks SQL runs analytical workloads on SQL warehouses. Whether Databricks replaces both Synapse and Fabric depends on your workloads, and these docs do not establish it.

Databricks runs Spark, so PySpark and Scala code can move with limited rewriting, and the migration guidance covers SQL and Hive ETL workloads. Hive tables can be queried through External Hive metastore federation during the move, and data moves to Delta Lake on cloud object storage. Once the data is in Delta Lake, Databricks SQL serves the BI workloads that Impala handled.

Serverless SQL warehouses provide instant, elastic compute on Databricks-managed infrastructure. The warehouse docs say to configure a sufficient maximum number of clusters for peak load and to monitor Peak Queued Queries, and Intelligent Workload Management predicts query needs and queues work when capacity is short.

Databricks SQL replaces Synapse SQL pools with serverless compute that scales on demand and is accelerated by Photon. Databricks' agentic code converter converts T-SQL to ANSI SQL, and the migration guidance names T-SQL among its source dialects; the guidance gives no status for the converter. Migrated data lands in open Delta Lake storage, and Unity Catalog takes over access control after cutover.

H3 can prefilter candidates for point-in-polygon work: h3_coverash3 returns the H3 cells that fully cover a polygon at a chosen resolution, so you can explode them, join to the point H3 cells, and then apply an exact polygon check because covering cells extend beyond the boundary.

Databricks H3 functions such as h3_longlatash3 index longitude and latitude into H3 hexagonal cells, and joining indexed datasets on cell IDs performs a spatial join without a spatial predicate.

The query_tags column in system.query.history stores custom key-value tags set through session configuration or the SET QUERY_TAGS statement, and it is populated for SQL warehouses and serverless compute. You can use those tags to attribute query activity to models or teams. The docs do not describe dbt-databricks tagging, so confirm that in the dbt-databricks docs.

Unity Catalog metric views define governed business measures in one place, and the metric views docs say they can be queried from SQL editors, notebooks, dashboards, Genie Agents, and alerts. Metric views can pre-compute and incrementally refresh aggregations through materialization. Confirm dbt-databricks support for metric views in the dbt docs.

Databricks SQL supports window functions including RANK, DENSE_RANK, LAG, LEAD, and CUME_DIST, and aggregate functions such as MIN can run over a window. Use PARTITION BY to define the rows each window function operates on, and ORDER BY to sequence rows within each partition. Ranking functions require an ORDER BY clause. Together these cover rankings and lead/lag comparisons in a single query.

Databricks SQL provides array_distinct, array_remove, array_contains, and array_intersect for arrays, and map_keys, map_values, map_concat, map_filter, and map_zip_with for maps. Higher-order functions transform, filter, and reduce apply functions to array elements directly in SQL. These built-in functions let you manipulate nested ARRAY and MAP columns without writing custom code.

Databricks SQL provides the collect_set and collect_list aggregate functions to gather values from rows into arrays, and flatten and explode to work with nested arrays. explode is a table-valued generator function, and explode_outer is also available. Use these functions together to reshape nested data inside SQL queries.

Connect Power BI Desktop to Databricks clusters and SQL warehouses using Partner Connect or manual configuration. The ADBC driver can replace the ODBC driver for this connection. A Power BI task in a Databricks job can orchestrate your Power BI semantic models automatically.

Databricks SQL supports common table expressions (WITH) that define named subqueries, PIVOT for rotating rows into columns, and HAVING for filtering grouped rows. The SQL language reference documents query optimizer hints, and Databricks SQL supports ANSI SQL with Delta Lake extensions. Together these let you write multi-stage analytical queries in SQL.

Use common table expressions to break a transformation into named subqueries, and PIVOT to rotate rows into columns. Filter grouped results with HAVING after aggregation. Query optimizer hints are documented in the SQL language reference for influencing how a query is executed.

AI/BI dashboards feature AI-assisted authoring for building dashboards on your data. You can set up automated refresh schedules and email or Slack subscriptions for published dashboards, and schedule dashboard updates with Lakeflow Jobs. Together these keep business users working from current results.

Yes, you can keep using Snowflake while adopting Databricks. Lakehouse Federation lets Databricks query Snowflake data in place through a Unity Catalog connection, so you can start analytics and AI on Databricks without moving everything first. Federated tables are governed alongside your lakehouse data, which gives you one access layer. This supports a phased adoption where you federate first and migrate selected workloads over time.

BigQuery is Google Cloud's serverless data warehouse. Databricks SQL runs on SQL warehouses with Photon as the default engine, and Lakehouse Federation lets Databricks query BigQuery without migrating the data first. The docs do not compare the two, so decide through a proof of concept on your workload and cloud.

Yes. Lakehouse Federation lets Databricks run federated queries on BigQuery data that Databricks does not manage, through a foreign catalog, and Unity Catalog query syntax and governance tools apply to it. Some operations, such as queries on views and external tables, are materialized in BigQuery and incur additional BigQuery compute charges. This is useful for exploring BigQuery data, joining it with lakehouse tables, or running a phased migration.

Photon is the native vectorized query engine for Databricks, written in C++. It is the default engine for all SQL warehouses, and Databricks documents up to 5x better price/performance on TPC-DS against other cloud data warehouses. The docs do not compare Databricks with Amazon Redshift, so benchmark your own workload.

Yes. Lakehouse Federation connects Databricks to Redshift and queries Redshift database tables in place, with no copy or ingestion step. Unity Catalog permissions and governance tools apply through a foreign catalog that mirrors the Redshift database. Federation covers Redshift database tables but not Redshift external (Spectrum) data, so plan around that limit.

Databricks SQL runs on SQL warehouses with Photon as the default engine, and it connects to BI tools such as Power BI and Tableau. Lakehouse Federation supports Teradata as a source, so you can query it from Databricks during a migration. The docs do not compare Databricks with Teradata, so assess your workloads with a proof of concept.

Yes, Databricks can query an Oracle database directly through Lakehouse Federation, a capability similar to an Oracle database link. Lakehouse Federation runs federated queries on Oracle data that Databricks does not manage, through a foreign catalog that mirrors the Oracle database. Unity Catalog query syntax and governance tools control access, so access is governed centrally. Oracle can remain the source of record while analytics and AI run on Databricks.

Databricks SQL runs on SQL warehouses with Photon as the default engine, and the same governed data can serve ETL, streaming, and AI. Lakehouse Federation supports Oracle as a source, so you can query it from Databricks during a migration. The docs do not compare Databricks with Oracle, so validate with a proof of concept.

Databricks SQL can serve BI workloads on the same governed data used for ETL and AI, and it connects to BI tools such as Power BI and Tableau. Whether it replaces a separate warehouse depends on your SQL and pipeline complexity, so assess each workload before migrating. Serverless SQL warehouses are recommended when available.

Yes. Lakehouse Federation lets Databricks query sources such as PostgreSQL, Oracle, Teradata, Snowflake, Redshift, and BigQuery, pushing queries down to external databases. Unity Catalog governance applies to those federated sources, so you can start analyzing and building on Databricks while legacy systems stay live. Migrating workloads incrementally reduces risk compared with a single large cutover.

Yes. Lakehouse Federation connects Databricks to PostgreSQL, so you can query Postgres tables through a foreign catalog without ingesting them first. Unity Catalog governance and query syntax apply to the federated tables. This helps you analyze operational data alongside lakehouse tables, or run a phased migration while Postgres stays the source of record.

Yes. Lakehouse Federation includes a Teradata connector that lets Databricks query Teradata data that Databricks does not manage, through a foreign catalog that mirrors the Teradata database. Unity Catalog query syntax and governance tools control access. Teradata can stay the system of record while you analyze and build on Databricks, then migrate workloads on your own schedule.

Trino and Presto are strong distributed SQL engines for querying across sources. Databricks Lakehouse Federation queries many external databases and catalogs in place, including PostgreSQL, Oracle, Teradata, and Snowflake, under Unity Catalog governance. Databricks also brings Spark engineering, streaming, and AI to the same platform, so the choice depends on whether you need a query engine alone or a full platform.

Starburst, built on Trino, specializes in federated SQL across many sources. Databricks Lakehouse Federation queries many external systems in place under Unity Catalog governance, and Databricks adds Spark engineering, streaming, and AI on the same platform. Organizations that want federation together with a full data and AI platform will find Databricks covers more of the stack in one place.

Databricks SQL runs on Photon, a native vectorized query engine that accelerates SQL workloads. The docs do not compare Databricks with ClickHouse, so validate with your own workload.

Yes. Tableau can connect to Unity Catalog data through the Databricks ODBC driver (version 2.6.19 or later), and Power BI has a Databricks connector with Power BI connections set up in Unity Catalog. Confirm the authentication setup for each tool before cutover.

Databricks SQL runs on SQL warehouses with Photon as the default engine, and it connects to BI tools such as Power BI and Tableau. The same governed Unity Catalog tables can serve ETL, streaming, and AI workloads, and serverless SQL warehouses are recommended when available. The docs do not benchmark it against other warehouses.

A warehouse scales in two dimensions. You pick a cluster size for heavier queries and set a minimum and maximum number of clusters for concurrent users. For classic and pro warehouses, the autoscaler provisions more clusters when queries queue, and serverless warehouses add or remove clusters as needed.

Yes, per cluster, which is why warehouses scale out. Classic and pro warehouses are sized at about one cluster per 10 concurrent queries, which the docs present as a recommendation rather than a fixed limit, and queued queries wait in a queue of up to 1,000 while the autoscaler provisions more clusters under queue pressure. Serverless warehouses add or remove clusters as needed, so concurrency follows demand.

Yes. Databricks SQL supports ANSI SQL with Delta Lake extensions, so a SQL-first team can work in standard SQL for warehousing and analytics. Databricks SQL also powers interactive AI/BI dashboards for sharing insights across your organization. Add Python or pipelines later if your needs grow.

No, Databricks supports SQL stored procedures as Unity Catalog objects. You create them with CREATE PROCEDURE and run them with CALL on Databricks SQL and Databricks Runtime 17.0 and above. Procedures accept IN, OUT, and INOUT parameters and can contain SQL scripting such as variables and control flow, so logic can run natively on the platform.

Materialized views are Unity Catalog managed tables that physically store the results of a query. They refresh incrementally when possible and fall back to a full refresh when an incremental refresh is not possible or cost-effective. They can be created and refreshed from a Databricks SQL warehouse or a serverless notebook, and incremental refresh works best with Delta sources that have row tracking enabled.

No. Photon accelerates SQL and DataFrame workloads, including ETL and stateless streaming, and falls back to the Spark runtime for operations it does not support, such as Python UDFs, RDD and Dataset APIs, and stateful streaming. The cited Photon page does not list window functions or sorts individually, so confirm acceleration on your own queries. Photon is on by default on serverless compute and SQL warehouses.

No. Photon is the default engine for all Databricks SQL warehouses, and it is enabled by default on serverless compute and serverless Lakeflow pipelines. Databricks also enables Photon by default for classic all-purpose compute, jobs compute, and classic pipelines, and if you create those through the Clusters or Jobs API, you set runtime_engine to PHOTON explicitly.

Not with serverless. A serverless SQL warehouse has rapid startup, typically 2 to 6 seconds, so interactive queries and dashboards see little cold-start wait. Classic and pro SQL warehouses take several minutes to start, typically about 4 minutes, which is the delay older articles describe. Serverless warehouses were built to remove that wait.

Databricks SQL runs queries on SQL warehouses, which are managed compute for querying and exploring data. Photon, the Databricks-native vectorized query engine, is the default engine for all SQL warehouses, so warehouse queries get it without a separate runtime choice. Photon also accelerates batch ETL and DataFrame workloads, stateless streaming, and Lakeflow pipelines on compute where Photon is enabled.

Yes. Photon is the default engine for all SQL warehouses and accelerates SQL and DataFrame workloads, including scans, joins, aggregations, and writes. Databricks cites up to 5x better price/performance for data and analytics workloads compared with other cloud data warehouses, based on TPC-DS benchmarks. Photon falls back to Spark for unsupported operations, and the gains are smallest for queries that normally run in under two seconds, so measure runtimes on your own workload.

Serverless SQL warehouses start in typically 2 to 6 seconds, while pro and classic SQL warehouses typically take about 4 minutes, according to the warehouse types documentation. Choosing serverless is therefore the fastest path to near-instant startup, and Databricks recommends serverless SQL warehouses when they are available. This page is the AWS version and does not list Azure-specific startup times, so check the Azure documentation for Azure figures.

Lakehouse Federation supports SQL Server, Azure SQL Database, and Azure SQL Managed Instance. Create a Unity Catalog connection with CREATE CONNECTION ... TYPE sqlserver, providing the host, port, and authentication details, either in Catalog Explorer or in the SQL editor. Then run CREATE FOREIGN CATALOG <catalog> USING CONNECTION <connection> with OPTIONS (database '<database-name>') so the SQL Server tables are queryable with Unity Catalog governance. You need a Unity Catalog-enabled workspace, the CREATE CONNECTION privilege to create the connection, and CREATE CATALOG on the metastore plus connection ownership or CREATE FOREIGN CATALOG to create the foreign catalog. Compute must be Databricks Runtime 13.3 LTS or above in Standard or Dedicated access mode, or Pro or serverless SQL warehouses on 2023.40 or above.

Databricks stores the cached data of a materialized view in cloud storage, and Unity Catalog holds the view's metadata. Some backing data lives in the __databricks_internal catalog, which is used for incremental refresh and does not appear in Catalog Explorer. Because the results are precomputed, queries against the view can run faster than queries against a regular view. A pipeline maintains the output, keeping it in sync with the source data.

Not exactly. A materialized view is a declarative pipeline object that combines its defining query, the flow that updates it, and its cached results, and you query it like a table. Databricks stores the cached data in cloud storage and keeps it in sync with the source data, usually through incremental updates that can cost far less than a full recomputation.

An incremental refresh detects the changes in the source data since the last refresh and recomputes only the affected results, then adds them to the table, so results match batch query semantics. By default, a serverless refresh runs a cost analysis and chooses the cheaper of incremental refresh and full recompute. Incremental refresh works best when source tables have row tracking, deletion vectors, and change data feed enabled, which Databricks recommends. A full refresh resets all checkpoints.

Databricks does not choose the more expensive option. It runs a cost analysis and chooses the cheaper of incremental refresh and full recompute. The AUTO policy is the default, INCREMENTAL STRICT requires incremental refresh and fails if it is not possible, and FULL always recomputes. Incremental refresh requires serverless compute, and classic compute always performs a full recompute.

Connect to serverless compute, because incremental refresh is only available when the compute is serverless. The query has to be eligible for incremental refresh, and Databricks decides per refresh based on cost. Enable row tracking on the queried source tables, and enable deletion vectors and change data feed as Databricks recommends. Some incremental refresh operations require row tracking, so check your sources before relying on it.

Keep declarative pipelines on serverless compute. Materialized views on serverless can refresh incrementally, processing only the changes since the last refresh, which is far cheaper than recomputing large tables. Materialized views on classic compute always perform a full recompute, so moving the pipeline to serverless avoids that cost. For data that is ingested once and not reprocessed, streaming tables guarantee exactly-once processing and are the better fit, and row tracking, deletion vectors, and change data feed on source tables further improve incremental refresh.

The local cache is checked first and stores query results for the lifetime of the cluster, or until the cache is full, whichever comes first. The remote result cache is checked next and is shared across the workspace's warehouses. A query runs only if neither cache holds its result. Each result cache lasts 24 hours from when the entry is created, both are invalidated when underlying tables are updated, and the remote result cache persists through warehouse stops and restarts.

Use the query profile, which visualizes the details of a query execution by operator. The DAG shows metrics such as time spent, memory peak, and rows, and the Top operators panel lists the longest-running operators. The Performance insights tab ranks opportunities by their projected effect on total task duration, and filter icons show the percent of data pruned during scanning. Look for exploding joins, full table scans, and expensive shuffles, which are common causes of slow queries, and if a query runs from the query cache it has no profile, so change it slightly, such as the LIMIT, to rerun it.

Yes. The system.query.history table records queries run on SQL warehouses and on serverless compute used by notebooks, jobs, and Lakeflow pipelines, and records are typically available within one hour. It includes columns such as total_duration_ms, execution_status (FINISHED, FAILED, or CANCELED), read_bytes, spilled_local_bytes, and query_tags for cost attribution. The statement_text column is redacted unless you are an account admin or a member of the databricks_pii_access group, and by default only admins can access system tables, so share access with dynamic views for other users.

The system.billing.usage table centralizes your account's billable usage and routes it to all regions, and records are typically available within 12 hours. Each record includes the quantity, unit, time range, SKU, cloud, and tags, along with IDs for compute resources and jobs where applicable. Databricks provides a guide with sample queries for monitoring costs with system tables, including an hourly usage query by job_id that stays correct after corrections.

A collation is a set of rules that determines how string comparisons are performed. The default collation is UTF8_BINARY, which compares strings byte by byte on their UTF-8 representation, and UTF8_LCASE is a lightweight case-insensitive collation. Locale collations such as DE or FR_CAN, or UNICODE, can be combined with CS or CI, AS or AI, and RTRIM modifiers. Apply a collation with the COLLATE clause, for example ORDER BY col COLLATE UTF8_LCASE, and collations require Databricks Runtime 16.1 and above, with RTRIM requiring 16.2 and above.

Databricks recommends the VARIANT data type over JSON strings for semi-structured data, and VARIANT is available in Databricks Runtime 15.4 and above. Convert JSON text into a VARIANT with parse_json, then extract fields with colon paths such as raw:owner, bracket notation, or variant_get with a target type, noting that field names are case sensitive. Use try_variant_get when a cast might fail, and cast with :: or CAST for supported types. VARIANT columns cannot be used as clustering, partition, or Z-order keys, nor for comparisons, grouping, or ordering.

Store the data in a VARIANT column rather than as JSON strings, which Databricks recommends for semi-structured data, and convert the JSON with parse_json. Query fields with colon paths or variant_get, and use try_variant_get where types might conflict. The VARIANT documentation does not publish size limits or specific guidance for files of 50 GB or more, so benchmark ingestion and queries on a representative slice of the data before you commit to a design.

The Statement Execution API 2.0 runs SQL statements on Databricks SQL warehouses over REST. Authenticate with a personal access token or, for automation, OAuth, and make sure the user has the Databricks SQL entitlement and CAN USE permission on the warehouse. Send a POST to /api/2.0/sql/statements with warehouse_id and statement, then poll GET /api/2.0/sql/statements/{statement_id} for status and results.

POST the statement to /api/2.0/sql/statements with the required warehouse_id and statement fields, and optionally catalog, schema, parameters, and wait_timeout. wait_timeout accepts 0s or a value from 5 to 50 seconds, with a default of 10 seconds, and if the statement is still running the response returns a PENDING state and a statement ID. Results come back inline in JSON_ARRAY format by default, and inline results over 25 MiB fail the statement, so large results should use external links.

Set the disposition field to choose how results come back. INLINE returns JSON_ARRAY data in the response, and EXTERNAL_LINKS returns presigned cloud-storage URLs, valid for 15 minutes or less, for JSON, CSV, or ARROW_STREAM results. Download each external_link with plain HTTP, without sending the Databricks authorization header, and protect these URLs because they embed temporary credentials. Only the user who ran the statement can fetch its results.

Run SHOW CREATE TABLE table_name, or SHOW CREATE TABLE view_name, to return the DDL statement that creates the table or view. The output is CREATE TABLE, CREATE VIEW, CREATE MATERIALIZED VIEW, or CREATE STREAMING TABLE, depending on the object. For materialized views and streaming tables, you need Databricks Runtime 14.1 or above.

Yes, alerts can support data quality checks. A scheduled alert runs its query, evaluates the condition, and resolves each evaluation to OK, TRIGGERED, or ERROR, and you can review past evaluations in the alert history. Write the condition as a data quality check on the query result, such as a row count or metric threshold, so a bad value triggers a notification.

Yes. After you publish a dashboard, you can share it with specific users or groups, or with everyone in your account, and account users do not need workspace access to view it. Users who do not have workspace access are limited to CAN RUN permission. Shared data permissions are the default, where viewers run queries with the publisher's permissions, or you can publish with individual data permissions so each viewer's own credentials apply.

Yes. Published AI/BI dashboards can be embedded in external sites and apps. Basic embedding lets viewers who sign in with Databricks credentials see the dashboard once they have been granted access, and workspace admins define which surfaces are allowed to embed dashboards. Embedding for external users lets people without Databricks accounts view dashboards, with the embedding application authenticating through a service principal using an OAuth secret, and Databricks recommends publishing with individual data permissions so each viewer's own access applies.

AI/BI dashboards let you build data visualizations and share reports with your team, with AI-assisted authoring, an enhanced visualization library, and a streamlined configuration experience. Datasets can be defined from tables, views, or custom queries, and filters can be scoped globally, per page, or per widget. Published dashboards can be shared with anyone registered to your Databricks account, even without workspace access, and they can be embedded in external sites and apps.

Connect Power BI Desktop to Databricks clusters and SQL warehouses with Partner Connect or a manual configuration. For authentication, you can set up a service principal for machine-to-machine (M2M) OAuth, configure Microsoft Entra ID as the single sign-on identity provider, or authorize OAuth as a role if you have Assume permission on it. Databricks recommends switching from the ODBC driver to the ADBC driver, which is also required for Lakehouse Real-Time warehouses.

Databricks recommends one cluster for every 10 concurrent queries. A warehouse scales between a minimum and a maximum number of clusters, which default to one each, so raising the maximum lets it serve more concurrent users. Increasing the cluster size reduces query latency, and clusters that run longer than 24 hours are recycled, which can briefly show more active clusters than the configured maximum.

Databricks recommends serverless SQL warehouses when they are available, citing instant, elastic compute, minimal management overhead, and lower total cost of ownership. Pro and classic warehouses are the other warehouse types. The cited page does not state startup times for any warehouse type.

A stopped SQL warehouse starts automatically when you run a query against it, provided you have access to it. It also auto-restarts when a job assigned to it is scheduled, when a JDBC or ODBC connection is made to it, or when a dashboard tied to a dashboard-level warehouse is opened. Starting a warehouse manually requires at least CAN MONITOR permission on it.

Workspace admins and users with unrestricted cluster creation permissions can create SQL warehouses, while most users can restart any warehouse they can connect to. New warehouses come with a usable starting configuration: the default cluster size is X-Large, and Auto Stop defaults to 45 minutes for pro and classic warehouses and 10 minutes for serverless. You can adjust these settings after creation.

The default cluster size for a new SQL warehouse is X-Large. Increasing the cluster size reduces query latency, so step up the size when latency matters more than cost. Cluster size is one of the warehouse settings you can tune alongside the minimum and maximum cluster counts.

For pro and classic SQL warehouses, Auto Stop defaults to 45 minutes and has a 10-minute minimum. Serverless warehouses default to 10 minutes, with a 5-minute minimum in the UI, and the API accepts serverless values as low as 1 minute. Idle warehouses continue to accumulate DBU and cloud instance charges until they are stopped, so a shorter Auto Stop reduces idle cost.

Scaling sets the minimum and maximum number of clusters a warehouse uses for a query, and the defaults are one minimum and one maximum cluster. Raise the maximum to serve more concurrent users, and Databricks recommends about one cluster for every 10 concurrent queries. Clusters that run longer than 24 hours are recycled, which can temporarily make the active cluster count exceed the configured maximum.

Cluster policies, including spot instance policies, are not supported on serverless SQL warehouses, and customer-managed VPCs do not apply to serverless SQL warehouse compute. Serverless warehouses do not have public IP addresses and use private connectivity between the control plane and the serverless compute plane in nearly all cases. Serverless SQL warehouses are supported in all AWS regions.

Serverless is enabled by default in most accounts, but the account must accept the additional serverless terms in the account console under the feature enablement tab. The workspace must be on the Premium plan or above, and the account must not be on a free trial. The workspace must not use S3 access policies or an external Hive legacy metastore, although AWS Glue can be used as the workspace legacy metastore.

Click the name of a SQL warehouse and then the Monitoring tab to watch its live activity. Live statistics near the top of the page show warehouse status, running queries, queued queries, and the current cluster count. The peak query chart shows the maximum number of concurrent running and queued queries in each 5-minute window, and the default time range is 8 hours, with 24 hours, 7 days, 14 days, or a custom period also available.

On serverless warehouses, the workload manager (IWM) predicts each query's resource needs and checks available capacity. If capacity is available, the query starts immediately, and if it is not, the query is placed in a queue. When queued wait times increase, the autoscaler provisions more clusters to work through the queue.

Classic and pro warehouses scale based on the estimated time needed to process their running and queued queries. A load of 2 to 6 minutes adds one cluster, 6 to 12 minutes adds two, and 12 to 22 minutes adds three, with one more cluster for every additional 15 minutes of load beyond 22. A query that waits in the queue for 5 minutes triggers a scale-up, and after 15 consecutive minutes of low load the warehouse scales down to the minimum needed for its peak load.

The maximum number of queries that can wait in a queue is 1,000 for all SQL warehouse types. Autoscaling adds clusters as estimated queue load grows, so the queue drains faster when demand rises. Watch the Peak Queued Queries metric on the monitoring page to see whether a queue is building.

A consistently non-zero Peak Queued Queries value on the monitoring page indicates that you may need a larger cluster size or more clusters. The peak chart shows the highest concurrent count of running and queued queries in each 5-minute window. Act on it by increasing the warehouse size or raising the maximum cluster count so the warehouse can handle the concurrency.

Photon is the Databricks-native vectorized query engine. It processes data in columnar batches on a C++ runtime, while Spark's Catalyst optimizer still plans the query, and it works with Spark APIs so existing code generally needs no changes. Photon is the default engine for all SQL warehouses and powers dashboards, ad hoc queries, and scheduled reports. Batch ETL and DataFrame workloads that use SQL or the DataFrame API also benefit from faster scans, joins, aggregations, and writes, and unsupported operations fall back to Spark automatically.

Photon accelerates stateless streaming, including workloads that write to a Delta or Parquet sink from sources such as Delta, Parquet, CSV, JSON, Kafka, and Kinesis. Stateful streaming is not supported by Photon, so stateful queries run on the standard Spark runtime. Photon also accelerates Lakeflow pipelines when it is enabled in the pipeline configuration.

QUALIFY filters the results of window functions directly, so you can keep only the top-ranked row per group without writing a subquery. The syntax is QUALIFY boolean_expression, and the query must include at least one window function in the SELECT list or the QUALIFY clause. The QUALIFY expression can combine conditions with AND and OR, but it cannot contain aggregate functions. It is supported in Databricks SQL and on Databricks Runtime 10.4 LTS and above.

A SQL script starts with a compound statement block, BEGIN ... END, which can declare local variables, cursors, user-defined conditions, and condition handlers before its body. The body can contain flow control, DDL, DML, queries, SET statements, EXECUTE IMMEDIATE, and nested compound statements. SQL scripting is available on Databricks Runtime 16.3 and above, and cursor statements require Databricks Runtime 18.1 or above.

SQL scripting supports IF THEN ELSE, CASE, WHILE, REPEAT, LOOP, and FOR for control flow, with LEAVE to exit a loop and ITERATE to skip to the next iteration. It also includes SIGNAL and RESIGNAL for raising conditions, GET DIAGNOSTICS for condition details, and nested BEGIN ... END blocks. The cursor statements OPEN, FETCH, and CLOSE require Databricks Runtime 18.1 or above. Exception handlers can target named conditions, SQLSTATE values, a generic SQLEXCEPTION, or NOT FOUND.

Lakehouse Federation is the Databricks query federation platform, providing governed, read-only access to external data through Unity Catalog foreign catalogs. Query federation uses a Unity Catalog connection that stores your credentials and the JDBC URL, and queries are pushed down to the foreign database over JDBC. Catalog federation is the alternative for data in object storage, where queries run directly on Databricks compute.

Query federation supports MySQL, PostgreSQL, Teradata, Oracle, Amazon Redshift, Salesforce Data 360, Snowflake, Microsoft SQL Server, Azure Synapse (SQL Data Warehouse), Google BigQuery, and Databricks. Catalog federation supports the legacy Databricks Hive metastore, external Hive metastores, AWS Glue, Salesforce Data 360, Snowflake, Palantir Foundry, and Workday Data Connect. Federated access is read-only, and it follows Unity Catalog privileges.

A foreign catalog mirrors an external database so you can use Unity Catalog query syntax and governance on it, and it is created as part of setting up a connection, with the foreign catalog step included in the UI flow. Catalog Explorer shows the connection a foreign catalog uses along with the external database it mirrors. On dedicated access mode compute you must own the connection to query the foreign catalog, while standard access mode and serverless compute use standard catalog privileges. Queries against foreign catalogs are read-only and must use Apache Spark data types.

COPY INTO loads data from a file location into a Delta table, and it is a retryable and idempotent operation. Files in the source location that have already been loaded are skipped, even if they have been modified since they were loaded. Supported file formats include CSV, JSON, AVRO, ORC, PARQUET, TEXT, and BINARYFILE.

Yes. COPY INTO is documented as retryable and idempotent, so re-running it after a failure skips the files that were already loaded and loads the rest. Setting the force copy option to true disables this skipping and loads files regardless of whether they were loaded before. For very large directories, Databricks recommends Auto Loader, and it advises against invoking COPY INTO concurrently for performance.

Parameter markers are typed placeholders whose values the invoking API supplies. Named markers use the form :parameter_name, can be referenced more than once, and are available on Databricks Runtime 12.1 and above. Unnamed markers use ? and consume the supplied array of arguments in order of appearance, and they are available on Databricks Runtime 13.3 and above.

No. You cannot mix named and unnamed parameter markers in the same SQL statement, so choose one style for each statement. Every marker needs a bound value, because an unbound marker raises an UNBOUND_SQL_PARAMETER error. Markers can be passed through the Statement Execution API and through SparkSession.sql() in Python, Scala, and Java.

Materialized views support manual refresh with REFRESH MATERIALIZED VIEW, which is synchronous by default and can be made asynchronous with the ASYNC keyword. Standalone materialized views run only in triggered mode, so verify whether a scheduled refresh applies to your object type before you rely on it; TRIGGER ON UPDATE refreshes when upstream data changes. The owner and any user granted the REFRESH privilege can refresh the view, and a REFRESH POLICY in the view definition sets whether incremental or full refresh is preferred.

By default, Databricks uses a cost model to choose the more cost-effective of incremental and full refresh. A full refresh runs when an incremental refresh cannot be performed or is not cost-effective. To check whether a query can be incrementalized, run EXPLAIN CREATE MATERIALIZED VIEW. Incremental refresh needs source Delta tables with row tracking enabled, and change data feed is recommended for better incremental refresh performance.

A streaming table is a Delta table with additional support for streaming and incremental data processing, and it is a good choice for data ingestion. It handles large volumes of append-only data and processes each input only once, which fits most ingestion workloads, and it can read from cloud object storage with Auto Loader or from message buses such as Apache Kafka. When source records change over time, Databricks recommends AUTO CDC instead of plain appends, and a single pipeline owns and updates each streaming table.

A scheduled alert runs its own query and evaluates the condition, resolving each evaluation to OK, TRIGGERED, or ERROR, and you can review past evaluations in the alert history. After a state change, the alert notifies its configured notification destinations.

Query History is the record of previous query runs, and it helps you debug query issues by showing each query's status, duration, user and compute details, and full statement. Query owners can see their own runs, and other users need at least CAN VIEW permission on the SQL warehouse that ran the query. Cancel a running query with the Cancel button next to its status, and open its query profile from the details panel for execution details.

Use the filters near the top of Query History to narrow the list by user, date range, compute, duration, query status (Queued, Running, Finished, Failed, or Cancelled), statement type, statement ID, and query tags. Query tags are one of these filters, so you can isolate workloads that you tagged for cost attribution or team tracking. Statements that run on Lakeflow pipelines compute can only be canceled from the Pipelines UI.

The local cache stores query results in memory for the lifetime of the cluster, or until the cache is full, whichever comes first. It is checked before the remote result cache, and a query runs only when neither cache holds its result. The local cache is cleared when the cluster stops or restarts, and each cache entry lasts 24 hours from when it was created.

The remote result cache is a serverless-only cache that persists results as workspace system data, so it is shared across all warehouses in the workspace. It survives warehouse stops and restarts, and its entries last 24 hours from creation, although a running warehouse is required to access it. It is available to ODBC and JDBC clients and the SQL Statement API, and it is invalidated when the underlying tables are updated.

Run SET use_cached_result = false in the SQL editor to skip result caching for a query. Databricks documents this setting for testing or benchmarking. The disk cache is separate: it stores copies of data files on the local SSD storage of compute nodes, and it invalidates itself when it detects changes to the underlying data files.

The Databricks SQL Statement Execution API 2.0 runs SQL statements on Databricks SQL warehouses over REST, with endpoints to submit a statement, get its status and results, fetch result chunks, and cancel it. A request can wait for the statement to finish up to a configured timeout, or return a statement ID immediately and let you poll for status. Results can come back inline or as external links, and the API supports parameterized queries with :name placeholders, which Databricks strongly recommends to reduce SQL injection risk.

INLINE returns results in the JSON response as JSON_ARRAY, and results over 25 MiB cause the statement to fail. EXTERNAL_LINKS returns presigned cloud-storage URLs, valid for 15 minutes or less, which suits large results and supports JSON, CSV, and ARROW_STREAM formats. Once a statement's format and disposition are set, they cannot be changed.

wait_timeout sets how long the request waits for the statement to finish, from 5 to 50 seconds with a default of 10 seconds, and 0s returns the statement ID immediately in a PENDING state. on_wait_timeout defaults to CONTINUE, which keeps the statement running and returns an ID for polling, while CANCEL stops the statement and returns a CANCELED state. The statement state can be PENDING, RUNNING, SUCCEEDED, FAILED, CANCELED, or CLOSED.

Query tags are custom key-value pairs that group queries by business context, track compute costs, and identify sources of long-running queries. Tags are stored in the query_tags column of system.query.history, where you can filter them with MAP_CONTAINS_KEY or query_tags['key'], and they appear on the Query History page. Tag data is stored as plain text, so keep passwords and personal data out of tag keys and values.

Session-level tags apply to all subsequent statements in the session, and in SQL you set them with SET QUERY_TAGS['team'] = 'marketing'. Statement-level tags apply to a single statement, and the Statement Execution API accepts them in a query_tags list of key and value objects in the request body. Each session can hold up to 10 KB of tags, and each query can have up to 20 user-specified tags.

The SQL editor lets you write, run, and analyze SQL in the Databricks UI, with autocomplete and Genie Code assistance. You can run single or multi-statement queries, browse catalogs and schemas after connecting to a SQL warehouse, and visualize, filter, and download results. Share queries with collaborators, who can edit together in real time, and review changes with version history.

The SQL editor lets you add parameter widgets to a query. The editor overview cited here does not document parameter syntax, types, or default values, so use the parameter documentation for those details. You can also customize SQL auto-formatting options from the editor.

Connect Power BI Desktop to Databricks clusters and SQL warehouses with Partner Connect or a manual configuration. For Power BI tasks that orchestrate semantic models, first create a Power BI connection in Unity Catalog. Databricks recommends the ADBC driver for Power BI, which is also required for Lakehouse Real-Time warehouses.

The Databricks Power BI integration documents three main authentication options: a service principal for machine-to-machine (M2M) OAuth, Microsoft Entra ID as the identity provider for single sign-on, and OAuth with role assumption. With role assumption, you need Assume permission on a role, and Power BI then runs queries as that role. Setting up the service principal for M2M OAuth is the documented first step before you connect.

For metric views in DirectQuery mode, use the connector's Native query option with the MEASURE() function. Microsoft removed the Metric View BI Compatibility Mode option, so reports that used it no longer work and should be rewritten to use Native query. AI/BI dashboards also work natively with metric views.

For Unity Catalog data, Tableau Desktop 2021.4 or above works with the Databricks ODBC driver version 2.6.19 or above. Tableau also offers the Databricks OSS ODBC Driver (Beta) as an alternative for SQL warehouses. OAuth is the recommended authentication method for Tableau Desktop and Tableau Cloud.

The Tableau integration documents OAuth as the recommended method, with Tableau enabled as an OAuth application in your account, and Tableau Desktop and Tableau Cloud have it enabled by default. Personal access tokens are also supported. Service principal OAuth for machine-to-machine authentication requires Tableau Cloud or Desktop 2025.2 or above, or Tableau Server or Bridge 2025.3 or above, along with Databricks ODBC driver 2.8.2 or above.

Datasets are defined from tables, views, or custom queries and power the visualizations in an AI/BI dashboard. AI/BI dashboards add AI-assisted authoring, an enhanced visualization library, and a streamlined configuration experience. Dataset-scoped filters apply to the entire dataset, so every visualization that uses that dataset responds to them.

Publish the dashboard first, then use Share to add users or groups with a permission level, or choose Anyone in my account can view to share with everyone in your account. Account users must first be registered with the account by an administrator, and users without workspace access are limited to CAN RUN permission. To share with everyone in the workspace, use the All workspace users system group.

Shared data permissions are the default, meaning viewers run queries using the publisher's data permissions, so viewers can see the dashboard without direct access to the underlying data. Individual data permissions mean viewers run queries with their own credentials, so they need access to the underlying data but do not need workspace membership. Under individual data permissions, compute access is always granted by the publisher's credentials.

Yes. Only published dashboards can be embedded in external applications, and published dashboards can be embedded in external sites and apps using iframes. Embedding for external users lets people without Databricks accounts view a dashboard, with the embedding application authenticating through a service principal with an OAuth secret. Databricks recommends publishing with individual data permissions so queries run with each viewer's access.

Embedding for external users lets you integrate dashboards into external systems without requiring viewers to have Databricks accounts. The embedding application authenticates with a service principal using an OAuth secret, and tokens for this mode are valid for one hour. Ask Genie is not supported in this mode, so use the Genie Conversation API for conversational features.

Workspace admins define which surfaces are allowed to embed dashboards. For basic embedding, viewers sign in with Databricks credentials, and only users who have been explicitly granted access can view the embedded dashboard. Third-party cookies must be enabled in the browser, or the embedded iframe may appear blank.

Embedded dashboards display in light mode by default. Set the colorScheme option when you initialize the dashboard to light, dark, or light dark, where light dark follows the user's system or browser preference. The option maps to the CSS color-scheme property on the iframe.

AI/BI dashboards support global filters that apply across all pages to visualizations that share a dataset, page-level filters that apply to every visualization on one page that shares a dataset, and widget-level filters that the author sets and viewers cannot change. Authors can also mark a widget as ignoring selected filters or cross-filter selections, which keeps that visualization from responding to them. Dataset scope means a filter applies to the entire dataset, so every visualization built on that dataset is affected.

Cross-filtering lets viewers filter dashboard data by interacting directly with visualizations. Selecting a data point or a legend item filters the dashboard, and it is automatically applied to supported visualizations that use the same dataset, so all other visualizations based on that dataset update. In pivot tables with nested fields, viewers can use Drill down and Drill up on row or column headers.

Field filters support single value, multiple values, date picker, date range picker, text entry, and range slider. Parameter filters support single value, multiple values, date picker, and date range. Both appear as interactive filter widgets, and the filter scope determines which visualizations each filter affects.

Genie is the Databricks AI experience for users, giving everyone in an organization one place to ask data questions in natural language. Genie One is the simplified interface for business users to discover and interact with data assets, while Genie Agents let data teams configure trusted data, metrics, business rules, and verified answers. Genie Code is the AI coding and data assistant for developers and technical practitioners, and Genie One and Genie Agents usage by users is free through January 31, 2027.

PIVOT rotates the unique values of a specified column list into separate columns, with aggregate expressions computing each cell. Its syntax is PIVOT (aggregate_expression FOR column_list IN (expression_list)), and each pivot value must be a literal whose type shares a least common type with its column. Columns that are not used in an aggregate or in the column list become grouping columns, which gives you a wide, aggregated view of the data.

UNPIVOT rotates groups of columns into rows, producing an unpivot column of type STRING that holds the column names and a value column that holds their values. The syntax takes an optional INCLUDE NULLS or EXCLUDE NULLS clause, and the default is EXCLUDE NULLS, which drops rows whose value is NULL. The multi-value form unpivots several value columns at once, and the feature is available on Databricks Runtime 12.2 LTS and above.

When spark.sql.ansi.enabled is true, Spark throws runtime exceptions for invalid operations such as integer overflow and invalid casts like CAST('a' AS INT), instead of returning null. The property is enabled by default in Apache Spark 4.0 and Databricks Runtime 17.0 and above. If you want null results instead, try_cast, try_add, and try_divide return NULL rather than throwing an exception.

VARIANT stores semi-structured data and is available in Databricks Runtime 15.4 and above. Extract top-level fields with colon syntax such as raw:owner, and nested fields with dot or bracket notation, with array indices starting at 0. Field names are case sensitive. variant_get extracts a field by path with a target type, and try_variant_get returns NULL instead of failing when a cast does not succeed.

Window functions compute a result for each row from a window of related rows, defined with OVER and optional PARTITION BY, ORDER BY, and frame clauses. Ranking functions require an ORDER BY in the window specification, and analytic functions access values from other rows by position relative to the current row, such as LAG and LEAD. Aggregate functions can also be used as window functions, but they cannot include a FILTER clause.

transform(expr, func) is a higher-order function that applies a lambda function to each element of an array and returns the resulting array. The lambda takes the element as its first parameter, and an optional second parameter gives the element's index. Use it to change array elements in a single expression, without exploding the array into rows.

h3_longlatash3 returns the H3 cell ID as a BIGINT for the given longitude and latitude in degrees, using WGS84 coordinates. The resolution argument is an INT from 0 to 15 inclusive, and values outside that range raise the H3_INVALID_RESOLUTION_VALUE error. The function returns NULL if any input is NULL, and it is available on Databricks Runtime 11.3 LTS and above.

GROUP BY supports GROUPING SETS, ROLLUP, and CUBE. ROLLUP(a, b) produces N+1 grouping sets for N elements, CUBE produces 2^N grouping sets, and GROUPING SETS lets you list the exact combinations you need. Mixed and nested grouping clauses can appear in one query, which lets a single query return several levels of subtotals.

ILIKE returns true when str matches pattern case-insensitively, and it is the case-insensitive counterpart of LIKE. An underscore matches one character and a percent sign matches zero or more characters, and the default escape character is the backslash. It supports ANY, SOME, and ALL quantifiers when you check several patterns, and NOT ILIKE negates the match.

A common table expression defines a temporary result set that you can reference multiple times within a SQL statement, most commonly in a SELECT. Adding RECURSIVE lets a CTE refer to itself, which supports graph and tree traversal, and recursive CTEs are available on Databricks Runtime 17.0 and later. The default recursion limit is 100 levels, and MAX RECURSION LEVEL changes it.

ai_query calls an existing Model Serving endpoint and returns its parsed response, using the form ai_query(endpoint, request). Endpoints can be Databricks-hosted Foundation Model endpoints, external model endpoints, or custom model endpoints in the same workspace, and the definer needs CAN QUERY permission on the endpoint. With failOnError set to false, rows that fail because of endpoint errors return a null response instead of failing the whole query. It requires Databricks Runtime 15.4 LTS or above and is not available on Databricks SQL Classic.

Databricks provides built-in AI Functions for SQL, which require Databricks Runtime 15.4 LTS or above and are not available on Classic SQL warehouses. The general-purpose function is ai_query, and document functions include ai_parse_document, ai_extract, and ai_classify. Text functions include ai_summarize, ai_translate, ai_fix_grammar, and ai_analyze_sentiment, and ai_forecast handles time series forecasting. Several functions, including ai_search, ai_enrich, and ai_decide, are in Beta.

Yes, you can create a SQL user-defined function with CREATE FUNCTION. It creates a scalar function that returns a single value, or a table function declared with RETURNS TABLE whose body is a query. DEFAULT parameter values are supported for SQL functions on Databricks Runtime 10.4 LTS and above.

CREATE PROCEDURE persists a SQL script in Unity Catalog as a procedure that takes or modifies arguments, runs SQL statements, and optionally returns a result set. It applies to Databricks SQL and Databricks Runtime 17.0 and above, and you invoke it with CALL, as in CALL add(1, 2, sum, total). Access is granted with GRANT, and the SQL SECURITY INVOKER or DEFINER characteristic controls whose authority the body runs under.

Databricks provides an ODBC driver (the Databricks ODBC Driver) and a JDBC driver (the Databricks JDBC Driver) to connect your tools and clients to Databricks SQL warehouses. Each driver has getting-started guidance linked from the page, and instructions for specific tools appear under technology partners or in each tool's own documentation. The page lists both drivers together, so choose the one your client or tool requires.

STATEMENT_TIMEOUT limits how long a statement can run, in seconds. Session values set with SET STATEMENT_TIMEOUT apply first, then warehouse-level values, then workspace values, then the system default of 172800 seconds (48 hours). Warehouse-level timeouts are in Beta and require a workspace admin to enable the Warehouse Statement Timeout preview. For example, SET STATEMENT_TIMEOUT = 900 sets a 15-minute limit for the current session.

Check waiting_at_capacity_duration_ms in the query history: high values mean queries are queued instead of running, and the remedy is to increase the warehouse max_clusters setting so it can scale. Excessive spill to disk points to memory pressure, which is addressed by optimizing the query or increasing the warehouse size. These are different fixes, so scaling out clusters will not solve a spill problem.

Query Profile visualizes query execution details and lets you inspect each operator's metrics, such as time and rows processed, to find bottlenecks like exploding joins or full table scans. From Query History, click the query's name and then See query profile.

Materialization for metric views uses materialized views to pre-compute aggregations, and Databricks refreshes them on a schedule set in the metric view's materialization configuration. The optimizer rewrites queries to a suitable materialization automatically, trying an exact match, then a rollup match, then an unaggregated match, and reading source tables if none applies. An aggregated materialization pre-computes specific dimensions and measures, while an unaggregated one materializes the full data model, and each metric view allows only one unaggregated materialization.

Serverless warehouses use intelligent workload management, which predicts query resource needs and manages compute automatically. When no capacity is available, the query is placed in a queue, and the autoscaler adds clusters when wait times increase and scales down when demand drops. Configure a sufficient maximum number of clusters to handle your peak load.

A stopped warehouse restarts automatically when a query runs against it, when a scheduled job targets it, when a JDBC or ODBC connection is made, or when a dashboard that uses it is opened. Keep these triggers in mind when you schedule dashboards and jobs that point at a warehouse you expect to stay stopped.

The wait_timeout parameter defaults to 10 seconds, and on_wait_timeout defaults to CONTINUE, so execution continues asynchronously and the call returns a statement ID you can poll. Setting on_wait_timeout to CANCEL instead cancels the statement when the wait deadline passes and returns a CANCELED state. Use CONTINUE for long-running queries and CANCEL when you want a request bounded by its wait time.

SQL pipe syntax lets you write a query as a chain of operators joined by |>, such as FROM, JOIN, AGGREGATE, and ORDER BY, with each stage working on the previous stage's result. Databricks documents it as supported in Databricks SQL and Databricks Runtime 16.2 and above.

Declare a session variable with DECLARE VARIABLE, for example DECLARE OR REPLACE VARIABLE myvar INT DEFAULT 17, and change it with SET VAR. Variables persist across statements, can be referenced in temporary views and SQL functions, and work with the IDENTIFIER clause. They are dropped when the session ends, or earlier with DROP VARIABLE. The feature applies to Databricks Runtime 14.1 and above.

Yes. Python user-defined table functions (UDTFs) return rows instead of a single value. Decorate a class with @udtf, define an eval method that yields rows, register it with spark.udtf.register, then call it in a FROM clause. Python UDTFs are in Public Preview on Databricks Runtime 14.3 LTS and above, and Unity Catalog UDTFs require Runtime 18 or above. Table inputs are passed with TABLE(t) and can be partitioned with PARTITION BY.

Yes, scheduled AI/BI dashboard snapshots can be delivered to Slack and Microsoft Teams channels as well as email. Workspace admins set up the Slack and Teams destinations first. A subscriber list holds up to 100 subscribers, and Excel data exports are capped at 100,000 rows.

Use dataset materialization for AI/BI dashboards, which is in Beta. A workspace admin enables AI/BI Dashboard Materialization under Previews. When you publish with materialization on, the dataset results are stored, so viewers read stored results instead of rerunning expensive queries. Refreshes follow the dashboard schedule, a daily one is added if none exists, and it requires serverless compute and datasets without parameters.

Yes, on Databricks Runtime 19 and above with YAML specification version 1.1 or later. Metric views support window measures, which define rolling, cumulative and current-row aggregations in one governed definition. You set an order field and a range: trailing for a rolling window, cumulative for a running total, or current with semiadditive last for a snapshot balance. Numeric offsets and ranges require an integral order column (TINYINT, SMALLINT, INT, or BIGINT), so check your order column against that rule for the range you choose.

Yes. Local metric views let you define fields, measures and join relationships visually inside an AI/BI dashboard, without publishing to Unity Catalog first. The logic stays scoped to that dashboard. When a definition is ready for wider use, you can export it to Unity Catalog as a global metric view that other dashboards and tools can use.

Use the /importBI command in Genie Code. Tableau workbooks (.twb, .twbx) and Power BI templates (.pbit) become an AI/BI dashboard plus metric views, and Power BI data models (.tmdl) become metric views. Direct uploads are limited to 100 MB, larger files can be referenced from a Unity Catalog volume, and partner-powered AI must be enabled for both the account and the workspace.

Lakehouse Real-Time (Beta) is a warehouse type for AI/BI dashboards that need sub-second responses for hundreds to thousands of concurrent users. It works best for selective queries that read small amounts of data. Its Beta guardrails are a 2-minute query runtime, a 1-minute default statement timeout (configurable up to 2 minutes), 300 GB of shuffle bytes written per stage, 30 GB of spill when enabled, and 10,000 files after pruning (adjustable up to 50,000); these values may change. It requires account enablement, supports only Unity Catalog managed tables and ANSI-compliant SELECT statements, and does not support Genie or local metric view materialization.

Use time_bucket(bucketSize, ts, origin). It returns the start of the bucket that contains the timestamp. The origin defaults to 1970-01-01 00:00:00, and it does not need to precede the timestamp, so you can shift bucket boundaries. For example, aligning hourly buckets to 5 minutes past each hour returns 11:05 for 11:27. Buckets include their start and exclude their end. The function is available in Databricks SQL and Databricks Runtime 19 and above.

Use the counter_diff window function, partitioned by the metric name and its attributes and ordered by time. It converts cumulative counter readings into differences. When a counter resets, detected when the value decreases or a later start_time advances, it returns NULL instead of a misleading delta. An ORDER BY clause is required. The function is available in Databricks SQL and Databricks Runtime 19 and above.

On classic compute, Databricks Runtime 18.1 and above supports more than five UDF calls in one query. On serverless compute and SQL warehouses, some features require an explicit environment_version, set to the required version or higher in each UDF definition. The ENVIRONMENT clause requires environment_version whenever it is present.

Run time varies with data layout, query plan, and how long a query waits for compute. Databricks exposes both. The query history system table (system.query.history) reports waiting_for_compute_duration_ms and waiting_at_capacity_duration_ms separately from execution_duration_ms, so queue wait and actual work can be told apart. It also reports read_bytes, spilled_local_bytes and read_io_cache_percent, which help explain slow runs. Query Profile shows bytes spilled to disk for a query.

Not always. A larger warehouse adds compute to each cluster, which helps most when queries spill to disk. Databricks guidance is to start with a single larger serverless warehouse and let serverless features manage concurrency, to increase size if queries spill to disk, and to size down if necessary, since it is usually more efficient to size down than to start small and scale up. Check Bytes spilled to disk in Query Profile to see whether a warehouse is too small.

Yes, in Beta. Dataset materialization stores the full result of a SQL dataset and refreshes it on the dashboard's refresh schedule, so published dashboards render from stored results instead of re-running heavy joins on every load or filter. Publishing with materialization adds a daily schedule if the dashboard has none. A workspace admin enables AI/BI Dashboard Materialization from the Previews page.

Yes, in Public Preview. Scheduled refresh reruns a saved query, table, or pivot table import so the workbook stays current. The workbook must be saved in OneDrive or SharePoint, and each schedule uses a SQL warehouse you choose. Refreshes run under the Databricks identity and Microsoft credentials of the person who created the schedule, and run history shows each run as OK, Running, or Failed.

Yes. You can save the imports in a workbook as a template stored in Unity Catalog, which keeps the query definitions but not the imported data. Colleagues import the template into any sheet, and each person runs the queries with their own Unity Catalog permissions. The connector grants SELECT on the template, and an admin must grant users USE CATALOG and USE SCHEMA. To change a query, recreate the template.

ADBC, Arrow Database Connectivity, is the default for all new Power BI connections created in the UI. Existing connections keep using ODBC until you switch them, and Databricks recommends moving to ADBC. It moves data in the Apache Arrow columnar format and requires Power BI Desktop 2.145.1105.0 or later. ADBC is also required for Lakehouse//RT, which accepts only the REST-based Statement Execution API.

The query history system table splits each statement's time into parts: waiting_for_compute_duration_ms, waiting_at_capacity_duration_ms, compilation_duration_ms, and execution_duration_ms. total_duration_ms excludes result fetch time, which the table reports separately in result_fetch_duration_ms, so compare it with your client's wall-clock timing. If the waiting columns are near zero, the remaining time sits in compilation, execution, or result fetch, and the breakdown shows which one.

Databricks documents serverless SQL warehouse startup as typically 2 to 6 seconds, versus approximately 4 minutes for pro and classic warehouses. The word typically matters: startup is not a guaranteed ceiling, and first-query latency also depends on your data, caching, and concurrency. This page does not include a head-to-head against Snowflake, so the comparison is best run on your own queries. Measure first-query latency on a stopped warehouse with a representative workload, and compare that number rather than a vendor claim.

Databricks SQL runs on the same lakehouse data as your notebooks and pipelines, using serverless SQL warehouses with Photon as the execution engine. Photon is a native C++ vectorized runtime: Spark's Catalyst optimizer still plans the query, and Photon executes it, with the greatest benefit on longer-running queries over large datasets. If Photon cannot run part of an operation, it falls back to the Spark runtime for the rest of that operation, so you do not rewrite SQL to use it. The honest gap is that Photon does not accelerate UDFs or RDD and Dataset APIs, so check feature coverage and performance against your own SQL and procedural workload with a proof of concept.