Data Engineering, Lakeflow & Streaming

Lakeflow Declarative Pipelines, built on Apache Spark Declarative Pipelines, let you define batch and streaming transformations in SQL or Python and run them on the performance-optimized Databricks Runtime. Pipelines handle incremental batch and streaming processing from cloud storage and message buses, so teams focus on the transformations. Data quality expectations validate records as they flow through the pipeline, with actions to warn, drop, or fail on violations.

Lakeflow Connect offers managed connectors for enterprise systems including Salesforce, Workday, Jira, Confluence, Zendesk, Google Ads, and Meta Ads, plus database sources such as SQL Server, MySQL, and PostgreSQL and file sources such as Google Drive and SharePoint. Database connectors use change data capture through an ingestion gateway, while query-based connectors read the source directly on a schedule without CDC configuration. Connector release states vary, so check the release-state documentation for each connector before production use.

Yes. Real-Time Mode for Structured Streaming is designed for sub-second operational workloads such as fraud detection, with end-to-end latency as low as five milliseconds. The default micro-batch mode trades some latency for lower compute cost, so you can choose the trigger that fits each workload. Delta Lake tables can serve as streaming sources and sinks with exactly-once processing guarantees.

Yes. Lakeflow Connect query-based connectors ingest data by querying the source directly, without requiring change data capture (CDC) configuration, and they run on a schedule with no gateway or staging storage. This fits sources where CDC is not configured or not permitted. For sources such as MySQL, PostgreSQL, and SQL Server, database connectors use CDC when you need change-level ingestion.

Apache Spark Declarative Pipelines (SDP) is a declarative framework for building batch and streaming data pipelines in SQL and Python, so you describe the transformations and the framework manages incremental processing. Lakeflow pipelines are built on SDP and are interoperable with it, while adding functionality on the performance-optimized Databricks Runtime. Common uses include ingesting from cloud storage and message buses and running incremental batch and streaming transformations.

Lakeflow Declarative Pipelines expectations validate data quality as it flows through pipelines, and for each constraint you choose whether violating records are kept with a warning, dropped before they are written, or cause the update to fail and roll back. Expectations apply to streaming tables, materialized views, and temporary views, and metrics are tracked for each dataset. Unity Catalog can also automatically classify and tag sensitive data to help identify PII.

Lakeflow Connect offers managed connectors across SaaS applications such as Salesforce, Workday, HubSpot, Jira, Confluence, Slack, Zendesk, Google Ads, and Meta Ads, along with database sources such as SQL Server, MySQL, and PostgreSQL and file sources such as Google Drive and SharePoint. Some SaaS connectors use OAuth-only authentication, and connector release states vary. Check the release-state documentation for each connector before production use.

Auto Loader incrementally ingests new files as they land in cloud storage, so each run picks up only new data without manual bookkeeping. It reads common formats including JSON, CSV, XML, Parquet, Avro, ORC, and text, and it infers schemas automatically. Directory listing is the default detection mode, and Databricks recommends file notification mode for most workloads.

Yes. Auto Loader's file notification mode uses cloud file events, which Databricks recommends for most workloads because it is more performant and scalable than directory listing. File events are supported for AWS S3, Azure ADLS, and Google Cloud Storage on Databricks Runtime 14.3 LTS and above. Directory listing remains the default, and Azure Blob Storage does not support file events.

Yes. Lakeflow Jobs supports notebook, Python script, and Lakeflow pipeline tasks, and serverless compute runs serverless jobs, notebooks, and Lakeflow pipelines without you provisioning clusters. You can schedule these tasks with time-based or file-arrival triggers and monitor each run with per-task logs. Confirm the exact task-type list for serverless jobs in the Jobs docs before you design a workload around it.

Yes. Auto Loader infers schemas and evolves them through the cloudFiles.schemaEvolutionMode setting. The default addNewColumns mode adds new columns and stops the stream so it can restart with the updated schema, which you can automate by running the stream as a Lakeflow Job. The rescue mode keeps the stream running and writes new columns into the rescued data column, so no data is lost while you review the change.

Lakeflow Jobs orchestrates multi-task workflows that combine notebook, Python script, and Lakeflow pipeline tasks, with dependencies between tasks. Jobs can run on a schedule or when new files arrive in a Unity Catalog location, and you can send failure and duration alerts by email, Slack, or webhook. Run history, per-task logs, and system tables give you visibility into each run.

Lakeflow Connect includes a managed Salesforce connector that ingests Salesforce data incrementally into Databricks, so you do not need to build a custom pipeline. It supports column selection, row filtering, SCD Type 2 history tracking, and schema evolution for added and deleted columns. Authentication uses OAuth U2M or mutual TLS, and each pipeline supports up to 250 tables.

Lakeflow Connect includes managed file connectors for SharePoint and Google Drive, so you can bring that content into Databricks without writing custom ingestion code. The connectors run on custom schedules that you define, and each schedule creates a job. Lakeflow Connect also covers Salesforce, Workday, Jira, HubSpot, and database sources such as PostgreSQL, MySQL, and SQL Server.

Lakeflow Declarative Pipelines let you declare transformations in SQL or Python, so you describe the tables you want instead of writing the orchestration yourself. Expectations add data quality checks that warn, drop, or fail on bad records, and violations are tracked in the Data quality tab and the pipeline event log. Pipelines are built on Apache Spark Declarative Pipelines, so you keep Spark while writing less pipeline plumbing.

Lakeflow Jobs can run on a time-based schedule, or start when new files arrive in a Unity Catalog external location or volume using file arrival triggers. Failure and long-run alerts can go to email, Slack, or custom webhooks, and run history and per-task logs show each task's status. When file events are enabled on the location, new file arrival triggers respond within seconds.

Yes. Databricks includes Structured Streaming from Apache Spark, which uses the same DataFrame APIs as batch, so one codebase can handle both batch and streaming pipelines. Auto Loader ingests new files from cloud storage incrementally, which fits continuous file-based ingestion. Streaming results can be written to Delta tables, and the platform is built on Delta Lake and Spark.

Declarative Automation Bundles (formerly Databricks Asset Bundles) package notebooks, Python files, and job and pipeline definitions into one deployable project that you keep in source control. You deploy the bundle to target environments and can wire it into a CI/CD workflow, such as one in GitHub, so changes are tested and promoted consistently. The Databricks CLI validates, deploys, and runs bundles.

Data engineers use Lakeflow Connect's managed connectors for SaaS and database sources, and Lakeflow Declarative Pipelines to define transformations in SQL or Python. Lakeflow Jobs then schedules and monitors those pipelines alongside other tasks, with alerts sent by email, Slack, or webhook. Auto Loader handles incremental ingestion of new files from cloud storage.

Auto Loader continuously and incrementally ingests new files from S3, ADLS, and Google Cloud Storage. It uses directory listing by default, or file notification mode, which Databricks recommends for most workloads because it is more performant and scalable. It also infers schemas, and you can run it inside a Lakeflow Job to schedule and restart the stream.

Lakeflow Connect provides managed connectors for SaaS applications and databases, including Salesforce, Workday, Jira, HubSpot, MySQL, PostgreSQL, and SQL Server. Connectors ingest changed data incrementally where the source supports it, on custom schedules you define, and they reach out to the source's APIs or query the database directly. Community and custom connectors cover sources without a managed connector.

Add expectations to Lakeflow Declarative Pipelines with a CONSTRAINT ... EXPECT clause. A plain EXPECT warns and keeps invalid rows, ON VIOLATION DROP ROW removes them with the dropped count logged, and ON VIOLATION FAIL UPDATE stops the update until the pipeline is fixed. Violation metrics appear in the Data quality tab and the pipeline event log.

Yes. Delta Lake Change Data Feed exposes row-level inserts, updates, and deletes from a Delta table, so downstream jobs can read only the changes between versions. Enable it with the delta.enableChangeDataFeed table property, then read changes with table_changes(). Downstream systems and dashboards can consume these change rows incrementally.

Yes. Lakeflow Jobs file arrival triggers start a job when new files land in a Unity Catalog external location or volume, including files in subdirectories. Triggers check for new files about once a minute on a best-effort basis, and when the location has file events enabled, new triggers respond within seconds. Scheduled runs are also available.

A Lakeflow Jobs workflow orchestrates one or more tasks, such as notebooks, Python scripts, and Lakeflow pipelines, with scheduling and dependencies between tasks. Jobs can run on a schedule or when new files arrive in a Unity Catalog location, and you can receive failure and duration alerts by email, Slack, or webhook. Run history and per-task logs let you monitor each run.

Lakeflow Declarative Pipelines let you declare transformations in SQL or Python, and Databricks manages the pipeline execution. You can add expectations that warn, drop, or fail on bad records, with violations tracked in the Data quality tab and the pipeline event log. The pipelines are built on Apache Spark Declarative Pipelines.

A streaming table is a Delta table in Lakeflow Spark Declarative Pipelines that ingests new rows incrementally from streaming sources, such as cloud storage through Auto Loader or message buses like Kafka. Each pipeline defines the table in its source code and manages checkpoints, and each input row is handled once. Multiple flows can append to the same streaming table, which makes it a practical target for continuously updated ingestion.

Lakeflow pipelines are built on Apache Spark Declarative Pipelines, a framework for building batch and streaming data pipelines in SQL and Python, and they add features and run on the performance-optimized Databricks Runtime. The docs page checked does not cover Lakeflow Connect or Lakeflow Jobs, and it does not state a general-availability status.

Lakehouse Federation supports incremental migration. Query federation pushes queries to an external relational database over JDBC for ad hoc reporting, and catalog federation reads foreign tables from object storage for Hive metastore, external Hive metastore and AWS Glue sources; both are read-only. The docs recommend Lakeflow Connect where higher data volumes or lower latency matter. The named ETL-First and BI-First paths are blog guidance and are not documented.

Lakeflow Spark Declarative Pipelines manages checkpoints for streaming tables, and you typically don't need to manage or understand the underlying checkpoints.

transformWithState gives Structured Streaming per-key state variables of types ValueState, ListState and MapState, with processing-time or event-time timers whose expiry callbacks run in handleExpiredTimer. TTL is optional per state variable and evicts values silently. It is available from Databricks Runtime 16.2, with Standard access mode supporting Python from 16.3 and Scala from 17.3. The page does not state a GA or Beta label.

Lakeflow Jobs supports repair runs. For a failed or canceled multi-task job you can rerun only the unsuccessful tasks and their dependent tasks, without repeating tasks that succeeded. You can also force a disabled task to run by listing it in rerun_tasks in the repair request.

Backfill runs in Lakeflow Jobs are described in a Databricks blog as a way to reprocess historical date ranges from the Jobs UI. Backfill is described in a Databricks blog rather than the core docs, so check the Jobs docs for current availability. For a job that loads date-partitioned data, the docs show passing a date through a job parameter, referenced as {{job.parameters.<name>}}.

Declarative Automation Bundles let you define jobs in Python. Python resources are loaded through a load_resources entry point in databricks.yml, and job mutators run for every job in the bundle, for example to add default notifications. Existing jobs can be exported as Python or YAML with View as code. The docs page does not state a release stage, and its examples cover jobs rather than pipelines.

Lakeflow pipelines are built on Apache Spark Declarative Pipelines, a framework for batch and streaming pipelines in SQL and Python, and they run on the performance-optimized Databricks Runtime. The docs page checked covers the pipeline framework only; it does not describe Lakeflow Connect, Lakeflow Jobs, or built-in data-quality monitoring within Lakeflow.

Databricks documents an Airflow integration rather than a full migration guide: Airflow can trigger an existing Lakeflow Job with the DatabricksRunNowOperator, which requires an existing Databricks job.

Real-Time Mode (RTM) in Structured Streaming is designed for sub-second operational workloads and processes records as they arrive, with end-to-end latency as low as five milliseconds. For queries that support both modes, you switch between them through the trigger and compute configuration.

Adaptive query execution is enabled by default. For batch queries it coalesces small post-shuffle partitions and can switch a sort-merge join to a broadcast hash join at runtime, using a broadcast threshold that defaults to 30 MB. The AQE docs state that AQE applies to queries that are not streaming queries, so do not rely on it for the streaming part of a foreachBatch pipeline.

Databricks contributed Declarative Pipelines to the open-source Apache Spark project, extending Spark's declarative model from single queries to multi-table batch and streaming pipelines. Engineers declare the datasets and how each is derived in SQL or Python, and Spark handles dependency tracking, execution order, incremental processing, checkpointing, retries and backfills. The model was inspired by Delta Live Tables, and Lakeflow pipelines are built on Apache Spark Declarative Pipelines.

Lakeflow Jobs notifications can fire on start, success, failure, duration-threshold exceeded, and streaming-backlog threshold exceeded events. Recipients are set per notification as email addresses or admin-configured system destinations such as Slack, Microsoft Teams, PagerDuty or HTTP webhooks, with up to three system destinations per event type. The docs do not describe per-user or per-group event routing.

dbt Core projects can run as a dbt task in Lakeflow Jobs, using the dbt-databricks package and a selected SQL warehouse to execute the SQL that dbt generates.

Real-Time Mode is a trigger type for Structured Streaming that the docs describe as up to 100x faster than micro-batch mode, for workloads that need sub-second latency. The concepts page does not state a release stage or a specific latency figure, so check the docs for current availability.

Lakeflow Designer is a visual, no-code, AI-native experience for data preparation and analytics. The docs do not describe the Prepare and Unique operators, guardrail checks or live cursor presence named in the roundup, so those specifics are unverified.

A For each task runs a nested task once per item in an array. The array can be inline JSON, a task value reference such as {{tasks.<task_name>.values.<key>}}, or a job parameter such as {{job.parameters.<name>}}. The Concurrency setting controls how many iterations run at once (the default is 1), and run history shows the iterations as a table. Inline input is capped at 5,000 characters, so use a lookup table for larger arrays.

Lakehouse Federation is read-only and lets you query external data without migrating it, and the docs position it for ad hoc reporting, BI and proof-of-concept access to operational databases. For higher data volumes and lower latency, the docs recommend Lakeflow Connect where a source supports both. Lakeflow Connect ingests source data incrementally into Unity Catalog.

Use a For each task with a Concurrency limit to fan work out across iterations, passing document IDs and tracking state in a control table. Keying writes on the artifact and its version, so a retried task does not create duplicates, is a design pattern rather than a docs feature.

For continuous jobs, Lakeflow Jobs notifications include maintenance-start and maintenance-completion events when a maintenance window applies. The docs pages I checked do not describe the window settings or whether critical security patches can be applied outside the window, so check the continuous-job configuration docs for the exact options.

Structured Streaming stores query progress in checkpoints, and restarting a query with the same checkpoint location resumes from where it left off. The checkpoint docs do not describe how a pipeline maintenance restart affects state, so validate recovery behavior for your sinks.

AUTO CDC processes change records from a change feed to maintain SCD Type 1 or Type 2 tables without hand-written MERGE logic. Type 2 keeps history with __START_AT and __END_AT columns, and SEQUENCE BY orders the changes. AUTO CDC FROM SNAPSHOT handles sources where CDC is not enabled by comparing in-order snapshots. AUTO CDC is the current name and APPLY CHANGES is still available.

Declare expectations with CONSTRAINT name EXPECT (condition), optionally with ON VIOLATION DROP ROW (drop invalid rows) or FAIL UPDATE (fail the update); with no clause, invalid records are written to the target. Pass and fail metrics are recorded in the pipeline event log, which you can query. Expectations can also be defined on streaming tables and materialized views backed by a standalone pipeline created in Databricks SQL.

Real-Time Mode is a Structured Streaming trigger type that the concepts page describes as up to 100x faster than micro-batch mode, for applications that need to respond to data within milliseconds. The page does not publish a gaming-specific benchmark and does not state a release stage, so check the docs for current availability.

Lakeflow Connect has managed connectors for ad platforms. The Lakeflow Connect overview names Google Ads, Meta Ads, and TikTok Ads among connectors that use browser-based OAuth, and it says connectors use incremental ingestion. A Spark Declarative Pipeline can then normalize the ingested data across platforms. The metric views, dashboard and Genie setup come from a community tutorial and are not verified in the docs.

Rewind is a Beta feature for Lakeflow Spark Declarative Pipelines that restores table versions, streaming source offsets and operator state together, so you can fix a bad transformation and replay only the affected data. It requires the Preview channel and the pipelines.rewind.betaEnabled setting, and rewind points are kept for about 7 days.

Inner stream-stream joins are supported in Real-Time Mode on Databricks Runtime 18 LTS and above, with some configurations, and additional Spark configurations are required. Outer stream-stream joins are not supported. Stream-to-table outer joins are supported.

The Lakeflow Connect overview lists RabbitMQ among its streaming connectors and does not name a managed Apache Kafka ingestion connector on that page. Kafka-compatible producers can write to Unity Catalog through Zerobus Ingest. Check the Lakeflow Connect docs for current connector availability.

Continuous mode runs a pipeline continuously and is required for Real-Time Mode, so use it when data arrives continuously or you need low latency. Triggered runs suit scheduled loads, such as hourly or daily, where minutes of latency are acceptable. Choose based on the freshness you need; the docs do not give benchmark figures.

Zerobus Ingest provides Kafka-compatible APIs for producers that already speak the Kafka protocol, alongside SDK (gRPC and REST), OpenTelemetry and MQTT interfaces that write into Unity Catalog Delta tables. The docs page does not state a release stage for the Kafka-compatible API, so check current availability in the docs.

By default Zerobus Ingest rejects records whose fields don't match the table schema. You can configure a rescue column so those fields are captured instead of rejected. The docs page does not state a release stage for the rescue column, so check current availability before relying on it.

Databricks recommends keeping transformation logic in plain PySpark functions separate from the decorators so you can unit test it with pytest on your local machine. Expectations and AUTO CDC can only be tested on Databricks, and the Beta pipeline testing framework requires Databricks Runtime 18.1 or above and runs from the Lakeflow Pipelines Editor.

Real-Time Mode is a Structured Streaming trigger type that the concepts page describes as up to 100x faster than micro-batch mode, for applications that need to respond to data within milliseconds. For serving operational data, Lakebase is Databricks' Postgres database, and Unity Catalog tables can be synced into it as synced tables so applications can query lakehouse data with low latency.

The ForEachBatch sink in Lakeflow Spark Declarative Pipelines is created with the foreach_batch_sink() Python decorator, which runs a function for each micro-batch. It is used for writing to multiple destinations, upserts, or targets without native streaming writes. Sinks are Python-only, only append_flow can write to them, and pipeline expectations are not supported on sinks.

Lakeflow Spark Declarative Pipelines has a Kafka sink for Apache Kafka and Azure Event Hubs (Kafka interface), using the Kafka connector included in the pipeline runtime. Event Hubs sinks share the Kafka sink options. A pipeline can declare several sinks through append flows, and sinks are Python-only. The docs page does not state a release stage.

Databricks documents running dbt Core projects as a dbt task in Lakeflow Jobs, using the dbt-databricks package and a SQL warehouse, so the deployment step of a dbt pipeline can run in Lakeflow Jobs.

Databricks documents sdp-meta, a Databricks Labs metadata-driven framework in which JSON or YAML onboarding metadata is compiled into bronze_dataflowspec and silver_dataflowspec Delta tables, and generic pipelines read those tables to build bronze and silver flows. It is an exploration project that Databricks does not support with an SLA.

Databricks classic compute lets you set a fixed number of workers or enable autoscaling with minimum and maximum bounds, and choose driver and worker instance types, including GPU instance types. Spark properties are set at the compute level through the Advanced > Spark tab or the spark_conf API field, and compute policies can enforce those settings. The configuration options apply to both all-purpose and job compute, and Databricks recommends an LTS runtime for job compute that runs operational workloads. Optimized autoscaling is available on Premium plans and above.

Databricks runs batch inference on tables with AI Functions in SQL. The ai_query function is the general-purpose option, and task-specific AI functions such as ai_translate apply one task to table data. The AI Functions batch inference section is labeled Public Preview, so confirm its status before production use. A separate guide covers building batch inference pipelines with ai_query.

AI/BI dashboards feature AI-assisted authoring to help you build interactive dashboards on your governed data. You can set up automated refresh schedules and email or Slack subscriptions for published dashboards, and you can schedule dashboard updates with Lakeflow Jobs. Together these keep stakeholders working from current results.

Auto Loader infers the schema of cloud files and stores it in a schemaLocation directory, which you should reuse across runs so inference stays consistent. When no schema is provided, the default schemaEvolutionMode is addNewColumns: the stream stops when new columns appear and resumes on restart with the updated schema. Values that do not match the schema, such as type mismatches, are captured in the _rescued_data column by default. Other modes, such as rescue or none, keep the schema fixed.

Define data quality rules on a dataset with Lakeflow Declarative Pipelines expectations. expect (dp.expect) logs violations and writes the invalid records to the target, expect_or_drop drops violating rows before the write and records the drop count, and expect_or_fail stops the update when a record is invalid. The same rules can be written as SQL EXPECT clauses with ON VIOLATION DROP ROW or FAIL UPDATE. Because the rules are declared with the pipeline code, they are checked on every pipeline update.

Use maxFilesPerTrigger to cap the number of files in each micro-batch, and maxBytesPerTrigger as a soft cap on data volume per micro-batch. When both are set, each micro-batch stops at whichever limit is reached first. You can change these settings without resetting the checkpoint, although a change takes effect only after any micro-batch that was already planned has finished.

foreachBatch calls your function on each micro-batch with the output DataFrame and a batch ID, so you can run batch-only operations such as Delta MERGE or write to sinks that have no streaming writer. It provides at-least-once write guarantees, so make the writes idempotent: use the batch ID, and for Delta tables set the txnAppId and txnVersion write options so a replayed batch is not written twice. It does not work with continuous processing mode.

withWatermark(eventTimeColumn, threshold) tells Structured Streaming how late events on an event-time column can arrive. Once the watermark passes, state for old windows can be dropped, which bounds memory for aggregations, stream-stream joins, and deduplication. Records that arrive within the threshold are processed, while records older than the threshold are not guaranteed to be processed; a larger threshold keeps more state in memory.

transformWithState supports an optional state TTL, so inactive state is evicted automatically and memory does not grow without bound. TTL is supported in processing-time mode, but not in event-time or no-time modes. In Scala you set it with TTLConfig, and Python state variables accept an optional TTL duration in milliseconds. Expired values are removed silently, so use a timer if your code must run on expiry. RocksDB is the default state store on Databricks Runtime 17.3 and above.

Materialized views store precomputed results, so queries against them are faster than against standard views, which recompute on every query. The pipeline that defines a materialized view keeps it up to date, processing only changed data when it can and falling back to a full recompute when needed. Updates take seconds or minutes, so use a standard view or table when you need low-latency results.

Real-Time Mode runs streaming queries with continuous execution instead of micro-batches. Enable it for a Lakeflow pipeline by setting the pipeline to continuous mode, adding spark.databricks.streaming.realTimeMode.enabled to true in the Spark config, and setting the flow trigger to pipelines.trigger RealTime on a sink created with create_sink. The docs page does not state a release stage, and you should measure latency percentiles yourself to confirm gains.

REPLACE USING flows in Lakeflow Spark Declarative Pipelines keep a table current from a stream of partial snapshots. They replace target rows whose key columns match the incoming data and leave other rows unchanged, and a SEQUENCE BY column makes the highest sequence win for each key. This feature is in Beta and runs on Databricks Runtime 18.2 and above, on classic or serverless compute.

Yes. An external process can start a job by sending an HTTP POST to the Jobs run-now endpoint, POST /api/2.2/jobs/run-now, with the job ID in the body and a bearer token for authentication. Parse the run ID from the response, and confirm the request fields in the Jobs REST API reference before you activate the flow.

Zerobus Ingest writes directly into Unity Catalog Delta tables through SDK (gRPC and REST), OpenTelemetry, MQTT and Kafka-compatible interfaces. The docs page gives no throughput figures. The 1 PB, 12 GB/s and 12 million rows-per-second result in the source post is a single community test run, not a documented limit.

The Lakeflow Connect Jira connector is in Beta per its docs page. The docs describe an admin configuring Jira to allow authentication from Databricks (OAuth U2M is supported; OAuth M2M and basic authentication are not), an admin creating a connection, and non-admins creating pipelines from that connection. A pipeline supports up to 250 tables.

Real-Time Mode is a Structured Streaming trigger type that the concepts page describes as up to 100x faster than micro-batch mode. The page links to a separate Real-time mode examples page for implementation details. It does not state whether Real-Time Mode is in Preview or GA, so check the current release status in the docs before production use.

The Lakeflow Pipelines Editor is an IDE built for pipeline development. Pipelines automatically infer dependencies between datasets and arrange them in a directed acyclic graph (DAG), which you can view and interact with in the editor. The editor shows execution insights and an issues pane with results from the latest run. It also supports data previews for streaming tables and materialized views, and selective execution to refresh individual files or tables without running the full pipeline.

The Databricks Runtime 18.2 release notes list TwsTester, a testing utility for transformWithState StatefulProcessor logic, as implemented in both Scala and PySpark (Apache Spark changes SPARK-54122 and SPARK-54805). The transformWithState docs do not describe how to use it, so check the Spark testing docs and your runtime version before relying on it.

Zerobus Ingest accepts OpenTelemetry (OTLP) data for traces, logs and metrics, with each signal written to predefined table schemas in Unity Catalog. The docs page does not state a release stage for the OpenTelemetry interface, so check current availability before relying on it. The OTLP/gRPC-only limit and the table-creation steps are from a community post and are not verified in the docs.

Workflows parameterization uses job parameters, referenced as {{job.parameters.<name>}}, and task values, referenced as {{tasks.<task_name>.values.<key>}}. For each task inputs can take these references. Dynamic values such as {{job.start_time.iso_date}} and {{backfill.iso_date}}, and the rule that a job parameter overrides a task parameter with the same key, come from a community post and are not verified in the docs.

Lakeflow Connect provides managed connectors for sources such as Salesforce, Workday, MySQL, PostgreSQL, and SQL Server, and it loads only changed data after the first run when the source supports incremental ingestion. Ingestion runs inside Databricks, so you do not operate a separate connector service, and the data lands in Delta tables that your pipelines and Unity Catalog governance already cover. Confirm incremental support per connector, because some sources and tables do not support it yet.

Databricks processes streams with Structured Streaming, which gives exactly-once processing when Delta tables are the source and sink, and Lakeflow Declarative Pipelines let you define streaming transformations declaratively. Files that land in cloud storage are ingested with Auto Loader, so streaming and file-based sources feed the same governed Delta tables. Because streams are processed and stored on the same platform as your analytics, you avoid operating a separate streaming layer between the source and the warehouse.

Databricks runs Structured Streaming with a real-time mode for sub-second workloads, and transformWithState supports custom stateful processing with state TTL. Teams that already run Spark and Delta can add real-time analytics on the same governed Delta tables without adopting a separate streaming database.

Yes. Lakeflow Connect provides managed connectors for SaaS sources including Salesforce and Workday, plus databases such as MySQL, PostgreSQL, and SQL Server. After the first run it loads only changed data when the source supports incremental ingestion, and the work runs inside Databricks rather than in a separate ingestion service. Some sources and tables do not support incremental ingestion yet, so confirm this per connector before you set a schedule.

Yes. AUTO CDC in Lakeflow Declarative Pipelines applies change records to a table as Type 1 or Type 2 updates, using the keys and sequencing column you specify to handle inserts, updates, and deletes. AUTO CDC FROM SNAPSHOT detects changes itself by comparing ordered snapshots, so a change feed is not required for that pattern. Because the CDC logic lives in the same pipeline as your transformations, you do not maintain it as a separate job.

Databricks documents deploying Declarative Automation Bundles from Azure DevOps with a staging deployment that runs when a feature branch merges to main and a production deployment that waits for a successful staging run. The setup uses an Azure Resource Manager service connection with workload identity federation, and production uses an approval check.

Databricks documents running pytest locally with Databricks Connect against remote compute, including a test that reads a remote table and asserts it contains data. Connect-based tests exercise the Spark code, so they do not replace an end-to-end run of a deployed Lakeflow Job.

A Databricks community post describes Freshworks moving ingestion to Spark Structured Streaming reading from Kafka with Delta Lake as the storage layer. Deletion vectors, which the docs describe as marking rows as modified in metadata rather than rewriting Parquet files, are one way to make updates cheaper. The post's 3x, 4x and SLA figures are not verified in the docs.

Bundles have no dedicated metric view resource, so you can deploy a metric view by adding a SQL task whose .sql file runs CREATE OR REPLACE VIEW ... WITH METRICS LANGUAGE YAML against a SQL warehouse, which the job executes after bundle deploy.

Databricks pipeline move lets you reassign a Streaming Table to another pipeline without a full refresh. Stop the source pipeline, remove the table definition from it, run ALTER STREAMING TABLE to set the pipelines.pipelineId property to the destination, and then add the definition to the destination pipeline. Both source and destination must be Unity Catalog pipelines in the same workspace, owned by you, with the destination using the default publishing mode.

In Lakeflow Spark Declarative Pipelines, AUTO CDC maintains SCD Type 2 history natively, tracking versions with __START_AT and __END_AT columns, and AUTO CDC FROM SNAPSHOT handles sources where CDC is not enabled. The Lakeflow Connect overview lists SQL Server among database CDC connectors.

Pipeline expectations (CONSTRAINT name EXPECT with ON VIOLATION warn, drop or fail) record pass and fail metrics in the pipeline event log, which you can query and alert on. Expectations are a good first layer of data testing; Unity Catalog lineage and monitoring and the RACI model in the source post are not verified in the docs.

Yes, for dbt Core. The Jobs UI has a dbt task type, so dbt runs as a task in a Lakeflow Jobs workflow, with retries, run-duration and streaming-backlog thresholds, and notifications set in advanced task settings. dbt commands run in sequential order by default, and Declarative Automation Bundles can define the dbt task as code.

Auto Loader detects new columns in incoming cloud storage files and stops the stream with an UnknownFieldException, and restarting the stream resumes processing with the updated schema. The schemaEvolutionMode option controls the behavior: addNewColumns adds new columns, addNewColumnsWithTypeWidening also widens supported types such as int to long and routes unsupported type changes to the rescued data column, and failOnNewColumns fails the stream without updating the schema. Auto Loader is the Databricks feature for incremental ingestion of files as they arrive.

Spark Declarative Pipelines on Lakeflow is a declarative framework for building batch and streaming data pipelines in SQL and Python. You define pipelines with streaming tables and materialized views rather than writing the pipeline orchestration yourself. Lakeflow pipelines are built on Apache Spark Declarative Pipelines.

Lakeflow Jobs organize tasks as a directed acyclic graph and support custom control flow such as if/else branching. Job-level run-time parameters are pushed to the tasks within the job, and you can set notifications for job events via email, Slack, or custom webhooks, including when a job fails or takes too long. Jobs can also be scheduled with external tools such as Apache Airflow.

Structured Streaming uses watermarks to define how late data is handled. Declare one with withWatermark by specifying an event-time column and a lateness threshold, such as 10 minutes, and queries process records that arrive within that threshold. Watermarks automatically remove old aggregation state after the threshold passes, which prevents memory growth and rising processing latency.

Declare pipeline expectations to check data quality, and choose an ON VIOLATION behavior for invalid records. The default warns and still writes invalid records to the target, ON VIOLATION DROP ROW drops them before they are written, and ON VIOLATION FAIL UPDATE stops the update. For quarantine patterns that keep invalid records for debugging without failing or dropping them, the expectations documentation links to advanced recommendations.

Delta Lake's transaction log lets you review each table modification and query previous table versions with time travel, which helps pinpoint when data changed. Change data feed records row-level changes between versions, so recovery work can target the rows that changed since a known good version.

Real-Time Mode for Structured Streaming processes events with end-to-end latency as low as five milliseconds for sub-second operational workloads such as fraud detection. Lakeflow Declarative Pipelines expectations validate incoming records, and SQL or Python can compute baseline statistics to score events against normal behavior. Data profiling in Unity Catalog profiles tables over time and writes drift statistics to a metrics table, which can flag when a data distribution changes.

Structured Streaming runs in micro-batch mode by default, processing streams in small batches for analytics and ETL at a lower compute cost than continuous processing. Auto Loader incrementally ingests new files as they land in cloud storage and tracks them for exactly-once processing. Delta Lake tables can serve as streaming sources and sinks with exactly-once processing guarantees, so results land reliably in Delta.

No, DLT was renamed, not deprecated. The product formerly known as Delta Live Tables is now called Lakeflow pipelines, and Databricks states there is no migration required because your code still works. Python APIs that use the dlt name can still be used, the classic SKUs still begin with DLT, and Databricks recommends adopting the new names over time.

It was updated to Lakeflow pipelines, part of the Lakeflow data engineering suite. Databricks documents that the product formerly known as DLT has been updated to Lakeflow pipelines, that existing code keeps working with no migration, and that dlt Python APIs can still be used. Databricks recommends moving to the new names over time.

No. Databricks says no migration is required: existing DLT code continues to work under Lakeflow pipelines, and Python APIs that use the dlt name still run. The classic SKUs still begin with DLT, and Databricks recommends adopting the new names over time for future compatibility. Plan the rename as a naming update rather than a rewrite.

Lakeflow is Databricks' data engineering suite. Lakeflow Connect provides managed ingestion connectors, Lakeflow Spark Declarative Pipelines handles transformations (built on Apache Spark Declarative Pipelines), and Lakeflow Jobs provides orchestration. The Lakeflow docs do not describe how the suite relates to Delta Live Tables, so the claims about DLT evolution and backward compatibility were not verified.

The renames are documented with explicit mappings rather than left implicit. Delta Live Tables became Lakeflow pipelines, and the documentation states that existing DLT code needs no migration, that dlt Python APIs still work, and that the classic SKUs still begin with DLT. Databricks recommends the new names over time for future compatibility.

The Lakeflow Spark Declarative Pipelines docs page does not state a release stage. A Databricks blog says Lakeflow, including Lakeflow Declarative Pipelines, reached general availability in June 2025, but the docs do not confirm it. Check the docs for current availability.

Lakeflow Connect is Databricks' set of managed connectors for database, SaaS, file and streaming sources, with incremental ingestion into Unity Catalog governed tables. It works alongside Lakeflow pipelines and Lakeflow Jobs. The docs do not describe it as replacing an earlier product.

Both. Micro-batch processing handles streams in small batches with latency measured in seconds or minutes, and Real-Time Mode is designed for sub-second operational workloads such as fraud detection and real-time personalization, with end-to-end latency as low as five milliseconds. You choose the latency and cost trade-off for each pipeline.

Yes. Structured Streaming stores processing state in checkpoints for fault tolerance and exactly-once delivery semantics, and Delta Lake tables work as streaming sources and sinks with exactly-once processing guarantees. Watermarks control how long a stream waits for late-arriving data in stateful operations. Together these features support production streaming pipelines.

For most pipelines, micro-batch processing delivers latency measured in seconds, with exactly-once guarantees on Delta sources and sinks. For sub-second operational workloads, Real-Time Mode is designed to deliver end-to-end latency as low as five milliseconds. Both options run on the same Structured Streaming engine, so you can choose the latency your use case needs.

Spark Real-Time Mode is a Structured Streaming mode that processes records as they arrive instead of in micro-batches. The Structured Streaming docs describe it for sub-second operational workloads, with end-to-end latency as low as five milliseconds. Its release stage is not stated on the Structured Streaming overview, so check the current release notes before adopting it for production, since Databricks posts about it give conflicting statuses.

Lakeflow Declarative Pipelines provide a declarative model for batch and streaming transformations in SQL and Python. They are built on Apache Spark Declarative Pipelines and run on Databricks Runtime. Whether you use dbt or native declarative pipelines, the choice is about whether you want an integrated platform underneath your transformations.

No. Job and pipeline run history is exposed as governed system tables in the system.lakeflow schema, including jobs, job_tasks, job_run_timeline, job_task_run_timeline, pipelines, and pipeline_update_timeline. You can query runs, retries, and failures across the workspaces in your account in the same cloud region with SQL. Duration columns are fully populated for single-task jobs created with the legacy Jobs APIs, and job_task_run_timeline covers the other runs.

No. In the default publishing mode, a Lakeflow Declarative Pipeline can publish tables to multiple catalogs and schemas. The catalog and schema you configure are defaults, and you can fully qualify names such as main.stores.regional_sales to write individual datasets to other schemas. The legacy publishing mode is the one that uses the LIVE virtual schema, and in default mode the LIVE keyword is ignored.

No. A single Lakeflow Declarative Pipeline can contain both SQL and Python source files. The pipeline analyzes dataset dependencies across all of its files to build one processing graph, so SQL and Python transformations can sit side by side and reference each other. The language you choose at creation only seeds the sample code and does not lock the pipeline to one language.

Yes. Lakeflow pipelines and their source files can be source-controlled together using Declarative Automation Bundles, so pipeline definitions are version-controlled alongside the rest of your project and deployed as code. The bundle packages the pipeline code and its YAML configuration for CI/CD.

No. The Lakeflow Pipelines Editor is an IDE built for pipeline development. It provides a multi-file editor for Python and SQL source files, an interactive pipeline graph of dataset dependencies and state, data previews for streaming tables and materialized views, and an issues pane with results from the latest run. Selective execution refreshes individual files or tables without running the whole pipeline, and editor-triggered updates reuse the cluster and turn off automatic retries for fast iteration.

With managed file events, Auto Loader no longer requires you to set up your own cloud notification services. You enable file events on a Unity Catalog external location and set cloudFiles.useManagedFileEvents to true on Databricks Runtime 14.3 LTS or above. Databricks sets up the subscriptions and file events in your cloud storage account, with one queue and storage event subscription per external location instead of one per stream.

No. 1,000 files per micro-batch is Auto Loader's default, not a fixed ceiling. You tune throughput with cloudFiles.maxFilesPerTrigger, the maximum number of files per batch, and cloudFiles.maxBytesPerTrigger, a soft byte limit. When both are set, Auto Loader processes files until it hits whichever limit comes first, so micro-batch size is a rate-limiting setting you configure for your workload.

dbt is a transformation tool, not an extract-and-load tool: it compiles your project into raw SQL and runs that SQL on Databricks compute. Install the dbt-databricks adapter (Databricks recommends version 1.8.0 or greater) in a Python virtual environment, configure profiles.yml with your connection details, and confirm the connection with dbt debug. You can run dbt Core projects from the command line or as dbt tasks in Lakeflow Jobs, and the hosted dbt Cloud service is also available.

Databricks defines three layers. Bronze holds raw, unvalidated data in its original form, grows incrementally, and keeps history for reprocessing and auditing. Silver holds validated, cleaned, and enriched data, and should keep at least one validated record-level representation that is not aggregated. Gold holds highly refined, aggregated, business-aligned datasets tuned for dashboards, ML, and applications.

Lakeflow Connect is Databricks' managed ingestion framework, with database, SaaS, file, streaming and community connectors, and you can build a custom connector for a source that no managed or community connector supports. SaaS connectors cover sources such as Salesforce, HubSpot, Jira and Workday. Managed connectors are governed by Unity Catalog. Their ingestion pipelines run on serverless compute, while standard database CDC gateways run on classic compute. The first run ingests all selected data, and later runs use change tracking to ingest only changed data when possible.

Bronze stores raw, unvalidated source data incrementally and keeps its history for reprocessing and auditing; it feeds the silver layer. Silver turns that data into validated, cleaned, and enriched records through steps such as cleansing, deduplication, normalization, and quality enforcement. Gold produces aggregated, business-aligned datasets optimized for analytics, dashboards, ML, and applications.

Set cloudFiles.maxFilesPerTrigger to 1 on your cloudFiles stream, which caps each trigger at one new file. The option defaults to 1000 when you do not set it. Auto Loader tracks processed files in the checkpoint, so each file is still processed exactly once.

Yes. Declarative Automation Bundles (formerly Databricks Asset Bundles) define jobs, Lakeflow pipelines, dashboards, and model serving resources as YAML alongside your source files. You use the Databricks CLI (v0.218.0 or later) to initialize a project, validate the configuration, deploy it to a target workspace, and run the deployed workflows from the command line. Bundles can also package source files along with unit and integration tests.

dbt handles the transformation layer: it compiles your modeling project into SQL and runs that SQL on Databricks compute, so the transformations execute on the lakehouse. The dbt-databricks adapter connects dbt to Databricks (Databricks recommends version 1.8.0 or greater), and dbt Core projects can run as dbt tasks inside Lakeflow Jobs. Ingestion is a separate step, which Lakeflow Connect covers with managed connectors.

Create a project with databricks bundle init, then define resources and configuration in databricks.yml and resource files. Use databricks bundle validate to check the configuration, databricks bundle deploy to push the bundle to a target workspace, and databricks bundle run to run the deployed workflow. A bundle can also package source files, unit tests, and integration tests with the resource definitions.

dbt Core is the open-source CLI that you run from a local IDE with the dbt-databricks adapter, and dbt debug confirms the connection. dbt Cloud is the hosted version and is also available, and Databricks points to separate dbt Cloud documentation for its details. dbt Core projects can also run as dbt tasks in Lakeflow Jobs, so you can choose the option that fits your team.

Start by using the Spark UI to find the bottleneck before changing configuration. Databricks recommends liquid clustering, which replaces table partitioning and ZORDER, using the latest Databricks Runtime, and Photon-enabled compute. Keep table statistics up to date because the cost-based optimizer depends on them. Adaptive query execution re-optimizes plans at runtime from actual data sizes, and predictive optimization can automate OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables.

Organize data into bronze (raw as ingested), silver (validated, cleaned, and enriched), and gold (aggregated, business-aligned) layers, following the Databricks medallion definitions. In Lakeflow Spark Declarative Pipelines, streaming tables are designed for append-only sources and process each input once, while materialized views serve refreshed results for downstream consumers. For changing source data such as CDC feeds, use AUTO CDC rather than appending.

Use Lakeflow Spark Declarative Pipelines for the data transformations: you declare streaming tables and materialized views in source code, and the pipeline works out their dependencies and execution order. Use Lakeflow Jobs to orchestrate the steps around those pipelines, since a job can combine notebook, Python script, and pipeline tasks with dependencies and if/else branching. Python script tasks run a Python file, so reusable Python code can run as a job step, and a Run Job task lets one job trigger another.

Auto Loader stores metadata for each discovered file in a RocksDB key-value store in the stream's checkpoint location, which gives exactly-once processing and lets it resume after a failure. When you start a stream for the first time, the cloudFiles.includeExistingFiles option (default true) controls whether files already in the directory are processed. In file notification mode, Auto Loader learns about new files from cloud file events instead of listing directories.

XML support in Auto Loader requires Databricks Runtime 14.3 or above and is in Public Preview. Set the rowTag option to the XML element that maps to each row. In Auto Loader, set cloudFiles.format to xml and enable cloudFiles.inferColumnTypes, because XML columns are inferred as strings by default. Set cloudFiles.schemaLocation to store the inferred schema, and use cloudFiles.schemaEvolutionMode to control how new columns are handled. You can also validate rows against an XSD with rowValidationXSDPath.

Create one job with a task for each bronze, silver, and gold step, and set dependencies so that downstream tasks run after the tasks they depend on. Tasks can also run conditionally, and the job can be scheduled or triggered by events such as new data in cloud storage. Lakeflow Jobs also supports pipeline tasks, so a Lakeflow pipeline can be one step in the same workflow.

Read the stream with spark.readStream and format kafka, setting kafka.bootstrap.servers and one of subscribe, subscribePattern, or assign. The Kafka key and value arrive as byte arrays, so cast them to strings or parse them with from_json or from_avro. Write the result to a Delta table with a checkpointLocation, and use trigger(availableNow=True) for incremental loads, or create a streaming table with read_kafka in SQL.

Install the dbt-databricks adapter in a Python virtual environment, where Databricks recommends version 1.8.0 or greater. Configure a profiles.yml with your Databricks connection details, then run dbt debug to confirm the connection. You can then run the project from the command line or as a dbt task in Lakeflow Jobs.

Setting cloudFiles.schemaLocation turns on schema inference and evolution, and Auto Loader stores the schema history in that location. For JSON, CSV, and XML, inferred columns default to strings, while Avro and Parquet use the types in the file. Use cloudFiles.schemaHints to override specific types in SQL syntax, such as date DATE, and records that don't fit the schema go to the _rescued_data column.

Keep transformation logic in plain DataFrame functions and unit test them with pytest, separate from the pipeline decorators. Pipeline code imports the decorators with from pyspark import pipelines as dp, and @dp.table and @dp.materialized_view build a dataflow graph, so dataset functions are not called directly. Lakeflow also offers a Beta unit testing capability on Databricks Runtime 18.1 and above that covers mock inputs, AUTO CDC specs and expectations; its tests must run from the Lakeflow Pipelines Editor.

Expectations are SQL Boolean constraints applied to each record, and each one has a name and an action. The default action, warn, writes invalid records to the target and tracks how many records pass or fail. drop removes invalid records before they are written and logs the dropped count, while fail stops the update when an invalid record is detected, so no metrics are recorded for that run. In a continuous pipeline, an expectation failure stops the flow and all dependent flows.

Partly. Code written against the open-source pyspark.pipelines module can run locally: the spark-pipelines CLI can initialize, validate and run a pipeline on local Apache Spark, and plain PySpark transformation functions can be unit tested with pytest. Lakeflow-specific features such as expectations and AUTO CDC cannot run or be tested locally, so validate them in a Databricks workspace. Import the decorators with from pyspark import pipelines as dp.

Auto Loader, the cloudFiles source, is the way to ingest files from S3 and other cloud storage incrementally, processing new data files as they arrive without additional setup. Its default discovery mode is directory listing, and Databricks recommends file notification mode with file events for most workloads. Auto Loader stores file metadata in the checkpoint, so each file is processed exactly once.

Configure a file arrival trigger on the Lakeflow Job, which starts a run when new files land in a Unity Catalog external location or volume. You can point the trigger at the root of that location or volume, or at a subpath, and it recursively checks all subdirectories. Jobs also support time-based triggers for scheduled runs.

Use the AUTO CDC API, or AUTO CDC FROM SNAPSHOT when only in-order snapshots are available, and the pipeline computes SCD Type 1 or Type 2 for you. SCD Type 2 keeps history, and you can track changes on all updates or only on a specified set of columns. AUTO CDC replaces the older APPLY CHANGES APIs with the same syntax, and it requires serverless pipelines or the Pro or Advanced edition.

A streaming table is a Delta table with added support for streaming and incremental processing, and a single pipeline owns and updates it. It is designed for append-only sources, and each input is processed only once. Because of that design, joins in a streaming table do not recompute when dimension tables change.

Adaptive query execution re-optimizes query plans while queries run, based on the actual size of the data. In sort-merge and shuffle hash joins, it splits skewed tasks into roughly evenly sized tasks, replicating them when needed. Keep table statistics current as well, since missing or stale statistics can produce inaccurate estimates and oversized tasks.

Yes. When the pipeline runs, Databricks attempts to process only the data needed to keep the materialized view up to date, and in many cases it makes only incremental updates. Some input changes, non-incremental computations, and UDF changes require a full recomputation, and the view still returns correct results when that happens. If a UDF's behavior changes in a way Databricks cannot detect, such as when an external library it calls is upgraded, you run a full refresh yourself.

Yes. The cloudFiles.schemaHints option overrides inferred column types using SQL schema syntax, such as date DATE, and it can also add columns that are missing from the initial stream. Auto Loader uses schema hints only when you do not provide a schema. Setting cloudFiles.schemaLocation enables schema inference and evolution for the stream.

In SQL, CREATE STREAMING TABLE accepts a column specification where you declare column names, types, comments, generated columns, identity columns, defaults, and masks, along with data quality expectations. Table clauses include PARTITIONED BY or CLUSTER BY, comments, and TBLPROPERTIES, and each table defines its own column list. Table constraints such as primary and foreign keys are informational and not enforced.

Yes. A Lakeflow Job can include a Run Job task that triggers another job in the same workspace, and job parameters can override the triggered job's defaults. Databricks composes these workflows within two limits: it does not support jobs with circular dependencies, and it does not support jobs that nest more than three Run Job tasks. Stay within those limits when you build workflows from several jobs.

Yes, with a caveat: XML file format support requires Databricks Runtime 14.3 or above, and the docs label it Public Preview, so confirm preview status for your workspace before production. Set cloudFiles.format to xml with a rowTag option. Optionally enable cloudFiles.inferColumnTypes for typed columns and cloudFiles.schemaLocation to store the inferred schema, since Auto Loader infers XML columns as strings by default.

Use a Lakeflow Connect managed database connector, which ingests relational sources such as MySQL, PostgreSQL and SQL Server through change data capture. Integrated CDC for MySQL runs extraction and application in a single pipeline with no separate gateway, configured with a Unity Catalog connection. Standard CDC uses an ingestion gateway that runs on classic compute, with an ingestion pipeline that moves data to the destination. Query-based connectors can query a database on a schedule without CDC setup.

A checkpoint stores the query's source offsets, committed micro-batches, state, and query ID, so a restart with the same checkpoint resumes where it left off. Changes to input sources, output sinks, or stateful operations generally require a new checkpoint, while adding or removing filters and changing trigger intervals are generally safe. Deleting the checkpoint or switching to a new location makes the query start fresh, and each query should have its own checkpoint location.

Auto Loader's cloudFiles.schemaEvolutionMode option controls this. With failOnNewColumns, the stream stops and the schema stays unchanged when new columns appear, until you update the schema or remove the file. Values that do not match the schema can be routed to the _rescued_data column as a JSON blob with the source file path, which lets you find and handle mismatches.

By default, Auto Loader skips file and folder names that start with an underscore or a period, controlled by the ignoredPathSegmentRegex option with the default ^[._]. Setting ignoredPathSegmentRegex to an empty string surfaces those hidden files. The docs list this option under the DataFrameReader options and note it requires Databricks Runtime 19 or above.

Declare bundle variables in the variables mapping and reference them as ${var.name}, which Databricks resolves at deployment time. You can set a different value per environment in the targets mapping or in variable-overrides.json. The CLI checks sources in order: the --var option, BUNDLE_VAR_ environment variables, variable-overrides.json, the target's variables, and then the default. Use job parameters for values that change per run, since bundle variables are fixed at deploy time.

Auto Loader is designed to incrementally process files as they arrive in cloud storage, which makes it a strong default for file landing zones. File notification mode subscribes to file events, scales to millions of files an hour, and is faster and more scalable than directory listing. You can switch between file notification and directory listing at any time while keeping exactly-once processing, and file notification mode does not guarantee the order in which files are discovered or processed.

Yes. The CREATE STREAMING TABLE column specification supports identity columns, alongside generated columns, defaults, and masks. Primary and foreign key constraints are informational and not enforced, and defining table constraints requires a Unity Catalog-enabled pipeline. Identity columns must be BIGINT on Delta tables, and liquid clustering can be enabled on the table with CLUSTER BY.

By default, Auto Loader uses directory listing mode to discover new files in cloud storage, and Databricks recommends file notification mode for most workloads. Auto Loader stores file metadata in a key-value store in the checkpoint location, which ensures each file is processed exactly once and lets ingestion resume from checkpoint information after failures. You can switch between directory listing and file notification at any time while keeping exactly-once processing.

Yes. A Lakeflow Job consists of one or more tasks, and a single job can include multiple notebook, pipeline, or Python script tasks. Tasks can be ordered with dependencies between them, and jobs support custom control flow logic such as branching.

Databricks supports dbt through the dbt-databricks adapter. You install the adapter and connect a dbt Core project to Databricks, and dbt compiles your models into raw SQL that runs on Databricks. This lets teams keep their dbt transformation workflows while executing on the Databricks platform.

read_files is a SQL table-valued function that reads files under a provided location, including cloud paths on S3, ADLS, and GCS, and returns the data in tabular form. When you use it in a streaming table query with the STREAM keyword, it leverages Auto Loader. It supports JSON, CSV, XML, TEXT, BINARYFILE, PARQUET, AVRO, and ORC files.

Yes. AUTO CDC FROM SNAPSHOT processes periodic full snapshots by comparing consecutive in-order snapshots to determine changes, such as detecting that a record was deleted between snapshots. It can also retain SCD Type 2 history, keeping prior versions of records with start and end markers, either on all updates or on a specified set of columns.

Yes. AUTO CDC FROM SNAPSHOT is designed for this case: it compares each new full snapshot to the previous one, detects changes between them including deletions, and can store SCD Type 2 history so each version of a record carries start and end markers. This avoids hand-written merge logic for snapshot-based dimensions.

You can reprocess data for only selected tables in a pipeline update. In the pipeline UI, choose the tables in the Select tables for refresh dialog and start the update with Refresh selection. To fully reprocess data already ingested for those tables, use the Full Refresh selection option.

Auto Loader is designed to avoid reprocessing files. It stores file metadata in a key-value store in the checkpoint location, which ensures data is processed exactly once, and it resumes from checkpoint information after failures. By default it uses directory listing mode to find new files.

Pipeline settings can be provided in the Configuration field, and for Python pipelines these values are exposed to spark.conf.get(). For SQL source code, declare values in the Parameters field and reference them with a colon prefix, such as :source_catalog. You can also override parameters for a single update from the UI, from a job's pipeline task, or through the Start update API.

The bronze layer holds raw, unvalidated data in its original form, appended incrementally, and it mainly feeds the silver layer. The silver layer holds validated, cleaned, and enriched data, where deduplication and normalization happen, and it keeps at least one validated non-aggregated representation of each record. The gold layer holds highly refined, aggregated, business-aligned datasets tuned for analytics, dashboards, ML, and applications.

Organize the pipeline into bronze, silver, and gold layers. Bronze keeps raw, unvalidated source data appended over time, silver holds validated, cleaned, deduplicated, and normalized records, and gold holds aggregated, business-aligned datasets for dashboards, analytics, ML, and applications. Each layer builds on the one before it, moving data from raw to consumption-ready.

Yes. In Python pipelines you can use standard Python for loops to create multiple tables programmatically, so table definitions can be generated from configuration or metadata rather than written one by one. This supports metadata-driven pipeline patterns.

Databricks supports Apache Kafka as both a Structured Streaming source and a sink. To read, use format kafka with kafka.bootstrap.servers and one topic option: subscribe, subscribePattern, or assign. The Kafka key and value arrive as binary, so cast them to strings or parse them with from_json or from_avro. For incremental batch loading, Databricks recommends Kafka with Trigger.AvailableNow.

dbt is a SQL transformation framework, and the dbt-databricks adapter runs dbt projects on Databricks. For deploying Databricks resources such as jobs and pipelines as code with CI/CD, Databricks points to Declarative Automation Bundles (formerly Databricks Asset Bundles). The docs include a migration guide from dbx to bundles, so the two tools cover different jobs.

Databricks enables Adaptive Query Execution (AQE) by default, and AQE handles skew in sort-merge and shuffle hash joins. It splits skewed partitions into smaller tasks and replicates them when needed, so the resulting tasks are roughly equal in size. Keep table statistics current as well, since missing or stale statistics can produce inaccurate estimates and oversized tasks.

Yes. Lakeflow Connect includes file source connectors for enterprise file storage services, including Google Drive and SharePoint. These connectors ingest both unstructured and structured files into Databricks. Lakeflow Connect also offers database and SaaS connector families for other source types.

Database CDC connectors in Lakeflow Connect use an ingestion gateway that extracts snapshots, change logs, and metadata from the source database, along with staging storage. A separate ingestion pipeline then moves that data from staging into the destination tables. Query-based connectors skip the gateway and staging storage because they query the source directly on a schedule.

Yes. The Databricks provider for Apache Airflow implements operators for triggering Lakeflow Jobs. DatabricksRunNowOperator triggers an existing job, and DatabricksSubmitRunOperator submits a run without requiring a job to exist in Databricks. The documentation includes a sample DAG that triggers a Databricks job.

Lakeflow Declarative Pipelines provides an event log, a structured Delta table you can query for programmatic and historical information about pipeline runs. The pipeline UI shows the pipeline graph, per-table row counts, data quality metrics, and update history. The monitoring docs also cover troubleshooting topics such as recovering from streaming checkpoint failures and fixing high initialization times.

Yes. read_files supports JSON, along with CSV, XML, TEXT, BINARYFILE, PARQUET, AVRO, and ORC files. When read_files is used in a streaming table query with the STREAM keyword, it leverages Auto Loader to ingest new files. This lets a Lakeflow Declarative Pipeline ingest JSON sources with SQL.

Auto Loader infers JSON columns as strings by default. Supplying a full schema avoids this inference, because Auto Loader uses schema hints only when you do not provide a schema. If you want to keep inference, cloudFiles.schemaHints lets you override specific column types, such as turning a string into a date, or add columns that are missing when the stream starts.

Use from_avro to decode a BINARY Avro payload into a STRUCT, passing the Avro schema as a JSON-format string. The function takes a BINARY expression containing Avro data as its input, so apply it to the binary value column of your message stream. The reference includes SQL examples, so use it in the SQL transformations of your pipeline.

Databricks imports .scala, .py, .sql, and .r source files as notebooks. A .py file is recognized as a notebook when its first line contains the comment # Databricks notebook source, so add that header to the top of the file before importing it. Plain .py files without that header are not identified as notebooks.

Genie spaces are a supported Databricks Asset Bundles resource (genie_spaces), so you can deploy them with your bundle rather than a separate setup job. SQL warehouses are also a supported bundle resource (sql_warehouse). Metric views are not in the bundle resource list, so do not assume they can be deployed through bundles.

Workflows support task values. An upstream task calls dbutils.jobs.taskValues.set() with a key-value pair, and downstream tasks reference the value with the dynamic value reference pattern {{tasks.<task_name>.values.<value_name>}}, which Databricks recommends for passing values as task parameters. dbutils.jobs.taskValues.get() is also available in Python notebooks, though the docs recommend the dynamic reference pattern.

A streaming table is a Delta table with additional support for streaming or incremental data processing. One or more flows in a pipeline can target it, and on each update the flows read new changes from a streaming source and append them to the table. Streaming tables are designed for append-only sources, where each input record is processed only once.

Streaming tables can ingest from cloud object storage and from streaming message buses. The docs list Kafka, Azure Event Hubs, and Google Pub/Sub as message bus sources. For sources whose records change or get deleted, the docs recommend AUTO CDC instead of plain appends.

Multiple flows can append to the same streaming table. A single pipeline owns and updates each streaming table, and other pipelines cannot change it. The streaming table itself is defined in the pipeline's source code.

Changes to the query apply to data processed from that point on, because rows already appended are not reprocessed automatically. To recompute existing rows, run a full refresh of the streaming table. Streaming tables are designed so that each input is processed only once, which is what they are built for with append-only sources.

Streams should be naturally bounded or bounded with a watermark. The docs warn that unbounded streams without a watermark can cause memory-pressure failures. Adding a watermark to the source lets the engine drop old state.

Streaming tables are owned and updated by a single pipeline. The streaming tables docs list CLONE among the limitations, since streaming tables cannot be used as the source or target of a deep or shallow clone. Other documented limitations include restricted query evolution and joins that do not recompute when dimension tables change, so plan for these when you design the pipeline.

Non-admin users need the REFRESH privilege on the streaming table in addition to permissions on the pipeline that backs it. Admins can grant both, so non-admins get access to the backing pipeline in a controlled way. The docs list this requirement among the streaming table limitations.

When the pipeline is triggered, Databricks attempts to process only the data that must be processed to keep the materialized view up to date. Some input changes require recomputing the whole result from scratch, which costs more. A materialized view always shows the correct result either way.

Queries against a materialized view can run much faster than against a regular view because the view stores precomputed results. The cached data is materialized in cloud storage, and Unity Catalog stores its metadata. The pipeline keeps those results up to date during each update.

A materialized view is a form of Unity Catalog managed table and is a batch target. The pipeline concepts documentation describes the three dataset types as streaming tables, materialized views, and views. Views are evaluated on demand and are not stored.

A flow is the foundational data processing concept in a pipeline. It reads from a source, applies your logic, and writes to a target. Flows support both streaming and batch semantics.

The pipeline is the container for the flows, streaming tables, materialized views, and sinks that you define. While it runs, it works out their dependencies and sets the execution order. You define these objects in source code rather than orchestrating each step manually.

AUTO CDC uses a set of keys to identify each record, specified as KEYS in SQL or keys in Python. The keys let the flow match incoming changes to existing rows. Pair them with a sequencing column so that changes are applied in order.

The sequencing column is set with SEQUENCE BY in SQL or sequence_by in Python, and it must be a sortable data type. NULL sequencing values are not supported. To sequence by several columns, you can wrap them in a STRUCT.

AUTO CDC supports SCD Type 1 and SCD Type 2, selected with STORED AS SCD TYPE 1 or STORED AS SCD TYPE 2. Type 1 keeps only the latest version of each record. Type 2 retains history on all updates or on updates to a specified set of columns.

Mark deletes with a condition such as APPLY AS DELETE WHEN operation = "DELETE". With SCD Type 1, the deleted key is removed from the target table. With SCD Type 2, the deleted record's version is closed, as shown by the __END_AT values in the docs example.

The warn action is the default. Invalid records are written to the target, and the pipeline tracks how many records pass or fail the expectation. Use it when you want visibility into quality problems without removing data.

The drop action removes invalid records before data is written to the target. The number of dropped records appears in the dataset metrics. Records that pass the expectation are written as normal.

Invalid records prevent the update from succeeding. For table updates, the transaction is rolled back, and someone must fix the issue before the data is reprocessed. This makes fail the right choice for critical quality rules.

Triggered is the default pipeline mode, and Continuous is the alternative. The configure-pipeline docs recommend the default ETL configuration for many use cases, including development, testing, and scheduled production workloads. You set the mode with the Pipeline mode option in the pipeline settings when you create or edit a pipeline.

Databricks recommends developing new pipelines using serverless. Serverless pipelines remove the compute configuration options, so you do not size clusters yourself. Serverless pipelines require the Advanced product edition.

Parameters are default key-value pairs that you declare in the pipeline settings, either in the Parameters section of the UI, in a parameters map in the JSON definition, or in the parameters mapping in YAML. In the UI, open the Parameters section and click Edit to set them. You can override them for a single update from the UI, from a job's pipeline task, or through the Start update API.

In SQL, prefix the parameter key with a colon, such as :source_catalog, and wrap identifiers such as catalog or schema names in IDENTIFIER(). Parameters are available to SQL source code, so this is the reference syntax to use there. For Python, set the key in the pipeline Configuration field and read it with spark.conf.get, which is the pattern the docs show.

Ad-hoc updates started from the UI reuse a cluster to avoid restart overhead, and that cluster runs for two hours by default, a value you can change with pipelines.clusterShutdown.delay. For these UI runs, pipeline retries are disabled so that errors surface immediately. Updates triggered by Jobs or the API, and all continuous pipeline updates, automatically retry specific errors such as a cluster that fails to start, and the cluster shuts down immediately after a job-triggered run completes.

Auto Loader is a Structured Streaming source, exposed as cloudFiles, for picking up new files as they land in cloud storage. It can also process files already in the directory. It records file metadata in a checkpoint so each file is processed exactly once.

Auto Loader ingests JSON, CSV, XML, PARQUET, AVRO, ORC, TEXT, and BINARYFILE files, including pre-compressed versions of these. Set the format with the cloudFiles.format option. This lets one streaming source handle many common file types from cloud storage.

Directory listing is the default discovery mode, while file notification mode uses cloud storage events to find new files without listing directories. Databricks recommends file notification mode using file events for most workloads. You can switch between file notifications and directory listing at any time.

On Amazon S3, Auto Loader uses AWS SNS and AWS SQS. On ADLS and Azure Blob Storage it uses Azure Event Grid and Azure Queue Storage, and on GCS it uses Google Pub/Sub. Auto Loader sets up the notification and queue service automatically.

With file events, Databricks manages one notification queue per external location instead of one per stream. Because the queue is shared, you no longer need to create a queue for each Auto Loader stream. This keeps notification setup simple as the number of streams grows.

Databricks file events is a service that sets up cloud resources to listen for storage events, processes those notifications, and caches file metadata. Setting cloudFiles.useManagedFileEvents to true makes Auto Loader discover files from that cache, so subsequent runs do not need a directory listing. One shared queue serves all Auto Loader streams on a bucket. Full listings still happen on the first run, after the stream has been idle for more than seven days, and when the read position is invalidated.

Specifying a target directory for the cloudFiles.schemaLocation option enables schema inference and evolution. Auto Loader stores the schema history in a _schemas directory under that location. For JSON, CSV, and XML, inferred columns are strings by default, while Avro and Parquet use the types encoded in the file.

In addNewColumns mode, which is the default when no schema is provided, Auto Loader adds the new columns to the schema. The stream then fails with an UnknownFieldException, and restarting it picks up the expanded schema. This lets the schema grow without manual edits.

In rescue mode, Auto Loader never evolves the schema and the stream does not fail due to schema changes. Columns that do not match the schema are preserved in the rescued data column instead of being dropped. This suits pipelines that must keep running while you review incoming changes.

In failOnNewColumns mode, the stream fails when it sees new columns and does not restart until you update the provided schema or remove the offending data file. The schema is never updated automatically in this mode. It is useful when schema changes must be reviewed before they are accepted.

When you provide a schema, the default schema evolution mode is none, so Auto Loader does not evolve the schema. Choose a different behavior through cloudFiles.schemaEvolutionMode when you want the schema to change. Your provided schema stays fixed unless you pick a mode that evolves it.

The _rescued_data column contains a JSON blob with the rescued columns and the source file path of the record. It holds values that could not be parsed into the schema. This prevents Auto Loader from silently dropping data.

Schema hints let you enforce the column types you know and expect on an inferred schema. The option takes SQL-style column type declarations. Auto Loader applies schema hints only when you do not provide a schema.

It controls whether files already in the input path are ingested when the stream starts. The default is true, so existing files are processed. Setting it to false processes only files that arrive after the initial setup.

The cloudFiles.maxFilesPerTrigger option caps the number of new files processed in each trigger, and its default is 1000. Lowering it limits the number of files handled in each micro-batch, which helps control batch size. Set it to 1 to process one new file per trigger.

The option defaults to false, so JSON and CSV columns are inferred as strings. When set to true, schema inference detects exact column types instead. Enable it when you want typed columns without writing an explicit schema.

The backfillInterval option runs asynchronous backfills at a set interval, such as 1 day. The docs advise against using it together with cloudFiles.useManagedFileEvents set to true. For streams that use managed file events, leave backfillInterval unset.

Each streaming query needs its own checkpoint location, because the checkpoint tracks the information that identifies the query, including state and processed records. The docs state that multiple queries should never share the same location. A dedicated checkpoint per query is what lets each query resume where it left off.

withWatermark declares a timestamp field and a lateness threshold, for example withWatermark('event_time', '10 minutes'). The engine uses it to automatically remove old state data, which prevents memory errors and increased processing latency. Records inside the threshold are always processed.

Records inside the watermark threshold are always processed. Records that arrive outside the threshold might still be processed, but the docs say this is not guaranteed. Choose a threshold that matches how late your data can arrive.

In append mode, the query writes rows to the target table after the watermark threshold has passed. In update mode, the query writes rows as results are calculated, and it can update and overwrite those rows as new data arrives. Choose the mode based on whether you need results only after they are complete or as they change.

The AvailableNow trigger consumes all available records as an incremental batch, processing the data present when the query starts. You can size batches with options such as maxBytesPerTrigger, depending on the source. Databricks recommends Trigger.AvailableNow for all incremental batch processing workloads.

Databricks recommends Trigger.AvailableNow for all incremental batch processing workloads. Trigger.Once is deprecated in Databricks Runtime 11.3 LTS and above. On serverless notebooks and jobs, only Trigger.AvailableNow and Trigger.Once are supported.

For low-latency workloads, Databricks recommends real-time mode, which processes records as they arrive. Databricks does not support or recommend the continuous trigger, which is an experimental Spark OSS feature. Use real-time mode instead for low-latency use cases.

processingTime runs micro-batches at a fixed time interval, for example .trigger(processingTime='10 seconds'), and a longer interval checks for new data less often. If you do not set a trigger, Structured Streaming uses processingTime with a 0 interval, which checks for new data every few milliseconds. Choose an interval that matches your latency and cost needs.

Databricks recommends RocksDB with changelog checkpointing to manage state. The docs say this reduces checkpoint duration and end-to-end latency. The state management scheme cannot change between restarts without starting a new checkpoint.

Stateful operations include streaming aggregations, deduplication, stream-stream joins, and custom stateful logic. In Databricks Runtime 16.2 and later, transformWithState supports multiple stateful operators in one workload. The legacy FlatMapGroupWithState and applyInPandasWithState operators are not supported in that chained setup.

Use format('kafka') on spark.readStream and set kafka.bootstrap.servers as a comma-separated list of host:port values. You must also specify one topic option: subscribe for a list of topics, subscribePattern for a Java regex, or assign for specific partitions as JSON. The bootstrap servers option is required for both batch and streaming queries.

Yes. Databricks supports writing to Kafka with the kafka format, using kafka.bootstrap.servers and an optional topic option. The Kafka page shows batch write semantics for Kafka sinks, and streaming writes use the same format. Kafka is supported as both a source and a sink for Structured Streaming.

Set the format to kinesis with spark.readStream.format('kinesis') and identify the stream with the streamName option. In Databricks Runtime 16.1 and above you can use streamARN instead, but you must specify either streamName or streamARN, not both. Records include fields such as partitionKey, data, shardId, sequenceNumber, and approximateArrivalTimestamp.

Databricks supports Apache Pulsar as a Structured Streaming source in Databricks Runtime 14.1 and above. You set the service.url option, such as pulsar://broker.example.com:6650, and identify topics with topic, topics, or topicsPattern. The connector is in Public Preview.

Structured Streaming supports real-time mode, which processes records as they arrive, and micro-batch mode, which processes streams in small batches. The docs describe it as a way to build streaming applications and pipelines with the same Spark APIs used for batch processing. Use it to incrementally ingest data from supported data sources.

Yes. Files in the source location that have already been loaded are skipped on subsequent runs, so repeated scheduled runs only load new data into your Delta table. This makes COPY INTO safe to rerun for incremental file loading. Its source locations include S3, ADLS, ABFS, GCS, and Unity Catalog volumes.

COPY INTO supports CSV, JSON, XML, Avro, ORC, Parquet, text, and binary files as source formats. It can read from cloud storage locations on S3, ADLS, ABFS, and GCS, as well as Unity Catalog volumes. It also supports file and folder filters and schema inference for the target table.

The COPY INTO docs recommend that SQL users use streaming tables for file ingestion. COPY INTO remains a useful SQL command for idempotent loads of new files into a Delta table, since files already loaded are skipped. Choose streaming tables when you are setting up ongoing file ingestion, and COPY INTO for repeatable batch loads.

The bronze layer contains raw, unvalidated data stored in its original form. Data is appended incrementally, and the layer preserves fidelity so data can be reprocessed and audited. It mainly feeds the silver layer rather than analysts.

The silver layer holds validated, cleaned, and enriched versions of the data, where cleansing, deduplication, and normalization happen. The docs recommend keeping at least one non-aggregated version of each record in silver. Silver is the layer for trustworthy row-level data.

The gold layer contains refined, often aggregated data modeled around business needs. It holds semantically meaningful datasets that map to business functions, and it is optimized for query performance. Gold data supports reporting, analytics, dashboards, and ML.

Lakeflow Connect organizes connectors by source type, including database, SaaS, file, and streaming connectors. Database connectors include MySQL, PostgreSQL, and SQL Server, and SaaS connectors include Salesforce, HubSpot, and Workday. Community and custom connectors are also documented.

Each database CDC connector efficiently tracks changes in the source database and applies them incrementally to Delta tables. An ingestion gateway extracts snapshots, change logs, and metadata from the source. A separate ingestion pipeline then moves that data from staging storage into the destination tables.

Yes. PostgreSQL is a supported database CDC connector in Lakeflow Connect, alongside MySQL, SQL Server, and Oracle. The connector uses change data capture to ingest changes incrementally from the source database.

Standard CDC is gateway-based, where a continuously running ingestion gateway works with a separate ingestion pipeline. Integrated CDC uses one unified pipeline that references a Unity Catalog connection and extracts changes on each update. Admins can create that Unity Catalog connection so that non-admin users can create gateway and pipeline pairs.

When you ingest the entire schema, the SQL Server connector is limited to 250 tables per pipeline. The connector also handles new and deleted columns and new tables in that mode. Incremental ingestion is supported.

The connector handles new and deleted columns automatically when you ingest the entire schema. Column renames are not handled automatically and require a full refresh, and data type changes are not supported. Plan for a full refresh when a source column is renamed.

The connector supports OAuth U2M, OAuth M2M, and basic username and password authentication. Manual refresh-token OAuth, API keys, and service account JSON keys are not supported. Admins typically create the Unity Catalog connection, which lets non-admin users create gateway and pipeline pairs.

Yes. The Salesforce connector ingests CRM records and customer data using incremental reads. Each Lakeflow Connect SaaS connector handles source-specific authentication, incremental reads, schema evolution, and automated retries. Salesforce is one of several SaaS sources, alongside HubSpot, Jira, and Workday.

The Workday HCM connector ingests HR, payroll, and workforce data from Workday Human Capital Management. Lakeflow Connect also lists Workday Activity Logging and Workday Reports as separate Workday connectors. The SaaS connectors write to streaming tables.

Lakeflow Jobs supports notebook, Lakeflow designer, Lakeflow Integrations (Beta), Clean Room notebook, Python script, Python wheel, SQL, pipeline (including database table sync and ingestion variants), SQL Alert (Public Preview), dashboard, Power BI, dbt, dbt platform (Public Preview), JAR, Spark Submit, and Genie Code tasks. Control-flow tasks include Run Job, If/else, and For each. Each task can be configured with its own dependencies and settings.

To set retries, you add a retry policy by clicking Add next to Retries in the task's advanced settings. The policy determines when and how many times failed task runs are retried. If you configure both a timeout and retries, the timeout applies to each retry.

The Run Job task triggers another job in the same workspace. You select the child job from the Job drop-down and can pass optional job parameters, which override the child job's default values. Databricks does not support circular dependencies between Run Job tasks, and you cannot nest more than three of them.

The If/else condition task takes a left operand, a boolean operator chosen from a drop-down such as ==, !=, >, >=, <, or <=, and a right operand. Operands can reference job parameters, task parameters, or task values. Comparisons with == and != are string comparisons, so 12.0 == 12 evaluates to false.

A new task's Depends on field defaults to the condition task with the true outcome, so it runs when the condition evaluates to true. Selecting the false outcome makes a task run when the condition evaluates to false. You can chain multiple tasks serially or in parallel on either branch.

The For each task runs a nested task once per item in an input JSON array, a task value reference, or a job parameter reference. Each element is passed to one iteration, and the nested task references it with {{input}} or {{input.<key>}}. This lets one task definition process a dynamic list of items.

The concurrency setting controls how many iterations run in parallel. If you do not change it, the default is 1, so iterations run one at a time. You can raise the Concurrency value to run iterations in parallel.

An upstream Python notebook task stores a JSON-serializable value under a string key with dbutils.jobs.taskValues.set. The JSON form of the value is capped at 48 KiB. Downstream tasks should read it with a dynamic value reference such as {{tasks.<task_name>.values.<value_name>}} in a task parameter.

Databricks recommends referencing task values as task parameters with the dynamic value reference pattern, {{tasks.<task_name>.values.<value_name>}}. The docs discourage the dbutils.jobs.taskValues.get pattern because a task value can be used by many downstream tasks, so renaming the upstream task would require numerous updates. Use the dynamic reference to keep downstream tasks maintainable.

Lakeflow Jobs can send notifications when a job or task run starts, succeeds, or fails, and when a run exceeds a configured duration threshold. Streaming backlog events and continuous-job maintenance window start and completion also trigger notifications. Duration alerts require you to set an expected duration limit.

Notifications can go to email addresses or third-party destinations such as Slack, Microsoft Teams, PagerDuty, and HTTP webhooks. An administrator must set up the third-party destinations. Each event type can have a maximum of three system destinations per job or task.

Job-level notifications are not sent when failed tasks are retried. To get a failure alert for every failed task, use task notifications on the specific task instead. Retry-level alerts therefore come from task notifications, which you configure per task.

Serverless compute for Lakeflow Jobs supports notebook, Python script, Python wheel, dbt, and JAR task types. Databricks provisions and scales the compute, and autoscaling and Photon are enabled automatically. Your workspace must have Unity Catalog enabled, and workloads must support standard access mode.

Because cluster creation permission is not required, all workspace users can use serverless compute to run their workflows. Databricks provisions and manages the compute for serverless jobs. Databricks also upgrades the serverless runtime automatically to add platform enhancements while keeping jobs stable.

A file arrival trigger starts a job run when new files land in a Unity Catalog external location or volume. You can configure it to monitor the root of that location or volume, or a subpath. It recursively checks for new files in all subdirectories of the configured location.

Minimum time between triggers is a cooldown that limits the job to at most one run per interval, so frequent arrivals cannot start back-to-back runs. Wait after last change is a debounce: the trigger waits until no new files arrive for that period, and each new arrival resets the timer. You can set either option on its own or set both together.

Job parameters are key-value pairs defined at the job level. You can override their default values with Run now with different parameters or through the REST API. Job parameters are pushed down to tasks that use key-value parameters, and other tasks can reference them through dynamic value references.

Job parameter default values are not a security control. Anyone with CAN MANAGE RUN permission or higher on the job can override them for a manual run. Use job permissions, not default values, to control who can change parameters.

PySpark on Databricks is the Python interface to Apache Spark, which Databricks is built on. It lets you use Spark's distributed processing from Python. The PySpark docs link to a DataFrames tutorial, PySpark basics, a DataFrames quickstart, and the reference.

DataFrames are the primary objects in Apache Spark, a dataset organized into named columns that is comparable to a spreadsheet or SQL table. PySpark on Databricks exposes the DataFrame API from Python. The docs link to a DataFrames tutorial and a PySpark DataFrames quickstart for hands-on practice.

foreachBatch provides only at-least-once write guarantees, so the sink write has to be made replay-safe. Using the batchId to deduplicate output in the sink gives you an exactly-once guarantee, though you must reason about end-to-end semantics yourself. Each batch must consume the entire DataFrame, because for stateful operators a partial action such as batch_df.show(2) can make the query fail on the next batch.

With the default addNewColumns schema evolution mode, Auto Loader adds the new columns to the tracked schema and the stream then fails with an UnknownFieldException. Restarting the stream resumes processing with the updated schema, so Databricks recommends running the stream as a Lakeflow Job that restarts automatically. The schema evolution behavior is selected through the schema evolution mode option.

Auto Loader places values that don't match the schema, such as missing columns, type mismatches, and case mismatches, into a rescued data column. When the schema is inferred, this column is named _rescued_data by default. You can rename it, or include it when you provide your own schema, by setting the rescuedDataColumn option.

Yes, addNewColumnsWithTypeWidening behaves like addNewColumns and also widens supported types, for example int to long. The documentation lists it as Public Preview in Databricks Runtime 16.4 and above, so check the preview status for your runtime before relying on it. Use it when you expect numeric columns in source files to grow over time.

Auto Loader supports two file detection modes, directory listing and file notification, and it uses directory listing by default. File notification mode can automatically set up a notification service and queue service in your cloud account, and Databricks describes file notification mode with file events as more performant and scalable than directory listing. Consider it when directory listing becomes a bottleneck on high-volume landing paths.

Auto Loader keeps metadata about discovered files in RocksDB inside the checkpoint location, which supports exactly-once processing of files. It processes new files as they arrive and can also pick up files that already exist in the input directory. Because this state lives in the checkpoint location, keep that location intact across restarts.

No. Repair run reruns only the unsuccessful tasks and any tasks that depend on them, and it applies to failed or canceled runs of multi-task jobs. You start it from the Repair run button on the Job run details page. Repaired tasks restart from the beginning, so a task that wrote partial output before failing may duplicate data, and single-task jobs should use Run now instead.

Set the task's Run if dependency to one of the documented options: All succeeded (the default), At least one succeeded, None failed, All done, At least one failed, or All failed. All done runs the task regardless of the status of its dependencies, while At least one failed and All failed are the options for failure-path tasks. If the condition is not met, the task is marked Upstream failed (for the success-based options) or Excluded (for At least one failed and All failed) instead of running.

Databricks recommends the AUTO CDC APIs in place of APPLY CHANGES, which remain available with the same syntax. SCD Type 1 updates records in place without keeping history, while SCD Type 2 keeps history either for all updates or only for the columns you specify. Use SEQUENCE BY to order change events, and wrap multiple sequencing columns in a STRUCT, because the sequencing column must be sortable and cannot be NULL.

By default an expectation only warns: invalid records are retained, and the pipeline tracks the violations in its metrics. Adding ON VIOLATION DROP ROW removes invalid records before they are written to the target, and the dropped count is logged with the other dataset metrics. ON VIOLATION FAIL UPDATE instead makes the update fail when a record is invalid, and manual intervention is needed before reprocessing.

Databricks processes only the data that needs updating whenever it can, so materialized views are often refreshed incrementally. A materialized view always shows a correct result, which can require fully recomputing the query from scratch when incremental processing is not possible, and that recompute can be costly. Each materialized view is defined and updated by a single pipeline, and refreshes happen when that pipeline is triggered.

Databricks recommends the RocksDB state store with changelog checkpointing for managing state in stateful streams, and recommends enabling changelog checkpointing for all stateful Structured Streaming queries on Databricks Runtime 13.3 LTS or later. Stateful operations include streaming aggregations, distinct, dropDuplicates, and stream-stream joins. Follow the linked configuration guides to enable RocksDB and changelog checkpointing for your queries.

A watermark on an event-time timestamp column, such as .withWatermark('event_time', '10 minutes'), bounds the state kept for stateful operations by removing old state once the threshold passes. Records that arrive within the threshold are always processed, while records arriving after it might still be processed, but that is not guaranteed. Choose the threshold by balancing state size against how late your data can reasonably arrive.

Databricks recommends Trigger.AvailableNow, which consumes all available records as an incremental batch. Trigger.Once is deprecated in Databricks Runtime 11.3 LTS and above, although serverless notebooks and jobs still support it. Use processingTime for fixed-interval micro-batches, such as every 10 seconds, since the continuous trigger is not supported on Databricks.

No. Databricks recommends giving each streaming query its own checkpoint directory in durable cloud storage. Configuration changes made between runs apply starting with the first new micro-batch planned after a restart. The page links to a separate topic on recovering after more significant changes.

Set the cloudFiles.maxBytesPerTrigger option on the Auto Loader stream. Databricks documents it as the maximum number of new bytes processed in every trigger. This lets you bound batch size by data volume rather than by file count. The limit is soft, and an individual file is never split across micro-batches.

Auto Loader stores the inferred schema and subsequent schema changes in the directory set by cloudFiles.schemaLocation. The stream's progress is tracked separately in the checkpoint, which you set with checkpointLocation on writeStream. Keep the two locations distinct so schema history and streaming state are managed independently.

Set the ignoreCorruptFiles option to true on the reader. This is a shared file-source option, so it has no cloudFiles. prefix, and its default is false. When true, the Spark job continues past corrupted files and still returns the data it read successfully. Leave it false if corrupt inputs should fail the job so you can investigate them.

A Lakeflow Declarative Pipeline writes its event log to a hidden Delta table in the pipeline's default catalog and schema, named event_log_<pipeline_id>, and you access it through the event_log() function. The log covers audit, data quality checks, progress, and lineage. By default only the pipeline's run-as user can query it, and you can publish it under a name you choose.

A full refresh truncates the streaming table, clears the checkpoints for its flows, and reprocesses every record still available in the source. A normal refresh evaluates only records that arrived since the last update and appends them. Because a full refresh can drop records when the source no longer retains the original data, Databricks recommends running it only when necessary.

Use dropDuplicatesWithinWatermark on a streaming DataFrame that has a watermark defined, available in Databricks Runtime 13.3 LTS and above. Duplicates are compared within the watermark threshold. To guarantee that all duplicates are removed, set the watermark delay higher than the largest event-timestamp gap between duplicate records.

Yes. Set cloudFiles.cleanSource to MOVE or DELETE, available in Databricks Runtime 16.4 and above. The default is OFF. MOVE archives processed files to cloudFiles.cleanSource.moveDestination. DELETE requires a retention period greater than seven days, and the default retention is 30 days. Databricks does not recommend the option when multiple streams read the same directory.

A private table in Lakeflow pipelines was previously called a temporary table. Adding the PRIVATE clause when creating a materialized view or streaming table makes it available to the pipeline but not for access outside it. It persists for the lifetime of the pipeline rather than a single update, which reduces processing time. It can share a name with a catalog table.

Temporary views in a Lakeflow pipeline are private to the defining pipeline and are not added to the catalog, so they can share a name with a catalog view. They persist only for the lifetime of the pipeline. Inside the pipeline, a name shared by a temporary view and a catalog view or table resolves to the temporary view.

Use the quarantine pattern. Apply the rules with expect_all so every row is kept, then compute an is_quarantined flag as NOT(rule_1 AND rule_2 ...). Store the flagged rows in one streaming table, then define two views, one where is_quarantined is false and one where it is true, so invalid rows can be reviewed separately.

Set cloudFiles.format to binaryFile, a format Auto Loader supports. The format produces a DataFrame with path, modificationTime, length, and content (raw bytes) columns. Use pathGlobFilter to select file extensions such as *.pdf or *.png. The format loads file bytes only, so text extraction or OCR must be a separate downstream step.

Lakeflow pipelines run on the current channel by default, which uses the current runtime version. Set the channel setting to preview to test your pipeline against upcoming runtime changes. Use current for production pipelines, since it is the default. The preview channel exists for testing, so keep production pipelines on current.

Data profiling supports three profile types: Time series for tables with a timestamp column, Inference for tables that hold a model's request log, and Snapshot for all other tables. Snapshot profiles support tables up to 4 TB. Profiling creates two metric tables, one with summary statistics and one with drift statistics, plus a dashboard.

Spark Connect is a gRPC-based client-server protocol in Apache Spark that runs DataFrame workloads on a remote server. Databricks uses it for serverless compute, Databricks Connect, Lakeflow pipelines with environment versions configured, and notebooks on standard compute (Scala on Runtime 13.3 and above, Python on 14.3 and above). Analysis is deferred to execution time, so some errors surface later than in Spark Classic.

Run if conditions decide whether a task runs based on how its upstream tasks finished. The default, All succeeded, requires every dependency to succeed. Other options include At least one succeeded, None failed, and All done, which runs the task regardless of upstream outcome. At least one failed and All failed let you run error-handling tasks when upstream tasks fail.

A continuous job is scheduled to run without stopping, and Databricks recommends continuous mode for always-on streaming workloads. With task retry mode set to On failure, failed tasks retry with an exponentially increasing delay. If repeated job runs fail, the job restarts after a retry period that grows up to a system maximum, and the backoff resets after a successful run.

A table update trigger can monitor up to 10 managed or Delta tables. With several tables, choose Any table is updated to start a run when one changes, or All tables are updated to start only after every monitored table has changed. Advanced settings add a minimum time between triggers and a wait after the last change. The job's Run as identity needs SELECT on each table, plus USE CATALOG and USE SCHEMA.

Use the trigger's advanced rate controls. Minimum time between triggers sets a cooldown between runs, so 900 seconds allows at most one run every 15 minutes. Wait after last change debounces arrivals, so a run starts only after no new files have landed for the set period, such as 60 seconds. You can use either option alone or combine them. Auto Loader processes new files with exactly-once guarantees.

Use a REPLACE USING flow. Each incoming snapshot replaces every row whose key columns match, and rows for other keys stay unchanged. You specify exactly one SEQUENCE BY column, and the highest sequence per key wins even when updates arrive out of order. REPLACE USING is in Beta and runs on Databricks Runtime 18.2 and above, on classic or serverless compute.

Yes, during the Multiple Triggers Beta. A job can have up to five triggers, and they can be of the same or different types, such as a schedule and a file arrival trigger. When any active trigger evaluates to true, the job starts a run. Each trigger can be paused and resumed independently, and pausing does not stop active runs. A continuous trigger cannot be combined with other triggers. Admins enable the feature under Previews.

With the default publishing mode, deleting a Unity Catalog pipeline retains its materialized views, streaming tables and views. Retained tables stay queryable but cannot be updated until they are moved into an active pipeline. To delete the pipeline and its tables together, set cascade to true in the API request. Hive metastore pipelines are not affected by this behavior.

Reference the project directory by its absolute workspace or Unity Catalog volume path in the job environment dependencies. The directory should contain pyproject.toml or setup.py. Python script and wheel tasks need a configured environment, while notebook tasks use the notebook's own environment unless you override it with a job environment.

Use the pipeline events system table, which is in Beta. It holds event-log entries for all pipelines and workspaces within a region. The current path is system.lakeflow_pipeline_events_preview.pipeline_events. You can filter by workspace_id, pipeline_id, level and event_time to find errors across your estate. At general availability the table moves to system.lakeflow.pipeline_events, so update queries then.

Define named entries under environment_variables at the job level, then give each task an environment_variables_key that selects one entry. Tasks read the values with os.environ. Entries are never merged or inherited, and a task without a key receives none. The feature is in Beta. An admin must enable Environment variables in Lakeflow Jobs under Previews, and tasks need serverless environment version 5 or above.

Yes. You can set a Lakeflow Spark Declarative Pipeline's Run as identity to an account group instead of an individual user, so the pipeline runs under a shared group identity. The group must be an account group assigned to the workspace. Workspace-local groups and the built-in admins, users, and account users groups cannot be used, and changing the run-as group does not change the pipeline owner.

Yes. The pipeline_events system table, which is in Beta, records Lakeflow pipeline event log entries for all pipelines and workspaces within a region. It captures lifecycle transitions, flow progress, data quality metrics, errors, and cluster resources, so you can query pipeline activity across your account from one table.

Yes, in Beta. In a Lakeflow Spark Declarative Pipeline, create_table() creates a managed table that is written by one or more append_flow declarations. Multiple flows can target the same managed table, so several sources can feed one table within a single pipeline.

Yes, Databricks recommends it. Run a continuous pipeline under a continuous job rather than setting the pipeline's Pipeline mode to continuous, because the job unlocks serverless performance modes such as Standard mode that the built-in continuous mode does not support. The built-in continuous setting is not being removed, so existing configurations keep working.

Use dynamic value references in the task configuration to pass job parameters, run details, and upstream task values without writing code. Double-brace expressions such as {{job.parameters.<name>}}, {{job.run_id}}, {{job.start_time.iso_date}} and {{tasks.<task_name>.values.<value_name>}} resolve at runtime. They are resolved values, not expressions, so you cannot run functions or arithmetic inside the braces.

Yes. Set a duration threshold on the job or task, then add a Duration Warning notification. Lakeflow Jobs sends an email or system notification when the job or task exceeds the configured threshold. Select Duration Warning when you add or edit a notification, and set the limit it uses.

Yes, in Public Preview. You set a streaming backlog threshold in the Job details pane. When the threshold is exceeded, an event fires that you can use to send a notification, so operators hear about a growing backlog early. Streaming backlog thresholds are not supported for pipeline tasks, so configure them on job tasks.

Yes. Add Git settings to the job and specify exactly one reference: a branch, a tag, or a commit. Databricks captures that reference when the run starts, so every task in the run uses the same code version. Supported task sources include notebooks, Python scripts, SQL files, and dbt projects. For staging and production, Databricks recommends a remote Git repository rather than a workspace Git folder.

Development mode favors iteration: it adds a [dev username] prefix to deployed resources, pauses schedules and triggers, enables concurrent runs, disables the deployment lock, and allows cluster overrides. Production mode enforces safeguards: pipelines must be marked development: false, and cluster overrides are not allowed. Mode is optional, and a target gets these behaviors only when you set its mode mapping.

databricks bundle validate checks the bundle configuration. databricks bundle deploy -t dev deploys it to the dev target, and databricks bundle run with the job's resource key runs it, accepting --params and --no-wait. databricks bundle destroy removes deployed resources and prompts for confirmation unless you pass --auto-approve.

On Databricks Runtime 13.3 LTS and above, Databricks Connect for Python does not support SparkContext, RDDs, libraries that use RDDs, Spark Context, or the Spark JVM, or the dataframe.display() API. Some Databricks Utilities are also unavailable, including widgets and notebook workflow. Write the logic with DataFrame operations instead of RDDs.

Yes. For Python UDFs, the local Python minor version must match the minor version of Python on the Databricks Runtime. Serverless has its own entry in the compatibility matrix. To find the remote Python version, check the System environment section of the Databricks Runtime release notes for that version.

Spark Connect defers analysis and name resolution to execution time, so a filter on a missing column that fails immediately in classic Spark builds without error in Spark Connect. The error surfaces when you access df.columns or df.schema, or trigger analysis with an action such as df.collect(). To validate early, trigger analysis explicitly with one of those calls.

Not if you structure it for testing. Databricks recommends storing Python and R functions and their unit tests outside notebooks, in modules run with pytest or testthat. You can also unit test functions in SQL notebooks as SQL UDFs. The docs say unit tests can be run manually or on a schedule, and advise against running unit tests against functions that work with production data, so use data close to production instead.

Yes. Pipeline email notifications are configured in the pipeline settings under Notifications and can go to one or more addresses. They can fire when an update completes successfully, when an update fails with a retryable or non-retryable error, when it fails with a fatal-only error, and when a single data flow fails. For custom handling beyond email, the monitoring docs point to Python event hooks.

Yes, with Python event hooks, which are in Public Preview. A hook is a Python callback that runs when an event is persisted to the pipeline event log, and Python is the only supported language for hooks. The docs state the limits: hooks fire only for STABLE-maturity events, one hook runs at a time so a slow hook blocks the others, and the pipeline does not guarantee that every hook sees every event before compute shuts down. A hook is disabled after max_allowable_consecutive_failures consecutive exceptions until the pipeline restarts.

Yes. The Lakeflow Connect GitLab connector ingests repository, project, group, CI/CD, and security data into Databricks. It supports incremental ingestion and SCD Type 2 history (except for tables with VARIANT columns), and it handles new and deleted columns automatically. Pipelines can be created in the UI, the API, or Declarative Automation Bundles, using OAuth U2M authentication. The connector is in Beta and a workspace admin must enable it on the Previews page.

Lakeflow Connect's Productboard connector ingests product hierarchy, roadmap, objectives, customer feedback notes, and workspace member data into Databricks. Only the notes table syncs incrementally. The connector is in Beta, and a workspace admin turns it on from the Previews page before use.

Yes, in Beta. The Lakeflow Connect Oracle Eloqua connector ingests marketing data including contacts, accounts, campaigns, emails, forms, form submissions, and website visitors. The connector supports incremental ingestion and Unity Catalog governance. A workspace admin enables Lakeflow Connect for Oracle Eloqua from the Previews page before creating a pipeline.

The Lakeflow Connect Asana connector, in Beta, ingests workspaces, projects, tasks, portfolios, and goals into Databricks. Every table is a full-refresh snapshot, so each pipeline update reingests the complete dataset and no incremental logic is needed. A workspace admin enables the connector from the Previews page before use.

Yes, in Beta. The Lakeflow Connect SurveyMonkey connector ingests surveys, survey structure, collectors, survey responses and contacts into Databricks. A workspace admin must turn on Lakeflow Connect for SurveyMonkey on the Previews page before creating a pipeline.

Yes, in Beta. The Lakeflow Connect Cribl connector ingests Cribl.Cloud configuration data such as workers, pipelines, sources, destinations, parsers, and packs, along with worker and input or output status. It also brings in leader audit logs and access logs. A workspace admin enables Lakeflow Connect for Cribl from the Previews page before use.

Yes, in Beta. The Lakeflow Connect Box Audit Logs connector loads Box enterprise audit events, such as user, file, and configuration activity, into Databricks. Incremental ingestion is supported, and authentication uses OAuth M2M through Box Server Authentication with the Client Credentials Grant. A workspace admin enables the connector on the Previews page before setup.

Yes, in Beta. The Lakeflow Connect Google Workspace connector loads audit activity from Google Workspace applications and services into Databricks. It supports incremental ingestion and Unity Catalog governance. Row filtering and SCD Type 2 are not supported. A workspace admin enables Lakeflow Connect for Google Workspace from the Previews page.

Yes, in Beta. The Lakeflow Connect Google Analytics Aggregated connector pulls aggregated GA4 report data, including users, sessions, events, acquisition, ecommerce, and demographics metrics. You can also build custom reports by selecting the dimensions and metrics you need, and ingestion is incremental. Raw event-level GA4 data is handled by the separate Google Analytics Raw Data connector.

Yes, in Beta. The Lakeflow Connect Azure DevOps connector ingests project, repository, work item, build, release, and identity data into Databricks. It supports OAuth M2M authentication. A workspace admin enables Lakeflow Connect for Azure DevOps from the Previews page before creating a pipeline.

Yes, in Beta. The Lakeflow Connect Microsoft Entra ID connector ingests directory objects, role assignments, identity configuration, and audit and sign-in logs into Databricks. A workspace admin must switch on Lakeflow Connect for Microsoft Entra ID from the Previews page before use.

No. The MySQL integrated CDC pipeline in Lakeflow Connect runs extraction and application in a single pipeline, so no separate ingestion gateway is required, and setup uses a Unity Catalog connection. The MySQL source needs binary logging with binlog_format set to ROW and binlog_row_image set to FULL. By default, integrated CDC pipelines run in triggered mode, and you schedule them with a Lakeflow Jobs task.

Add TRIGGER ON UPDATE to a standalone materialized view or streaming table, and the pipeline refreshes when upstream changes are detected. The minimum trigger interval is one minute, and AT MOST EVERY INTERVAL caps how often it runs. A pipeline can monitor up to 10 upstream tables and 30 upstream views, and a workspace can have up to 1,000 trigger-enabled pipelines. The docs recommend this trigger for production workloads with unpredictable upstream schedules.

Yes. The system.lakeflow.pipelines table tracks pipelines across your account as a slowly changing history of configuration, and system.lakeflow.pipeline_update_timeline records each pipeline update with its timing and related metadata. Both tables cover all workspaces in the account that share a cloud region, support streaming reads, and retain 365 days of records, with the latest pipeline record kept beyond that window.

Yes. An AsyncStatefulProcessor with async def methods lets state operations and user logic run concurrently across grouping keys, which raises throughput over the synchronous API with only minor code changes. It needs Databricks Runtime 19 or above, or serverless compute with environment version 6 or above, and it supports only the Python row-based API. In Lakeflow pipelines it requires the PREVIEW channel.

Yes, in Public Preview. On serverless compute, df.cache() computes the DataFrame immediately and stores a session-local snapshot that derived DataFrames read. It needs environment version 5 or above, and Databricks Connect 18.1.8 or later for Python. Cached data lasts until the session ends, for at most seven days, and df.unpersist() frees it early. Admins can turn the preview off on the Previews page, and it is not available in compliance security profile workspaces.

Databricks SQL has built-in IP functions for IPv4 and IPv6 addresses and CIDR blocks, such as ip_cidr_contains, ip_network and ip_version, and they apply to Databricks Runtime 18.2 and above.

The ingestion overview groups options by source. Auto Loader handles files landing in cloud object storage, Structured Streaming connectors cover streaming sources such as Kafka and Kinesis, and fully-managed connectors cover supported enterprise sources such as Salesforce and SQL Server without custom ingestion code. Pick the option that matches where your data lives rather than the one that sounds most powerful. If a managed connector covers your source, start there and write custom code only for sources it does not support.

Databricks' Spark UI guide gives a sequence: start with the Jobs Timeline to find major issues, look at the longest stage, check for skew or spill, determine whether that stage is I/O bound, and then look for other causes of slow stage runtime. If the timeline shows failing jobs, long gaps between jobs, or many small jobs, the guide says to follow the link for that pattern instead. You open the Spark UI from your cluster's page by clicking Spark UI. Working through this order helps you narrow the problem before you change any configuration.

The stage page's Summary Metrics compare task durations. The guide says that if the maximum duration exceeds the 75th percentile by 50% or more, the stage might have skew, which is a signal to investigate rather than proof. For sort-merge and shuffle-hash joins, adaptive query execution (AQE) splits skewed partitions automatically. Use the stage metrics to decide whether the skew comes from your data or from how your query distributes it.

The stage detail page shows spill statistics, and the guide's options include reducing the number of rows or columns the query processes, or increasing executor memory. Adding memory is one option, not the only one, so check whether your query is carrying more data than it needs first. The same page is where you confirm whether spill and skew appear together.

No. Under its high shuffle guidance, Databricks recommends setting spark.sql.shuffle.partitions to auto, which lets Spark set the partition count from the query plan and input size. When a stage is slow, the Spark UI guide has you check the Shuffle Read and Shuffle Write columns to see whether shuffle volume is the issue. Auto is the documented starting point.

The failing-jobs guide says to go past the summary to the failed stage and the failed task, where the failure reason is shown. On classic compute the path is Spark UI, then the failed job, then the failed stage, then the failed task. Start from the task's failure reason and trace back to the code that produced the data it failed on.

Databricks errors carry a named error condition, such as TABLE_OR_VIEW_NOT_FOUND, and the error-classes reference lists each one with its message template and SQLSTATE. For example, TABLE_OR_VIEW_NOT_FOUND has SQLSTATE 42P01, and its template says the table or view cannot be found. Searching the reference for the bracketed name is the quickest way to decode an unfamiliar error before you dig into the stack trace.

Databricks recommends matching on the error condition, SQLSTATE, and message parameters rather than message text, because message text is not stable across releases and may be changed or localized without notice. In Python, catch PySparkException and use getErrorClass, getSqlState, and getMessageParameters. In Scala, use SparkThrowable with the same accessors. In SQL, define a compound-statement condition handler and use GET DIAGNOSTICS to read the condition. Building error handling on those fields keeps it working through upgrades.

Not necessarily. The UDF guidance says built-in Spark functions and SQL UDFs are the most efficient options, and that UDFs should be used only for logic that is difficult to express with built-in functions. Pandas UDFs use Apache Arrow to transfer data and support vectorized operations that can vastly increase performance over row-by-row scalar UDFs. A row-by-row Python UDF still runs your Python function once per row, so rewriting it with built-in functions or a pandas UDF is the documented direction.

Avoid raising spark.driver.maxResultSize as a first fix, because very high values can cause out-of-memory errors. Refactor the code so the driver does not collect a large amount of data, limit what the driver collects, or increase driver memory if needed. The error message reports the total serialized size of the results from the failing tasks.

Adaptive Query Execution can dynamically change a sort-merge join into a broadcast hash join at runtime. The switch is governed by spark.databricks.adaptive.autoBroadcastJoinThreshold, which defaults to 30MB and is the size threshold for the switch. Because it uses runtime sizes, the plan can change during execution. AQE does not make the switch if the share of non-empty partitions is below spark.sql.adaptive.nonEmptyPartitionRatioForBroadcastJoin, or if the join type cannot broadcast that side, such as the left side of a LEFT OUTER JOIN. If a small table still goes through sort-merge, check its size after filters and the join type.

When a stage is slow with little I/O, the Spark UI guide sends you to the associated SQL query and its SQL DAG. Look for slow UDFs, which you can test by commenting them out to see whether the pipeline speeds up, and for small-file overhead on scan or write operators. Also check for Cartesian or nested loop joins, which the guide calls very expensive, and for explode() or exploding joins, where a few input rows multiply into many outputs. Each check has a fix the guide describes, such as native functions, OPTIMIZE, or range join optimization.

Yes. The local development guide recommends keeping transformation logic in plain PySpark functions, separate from pipeline decorators, so you can unit test those functions with pytest on your local machine. Lakeflow pipeline-specific behavior such as expectations and AUTO CDC cannot be run or tested locally, so validate those parts on Databricks. Combining local pytest for transformation logic with workspace runs for pipeline-specific behavior gives a faster feedback loop.

Databricks offers a unit testing capability for Lakeflow pipelines that covers mock inputs, dependent tables, Auto CDC specifications, and expectations. The page labels this feature Beta and requires Databricks Runtime 18.1 or above. Tests must be run from the web-based Lakeflow Pipelines Editor, and Spark Connect is not supported, so this differs from running tests in a local IDE with Databricks Connect. Check the status on the documentation page before relying on it in production workflows.

Serverless pipelines offer a performance mode setting. Standard performance mode is designed to reduce costs for workloads where a slightly higher launch latency is acceptable. Both modes use the same SKU, but standard mode consumes fewer DBUs. For triggered pipelines you select the mode in the schedule settings, and to use standard mode with a continuous pipeline you clear the Performance optimized checkbox and run the pipeline with a continuous job. Compare pipelines in the same mode before drawing conclusions.

Serverless compute supports cost attribution through custom tags applied with a serverless usage policy, which lets you tag spend by team or project. The compute comparison page lists this as the cost attribution option for serverless. Setting up tagging early makes spend easier to attribute before it becomes a surprise.

Yes, for the open-source subset of the pipeline API. The spark-pipelines command line interface can initialize, validate, and run a pipeline with local Apache Spark. Lakeflow pipeline features such as expectations and AUTO CDC cannot be run or tested locally, so validate those in the workspace.

This is documented behavior for the default addNewColumns schema evolution mode, which applies when no schema is provided. Auto Loader adds the new columns to the stored schema and the stream fails with an UnknownFieldException. Restarting the stream resumes processing with the updated schema. The docs recommend configuring Auto Loader streams with Lakeflow Jobs so they restart automatically. If you prefer the stream not to fail on schema changes, rescue mode never evolves the schema and captures unexpected data in the rescued data column.

Ordinary addNewColumns mode adds new columns but keeps the data types of existing columns. Rescue mode never evolves the schema and does not fail on schema changes, so unexpected data lands in the rescued data column for review. The addNewColumnsWithTypeWidening mode also widens supported types and is in Public Preview on Databricks Runtime 16.4 and above; changes it does not support go to the rescued data column. Choose the mode that matches your tolerance for schema drift.

Providing an explicit schema changes the default evolution mode to none, and addNewColumns is not allowed when the stream's schema is provided directly. If you want schema evolution with a known starting schema, supply the columns as schema hints instead, which the docs say work with addNewColumns. This keeps the types you care about pinned while still letting new columns be added. The schema evolution page documents the rules for each mode.

Databricks provides built-in monitoring for Structured Streaming through the Streaming tab in the Spark UI. Naming each query with queryName() makes it easy to tell queries apart there. For alerting or dashboards, a StreamingQueryListener can push metrics to external services, and its onQueryProgress and onQueryTerminated callbacks report status updates and stops, with or without error. Keep listener logic lightweight, since the docs warn that processing latency with listeners can significantly affect query processing speeds.

Several controls work together. Photon, the native vectorized engine, replaces the JVM-based Spark SQL execution engine with a native C++ runtime, and it falls back to Spark for any operation it does not handle, so existing Spark code keeps working. Photon is enabled on serverless compute, SQL warehouses, and serverless Lakeflow pipelines, and by default on classic jobs compute and classic Lakeflow pipelines. Serverless compute for jobs and Lakeflow pipelines removes cluster provisioning, and compute policies can limit which instance types users select for classic compute. The honest limit is that Photon does not accelerate UDFs or RDD and Dataset APIs, so check whether those hot paths can be replaced with built-in DataFrame or SQL operations.

Spark earns its place when data outgrows one machine or when you need the managed platform around the work. For smaller jobs, Databricks documents single-node compute, intended for jobs that use small amounts of data, where the driver acts as both master and worker and there are no worker nodes. Multi-node compute is intended for larger, distributed jobs. DuckDB and Polars are excellent single-node engines and remain a good choice for purely local analysis. Databricks does not state a GB threshold in the pages checked, so choose based on your workload rather than a rule of thumb.

Usually not. Serverless compute for notebooks, jobs and Lakeflow pipelines is a Databricks-managed service that allocates and manages the compute resources, so you do not size a cluster for a small job. Single-node compute covers jobs that use small amounts of data, with no worker nodes. Choose the simplest option that fits your workload, and move to multi-node compute when data volume or concurrency actually demands it.

Single-machine libraries are fast for data that fits on one machine, and that is a fair point. On Databricks, Spark is the technology powering compute clusters and SQL warehouses, Databricks Runtime extends open-source Spark with additional optimizations such as Photon, and Databricks manages the Spark context and session for you. The case for Spark is about data that outgrows one machine and about the managed platform around the work. Compare on your own workload, including engineering and governance effort, rather than on a single benchmark.

The Databricks documentation presents Spark as the technology powering compute clusters and SQL warehouses. Databricks Runtime extends open-source Spark with additional optimizations and proprietary features, and Databricks manages the Spark context and session for you. Market-share and trend questions are outside what product documentation can settle, so judge Spark by your own workload needs rather than by a trend. Spark is one option among several, and the choice should follow the problem.

Not in the default publishing mode, which is used by all new pipelines. The pipeline's configured catalog and schema are the defaults for dataset definitions and table reads unless a query overrides them, and Databricks recommends fully qualified identifiers for datasets outside those defaults. In the default publishing mode the LIVE keyword is ignored. Pipelines still on the legacy publishing mode use the LIVE virtual schema, so check your pipeline's publishing mode before concluding that medallion layers are blocked.

Use the dbt task in Lakeflow Jobs. Point the task at a dbt project in a workspace folder or a Git provider, enter dbt commands such as dbt deps and dbt run, which run in sequential order, and choose the SQL warehouse that runs the SQL dbt generates, plus the target catalog and schema. Databricks recommends serverless compute for jobs, and recommends pinning the dbt tasks to a specific dbt-databricks version so the same version runs in development and production. The honest limit is that this is a job-run workflow, so for iteration you still edit the project and then run it through the job.

Build it when you need history, not just the latest state. Lakeflow's AUTO CDC APIs handle keys, sequencing, deletes and SCD Type 2 for you: with STORED AS SCD TYPE 2, the target keeps each version with __START_AT and __END_AT columns, and the SEQUENCE BY column decides which change applies last, so late or out-of-order changes land correctly. If you only need the current row, a Delta MERGE upsert is less machinery and is often enough. Databricks recommends the AUTO CDC APIs over the APPLY CHANGES APIs, which are still available, so use AUTO CDC for new work.