Databricks Certified Data Engineer Professional Sample Questions:

1. In order to facilitate near real-time workloads, a data engineer is creating a helper function to leverage the schema detection and evolution functionality of Databricks Auto Loader. The desired function will automatically detect the schema of the source directly, incrementally process JSON files as they arrive in a source directory, and automatically evolve the schema of the table when new fields are detected.
The function is displayed below with a blank:

Which response correctly fills in the blank to meet the specified requirements?

A)

B)

C)

D)

E)

2. A company processes semi-structured JSON files from an external source using Auto Loader in a classic Databricks job. Occasionally, records arrive with null critical fields, invalid types, or unexpected nested schema variations. The engineer must ensure that malformed or non- conforming records are not dropped silently and are captured in a separate quarantine table. The pipeline should continue processing good records into the Bronze layer without failing the job, and the approach must support both batch and streaming ingestion.
The data engineer needs to build a robust ingestion pattern that automatically routes bad records to a quarantine Delta table, while still ingesting good records into the Bronze layer for further processing.
Which approach fulfills the quarantine mechanism in this ingestion architecture?

A) Use Lakeflow Spark Declarative Pipelines with a SQL pipeline; configure it to drop rows with nulls using where critical_fields is not null, and rely on audit logs for malformed data.
B) Create a notebook job with inferSchema=True, write a streaming query with .foreachBatch() and catch exceptions using try/except to redirect failed batches to quarantine.
C) Use Auto Loader with failFast mode to set to false, and enable schema evolution; invalid records will be silently ignored during ingestion.
D) Use Auto Loader with LDP and implement an EXPECT () constraint with a record audit logic to route bad records.

3. A data engineering team is configuring access controls in Databricks Unity Catalog. They grant the SELECT privilege on the sales catalog to the analyst_group, expecting that members of this group will automatically have SELECT access to all current and future schemas, tables, and views within the catalog. What describes the privilege inheritance behavior in Unity Catalog?

A) Privileges in Unity Catalog do not cascade; SELECT must be explicitly granted on each schema and table, even if granted at the catalog level.
B) Privileges granted at the schema level override any catalog-level privileges and prevent access unless explicitly revoked.
C) Granting SELECT at the catalog level applies to existing schemas and tables but not to those created in the future.
D) Granting SELECT on a catalog automatically applies SELECT to all current and future schemas, tables, and views within that catalog.

4. A data engineer is setting up a pipeline to ingest data from a message bus system that occasionally delivers duplicate messages. The duplicate messages can be a week apart. The target is a Databricks Delta Lake table where each record should appear exactly once. Which Databricks ingestion pattern should be implemented to handle potential duplicates where events can arrive outside of the configured watermark?

A) Implement a write operation using MERGE INTO with a unique key
B) Use Delta Lake time travel to identify and remove duplicates
C) Use Delta Lake's change data feed to filter duplicate records
D) Configure Structured Streaming with dropDuplicates transformation

5. While reviewing a query's execution in the Databricks Query Profiler, a data engineer observes that the Top Operators panel shows a Sort operator with high Time Spent and Memory Peak metrics. The Spark UI also reports frequent data spilling. How should the data engineer address this issue?

A) Repartition the DataFrame to a single partition before sorting.
B) Switch to a broadcast join to reduce memory usage.
C) Convert the sort operation to a filter operation.
D) Increase the number of shuffle partitions to better distribute data.

Solutions: