Data Engineer - Professional Practice Exam — Data Engineer - Professional

This question assesses knowledge of how to retrieve input parameters passed to a Databricks notebook via the Jobs API for batch pipeline parameterization. The standard, supported mechanism for accepting input parameters in Databricks notebooks, whether for interactive runs, scheduled jobs, or API-triggered jobs, is notebook widgets. When a parameter is passed to a notebook task via the Jobs API, it is automatically bound to a widget with a matching name defined in the notebook. Option E first defines a text widget named "date" that matches the parameter passed by the upstream system, then retrieves the value of that widget into the date variable. This correctly captures the input parameter, making it available for use in the dynamic parquet load path as specified in the scenario. Option Analysis: A. Incorrect. spark.conf.get() is used to retrieve Spark cluster configuration properties, not notebook-level input parameters passed via the Jobs API. The date parameter is a pipeline-specific input, not a Spark runtime configuration, so this method will not return the expected value. B. Incorrect. The standard Python input() function reads input from standard input, which requires interactive user input. Scheduled Databricks jobs run in a headless environment with no interactive input stream, so this approach will fail for batch jobs triggered via the Jobs API. C. Incorrect. sys.argv retrieves command-line arguments passed to a standalone Python script. Databricks notebooks do not receive job parameters as command-line arguments, so this method will not access the parameter passed via the Jobs API. D. Incorrect. dbutils.notebooks.getParam() is not a valid method in the Databricks utility (dbutils) library. There is no getParam method under the dbutils.notebooks namespace, so this code will throw a runtime error. E. Correct. This option follows the standard Databricks pattern for accepting job parameters: it first declares a text widget with the name "date" matching the parameter passed via the Jobs API, then retrieves the widget's value using dbutils.widgets.get(). This correctly captures the input date parameter for use in the load path, and works for both interactive and scheduled job runs. Key Concepts: 1. Databricks Notebook Widgets: Widgets are the native Databricks feature for parameterizing notebooks, supporting text, dropdown, and other input types that can be populated manually during interactive runs or automatically via job API calls. 2. Databricks Jobs API Parameter Binding: When passing parameters to a notebook task via the Jobs API, parameters are automatically mapped to widgets with matching names in the target notebook, eliminating the need for custom input handling. 3. Parameterized Batch Pipelines: Using job parameters to dynamically define data load paths is a core best practice for building reusable, idempotent batch pipelines, as it allows the same notebook code to process data for different batch dates without modification. References: Databricks Notebook Widgets Documentation, https://docs.databricks.com/notebooks/widgets.html Pass Parameters to Databricks Job Tasks

The scenario describes pre-configured interactive clusters with auto-termination enabled, requiring users to have the ability to start stopped clusters and attach to them to run workloads with the least possible permissions. Per Databricks access control rules, the Can Restart cluster permission is explicitly designed to grant users the ability to start terminated clusters, restart running clusters, and attach to the cluster to execute workloads with no additional unnecessary privileges. This meets all requirements of the scenario without overpermissioning, aligning with the minimal permission requirement stated in the question. Option Analysis: A. Incorrect. "Can Manage" privileges include permissions to edit cluster configuration, delete clusters, modify cluster permissions, and manage all cluster operations in addition to restart and attach capabilities. This is far more permission than required for the scenario, so it violates the minimal permission requirement. B. Incorrect. Workspace Admin privileges grant full access to all workspace resources, which is completely unnecessary for this use case. Additionally, cluster creation permission is not required as the clusters are already pre-configured by the workspace administrator, so this option includes multiple excessive permissions. C. Incorrect. First, cluster creation permission is not required for interacting with pre-existing clusters. Second, "Can Attach To" privileges only allow users to attach to a running cluster, but do not grant permission to start a terminated cluster, so this option fails to meet the requirement of allowing users to start the cluster after auto-termination. D. Correct. "Can Restart" privileges on the target cluster grant exactly the required capabilities: starting a terminated (auto-stopped) cluster, restarting a running cluster if needed, and attaching to the cluster to run workloads. This is the lowest permission level that satisfies all requirements in the scenario, aligning with the least privilege principle. E. Incorrect. While "Can Restart" privileges are sufficient for the use case, the added cluster creation permission is unnecessary, as the clusters are already configured by the administrator. This makes the permission set non-minimal, so this option is incorrect. Key Concepts: 1. Databricks Cluster Access Control Tiers: This core concept covers the granular permission levels available for Databricks clusters, including Can Attach To, Can Restart, and Can Manage, each with distinct allowed operations to support least privilege access. 2. Least Privilege Security Principle: A foundational access control principle requiring that users are granted only the minimum set of permissions necessary to complete their required tasks, reducing security risk and excessive access. 3. Databricks Interactive Cluster Cost Management: This covers configuration settings like auto-termination to reduce costs from unused compute, paired with access controls that ensure users can access required compute resources without needing to provision or manage clusters unnecessarily. References: Databricks Documentation - Cluster access control, https://docs.databricks.com/en/security/access-control/cluster-acl.html Databricks Certified Data Engineer Professional Exam Guide, https://www.databricks.com/learn/certification/data-engineer-professional

The question evaluates core knowledge of production Structured Streaming job scheduling requirements, specifically automatic failure recovery and cost optimization. First, automatic failure recovery for streaming queries requires enabled retries to restart failed workloads from their persisted checkpoint state without manual intervention. Second, cost optimization requires use of the lowest-cost appropriate cluster type and elimination of unnecessary concurrent runs that waste compute resources. Third, Structured Streaming queries require a maximum of 1 concurrent run to prevent checkpoint state corruption, duplicate processing, and violation of exactly-once processing guarantees. The suggested answer D meets all three criteria: new job clusters minimize cost by only running for the duration of the job, unlimited retries enable automatic failure recovery, and a maximum of 1 concurrent run preserves processing integrity while avoiding excess cost. Option Analysis: A. Incorrect. Maximum Concurrent Runs set to Unlimited is invalid for Structured Streaming jobs, as multiple concurrent runs of the same query will conflict on shared checkpoint state, cause duplicate processing, and incur unnecessary additional compute costs from multiple parallel clusters. This configuration also fails to enforce core processing guarantees for streaming workloads. B. Incorrect. Retries set to None eliminates automatic failure recovery, which is an explicit requirement of the question. This configuration requires manual intervention to restart failed queries, making it unsuitable for unmonitored production workloads. C. Incorrect. Existing All-Purpose Clusters have significantly higher costs than new job clusters, as all-purpose clusters are long-running and accrue compute costs even during idle periods, while job clusters are terminated immediately after job completion or failure. This configuration fails the cost efficiency requirement stated in the question. D. Correct. New Job Clusters are the lowest cost option for scheduled production jobs, as they are ephemeral and do not incur idle compute costs. Unlimited retries enable automatic recovery from all query failures by restarting the stream from the last committed offset in the checkpoint directory. Maximum Concurrent Runs set to 1 ensures only one instance of the query runs at a time, preventing state corruption, ensuring exactly-once processing, and avoiding unnecessary extra costs from multiple concurrent runs. This configuration meets all requirements specified in the question. E. Incorrect. This configuration has two critical flaws: Existing All-Purpose Clusters are higher cost than job clusters, and Retries set to None removes automatic failure recovery capabilities. It fails both core requirements of the question. Key Concepts: 1. Structured Streaming Fault Tolerance: Structured Streaming relies on a persistent checkpoint directory to store query state and processed offset metadata, allowing failed queries to resume execution exactly where they left off when restarted. Only one active instance of a given streaming query can run against a single checkpoint directory to avoid state conflicts and processing errors. 2. Databricks Cluster Cost Model: Job clusters are purpose-built for scheduled workloads, created at job start and terminated immediately after job completion or failure, resulting in lower total cost of ownership compared to long-running all-purpose clusters that are designed for interactive ad-hoc work and incur costs during idle periods. 3. Production Streaming Job Scheduling Best Practices: Production streaming jobs require enabled retries for hands-off failure recovery, and a maximum of 1 concurrent run to enforce exactly-once processing guarantees and prevent unnecessary resource consumption. References: Databricks Structured Streaming Production Guide, Databricks Clusters Configuration Guide, https://docs.databricks.com/clusters/configure.html#cluster-types

This question assesses understanding of Databricks SQL alert functionality and time-series sensor data monitoring workflows. The alert is configured to trigger when mean(temperature) > 120, runs every minute, and sends notifications at most every 1 minute. Databricks SQL alerts evaluate the threshold condition against every row returned by the underlying query. For sensor monitoring use cases with a table containing sensor_id as a unique identifier, the aggregation query is logically grouped by sensor_id to calculate per-sensor average temperature, rather than a single global average across all sensors. When an alert is set on this aggregated output, it triggers if any row (i.e., any individual sensor's aggregated value) meets the threshold. Three consecutive notifications confirm three consecutive successful query executions each had at least one sensor with average temperature exceeding 120, after which no sensors met the threshold, so notifications stopped. This directly aligns with the correct answer, option E. Option Analysis: A. Incorrect. A global average across all sensors exceeding 120 would only be the trigger condition if the query aggregated all records without grouping by sensor_id. The scenario specifies the table tracks individual sensor records for monitoring purposes, which implies per-sensor aggregation. There is no evidence the alert uses a cross-sensor global mean, so this is not a required condition. B. Incorrect. If the recent_sensor_recordings table was unresponsive, the query would fail to complete successfully. Databricks SQL alerts do not trigger threshold-based notifications for failed query runs by default, so three consecutive successful alert notifications confirm the table was responsive during those runs. C. Incorrect. Query update failures would result in invalid or missing query results, which do not trigger threshold-based alert notifications. The three consecutive alert triggers confirm the query ran successfully and returned valid results each time, so this statement is false. D. Incorrect. The alert trigger condition is based on mean (average) temperature, not maximum temperature. Even if a sensor's maximum temperature exceeded 120, if its average temperature for the time window was below 120, the alert would not trigger, so this is not a required condition. E. Correct. As Databricks SQL alerts trigger if any row returned by the query meets the threshold condition, the three consecutive notifications confirm that for each of the three query runs, at least one sensor had an average temperature exceeding 120. The subsequent stop in notifications confirms no sensors met this condition in later runs, so this statement must be true. Key Concepts: 1. Databricks SQL Alert Threshold Evaluation: Databricks SQL alerts check the configured threshold against every row returned by the associated scheduled query. An alert triggers if any row meets the threshold, not only if a single global aggregated result meets the threshold. 2. Time-Series IoT Data Aggregation: For sensor monitoring use cases, temperature aggregations are typically grouped by sensor_id to track individual device performance, rather than calculating a single global average across all sensors. 3. Alert Notification Frequency Logic: Notification frequency settings in Databricks SQL define the minimum interval between alert notifications. A notification sent every 1 minute for three consecutive minutes confirms three separate, successful query executions met the trigger condition. References: Databricks SQL Alerts Guide, https://docs.databricks.com/sql/user/alerts/index.html Create Alerts in Databricks SQL

The scenario describes a developer using Databricks Repos who cannot see the desired dev-2.3.9 branch in their local branch selection dropdown. This occurs because the local Databricks Repos clone has not been synced with the remote Git repository to retrieve new branch references added to the remote since the last sync. The suggested answer B directly resolves this root cause: pulling changes from the remote fetches all new branch references, commits, and updates from the remote, making the dev-2.3.9 branch visible in the dropdown. The developer can then select the branch to review the unmodified current logic for the notebook, without unnecessary or high-risk code changes. This aligns with standard Databricks Repos and Git workflows, a core knowledge domain for the Data Engineer Professional certification. Option Analysis: A. Incorrect. Creating a pull request is a process for reviewing and merging code changes between branches, not for retrieving missing branch references from the remote. Additionally, the dev-2.3.9 branch is not available locally, so it cannot be selected via the REST API until remote changes are pulled first. This approach does not address the root cause of the missing branch. B. Correct. Databricks Repos maintains a local clone of the connected remote Git repository. Pulling changes from the remote fetches all new branch references, commits, and updates pushed to the remote since the last sync. After pulling, the dev-2.3.9 branch will appear in the branch selection dropdown, allowing the developer to select it and review the current, unmodified logic for the notebook as required. This follows official Databricks best practices tested in the certification. C. Incorrect. A branch cannot be checked out in Databricks Repos if it has not been synced to the local clone from the remote first. Since dev-2.3.9 is not present in the dropdown, it is not available locally to checkout. Additionally, auto-resolving conflicts is unnecessary here, as the goal is to review the existing logic on dev-2.3.9, not merge it with the developer's personal branch. D. Incorrect. Merging unvetted personal changes to the main branch is a high-risk operation that violates standard change management practices, and is completely unnecessary for the goal of reviewing an existing feature branch. Cloning the repository again is redundant and inefficient, as pulling changes achieves the same result without the risk of losing uncommitted local work. E. Incorrect. Merging the developer's personal branch with dev-2.3.9 would alter the state of the code on dev-2.3.9, which defeats the purpose of reviewing the unmodified current logic on that branch. Creating a pull request is also unnecessary for the task of accessing an existing remote branch to review its contents. Key Concepts: 1. Databricks Repos Local Sync: Databricks Repos stores a local clone of the connected remote Git repository. All branch references, commits, and changes must be synced between the local clone and remote via pull or push operations to ensure the local environment reflects the latest remote state. 2. Git Branch Fetching: New branches created on a remote Git repository are not automatically available in local clones. Users must run a fetch or pull operation to retrieve new branch references from the remote before they can be accessed locally. 3. Databricks Repos Branch Selection: The branch dropdown in Databricks Repos only displays branches that have been synced to the local clone. Missing branches indicate the local clone has not been updated with recent remote changes. References: Git operations with Databricks Repos, https://docs.databricks.com/repos/git-operations-with-repos.html Work with branches in Databricks Repos