Microsoft DP-750 - Questions & Answers

Enabling Photon acceleration on a job compute cluster is the best option. It significantly speeds up data engineering pipelines, handles JSON file ingestion seamlessly, and optimizes cost-efficiency for bursty production workloads, which are key for processing rapidly arriving IoT sensor events.

Enable Photon acceleration for a job compute cluster: Databricks Auto Loader is designed for low-latency, high-volume file ingestion. Running Auto Loader on a job compute cluster provides the lowest operational cost because job clusters are billed at a much lower rate than all-purpose clusters.

Enabling Photon acceleration heavily optimizes the ingestion, parsing, and processing of raw JSON data strings, allowing the cluster to handle bursty workloads faster and auto-scale more efficiently.
Scenario, Data Environment, Contoso ingests the following operational and business data: Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files frequently experience schema drift. Contoso identifies the following data ingestion and processing requirements:
-> Auto-scale ingestion pipelines to handle bursty workloads. Handle schema drift for the maintenance and
telemetry data.
-> Ingest file-based telemetry data by using minimal operational effort.

References:
https://medium.com/@krthiak/10-days-of-data-engineering-interview-qna-day-5-db1c0b58bf86
Set up and configure an Azure Databricks environment Question Set 2

Create separate tasks for ingestion, cleansing, and curation is the best architectural fit. This modular workflow approach natively addresses the scenario's technical challenges:

Modularity and Resource Efficiency: Splitting the pipeline into distinct, sequential tasks allows you to configure dedicated, non-interactive compute clusters tailored to the specific resource requirements of each phase (e.g., lightweight for ingestion, heavier memory for curation).

Handling Peak Loads: Independent task scaling ensures that the heavy ingestion phase can scale up to handle event hub spikes without dragging down or over-allocating resources for downstream processing.

Checkpointing & Schema Drift: Separate tasks allow structured streaming checkpoints to be cleanly isolated for each step, ensuring that if a pipeline restarts, it continues exactly where it left off without reprocessing old events. Schema evolution can also be intercepted and handled gracefully between stages rather than breaking a monolith script.

Scenario: Technical Requirements, Contoso identifies the following environment and compute requirements:
-> Production ingestion workloads must run as scheduled, non-interactive pipelines rather than on shared
interactive development clusters.

Technical Requirements, Contoso identifies the following data ingestion and processing requirements:
-> Build a new telemetry pipeline that ingests raw events from the event hubs, cleanses the data, and publishes curated tables to Unity Catalog. Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files frequently experience schema drift.

Problem Statements, The company's existing analytics environment has several issues:

Ingestion Telemetry pipelines fall behind during peak loads. Telemetry ingestion fails when schema drift occurs. Streaming pipelines reprocess events after a pipeline restarts.

References:
https://www.meegle.com/en_us/topics/etl-pipeline/etl-pipeline-for-hadoop-ecosystems
Deploy and maintain data pipelines and workloads Question Set 2

The best option for a Lakeflow Spark Declarative Pipelines (SDP) pipeline that scales automatically while keeping costs and administrative effort low is a job cluster that uses autoscaling.

Lowest Costs: Job clusters (also called automated compute) are billed at a significantly lower Data Processing Unit (DBU) rate compared to all-purpose clusters. By enabling autoscaling, Databricks dynamically allocates or removes worker nodes based on real-time pipeline demand, ensuring you never pay for unutilized resources.

Low Administrative Effort: While Databricks generally recommends Serverless compute as the absolute ideal for zero-admin pipelines, when selecting from classic compute options, a job cluster automatically handles its own lifecycle. It deploys when the pipeline starts executing and terminates automatically when processing is finished.

Incorrect: [Not A] Databricks SQL warehouses are designed to run standalone materialized views and streaming tables via standard SQL. They are not the native compute vehicle for running a fully automated, dedicated Lakeflow Spark Declarative Pipelines (SDP) deployment framework.

[Not B] All-purpose compute is meant for interactive development, debugging, and ad-hoc analysis. It is billed at a much higher DBU rate, which violates the requirement to keep costs low.

[Not D] Aside from the higher billing rate of all-purpose compute, a single-node configuration does not scale horizontally. This directly conflicts with your requirement to build a pipeline that scales automatically.

References:
https://docs.databricks.com/gcp/en/ldp/auto-scaling

To create a new schema with a specific managed location in Azure Databricks Unity Catalog, use the CREATE SCHEMA command.

CREATE SCHEMA catalog_name.schema_name MANAGED LOCATION
'abfss://[email protected]/data';

Catalog Name: Replace catalog_name with your existing catalog.

Schema Name: Replace schema_name with your desired new schema name.Quotes: The storage path must be enclosed in single quotes.

Privileges: You must have CREATE SCHEMA privileges on the parent catalog and ownership (or adequate permissions) on the external location.

References:
https://dev.to/encorepartners/creating-your-first-catalog-schema-and-tables-in-databricks-20p3

The best action is uploading the libraries to the workspace and installing the libraries on the cluster (or ideally uploading them to Unity Catalog volumes).

Unity Catalog Compatibility: When using Unity Catalog for access control, compute clusters are typically configured with Standard (Shared) access mode. In this mode, traditional cluster init scripts [Not B.] face strict execution restrictions or are completely blocked to maintain secure user isolation.

Governance: Uploading your packages as Workspace Files or to Unity Catalog volumes allows administrators to manage access permissions directly and add them to an allowlist if needed.

Cluster-Wide Availability: Installing the libraries via the cluster's Libraries tab ensures that the required ingestion packages are automatically pre-installed and available across all nodes and notebooks running on that cluster.

Incorrect: [Not A] Running pip3 install manually on a cluster terminal or inside a notebook only applies to the specific notebook session (notebook-scoped). It does not natively persist across cluster restarts or handle cross-node execution effectively for data ingestion pipelines.

[Not B] Running a custom script or a legacy init script to modify system-level paths introduces security risks and is generally incompatible with Unity Catalog's strict execution isolation policies for shared compute.

References:
https://docs.databricks.com/aws/en/libraries/

To prevent your Apache Spark Structured Streaming job from reprocessing previously ingested data after a cluster restart, you must configure a streaming checkpoint directory.

Core Solution Enable Checkpointing: Define the checkpointLocation option in your streaming write configuration.

Track Progress: Spark uses this directory to save the exact offset ranges of processed data.

Automatic Recovery: Upon restart, the engine reads the checkpoint and resumes precisely where it left off.

Implementation Example in python
# Configure the streaming write with a checkpoint path (df.writeStream.format("delta").outputMode("append").option("checkpointLocation",
"/Volumes/catalog/schema/volume_name/checkpoints/job_name").toTable("catalog.schema.target_table"))

References:
https://medium.com/@salah.uddin_75300/architecture-of-a-streaming-machine-learning-data-pipeline-
042200c8e7ff

To ensure a service principal can query data in two specific catalogs while maintaining the principle of least privilege, you must assign the following permissions for each catalog:

Unified Catalog Permissions USE CATALOG on the specific catalog: This is a prerequisite that allows the principal to see the catalog and its child objects.

USE SCHEMA on the schemas containing the data: This allows the principal to interact with the schemas within that catalog.

SELECT on the specific tables or views: This provides the actual read access required to query the data.

References:
https://learn.microsoft.com/en-us/purview/data-governance-roles-permissions