Databricks Databricks Certified Associate Developer for Apache Spark 3.5 – Python Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam Dumps: Updated Questions & Answers (October 2025)

Comprehensive and Detailed Explanation From Exact Extract:

Shuffling occurs in operations likegroupBy,reduceByKey, orjoin—which cause data to be moved across partitions. Therepartition()operation can also cause a shuffle, but in this context, it follows an aggregation.

InOption D, thegroupByfollowed byaggresults in a shuffle due to grouping across nodes.

Therepartition(10)is a partitioning transformation but does not involve a new shuffle since the data is already grouped.

This sequence — shuffle (groupBy) followed by non-shuffling (repartition) — is correct.

Option A does the opposite: thefilterdoes not cause a shuffle, butgroupBydoes — this makes it the wrong order.

dedup_df = iot_bronze_df.dropDuplicates(["event_ts","sensor_id","metric_value"])

dropDuplicates accepts a list of columns to use for deduplication.

This ensures only unique records based on the specified keys are retained.

Comprehensive and Detailed Explanation From Exact Extract:

In PySpark, when performing a join between two DataFrames, the columns do not have to share the same name. You can explicitly provide a join condition by comparing specific columns from each DataFrame.

This syntax is correct and fully supported:

df1.join(df2, df1.employee_id == df2.emp_id, how='inner')

This will perform an inner join betweendf1anddf2using theemployee_idfromdf1andemp_idfromdf2.

Comprehensive and Detailed Explanation:

The approx_percentile function in Spark is a performance-optimized alternative to percentile. It takes an optional accuracy parameter:

approx_percentile(column, percentage, accuracy)

Higher accuracy values → more precise results, but increased memory/computation.

Lower values → faster but less accurate.

From the documentation:

“Increasing the accuracy improves precision but increases memory usage.”

Final Answer: D

Temp views likecreateOrReplaceTempVieware session-scoped.

They disappear once the Spark session ends.

To retain data across sessions, it must be persisted:

df.write.saveAsTable("users_vw")

Thus, the view needs to be persisted as a table to survive session termination.

Spark uses lazy evaluation. Transformations like filter, select, and groupBy only define the DAG (Directed Acyclic Graph). No execution occurs until an action is triggered.

The first action in the code is:reduced_df.count()

So Spark starts processing data at this line.

The.mode("overwrite")ensures that existing files at the path will be replaced.

partitionBy("country")optimizes queries by writing data into partitioned folders.

Correct syntax:

df.write.mode("overwrite").partitionBy("country").parquet("/data/output")

— Source:Spark SQL, DataFrames and Datasets Guide

Comprehensive and Detailed Explanation From Exact Extract:

The message"GC overhead limit exceeded"typically indicates that the JVM is spending too much time in garbage collection with little memory recovery. This suggests that the driver or executor is under-provisioned in memory.

The most effective remedy is to increase the driver memory using:

--driver-memory 4g

This is confirmed in Spark's official troubleshooting documentation:

“If you see a lot ofGC overhead limit exceedederrors in the driver logs, it’s a sign that the driver is running out of memory.”

—Spark Tuning Guide

Why others are incorrect:

Amay help but does not directly address the driver memory shortage.

Bis not a valid action; GC cannot be disabled.

Dincreases memory usage, worsening the problem.

Comprehensive and Detailed Explanation From Exact Extract:

To filter rows based on a condition and display them in Spark, usefilter(...).show():

employees_df.filter(employees_df.tenure >= 5).show()

Option A is correct and shows the results.

Option B filters but doesn’t display them.

Option C uses Python’s built-infilter, not Spark.

Option D collects the results to the driver, which is unnecessary if.show()is sufficient.

Final Answer: A

Comprehensive and Detailed Explanation From Exact Extract:

The default broadcast join threshold in Spark is:

spark.sql.autoBroadcastJoinThreshold = 10MB

Sincedf2is only 8 MB (less than 10 MB), Spark will automatically apply a broadcast join without requiring explicit hints.

From the Spark documentation:

“If one side of the join is smaller than the broadcast threshold, Spark will automatically broadcast it to all executors.”

A is incorrect because Spark does support auto broadcast even with static plans.

B is correct: Spark will automatically broadcast df2.

C and D are incorrect because Spark's default logic handles this optimization.

Final Answer: B