• Home
  • Blogs
  • [2023] Practice with these Associate-Developer-Apache-Spark dumps Certification Sample Questions [Q104-Q129]

[2023] Practice with these Associate-Developer-Apache-Spark dumps Certification Sample Questions [Q104-Q129]

Share

[2023] Practice with these Associate-Developer-Apache-Spark dumps Certification Sample Questions

Get Instant Access of 100% REAL Associate-Developer-Apache-Spark DUMP Pass Your Exam Easily


Why you should take Databricks Associate Developer Apache Spark Exam?

If you are a developer who is interested in learning more about Spark and Big Data technologies, then you should definitely consider taking the Databricks Associate Developer Apache Spark Exam. This exam will help you learn how to use the technologies that are being used in the real world. Databricks Associate Developer Apache Spark exam dumps are the best way to prepare for this exam.

Today's modern businesses need to be agile and nimble to adapt to the fast-paced business environment. With the advent of big data, cloud computing, and the Internet of Things, enterprises now face the challenge of managing, processing, analyzing, and integrating vast amounts of data. These challenges require new skills and a new approach to problem solving. The Apache Spark is a high-performance analytics engine that allows you to analyze and process large datasets in a fraction of the time. The Databricks Associate Developer Apache Spark Exam will help you master the skills required to build data-driven applications using Apache Spark.


Prerequisites for the Databricks Associate Developer Apache Spark Exam?

  • You need to be able to complete the individual data manipulation task with the help of the Spark DataFrameAPI.
  • If you have a basic understanding of the architecture, you can use adaptive query execution.

 

NEW QUESTION # 104
Which of the following code blocks reads in parquet file /FileStore/imports.parquet as a DataFrame?

  • A. spark.read().parquet("/FileStore/imports.parquet")
  • B. spark.read.parquet("/FileStore/imports.parquet")
  • C. spark.mode("parquet").read("/FileStore/imports.parquet")
  • D. spark.read.path("/FileStore/imports.parquet", source="parquet")
  • E. spark.read().format('parquet').open("/FileStore/imports.parquet")

Answer: B

Explanation:
Explanation
Static notebook | Dynamic notebook: See test 1
(https://flrs.github.io/spark_practice_tests_code/#1/23.html ,
https://bit.ly/sparkpracticeexams_import_instructions)


NEW QUESTION # 105
Which of the following code blocks returns a copy of DataFrame itemsDf where the column supplier has been renamed to manufacturer?

  • A. itemsDf.withColumnsRenamed("supplier", "manufacturer")
  • B. itemsDf.withColumn("supplier").alias("manufacturer")
  • C. itemsDf.withColumn(["supplier", "manufacturer"])
  • D. itemsDf.withColumnRenamed(col("manufacturer"), col("supplier"))
  • E. itemsDf.withColumnRenamed("supplier", "manufacturer")

Answer: E

Explanation:
Explanation
itemsDf.withColumnRenamed("supplier", "manufacturer")
Correct! This uses the relatively trivial DataFrame method withColumnRenamed for renaming column supplier to column manufacturer.
Note that the question asks for "a copy of DataFrame itemsDf". This may be confusing if you are not familiar with Spark yet. RDDs (Resilient Distributed Datasets) are the foundation of Spark DataFrames and are immutable. As such, DataFrames are immutable, too. Any command that changes anything in the DataFrame therefore necessarily returns a copy, or a new version, of it that has the changes applied.
itemsDf.withColumnsRenamed("supplier", "manufacturer")
Incorrect. Spark's DataFrame API does not have a withColumnsRenamed() method.
itemsDf.withColumnRenamed(col("manufacturer"), col("supplier"))
No. Watch out - although the col() method works for many methods of the DataFrame API, withColumnRenamed is not one of them. As outlined in the documentation linked below, withColumnRenamed expects strings.
itemsDf.withColumn(["supplier", "manufacturer"])
Wrong. While DataFrame.withColumn() exists in Spark, it has a different purpose than renaming columns.
withColumn is typically used to add columns to DataFrames, taking the name of the new column as a first, and a Column as a second argument. Learn more via the documentation that is linked below.
itemsDf.withColumn("supplier").alias("manufacturer")
No. While DataFrame.withColumn() exists, it requires 2 arguments. Furthermore, the alias() method on DataFrames would not help the cause of renaming a column much. DataFrame.alias() can be useful in addressing the input of join statements. However, this is far outside of the scope of this question. If you are curious nevertheless, check out the link below.
More info: pyspark.sql.DataFrame.withColumnRenamed - PySpark 3.1.1 documentation, pyspark.sql.DataFrame.withColumn - PySpark 3.1.1 documentation, and pyspark.sql.DataFrame.alias - PySpark 3.1.2 documentation (https://bit.ly/3aSB5tm , https://bit.ly/2Tv4rbE , https://bit.ly/2RbhBd2) Static notebook | Dynamic notebook: See test 1 (https://flrs.github.io/spark_practice_tests_code/#1/31.html ,
https://bit.ly/sparkpracticeexams_import_instructions)


NEW QUESTION # 106
Which of the following code blocks returns a DataFrame showing the mean value of column "value" of DataFrame transactionsDf, grouped by its column storeId?

  • A. transactionsDf.groupBy("storeId").agg(avg("value"))
  • B. transactionsDf.groupBy("value").average()
  • C. transactionsDf.groupBy("storeId").agg(average("value"))
  • D. transactionsDf.groupBy("storeId").avg(col("value"))
  • E. transactionsDf.groupBy(col(storeId).avg())

Answer: A

Explanation:
Explanation
This question tests your knowledge about how to use the groupBy and agg pattern in Spark. Using the documentation, you can find out that there is no average() method in pyspark.sql.functions.
Static notebook | Dynamic notebook: See test 2


NEW QUESTION # 107
Which of the following code blocks returns a single-row DataFrame that only has a column corr which shows the Pearson correlation coefficient between columns predError and value in DataFrame transactionsDf?

  • A. transactionsDf.select(corr(predError, value).alias("corr"))
  • B. transactionsDf.select(corr(["predError", "value"]).alias("corr")).first()
  • C. transactionsDf.select(corr(col("predError"), col("value")).alias("corr")).first()
  • D. transactionsDf.select(corr("predError", "value"))
  • E. transactionsDf.select(corr(col("predError"), col("value")).alias("corr")) (Correct)

Answer: E

Explanation:
Explanation
In difficulty, this question is above what you can expect from the exam. What this question NO:
wants to teach you, however, is to pay attention to the useful details included in the documentation.
pyspark.sql.corr is not a very common method, but it deals with Spark's data structure in an interesting way.
The command takes two columns over multiple rows and returns a single row - similar to an aggregation function. When examining the documentation (linked below), you will find this code example:
a = range(20)
b = [2 * x for x in range(20)]
df = spark.createDataFrame(zip(a, b), ["a", "b"])
df.agg(corr("a", "b").alias('c')).collect()
[Row(c=1.0)]
See how corr just returns a single row? Once you understand this, you should be suspicious about answers that include first(), since there is no need to just select a single row. A reason to eliminate those answers is that DataFrame.first() returns an object of type Row, but not DataFrame, as requested in the question.
transactionsDf.select(corr(col("predError"), col("value")).alias("corr")) Correct! After calculating the Pearson correlation coefficient, the resulting column is correctly renamed to corr.
transactionsDf.select(corr(predError, value).alias("corr"))
No. In this answer, Python will interpret column names predError and value as variable names.
transactionsDf.select(corr(col("predError"), col("value")).alias("corr")).first() Incorrect. first() returns a row, not a DataFrame (see above and linked documentation below).
transactionsDf.select(corr("predError", "value"))
Wrong. Whie this statement returns a DataFrame in the desired shape, the column will have the name corr(predError, value) and not corr.
transactionsDf.select(corr(["predError", "value"]).alias("corr")).first() False. In addition to first() returning a row, this code block also uses the wrong call structure for command corr which takes two arguments (the two columns to correlate).
More info:
- pyspark.sql.functions.corr - PySpark 3.1.2 documentation
- pyspark.sql.DataFrame.first - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3


NEW QUESTION # 108
Which of the following statements about data skew is incorrect?

  • A. Broadcast joins are a viable way to increase join performance for skewed data over sort-merge joins.
  • B. In skewed DataFrames, the largest and the smallest partition consume very different amounts of memory.
  • C. To mitigate skew, Spark automatically disregards null values in keys when joining.
  • D. Salting can resolve data skew.
  • E. Spark will not automatically optimize skew joins by default.

Answer: C

Explanation:
Explanation
To mitigate skew, Spark automatically disregards null values in keys when joining.
This statement is incorrect, and thus the correct answer to the question. Joining keys that contain null values is of particular concern with regard to data skew.
In real-world applications, a table may contain a great number of records that do not have a value assigned to the column used as a join key. During the join, the data is at risk of being heavily skewed. This is because all records with a null-value join key are then evaluated as a single large partition, standing in stark contrast to the potentially diverse key values (and therefore small partitions) of the non-null-key records.
Spark specifically does not handle this automatically. However, there are several strategies to mitigate this problem like discarding null values temporarily, only to merge them back later (see last link below).
In skewed DataFrames, the largest and the smallest partition consume very different amounts of memory.
This statement is correct. In fact, having very different partition sizes is the very definition of skew. Skew can degrade Spark performance because the largest partition occupies a single executor for a long time. This blocks a Spark job and is an inefficient use of resources, since other executors that processed smaller partitions need to idle until the large partition is processed.
Salting can resolve data skew.
This statement is correct. The purpose of salting is to provide Spark with an opportunity to repartition data into partitions of similar size, based on a salted partitioning key.
A salted partitioning key typically is a column that consists of uniformly distributed random numbers. The number of unique entries in the partitioning key column should match the number of your desired number of partitions. After repartitioning by the salted key, all partitions should have roughly the same size.
Spark does not automatically optimize skew joins by default.
This statement is correct. Automatic skew join optimization is a feature of Adaptive Query Execution (AQE).
By default, AQE is disabled in Spark. To enable it, Spark's spark.sql.adaptive.enabled configuration option needs to be set to true instead of leaving it at the default false.
To automatically optimize skew joins, Spark's spark.sql.adaptive.skewJoin.enabled options also needs to be set to true, which it is by default.
When skew join optimization is enabled, Spark recognizes skew joins and optimizes them by splitting the bigger partitions into smaller partitions which leads to performance increases.
Broadcast joins are a viable way to increase join performance for skewed data over sort-merge joins.
This statement is correct. Broadcast joins can indeed help increase join performance for skewed data, under some conditions. One of the DataFrames to be joined needs to be small enough to fit into each executor's memory, along a partition from the other DataFrame. If this is the case, a broadcast join increases join performance over a sort-merge join.
The reason is that a sort-merge join with skewed data involves excessive shuffling. During shuffling, data is sent around the cluster, ultimately slowing down the Spark application. For skewed data, the amount of data, and thus the slowdown, is particularly big.
Broadcast joins, however, help reduce shuffling data. The smaller table is directly stored on all executors, eliminating a great amount of network traffic, ultimately increasing join performance relative to the sort-merge join.
It is worth noting that for optimizing skew join behavior it may make sense to manually adjust Spark's spark.sql.autoBroadcastJoinThreshold configuration property if the smaller DataFrame is bigger than the 10 MB set by default.
More info:
- Performance Tuning - Spark 3.0.0 Documentation
- Data Skew and Garbage Collection to Improve Spark Performance
- Section 1.2 - Joins on Skewed Data * GitBook


NEW QUESTION # 109
The code block displayed below contains an error. The code block is intended to join DataFrame itemsDf with the larger DataFrame transactionsDf on column itemId. Find the error.
Code block:
transactionsDf.join(itemsDf, "itemId", how="broadcast")

  • A. broadcast is not a valid join type.
  • B. Spark will only perform the broadcast operation if this behavior has been enabled on the Spark cluster.
  • C. The larger DataFrame transactionsDf is being broadcasted, rather than the smaller DataFrame itemsDf.
  • D. The join method should be replaced by the broadcast method.
  • E. The syntax is wrong, how= should be removed from the code block.

Answer: A

Explanation:
Explanation
broadcast is not a valid join type.
Correct! The code block should read transactionsDf.join(broadcast(itemsDf), "itemId"). This would imply an inner join (this is the default in DataFrame.join()), but since the join type is not given in the question, this would be a valid choice.
The larger DataFrame transactionsDf is being broadcasted, rather than the smaller DataFrame itemsDf.
This option does not apply here, since the syntax around broadcasting is incorrect.
Spark will only perform the broadcast operation if this behavior has been enabled on the Spark cluster.
No, it is enabled by default, since the spark.sql.autoBroadcastJoinThreshold property is set to 10 MB by default. If that property would be set to -1, then broadcast joining would be disabled.
More info: Performance Tuning - Spark 3.1.1 Documentation (https://bit.ly/3gCz34r) The join method should be replaced by the broadcast method.
No, DataFrame has no broadcast() method.
The syntax is wrong, how= should be removed from the code block.
No, having the keyword argument how= is totally acceptable.


NEW QUESTION # 110
Which of the following code blocks reads all CSV files in directory filePath into a single DataFrame, with column names defined in the CSV file headers?
Content of directory filePath:
1._SUCCESS
2._committed_2754546451699747124
3._started_2754546451699747124
4.part-00000-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-298-1-c000.csv.gz
5.part-00001-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-299-1-c000.csv.gz
6.part-00002-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-300-1-c000.csv.gz
7.part-00003-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-301-1-c000.csv.gz spark.option("header",True).csv(filePath)

  • A. spark.read.format("csv").option("header",True).option("compression","zip").load(filePath)
  • B. spark.read.format("csv").option("header",True).load(filePath)
  • C. spark.read.load(filePath)
  • D. spark.read().option("header",True).load(filePath)

Answer: B

Explanation:
Explanation
The files in directory filePath are partitions of a DataFrame that have been exported using gzip compression.
Spark automatically recognizes this situation and imports the CSV files as separate partitions into a single DataFrame. It is, however, necessary to specify that Spark should load the file headers in the CSV with the header option, which is set to False by default.


NEW QUESTION # 111
Which of the following code blocks writes DataFrame itemsDf to disk at storage location filePath, making sure to substitute any existing data at that location?

  • A. itemsDf.write().parquet(filePath, mode="overwrite")
  • B. itemsDf.write.mode("overwrite").path(filePath)
  • C. itemsDf.write.option("parquet").mode("overwrite").path(filePath)
  • D. itemsDf.write(filePath, mode="overwrite")
  • E. itemsDf.write.mode("overwrite").parquet(filePath)

Answer: E

Explanation:
Explanation
itemsDf.write.mode("overwrite").parquet(filePath)
Correct! itemsDf.write returns a pyspark.sql.DataFrameWriter instance whose overwriting behavior can be modified via the mode setting or by passing mode="overwrite" to the parquet() command.
Although the parquet format is not prescribed for solving this question, parquet() is a valid operator to initiate Spark to write the data to disk.
itemsDf.write.mode("overwrite").path(filePath)
No. A pyspark.sql.DataFrameWriter instance does not have a path() method.
itemsDf.write.option("parquet").mode("overwrite").path(filePath)
Incorrect, see above. In addition, a file format cannot be passed via the option() method.
itemsDf.write(filePath, mode="overwrite")
Wrong. Unfortunately, this is too simple. You need to obtain access to a DataFrameWriter for the DataFrame through calling itemsDf.write upon which you can apply further methods to control how Spark data should be written to disk. You cannot, however, pass arguments to itemsDf.write directly.
itemsDf.write().parquet(filePath, mode="overwrite")
False. See above.
More info: pyspark.sql.DataFrameWriter.parquet - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3


NEW QUESTION # 112
Which of the following code blocks returns a single-column DataFrame of all entries in Python list throughputRates which contains only float-type values ?

  • A. spark.createDataFrame(throughputRates, FloatType())
  • B. spark.createDataFrame(throughputRates, FloatType)
  • C. spark.createDataFrame(throughputRates)
  • D. spark.createDataFrame((throughputRates), FloatType)
  • E. spark.DataFrame(throughputRates, FloatType)

Answer: A

Explanation:
Explanation
spark.createDataFrame(throughputRates, FloatType())
Correct! spark.createDataFrame is the correct operator to use here and the type FloatType() which is passed in for the command's schema argument is correctly instantiated using the parentheses.
Remember that it is essential in PySpark to instantiate types when passing them to SparkSession.createDataFrame. And, in Databricks, spark returns a SparkSession object.
spark.createDataFrame((throughputRates), FloatType)
No. While packing throughputRates in parentheses does not do anything to the execution of this command, not instantiating the FloatType with parentheses as in the previous answer will make this command fail.
spark.createDataFrame(throughputRates, FloatType)
Incorrect. Given that it does not matter whether you pass throughputRates in parentheses or not, see the explanation of the previous answer for further insights.
spark.DataFrame(throughputRates, FloatType)
Wrong. There is no SparkSession.DataFrame() method in Spark.
spark.createDataFrame(throughputRates)
False. Avoiding the schema argument will have PySpark try to infer the schema. However, as you can see in the documentation (linked below), the inference will only work if you pass in an "RDD of either Row, namedtuple, or dict" for data (the first argument to createDataFrame). But since you are passing a Python list, Spark's schema inference will fail.
More info: pyspark.sql.SparkSession.createDataFrame - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3


NEW QUESTION # 113
Which of the following describes a valid concern about partitioning?

  • A. Short partition processing times are indicative of low skew.
  • B. The coalesce() method should be used to increase the number of partitions.
  • C. A shuffle operation returns 200 partitions if not explicitly set.
  • D. No data is exchanged between executors when coalesce() is run.
  • E. Decreasing the number of partitions reduces the overall runtime of narrow transformations if there are more executors available than partitions.

Answer: C

Explanation:
Explanation
A shuffle operation returns 200 partitions if not explicitly set.
Correct. 200 is the default value for the Spark property spark.sql.shuffle.partitions. This property determines how many partitions Spark uses when shuffling data for joins or aggregations.
The coalesce() method should be used to increase the number of partitions.
Incorrect. The coalesce() method can only be used to decrease the number of partitions.
Decreasing the number of partitions reduces the overall runtime of narrow transformations if there are more executors available than partitions.
No. For narrow transformations, fewer partitions usually result in a longer overall runtime, if more executors are available than partitions.
A narrow transformation does not include a shuffle, so no data need to be exchanged between executors.
Shuffles are expensive and can be a bottleneck for executing Spark workloads.
Narrow transformations, however, are executed on a per-partition basis, blocking one executor per partition.
So, it matters how many executors are available to perform work in parallel relative to the number of partitions. If the number of executors is greater than the number of partitions, then some executors are idle while other process the partitions. On the flip side, if the number of executors is smaller than the number of partitions, the entire operation can only be finished after some executors have processed multiple partitions, one after the other. To minimize the overall runtime, one would want to have the number of partitions equal to the number of executors (but not more).
So, for the scenario at hand, increasing the number of partitions reduces the overall runtime of narrow transformations if there are more executors available than partitions.
No data is exchanged between executors when coalesce() is run.
No. While coalesce() avoids a full shuffle, it may still cause a partial shuffle, resulting in data exchange between executors.
Short partition processing times are indicative of low skew.
Incorrect. Data skew means that data is distributed unevenly over the partitions of a dataset. Low skew therefore means that data is distributed evenly.
Partition processing time, the time that executors take to process partitions, can be indicative of skew if some executors take a long time to process a partition, but others do not. However, a short processing time is not per se indicative a low skew: It may simply be short because the partition is small.
A situation indicative of low skew may be when all executors finish processing their partitions in the same timeframe. High skew may be indicated by some executors taking much longer to finish their partitions than others. But the answer does not make any comparison - so by itself it does not provide enough information to make any assessment about skew.
More info: Spark Repartition & Coalesce - Explained and Performance Tuning - Spark 3.1.2 Documentation


NEW QUESTION # 114
Which of the following describes properties of a shuffle?

  • A. A shuffle is one of many actions in Spark.
  • B. Shuffles involve only single partitions.
  • C. In a shuffle, Spark writes data to disk.
  • D. Operations involving shuffles are never evaluated lazily.
  • E. Shuffles belong to a class known as "full transformations".

Answer: C

Explanation:
Explanation
In a shuffle, Spark writes data to disk.
Correct! Spark's architecture dictates that intermediate results during a shuffle are written to disk.
A shuffle is one of many actions in Spark.
Incorrect. A shuffle is a transformation, but not an action.
Shuffles involve only single partitions.
No, shuffles involve multiple partitions. During a shuffle, Spark generates output partitions from multiple input partitions.
Operations involving shuffles are never evaluated lazily.
Wrong. A shuffle is a costly operation and Spark will evaluate it as lazily as other transformations. This is, until a subsequent action triggers its evaluation.
Shuffles belong to a class known as "full transformations".
Not quite. Shuffles belong to a class known as "wide transformations". "Full transformation" is not a relevant term in Spark.
More info: Spark - The Definitive Guide, Chapter 2 and Spark: disk I/O on stage boundaries explanation - Stack Overflow


NEW QUESTION # 115
The code block displayed below contains at least one error. The code block should return a DataFrame with only one column, result. That column should include all values in column value from DataFrame transactionsDf raised to the power of 5, and a null value for rows in which there is no value in column value. Find the error(s).
Code block:
1.from pyspark.sql.functions import udf
2.from pyspark.sql import types as T
3.
4.transactionsDf.createOrReplaceTempView('transactions')
5.
6.def pow_5(x):
7. return x**5
8.
9.spark.udf.register(pow_5, 'power_5_udf', T.LongType())
10.spark.sql('SELECT power_5_udf(value) FROM transactions')

  • A. The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and the SparkSession cannot access the transactionsDf DataFrame.
  • B. The pow_5 method is unable to handle empty values in column value, the name of the column in the returned DataFrame is not result, and Spark driver does not call the UDF function appropriately.
  • C. The returned DataFrame includes multiple columns instead of just one column.
  • D. The pow_5 method is unable to handle empty values in column value, the UDF function is not registered properly with the Spark driver, and the name of the column in the returned DataFrame is not result.
  • E. The pow_5 method is unable to handle empty values in column value and the name of the column in the returned DataFrame is not result.

Answer: B

Explanation:
Explanation
Correct code block:
from pyspark.sql.functions import udf
from pyspark.sql import types as T
transactionsDf.createOrReplaceTempView('transactions')
def pow_5(x):
if x:
return x**5
return x
spark.udf.register('power_5_udf', pow_5, T.LongType())
spark.sql('SELECT power_5_udf(value) AS result FROM transactions')
Here it is important to understand how the pow_5 method handles empty values. In the wrong code block above, the pow_5 method is unable to handle empty values and will throw an error, since Python's ** operator cannot deal with any null value Spark passes into method pow_5.
The order of arguments for registering the UDF function with Spark via spark.udf.register matters. In the code snippet in the question, the arguments for the SQL method name and the actual Python function are switched. You can read more about the arguments of spark.udf.register and see some examples of its usage in the documentation (link below).
Finally, you should recognize that in the original code block, an expression to rename column created through the UDF function is missing. The renaming is done by SQL's AS result argument.
Omitting that argument, you end up with the column name power_5_udf(value) and not result.
More info: pyspark.sql.functions.udf - PySpark 3.1.1 documentation


NEW QUESTION # 116
Which of the following statements about storage levels is incorrect?

  • A. DISK_ONLY will not use the worker node's memory.
  • B. Caching can be undone using the DataFrame.unpersist() operator.
  • C. The cache operator on DataFrames is evaluated like a transformation.
  • D. MEMORY_AND_DISK replicates cached DataFrames both on memory and disk.
  • E. In client mode, DataFrames cached with the MEMORY_ONLY_2 level will not be stored in the edge node's memory.

Answer: D

Explanation:
Explanation
MEMORY_AND_DISK replicates cached DataFrames both on memory and disk.
Correct, this statement is wrong. Spark prioritizes storage in memory, and will only store data on disk that does not fit into memory.
DISK_ONLY will not use the worker node's memory.
Wrong, this statement is correct. DISK_ONLY keeps data only on the worker node's disk, but not in memory.
In client mode, DataFrames cached with the MEMORY_ONLY_2 level will not be stored in the edge node's memory.
Wrong, this statement is correct. In fact, Spark does not have a provision to cache DataFrames in the driver (which sits on the edge node in client mode). Spark caches DataFrames in the executors' memory.
Caching can be undone using the DataFrame.unpersist() operator.
Wrong, this statement is correct. Caching, as achieved via the DataFrame.cache() or DataFrame.persist() operators can be undone using the DataFrame.unpersist() operator. This operator will remove all of its parts from the executors' memory and disk.
The cache operator on DataFrames is evaluated like a transformation.
Wrong, this statement is correct. DataFrame.cache() is evaluated like a transformation: Through lazy evaluation. This means that after calling DataFrame.cache() the command will not have any effect until you call a subsequent action, like DataFrame.cache().count().
More info: pyspark.sql.DataFrame.unpersist - PySpark 3.1.2 documentation


NEW QUESTION # 117
Which of the following describes the role of the cluster manager?

  • A. The cluster manager schedules tasks on the cluster in local mode.
  • B. The cluster manager allocates resources to Spark applications and maintains the executor processes in remote mode.
  • C. The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode.
  • D. The cluster manager schedules tasks on the cluster in client mode.
  • E. The cluster manager allocates resources to the DataFrame manager.

Answer: C

Explanation:
Explanation
The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode.
Correct. In cluster mode, the cluster manager is located on a node other than the client machine. From there it starts and ends executor processes on the cluster nodes as required by the Spark application running on the Spark driver.
The cluster manager allocates resources to Spark applications and maintains the executor processes in remote mode.
Wrong, there is no "remote" execution mode in Spark. Available execution modes are local, client, and cluster.
The cluster manager allocates resources to the DataFrame manager
Wrong, there is no "DataFrame manager" in Spark.
The cluster manager schedules tasks on the cluster in client mode.
No, in client mode, the Spark driver schedules tasks on the cluster - not the cluster manager.
The cluster manager schedules tasks on the cluster in local mode.
Wrong: In local mode, there is no "cluster". The Spark application is running on a single machine, not on a cluster of machines.


NEW QUESTION # 118
Which of the following describes Spark's standalone deployment mode?

  • A. Standalone mode uses a single JVM to run Spark driver and executor processes.
  • B. Standalone mode uses only a single executor per worker per application.
  • C. Standalone mode is how Spark runs on YARN and Mesos clusters.
  • D. Standalone mode is a viable solution for clusters that run multiple frameworks, not only Spark.
  • E. Standalone mode means that the cluster does not contain the driver.

Answer: B

Explanation:
Explanation
Standalone mode uses only a single executor per worker per application.
This is correct and a limitation of Spark's standalone mode.
Standalone mode is a viable solution for clusters that run multiple frameworks.
Incorrect. A limitation of standalone mode is that Apache Spark must be the only framework running on the cluster. If you would want to run multiple frameworks on the same cluster in parallel, for example Apache Spark and Apache Flink, you would consider the YARN deployment mode.
Standalone mode uses a single JVM to run Spark driver and executor processes.
No, this is what local mode does.
Standalone mode is how Spark runs on YARN and Mesos clusters.
No. YARN and Mesos modes are two deployment modes that are different from standalone mode. These modes allow Spark to run alongside other frameworks on a cluster. When Spark is run in standalone mode, only the Spark framework can run on the cluster.
Standalone mode means that the cluster does not contain the driver.
Incorrect, the cluster does not contain the driver in client mode, but in standalone mode the driver runs on a node in the cluster.
More info: Learning Spark, 2nd Edition, Chapter 1


NEW QUESTION # 119
Which of the following describes Spark's way of managing memory?

  • A. As a general rule for garbage collection, Spark performs better on many small objects than few big objects.
  • B. Disabling serialization potentially greatly reduces the memory footprint of a Spark application.
  • C. Spark's memory usage can be divided into three categories: Execution, transaction, and storage.
  • D. Spark uses a subset of the reserved system memory.
  • E. Storage memory is used for caching partitions derived from DataFrames.

Answer: E

Explanation:
Explanation
Spark's memory usage can be divided into three categories: Execution, transaction, and storage.
No, it is either execution or storage.
As a general rule for garbage collection, Spark performs better on many small objects than few big objects.
No, Spark's garbage collection runs faster on fewer big objects than many small objects.
Disabling serialization potentially greatly reduces the memory footprint of a Spark application.
The opposite is true - serialization reduces the memory footprint, but may impact performance in a negative way.
Spark uses a subset of the reserved system memory.
No, the reserved system memory is separate from Spark memory. Reserved memory stores Spark's internal objects.
More info: Tuning - Spark 3.1.2 Documentation, Spark Memory Management | Distributed Systems Architecture, Learning Spark, 2nd Edition, Chapter 7


NEW QUESTION # 120
Which of the following is a problem with using accumulators?

  • A. Only numeric values can be used in accumulators.
  • B. Accumulators are difficult to use for debugging because they will only be updated once, independent if a task has to be re-run due to hardware failure.
  • C. Only unnamed accumulators can be inspected in the Spark UI.
  • D. Accumulators do not obey lazy evaluation.
  • E. Accumulator values can only be read by the driver, but not by executors.

Answer: E

Explanation:
Explanation
Accumulator values can only be read by the driver, but not by executors.
Correct. So, for example, you cannot use an accumulator variable for coordinating workloads between executors. The typical, canonical, use case of an accumulator value is to report data, for example for debugging purposes, back to the driver. For example, if you wanted to count values that match a specific condition in a UDF for debugging purposes, an accumulator provides a good way to do that.
Only numeric values can be used in accumulators.
No. While pySpark's Accumulator only supports numeric values (think int and float), you can define accumulators for custom types via the AccumulatorParam interface (documentation linked below).
Accumulators do not obey lazy evaluation.
Incorrect - accumulators do obey lazy evaluation. This has implications in practice: When an accumulator is encapsulated in a transformation, that accumulator will not be modified until a subsequent action is run.
Accumulators are difficult to use for debugging because they will only be updated once, independent if a task has to be re-run due to hardware failure.
Wrong. A concern with accumulators is in fact that under certain conditions they can run for each task more than once. For example, if a hardware failure occurs during a task after an accumulator variable has been increased but before a task has finished and Spark launches the task on a different worker in response to the failure, already executed accumulator variable increases will be repeated.
Only unnamed accumulators can be inspected in the Spark UI.
No. Currently, in PySpark, no accumulators can be inspected in the Spark UI. In the Scala interface of Spark, only named accumulators can be inspected in the Spark UI.
More info: Aggregating Results with Spark Accumulators | Sparkour, RDD Programming Guide - Spark 3.1.2 Documentation, pyspark.Accumulator - PySpark 3.1.2 documentation, and pyspark.AccumulatorParam - PySpark 3.1.2 documentation


NEW QUESTION # 121
Which of the following code blocks returns approximately 1000 rows, some of them potentially being duplicates, from the 2000-row DataFrame transactionsDf that only has unique rows?

  • A. transactionsDf.sample(False, 0.5)
  • B. transactionsDf.take(1000).distinct()
  • C. transactionsDf.take(1000)
  • D. transactionsDf.sample(True, 0.5)
  • E. transactionsDf.sample(True, 0.5, force=True)

Answer: D

Explanation:
Explanation
To solve this question, you need to know that DataFrame.sample() is not guaranteed to return the exact fraction of the number of rows specified as an argument. Furthermore, since duplicates may be returned, you should understand that the operator's withReplacement argument should be set to True. A force= argument for the operator does not exist.
While the take argument returns an exact number of rows, it will just take the first specified number of rows (1000 in this question) from the DataFrame. Since the DataFrame does not include duplicate rows, there is no potential of any of those returned rows being duplicates when using take(), so the correct answer cannot involve take().
More info: pyspark.sql.DataFrame.sample - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2


NEW QUESTION # 122
Which of the following is not a feature of Adaptive Query Execution?

  • A. Replace a sort merge join with a broadcast join, where appropriate.
  • B. Reroute a query in case of an executor failure.
  • C. Split skewed partitions into smaller partitions to avoid differences in partition processing time.
  • D. Collect runtime statistics during query execution.
  • E. Coalesce partitions to accelerate data processing.

Answer: B

Explanation:
Explanation
Reroute a query in case of an executor failure.
Correct. Although this feature exists in Spark, it is not a feature of Adaptive Query Execution. The cluster manager keeps track of executors and will work together with the driver to launch an executor and assign the workload of the failed executor to it (see also link below).
Replace a sort merge join with a broadcast join, where appropriate.
No, this is a feature of Adaptive Query Execution.
Coalesce partitions to accelerate data processing.
Wrong, Adaptive Query Execution does this.
Collect runtime statistics during query execution.
Incorrect, Adaptive Query Execution (AQE) collects these statistics to adjust query plans. This feedback loop is an essential part of accelerating queries via AQE.
Split skewed partitions into smaller partitions to avoid differences in partition processing time.
No, this is indeed a feature of Adaptive Query Execution. Find more information in the Databricks blog post linked below.
More info: Learning Spark, 2nd Edition, Chapter 12, On which way does RDD of spark finish fault-tolerance?
- Stack Overflow, How to Speed up SQL Queries with Adaptive Query Execution


NEW QUESTION # 123
Which is the highest level in Spark's execution hierarchy?

  • A. Slot
  • B. Task
  • C. Executor
  • D. Stage
  • E. Job

Answer: E


NEW QUESTION # 124
Which of the following code blocks creates a new 6-column DataFrame by appending the rows of the
6-column DataFrame yesterdayTransactionsDf to the rows of the 6-column DataFrame todayTransactionsDf, ignoring that both DataFrames have different column names?

  • A. todayTransactionsDf.concat(yesterdayTransactionsDf)
  • B. union(todayTransactionsDf, yesterdayTransactionsDf)
  • C. todayTransactionsDf.unionByName(yesterdayTransactionsDf, allowMissingColumns=True)
  • D. todayTransactionsDf.unionByName(yesterdayTransactionsDf)
  • E. todayTransactionsDf.union(yesterdayTransactionsDf)

Answer: E

Explanation:
Explanation
todayTransactionsDf.union(yesterdayTransactionsDf)
Correct. The union command appends rows of yesterdayTransactionsDf to the rows of todayTransactionsDf, ignoring that both DataFrames have different column names. The resulting DataFrame will have the column names of DataFrame todayTransactionsDf.
todayTransactionsDf.unionByName(yesterdayTransactionsDf)
No. unionByName specifically tries to match columns in the two DataFrames by name and only appends values in columns with identical names across the two DataFrames. In the form presented above, the command is a great fit for joining DataFrames that have exactly the same columns, but in a different order. In this case though, the command will fail because the two DataFrames have different columns.
todayTransactionsDf.unionByName(yesterdayTransactionsDf, allowMissingColumns=True) No. The unionByName command is described in the previous explanation. However, with the allowMissingColumns argument set to True, it is no longer an issue that the two DataFrames have different column names. Any columns that do not have a match in the other DataFrame will be filled with null where there is no value. In the case at hand, the resulting DataFrame will have 7 or more columns though, so it this command is not the right answer.
union(todayTransactionsDf, yesterdayTransactionsDf)
No, there is no union method in pyspark.sql.functions.
todayTransactionsDf.concat(yesterdayTransactionsDf)
Wrong, the DataFrame class does not have a concat method.
More info: pyspark.sql.DataFrame.union - PySpark 3.1.2 documentation,
pyspark.sql.DataFrame.unionByName - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3


NEW QUESTION # 125
The code block shown below should store DataFrame transactionsDf on two different executors, utilizing the executors' memory as much as possible, but not writing anything to disk. Choose the answer that correctly fills the blanks in the code block to accomplish this.
1.from pyspark import StorageLevel
2.transactionsDf.__1__(StorageLevel.__2__).__3__

  • A. 1. persist
    2. DISK_ONLY_2
    3. count()
  • B. 1. persist
    2. MEMORY_ONLY_2
    3. select()
  • C. 1. cache
    2. DISK_ONLY_2
    3. count()
  • D. 1. cache
    2. MEMORY_ONLY_2
    3. count()
  • E. 1. persist
    2. MEMORY_ONLY_2
    3. count()

Answer: E

Explanation:
Explanation
Correct code block:
from pyspark import StorageLevel
transactionsDf.persist(StorageLevel.MEMORY_ONLY_2).count()
Only persist takes different storage levels, so any option using cache() cannot be correct. persist() is evaluated lazily, so an action needs to follow this command. select() is not an action, but count() is - so all options using select() are incorrect.
Finally, the question states that "the executors' memory should be utilized as much as possible, but not writing anything to disk". This points to a MEMORY_ONLY storage level. In this storage level, partitions that do not fit into memory will be recomputed when they are needed, instead of being written to disk, as with the storage option MEMORY_AND_DISK. Since the data need to be duplicated across two executors, _2 needs to be appended to the storage level.
Static notebook | Dynamic notebook: See test 2


NEW QUESTION # 126
Which of the following code blocks returns a single-column DataFrame showing the number of words in column supplier of DataFrame itemsDf?
Sample of DataFrame itemsDf:
1.+------+-----------------------------+-------------------+
2.|itemId|attributes |supplier |
3.+------+-----------------------------+-------------------+
4.|1 |[blue, winter, cozy] |Sports Company Inc.|
5.|2 |[red, summer, fresh, cooling]|YetiX |
6.|3 |[green, summer, travel] |Sports Company Inc.|
7.+------+-----------------------------+-------------------+

  • A. itemsDf.select(size(split("supplier", " ")))
  • B. itemsDf.split("supplier", " ").size()
  • C. spark.select(size(split(col(supplier), " ")))
  • D. itemsDf.select(word_count("supplier"))
  • E. itemsDf.split("supplier", " ").count()

Answer: A

Explanation:
Explanation
Output of correct code block:
+----------------------------+
|size(split(supplier, , -1))|
+----------------------------+
| 3|
| 1|
| 3|
+----------------------------+
This question shows a typical use case for the split command: Splitting a string into words. An additional difficulty is that you are asked to count the words. Although it is tempting to use the count method here, the size method (as in: size of an array) is actually the correct one to use. Familiarize yourself with the split and the size methods using the linked documentation below.
More info:
Split method: pyspark.sql.functions.split - PySpark 3.1.2 documentation Size method: pyspark.sql.functions.size - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2


NEW QUESTION # 127
Which of the following code blocks returns a copy of DataFrame transactionsDf where the column storeId has been converted to string type?

  • A. transactionsDf.withColumn("storeId", col("storeId").cast("string"))
  • B. transactionsDf.withColumn("storeId", convert("storeId", "string"))
  • C. transactionsDf.withColumn("storeId", convert("storeId").as("string"))
  • D. transactionsDf.withColumn("storeId", col("storeId").convert("string"))
  • E. transactionsDf.withColumn("storeId", col("storeId", "string"))

Answer: A

Explanation:
Explanation
This question asks for your knowledge about the cast syntax. cast is a method of the Column class. It is worth noting that one could also convert a column type using the Column.astype() method, which is just an alias for cast.
Find more info in the documentation linked below.
More info: pyspark.sql.Column.cast - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2


NEW QUESTION # 128
The code block displayed below contains an error. The code block should configure Spark so that DataFrames up to a size of 20 MB will be broadcast to all worker nodes when performing a join.
Find the error.
Code block:

  • A. Spark will only broadcast DataFrames that are much smaller than the default value.
  • B. The correct option to write configurations is through spark.config and not spark.conf.
  • C. The passed limit has the wrong variable type.
  • D. spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 20)
  • E. The command is evaluated lazily and needs to be followed by an action.
  • F. Spark will only apply the limit to threshold joins and not to other joins.

Answer: A

Explanation:
Explanation
This is question is hard. Let's assess the different answers one-by-one.
Spark will only broadcast DataFrames that are much smaller than the default value.
This is correct. The default value is 10 MB (10485760 bytes). Since the configuration for spark.sql.autoBroadcastJoinThreshold expects a number in bytes (and not megabytes), the code block sets the limits to merely 20 bytes, instead of the requested 20 * 1024 * 1024 (= 20971520) bytes.
The command is evaluated lazily and needs to be followed by an action.
No, this command is evaluated right away!
Spark will only apply the limit to threshold joins and not to other joins.
There are no "threshold joins", so this option does not make any sense.
The correct option to write configurations is through spark.config and not spark.conf.
No, it is indeed spark.conf!
The passed limit has the wrong variable type.
The configuration expects the number of bytes, a number, as an input. So, the 20 provided in the code block is fine.


NEW QUESTION # 129
......

Free Exam Files Downloaded Instantly: https://passking.actualtorrent.com/Associate-Developer-Apache-Spark-exam-guide-torrent.html

Related Blogs

more
Databricks Associate-Developer-Apache-Spark Certification Practice Exam

Copyright © 2026 ACTUALTORRENT.COM. All rights reserved. All trademarks are the property of their respective owners and we don't provide actual questions from any vendor.