2023 Latest 100% Exam Passing Ratio - Associate-Developer-Apache-Spark Dumps PDF
Pass Exam With Full Sureness - Associate-Developer-Apache-Spark Dumps with 179 Questions
What is the Databricks Associate Developer Apache Spark Exam?
The Databricks Associate Developer Apache Spark Exam is a certification that can be earned by anyone who has successfully completed the Databricks Associate Developer Apache Spark Certification Training. The exam covers all the material that was covered in the training. The exam is designed to test your knowledge of the concepts, skills, and abilities that you learned during the course.
Do you want to become a Data Engineer or a Spark Architect? If so, then the Databricks Associate Developer Apache Spark Exam is a must-pass. The Databricks Associate Developer Apache Spark Exam is designed to help you develop a complete understanding of the technology used by the Databricks platform. You will learn about the basics of Spark, including the Spark programming language, Spark SQL, Spark Streaming, and the Spark ecosystem. Databricks Associate Developer Apache Spark exam dumps are the choice of champions.
The Databricks Associate Developer Apache Spark Exam is a test that aims to assess whether you have the knowledge required to become a certified Apache Spark developer. The Databricks Associate Developer Apache Spark Exam consists of two parts: the first part tests your knowledge of the fundamentals of the Apache Spark framework and the second part tests your ability to apply this knowledge. This post will help you get a head start in preparing for the Databricks Associate Developer Apache Spark Exam. The executors disk division actions documentation frame for the executor syntax variables object return allowed partition for the fit output transformation to induce couple of manager and evaluated expected safely, lazily named nodes broadcast operations for correctly mock driver.
How to become Databricks Associate Developer Apache Spark certified
The Databricks Associate Developer Apache Spark certification is a great opportunity to get hands-on experience with the most popular big data platform. It's a two-day certification course that teaches you how to use the latest version of Apache Spark. Databricks Associate Developer Apache Spark exam dumps are the key of success.
To create applications that run on the Databricks platform. You'll learn how to use the Spark shell to interact with the data stored in Databricks, the Databricks DataFrame API, and the Databricks Query Language (DQL). You'll also gain exposure to a variety of machine learning and data mining algorithms that can be used to solve real-world problems.
NEW QUESTION 80
Which of the following code blocks reads all CSV files in directory filePath into a single DataFrame, with column names defined in the CSV file headers?
Content of directory filePath:
1._SUCCESS
2._committed_2754546451699747124
3._started_2754546451699747124
4.part-00000-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-298-1-c000.csv.gz
5.part-00001-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-299-1-c000.csv.gz
6.part-00002-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-300-1-c000.csv.gz
7.part-00003-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-301-1-c000.csv.gz spark.option("header",True).csv(filePath)
- A. spark.read.load(filePath)
- B. spark.read.format("csv").option("header",True).option("compression","zip").load(filePath)
- C. spark.read().option("header",True).load(filePath)
- D. spark.read.format("csv").option("header",True).load(filePath)
Answer: D
Explanation:
Explanation
The files in directory filePath are partitions of a DataFrame that have been exported using gzip compression.
Spark automatically recognizes this situation and imports the CSV files as separate partitions into a single DataFrame. It is, however, necessary to specify that Spark should load the file headers in the CSV with the header option, which is set to False by default.
NEW QUESTION 81
The code block shown below should return a single-column DataFrame with a column named consonant_ct that, for each row, shows the number of consonants in column itemName of DataFrame itemsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.
DataFrame itemsDf:
1.+------+----------------------------------+-----------------------------+-------------------+
2.|itemId|itemName |attributes |supplier |
3.+------+----------------------------------+-----------------------------+-------------------+
4.|1 |Thick Coat for Walking in the Snow|[blue, winter, cozy] |Sports Company Inc.|
5.|2 |Elegant Outdoors Summer Dress |[red, summer, fresh, cooling]|YetiX |
6.|3 |Outdoors Backpack |[green, summer, travel] |Sports Company Inc.|
7.+------+----------------------------------+-----------------------------+-------------------+ Code block:
itemsDf.select(__1__(__2__(__3__(__4__), "a|e|i|o|u|\s", "")).__5__("consonant_ct"))
- A. 1. lower
2. regexp_replace
3. length
4. "itemName"
5. alias - B. 1. size
2. regexp_extract
3. lower
4. col("itemName")
5. alias - C. 1. size
2. regexp_replace
3. lower
4. "itemName"
5. alias - D. 1. length
2. regexp_replace
3. lower
4. col("itemName")
5. alias - E. 1. length
2. regexp_extract
3. upper
4. col("itemName")
5. as
Answer: D
Explanation:
Explanation
Correct code block:
itemsDf.select(length(regexp_replace(lower(col("itemName")), "a|e|i|o|u|\s", "")).alias("consonant_ct")) Returned DataFrame:
+------------+
|consonant_ct|
+------------+
| 19|
| 16|
| 10|
+------------+
This question tries to make you think about the string functions Spark provides and in which order they should be applied. Arguably the most difficult part, the regular expression "a|e|i|o|u|
\s", is not a numbered blank. However, if you are not familiar with the string functions, it may be a good idea to review those before the exam.
The size operator and the length operator can easily be confused. size works on arrays, while length works on strings. Luckily, this is something you can read up about in the documentation.
The code block works by first converting all uppercase letters in column itemName into lowercase (the lower() part). Then, it replaces all vowels by "nothing" - an empty character "" (the regexp_replace() part). Now, only lowercase characters without spaces are included in the DataFrame. Then, per row, the length operator counts these remaining characters. Note that column itemName in itemsDf does not include any numbers or other characters, so we do not need to make any provisions for these. Finally, by using the alias() operator, we rename the resulting column to consonant_ct.
More info:
- lower: pyspark.sql.functions.lower - PySpark 3.1.2 documentation
- regexp_replace: pyspark.sql.functions.regexp_replace - PySpark 3.1.2 documentation
- length: pyspark.sql.functions.length - PySpark 3.1.2 documentation
- alias: pyspark.sql.Column.alias - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2
NEW QUESTION 82
Which is the highest level in Spark's execution hierarchy?
- A. Slot
- B. Stage
- C. Task
- D. Executor
- E. Job
Answer: E
NEW QUESTION 83
Which of the following code blocks reads the parquet file stored at filePath into DataFrame itemsDf, using a valid schema for the sample of itemsDf shown below?
Sample of itemsDf:
1.+------+-----------------------------+-------------------+
2.|itemId|attributes |supplier |
3.+------+-----------------------------+-------------------+
4.|1 |[blue, winter, cozy] |Sports Company Inc.|
5.|2 |[red, summer, fresh, cooling]|YetiX |
6.|3 |[green, summer, travel] |Sports Company Inc.|
7.+------+-----------------------------+-------------------+
- A. 1.itemsDfSchema = StructType([
2. StructField("itemId", IntegerType()),
3. StructField("attributes", ArrayType([StringType()])),
4. StructField("supplier", StringType())])
5.
6.itemsDf = spark.read(schema=itemsDfSchema).parquet(filePath) - B. 1.itemsDfSchema = StructType([
2. StructField("itemId", IntegerType()),
3. StructField("attributes", StringType()),
4. StructField("supplier", StringType())])
5.
6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath) - C. 1.itemsDfSchema = StructType([
2. StructField("itemId", IntegerType),
3. StructField("attributes", ArrayType(StringType)),
4. StructField("supplier", StringType)])
5.
6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath) - D. 1.itemsDfSchema = StructType([
2. StructField("itemId", IntegerType()),
3. StructField("attributes", ArrayType(StringType())),
4. StructField("supplier", StringType())])
5.
6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath) - E. 1.itemsDf = spark.read.schema('itemId integer, attributes <string>, supplier string').parquet(filePath)
Answer: D
Explanation:
Explanation
The challenge in this question comes from there being an array variable in the schema. In addition, you should know how to pass a schema to the DataFrameReader that is invoked by spark.read.
The correct way to define an array of strings in a schema is through ArrayType(StringType()). A schema can be passed to the DataFrameReader by simply appending schema(structType) to the read() operator. Alternatively, you can also define a schema as a string. For example, for the schema of itemsDf, the following string would make sense: itemId integer, attributes array<string>, supplier string.
A thing to keep in mind is that in schema definitions, you always need to instantiate the types, like so:
StringType(). Just using StringType does not work in pySpark and will fail.
Another concern with schemas is whether columns should be nullable, so allowed to have null values. In the case at hand, this is not a concern however, since the question just asks for a
"valid"
schema. Both non-nullable and nullable column schemas would be valid here, since no null value appears in the DataFrame sample.
More info: Learning Spark, 2nd Edition, Chapter 3
Static notebook | Dynamic notebook: See test 3
NEW QUESTION 84
Which of the following code blocks reads in the two-partition parquet file stored at filePath, making sure all columns are included exactly once even though each partition has a different schema?
Schema of first partition:
1.root
2. |-- transactionId: integer (nullable = true)
3. |-- predError: integer (nullable = true)
4. |-- value: integer (nullable = true)
5. |-- storeId: integer (nullable = true)
6. |-- productId: integer (nullable = true)
7. |-- f: integer (nullable = true)
Schema of second partition:
1.root
2. |-- transactionId: integer (nullable = true)
3. |-- predError: integer (nullable = true)
4. |-- value: integer (nullable = true)
5. |-- storeId: integer (nullable = true)
6. |-- rollId: integer (nullable = true)
7. |-- f: integer (nullable = true)
8. |-- tax_id: integer (nullable = false)
- A. 1.nx = 0
2.for file in dbutils.fs.ls(filePath):
3. if not file.name.endswith(".parquet"):
4. continue
5. df_temp = spark.read.parquet(file.path)
6. if nx == 0:
7. df = df_temp
8. else:
9. df = df.union(df_temp)
10. nx = nx+1
11.df - B. spark.read.parquet(filePath)
- C. spark.read.parquet(filePath, mergeSchema='y')
- D. 1.nx = 0
2.for file in dbutils.fs.ls(filePath):
3. if not file.name.endswith(".parquet"):
4. continue
5. df_temp = spark.read.parquet(file.path)
6. if nx == 0:
7. df = df_temp
8. else:
9. df = df.join(df_temp, how="outer")
10. nx = nx+1
11.df - E. spark.read.option("mergeSchema", "true").parquet(filePath)
Answer: E
Explanation:
Explanation
This is a very tricky question and involves both knowledge about merging as well as schemas when reading parquet files.
spark.read.option("mergeSchema", "true").parquet(filePath)
Correct. Spark's DataFrameReader's mergeSchema option will work well here, since columns that appear in both partitions have matching data types. Note that mergeSchema would fail if one or more columns with the same name that appear in both partitions would have different data types.
spark.read.parquet(filePath)
Incorrect. While this would read in data from both partitions, only the schema in the parquet file that is read in first would be considered, so some columns that appear only in the second partition (e.g. tax_id) would be lost.
nx = 0
for file in dbutils.fs.ls(filePath):
if not file.name.endswith(".parquet"):
continue
df_temp = spark.read.parquet(file.path)
if nx == 0:
df = df_temp
else:
df = df.union(df_temp)
nx = nx+1
df
Wrong. The key idea of this solution is the DataFrame.union() command. While this command merges all data, it requires that both partitions have the exact same number of columns with identical data types.
spark.read.parquet(filePath, mergeSchema="y")
False. While using the mergeSchema option is the correct way to solve this problem and it can even be called with DataFrameReader.parquet() as in the code block, it accepts the value True as a boolean or string variable. But 'y' is not a valid option.
nx = 0
for file in dbutils.fs.ls(filePath):
if not file.name.endswith(".parquet"):
continue
df_temp = spark.read.parquet(file.path)
if nx == 0:
df = df_temp
else:
df = df.join(df_temp, how="outer")
nx = nx+1
df
No. This provokes a full outer join. While the resulting DataFrame will have all columns of both partitions, columns that appear in both partitions will be duplicated - the question says all columns that are included in the partitions should appear exactly once.
More info: Merging different schemas in Apache Spark | by Thiago Cordon | Data Arena | Medium Static notebook | Dynamic notebook: See test 3
NEW QUESTION 85
The code block shown below should add a column itemNameBetweenSeparators to DataFrame itemsDf. The column should contain arrays of maximum 4 strings. The arrays should be composed of the values in column itemsDf which are separated at - or whitespace characters. Choose the answer that correctly fills the blanks in the code block to accomplish this.
Sample of DataFrame itemsDf:
1.+------+----------------------------------+-------------------+
2.|itemId|itemName |supplier |
3.+------+----------------------------------+-------------------+
4.|1 |Thick Coat for Walking in the Snow|Sports Company Inc.|
5.|2 |Elegant Outdoors Summer Dress |YetiX |
6.|3 |Outdoors Backpack |Sports Company Inc.|
7.+------+----------------------------------+-------------------+
Code block:
itemsDf.__1__(__2__, __3__(__4__, "[\s\-]", __5__))
- A. 1. withColumn
2. itemNameBetweenSeparators
3. str_split
4. "itemName"
5. 5 - B. 1. withColumn
2. "itemNameBetweenSeparators"
3. split
4. "itemName"
5. 4
(Correct) - C. 1. withColumnRenamed
2. "itemName"
3. split
4. "itemNameBetweenSeparators"
5. 4 - D. 1. withColumnRenamed
2. "itemNameBetweenSeparators"
3. split
4. "itemName"
5. 4 - E. 1. withColumn
2. "itemNameBetweenSeparators"
3. split
4. "itemName"
5. 5
Answer: B
Explanation:
Explanation
This question deals with the parameters of Spark's split operator for strings.
To solve this question, you first need to understand the difference between DataFrame.withColumn() and DataFrame.withColumnRenamed(). The correct option here is DataFrame.withColumn() since, according to the question, we want to add a column and not rename an existing column. This leaves you with only 3 answers to consider.
The second gap should be filled with the name of the new column to be added to the DataFrame. One of the remaining answers states the column name as itemNameBetweenSeparators, while the other two state it as "itemNameBetweenSeparators". The correct option here is
"itemNameBetweenSeparators", since the other option would let Python try to interpret itemNameBetweenSeparators as the name of a variable, which we have not defined. This leaves you with 2 answers to consider.
The decision boils down to how to fill gap 5. Either with 4 or with 5. The question asks for arrays of maximum four strings. The code in gap 5 relates to the limit parameter of Spark's split operator (see documentation linked below). The documentation states that "the resulting array's length will not be more than limit", meaning that we should pick the answer option with 4 as the code in the fifth gap here.
On a side note: One answer option includes a function str_split. This function does not exist in pySpark.
More info: pyspark.sql.functions.split - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
NEW QUESTION 86
Which of the following code blocks reads in parquet file /FileStore/imports.parquet as a DataFrame?
- A. spark.read().parquet("/FileStore/imports.parquet")
- B. spark.read().format('parquet').open("/FileStore/imports.parquet")
- C. spark.read.path("/FileStore/imports.parquet", source="parquet")
- D. spark.read.parquet("/FileStore/imports.parquet")
- E. spark.mode("parquet").read("/FileStore/imports.parquet")
Answer: D
Explanation:
Explanation
Static notebook | Dynamic notebook: See test 1
(https://flrs.github.io/spark_practice_tests_code/#1/23.html ,
https://bit.ly/sparkpracticeexams_import_instructions)
NEW QUESTION 87
Which of the following code blocks generally causes a great amount of network traffic?
- A. DataFrame.rdd.map()
- B. DataFrame.select()
- C. DataFrame.collect()
- D. DataFrame.count()
- E. DataFrame.coalesce()
Answer: C
Explanation:
Explanation
DataFrame.collect() sends all data in a DataFrame from executors to the driver, so this generally causes a great amount of network traffic in comparison to the other options listed.
DataFrame.coalesce() just reduces the number of partitions and generally aims to reduce network traffic in comparison to a full shuffle.
DataFrame.select() is evaluated lazily and, unless followed by an action, does not cause significant network traffic.
DataFrame.rdd.map() is evaluated lazily, it does therefore not cause great amounts of network traffic.
DataFrame.count() is an action. While it does cause some network traffic, for the same DataFrame, collecting all data in the driver would generally be considered to cause a greater amount of network traffic.
NEW QUESTION 88
In which order should the code blocks shown below be run in order to read a JSON file from location jsonPath into a DataFrame and return only the rows that do not have value 3 in column productId?
1. importedDf.createOrReplaceTempView("importedDf")
2. spark.sql("SELECT * FROM importedDf WHERE productId != 3")
3. spark.sql("FILTER * FROM importedDf WHERE productId != 3")
4. importedDf = spark.read.option("format", "json").path(jsonPath)
5. importedDf = spark.read.json(jsonPath)
- A. 4, 1, 2
- B. 5, 1, 2
- C. 5, 2
- D. 4, 1, 3
- E. 5, 1, 3
Answer: B
Explanation:
Explanation
Correct code block:
importedDf = spark.read.json(jsonPath)
importedDf.createOrReplaceTempView("importedDf")
spark.sql("SELECT * FROM importedDf WHERE productId != 3")
Option 5 is the only correct way listed of reading in a JSON in PySpark. The option("format", "json") is not the correct way to tell Spark's DataFrameReader that you want to read a JSON file. You would do this through format("json") instead. Also, you can communicate the specific path of the JSON file to the DataFramReader using the load() method, not the path() method.
In order to use a SQL command through the SparkSession spark, you first need to create a temporary view through DataFrame.createOrReplaceTempView().
The SQL statement should start with the SELECT operator. The FILTER operator SQL provides is not the correct one to use here.
Static notebook | Dynamic notebook: See test 2
NEW QUESTION 89
Which of the following code blocks applies the Python function to_limit on column predError in table transactionsDf, returning a DataFrame with columns transactionId and result?
- A. 1.spark.udf.register(to_limit, "LIMIT_FCN")
2.spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf") - B. 1.spark.udf.register("LIMIT_FCN", to_limit)
2.spark.sql("SELECT transactionId, to_limit(predError) AS result FROM transactionsDf") spark.sql("SELECT transactionId, udf(to_limit(predError)) AS result FROM transactionsDf") - C. 1.spark.udf.register("LIMIT_FCN", to_limit)
2.spark.sql("SELECT transactionId, LIMIT_FCN(predError) FROM transactionsDf AS result") - D. 1.spark.udf.register("LIMIT_FCN", to_limit)
2.spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf") (Correct)
Answer: D
Explanation:
Explanation
spark.udf.register("LIMIT_FCN", to_limit)
spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf") Correct! First, you have to register to_limit as UDF to use it in a sql statement. Then, you can use it under the LIMIT_FCN name, correctly naming the resulting column result.
spark.udf.register(to_limit, "LIMIT_FCN")
spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf") No. In this answer, the arguments to spark.udf.register are flipped.
spark.udf.register("LIMIT_FCN", to_limit)
spark.sql("SELECT transactionId, to_limit(predError) AS result FROM transactionsDf") Wrong, this answer does not use the registered LIMIT_FCN in the sql statement, but tries to access the to_limit method directly. This will fail, since Spark cannot access it.
spark.sql("SELECT transactionId, udf(to_limit(predError)) AS result FROM transactionsDf") Incorrect, there is no udf method in Spark's SQL.
spark.udf.register("LIMIT_FCN", to_limit)
spark.sql("SELECT transactionId, LIMIT_FCN(predError) FROM transactionsDf AS result") False. In this answer, the column that results from applying the UDF is not correctly renamed to result.
Static notebook | Dynamic notebook: See test 3
NEW QUESTION 90
Which of the following statements about lazy evaluation is incorrect?
- A. Lineages allow Spark to coalesce transformations into stages
- B. Spark will fail a job only during execution, but not during definition.
- C. Execution is triggered by transformations.
- D. Accumulators do not change the lazy evaluation model of Spark.
- E. Predicate pushdown is a feature resulting from lazy evaluation.
Answer: C
Explanation:
Explanation
Execution is triggered by transformations.
Correct. Execution is triggered by actions only, not by transformations.
Lineages allow Spark to coalesce transformations into stages.
Incorrect. In Spark, lineage means a recording of transformations. This lineage enables lazy evaluation in Spark.
Predicate pushdown is a feature resulting from lazy evaluation.
Wrong. Predicate pushdown means that, for example, Spark will execute filters as early in the process as possible so that it deals with the least possible amount of data in subsequent transformations, resulting in a performance improvements.
Accumulators do not change the lazy evaluation model of Spark.
Incorrect. In Spark, accumulators are only updated when the query that refers to the is actually executed. In other words, they are not updated if the query is not (yet) executed due to lazy evaluation.
Spark will fail a job only during execution, but not during definition.
Wrong. During definition, due to lazy evaluation, the job is not executed and thus certain errors, for example reading from a non-existing file, cannot be caught. To be caught, the job needs to be executed, for example through an action.
NEW QUESTION 91
The code block displayed below contains an error. The code block should read the csv file located at path data/transactions.csv into DataFrame transactionsDf, using the first row as column header and casting the columns in the most appropriate type. Find the error.
First 3 rows of transactions.csv:
1.transactionId;storeId;productId;name
2.1;23;12;green grass
3.2;35;31;yellow sun
4.3;23;12;green grass
Code block:
transactionsDf = spark.read.load("data/transactions.csv", sep=";", format="csv", header=True)
- A. Spark is unable to understand the file type.
- B. The resulting DataFrame will not have the appropriate schema.
- C. The transaction is evaluated lazily, so no file will be read.
- D. The code block is unable to capture all columns.
- E. The DataFrameReader is not accessed correctly.
Answer: B
Explanation:
Explanation
Correct code block:
transactionsDf = spark.read.load("data/transactions.csv", sep=";", format="csv", header=True, inferSchema=True) By default, Spark does not infer the schema of the CSV (since this usually takes some time). So, you need to add the inferSchema=True option to the code block.
More info: pyspark.sql.DataFrameReader.csv - PySpark 3.1.2 documentation
NEW QUESTION 92
Which of the following DataFrame operators is never classified as a wide transformation?
- A. DataFrame.select()
- B. DataFrame.sort()
- C. DataFrame.repartition()
- D. DataFrame.aggregate()
- E. DataFrame.join()
Answer: A
Explanation:
Explanation
As a general rule: After having gone through the practice tests you probably have a good feeling for what classifies as a wide and what classifies as a narrow transformation. If you are unsure, feel free to play around in Spark and display the explanation of the Spark execution plan via DataFrame.[operation, for example sort()].explain(). If repartitioning is involved, it would count as a wide transformation.
DataFrame.select()
Correct! A wide transformation includes a shuffle, meaning that an input partition maps to one or more output partitions. This is expensive and causes traffic across the cluster. With the select() operation however, you pass commands to Spark that tell Spark to perform an operation on a specific slice of any partition. For this, Spark does not need to exchange data across partitions, each partition can be worked on independently. Thus, you do not cause a wide transformation.
DataFrame.repartition()
Incorrect. When you repartition a DataFrame, you redefine partition boundaries. Data will flow across your cluster and end up in different partitions after the repartitioning is completed. This is known as a shuffle and, in turn, is classified as a wide transformation.
DataFrame.aggregate()
No. When you aggregate, you may compare and summarize data across partitions. In the process, data are exchanged across the cluster, and newly formed output partitions depend on one or more input partitions. This is a typical characteristic of a shuffle, meaning that the aggregate operation may classify as a wide transformation.
DataFrame.join()
Wrong. Joining multiple DataFrames usually means that large amounts of data are exchanged across the cluster, as new partitions are formed. This is a shuffle and therefore DataFrame.join() counts as a wide transformation.
DataFrame.sort()
False. When sorting, Spark needs to compare many rows across all partitions to each other. This is an expensive operation, since data is exchanged across the cluster and new partitions are formed as data is reordered. This process classifies as a shuffle and, as a result, DataFrame.sort() counts as wide transformation.
More info: Understanding Apache Spark Shuffle | Philipp Brunenberg
NEW QUESTION 93
Which of the following code blocks produces the following output, given DataFrame transactionsDf?
Output:
1.root
2. |-- transactionId: integer (nullable = true)
3. |-- predError: integer (nullable = true)
4. |-- value: integer (nullable = true)
5. |-- storeId: integer (nullable = true)
6. |-- productId: integer (nullable = true)
7. |-- f: integer (nullable = true)
DataFrame transactionsDf:
1.+-------------+---------+-----+-------+---------+----+
2.|transactionId|predError|value|storeId|productId| f|
3.+-------------+---------+-----+-------+---------+----+
4.| 1| 3| 4| 25| 1|null|
5.| 2| 6| 7| 2| 2|null|
6.| 3| 3| null| 25| 3|null|
7.+-------------+---------+-----+-------+---------+----+
- A. print(transactionsDf.schema)
- B. transactionsDf.schema.print()
- C. transactionsDf.printSchema()
- D. transactionsDf.rdd.formatSchema()
- E. transactionsDf.rdd.printSchema()
Answer: C
Explanation:
Explanation
The output is the typical output of a DataFrame.printSchema() call. The DataFrame's RDD representation does not have a printSchema or formatSchema method (find available methods in the RDD documentation linked below). The output of print(transactionsDf.schema) is this:
StructType(List(StructField(transactionId,IntegerType,true),StructField(predError,IntegerType,true),StructField (value,IntegerType,true),StructField(storeId,IntegerType,true),StructField(productId,IntegerType,true),StructFiel It includes the same information as the nicely formatted original output, but is not nicely formatted itself. Lastly, the DataFrame's schema attribute does not have a print() method.
More info:
- pyspark.RDD: pyspark.RDD - PySpark 3.1.2 documentation
- DataFrame.printSchema(): pyspark.sql.DataFrame.printSchema - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2
NEW QUESTION 94
Which of the following code blocks returns all unique values across all values in columns value and productId in DataFrame transactionsDf in a one-column DataFrame?
- A. tranactionsDf.select('value').join(transactionsDf.select('productId'), col('value')==col('productId'),
'outer') - B. transactionsDf.select(col('value'), col('productId')).agg({'*': 'count'})
- C. transactionsDf.select('value', 'productId').distinct()
- D. transactionsDf.agg({'value': 'collect_set', 'productId': 'collect_set'})
- E. transactionsDf.select('value').union(transactionsDf.select('productId')).distinct()
Answer: E
Explanation:
Explanation
transactionsDf.select('value').union(transactionsDf.select('productId')).distinct() Correct. This code block uses a common pattern for finding the unique values across multiple columns: union and distinct. In fact, it is so common that it is even mentioned in the Spark documentation for the union command (link below).
transactionsDf.select('value', 'productId').distinct()
Wrong. This code block returns unique rows, but not unique values.
transactionsDf.agg({'value': 'collect_set', 'productId': 'collect_set'}) Incorrect. This code block will output a one-row, two-column DataFrame where each cell has an array of unique values in the respective column (even omitting any nulls).
transactionsDf.select(col('value'), col('productId')).agg({'*': 'count'}) No. This command will count the number of rows, but will not return unique values.
transactionsDf.select('value').join(transactionsDf.select('productId'), col('value')==col('productId'), 'outer') Wrong. This command will perform an outer join of the value and productId columns. As such, it will return a two-column DataFrame. If you picked this answer, it might be a good idea for you to read up on the difference between union and join, a link is posted below.
More info: pyspark.sql.DataFrame.union - PySpark 3.1.2 documentation, sql - What is the difference between JOIN and UNION? - Stack Overflow Static notebook | Dynamic notebook: See test 3
NEW QUESTION 95
The code block displayed below contains an error. The code block should return a copy of DataFrame transactionsDf where the name of column transactionId has been changed to transactionNumber. Find the error.
Code block:
transactionsDf.withColumn("transactionNumber", "transactionId")
- A. The method withColumn should be replaced by method withColumnRenamed and the arguments to the method need to be reordered.
- B. The arguments to the withColumn method need to be reordered.
- C. Each column name needs to be wrapped in the col() method and method withColumn should be replaced by method withColumnRenamed.
- D. The copy() operator should be appended to the code block to ensure a copy is returned.
- E. The arguments to the withColumn method need to be reordered and the copy() operator should be appended to the code block to ensure a copy is returned.
Answer: A
Explanation:
Explanation
Correct code block:
transactionsDf.withColumnRenamed("transactionId", "transactionNumber")
Note that in Spark, a copy is returned by default. So, there is no need to append copy() to the code block.
More info: pyspark.sql.DataFrame.withColumnRenamed - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2
NEW QUESTION 96
......
Verified Associate-Developer-Apache-Spark dumps Q&As - 100% Pass from TestkingPDF: https://www.testkingpdf.com/Associate-Developer-Apache-Spark-testking-pdf-torrent.html
Pass Associate-Developer-Apache-Spark Exam in First Attempt Guaranteed 2023 Dumps: https://drive.google.com/open?id=1QHWGhOnpNnvnV6D2oIngOeiqdr-u0TcK
