Title: 544
Tags: lec21

Q: How do you create a SparkData corresponding to a CSV at location PATH?
A: df = spark.read.format("csv").load(PATH)

Q: How can you see the number of RDD partitions for a Spark DataFrame df?
A: df.rdd.getNumPartitions()

Q: Say you run this:
spark.read.format("csv").load("test.csv")
And test.csv looks like this:
a,b
1,2
3,4
What will be the first row of data?
A: a,b
Because headers are off by default.  Add this:
spark.read.format("csv").option("header", True).load("test.csv")

Q: Say you run this:
spark.read.format("csv").option("header", True).load("test.csv")
And test.csv looks like this:
a,b
1,2
3,4
What will be the data types for the columns?
A: strings, because everything in a CSV is a string.
To get ints, use this:
spark.read.format("csv").option("header", True).option("inferSchema", True).load("test.csv")

Q: If you want to select the "x" column of a Spark DataFrame df, add 1 to it, and call the result column y, how would you do that?
A: df.select(expr("x+1").alias("y"))

Q: Spark gives an error about a file already existing when you run this:
df.select(columns).write.format("parquet").save(PATH)
What can you add to replace the previous file?
A: .mode("overwrite")
Like this:
df.select(columns).write.format("parquet").mode("overwrite").save(PATH)