Title: 544
Tags: lec19

Q: What is the difference between a data warehouse and data lake?
A: Both are a place where we collect data for analysis (OLAP workloads).
A warehouse is a database with integrated storage+computation.
A data lake is a combination of distributed storage (for example, HDFS) and decoupled analytics engine (for example, Spark)

Q: In MapReduce, what parameters do map and reduce functions take?
A: def map(key, value)
def reduce(key, values)
Note that with reduce, all values with the same key will go to a single reduce call on a single machine.  During map, different values with the same key go to different map calls, perhaps even on different machines.

Q: A reduce task might have the reduce function called several times.  What can we say about the keys across these multiple calls for a single task?
A: The keys will be sorted

Q: What determines the number of output files in a MapReduce job?
A: The number of reducers, because each reducer produces one output file.

Q: What are some performance advantages Spark has over MapReduce?
A: intermediate data is handled better (could directly pass from one computation to the next, without materializing in HDFS)
a chain of RDDs lets us specify many steps to get to a desired end result, but lazy execution means that an optimizer could find more efficient ways to get to that same result

Q: If you have a regular Python list, how can you convert it into a Spark RDD?
A: rdd = sc.parallelize(YOUR LIST)

Q: What is the relationship between actions, operations, and transformations?
A: actions and transformations are both kinds of operations
transformations are lazy (no work done yet), and describe how to produce an output RDD given an input RDD
actions actually materialize some output (on the screen, in an HDFS file, etc), triggering whatever work is necessary to compute that output.  An action is often at the end of a chain of transformations.

Q: A Spark task:
  - runs on a _________
  - operates on a _________
A: runs on a single CPU core
operates on a single partition, which is loaded entirely to memory

Q: Why might you want bigger Spark partitions?
A: Each partition corresponds to a task, and there is an overhead to starting tasks.  Bigger partitions => fewer partitions => less overhead.

Q: Why might you want smaller Spark partitions?
A: - use more cores that are available in the cluster
- balance work more evenly across executors
- use less memory at any given time

Q: What are two categories of Spark transformations?
A: narrow transformation: one input partition corresponds to one output partition
wide transformation: each output partition might need data from multiple input partitions (and vice versa)
if partitions are on different physical machines, wide transformations often cause network I/O (rows need to be shuffled/exchanged across the network)

Q: If df is a Spark DataFrame, how would we start/stop caching it?
A: df.cache() # start
df.unpersist() # stop

Q: What does RDD stand for, in the context of Spark?
A: Resilient Distributed Dataset