Title: 544
Tags: lec35

Q: What does "stateless" mean in the context of Spark streaming?
A: Each microbatch of rows can be processed without referring back to results computed from previous microbatches.

Q: What are some examples of sources/sinks in Spark streaming?
A: Kafka, Parquet files in HDFS, Cassandra, console, etc.

Q: What are the three output modes available for Spark streaming queries?
A: append
complete
update

Q: Most systems use replication to implement fault tolerance.  Why doesn't Spark streaming need to do that?
A: Spark streaming is based on RDDs, which describe how to compute the data on demand, as needed.  So as long as data can be re-read from the source, we can re-materialize any output previously computed for an RDD.

Q: Why is spark.sql.shuffle.partitions an important configuration for Spark streaming, but not for Spark batch jobs?
A: In recent versions of Spark, a "partition coalescing" optimization was added that combines small partitions into bigger partitions.  But it is only supported for batch jobs.  With streams, we don't know the future distribution of data across partitions, so we cannot readily decide which partitions to combine to get large, balanced partitions.

Q: What do watermarks do?
A: They specify an upper bound on how "late" data could conceivably arrive.  If we have a streaming GROUP BY that groups on time intervals, this means we can eventually discard our running count for a given time window because we can assume no more data will arrive that could change that count.
This prevents memory consumption from growing indefinitely.

Q: Spark streaming supports most of the operations supported by batch jobs, but a "pivot" operation is an example of one that only works for batch (not streaming).  Why doesn't it work?
A: If you allowed pivot for streaming, there are some new inputs that could appear that would cause an additional column to be added to the output.  But Spark DataFrames (whether streaming or not) need to have a well-defined set of columns so that SQL queries and other operations can operate on them.

Q: Say you have "A ???? JOIN B", where A is static and B is a stream.  What kind of JOINs are supported?
A: INNER and RIGHT are supported.
LEFT is NOT supported.  The reason is that a left join requires us to output a row of A with NULL columns from B if there is no B row matching.  But we can never conclude there is no B row matching because B keeps growing, and we don't know the future.

Q: When do Spark streaming jobs provide exactly-once semantics?
A: You need 3 conditions:
1. deterministic code
2. re-readable source
3. idempotent sink (this means that it can suppress duplicates)