Title: 544
Tags: lec24

Q: Is this deterministic?
df.randomSplit([0.75, 0.25], seed=544)
A: Not completely, because the deterministic split happens at the partition level.  If the number of partitions changes (e.g., because the number of workers changes and Spark decides a different partition count would be ideal), then the overall split won't be deterministic.

Q: Consider Spark's DecisionTreeRegressionModel and DecisionTreeRegressor.  Which one is fitted?
A: DecisionTreeRegressionModel is fitted.
DecisionTreeRegressor is NOT fitted.
Usually, the class names ending in "Model" represent fitted models in Spark.

Q: dt = DecisionTreeRegressor(featuresCol="????", labelCol="????")
We can only pass in one featuresCol -- how do we handle multiple features?
A: Use a vector assembler to transform a DataFrame we want to train/predict on, adding a column of vectors that contains data from multiple columns:
va = VectorAssembler(inputCols=["col1", "col2", ...], outputCol="features")

Q: If we have a fitted model in Spark, how do we used it to make predictions?
A: df_WITH_predictions = model.transform(df_WITHOUT_predictions)
.transform(...) in Spark is used both for preprocessing AND prediction.

Q: A Spark "Pipeline" object is an unfit model, consisting of multiple transformers and ending in an estimator.  When we call "fit" on it, we get a fitted PipelineModel.  What does it contain?
A: All the transformers of Pipeline are copied to PipelineModel.
"fit" is called on the estimator at the end of Pipeline, and this "fit" returns a transformer.  The transformer is copied to the end of PipelineModel, such that PipelineModel is a series of transformers.

Q: The traditional decision tree training algorithm considers splits on every column, for every possible split between rows (a threshold on a column is a split).  Where does Spark consider splits?
A: It still looks at every column, but we configure a number of bins on each column, for example: dt.setMaxBins(N).  A histogram is computed on this with N buckets, and the thresholds considered are those that split the buckets (so N-1 possible split points per column).

Q: What is an "equi-depth" histogram?
A: Each bucket contains a similar number of data points, even if that means some buckets cover a wider range of values than others (e.g., 1-3, 3-10, 10-100, etc).
In contrast, most histograms have buckets with fixed intervals (e.g., 0-10, 10-20, 20-30, etc).