Title: 544
Tags: lec14

Q: What does it mean for a format to be "column oriented"?
A: The data in the column is layout out consecutively in memory or within a file.

Q: What is the orientation of CSVs and parquet files?
A: CSVs: row oriented
Parquet: column oriented

Q: What is the tradeoff between text encoded and binary encoded formats?
A: text: easier to debug (because you can just look at it)
binary: usually more efficient (often less space, and numbers usually already in a format on which CPUs can directly operate without further conversion)

Q: Compression works by looking for repetition over some window, say 32 KB.  What would be the advantage and disadvage of increasing the window size?
A: Advantage: find more repetition, and use that to get a smaller file size
Disadvantage: spend more computation time finding the repetition

Q: What does snappy prioritize, high speed, or high compression ratios?
A: high speed
from the docs: "[Snappy] does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression."

Q: Considering CSVs and parquets: when do we need to infer the schema?
A: With CSVs we need to infer it, because everything in a CSV is a string (need to detect that all the strings in a given column look like integers, for example).
Parquets have schema information built in, so there is no need to infer the schema.

Q: What is transactions processing?
A: Operating on individual rows or a small number of rows at a time (e.g., adding, modifying, deleting rows).

Q: What is analytics processing?
A: Trying to answer questions about the data, which involves computation over whole columns of data.

Q: What does OLTP stand for?
A: Online Transactions Processing

Q: What does OLAP stand for?
A: Online Analytics Processing

Q: What is a data warehouse?
A: An OLAP database where we collect data from multiple sources (e.g., OLTP databases) for analysis.

Q: What does ETL stand for?
A: Extract Transform Load
Often used to get data into a data warehouse.  For example, an ETL process might extract data from an OLTP database, transform it in some way, then load it into an OLAP database (the data warehouse).

Q: For analytics work, what are some tradeoffs between storing data in a database vs. storing it in some files (e.g., parquet files)?
A: Database advantage: a database will often have a special storage format co-designed to work with the analytics engine -- the tight integration can lead to great performance.
Files advantage: if you want to go beyond doing SQL queries, having your data in a bunch of files makes it easier to use other tools.  For example, it will probably be faster for some machine learning package to pull data from Parquet files (than do a big query on the database to extract everything).