Title: CS544 / 11-cloud
Tags: cs544-cloud

Q: What does EC2 stand for, and what is it?
A: Elastic Compute Cloud
AWS's VM service (AWS is Amazon Web Services).

Q: What is usually more expensive to rent in the cloud, 1 VM for 100 hours, or 100 VMs for 1 hour?
A: The cost is usually the same.

Q: What is the difference between IaaS and PaaS?
A: IaaS: Infrastructure as a Service.  You rent low-level hardware-like resources (e.g., VMs, virtual disks, private networks, etc).  You use these to deploy systems yourself.  For example, you might launch a bunch of VMs and start running HDFS DataNodes on them.
PaaS: Platform as a Service.  The cloud provider runs a specific system on your behalf.  It could be something they built (e.g., S3), or something open source (e.g., MySQL).  This is usually more convenient, but you often pay more for that convenience.

Q: What does cloud "lock in" mean?
A: It means it is difficult to move to another cloud (or out of the cloud), even if if that's desirable in the long run.  Reasons for the difficulty:
1. need to pay a high network egress cost to get your data out
2. your code might be written to specifically use an API not available in other clouds

Q: What is a "tenant", and what is the difference between multi-tenant hosts and single-tenant hosts?
A: A tenant is a customer paying for cloud services, and might include multiple users (e.g., employees in the same company).
Multi-tenant: different tenants rent VMs on the same physical machine.
Single-tenant: all the VMs on a single physical machine will belong to the same tenant.  Usually more expensive, especially if that tenant doesn't have enough VMs to keep that physical machine fully utilized.

Q: What is a cloud cost you typically pay when VMs are powered off, but not deleted?
A: You'll typically pay for the virtual disk attached to the VM (because the cloud provider needs to expend storage resources to keep your data around so your VM can be booted up again later).

Q: What is a typical geographic hierarchy in clouds?
A: continents (approximate) contain regions, which contain zones.
Zones in the same region will be geographically near each other, but generally have different power and network resources, making correlated failures between zones unlikely.

Q: What is ingress/egress in terms of cloud resources, and what is more expensive?
A: Ingress: incoming network I/O (e.g., data being uploaded to a VM).
Egress: outgoing network I/O (e.g., data being downloaded from a VM).
Ingress is typically free.  Egress is sometimes free (perhaps when egress is to another VM in the same zone), but there is usually a cost per GB of data transferred.  Transferring out of the cloud is usually most expensive.  Within the cloud, the cost depends on whether it is the same region, same continent, etc.

Q: You can either pay a fixed price to get a VM (with high chances of success), or you can pay a fluxuating price (usually cheaper), and potentially get pre-empted (VM is taken away so cloud can use resources for something of higher priority).
What are these two billing models called?
A: "on-demand" instances and "spot" instances, respectively.

Q: The HDFS file system was modeled on a paper about the GFS file system at Google.  What is the newer distributed file system Google built and started using after GFS?
A: Colossus File System

Q: Is BigQuery a data warehouse or a data lake?
A: The distinction is blurred.  The BigQuery engine is integrated with a BigQuery storage engine (as in a warehouse), but BigQuery can also execute on external data sources such as parquet files in HDFS (as in a lake).

Q: The parquet file format was inspired by the ______ format developed at Google, which was eventually replaced internally at Google with the _______ format.
A: ColumnIO, Capacitor

Q: When a value in a column is repeated many times consecutively, we can record just the value and count (instead of many copies of the value).  This is called ________ encoding.
A: run-length

Q: When a column contains large strings, and the same values show up multiple times, we can replace those strings with shorter numeric codes.  We can use the numeric codes to lookup the original strings as needed.  This is called ______ encoding.
A: dictionary

Q: What is something that a GPL open-source license requires that is not required by MIT/BSD/Apache?
A: If you are making changes to the source code, you generally have to release your changes publicly under the GPL too (there are some exceptions/workarounds).

Q: What is a type of intellectual property the Apache open-source license focuses on that the MIT and BSD licenses do not talk much about?
A: They all talk about copyright; Apache additionally talks about patents

Q: What are some factors you might consider when choosing a GCS bucket region?
A: Cost, carbon emissions, proximity to users, etc.

Q: What is the GCP marketplace?
A: Collection of datasets (some free) that can directly be queried with BigQuery (the marketplace also includes software and other things).

Q: How can you activate/deactivate your GCP credentials on a GCP VM?
A: gcloud auth ...

Q: What does this let you do?
%load_ext bigquery_magics
A: It lets you write BiqQuery queries directly in Jupyter cells, like this:
%%bigquery
SELECT * FROM some_table

Q: You run this:
q = bq.query("...")
Why might q.total_bytes_processed be different than q.total_bytes_billed?
A: Billing gets rounded up to the nearest MB, and there is a 10 MB min overall.

Q: What is a cross join?
A: Every row in table A is matched with every row in table B (there is no "ON" expression to limit the matches between tables).

Q: What is a correlated cross join?
A: Some fields in a row might contain multiple values -- for example, it could be an an array of structs.
A correlated cross join does a cross join on a row by row basis.  For example, say we have two rows like this, and we are doing a correlated cross join between the two columns:
[1], [2,3]
[4,5], [6,7]
We would get these outputs:
1,2
1,3
4,6
5,6
4,7
5,7
Notice we don't join across rows (for example, 1 does NOT get paired with 6).

Q: If you want to do a correlated cross join against a field C containing an array of structs, how do you first "expand" the arrays so each entry becomes its own row?
A: UNNEST(C)

Q: What are some examples of external tables we can register in BigQuery?
A: Parquet files in GCS, Google sheets, etc.

Q: If column C is an array type, how do you get the Nth item from every array using BigQuery?
A: SELECT C[OFFSET(N)]
FROM some_table

Q: What can we call if we want to get a pandas DataFrame from a BigQuery query?  What about a geopandas GeoDataFrame?
bq.query(...).?????()
A: to_dataframe
to_geodataframe

Q: Say you have a query like this:
SELECT * FROM myproj.mydataset.tbl1 WHERE x=3
And you want to put the results in a new BigQuery table named myproj.mydataset.tbl2
How do you do this?
A: CREATE OR REPLACE TABLE `myproj.mydataset.tbl2`
AS
SELECT * FROM myproj.mydataset.tbl1 WHERE x=3

Q: What is the difference between ST_CENTROID_AGG(geo_column) and ST_CENTROID(geo_column)?
A: ST_CENTROID_AGG will give a single centroid for the shape representing the union of every shape in geo_column.
ST_CENTROID will give many centroids, one for each shape in a cell of geo_column.

Q: If you have floats long and lat, what can you call in BigQuery to convert this to a geographic point that can be used with other spatial functions?
A: ST_GEOGPOINT(long, lat)

Q: In BigQuery, what are two reasons to do train/test splitting yourself (for ML) instead of letting BigQuery do it?
A: 1. BigQuery has odd defaults depending on dataset size.  For example, for <500 rows, all data is used for training.
2. BigQuery remembers the train/test split for 48 hours, so it will feel deterministic at first, but you'll be surprised later

Q: In BigQuery, what must be passed to ML.WEIGHTS(...)?
A: MODEL your_model_name

Q: In BigQuery, what must be passed to ML.PREDICT(...)?
A: MODEL your_model_name, QUERY
The query must have all the features needed to predict.

Q: In BigQuery, what must be passed to ML.EVALUATE(...)?
A: MODEL your_model_name, QUERY
The query must return all the features needed to predict AND a label column indicating the correct label.

Q: In BigQuery, how do we do non-linear transformations on our features?
A: When creating the model, add a line something like this:
TRANSFORM(POWER(x, 2), ...)
In the parentheses, you can put things you would normally put after select: "SELECT ?????".  This lets you add columns that are transformations of raw columns in the input data.

Q: In BigQuery, how do you handle OneHot encoding?
A: Do nothing: BigQuery will automatically do it for you if your training data has categorical features.

Q: What is a BigQuery slot?
A: A unit of resources, consisting of approximately 1 GB of RAM and 0.5 CPUs

Q: What are two BigQuery billing models?
A: Capacity and On-Demand

Q: Which BigQuery billing model has priority access to resources (i.e., first "right of refusal")?
A: Capacity billing.  This is reserved capacity, so it should be usable whenever needed.
Any leftover resources (if any) are shared between on-demand jobs.

Q: What do on-demand and capacity billing charge for in BigQuery?
A: Both charge for Collosus storage.
On-demand additionally charges for Collosus I/O.
Capacity additionally charges for CPU+memory.

Q: What does it mean that a BigQuery task is preemptible?
A: It can be interrupted if resources are needed for something more important.  For example, an on-demand task might be preempted by a capacity task (capacity is considered more important).