Title: 544
Tags: lec2 lec1 lec13 lec4 lec5 lec30 lec11 lec35 lec6 lec7 lec32 lec15 lec19 lec24 lec26 lec28 lec29 lec17 lec23 lec8 lec9 lec16 lec34 lec14 lec27 lec31 lec36 lec41 lec10 lec25

Q: What do systems primarily manage?
A: Resources: compute, memory, storage, and network

Q: what does a compiler do?
A: translate high level code (something a programmer would write) to low level code, such as machine code that a CPU can directly execute

Q: what does an interpreter do?
A: loop over high-level code and execute corresponding operations on the CPU, as the program executes

Q: What is a language (other than Java) that compiles to bytecode for the JVM?
A: Scala is the best answer (important for us, because we'll learn Spark, which was written in Scala)
Python could technically be an answer too due to the Jython project, but this probably isn't used much in practice.

Q: Consider high level code (like Python), machine code, and byte code -- what can run directly on a CPU?
A: machine code

Q: What does JVM stand for?
A: Java Virtual Machine

Q: What is the common extension for files containing Java bytecode?
A: .class

Q: What is the common extension for files containing Python bytecode?
A: .pyc

Q: What does this Python code do?
import dis
dis.dis("????")
A: Shows you the Python bytecode corresponding to the Python code in the string.

Q: If you use more machines, what kind of scaling is it?
A: scaling out

Q: If you use a more powerful machine, what kind of scaling is it?
A: scale up

Q: What does CPU stand for?
A: Central Processing Unit

Q: What does GPU stand for?
A: Graphics Processing Unit

Q: What does RAM stand for?
A: Random Access Memory

Q: What does HDD stand for?
A: Hard Disk Drive

Q: What does SSD stand for?
A: Solid State Disk

Q: What does NIC stand for?
A: Network interface card/controller

Q: What does it mean for a device to provide random access?
A: You can jump around and read from different locations efficiently.

Q: What values can a bit hold?
A: 0 or 1

Q: How many different values can a byte hold?
A: A byte has 8 bits, so there are 2^8 different possible combinations of bits.  So 256 different values.

Q: How much is 1 KB?
A: 1024 (or 1000) bytes

Q: How much is 1 MB?
A: 1024 KB (or 1000 KB)

Q: How much is 1 GB?
A: 1024 (or 1000) MB

Q: How much is 1 TB?
A: 1024 (or 1000) GB

Q: What does it mean for a device to be "byte addressible"?
A: Each byte has it's own address, and the CPU can use that address to get the value.
Extracting a single bit from a byte actually involves more steps than using the whole byte.

Q: What is generally faster, storage or memory?
A: Memory

Q: What does volatile mean, and what is an example of hardware that is volatile?
A: Volatile means you lose the contents upon reboot.  Memory (e.g., RAM) is volatile, storage (e.g., SSD) is non-volatile.

Q: What do we mean with we call HDDs and SSDs "block devices"?
A: Data is read/written in blocks of many bytes (for example, 0.5 KB).
Accessing 1 byte in the block takes the same time as accessing the whole block.

Q: A device can store 1000 GB, can transfer 100 MB/s, and serve one request in 10 ms.  What kinds of metrics are these?
A: 1000 GB: capacity
100 MB/s: throughput
10 ms: latency

Q: If throughput can be expressed as X MB/s or Y Mbps, what is the relationship between X and Y?
A: Y = X * 8
(because the "b" in Mbps stands for bits, and there are 8 bits in a byte)

Q: often a bunch of server machines are stacked on top of each other in a metal box called a ______ and connected with a network switch
A: rack

Q: a large facility containing many racks of computers is called a ______
A: data center

Q: a cluster is a ______
A: bunch of computers working together

Q: what is a program you can use to run shell programs on another machine?
A: ssh

Q: what is the technical term for "folder"?
A: "directory" -- "d" shows up in many acronyms/commands for this reason (pwd, cd, etc)

Q: What Linux command tells you see what directory you're in?
A: pwd

Q: What Linux command shows you all the contents of a directory?
A: ls

Q: How do you create a new empty file, file.txt?
A: touch file.txt

Q: What Linux command do you use to install software on Ubuntu?
A: apt

Q: What is a Linux command you can use to download from a URL to a file on your computer?
A: wget SOME_URL
(curl is a similar tool)

Q: What Linux command lets you rename a file?
A: mv name1.txt name2.txt
It also lets you move a file, like this:
mv name.txt somedir

Q: How can you copy from file A to file B?
A: cp A B

Q: What Linux command lets you copy files between computers?
A: scp

Q: What Linux command lets you see everything in a file?
A: cat

Q: What Linux commands let you see just the start or end of a file?
A: head file.txt
tail file.txt

Q: What Linux command creates a new directory?
A: mkdir somedir

Q: What command lets you see entries in the Linux manual?
A: man SOMECOMMAND

Q: How do you navitage to a new directory?
A: cd SOMEDIR

Q: How do you run commands as the root user?
A: sudo COMMAND

Q: What Linux command lets you change the access permissions on a file?
A: chmod

Q: Given a program name, how can you find where it is on your computer?
A: which PROGRAMNAME

Q: How can you see an environment variable (for example, PATH)?
A: echo $PATH

Q: How can you pipe the standard output of A to the standard input of B?
A: A | B

Q: How do you redirect the standard output of A to file.txt, creating file.txt (or replacing any prior content)?
A: A > file.txt

Q: How do you redirect the standard output of A to file.txt, creating file.txt (or appending to the end of any prior content)?
A: A >> file.txt

Q: How do you send BOTH standard output and standard error to a file?
A: myprogram &> file.txt

Q: What programs counts lines, words, and characters?
A: wc

Q: What program filters standard input, writing matching lines to standard output?
A: grep

Q: What program recursively finds directories and files?
A: find

Q: How can you run a program asyncronously?  (meaning you can type other things on the prompt while waiting for it to finish)
A: longprogram &

Q: How can you see running programs and their PIDs?  (process IDs)
A: ps
To see more:
ps ax
a=other users
x=not just processes attached to a terminal

Q: How do you kill a process by its PID number?
A: kill PID

Q: How do you kill a process by its name?
A: pkill NAME

Q: How can you see CPU and memory usage on your machine?
A: htop

Q: What Linux command shows you disk utilization?
A: df
("df -h" shows human readable)

Q: What command shows you disk usage by everything in the current directory?
A: du -hs .

Q: How can you see most port numbers being used by processes on your machine?
A: sudo lsof -i tcp -P

Q: If you want to download an image X from DockerHub to your VM, what command do you use?
A: docker pull X

Q: How can you see all the Docker images on your VM?
A: docker images

Q: How can you give a Docker image "a" an additional second name, "b"?
A: docker tag a b

Q: How can you launch a container from image X?
A: docker run X

Q: How can you see existing containers?
A: docker ps

Q: How can you delete a container X?
A: docker rm X

Q: How can you delete an image X?
A: docker rmi X

Q: How can you see Docker's disk utilization?
A: docker system df

Q: How can you cleanup unnecessary Docker data to free up space?
A: docker system prune

Q: How can you see the output (for example prints) for an application running inside container X?
A: docker logs X

Q: How do you run a command (lets say "ls") inside a running container X?
A: docker exec X ls

Q: How do you see resource consumption of running containers?
A: docker stats

Q: How do you get a bash shell inside an existing container X?
A: docker exec -it X bash

Q: How do you cause a container X to stop running?
A: docker kill X

Q: How do you run the Dockerfile in the current directory to build an image X?
A: docker build . -t X

Q: What is the difference between "docker rm" and "docker kill"
A: After "docker kill", the container won't be running, but it will still exist (you can see it with "docker ps -a").  "docker rm" actually cleans up the space that was used by that container.

Q: In a Dockerfile, how do you specify a base image X?
A: FROM X

Q: In a Dockerfile, how do you execute a command, say "apt update"?
A: RUN apt update

Q: In a Dockerfile, how do you copy a file X.txt (in the same directory on the VM) to /Y.txt (in the image)?
A: COPY X.txt /Y.txt

Q: Why are the three main numeric types in Python?
A: integers, floating point numbers, and complex numbers

Q: Of integers and floats, what support is there for Infinity?
A: Floats support this, ints do not

Q: If you have a type like intN or floatN, how many bytes does it take?
A: N/8
For example, int32 takes 4 bytes

Q: What is the unsigned equivalent of int16?
A: uint16

Q: When we choose between different numeric types, what are three things we're trading off?
A: precision (specifying small differences)
range (difference between smallest and biggest possible numbers)
space (how many bytes does it take?)

Q: If you invent a new bit layout to represent your numbers, what will you need to worry about with respect to performance?
A: Existing CPUs won't have built-in operations for your type.  So you'll need to implement operations like multipy in software (which is slower than hardware), or convert to another type the CPU can directly work with on an as-needed basis (also slow)

Q: What happens when you add negative infinity and positive infinity?
A: You get NaN (not a number)

Q: What does sys.float_info.dig tell you?
A: The maximum number of decimal digits that can be faithfully represented in a float

Q: What does sys.float_info.epsilon tell you?
A: difference between 1.0 and the next number after 1.0 that we can represent with a Python float

Q: If you want torch.rand(...) to give you the same pseudorandom numbers each time, what must you do?
A: "seed" the randomizer
torch.manual_seed(SOME_NUMBER)

Q: If you want to move a CPU tensor A to a GPU tensor B, what do you do?
A: B = A.to("cuda")

Q: torch.sigmoid(x) will return numbers between _____ and _____
A: 0, 1

Q: How do we matrix multiply A and B in PyTorch?
A: A @ B

Q: If you want to create a callable type, what special method do you need in your class?
A: __call__

Q: x = torch.rand(a, b)
y = torch.rand(c, d)
When will x @ y work?
A: When b = c

Q: x = torch.rand(a, b)
y = torch.rand(c, d)
If x @ y works, what will be the resulting shape?
A: (a, d)

Q: How many columns will it have?
A = torch.rand(x, y)
A: y

Q: model = torch.nn.Linear(x, y)
preditions = model(data)
How many columns must data have?
A: x

Q: PyTorch: if y is a function of x, and you're trying to find the best x to optimize y, what do you need to do to let PyTorch know you'll later need a gradient?
A: x = torch.tensor(????, requires_grad=True)

Q: PyTorch: if y is a function of x, and you turned on gradient tracking for x, how can you get the y-over-x gradient (after you've calculated y)?
A: y.backward()
x.grad

Q: PyTorch: if you call .backward() on more than one result without zeroing gradients, what happens, say to x.grad?
A: x.grad will acculumate gradients: it will be the some of gradients with respect to all results on which you called .backward().

Q: PyTorch: say you've calculated y as a function of x, and you're trying to find a better x value.  You're using an optimizer opt for this purpose, and you've already called backward() to get gradients.
How do you make a small change to x?
A: opt.step()

Q: What does SGD stand for?
A: Stochastic Gradient Descent

Q: PyTorch: by default, the SGD optimizer is trying to get small results.  What should you pass to the SGD constructor if you want to get big results?
A: maximize=True

Q: PyTorch: what does the "lr" parameter for SGD stand for?
A: learning rate

Q: PyTorch: when you call optimizer.step(), how much will an input tensor x change?
A: x.grad * (learning rate of optimizer)
maximization: increase by that amount
minimization: decrease by that amount

Q: PyTorch: if you're optimizing y as a function of x and using an optimizer opt, how do you reset x.grad to zero?
A: opt.zero_grad()

Q: What are the big three areas of machine learning?
A: Reinforcement learning: agent makes series of actions to maximize reward
Unsupervised learning: looking for general patterns
Supervised learning: train models to predict unknowns

Q: What is the difference between regression and classification
A: regression: predict a number
classification: predict a category

Q: PyTorch: when training on a larger dataset, it is often efficient to use the whole dataset for each step of the optimizer.  What do we do?
A: We break the dataset into batches.  Within the loop over epochs, we have another loop over batches of the dataset.

Q: What does MSE stand for?
A: Mean Squared Error

Q: What PyTorch type lets us slice over our data, getting batches of features (x values) and corresponding batches of labels (y values).
A: Dataset.  An in-memory one is TensorDataset

Q: What PyTorch type provides additional features to a DataSet, like shuffling, batching, and iteration?
A: DataLoader

Q: There is one virtual address space per _______
A: process

Q: A process is a running ______
A: program

Q: Virtual memory maps to ______
A: physical memory

Q: Which address space can contain "holes" (invalid addresses)
A: Virtual address spaces can have holes (physical address spaces do not -- they are contiguous)

Q: What are three things that go in a virtual address space?
A: code
stack
heap

Q: Which pointer indicates what code to execute next?
A: The "instruction pointer"

Q: There is one stack per _____ and one heap per ______
A: thread, process

Q: What is it called when the scheduler stops running one thread and starts running a different one?
A: A "context switch"

Q: What are three states a thread can be in?
A: running, blocked, ready

Q: What are some reasons a thread might be blocked?
A: waiting on user input, disk operations, network operations, timer sleep, etc.

Q: How do you create and launch a thread object for running function f(x)?
A: t = threading.Thread()
t.start(target=f, args=[x])

Q: How do you wait for a thread t to finish running?
A: t.join()

Q: How can you print a thread's ID?
A: t.get_native_id()

Q: What is a "race condition"
A: A bug that only sometimes surfaces, depending on when context switches happened between threads

Q: When might a context switch occur?
A: Between instructions -- a single line of code might correspond to multiple bytecodes or machine instructions, so it's important to note that context switches often occur in the middle of a line of code being executed.

Q: What is a "race condition"?
A: A bug, where whether or not you get correct behavior depends on how threads get scheduled.
Race conditions are tricky to test+debug because they're non-deterministic (not same behavior each time), and you might only see the incorrect behavior rarely (say one in a million times).

Q: What is "deadlock"?
A: This is when threads can never make progress, because they each want to acquire a lock that will never be released.
One example: t1 has A and t2 and B; t1 is waiting to additionally acquire B, and t2 is waiting to additionally acquire A.  This will never happen because neither thread will release its first lock until it has acquired the second lock and done some work.

Q: How can a thread (a) get a lock and (b) let go of it?
A: lock.acquire()
lock.release()

Q: How can you rewrite the following code so that the release will happen, even if an exception occurs?
lock.acquire()
# code with possible exception
lock.release()
A: with lock:
    # code with possible exception

Q: What do you call a section of code where certain interleavings with other code would be a problem?
A: critical section

Q: What is "atomicity" in the context of multi-threaded programs?
A: If a thread is changing multiple things together, we never want to see (say from another thread) partial results, with only some changes made.

Q: What is an "invariant"?
A: A rule that applies to the data, often something a human might know from context.  For example, an invariant might be a list of ages cannot contain negative numbers.  Some invariants are known by the system (for example, that the length of the list itself cannot be negative).

Q: What is "consistency" in the context of threading?
A: All invariants are enforced.

Q: If thread t1 holds a lock, what could happen?
1. we could context switch to another thread t2
2. another thread t2 could acquire the lock
A: (1) is possible, (2) is not

Q: What happens in Python when you try to release a lock that has already been released?
A: There is an exception

Q: What is the GIL?
A: Global Interpreter Lock

Q: Before a CPU can add two values in RAM, what must it do?
A: Load those values to CPU registers

Q: Which CPU cache is fastest?
A: L1

Q: Which CPU cache is biggest?
A: L3

Q: When data in a cache is too old and no longer valid, we say it is _______
A: stale

Q: What does FIFO stand for?
A: First In First Out

Q: What does LRU stand for?
A: Least Recently Used

Q: How are FIFO and LRU similar?  How are they different?
A: They are the same for a miss: the data is added to the cash, in the "newest" position.
They are different for a hit: FIFO does not change how it prioritizes the item, but LRU resets it to the "newest" position.

Q: What do you call it when you find an item in a cache?
A: "hit"

Q: What do you call it when you CANNOT find an item in a cache?
A: "miss"

Q: What is a pattern where a random eviction policy would beat LRU and FIFO?
A: A common pattern is that the data is too big to fit completely in cache, and the program loops, repeatedly reading over the dataset in the same order.  LRU and FIFO will have a 0% hit rate, but random will at least get some hits.

Q: What is the name for a piece of data loaded to CPU cache?
A: cache line

Q: What is the most common cache line size for modern CPUs?
A: 64 bytes is common (other sizes like 128 used to be more common; new Macs haved started using 128 again)

Q: If you want to sum over column in a PyTorch matrix in a cache friendly way, how should you create the tensor?
A: First create a tensor where each column is actually stored as a row in the matrix.  Then transpose it.  This will get you columns that are contiguous in memory, resulting in fewer cache lines being loaded as the data is scanned.

Q: How does PyArrow indicate which strings/ints/etc are null/None in an array?
A: validity bitmap: 1 means valid, 0 means null/None

Q: How can you add a region to your virtual address space that is not backed by a file?
A: anonymous mmap

Q: How can you map a file into your virtual address space?
A: Open the file, get the f.fileno(), and pass it to mmap.

Q: If you want to be able to evict pages in anonymously mmap'd regions, what do you need to do?
A: enable a swap file in Linux

Q: How many bits are in a MAC address?
A: 48

Q: What does the prefix of a MAC address indicate?
A: The vendor/manufacter that produced the NIC
This is so that we can achieve globally unique MAC addresses (each vendor gets a prefix, and is expected not to reuse the same suffix for different devices).

Q: How can you see the MAC addresses and IP addresses for your NICs?
A: ifconfig
("if" is short for interface)

Q: What is a virtual NIC that is only reachable from the local machine, and what IP address does it use?
A: The "loopback" device, which has address 127.0.0.1

Q: What is a network?
A: A collection of computers that can send each other messages by MAC address

Q: Within a network, what device can forward data that is sent to a specific MAC address?
A: A switch

Q: What is the device that connects multiple networks to form an internet, or the Internet?
A: a router

Q: How many bits does an IPv4 address contain, and how many possible addresses are there?
A: 32 bits, and about 4 billion possible addresses

Q: What does a prefix of an IP address tell you?
A: It identifies the network

Q: Why do routers forward based on IP address rather than MAC address?
A: Because there are too many MAC addresses in the world, and forwarding tables would need to be too big if every router needed to remember where every MAC address lives.
With IP addresses, IPs with the same prefix are on the same network.  A router just needs to know where to forward data to get it closer to the correct network.

Q: The following are private IP ranges, meaning multiple networks can use the same IPs from these ranges internally:
192.168.0.0 to 192.168.255.255
172.16.0.0 to 172.31.255.255
10.0.0.0 to 10.255.255.255
What device/software can let us connect to machines using these IPs, given they are not unique?
A: A NAT (network address translation) box.
It can map a unique public to one of these private IPs within a network.

Q: What is something that both UDP and TCP add in addition to what IP provides?
A: The both provide port numbers, which let us direct communication to a specific process on a machine

Q: What is something that TCP provides that UDP does not?
A: TCP provides more reliable communication:
retry when a packet is dropped
buffer packets on receiving side and re-arrange so they arrive in the same order they were sent

Q: What does UDP stand for?
A: User Datagram Protocol

Q: What does TCP stand for?
A: Transmission Control Protocol

Q: Why would you ever prefer UDP over TCP?
A: TCP does extra work, which is sometimes not necessary.  For example, in a Zoom call, if some packets are dropped the session may freeze.  After that, it's more natural for humans to repeat what they were saying (it would be weird if it replayed video/audio from the few seconds when the connection was lost).

Q: What is the relationship between domain names in a URL and IP addresses?
Example:
https://tyler.caraza-harter.com:443/cs544/s23/schedule.html
A: When you enter a domain name into a browser, it gets translated to an IP address automatically for you.

Q: If you are using HTTP over TCP/IP, what functionality does TCP/IP provide, and what does HTTP provide?
A: TCP/IP gives you IP addresses and port numbers, letting you communicate with a specific process (a server on a specific machine.
HTTP additionally lets you specify what resource you want to access via that server.
For example, consider this URL:
https://tyler.caraza-harter.com:443/cs544/s23/schedule.html
The resource is "/cs544/s23/schedule.html"

Q: What does RPC stand for?
A: Remote Procedure Call
Procedure: another name for a function/method
Remote: we want it to run on a different process/container/computer

Q: What are the RPC implementations created by Meta and Google called, respectively?
A: Thrift and gRPC

Q: What is gRPC layered on top of?
A: HTTP
Every method call is an HTTP POST request.  The arguments go in the request body, the return value goes in the response body.

Q: What is it called when we convert various data types into bytes to send over the wire?  What about the reverse conversion?
A: serialization/deserialization

Q: What does gRPC use for serialization?
A: Protocol Buffers (protobufs for short)

Q: What is variable-length encoding, and what are the pros/cons of using it?
A: Use more space for bigger numbers.  E.g., if an int32 value is small enough, we could use 1, 2, or 3 bytes (instead of the usual 4).
Pros: save space.  Very important for sending data over the network, so protobufs do this.
Cons: CPUs cannot directly operate on int32's unless they are 4 bytes, so conversion is necessary before sending and after receiving this data.

Q: Port forwarding: if we want to forward port 300 of the loopback device on a VM to port 80 inside a container, how do we run the container?
A: docker run -p 127.0.0.1:300:80 myimg

Q: What does this do?
ssh USER@VM -L localhost:1234:localhost:5678
A: This is executed on a machine (say your laptop) to create an SSH tunnel to the VM.  While the tunnel is in use, any requests to localhost:1234 on the laptop will be forwarded to localhost:5678 on the VM ("localhost" is not redundant here because it is referring to two different loopback devices, the one on the laptop and the one on the VM).

Q: Consider this:
docker run -d -p 300:80 myimg
We specify the port on the VM (300) and container (80), but we don't specify which NIC.  What will be used?
A: By default, the command is equivalent to this:
docker run -d -p 0.0.0.0:300:80 myimg
This means traffic on all of the VM's NICs will forward to the container (possibly a security risk, depending on your firewall settings).
Inside the container, it will forward to whatever the main NIC is (e.g., eth0), NOT the loopback device (so anything inside the container should listen on 0.0.0.0, not localhost).

Q: What is container orchestration, and what is currently a very popular orchestrator?
A: Orchestration lets you deploy cooperating containers across many different machines.
Kubernetes is probably the most well known.

Q: What does "docker compose" do?
A: Lets you start a bunch of cooperating containers on the same machine.

Q: protobuf files (.proto) can support two different syntaxes, v2 and v3.  If you want v3, what should the first line of your .proto be?
A: syntax = "proto3";

Q: What are the two main things a .proto file contains?
A: messages
services
Example:
message MyMessage {
  ...
}
service MyService {
  ...
}

Q: In a .proto file, what does a message contain?
A: Fields, each with a type and a number, like this:
message MyMessage {
  int32 x = 1;
  int32 y = 2;
}

Q: In a .proto file, what does a service contain?
A: RPCs, like this:
service MyService {
  rpc MyRPC(Message1) returns (Message2);
}
Each RPC has one message containing arguments and another containing return values.

Q: When you generate code for a .proto file, you get two files:
????_pb2
????_pb2_grpc
What are they each for?
A: ????_pb2: code for serializing/deserializing the messages
????_pb2_grpc: servicer (class to be inherited from) and stubs for making remote procedure calls

Q: Consider this:
python3 -u server.py &> log.txt
What does "-u" do?
A: "force the stdout and stderr streams to be unbuffered"
Otherwise, if its buffered, any prints from server.py might not immediately appear in log.txt (that can be confusing if you're using log.txt to debug).

Q: If you have a docker-compose.yml in the current directory, and you want to start running the services defined, what can you do?
A: docker compose up

Q: If you want to see the compose containers currently running, what do you do?
A: docker compose ps

Q: If you want to stop running containers for a compose cluster, what can you do?
A: docker compose down

Q: If you want to delete all containers for a docker compose cluster, what can you do?
A: docker compose rm
(add -f to force it; for example, might be necessary if they're currently running)

Q: Memory is byte-addressible.  But what is the unit of access for an HDD, SSD, etc?
A: These are block devices, so the unit is a block.  A typical block size might be about 512 bytes.

Q: What are two ways we can use memory to make access to block devices more efficient?
A: caching: keep some disk data in memory, so we don't need to go back to disk if the data is reused
buffering: when doing many smaller reads/writes, accumulate more data in memory before doing the transfer between disk and memory (because block devices are more efficient for bigger accesses).

Q: What are the names for these two data access patterns?
1. reading lots of data in order
2. jumping around and reading from many different places
A: 1. sequential
2. random

Q: "block" and "page" have different meanings for SSD internals than we've seen for operating systems and interactions with block devices.  What are the differences?
A: block (for SSDs): unit of erasure (perhaps 100s of KBs)
block (for using a block device): unit of transfer to/from device, perhaps 512 bytes
page (for SSDs): unit of read/write within the define (few KBs)
page (for system memory): unit of mapping between virtual memory and physical memory (usually 4 KB)

Q: How can we break a block device into smaller sections?
A: with partitions

Q: How can we treat multiple individual block devices like one bigger, more reliabel one?
A: With RAID (redundant array of inexpensive disks)

Q: What is the data structure in a file system that points to blocks containing file data?
A: inode

Q: On UNIX, how do we make many smaller file systems appear as one big one?
A: By mounting them on top of each other

Q: What does it mean for a format to be "column oriented"?
A: The data in the column is layout out consecutively in memory or within a file.

Q: What is the orientation of CSVs and parquet files?
A: CSVs: row oriented
Parquet: column oriented

Q: What is the tradeoff between text encoded and binary encoded formats?
A: text: easier to debug (because you can just look at it)
binary: usually more efficient (often less space, and numbers usually already in a format on which CPUs can directly operate without further conversion)

Q: Compression works by looking for repetition over some window, say 32 KB.  Would would the the advantage and disadvage of increasing the window size?
A: Advantage: find more repetition, and use that to get a smaller file size
Disadvantage: spend more computation time finding the repetition

Q: What does snappy prioritize, high speed, or high compression ratios?
A: high speed
from the docs: "[Snappy] does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression."

Q: Considering CSVs and parquets: when do we need to infer the schema?
A: With CSVs we need to infer it, because everything in a CSV is a string (need to detect that all the strings in a given column look like integers, for example).
Parquets have schema information built in, so there is no need to infer the schema.

Q: What is transactions processing?
A: Operating on individual rows or a small number of rows at a time (e.g., adding, modifying, deleting rows).

Q: What is analytics processing?
A: Trying to answer questions about the data, which involves computation over whole columns of data.

Q: What does OLTP stand for?
A: Online Transactions Processing

Q: What does OLAP stand for?
A: Online Analytics Processing

Q: What is a data warehouse?
A: An OLAP database where we collect data from multiple sources (e.g., OLTP databases) for analysis.

Q: What does ETL stand for?
A: Extract Transform Load
Often used to get data into a data warehouse.  For example, an ETL process might extract data from an OLTP database, transform it in some way, then load it into an OLAP database (the data warehouse).

Q: For analytics work, what are some tradeoffs between storing data in a database vs. storing it in some files (e.g., parquet files)?
A: Database advantage: a database will often have a special storage format co-designed to work with the analytics engine -- the tight integration can lead to great performance.
Files advantage: if you want to go beyond doing SQL queries, having your data in a bunch of files makes it easier to use other tools.  For example, it will probably be faster for some machine learning package to pull data from Parquet files (than do a big query on the database to extract everything).

Q: What SQL clause lets you combine two tables?
A: JOIN

Q: What are two SQL clauses that let you filter down what rows are included based on some condition, and how do these two clauses differ?
A: WHERE and HAVING
WHERE is prior GROUP BY (if GROUP BY is used), so filters input rows
HAVING is after GROUP BY, so filters groups

Q: What SQL clause pulls together related rows?
A: GROUP BY

Q: In SQL, SUM, AVG, and other operations that work over all rows (or groups of rows) are called ______
A: aggregates

Q: What SQL clause sorts rows?
A: ORDER BY

Q: What SQL clause lets you limit results to the first N rows?
A: LIMIT

Q: What does ACID stand for?
A: - atomiticy: it all happens or nothing happens (partial progress is rolled back)
- consistency: application invariants (like no negative bank accounts) are supported
- isolation: others cannot see a transaction in progress (aka atomicity when talking about locks)
- durability: once finished, it persists (even if machine crashes+restarts)

Q: If conn is a SQLAlchemy connection, and you've made some changes as part of a transaction, how do you finalize that transaction?
A: conn.commit()

Q: If conn is a SQLAlchemy connection, and you've made some changes as part of a transaction, how do you abandon that transaction?
A: conn.rollback()

Q: What does GFS stand for?
A: Google File System

Q: What is the Hadoop system similar to GFS?
A: HDFS (Hadoop File System)

Q: What is a Hadoop system similar to Google's BigTable?
A: HBase

Q: What is a distributed analytics system that many people have started using instead of MapReduce?
A: Spark

Q: When Google MapReduce and BigTable were first created, where did they store their data?
A: GFS

Q: What is the technical word for having multiple copies of our data on different systems?
A: replication

Q: What is the technical word for breaking our data (for example, a file) into smaller pieces that are more manageable?
A: partitioning

Q: HDFS has a boss/worker architecture.  What are the specific names given to the boss and workers in an HDFS cluster?
A: boss: NameNode
worker: DataNode

Q: When a client wants to read an HDFS file, how does it interact with the NameNode and DataNodes?
A: First it tells the NameNode what it wants to read, and the NameNode responds with information about where the client can find the data.
Then, the client fetches the data directly from the DataNodes.
Data does not flow through the NameNode because otherwise having a single NameNode would be a major bottleneck.

Q: What are pipelined writes?
A: If a client wants to write 3 replicas of a block to DataNodes DN1, DN2, and DN3, it does not send the data directly to each of them.
Instead, it sends the data to DN1, which forwards to DN2, which forwards to DN3.
This prevents any one computer from having too much work (most importantly, the client isn't doing all the work, because the client may be a weaker machine than the servers).

Q: If you write 1 MB to a triply replicated file, how much disk I/O does that generate?  What if you read 1 MB from a triply replicated file?
A: Write: 3 MB (need a copy on each machine, for safety)
Read: 1 MB (the three machines have identical data, so there is no point in requesting it from more than one)

Q: How do you format an HDFS NameNode?
A: hdfs namenode -format

Q: How do you start running an HDFS NameNode with hostname "main" and port 9000?
A: hdfs namenode --fs hdfs://main:9000

Q: How can you get an HDFS cluster report for hdfs://main:9000/?
A: hdfs dfsadmin -fs hdfs://main:9000/ -report

Q: How can you start a DataNode and connect it to a NameNode with hostname "main" on port 9000?
A: hdfs datanode -fs hdfs://main:9000

Q: If you want to use commands like mkdir, cp, ls, cat, and du on HDFS, how do you do it?
A: hdfs dfs -COMMAND ARGS
For example, to run "ls" on the "data" directory:
hdfs dfs -ls hdfs://main:9000/data/

Q: What is the difference between a data warehouse and data lake?
A: Both are a place where we collect data for analysis (OLAP workloads).
A warehouse is a database with integrated storage+computation.
A lakehouse is a combination of distributed storage (for example, HDFS) and decoupled analytics engine (for example, Spark)

Q: In MapReduce, what parameters do map and reduce functions take?
A: def map(key, value)
def reduce(key, values)
Note that will reduce, all values with the same key will go to a single reduce call on a single machine.  During map, different values with the same key go to different map calls, perhaps even on different machines.

Q: A reduce task might have the reduce function called several times.  What can we say about the keys across these multiple calls for a single task?
A: The keys will be sorted

Q: What determines the number of output files in a MapReduce job?
A: The number of reducers, because each reducer produces one output file.

Q: What are some performance advantages Spark has over MapReduce?
A: intermediate data is handled better (could directly pass from one computation to the next, without materializing in HDFS)
a chain of RDDs lets us specify many steps to get to a desired end result, but lazy execution means that an optimizer could find more efficient ways to get to that same result

Q: If you have a regular Python list, how can you convert it into a Spark RDD?
A: rdd = sc.parallelize(YOUR LIST)

Q: What is the relationship between actions, operations, and transformations?
A: actions and transformations are both kinds of operations
transformations are lazy (no work done yet), and describe how to produce an output RDD given an input RDD
actions actually materialize some output (on the screen, in an HDFS file, etc), triggering whatever work is necessary to compute that output.  An action is often at the end of a chain of transformations.

Q: A Spark task:
  - runs on a _________
  - operates on a _________
A: runs on a single CPU core
operates on a single partition, which is loaded entirely to memory

Q: Why might you want bigger Spark partitions?
A: Each partition corresponds to a task, and there is an overhead to starting tasks.  Bigger partitions => fewer partitions => less overhead.

Q: Why might you want smaller Spark partitions?
A: - use more cores that are available in the cluster
- balance work more evenly across executors
- use less memory at any given time

Q: What are two categories of Spark transformations?
A: narrow transformation: one input partition corresponds to one output partition
wide transformation: each output partition might need data from multiple input partitions (and vice versa)
if partitions are on different physical machines, wide transformations often cause network I/O (rows need to be shuffled/exchanged across the network)

Q: If df is a Spark DataFrame, how would we start/stop caching it?
A: df.cache() # start
df.unpersist() # stop

Q: What does RDD stand for, in the context of Spark?
A: Resilient Distributed Dataset

Q: How do you create a SparkData corresponding to a CSV at location PATH?
A: df = spark.read.format("csv").load(PATH)

Q: How can you see the number of RDD partitions for a Spark DataFrame df?
A: df.rdd.getNumPartitions()

Q: Say you run this:
spark.read.format("csv").load("test.csv")
And test.csv looks like this:
a,b
1,2
3,4
What will be the first row of data?
A: a,b
Because headers are off by default.  Add this:
spark.read.format("csv").option("header", True).load("test.csv")

Q: Say you run this:
spark.read.format("csv").option("header", True).load("test.csv")
And test.csv looks like this:
a,b
1,2
3,4
What will be the data types for the columns?
A: strings, because everything in a CSV is a string.
To get ints, use this:
spark.read.format("csv").option("header", True).option("inferSchema", True).load("test.csv")

Q: If you want to select the "x" column of a Spark DataFrame df, add 1 to it, and call the result column y, how would you do that?
A: df.select(expr("x+1").alias("y"))

Q: Spark gives an error about a file already existing when you run this:
df.select(columns).write.format("parquet").save(PATH)
What can you add to replace the previous file?
A: .mode("overwrite")
Like this:
df.select(columns).write.format("parquet").mode("overwrite").save(PATH)

Q: If df is a Spark DataFrame, what are two ways you can make it appear as a view in Hive (for the purpose of upcoming SQL queries)?
A: df.createTempView("VIEW_NAME")
df.createOrReplaceTempView("VIEW_NAME")
The former raises an exception if the view already exists.

Q: If you have a query, say SELECT * FROM data, how can you run it with Spark SQL?  Assume your Spark session is in a variable "spark".
A: spark.sql("SELECT * FROM data")

Q: What does spark.sql("QUERY") return?
A: a Spark DataFrame

Q: How do you write a Spark DataFrame (say df) as a Hive table?
A: df.write.saveAsTable("TABLE_NAME")

Q: What does spark.sql("SHOW TABLES") do?
A: Gives you a DataFrame listing all the tables AND all the views.  You'll need to do .show(), .toPandas(), or similar to actually see it.

Q: If tv is a table OR view in Hive, how can you get the corresponding Spark DataFrame?
A: spark.table("tv")

Q: What kind of Spark operation is .count()?
A: It depends.
In df.count() where df is a DataFrame, then it is an ACTION, because it actually triggers the work to get a number back.
In groups.count() where groups is a GroupedData, then it is a TRANSFORMATION, because the counts per group might only be an intermediate result that will be further processed.

Q: How are grouping and windowing similar and different?
A: In each case, we pull together related data into groups/partitions.
In grouping, there is one output row per group (e.g., an AVG stat).
In windowing, all the rows in the partitions can be part of the output, but they can have stats relative to the rest of the partition (e.g., a row_number() relative to the row's position with the partition).

Q: Say you want to filter the results of a Spark SQL query based on a value computed with a windowing function.  Why is this tricky, and what are some solutions?
A: We can't use WHERE -- that runs prior to the windowing function.  We can't use HAVING -- that is only for GROUP BY.
Options:
1. nested SQL query (the outer query can have a WHERE clause to filter)
2. add a where filter using the DataFrame API after the SQL query (this is an excellent case for mixing and matching Spark SQL with Spark DataFrames).
Example of option 2:
spark.sql("""
SELECT area, Call_Number, row_number() OVER (PARTITION BY area ORDER BY Call_Number ASC) AS num
FROM calls
""").where("num <= 3").toPandas()

Q: JOINs typically have ON clauses.  Say your ON clause is something like this:
ON tbl1.colA = tbl2.colB
What is the technical name for this kind of JOIN?
A: "equi join"

Q: If you only want to get the unique values in column C of table T, what SQL can you write?
A: SELECT DISTINCT C FROM T;

Q: What is the difference between these Spark caching levels?
- MEMORY_ONLY
- MEMORY_ONLY_SER
A: MEMORY_ONLY: data is represented as JVM objects on which computation can be done directly.  Unfortunately, JVM objects are somewhat bloated (often using 2-5x more memory than the actual data).
MEMORY_ONLY_SER: data is seralized to a compact byte format.  This saves space, but data partitions must be transformed to JVM objects every time computation is done on them.

Q: Is data that is cached at the DISK_ONLY level serialized?
A: Yes, so it will actually use less storage space than MEMORY_ONLY (not serialized) uses of memory.

Q: What is df.cache() a shortcut for?
A: df.persist(StorageLevel.MEMORY_ONLY)

Q: What is the difference between these to Spark caching levels, and what are the tradeoffs?
- MEMORY_ONLY
- MEMORY_ONLY_2
A: MEMORY_ONLY only keeps the data in RAM on one worker.
MEMORY_ONLY_2 keeps it in RAM on two workers.  This uses more memory in the cluster (unfortunately), but it means we have more flexibility: when doing computation on an RDD partition, Spark has two workers with cached data it can choose from, so it can pick the worker that is less busy.
Also, if a worker with cached data crashes, we can use the other worker, avoiding a long delay to recompute the data that had been cached.

Q: What is the difference between normal partitioning and hash partitioning in Spark?
A: In normal partitioning, the rows are divided roughly evenly into partitions, and there is no guarantee about what row will be in what partition.
With hash partitioning, the user specifies a "key" (usually one or more columns).  For each row, we compute hash(key) % partition_count to get a partition number.  This means that a partition can contain rows with different keys, but all rows for the same key will end up in the same partition.

Q: What is the idea of partial aggregates in Spark?
A: For some aggregates (e.g., sum, count, avg), we can compute partial results prior to the exchange, often saving network I/O.
For example, instead of sending ("A", 1), ("A", 2), ("B", 4) when we're summing per key, a task could send this:
("A", 3), ("B", 4)
Of course, the receiving task might get "A" values from other executors too, so 3 is a partial sum, not the total sum.

Q: One optimization that is part of adaptive query execution (ADE) in Spark is "partition coallescing".  What is it?
A: For GROUP BY and similar queries, Spark needs to bring together related data with hash partitioning, using partition_num = hash(key) % partition_count, for each row.
partition_count could be anything, but needs to be the same for the entire Spark session.  It is 200 by default, which often creates many very small partitions (which is slow later).
The idea of partition coalescing is that the boss checks the sizes of fragments of partition data before those partitions are sent over the network.  It then combines some partitions together to form bigger partitions, ideally similar in size.  Executors are informed of the combining decisions.  So there will generally be <200 partitions in the end.

Q: What does bucketBy do when Spark is writing parquet files?
A: Each task that is writing data will bucket the data by key (this is basically the same as partitioning, but in Spark "partitioning" refers to memory/RDDs, and "bucketing" refers to files).  Data for each bucket is written to a different Parquet file.
This means that when we read the data back in, we can get an RDD that is already hash partitioned by that key.  If we want to do a GROUP BY (on the bucket key) followed by an aggregate, we can skip the network shuffle/exchange that would usually be necessary.

Q: What does SMJ stand for?
A: Shuffle Sort Merge Join

Q: What does BHJ stand for?
A: Broadcast Hash Join

Q: How does a distributed JOIN work with BHJ?
A: This is best for a join between a big table and small table.
The small table is loaded to a dictionary and sent to every machine that will work on a partition of the big table.
Tasks operating on partitions of the big table can independently loop over rows, matching keys with the dictionary.
The dictionary needs to fit in memory on every machine, so this is only a good strategy when the small table is small enough.

Q: How does a distributed JOIN work with SMJ?
A: Table A and table B are both hash partitioned on the column we are joining on.  Then we shuffle data to that we get pairs of partitions (one from A, one from B) with keys in common on the same executor.
The rows in these partitions are sorted by key, so tasks doing the JOIN can loop over the rows together, efficiently finding matches.

Q: Is this deterministic?
df.randomSplit([0.75, 0.25], seed=544)
A: Not completely, because the deterministic split happens at the partition level.  If the number of partitions changes (e.g., because the number of workers changes and Spark decides a different partition count would be ideal), then the overall split won't be deterministic.

Q: Consider Spark's DecisionTreeRegressionModel and DecisionTreeRegressor.  Which one is fitted?
A: DecisionTreeRegressionModel is fitted.
DecisionTreeRegressor is NOT fitted.
Usually, the class names ending in "Model" represent fitted models in Spark.

Q: dt = DecisionTreeRegressor(featuresCol="????", labelCol="????")
We can only pass in one featuresCol -- how do we handle multiple features?
A: Use a vector assembler to transform a DataFrame we want to train/predict on, adding a column of vectors that contains data from multiple columns:
va = VectorAssembler(inputCols=["col1", "col2", ...], outputCol="features")

Q: If we have a fitted model in Spark, how do we used it to make predictions?
A: df_WITH_predictions = model.transform(df_WITHOUT_predictions)
.transform(...) in Spark is used both for preprocessing AND prediction.

Q: A Spark "Pipeline" object is an unfit model, consisting of multiple transformers and ending in an estimator.  When we call "fit" on it, we get a fitted PipelineModel.  What does it contain?
A: All the transformers of Pipeline are copied to PipelineModel.
"fit" is called on the estimator at the end of Pipeline, and this "fit" returns a transformer.  The transformer is copied to the end of PipelineModel, such that PipelineModel is a series of transformers.

Q: The traditional decision tree training algorithm considers splits on every column, for every possible split between rows (a threshold on a column is a split).  Where does Spark consider splits?
A: It still looks at every column, but we configure a number of bins on each column, for example: dt.setMaxBins(N).  A histogram is computed on this with N buckets, and the thresholds considered are those that split the buckets (so N-1 possible split points per column).

Q: What is an "equi-depth" histogram?
A: Each bucket contains a similar number of data points, even if that means some buckets cover a wider range of values than others (e.g., 1-3, 3-10, 10-100, etc).
In contrast, most histograms have buckets with fixed intervals (e.g., 0-10, 10-20, 20-30, etc).

Q: What is the open source system most similar to Google's BigTable?
A: HBase

Q: How do HBase and Cassandra handle replication differently for N-times replicated data?
A: A row in HBase will belong to a single RegionServer, but that RegionServer will store that data in a N-times replicated HDFS file.
A row in Cassandra will belong to N different workers, each of which will store it in a single local file system (the local FS does not replicate it).
In other words: HBase replicates at the file system level, Cassandra replicates at the database level.

Q: HBase and BigTable are "versioned sparse tables" -- what does this mean?
A: There is not a fixed number of columns, and different rows can have different columns.  Looking up a cell is more like using a dictionary, something like table["row:column"].
Each cell may also have multiple versions, and if we want a specific one, we can use table["row:column:version"].
If we look at every possible combination of rows/columns in the table, many correspond to cells with no values -- in a normal database, these missing values would be represented as NULL and representing those NULLs would take a little storage/memory space.  In versioned sparse tables, no space is used to represent the missing values (hence "sparse").

Q: What happens to the regions an HBase RegionServer was responsible for if that server dies?
A: All of that dead server's files (which contain region data) should still be safely stored in HDFS, so other healthy RegionServers can be assigned to take over these regions.

Q: What is a Cassandra keyspace, and where is the data it contains stored?
A: A collection of tables.  Every keyspace has data spread across all workers in the cluster.

Q: Why might you want all data for a particular user to live in a single row in an HBase table?
A: A single row will belong to a single RegionServer, meaning requests for that users data will go to one server instead of many.

Q: At what granularity can a user specify the replication factor, in both HDFS and Cassandra?
A: HDFS: the user specifies replication per file
Cassandra: the user specifies replication per keyspace

Q: What is the primary key in a Cassandra table?
A: The combination of the partition key and cluster key.

Q: What about a row determines which Cassandra workers are responsible for that row?
A: The value in the partition key

Q: What determines the sort order of rows within a partition?
A: The values in the cluster key column(s)

Q: For what kinds of columns are the same values shared across many rows of data?
A: Two cases:
1. partition key
2. static column

Q: What are two systems that inspired the design of Cassandra?
A: Dyno (from Amazon)
BigTable (from Google)
Both these published papers, but not source code.  Facebook designed Cassandra using the ideas in the papers, then open-sourced Cassandra.

Q: Cassandra keyspaces contain tables.  If you want to access table T in keyspace K, but you don't want to type K.T every time you access T, what can you do?
A: Execute this first:
USE K

Q: What is the cluster and partition key for this Cassandra table?
CREATE TABLE ???? (
  ...
  PRIMARY KEY (x, y, z)
)
A: partition key: x
cluster key: (y,z)
If you wanted y to be part of the partition key, you would need to specify like this:
PRIMARY KEY ((x, y), z)

Q: What are two Cassandra functions to create a UUID value, and what is the difference between them?
A: UUID() and NOW()
NOW() does a better job of actually guaranteeing uniqueness because it uses several factors: MAC address (remember these are supposed to be globally unique), current time, and sequence number (in case it is called concurrently)

Q: What happens when you do a Cassandra INSERT with a key that already exists?
A: INSERT is really an upsert (update OR insert), so it will overwrite the previous value if already there.  No error.
The key could be the whole primary key, or just a partition key.

Q: Will this query be fast or slow in Cassandra if table T is large?
alter table T add (newcol sometype)
A: Fast because Cassandra tables are sparse.  Cassandra doesn't need to write a bunch of NULL values to fill out the new column.

Q: What are some advantages of prepared statements, in CQL or SQL?
A: 1. Only need to parse the query once.
2. Easier to call (just specify a few values to plug in).  We can also control default values more easily.

Q: When can you do GROUP BY with Cassandra?
A: When you're grouping on the columns of the primary key.

Q: If your Cassandra keyspace K has a table T, what does this do in Spark?
spark.conf.set("spark.sql.catalog.C", "com.datastax.spark.connector.datasource.CassandraCatalog")
spark.conf.set("spark.sql.catalog.C.spark.cassandra.connection.host", "????")
A: It creates a Spark catalog C containing all the keyspaces of the Cassandra cluster specified on the "host" line.  This means the T table can be queried from Spark as C.K.T.

Q: What is a weakness of HDFS's approach to partitioning?
A: You need a data structure somewhere that maps logical blocks to physical locations on DataNodes.  If there are many blocks in the system, the structure is large -- in the case of HDFS, it is stored in memory in the NameNode, which is expensive.

Q: What is elasticity?
A: It means you can efficiently scale up/down (for example, adding or removing workers in a cluster).  Being efficient means we can't move the majority of the data over the network each time we gain/lose a worker.

Q: What are the strengths and weaknesses of hash partitioning?
A: Strength: it's highly scalable because anybody can figure out where data goes with a little math: hash(key)%partition_count -- no need to ask a centralized boss.
Weakness: it's not elastic.  hash(key)%partition_count rarely equals hash(key)%(partition_count+1).  This means that if you add a worker, most data in the cluster will need to be shuffled over the network to a different worker.

Q: What are a couple famous systems that use consistent hashing to partition data across workers?
A: Dynamo and Cassandra

Q: In consistent hashing, how are tokens managed for (a) workers and (b) rows?
A: Workers: the system can choose tokens for workers (aka nodes) however it wants.  Some options: randomly, or trying to space tokens evenly.  Given the flexibility in the decision, whatever is decided needs to be remembered -- that is, stored in some data structure.  For Cassandra, that data structure is called the token map.  If there are only a few thousand workers in the cluster (a lot, actually), the token map won't take that much space in RAM.
Rows: we want to handle MANY rows (billions, trillions, ...) in the system, so it's not reasonable to store a map of tokens for every row in a table.  Instead, we take a hash of the row's key.  For Cassandra, the key used is the "partition key".  This means that all rows in the same partition will have the same token.

Q: Why does Cassandra have tokens for vnodes (virtual nodes) instead of just for nodes (the workers in the cluster)?
A: 1. when we add a worker to the cluster, it's better if the new worker can take some work away from a few other nodes.  If we only reassign some work from just one old node, it will take longer to hand off the data, and the cluster will not be as balanced in the end.
2. some workers might have more CPU/RAM/etc.  We can give these stronger nodes more vnodes, which will result in them receiving more work.

Q: Given a Cassandra row's token, how do we select a vnode to be responsible for that token?  Assume single replication.
A: If a vnode has exactly the same token as the row, then that vnode owns the row.  Having an exact match like this is rare.
Otherwise, we move to the right in the token ring, finding the first vnode token that is bigger than the row's token.

Q: What is the "wrapping range" in Cassandra, and how do we handle it?  Assume single replication.
A: This is the range of tokens that are bigger than any vnode token.
Having a row with a token in this range is a special case since we usually "walk the ring" to the right to find a vnode for a row.  We handle this corner case by assigning the row to the vnode with the smallest token in the cluster.

Q: What is "heterogeneity" in a cluster?
A: It means the workers are running on different hardware -- some machines might be more powerful than others (more CPU, RAM, etc.).

Q: Where is the token map stored in Cassandra?
A: We don't have a centralized boss, so every worker in the cluster stores a copy of the token map.

Q: How do we update the token map in Cassandra?  For example, we might need to insert some new vnodes to the token map if a new worker joins the cluster.
A: We can't plan to update all the workers at once (remember, they all have a copy of the token map).  At any given time, there are probably some machines offline (for example, rebooting) if we have a large cluster.
Instead, we update a few workers in the cluster.  All the workers engage in a "gossip" protocol.  Once every second (or N seconds), a worker chooses a peer, and they exchange gossip about any new information their copies of the token map.
As with gossip or epidemics, the information will quickly spread to all the nodes that are alive.  If a node was down or rebooting, it will get the information soon after it comes back online and starts gossiping with its peers.

Q: If a client wants to interact with a Cassandra cluster, how does it start?
A: The client can connect to any worker, which will serve as a coordinator.  Every worker has a copy of the token map, so the coordinator will know what node (or nodes) need to be involved for operations on a given partiton of rows.

Q: When multiple nodes in a system can be affected by the same fault, we say they are in the same _________
A: failure domain
these nodes are vulnerable to "correlated failure"

Q: What are some examples of "faults", and what does fault tolerance mean?
A: Few examples of faults: disk failure, network down, node dies, rack loses power, data center is destroyed.
Fault tolerance: the system can keep "working" even when certain faults occur.
Fault tolerance can take many forms, because different systems can handle different kinds of faults.  For example, HDFS can handle a DataNode failure, but it is not geographically distributed, so it couldn't handle loss of a whole data center.  Cassandra can be geographically distributed, so it could potentially.  Fault tolerance also varies based on what "working" mean -- for example, some systems might insure durability (your data is not lost), but not availability (so the system might be offline for a while).

Q: Given a token T for a row that is replicated N times, how do you "walk the ring" to find the N nodes necessary?
A: Start on T, and keep increasing it, recording every vnode you match or pass in a replica set.  Stop when the replica set contains N nodes.
Special cases:
1. if you get to the biggest possible token number, wrap around to the smallest
2. if you find a vnode that is in the same failure domain as another vnode already found, potentially skip it.  This depends on the policy, which is pluggable.  The simplest policy will just skip vnodes on the same physical node, but more complex policies might also skip nodes that are in the same rack or datacenter.

Q: What does "ack" stand for, and what does it mean in the context of writing data?
A: "ack" is short for "acknowledgement", and receiving an acknowledgement back from a system means our data is "committed" (in other words, "safe" even if certain bad things happen).
Different systems guarantee different levels of safety for committed data.  For example, for one system it might mean the data is in RAM on two workers.  On a system with a stronger guarantee, in contrast, it might mean the data is on disk on three workers.

Q: If you receive ACK, what does it mean?  If you don't receive an ACK, what does it mean?
A: ACK: the data is definitely committed
No ACK: the data MAY or MAY NOT be committed (perhaps the write works fine, but an issue arose with sending the message back to report that)

Q: Cassandra has RF, W, and R options.  How are these used when reading and writing data?
A: Writing: a coordinator will attempt to write RF copies of the data.  It will report success (ack) if at least W nodes successfully write it.
Reading: a coordinator will attempt to read the data from R nodes.  1 of these will be a full data read, and (R-1) workers will just send back a checksum of the data.

Q: In terms of RF, W, and R: what should you do if you want a stronger durability guarantee when you get an ack?
A: Increase W (RF must be >= W, so RF might need to increase too).

Q: In terms of RF, W, and R: what should you do if you want better write availability?
A: Decrease W -- then our write can be considered successful even if fewer nodes are currently working.

Q: In terms of RF, W, and R: what should you do if you want better read availability?
A: Decrease R: then we can read the data even if fewer nodes are currently working.

Q: In terms of RF, W, and R: what should you do if you want to guarantee that reads will see the last written data?
A: Make sure W+R > RF.
This guarantees the write and read quorums will overlap.  For example, if you have 5 nodes, and two subsets, of size 2 and 4, those two subsets are mathematically guaranteed to have some overlap.

Q: What does eventually consistent mean?
A: It means different versions of the data on different workers can temporarily "diverge" -- this means that neither version 1 or version 2 of a row is strictly "newest".
Perhaps version 1 has the latest data in column A, but version 2 has the latest data in column B.  Eventually, though, the rows should sync up.

Q: In eventually consistent systems, sometimes you do a read, and you get conflicting versions of the data back.  Neither version can be considered strictly better.  What are some approaches to handling this?  What does Cassandra do?
A: Approach 1: Pass all the different versions back to the client doing the reads -- let the client sort it out.
Approach 2: Automatically merge all the versions to get a new version.
Cassandra automatically merges.  It keeps timestamps on every cell, and uses a LWW policy (last writer wins), meaning we compare the versions cell by cell, picking the data with the freshest timestamp in each case.

Q: What are two communication systems we've learned, for synchronous and asynchronous communication, respectively?
A: gRPC: sync
Kafka: async
gRPC uses protobufs for messages.  Kafka can use any format for messages (protobufs are a reasonable option).

Q: What are the three kinds of processes that work together in Kafka?
A: producers => brokers => consumers

Q: What is cron?
A: A program that can run other programs on a schedule

Q: What are some advantages of using a streaming platform like Kafka for ETL jobs, rather than running ETL jobs on a schedule (say with cron) to move data from system A to system B?
A: 1. data will be fresher if we're constantly streaming it from transactions processing systems to analytics systems
2. if you have many transactions processing systems and many analytics systems, you may need to write a custom job for every pair of systems.  Having a centralized log (like Kafka) means you just need to write one producer for each transactions processing system and one consumer for each analytics system

Q: Where is topics data stored in Kafka?
A: on brokers

Q: Kafka topics consist of one or more _____, each of which contains a series of ________
A: partitions
messages

Q: Kafka topics can be divided into a fixed number of pieces called _____ that can potentially be assigned to different brokers
A: partitions

Q: When a producer sends a message, what are 3 things it typically specifies?  And of the 3 things, what is optional?
A: topic, key, value
key is optional

Q: If a message doesn't have a key, how does a Kafka consumer decide what topic partition it should go in?
A: A "round robin" policy is used, meaning the producer just rotates between different partitions each time there is a new message.

Q: If a Kafka producer specifies a key for a message, how does it determine what partition the message should go to?
A: partition = hash(key) % partition_count
Note that this is pluggable -- you could define a different policy to override the default if you wanted to.

Q: How does a Kafka consumer request a batch of messages from the brokers, with a timeout of N milliseconds?
A: batch = consumer.poll(N)

Q: What does a Kafka batch contain?
A: messages, divided by topic partition

Q: If Kafka message A was produced before message B, when can we guarantee that consumers will see A before B?
Assume we don't know the partition count, but that whatever it is won't change.
A: We can only guarantee this when A and B have the same topic AND key.

Q: Kafka consumers typically read messages at ever increasing offsets in each of the tracked topic partitions, but if consumers want to move to a different offset, they can call this:
consumer.?????(partition, offset)
A: consumer.seek(partition, offset)

Q: Why might you want to have different Kafka consumer groups?
A: There are cases you might want the same messages to be seen by multiple different applications.  You can achieve this by using different consumer groups for each application.
Example:
c = KafkaConsumer("clicks", group_id="g1", ...)

Q: Why might you want multiple Kafka consumers in the same consumer group?
A: If the consumer group is subscribed to a high-volume topic (meaning the topic is getting new messages at a high rate), it will help the system scale if different consumers running on different machines can be responsible for different partitions of that topic.

Q: What is the difference between log "rollover" and "deletion" in Kafka?  What does "retention" mean?
A: At any given time, new data for a topic partition is being written to an "active" file.  Due to limits on file size/age, we might want to switch to writing to a new file, while still keeping the old file around in a finalized state.  This is "rollover".
"deletion" refers to deleting non-active files that are too old, or meet some other criteria specified by a "retention" policy.

Q: If your want to connect to a Kafka broker "localhost:9092", how would you do it in these cases?
from kafka import KafkaAdminClient, KafkaProducer, KafkaConsumer
admin = KafkaAdminClient(????)
producer = KafkaProducer(????)
consumer = KafkaConsumer(????)
A: This can be pasted in where each "????" is:
bootstrap_servers=["localhost:9092"]

Q: How can you create a Kafka topic "T" with P partitions and R replicas?
A: admin.create_topics([NewTopic("T", num_partitions=P, replication_factor=R)])

Q: How can a Kafka producer send the string "hello" to topic "greetings"?
A: producer.send("greetings", value=bytes("hello", "utf-8"))
Values must always be represented as bytes!

Q: How does a Kafka consumer check for a new batch of messages, with timeout of 1 second?
A: batch = consumer.poll(1000)

Q: How can you see the topic partitions a Kafka consumer is responsible for?
A: consumer.assignment()

Q: If you want a Kafka consumer to go back the start of each topic partition, how can you do that?
A: consumer.seek_to_beginning()

Q: Assignment of topic partitions to a Kafka consumer can happen automatically or manually.  What does the code for each look like?
A: automatic:
consumer.subscribe(["some_topic"])  # Kafka decides which partitions of some_topic
manual:
consumer.assign([TopicPartition("some_topic", 3)])    # this consumer will only read from partition 3 of some_topic

Q: If tp is a topic partition, how can you check the next offset a Kafka consumer will read at for that partition?
A: offset = consumer.position(tp)

Q: When is a Kafka message considered "committed"?
A: When it is written to ALL in-sync replicas.

Q: In Kafka, both consumers and follower replicas fetch data from the leader replica.  But how do the fetches differ?
A: Consumers fetch only "committed" messages.
Followers fetch both committed and uncomitted messages.
This is because we only want consumers to see committed messages.  But the way messages become committed is by being received by more replicas, so the follower replicas need to fetch uncommitted data.

Q: If min.insync.replicas is X and the number of currently in-sync replicas is Y, why doesn't Kafka consider a message committed as soon as it is written to at least X replicas?  (assume X<Y in this question)
A: Hypothetical scenario showing why we can't do this.
Let's say X=2 and Y=3.
1. write occurs on two replicas (one "in-sync" replica doesn't have the message)
2. leader returns an "ack" indicating message is committed at offset 100
3. leader dies
4. Kafka selects a new leader from those that are in-sync -- let's say it selects the one that didn't receive the message
5. the new leader is the authority on what messages are at each offset, and might write a different messages at offset 100.  This will mean a previously committed message will have changed
That's why a Kafka leader WILL NOT return an ack in step 2.  It will wait until the message is on the third replica too.

Q: In order to be considered "in sync", does a follower replica in Kafka need to have ALL the messages the leader has seen?
A: No, it is allowed to lag a bit.  This is configurable -- for example, you might define "in sync" to mean the follower has retrieved a batch reading the end of the topic partition in the last N seconds.

Q: How do "elections" in distributed systems usually differ from a regular democratic election humans might participate in?
A: In a human election, we usually have a couple goals:
1. represent the will of the people
2. have a well defined process so it is clear who the leader is (it is problematic if there is disagreement about who the leader is)
In distributed systems, we usually just care about (2).  So in Kafka, for example, "election" of a replica leader is really just a special "controller" broker making a decision without input from other brokers.

Q: What is an example of a "write anomoly" in Kafka?
A: A message was acknowledged as successfully written, but it was not actually readable later.

Q: What is an example of a "read anomoly" in Kafka?
A: A message was read, but when you try to read it again later, it has dissapeared.

Q: Producers can be created like this: KafkaProducer(..., acks=????)
What are the options, and what do they each do?
A: 0: don't wait for the leader to send an ack
1: ack means leader has written message to its own log
"all": message has been written to all in-sync replicas

Q: How might we end up with "at-least-once" semantics?
A: This happens when a sender keeps retrying until it gets an acknowledgement of success.  If the ack is received on the Nth time, then for the first (N-1) times, the message may or may not have gone through.  This means the message will have been sent between 1 and N times.

Q: How might you end up with "at-most-once" semantics?
A: You try to send a message, and you never retry sending, even if it isn't acknowledged.  If not acknowledged, there's still a chance the message made it through, and the failure happened before the ack could be sent.  So the messages will have been received either 0 or 1 times.

Q: What does it mean for an operation to be idempotent?
A: the effect of performing the operation multiple times is equivalent to the effect of performing the operation a single time

Q: How might we make an operation idempotent, even if it not naturally so?
A: Associate the operation with a unique ID, and whatever system is receiving the operation requests is responsible for remembering previously seen IDs and supressing/ignoring duplicates.

Q: What is the problem with "auto commit" for Kafka consumers?
A: Auto commit happens on some timer, and it's possible the consumer created some output (based on messages in a batch), then crashed before commiting the offset after those messages.  When a new consumer takes over, it will start from slightly old offsets, and re-process messages that have already influenced the output.  This could result in "overcounting" or other incorrect output.

Q: If we have Kafka consumers (a) processing messages from a Kafka topic and (b) writing output based on that processing, how can we guarantee each message is considered exactly once in the output, even if the consumer crashes and restarts?
A: Whenever the consumer writes output, we want to simultaneously (that is, "atomically") write offsets to indicate what messages were used to create that output.  For example, if the consumer is writing to a MySQL database, it might do two INSERTs as part of the same transaction (one INSERT for the actual output, and another INSERT to record the corresponding offsets somewhere).

Q: How can you use spark.read in Spark to get rows corresponding to Kafka messages?
A: Use .format to specify data is coming from Kafka.  Use .option to specify the bootstrap servers for the Kafka cluster and the topic(s) to subscribe to.

Q: spark.read.???? returns a regular DataFrame.  How do you get a streaming DataFrame?
A: replace "read" with "readStream"

Q: When you read from a Kafka topic from Spark, what columns will your DataFrame have?
A: key, value, some others.  They columns you get will always be the same, regardless of the message format!
This usually means Spark has to do some extra work to extract different values out of the value, which probably contains some data structure serialized as bytes.

Q: Why is .config("spark.sql.shuffle.partitions", ????) important for Spark streaming, but not other Spark work?
A: The default number of shuffle partitions is 200, meaning that after group by or similar, there will be 200 partitions in the resulting RDD.  There is a 1-to-1 correspondence between tasks and partitions, so this means that further computation will (by default) cause hundreds of tasks to run -- the total startup costs of these tasks becomes a significant overhead.
In recent versions of Spark, a "partition coalescing" feature has been added that combines together many small partitions, usually reducing the results to far fewer than 200 partitions.  But the partition coalescing optimization is not supported for streaming queries, so Spark programmers need to choose this config carefully for streaming scenarios.

Q: What does a streaming Spark query do?
A: Moves data from a source to a sink, possibly with some transformations in between.

Q: How are source and sink specified in a Spark streaming query?
A: With readStream and writeStream, respectively:
q = spark.readStream(????).????.writeStream(????).start()

Q: When you uses Spark's from_json to deserialize to a struct/dictionary, you must specify a schema.  What if the schema lists a column not found in the dictionary?  What if the dictionary has a column not listed in the schema?
A: Schema has extra column: fill with NULL values
Dictionary has extra column: ignore it

Q: What does "stateless" mean in the context of Spark streaming?
A: Each microbatch of rows can be processed without referring back to results computed from previous microbatches.

Q: What are some examples of sources/sinks in Spark streaming?
A: Kafka, Parquet files in HDFS, Cassandra, console, etc.

Q: What are the three output modes available for Spark streaming queries?
A: append
complete
update

Q: Most systems use replication to implement fault tolerance.  Why doesn't Spark streaming need to do that?
A: Spark streaming is based on RDDs, which describe how to compute the data on demand, as needed.  So as long as data can be re-read from the source, we can re-materialize any output previously computed for an RDD.

Q: Why is spark.sql.shuffle.partitions an important configuration for Spark streaming, but not for Spark batch jobs?
A: In recent versions of Spark, a "partition coalescing" optimization was added that combines small partitions into bigger partitions.  But it is only supported for batch jobs.  With streams, we don't know the future distribution of data across partitions, so we cannot readily decide which partitions to combine to get large, balanced partitions.

Q: What do watermarks do?
A: They specify an upper bound on how "late" data could conceivably arrive.  If we have a streaming GROUP BY that groups on time intervals, this means we can eventually discard our running count for a given time window because we can assume no more data will arrive that could change that count.
This prevents memory consumption from growing indefinitely.

Q: Spark streaming supports most of the operations supported by batch jobs, but a "pivot" operation is an example of one that only works for batch (not streaming).  Why doesn't it work?
A: If you allowed pivot for streaming, there are some new inputs that could appear that would cause an additional column to be added to the output.  But Spark DataFrames (whether streaming or not) need to have a well-defined set of columns so that SQL queries and other operations can operate on them.

Q: Say you have "A ???? JOIN B", where A is static and B is a stream.  What kind of JOINs are supported?
A: INNER and RIGHT are supported.
LEFT is NOT supported.  The reason is that a left join requires us to output a row of A with NULL columns from B if there is no B row matching.  But we can never conclude there is no B row matching because B keeps growing, and we don't know the future.

Q: When do Spark streaming jobs provide exactly-once semantics?
A: You need 3 conditions:
1. deterministic code
2. re-readable source
3. idempotent sink (this means that it can suppress duplicates)

Q: What does EC2 stand for, and what is it?
A: Elastic Compute Cloud
AWS's VM service (AWS is Amazon Web Services).

Q: What is usually more expensive to rent in the cloud, 1 VM for 100 hours, or 100 VMs for 1 hour?
A: The cost is usually the same.

Q: What is the difference between IaaS and PaaS?
A: IaaS: Infrastructure as a Service.  You rent low-level hardware-like resources (e.g., VMs, virtual disks, private networks, etc).  You use these to deploy systems yourself.  For example, you might launch a bunch of VMs and start running HDFS DataNodes on them.
PaaS: Platform as a Service.  The cloud provider runs a specific system on your behalf.  It could be something they built (e.g., S3), or something open source (e.g., MySQL).  This is usually more convenient, but you often pay more for that convenience.

Q: What does cloud "lock in" mean?
A: It means it is difficult to move to another cloud (or out of the cloud), even if if that's desirable in the long run.  Reasons for the difficulty:
1. need to pay a high network egress cost to get your data out
2. your code might be written to specifically use an API not available in other clouds

Q: What is a "tenant", and what is the difference between multi-tenant hosts and single-tenant hosts?
A: A tenant is a customer paying for cloud services, and might include multiple users (e.g., employees in the same company).
Multi-tenant: different tenants rent VMs on the same physical machine.
Single-tenant: all the VMs on a single physical machine will belong to the same tenant.  Usually more expensive, especially if that tenant doesn't have enough VMs to keep that physical machine fully utilized.

Q: What is a cloud cost you typically pay when VMs are powered off, but not deleted?
A: You'll typically pay for the virtual disk attached to the VM (because the cloud provider needs to expend storage resources to keep your data around so your VM can be booted up again later).

Q: What is a typical geographic hierarchy in clouds?
A: continents (approximate) contain regions, which contain zones.
Zones in the same region will be geographically near each other, but generally have different power and network resources, making correlated failures between zones unlikely.

Q: What is ingress/egress in terms of cloud resources, and what is more expensive?
A: Ingress: incoming network I/O (e.g., data being uploaded to a VM).
Egress: outgoing network I/O (e.g., data being downloaded from a VM).
Ingress is typically free.  Egress is sometimes free (perhaps when egress is to another VM in the same zone), but there is usually a cost per GB of data transferred.  Transferring out of the cloud is usually most expensive.  Within the cloud, the cost depends on whether it is the same region, same continent, etc.

Q: You can either pay a fixed price to get a VM (with high chances of success), or you can pay a fluxuating price (usually cheaper), and potentially get pre-empted (VM is taken away so cloud can use resources for something of higher priority).
What are these two billing models called?
A: "on-demand" instances and "spot" instances, respectively.

Q: The HDFS file system was modeled on a paper about the GFS file system at Google.  What is the newer distributed file system Google built and started using after GFS?
A: Colossus File System

Q: Is BigQuery a data warehouse or a data lake?
A: The distinction is blurred.  The BigQuery engine is integrated with a BigQuery storage engine (as in a warehouse), but BigQuery can also execute on external data sources such as parquet files in HDFS (as in a lake).

Q: The parquet file format was inspired by the ______ format developed at Google, which was eventually replaced internally at Google with the _______ format.
A: ColumnIO, Capacitor

Q: When a value in a column is repeated many times consecutively, we can record just the value and count (instead of many copies of the value).  This is called ________ encoding.
A: run-length

Q: When a column contains large strings, and the same values show up multiple times, we can replace those strings with shorter numeric codes.  We can use the numeric codes to lookup the original strings as needed.  This is called ______ encoding.
A: dictionary

Q: What is something that a GPL open-source license requires that is not required by MIT/BSD/Apache?
A: If you are making changes to the source code, you generally have to release your changes publicly under the GPL too (there are some exceptions/workarounds).

Q: What is a type of intellectual property the Apache open-source license focuses on that the MIT and BSD licenses do not talk much about?
A: They all talk about copyright; Apache additionally talks about patents

Q: What are some factors you might consider when choosing a GCS bucket region?
A: Cost, carbon emissions, proximity to users, etc.

Q: What is the GCP marketplace?
A: Collection of datasets (some free) that can directly be queried with BigQuery (the marketplace also includes software and other things).

Q: How can you activate/deactivate your GCP credentials on a GCP VM?
A: gcloud auth ...

Q: What does this let you do?
%load_ext google.cloud.bigquery
A: It lets you write BiqQuery queries directly in Jupyter cells, like this:
%%bigquery
SELECT * FROM some_table

Q: You run this:
q = bq.query("...")
Why might q.total_bytes_processed be different than q.total_bytes_billed?
A: Billing gets rounded up to the nearest MB, and there is a 10 MB min overall.

Q: What is a cross join?
A: Every row in table A is matched with every row in table B (there is no "ON" expression to limit the matches between tables).

Q: What is a correlated cross join?
A: Some fields in a row might contain multiple values -- for example, it could be an an array of structs.
A correlated cross join does a cross join on a row by row basis.  For example, say we have two rows like this, and we are doing a correlated cross join between the two columns:
[1], [2,3]
[4,5], [6,7]
We would get these outputs:
1,2
1,3
4,6
5,6
4,7
5,7
Notice we don't join across rows (for example, 1 does NOT get paired with 6).

Q: If you want to do a correlated cross join against a field C containing an array of structs, how do you first "expand" the arrays so each entry becomes its own row?
A: UNNEST(C)

Q: What are some examples of external tables we can register in BigQuery?
A: Parquet files in GCS, Google sheets, etc.

Q: If column C is an array type, how do you get the Nth item from every array using BigQuery?
A: SELECT C[OFFSET(N)]
FROM some_table

Q: What can we call if we want to get a pandas DataFrame from a BigQuery query?  What about a geopandas GeoDataFrame?
bq.query(...).?????()
A: to_dataframe
to_geodataframe

Q: Say you have a query like this:
SELECT * FROM myproj.mydataset.tbl1 WHERE x=3
And you want to put the results in a new BigQuery table named myproj.mydataset.tbl2
How do you do this?
A: CREATE OR REPLACE TABLE `myproj.mydataset.tbl2`
AS
SELECT * FROM myproj.mydataset.tbl1 WHERE x=3

Q: What is the difference between ST_CENTROID_AGG(geo_column) and ST_CENTROID(geo_column)?
A: ST_CENTROID_AGG will give a single centroid for the shape representing the union of every shape in geo_column.
ST_CENTROID will give many centroids, one for each shape in a cell of geo_column.

Q: If you have floats long and lat, what can you call in BigQuery to convert this to a geographic point that can be used with other spatial functions?
A: ST_GEOGPOINT(long, lat)

Q: What does ST_SNAPTOGRID(point, 0.1) do, and what is the purpose?
A: It rounds the latitude and longitude of point to one decimal point.
The reason is that we sometimes want to ask a data scientist to do geographic analysis, but we want to preserve the privacy of customers to some extent, not giving away their exact location.

Q: What is a BigQuery slot?
A: A unit of resources, consisting of approximately 1 GB of RAM and 0.5 CPUs

Q: What are two BigQuery billing models?
A: Capacity and On-Demand

Q: Which BigQuery billing model has priority access to resources (i.e., first "right of refusal")?
A: Capacity billing.  This is reserved capacity, so it should be usable whenever needed.
Any leftover resources (if any) are shared between on-demand jobs.

Q: What do on-demand and capacity billing charge for in BigQuery?
A: Both charge for Collosus storage.
On-demand additionally charges for Collosus I/O.
Capacity additionally charges for CPU+memory.

Q: What does it mean that a BigQuery task is preemptible?
A: It can be interrupted if resources are needed for something more important.  For example, an on-demand task might be preempted by a capacity task (capacity is considered more important).

Q: In BigQuery, what are two reasons to do train/test splitting yourself (for ML) instead of letting BigQuery do it?
A: 1. BigQuery has odd defaults depending on dataset size.  For example, for <500 rows, all data is used for training.
2. BigQuery remembers the train/test split for 48 hours, so it will feel deterministic at first, but you'll be surprised later

Q: In BigQuery, what must be passed to ML.WEIGHTS(...)?
A: MODEL your_model_name

Q: In BigQuery, what must be passed to ML.PREDICT(...)?
A: MODEL your_model_name, QUERY
The query must have all the features needed to predict.

Q: In BigQuery, what must be passed to ML.EVALUATE(...)?
A: MODEL your_model_name, QUERY
The query must return all the features needed to predict AND a label column indicating the correct label.

Q: In BigQuery, how do we do non-linear transformations on our features?
A: When creating the model, add a line something like this:
TRANSFORM(POWER(x, 2), ...)
In the parentheses, you can put things you would normally put after select: "SELECT ?????".  This lets you add columns that are transformations of raw columns in the input data.

Q: In BigQuery, how do you handle OneHot encoding?
A: Do nothing: BigQuery will automatically do it for you if your training data has categorical features.

Q: What does COS stand for (in the context of Google?)
A: Container Optimized Operating System

Q: What does GKE stand for (in the context of Google)?
A: Google Kubernetes Engine

Q: What is a "service account" on Google cloud?
A: Basically a non-human user (e.g., a program that automates some stuff).

Q: What is bin packing?
A: Trying to place processes/containers/VMs carefully on machines to maximize how many things we can run at once without exhausting computational resources.

Q: What are cgroups?
A: Control Groups -- these control performance isolation (e.g., you could impose limits on how much CPU time processes in a cgroup have, so that other processes aren't starved of CPU time).

Q: What are namespaces, in the context of containers?
A: Features that limit what resources are visible to a process, to achieve namespace isolation.
For example, if you run "kill 123" in process-ID namespace A, then 123 can only refer to processes in A; a process with ID 123 in namespace B would be safe from being killed.

Q: What is something that two containers in a Kubernetes pod share that two regular Docker containers would not typically share?
A: network namespace (which contains NICs, physical or virtual)

Q: If you want to host a Docker image registry on Google's cloud, what's an easy service to use?
A: The Artifact Registry (Docker images are one kind of artifact that is supported).