Title: 544
Tags: lec2 lec1 lec13 lec4 lec5 lec11 lec6 lec7 lec15 lec19 lec17 lec8 lec9 lec16 lec14 lec10 lec20 lec18

Q: What do systems primarily manage?
A: Resources: compute, memory, storage, and network

Q: what does a compiler do?
A: translate high level code (something a programmer would write) to low level code, such as machine code that a CPU can directly execute

Q: what does an interpreter do?
A: loop over high-level code and execute corresponding operations on the CPU, as the program executes

Q: What is a language (other than Java) that compiles to bytecode for the JVM?
A: Scala is the best answer (important for us, because we'll learn Spark, which was written in Scala)
Python could technically be an answer too due to the Jython project, but this probably isn't used much in practice.

Q: Consider high level code (like Python), machine code, and byte code -- what can run directly on a CPU?
A: machine code

Q: What does JVM stand for?
A: Java Virtual Machine

Q: What is the common extension for files containing Java bytecode?
A: .class

Q: What is the common extension for files containing Python bytecode?
A: .pyc

Q: What does this Python code do?
import dis
dis.dis("????")
A: Shows you the Python bytecode corresponding to the Python code in the string.

Q: If you use more machines, what kind of scaling is it?
A: scaling out

Q: If you use a more powerful machine, what kind of scaling is it?
A: scale up

Q: What does CPU stand for?
A: Central Processing Unit

Q: What does GPU stand for?
A: Graphics Processing Unit

Q: What does RAM stand for?
A: Random Access Memory

Q: What does HDD stand for?
A: Hard Disk Drive

Q: What does SSD stand for?
A: Solid State Disk

Q: What does NIC stand for?
A: Network interface card/controller

Q: What does it mean for a device to provide random access?
A: You can jump around and read from different locations efficiently.

Q: What values can a bit hold?
A: 0 or 1

Q: How many different values can a byte hold?
A: A byte has 8 bits, so there are 2^8 different possible combinations of bits.  So 256 different values.

Q: How much is 1 KB?
A: 1024 (or 1000) bytes

Q: How much is 1 MB?
A: 1024 KB (or 1000 KB)

Q: How much is 1 GB?
A: 1024 (or 1000) MB

Q: How much is 1 TB?
A: 1024 (or 1000) GB

Q: What does it mean for a device to be "byte addressible"?
A: Each byte has it's own address, and the CPU can use that address to get the value.
Extracting a single bit from a byte actually involves more steps than using the whole byte.

Q: What is generally faster, storage or memory?
A: Memory

Q: What does volatile mean, and what is an example of hardware that is volatile?
A: Volatile means you lose the contents upon reboot.  Memory (e.g., RAM) is volatile, storage (e.g., SSD) is non-volatile.

Q: What do we mean with we call HDDs and SSDs "block devices"?
A: Data is read/written in blocks of many bytes (for example, 0.5 KB).
Accessing 1 byte in the block takes the same time as accessing the whole block.

Q: A device can store 1000 GB, can transfer 100 MB/s, and serve one request in 10 ms.  What kinds of metrics are these?
A: 1000 GB: capacity
100 MB/s: throughput
10 ms: latency

Q: If throughput can be expressed as X MB/s or Y Mbps, what is the relationship between X and Y?
A: Y = X * 8
(because the "b" in Mbps stands for bits, and there are 8 bits in a byte)

Q: often a bunch of server machines are stacked on top of each other in a metal box called a ______ and connected with a network switch
A: rack

Q: a large facility containing many racks of computers is called a ______
A: data center

Q: a cluster is a ______
A: bunch of computers working together

Q: what is a program you can use to run shell programs on another machine?
A: ssh

Q: what is the technical term for "folder"?
A: "directory" -- "d" shows up in many acronyms/commands for this reason (pwd, cd, etc)

Q: What Linux command tells you see what directory you're in?
A: pwd

Q: What Linux command shows you all the contents of a directory?
A: ls

Q: How do you create a new empty file, file.txt?
A: touch file.txt

Q: What Linux command do you use to install software on Ubuntu?
A: apt

Q: What is a Linux command you can use to download from a URL to a file on your computer?
A: wget SOME_URL
(curl is a similar tool)

Q: What Linux command lets you rename a file?
A: mv name1.txt name2.txt
It also lets you move a file, like this:
mv name.txt somedir

Q: How can you copy from file A to file B?
A: cp A B

Q: What Linux command lets you copy files between computers?
A: scp

Q: What Linux command lets you see everything in a file?
A: cat

Q: What Linux commands let you see just the start or end of a file?
A: head file.txt
tail file.txt

Q: What Linux command creates a new directory?
A: mkdir somedir

Q: What command lets you see entries in the Linux manual?
A: man SOMECOMMAND

Q: How do you navitage to a new directory?
A: cd SOMEDIR

Q: How do you run commands as the root user?
A: sudo COMMAND

Q: What Linux command lets you change the access permissions on a file?
A: chmod

Q: Given a program name, how can you find where it is on your computer?
A: which PROGRAMNAME

Q: How can you see an environment variable (for example, PATH)?
A: echo $PATH

Q: How can you pipe the standard output of A to the standard input of B?
A: A | B

Q: How do you redirect the standard output of A to file.txt, creating file.txt (or replacing any prior content)?
A: A > file.txt

Q: How do you redirect the standard output of A to file.txt, creating file.txt (or appending to the end of any prior content)?
A: A >> file.txt

Q: How do you send BOTH standard output and standard error to a file?
A: myprogram &> file.txt

Q: What programs counts lines, words, and characters?
A: wc

Q: What program filters standard input, writing matching lines to standard output?
A: grep

Q: What program recursively finds directories and files?
A: find

Q: How can you run a program asyncronously?  (meaning you can type other things on the prompt while waiting for it to finish)
A: longprogram &

Q: How can you see running programs and their PIDs?  (process IDs)
A: ps
To see more:
ps ax
a=other users
x=not just processes attached to a terminal

Q: How do you kill a process by its PID number?
A: kill PID

Q: How do you kill a process by its name?
A: pkill NAME

Q: How can you see CPU and memory usage on your machine?
A: htop

Q: What Linux command shows you disk utilization?
A: df
("df -h" shows human readable)

Q: What command shows you disk usage by everything in the current directory?
A: du -hs .

Q: How can you see most port numbers being used by processes on your machine?
A: sudo lsof -i tcp -P

Q: If you want to download an image X from DockerHub to your VM, what command do you use?
A: docker pull X

Q: How can you see all the Docker images on your VM?
A: docker images

Q: How can you give a Docker image "a" an additional second name, "b"?
A: docker tag a b

Q: How can you launch a container from image X?
A: docker run X

Q: How can you see existing containers?
A: docker ps

Q: How can you delete a container X?
A: docker rm X

Q: How can you delete an image X?
A: docker rmi X

Q: How can you see Docker's disk utilization?
A: docker system df

Q: How can you cleanup unnecessary Docker data to free up space?
A: docker system prune

Q: How can you see the output (for example prints) for an application running inside container X?
A: docker logs X

Q: How do you run a command (lets say "ls") inside a running container X?
A: docker exec X ls

Q: How do you see resource consumption of running containers?
A: docker stats

Q: How do you get a bash shell inside an existing container X?
A: docker exec -it X bash

Q: How do you cause a container X to stop running?
A: docker kill X

Q: How do you run the Dockerfile in the current directory to build an image X?
A: docker build . -t X

Q: What is the difference between "docker rm" and "docker kill"
A: After "docker kill", the container won't be running, but it will still exist (you can see it with "docker ps -a").  "docker rm" actually cleans up the space that was used by that container.

Q: In a Dockerfile, how do you specify a base image X?
A: FROM X

Q: In a Dockerfile, how do you execute a command, say "apt update"?
A: RUN apt update

Q: In a Dockerfile, how do you copy a file X.txt (in the same directory on the VM) to /Y.txt (in the image)?
A: COPY X.txt /Y.txt

Q: Why are the three main numeric types in Python?
A: integers, floating point numbers, and complex numbers

Q: Of integers and floats, what support is there for Infinity?
A: Floats support this, ints do not

Q: If you have a type like intN or floatN, how many bytes does it take?
A: N/8
For example, int32 takes 4 bytes

Q: What is the unsigned equivalent of int16?
A: uint16

Q: When we choose between different numeric types, what are three things we're trading off?
A: precision (specifying small differences)
range (difference between smallest and biggest possible numbers)
space (how many bytes does it take?)

Q: If you invent a new bit layout to represent your numbers, what will you need to worry about with respect to performance?
A: Existing CPUs won't have built-in operations for your type.  So you'll need to implement operations like multipy in software (which is slower than hardware), or convert to another type the CPU can directly work with on an as-needed basis (also slow)

Q: What happens when you add negative infinity and positive infinity?
A: You get NaN (not a number)

Q: What does sys.float_info.dig tell you?
A: The maximum number of decimal digits that can be faithfully represented in a float

Q: What does sys.float_info.epsilon tell you?
A: difference between 1.0 and the next number after 1.0 that we can represent with a Python float

Q: If you want torch.rand(...) to give you the same pseudorandom numbers each time, what must you do?
A: "seed" the randomizer
torch.manual_seed(SOME_NUMBER)

Q: If you want to move a CPU tensor A to a GPU tensor B, what do you do?
A: B = A.to("cuda")

Q: torch.sigmoid(x) will return numbers between _____ and _____
A: 0, 1

Q: How do we matrix multiply A and B in PyTorch?
A: A @ B

Q: If you want to create a callable type, what special method do you need in your class?
A: __call__

Q: x = torch.rand(a, b)
y = torch.rand(c, d)
When will x @ y work?
A: When b = c

Q: x = torch.rand(a, b)
y = torch.rand(c, d)
If x @ y works, what will be the resulting shape?
A: (a, d)

Q: How many columns will it have?
A = torch.rand(x, y)
A: y

Q: model = torch.nn.Linear(x, y)
preditions = model(data)
How many columns must data have?
A: x

Q: PyTorch: if y is a function of x, and you're trying to find the best x to optimize y, what do you need to do to let PyTorch know you'll later need a gradient?
A: x = torch.tensor(????, requires_grad=True)

Q: PyTorch: if y is a function of x, and you turned on gradient tracking for x, how can you get the y-over-x gradient (after you've calculated y)?
A: y.backward()
x.grad

Q: PyTorch: if you call .backward() on more than one result without zeroing gradients, what happens, say to x.grad?
A: x.grad will acculumate gradients: it will be the some of gradients with respect to all results on which you called .backward().

Q: PyTorch: say you've calculated y as a function of x, and you're trying to find a better x value.  You're using an optimizer opt for this purpose, and you've already called backward() to get gradients.
How do you make a small change to x?
A: opt.step()

Q: What does SGD stand for?
A: Stochastic Gradient Descent

Q: PyTorch: by default, the SGD optimizer is trying to get small results.  What should you pass to the SGD constructor if you want to get big results?
A: maximize=True

Q: PyTorch: what does the "lr" parameter for SGD stand for?
A: learning rate

Q: PyTorch: when you call optimizer.step(), how much will an input tensor x change?
A: x.grad * (learning rate of optimizer)
maximization: increase by that amount
minimization: decrease by that amount

Q: PyTorch: if you're optimizing y as a function of x and using an optimizer opt, how do you reset x.grad to zero?
A: opt.zero_grad()

Q: What are the big three areas of machine learning?
A: Reinforcement learning: agent makes series of actions to maximize reward
Unsupervised learning: looking for general patterns
Supervised learning: train models to predict unknowns

Q: What is the difference between regression and classification
A: regression: predict a number
classification: predict a category

Q: PyTorch: when training on a larger dataset, it is often efficient to use the whole dataset for each step of the optimizer.  What do we do?
A: We break the dataset into batches.  Within the loop over epochs, we have another loop over batches of the dataset.

Q: What does MSE stand for?
A: Mean Squared Error

Q: What PyTorch type lets us slice over our data, getting batches of features (x values) and corresponding batches of labels (y values).
A: Dataset.  An in-memory one is TensorDataset

Q: What PyTorch type provides additional features to a DataSet, like shuffling, batching, and iteration?
A: DataLoader

Q: There is one virtual address space per _______
A: process

Q: A process is a running ______
A: program

Q: Virtual memory maps to ______
A: physical memory

Q: Which address space can contain "holes" (invalid addresses)
A: Virtual address spaces can have holes (physical address spaces do not -- they are contiguous)

Q: What are three things that go in a virtual address space?
A: code
stack
heap

Q: Which pointer indicates what code to execute next?
A: The "instruction pointer"

Q: There is one stack per _____ and one heap per ______
A: thread, process

Q: What is it called when the scheduler stops running one thread and starts running a different one?
A: A "context switch"

Q: What are three states a thread can be in?
A: running, blocked, ready

Q: What are some reasons a thread might be blocked?
A: waiting on user input, disk operations, network operations, timer sleep, etc.

Q: How do you create and launch a thread object for running function f(x)?
A: t = threading.Thread()
t.start(target=f, args=[x])

Q: How do you wait for a thread t to finish running?
A: t.join()

Q: How can you print a thread's ID?
A: t.get_native_id()

Q: What is a "race condition"
A: A bug that only sometimes surfaces, depending on when context switches happened between threads

Q: When might a context switch occur?
A: Between instructions -- a single line of code might correspond to multiple bytecodes or machine instructions, so it's important to note that context switches often occur in the middle of a line of code being executed.

Q: What is a "race condition"?
A: A bug, where whether or not you get correct behavior depends on how threads get scheduled.
Race conditions are tricky to test+debug because they're non-deterministic (not same behavior each time), and you might only see the incorrect behavior rarely (say one in a million times).

Q: What is "deadlock"?
A: This is when threads can never make progress, because they each want to acquire a lock that will never be released.
One example: t1 has A and t2 and B; t1 is waiting to additionally acquire B, and t2 is waiting to additionally acquire A.  This will never happen because neither thread will release its first lock until it has acquired the second lock and done some work.

Q: How can a thread (a) get a lock and (b) let go of it?
A: lock.acquire()
lock.release()

Q: How can you rewrite the following code so that the release will happen, even if an exception occurs?
lock.acquire()
# code with possible exception
lock.release()
A: with lock:
    # code with possible exception

Q: What do you call a section of code where certain interleavings with other code would be a problem?
A: critical section

Q: What is "atomicity" in the context of multi-threaded programs?
A: If a thread is changing multiple things together, we never want to see (say from another thread) partial results, with only some changes made.

Q: What is an "invariant"?
A: A rule that applies to the data, often something a human might know from context.  For example, an invariant might be a list of ages cannot contain negative numbers.  Some invariants are known by the system (for example, that the length of the list itself cannot be negative).

Q: What is "consistency" in the context of threading?
A: All invariants are enforced.

Q: If thread t1 holds a lock, what could happen?
1. we could context switch to another thread t2
2. another thread t2 could acquire the lock
A: (1) is possible, (2) is not

Q: What happens in Python when you try to release a lock that has already been released?
A: There is an exception

Q: What is the GIL?
A: Global Interpreter Lock

Q: Before a CPU can add two values in RAM, what must it do?
A: Load those values to CPU registers

Q: Which CPU cache is fastest?
A: L1

Q: Which CPU cache is biggest?
A: L3

Q: When data in a cache is too old and no longer valid, we say it is _______
A: stale

Q: What does FIFO stand for?
A: First In First Out

Q: What does LRU stand for?
A: Least Recently Used

Q: How are FIFO and LRU similar?  How are they different?
A: They are the same for a miss: the data is added to the cash, in the "newest" position.
They are different for a hit: FIFO does not change how it prioritizes the item, but LRU resets it to the "newest" position.

Q: What do you call it when you find an item in a cache?
A: "hit"

Q: What do you call it when you CANNOT find an item in a cache?
A: "miss"

Q: What is a pattern where a random eviction policy would beat LRU and FIFO?
A: A common pattern is that the data is too big to fit completely in cache, and the program loops, repeatedly reading over the dataset in the same order.  LRU and FIFO will have a 0% hit rate, but random will at least get some hits.

Q: What is the name for a piece of data loaded to CPU cache?
A: cache line

Q: What is the most common cache line size for modern CPUs?
A: 64 bytes is common (other sizes like 128 used to be more common; new Macs haved started using 128 again)

Q: If you want to sum over column in a PyTorch matrix in a cache friendly way, how should you create the tensor?
A: First create a tensor where each column is actually stored as a row in the matrix.  Then transpose it.  This will get you columns that are contiguous in memory, resulting in fewer cache lines being loaded as the data is scanned.

Q: How does PyArrow indicate which strings/ints/etc are null/None in an array?
A: validity bitmap: 1 means valid, 0 means null/None

Q: How can you add a region to your virtual address space that is not backed by a file?
A: anonymous mmap

Q: How can you map a file into your virtual address space?
A: Open the file, get the f.fileno(), and pass it to mmap.

Q: If you want to be able to evict pages in anonymously mmap'd regions, what do you need to do?
A: enable a swap file in Linux

Q: How many bits are in a MAC address?
A: 48

Q: What does the prefix of a MAC address indicate?
A: The vendor/manufacter that produced the NIC
This is so that we can achieve globally unique MAC addresses (each vendor gets a prefix, and is expected not to reuse the same suffix for different devices).

Q: How can you see the MAC addresses and IP addresses for your NICs?
A: ifconfig
("if" is short for interface)

Q: What is a virtual NIC that is only reachable from the local machine, and what IP address does it use?
A: The "loopback" device, which has address 127.0.0.1

Q: What is a network?
A: A collection of computers that can send each other messages by MAC address

Q: Within a network, what device can forward data that is sent to a specific MAC address?
A: A switch

Q: What is the device that connects multiple networks to form an internet, or the Internet?
A: a router

Q: How many bits does an IPv4 address contain, and how many possible addresses are there?
A: 32 bits, and about 4 billion possible addresses

Q: What does a prefix of an IP address tell you?
A: It identifies the network

Q: Why do routers forward based on IP address rather than MAC address?
A: Because there are too many MAC addresses in the world, and forwarding tables would need to be too big if every router needed to remember where every MAC address lives.
With IP addresses, IPs with the same prefix are on the same network.  A router just needs to know where to forward data to get it closer to the correct network.

Q: The following are private IP ranges, meaning multiple networks can use the same IPs from these ranges internally:
192.168.0.0 to 192.168.255.255
172.16.0.0 to 172.31.255.255
10.0.0.0 to 10.255.255.255
What device/software can let us connect to machines using these IPs, given they are not unique?
A: A NAT (network address translation) box.
It can map a unique public to one of these private IPs within a network.

Q: What is something that both UDP and TCP add in addition to what IP provides?
A: The both provide port numbers, which let us direct communication to a specific process on a machine

Q: What is something that TCP provides that UDP does not?
A: TCP provides more reliable communication:
retry when a packet is dropped
buffer packets on receiving side and re-arrange so they arrive in the same order they were sent

Q: What does UDP stand for?
A: User Datagram Protocol

Q: What does TCP stand for?
A: Transmission Control Protocol

Q: Why would you ever prefer UDP over TCP?
A: TCP does extra work, which is sometimes not necessary.  For example, in a Zoom call, if some packets are dropped the session may freeze.  After that, it's more natural for humans to repeat what they were saying (it would be weird if it replayed video/audio from the few seconds when the connection was lost).

Q: What is the relationship between domain names in a URL and IP addresses?
Example:
https://tyler.caraza-harter.com:443/cs544/s23/schedule.html
A: When you enter a domain name into a browser, it gets translated to an IP address automatically for you.

Q: If you are using HTTP over TCP/IP, what functionality does TCP/IP provide, and what does HTTP provide?
A: TCP/IP gives you IP addresses and port numbers, letting you communicate with a specific process (a server on a specific machine.
HTTP additionally lets you specify what resource you want to access via that server.
For example, consider this URL:
https://tyler.caraza-harter.com:443/cs544/s23/schedule.html
The resource is "/cs544/s23/schedule.html"

Q: What does RPC stand for?
A: Remote Procedure Call
Procedure: another name for a function/method
Remote: we want it to run on a different process/container/computer

Q: What are the RPC implementations created by Meta and Google called, respectively?
A: Thrift and gRPC

Q: What is gRPC layered on top of?
A: HTTP
Every method call is an HTTP POST request.  The arguments go in the request body, the return value goes in the response body.

Q: What is it called when we convert various data types into bytes to send over the wire?  What about the reverse conversion?
A: serialization/deserialization

Q: What does gRPC use for serialization?
A: Protocol Buffers (protobufs for short)

Q: What is variable-length encoding, and what are the pros/cons of using it?
A: Use more space for bigger numbers.  E.g., if an int32 value is small enough, we could use 1, 2, or 3 bytes (instead of the usual 4).
Pros: save space.  Very important for sending data over the network, so protobufs do this.
Cons: CPUs cannot directly operate on int32's unless they are 4 bytes, so conversion is necessary before sending and after receiving this data.

Q: Port forwarding: if we want to forward port 300 of the loopback device on a VM to port 80 inside a container, how do we run the container?
A: docker run -p 127.0.0.1:300:80 myimg

Q: What does this do?
ssh USER@VM -L localhost:1234:localhost:5678
A: This is executed on a machine (say your laptop) to create an SSH tunnel to the VM.  While the tunnel is in use, any requests to localhost:1234 on the laptop will be forwarded to localhost:5678 on the VM ("localhost" is not redundant here because it is referring to two different loopback devices, the one on the laptop and the one on the VM).

Q: Consider this:
docker run -d -p 300:80 myimg
We specify the port on the VM (300) and container (80), but we don't specify which NIC.  What will be used?
A: By default, the command is equivalent to this:
docker run -d -p 0.0.0.0:300:80 myimg
This means traffic on all of the VM's NICs will forward to the container (possibly a security risk, depending on your firewall settings).
Inside the container, it will forward to whatever the main NIC is (e.g., eth0), NOT the loopback device (so anything inside the container should listen on 0.0.0.0, not localhost).

Q: What is container orchestration, and what is currently a very popular orchestrator?
A: Orchestration lets you deploy cooperating containers across many different machines.
Kubernetes is probably the most well known.

Q: What does "docker compose" do?
A: Lets you start a bunch of cooperating containers on the same machine.

Q: protobuf files (.proto) can support two different syntaxes, v2 and v3.  If you want v3, what should the first line of your .proto be?
A: syntax = "proto3";

Q: What are the two main things a .proto file contains?
A: messages
services
Example:
message MyMessage {
  ...
}
service MyService {
  ...
}

Q: In a .proto file, what does a message contain?
A: Fields, each with a type and a number, like this:
message MyMessage {
  int32 x = 1;
  int32 y = 2;
}

Q: In a .proto file, what does a service contain?
A: RPCs, like this:
service MyService {
  rpc MyRPC(Message1) returns (Message2);
}
Each RPC has one message containing arguments and another containing return values.

Q: When you generate code for a .proto file, you get two files:
????_pb2
????_pb2_grpc
What are they each for?
A: ????_pb2: code for serializing/deserializing the messages
????_pb2_grpc: servicer (class to be inherited from) and stubs for making remote procedure calls

Q: Consider this:
python3 -u server.py &> log.txt
What does "-u" do?
A: "force the stdout and stderr streams to be unbuffered"
Otherwise, if its buffered, any prints from server.py might not immediately appear in log.txt (that can be confusing if you're using log.txt to debug).

Q: If you have a docker-compose.yml in the current directory, and you want to start running the services defined, what can you do?
A: docker compose up

Q: If you want to see the compose containers currently running, what do you do?
A: docker compose ps

Q: If you want to stop running containers for a compose cluster, what can you do?
A: docker compose down

Q: If you want to delete all containers for a docker compose cluster, what can you do?
A: docker compose rm
(add -f to force it; for example, might be necessary if they're currently running)

Q: Memory is byte-addressible.  But what is the unit of access for an HDD, SSD, etc?
A: These are block devices, so the unit is a block.  A typical block size might be about 512 bytes.

Q: What are two ways we can use memory to make access to block devices more efficient?
A: caching: keep some disk data in memory, so we don't need to go back to disk if the data is reused
buffering: when doing many smaller reads/writes, accumulate more data in memory before doing the transfer between disk and memory (because block devices are more efficient for bigger accesses).

Q: What are the names for these two data access patterns?
1. reading lots of data in order
2. jumping around and reading from many different places
A: 1. sequential
2. random

Q: "block" and "page" have different meanings for SSD internals than we've seen for operating systems and interactions with block devices.  What are the differences?
A: block (for SSDs): unit of erasure (perhaps 100s of KBs)
block (for using a block device): unit of transfer to/from device, perhaps 512 bytes
page (for SSDs): unit of read/write within the define (few KBs)
page (for system memory): unit of mapping between virtual memory and physical memory (usually 4 KB)

Q: How can we break a block device into smaller sections?
A: with partitions

Q: How can we treat multiple individual block devices like one bigger, more reliabel one?
A: With RAID (redundant array of inexpensive disks)

Q: What is the data structure in a file system that points to blocks containing file data?
A: inode

Q: On UNIX, how do we make many smaller file systems appear as one big one?
A: By mounting them on top of each other

Q: What does it mean for a format to be "column oriented"?
A: The data in the column is layout out consecutively in memory or within a file.

Q: What is the orientation of CSVs and parquet files?
A: CSVs: row oriented
Parquet: column oriented

Q: What is the tradeoff between text encoded and binary encoded formats?
A: text: easier to debug (because you can just look at it)
binary: usually more efficient (often less space, and numbers usually already in a format on which CPUs can directly operate without further conversion)

Q: Compression works by looking for repetition over some window, say 32 KB.  Would would the the advantage and disadvage of increasing the window size?
A: Advantage: find more repetition, and use that to get a smaller file size
Disadvantage: spend more computation time finding the repetition

Q: What does snappy prioritize, high speed, or high compression ratios?
A: high speed
from the docs: "[Snappy] does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression."

Q: Considering CSVs and parquets: when do we need to infer the schema?
A: With CSVs we need to infer it, because everything in a CSV is a string (need to detect that all the strings in a given column look like integers, for example).
Parquets have schema information built in, so there is no need to infer the schema.

Q: What is transactions processing?
A: Operating on individual rows or a small number of rows at a time (e.g., adding, modifying, deleting rows).

Q: What is analytics processing?
A: Trying to answer questions about the data, which involves computation over whole columns of data.

Q: What does OLTP stand for?
A: Online Transactions Processing

Q: What does OLAP stand for?
A: Online Analytics Processing

Q: What is a data warehouse?
A: An OLAP database where we collect data from multiple sources (e.g., OLTP databases) for analysis.

Q: What does ETL stand for?
A: Extract Transform Load
Often used to get data into a data warehouse.  For example, an ETL process might extract data from an OLTP database, transform it in some way, then load it into an OLAP database (the data warehouse).

Q: For analytics work, what are some tradeoffs between storing data in a database vs. storing it in some files (e.g., parquet files)?
A: Database advantage: a database will often have a special storage format co-designed to work with the analytics engine -- the tight integration can lead to great performance.
Files advantage: if you want to go beyond doing SQL queries, having your data in a bunch of files makes it easier to use other tools.  For example, it will probably be faster for some machine learning package to pull data from Parquet files (than do a big query on the database to extract everything).

Q: What SQL clause lets you combine two tables?
A: JOIN

Q: What are two SQL clauses that let you filter down what rows are included based on some condition, and how do these two clauses differ?
A: WHERE and HAVING
WHERE is prior GROUP BY (if GROUP BY is used), so filters input rows
HAVING is after GROUP BY, so filters groups

Q: What SQL clause pulls together related rows?
A: GROUP BY

Q: In SQL, SUM, AVG, and other operations that work over all rows (or groups of rows) are called ______
A: aggregates

Q: What SQL clause sorts rows?
A: ORDER BY

Q: What SQL clause lets you limit results to the first N rows?
A: LIMIT

Q: What does ACID stand for?
A: - atomiticy: it all happens or nothing happens (partial progress is rolled back)
- consistency: application invariants (like no negative bank accounts) are supported
- isolation: others cannot see a transaction in progress (aka atomicity when talking about locks)
- durability: once finished, it persists (even if machine crashes+restarts)

Q: If conn is a SQLAlchemy connection, and you've made some changes as part of a transaction, how do you finalize that transaction?
A: conn.commit()

Q: If conn is a SQLAlchemy connection, and you've made some changes as part of a transaction, how do you abandon that transaction?
A: conn.rollback()

Q: What does GFS stand for?
A: Google File System

Q: What is the Hadoop system similar to GFS?
A: HDFS (Hadoop File System)

Q: What is a Hadoop system similar to Google's BigTable?
A: HBase

Q: What is a distributed analytics system that many people have started using instead of MapReduce?
A: Spark

Q: When Google MapReduce and BigTable were first created, where did they store their data?
A: GFS

Q: What is the technical word for having multiple copies of our data on different systems?
A: replication

Q: What is the technical word for breaking our data (for example, a file) into smaller pieces that are more manageable?
A: partitioning

Q: HDFS has a boss/worker architecture.  What are the specific names given to the boss and workers in an HDFS cluster?
A: boss: NameNode
worker: DataNode

Q: When a client wants to read an HDFS file, how does it interact with the NameNode and DataNodes?
A: First it tells the NameNode what it wants to read, and the NameNode responds with information about where the client can find the data.
Then, the client fetches the data directly from the DataNodes.
Data does not flow through the NameNode because otherwise having a single NameNode would be a major bottleneck.

Q: What are pipelined writes?
A: If a client wants to write 3 replicas of a block to DataNodes DN1, DN2, and DN3, it does not send the data directly to each of them.
Instead, it sends the data to DN1, which forwards to DN2, which forwards to DN3.
This prevents any one computer from having too much work (most importantly, the client isn't doing all the work, because the client may be a weaker machine than the servers).

Q: If you write 1 MB to a triply replicated file, how much disk I/O does that generate?  What if you read 1 MB from a triply replicated file?
A: Write: 3 MB (need a copy on each machine, for safety)
Read: 1 MB (the three machines have identical data, so there is no point in requesting it from more than one)

Q: How do you format an HDFS NameNode?
A: hdfs namenode -format

Q: How do you start running an HDFS NameNode with hostname "main" and port 9000?
A: hdfs namenode --fs hdfs://main:9000

Q: How can you get an HDFS cluster report for hdfs://main:9000/?
A: hdfs dfsadmin -fs hdfs://main:9000/ -report

Q: How can you start a DataNode and connect it to a NameNode with hostname "main" on port 9000?
A: hdfs datanode -fs hdfs://main:9000

Q: If you want to use commands like mkdir, cp, ls, cat, and du on HDFS, how do you do it?
A: hdfs dfs -COMMAND ARGS
For example, to run "ls" on the "data" directory:
hdfs dfs -ls hdfs://main:9000/data/

Q: What is the difference between a data warehouse and data lake?
A: Both are a place where we collect data for analysis (OLAP workloads).
A warehouse is a database with integrated storage+computation.
A lakehouse is a combination of distributed storage (for example, HDFS) and decoupled analytics engine (for example, Spark)

Q: In MapReduce, what parameters do map and reduce functions take?
A: def map(key, value)
def reduce(key, values)
Note that will reduce, all values with the same key will go to a single reduce call on a single machine.  During map, different values with the same key go to different map calls, perhaps even on different machines.

Q: A reduce task might have the reduce function called several times.  What can we say about the keys across these multiple calls for a single task?
A: The keys will be sorted

Q: What determines the number of output files in a MapReduce job?
A: The number of reducers, because each reducer produces one output file.

Q: What are some performance advantages Spark has over MapReduce?
A: intermediate data is handled better (could directly pass from one computation to the next, without materializing in HDFS)
a chain of RDDs lets us specify many steps to get to a desired end result, but lazy execution means that an optimizer could find more efficient ways to get to that same result

Q: If you have a regular Python list, how can you convert it into a Spark RDD?
A: rdd = sc.parallelize(YOUR LIST)

Q: What is the relationship between actions, operations, and transformations?
A: actions and transformations are both kinds of operations
transformations are lazy (no work done yet), and describe how to produce an output RDD given an input RDD
actions actually materialize some output (on the screen, in an HDFS file, etc), triggering whatever work is necessary to compute that output.  An action is often at the end of a chain of transformations.

Q: A Spark task:
  - runs on a _________
  - operates on a _________
A: runs on a single CPU core
operates on a single partition, which is loaded entirely to memory

Q: Why might you want bigger Spark partitions?
A: Each partition corresponds to a task, and there is an overhead to starting tasks.  Bigger partitions => fewer partitions => less overhead.

Q: Why might you want smaller Spark partitions?
A: - use more cores that are available in the cluster
- balance work more evenly across executors
- use less memory at any given time

Q: What are two categories of Spark transformations?
A: narrow transformation: one input partition corresponds to one output partition
wide transformation: each output partition might need data from multiple input partitions (and vice versa)
if partitions are on different physical machines, wide transformations often cause network I/O (rows need to be shuffled/exchanged across the network)

Q: If df is a Spark DataFrame, how would we start/stop caching it?
A: df.cache() # start
df.unpersist() # stop

Q: What does RDD stand for, in the context of Spark?
A: Resilient Distributed Dataset

Q: How do you create a SparkData corresponding to a CSV at location PATH?
A: df = spark.read.format("csv").load(PATH)

Q: How can you see the number of RDD partitions for a Spark DataFrame df?
A: df.rdd.getNumPartitions()

Q: Say you run this:
spark.read.format("csv").load("test.csv")
And test.csv looks like this:
a,b
1,2
3,4
What will be the first row of data?
A: a,b
Because headers are off by default.  Add this:
spark.read.format("csv").option("header", True).load("test.csv")

Q: Say you run this:
spark.read.format("csv").option("header", True).load("test.csv")
And test.csv looks like this:
a,b
1,2
3,4
What will be the data types for the columns?
A: strings, because everything in a CSV is a string.
To get ints, use this:
spark.read.format("csv").option("header", True).option("inferSchema", True).load("test.csv")

Q: If you want to select the "x" column of a Spark DataFrame df, add 1 to it, and call the result column y, how would you do that?
A: df.select(expr("x+1").alias("y"))

Q: Spark gives an error about a file already existing when you run this:
df.select(columns).write.format("parquet").save(PATH)
What can you add to replace the previous file?
A: .mode("overwrite")
Like this:
df.select(columns).write.format("parquet").mode("overwrite").save(PATH)