Title: 544
Tags: lec31

Q: What are two communication systems we've learned, for synchronous and asynchronous communication, respectively?
A: gRPC: sync
Kafka: async
gRPC uses protobufs for messages.  Kafka can use any format for messages (protobufs are a reasonable option).

Q: What are the three kinds of processes that work together in Kafka?
A: producers => brokers => consumers

Q: What is cron?
A: A program that can run other programs on a schedule

Q: What are some advantages of using a streaming platform like Kafka for ETL jobs, rather than running ETL jobs on a schedule (say with cron) to move data from system A to system B?
A: 1. data will be fresher if we're constantly streaming it from transactions processing systems to analytics systems
2. if you have many transactions processing systems and many analytics systems, you may need to write a custom job for every pair of systems.  Having a centralized log (like Kafka) means you just need to write one producer for each transactions processing system and one consumer for each analytics system

Q: Where is topics data stored in Kafka?
A: on brokers

Q: Kafka topics consist of one or more _____, each of which contains a series of ________
A: partitions
messages

Q: Kafka topics can be divided into a fixed number of pieces called _____ that can potentially be assigned to different brokers
A: partitions

Q: When a producer sends a message, what are 3 things it typically specifies?  And of the 3 things, what is optional?
A: topic, key, value
key is optional

Q: If a message doesn't have a key, how does a Kafka consumer decide what topic partition it should go in?
A: A "round robin" policy is used, meaning the producer just rotates between different partitions each time there is a new message.

Q: If a Kafka producer specifies a key for a message, how does it determine what partition the message should go to?
A: partition = hash(key) % partition_count
Note that this is pluggable -- you could define a different policy to override the default if you wanted to.

Q: How does a Kafka consumer request a batch of messages from the brokers, with a timeout of N milliseconds?
A: batch = consumer.poll(N)

Q: What does a Kafka batch contain?
A: messages, divided by topic partition

Q: If Kafka message A was produced before message B, when can we guarantee that consumers will see A before B?
Assume we don't know the partition count, but that whatever it is won't change.
A: We can only guarantee this when A and B have the same topic AND key.

Q: Kafka consumers typically read messages at ever increasing offsets in each of the tracked topic partitions, but if consumers want to move to a different offset, they can call this:
consumer.?????(partition, offset)
A: consumer.seek(partition, offset)

Q: Why might you want to have different Kafka consumer groups?
A: There are cases you might want the same messages to be seen by multiple different applications.  You can achieve this by using different consumer groups for each application.
Example:
c = KafkaConsumer("clicks", group_id="g1", ...)

Q: Why might you want multiple Kafka consumers in the same consumer group?
A: If the consumer group is subscribed to a high-volume topic (meaning the topic is getting new messages at a high rate), it will help the system scale if different consumers running on different machines can be responsible for different partitions of that topic.

Q: What is the difference between log "rollover" and "deletion" in Kafka?  What does "retention" mean?
A: At any given time, new data for a topic partition is being written to an "active" file.  Due to limits on file size/age, we might want to switch to writing to a new file, while still keeping the old file around in a finalized state.  This is "rollover".
"deletion" refers to deleting non-active files that are too old, or to meet some other criteria specified by a "retention" policy.