Course Schedule

Part 1: Resources

Week 1

Mon, Jan 19
MLK Day
Wed, Jan 21
Course Intro
Read: Syllabus
Slides: PDF
Watch: Lecture
Fri, Jan 23
Deployment (Linux Shell)
Slides: PDF
Watch: Lecture (Part 1)
Watch: Lecture (Part 2)

Week 2

Mon, Jan 26
Deployment (Linux Pipelines)
Read: Designing Data Intensive Applications, Kleppmann ("Batch Processing with Unix Tools" of Chapter 10)
Slides: PDF
Watch: Lecture
Wed, Jan 28
Deployment (Docker)
Slides: PDF
Watch: Lecture
Release: P1 (Docker)
Fri, Jan 30
Network (Overview)
Read: Designing Data Intensive Applications, Kleppmann (Chapter 4, "Encoding and Evolution")
Slides: PDF
Watch: Lecture

Week 3

Mon, Feb 2
Network (gRPC)
Read: gRPC Basics Tutorial
Slides: PDF
Watch: Lecture
Due: Shell Worksheet
Wed, Feb 4
Network (Distributed Computing)
Slides: PDF
Watch: Lecture
Fri, Feb 6

In-Class Midterm

Week 4

Mon, Feb 9
Memory (Cache Concepts)
Read: Systems Performance, Gregg (6.2.2; "CPU Caches" and "Latency" subsections of 6.4.1)
Slides: PDF
Watch: Lecture
Wed, Feb 11
Memory (Cache Implementation)
Watch: Lecture
Due: P1
Release: P2 (Network+Memory)
Fri, Feb 13
Memory (Cache Friendly)
Read: Gallery of Processor Cache Effects (Examples 1 and 2)
Slides: PDF
Watch: Lecture

Week 5

Mon, Feb 16
Compute (Threads)
Read: Fluent Python, 2nd Edition ("What's New in This Chapter" through "A Bit of Jargon" in chapter 19, "Concurrency Models in Python")
Slides: PDF
Watch: Lecture
Wed, Feb 18
Compute (Locks)
Read: Mastering Concurrency in Python ("Working With Threads In Python" chapter)
Slides: PDF
Watch: Lecture
Fri, Feb 20
Compute (Advanced Concurrency)
Storage (Block Devices)
Slides: PDF
Watch: Lecture

Week 6

Mon, Feb 23
Storage (File Systems and Formats)
Read: Designing Data Intensive Applications, Kleppmann ("Transaction Processing or Analytics?" and "Column-Oriented Storage" sections of Chapter 3, "Storage and Retrieval")
Slides: PDF
Watch: Lecture
Wed, Feb 25
SQL Databases (MySQL)
Read: MySQL Crash Course, Silva (Chapters 3+5), Designing Data-Intensive Applications, Kleppmann ("The Meaning of ACID" section in Chapter 7, "Transactions")
Slides: PDF
Watch: Lecture
Due: P2
Release: P3 (Compute+Storage)
Fri, Feb 27
SQL Practice
Watch: Lecture
Part 2: Clusters

Week 7

Mon, Mar 2

In-Class Midterm

Wed, Mar 4
Hadoop Ecosystem
HDFS
Read: Mastering Hadoop 3, Singh et al. ("Deep Dive Into the Hadoop Distributed File System" chapter)
Slides: PDF
Watch: Lecture
Due: Threads Worksheet
Fri, Mar 6
HDFS
MapReduce
Read: Learning Spark, 2nd edition by Damji et al. (sections "The Importance of an Optimal Storage Solution", "Databases", and "Data Lakes" of chapter 9, "Building Reliable Data Lakes with Apache Spark")
Slides: PDF
Watch: Lecture

Week 8

Mon, Mar 9
Spark RDDs
Watch: Lecture
Wed, Mar 11
Spark DataFrames
Read: Learning Spark, 2nd edition by Damji et al. (Chapter 4, "Spark SQL and DataFrames: Introduction to Built-in Data Sources")
Watch: Lecture
Release: P4 (HDFS)
Fri, Mar 13
Spark SQL
Read: Designing Data Intensive Applications, Kleppmann ("Reduce-Side Joins and Grouping" of Chapter 10, "Batch Processing")
Slides: PDF
Watch: Lecture
Due: P3

Week 9

Mon, Mar 16
Spark Internals and Performance
Read: Learning Spark, 2nd edition by Damji et al. (Chapter 7, "Optimizing and Tuning Spark Applications")
Slides: PDF
Watch: Lecture (Part 1)
Watch: Lecture (Part 2)
Watch: Lecture (Part 3)
Watch: Lecture (Part 4)
Wed, Mar 18
Spark Machine Learning API
Read: Learning Spark, 2nd edition by Damji et al. (Chapter 10, "Machine Learning with MLlib")
Slides: PDF
Watch: Lecture
Fri, Mar 20
Spark Machine Learning: PLANET
Watch: Lecture

Week 10

Mon, Mar 23

In-Class Midterm

Wed, Mar 25
Wide Tables: HBase and Cassandra
Read: Cassandra, The Definitive Guide, by Carpenter et al. (Chapter 4, "The Cassandra Query Language")
Slides: PDF
Watch: Lecture
Due: P4
Release: P5 (Spark)
Due: HDFS_Spark Worksheet
Fri, Mar 27
Cassandra Query Language (CQL)
Watch: Lecture

Week 11

Mon, Mar 30
Spring Break
Wed, Apr 1
Spring Break
Fri, Apr 3
Spring Break

Week 12

Mon, Apr 6
Cassandra Partitioning
Read: Cassandra, The Definitive Guide, by Carpenter et al. (sections "Data Centers and Racks" to "Hinted Handoff" of Chapter 6, "The Cassandra Architecture")
Slides: PDF
Watch: Lecture
Wed, Apr 8
Cassandra Replication
Slides: PDF
Watch: Lecture
Due: P5
Release: P6 (Cassandra)
Fri, Apr 10
Streaming: Kafka Concepts
Read: Kafka, The Definitive Guide, 2nd edition by Shapira et al. ("Enter Kafka" section of Chapter 1, "Meet Kafka")
Slides: PDF
Watch: Lecture

Week 13

Mon, Apr 13
Streaming: Kafka Demos
Watch: Lecture
Wed, Apr 15
Streaming: Kafka Reliability
Read: Kafka, The Definitive Guide, 2nd edition by Shapira et al. (Chapter 7, "Reliable Data Delivery")
Slides: PDF
Watch: Lecture
Release: P7 (Kafka)
Due: Cassandra Worksheet
Fri, Apr 17
The Cloud
Slides: PDF
Watch: Lecture
Part 3: Cloud

Week 14

Mon, Apr 20
Cloud Demos (Google)
Slides: PDF
Watch: Lecture
Due: P6
Wed, Apr 22
DataForm and BigQuery Types
Read: Google BigQuery: The Definitive Guide, by Lakshmanan et al. ("BigQuery Geographic Information Systems" section of Chapter 8, "Advanced Queries")
Slides: PDF
Watch: Lecture
Due: Kafka Worksheet
Fri, Apr 24
BigQuery: Geographic Data
Watch: Lecture
Release: P8 (Cloud)

Week 15

Mon, Apr 27
BigQuery: Machine Learning
Read: Google BigQuery: The Definitive Guide, by Lakshmanan et al. (Chapter 9, "Machine Learning in BigQuery")
Slides: PDF
Watch: Lecture
Due: P7
Wed, Apr 29
BigQuery 4: Cost and Performance
Slides: PDF
Watch: Lecture
Fri, May 1
Review
Watch: Review 1
Watch: Review 2
Due: P8