What Is This Course About?

Introduction

Dynamo: Introduction

High-Level Architecture

Data Partitioning

Replication

Vector Clocks and Conflicting Data

The Life of Dynamo’s put() & get() Operations

Anti-entropy Through Merkle Trees

Gossip Protocol

Dynamo Characteristics and Criticism

Summary: Dynamo

Quiz: Dynamo

Mock Interview: Dynamo

Dynamo: How to design a key value store?

Cassandra: Introduction

High-level Architecture

Cassandra Consistency Levels

Gossiper

Anatomy of Cassandra's Write Operation

Anatomy of Cassandra's Read Operation

Compaction

Tombstones

Summary: Cassandra

Quiz: Cassandra

Mock Interview: Cassandra

Cassandra: How to Design a Wide-column NoSQL Database?

Messaging Systems: Introduction

Kafka: Introduction

Kafka: Deep Dive

Consumer Groups

Kafka Workflow

Role of ZooKeeper

Controller Broker

Kafka Delivery Semantics

Kafka Characteristics

Summary: Kafka

Quiz: Kafka

Mock Interview: Kafka

Kafka: How to Design a Distributed Messaging System?

Chubby: Introduction

Design Rationale

How Chubby Works

File, Directories, and Handles

Locks, Sequencers, and Lock-delays

Sessions and Events

Master Election and Chubby Events

Caching

Database

Scaling Chubby

Summary: Chubby

Quiz: Chubby

Mock Interview: Chubby

Chubby: How to Design a Distributed Locking Service?

Google File System: Introduction

Single Master and Large Chunk Size

Metadata

Master Operations

Anatomy of a Read Operation

Anatomy of a Write Operation

Anatomy of an Append Operation

GFS Consistency Model and Snapshotting

Fault Tolerance, High Availability, and Data Integrity

Garbage Collection

Criticism on GFS

Summary: GFS

Quiz: GFS

Mock Interview: GFS

GFS: How to Design a Distributed File System Storage?

Hadoop Distributed File System: Introduction

Deep Dive

Data Integrity & Caching

Fault Tolerance

HDFS High Availability (HA)

HDFS Characteristics

Summary: HDFS

Quiz: HDFS

Mock Interview: HDFS

HDFS: How to Design File Storage System?

BigTable: Introduction

BigTable Data Model

System APIs

Partitioning and High-level Architecture

SSTable

GFS and Chubby

Bigtable Components

Working with Tablets

The Life of BigTable's Read & Write Operations

Fault Tolerance and Compaction

BigTable Refinements

BigTable Characteristics

 Summary: BigTable

Quiz: BigTable

Mock Interview: BigTable

BigTable: How to Design a Wide Column Storage System? 

Introduction: System Design Patterns

1. Bloom Filters

2. Consistent Hashing

3. Quorum

4. Leader and Follower

5. Write-ahead Log

6. Segmented Log

7. High-Water Mark

8. Lease

9. Heartbeat

10. Gossip Protocol

11. Phi Accrual Failure Detection

12. Split Brain

13. Fencing

14. Checksum

15. Vector Clocks

16. CAP Theorem

17. PACELC Theorem

18. Hinted Handoff

19. Read Repair

20. Merkle Trees

System Design Patterns

Quiz I

Quiz II

Final Assessment

Contact us

Appendix

Grokking the Advanced System Design Interview

Test your knowledge of GFS's architecture by answering these questions.


GFS scales by adding ChunkServers to the cluster. The master gradually moves data from existing ChunkServers to the newly added servers. Since the master stores all metadata in memory (for faster operations), GFS is limited by how much memory the master has. At Google, this is not a serious problem in practice as the master maintains less than 64 bytes of metadata for each 64 MB chunk, but as the system has grown in usage, problems have emerged with the centralized master scheme.Despite the separation of control and data flow and the performance optimization of the master, it is emerging as a bottleneck in the design. Despite the reduced amount of metadata (because of the large chunk size), the amount of metadata stored by the master is increasing to a level where it is getting difficult to keep all the metadata in main memory.