What is a System Design Interview?

Functional vs. Non-functional Requirements

What are Back-of-the-Envelope Estimations?

Things to Avoid During System Design Interview

Introduction to System Design Interview

System Design Interviews - A step by step guide

System Design Master Template

Designing a URL Shortening Service like TinyURL

Designing Pastebin

Designing Instagram

Designing Dropbox

Designing Facebook Messenger

Designing Twitter

Designing Youtube or Netflix

Designing Typeahead Suggestion

Designing an API Rate Limiter

Designing Twitter Search

Designing a Web Crawler

Designing Facebook’s Newsfeed

Designing Yelp or Nearby Friends

Designing Uber backend

Designing Ticketmaster

Additional Resources

System Design Problems

System Design Basics

Key Characteristics of Distributed Systems

Load Balancing

Caching

Data Partitioning

Indexes

Proxies

Redundancy and Replication

SQL vs. NoSQL

CAP Theorem

PACELC Theorem (New)

Consistent Hashing (New)

Long-Polling vs WebSockets vs Server-Sent Events

Bloom Filters (New)

Quorum (New)

Leader and Follower (New)

Heartbeat (New)

Checksum (New)

Glossary of System Design Basics

Importance of Discussing Trade-offs

Strong vs Eventual Consistency

Latency vs Throughput

ACID vs BASE Properties in Databases

Read-Through vs Write-Through Cache

Batch Processing vs Stream Processing

Load Balancer vs. API Gateway

API Gateway vs Direct Service Exposure

Proxy vs. Reverse Proxy

API Gateway vs. Reverse Proxy

Primary-Replica vs Peer-to-Peer Replication

Data Compression vs Data Deduplication

Server-Side Caching vs Client-Side Caching

REST vs RPC

Polling vs Long-Polling vs Webhooks

CDN Usage vs Direct Server Serving

Serverless Architecture vs Traditional Server-based

Stateful vs Stateless Architecture

Hybrid Cloud Storage vs All-Cloud Storage

Token Bucket vs Leaky Bucket

Read Heavy vs Write Heavy System

System Design Trade-offs

Contact Us

Other courses

Appendix

Grokking the System Design Interview

## 1. What is a Web Crawler?
A web crawler is a software program that browses the World Wide Web in a methodical and automated manner. It collects documents by recursively fetching links from a set of starting pages. Many sites, particularly search engines, use web crawling as a means of providing up-to-date data. Search engines download all the pages to create an index on them to perform faster searches.

Some other uses of web crawlers are:

* To test web pages and links for valid syntax and structure.
* To monitor sites to see when their structure or contents change.


Let's design a Web Crawler that will systematically browse and download the World Wide Web. Web crawlers are also known as web spiders, robots, worms, walkers, and bots.

Difficulty Level: Hard
