TL;DR Architecture

At high level:

1. Global Traffic Routing (Geo DNS / Anycast)
2. Edge Layer (CDN + WAF + Rate Limiting)
3. Load Balancers (L4 + L7)
4. Stateless App Tier (autoscaled microservices)
5. Multi layer Caching (edge, distributed, local)
6. Data Layer (sharded DB + replicas + queue based writes)
7. Async Processing (Kafka / stream workers)
8. Observability + Auto healing

1) Capacity Planning First (Quick Math)

Never start with boxes and arrows. Start with numbers.

Assume:

• 100K RPS peak
• 95th percentile response target: < 150ms
• Read heavy workload: 90% reads, 10% writes
• Payload average: 2KB response
• Peak egress = 100,000 * 2KB = ~200MB/s (~1.6 Gbps, ignoring overhead)

Now read/write split:

• Reads: 90K RPS
• Writes: 10K RPS

Without caching, DB is dead.

So design target: serve 80–95% reads from cache.

If 90K reads and 90% cache hit:

• DB reads drop to 9K RPS (huge difference)

2) Request Flow (End-to-End)

Client
  ↓
Geo DNS / Anycast
  ↓
CDN + WAF + DDoS Protection
  ↓
L4 Load Balancer (TCP/connection distribution)
  ↓
L7 Load Balancer / API Gateway (routing, auth, throttling)
  ↓
Stateless Service Pods (Kubernetes / ECS)
  ↓
Cache Layer (Redis Cluster + local in memory cache)
  ↓
Database Layer (primary + read replicas + shards)
  ↓
Event Bus (Kafka/Pulsar) for async writes and side effects

3) Load Balancing Strategy (100K RPS Safe)

L4 + L7 combo

• L4 LB handles raw connection distribution efficiently.
• L7 LB/API Gateway handles:
  • path based routing
  • JWT/auth checks
  • per client rate limits
  • request shaping

Algorithms

• Use least request or EWMA latency based routing over round robin.
• Enable slow start for new pods (avoid sending full traffic instantly).
• Circuit break unhealthy nodes quickly with active health checks.

Multi region failover

• Active in at least 2 regions for true HA.
• Traffic policy:
  • nearest region by latency
  • fail to secondary if SLA drops

4) Caching: Your Main Weapon Against DB Overload

At 100K RPS, caching is not optional.

Multi layer cache design

1. CDN Cache (public GET responses, static + semi dynamic)
2. Service side Distributed Cache (Redis Cluster)
3. In process Local Cache (hot keys, tiny TTL)

Recommended pattern

• Cache aside for most reads.
• Key format namespaced:
  • user:v3:12345:profile
• TTL with jitter:
  • e.g. 120s ± random(0–30s)
  • prevents thundering herd at exact expiration moment.

Prevent cache stampede

Use:

• Request coalescing (single flight per key)
• Soft TTL + background refresh
• Distributed lock for ultra hot keys

Hot key protection

If one key gets massive traffic:

• Replicate hot key across multiple Redis slots
• Or keep a near cache in every app instance
• Add tiny randomized response delays for abusive clients

5) Database Design Under High Load

Primary principles

• Keep app tier stateless, keep DB tier protected.
• Reads should mostly hit cache or read replicas.
• Writes should be controlled and often asynchronous where possible.

Read scaling

• Primary + N read replicas
• Route stale tolerant reads to replicas
• Use follower reads with max replication lag guard (e.g. < 200ms)

Write scaling

For 10K writes/sec:

• Partition by tenant/user/hash key
• Use sharding for horizontal scaling
• Avoid cross shard joins in hot paths

Index strategy

• Index only what query patterns demand
• Watch write amplification from too many indexes
• Add covering indexes for hottest reads

Connection management

• Use DB proxy/pooler (PgBouncer, RDS Proxy, etc.)
• Enforce max connections per service
• Backpressure before DB collapse

6) Async Architecture for Spikes

Not every request must complete all work synchronously.

Use an event stream (Kafka/Pulsar):

• API writes critical data quickly
• Emits event
• Background workers process:
  • emails
  • analytics
  • notifications
  • search indexing
  • cache warming

This decouples user latency from heavy downstream work.

Queue safety patterns

• Idempotency keys
• Retry with exponential backoff
• Dead letter queues (DLQ)
• Exactly once is hard; prefer effectively once with dedupe

7) Failure Handling & Resilience

At 100K RPS, partial failures are normal.

Must have patterns:

• Timeouts everywhere (client, service, DB, cache)
• Circuit breakers to isolate failures
• Bulkheads to isolate resource pools
• Rate limiting (per IP/token/tenant)
• Load shedding for non critical traffic
• Graceful degradation (return partial response instead of 500)

Example degradation:

• Personalized recommendations fail → show trending defaults from cache.

8) Observability (If You Can’t See It, You Can’t Scale It)

Track these golden signals per service:

• Latency (p50/p95/p99)
• Traffic (RPS)
• Errors (4xx/5xx)
• Saturation (CPU, memory, queue depth, DB connections)

Also monitor:

• Cache hit ratio (global + per endpoint)
• Redis latency + evictions
• DB replica lag
• Kafka consumer lag
• LB healthy target count

Use distributed tracing (OpenTelemetry) for cross service bottlenecks.

9) Security & Abuse Controls at Scale

High traffic attracts abuse.

Add:

• WAF managed rules
• Bot detection and challenge
• API keys/JWT scopes
• Per consumer quota tiers
• Payload size limits
• Schema validation at gateway

For sensitive operations:

• Idempotency tokens
• Signed requests
• Replay protection (nonce + timestamp)

10) Cost Aware Scaling

100K RPS can become very expensive very quickly.

Cost controls:

• Push cache hit ratio upward (biggest saver)
• Use autoscaling with smart cooldowns
• Rightsize pods and DB nodes continuously
• Use spot/preemptible nodes for stateless workers
• Tiered storage for old data
• Archive cold analytics out of primary DB

11) Example Technology Stack (Reference)

• Edge: Cloudflare / Fastly / Akamai
• LB/API: NGINX / Envoy / AWS ALB + API Gateway
• App: Go / Rust / Java (stateless containers)
• Cache: Redis Cluster (+ local LRU cache)
• DB: Postgres sharded / MySQL Vitess / Cassandra (depends on access patterns)
• Stream: Kafka
• Orchestration: Kubernetes
• Observability: Prometheus + Grafana + OpenTelemetry + Tempo/Jaeger

12) “Surprise Me” Section: Make It Sexy ✨

Here are advanced touches that make this architecture stand out:

A) Adaptive TTL Cache

Instead of fixed TTL, auto tune TTL based on key popularity + update frequency:

• hot and stable keys => longer TTL
• volatile keys => shorter TTL Result: higher hit ratio without stale data explosion.

B) Predictive Cache Warming

Use yesterday’s traffic patterns + upcoming events to pre warm top keys before peak hours.

C) Brownout Mode

When system stress is high, automatically disable non essential features:

• hide expensive widgets
• skip deep personalization
• defer non critical writes Users still get core functionality fast.

D) Cell Based Architecture

Split platform into isolated “cells” (mini stacks with their own app/cache/db shards).

If one cell fails, blast radius stays contained.

13) Minimal Launch Plan (Practical Roadmap)

Phase 1 (up to ~10K RPS)

• Single region
• L7 LB + autoscaled stateless services
• Redis + primary DB + replicas
• Basic metrics + alerts

Phase 2 (~10K to 50K RPS)

• Add CDN caching
• Add queue/event driven async workers
• Read/write splitting, stronger rate limits
• Better tracing and SLOs

Phase 3 (~50K to 100K+ RPS)

• Multi region active
• DB sharding + cell architecture
• Advanced cache protections
• Brownout + load shedding automation
• Game day chaos testing

Final Takeaway

At 100K RPS, success is less about one “super database” and more about traffic shaping + caching + graceful degradation.

If you remember one formula, make it this:

Scale reads with caches, scale writes with partitioning + async workflows, protect everything with backpressure.

That is how you survive high load without burning your database—or your team.