The engine room.

U.S. law (1996) governing Protected Health Information. The pieces that affect engineers are the Privacy Rule and Security Rule, which apply whenever your system touches PHI.

SOC 2 is a certification, not a law. It's an audit framework run by the AICPA. Companies get a SOC 2 report to show enterprise customers "we have our security house in order." Required to sell to almost any large enterprise.

SOC 2 evaluates controls against five Trust Service Criteria: Security (required), Availability, Processing Integrity, Confidentiality, Privacy.

Two report types:

• Type I — controls are designed correctly at a point in time.
• Type II — controls operated effectively over a period (usually 6-12 months). This is what enterprise customers ask for.

U.S. law for publicly-traded companies. SOX is about financial reporting accuracy. Section 404 requires management to assess and report on the effectiveness of internal controls over financial reporting (ICFR).

SOX touches engineers when you work on systems that produce or feed financial data: billing, revenue recognition, accounting integrations, subscription management, usage tracking, transaction processing.

EU law (since 2018). Applies to any company processing data of EU residents, regardless of where the company is based. Penalties: up to 4% of global annual revenue or €20M, whichever is higher.

Standard for any system that touches credit card data. Maintained by the card brands (Visa, Mastercard, etc.), enforced contractually.

The single most useful thing to know: don't touch card data if you can avoid it. Use Stripe / Adyen / Braintree, never see the PAN, and your PCI scope drops to almost nothing (SAQ A — short questionnaire). The moment your servers see a PAN, you're in full PCI-DSS scope (SAQ D, ~300 controls, full audit).

U.S. law (2013) requiring track-and-trace for prescription drugs through the supply chain. Manufacturers, wholesalers, dispensers, repackagers must all participate.

Engineering implications:

• Each drug package has a unique serial number (NDC + lot + serial).
• Every transfer of ownership generates a T3 transaction document (Transaction Information, Transaction History, Transaction Statement). EPCIS XML format.
• Suspicious products quarantined and reported within 24 hours.
• Full electronic interoperable tracing required by 2024.

The #1 issue for years running. Users access things they shouldn't be able to access — usually because the server checks authentication but forgets to check authorization.

The classic IDOR bug: GET /orders/12345. Server pulls order 12345. Returns it. Never checks whether the logged-in user owns that order.

# wrong:
@app.get("/orders/{order_id}")
def get_order(order_id: int, user: User = Depends(current_user)):
    return db.query(Order).get(order_id)

# right:
@app.get("/orders/{order_id}")
def get_order(order_id: int, user: User = Depends(current_user)):
    order = db.query(Order).filter(
        Order.id == order_id,
        Order.user_id == user.id  # ownership check
    ).first()
    if not order:
        raise HTTPException(404)  # 404 not 403 — don't leak existence
    return order

-- enable RLS on the orders table
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;

-- create a policy: users can only see their own orders
CREATE POLICY orders_owner_policy ON orders
    USING (user_id = current_setting('app.current_user_id')::INT);

-- in the app, set the user context per-connection:
SET LOCAL app.current_user_id = '12345';
SELECT * FROM orders;  -- only returns rows for user 12345, no WHERE needed

Sensitive data exposed because of weak or missing encryption. The big ones:

• Passwords stored in plaintext (or with MD5/SHA-1 — old broken hashes).
• Sensitive data in URLs (URLs leak via referer headers, server logs, browser history).
• HTTP instead of HTTPS for sensitive endpoints.
• Self-signed certs accepted in production.
• Keys hardcoded in source code or committed to git.

# Python with passlib
from passlib.hash import argon2

# at signup:
password_hash = argon2.using(rounds=4, memory_cost=65536).hash(password)
# store password_hash in DB; never store the password itself

# at login:
if argon2.verify(password, stored_hash):
    # success
else:
    # fail

The classic. User input gets interpreted as code. SQL injection is the famous one but the same principle applies anywhere user input meets an interpreter.

# wrong — string interpolation into SQL
query = f"SELECT * FROM users WHERE email = '{email}'"
db.execute(query)
# attacker submits: ' OR '1'='1
# query becomes: SELECT * FROM users WHERE email = '' OR '1'='1'
# returns all users

# right — parameterized
db.execute("SELECT * FROM users WHERE email = %s", (email,))
# the DB driver handles escaping, '%s' is NOT string formatting here

# Postgres (psycopg2)
cur.execute("SELECT * FROM orders WHERE user_id = %s AND status = %s", (uid, status))

# Postgres (asyncpg)
await conn.fetch("SELECT * FROM orders WHERE user_id = $1", uid)

# MongoDB (pymongo) — NoSQL injection prevention
# wrong: collection.find(json.loads(user_input))  # user can inject operators
# right: collection.find({"_id": user_input_id})  # explicit field

# Elasticsearch — queries built from user input need filters
# wrong: query_string with user input directly
# right: bool query with explicit term/match filters

# Prepared statements pre-parse the SQL once, then bind values:
stmt = conn.prepare("SELECT * FROM orders WHERE user_id = $1")
result = stmt.fetch(uid)  # only the binding changes per call, not the SQL

Design-level flaws that no amount of careful coding can fix. Examples:

• Password reset tokens that are predictable or never expire.
• "Forgot password" flows that confirm the email exists.
• Race conditions in critical operations (balance check, then transfer — separate from the actual subtraction).
• Missing rate limits on expensive or sensitive endpoints.

# wrong:
balance = get_balance(user_id)         # read: 100
if balance >= amount:                   # check
    deduct(user_id, amount)             # write: 100 - 50 = 50
    credit(recipient, amount)

# DB-level atomic update with check
UPDATE wallets
SET balance = balance - %s
WHERE user_id = %s AND balance >= %s
RETURNING balance;
-- if no row updated, transfer fails (insufficient funds OR concurrent)

BEGIN;
SELECT balance FROM wallets WHERE user_id = %s FOR UPDATE;  -- lock
-- check, then UPDATE
COMMIT;

Default credentials, exposed admin panels, verbose error messages, unnecessary services enabled, missing security headers. Boring but #1 source of breaches by volume.

The security headers you should set:

Content-Security-Policy: default-src 'self'; ...
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: geolocation=(), microphone=()

Your dependencies have CVEs. log4shell, Heartbleed, the npm supply chain attacks — all came in through dependencies.

What "good dependency hygiene" looks like:

• Dependabot / Renovate enabled, auto-PRs for updates.
• SCA (Software Composition Analysis) tool — Snyk, Trivy, GitHub Advanced Security.
• Lock files committed (package-lock.json, poetry.lock).
• SBOM (Software Bill of Materials) generated for compliance.
• CVE alerts routed to a triage queue with SLAs (critical = patch in 7 days, etc.).

Covered in the auth section of the original doc — JWT vs session, refresh tokens, MFA. The OWASP variant focuses on:

• Brute-force protection (rate limiting, account lockout, CAPTCHA after N failures).
• MFA (TOTP, WebAuthn).
• Session timeout and rotation.
• Credential stuffing protection (haveibeenpwned API integration).

Trusting code or data without verifying it. CI/CD pipelines that pull dependencies without checksum verification. Signed updates. Pickle deserialization in Python (a classic remote code execution vector).

import pickle
# pickle.loads(user_input)  # NEVER — attacker can run arbitrary code

# the malicious payload looks like:
class Exploit:
    def __reduce__(self):
        import os
        return (os.system, ('rm -rf /',))
# pickle.dumps(Exploit()) becomes a payload that runs `rm -rf /` on unpickle

Breaches go undetected for months because nobody was watching. Median time-to-detection for breaches is around 200 days.

What to log (separate from your application logs):

• Failed login attempts (per user and per IP).
• Privilege escalations.
• Access to sensitive data (PHI, PII, financial).
• Configuration changes.
• Admin actions.
• Unusual data access patterns (large exports, off-hours queries).

Where to send it: a SIEM (Security Information and Event Management) — Splunk, Datadog Cloud SIEM, AWS Security Lake. Logs go to write-only storage with object lock so an attacker who breaches the app can't tamper with the audit trail.

Your server makes outbound HTTP requests on behalf of users (image fetcher, webhook receiver, URL preview). User submits a URL pointing at internal infrastructure. Server happily fetches it and returns the response.

The protocol formerly known as SSL. Wraps any TCP connection in an encrypted, authenticated channel. The handshake:

1. ClientHello — client says "I support these cipher suites and TLS versions."
2. ServerHello — server picks a cipher, sends its certificate.
3. Certificate verification — client checks the cert is signed by a CA it trusts, matches the domain, isn't expired or revoked.
4. Key exchange — both sides derive a shared session key (typically via ECDHE).
5. Finished — both sides confirm they have the same key. Channel is now encrypted.

TLS 1.3 is significantly faster (1-RTT vs 2-RTT) and removes legacy crypto. If you can require TLS 1.3, do.

Data on disk is encrypted. If someone steals the disk (or grabs an unencrypted backup), they get ciphertext. The mechanisms:

Keys leak, get exposed in logs, leave with employees, or just age out of best practice. Rotation is the answer, but it has to be designed in.

• Master keys in KMS — auto-rotate annually. KMS handles the re-wrapping of DEKs.
• Application secrets (Stripe key, OpenAI key) — rotate quarterly minimum, immediately on compromise. Code must read from a secrets manager that supports rotation, not from env vars baked into a container image.
• JWT signing keys — rotate by issuing new tokens with a new key ID, accepting both old and new during a transition window, then retiring the old.
• Database passwords — rotate. Use IAM authentication where possible (RDS supports it, no password to rotate).

Microsoft's threat-modeling acronym. For each component in your system, walk through six threat categories:

STRIDE works best when applied at trust boundaries — the places where data crosses from one trust level to another. Common boundaries:

• Internet → your edge (public API surface)
• Your edge → backend services
• Backend → database
• Backend → third-party APIs
• Backend → message queue → other backend
• App → user's browser (XSS surface)

Inside a trust boundary, you can mostly trust the data. Crossing one, you can't.

# Threat Model: Photo Upload (Makro v2)

## System diagram
[client] -> [API Gateway] -> [Lambda] -> [S3]
                                      \-> [Postgres]
                                      \-> [Agent Pipeline]

## Trust boundaries
- Internet → API Gateway (HTTPS, JWT)
- Lambda → S3 (IAM)
- Lambda → Postgres (IAM auth)

## Threats (STRIDE)

### Upload endpoint
- T1 (Spoofing): User A uploads as User B
  Mitigation: presigned URL bound to authed user ID; S3 key includes user ID
  Severity: HIGH | Status: MITIGATED

- T2 (DoS): Attacker requests millions of URLs
  Mitigation: rate limit 100/min/user on /uploads endpoint
  Severity: MEDIUM | Status: MITIGATED

- T3 (Info disclosure): User A reads User B's photo
  Mitigation: presigned GET URLs scoped to owner; bucket-level deny public access
  Severity: HIGH | Status: MITIGATED

### Agent pipeline trigger
- T4 (Tampering): Attacker triggers agent on someone else's photo
  Mitigation: pipeline takes (user_id, s3_key) tuple; verifies key prefix matches user_id
  Severity: HIGH | Status: MITIGATED

## Open risks
- R1: No virus scan on uploads. Acceptable for MVP, planned for v3.
- R2: Cost runaway if user uploads 1000 photos/day. Mitigation: usage quotas (planned).

In a distributed system, you can have at most two of three when a network partition occurs:

• C — Consistency. Every read returns the most recent write, or an error.
• A — Availability. Every request gets a response (not necessarily the latest data).
• P — Partition tolerance. The system keeps working despite network failures between nodes.

P is not optional. Networks fail. So the real choice is C vs A during a partition.

CAP only addresses what happens during partitions. PACELC adds: even when there's no partition, you trade between latency and consistency. If your replicas need to agree before responding, every write costs round-trips. If they don't, reads can be stale.

"Strong vs eventual" is the cartoon version. There are subtler models:

• Strong (linearizable): reads always see the latest write. As if there's one machine.
• Sequential: all clients see operations in the same order, but not necessarily real-time order.
• Causal: if A causes B, everyone sees A before B. Unrelated operations may appear in different orders to different clients.
• Read-your-writes: after you write, you see your own write (but other clients may not yet).
• Monotonic reads: you don't see time go backwards. If you saw value V at time T, you won't later see a value older than V.
• Eventual: if writes stop, eventually all replicas converge.

How a group of nodes agree on a value when any of them might fail. The output is a single replicated log that all nodes agree on.

Paxos is the original (Lamport, 1989). Famous for being correct, hard to understand, and harder to implement.

Raft (2014) is the modern answer — same guarantees, designed to be understandable. The basic shape:

1. Cluster elects a leader. Only the leader accepts writes.
2. Leader appends each write to its log and replicates to followers.
3. Once a majority (quorum) acknowledges, the write is committed.
4. If the leader dies, followers detect it (heartbeat timeout) and elect a new one.
5. The new leader's log is at least as up-to-date as a quorum of nodes (election rules guarantee this).

Where you encounter consensus systems:

• etcd — Kubernetes uses it for cluster state.
• ZooKeeper — older, used by Kafka, HBase.
• Consul — service discovery.
• CockroachDB / Spanner / TiDB — distributed SQL with strong consistency, built on Raft.

Two nodes both think they're the leader. Both accept writes. When the partition heals, you have divergent histories that need to be reconciled — usually by losing data.

Prevention:

• Quorum requirements. A leader can't be elected without majority support. A minority can't make progress.
• Fencing tokens. Each leader gets a monotonically increasing token. Storage systems reject writes from older tokens.
• Lease-based leadership. Leader holds a lease that expires; must renew it. If you can't renew, you stop being leader before your replacement starts.

You can't have ACID transactions across services. The saga pattern compensates: each step has a "do" and an "undo." If a later step fails, you run the undo for previous steps.

Step 1: charge_card($1500)        — undo: refund($1500)
Step 2: reserve_flight()           — undo: cancel_flight()
Step 3: reserve_hotel()            — undo: cancel_hotel()
Step 4: reserve_car()               — undo: cancel_car()

Two dominant ways to organize data on disk. Different trade-offs.

Before any modification touches the actual data files, it's appended to a sequential log on disk. The log is the source of truth; the data files are an optimization.

Why:

• Durability. If the server crashes mid-write, recovery replays the WAL.
• Performance. Sequential disk writes are 100x faster than random ones. The WAL turns random user writes into sequential disk writes.
• Replication. Replicas don't replay user-level operations; they replay WAL entries. Cheap and identical.
• Point-in-time recovery. Keep the WAL around, you can replay to any moment.

How Postgres lets readers and writers coexist without blocking each other. Every row has hidden columns: xmin (transaction that created this version) and xmax (transaction that deleted/superseded it).

When you UPDATE a row, Postgres doesn't overwrite. It marks the old row's xmax and inserts a new row with a new xmin. Old readers (with transaction ID before xmax) still see the old row. New readers see the new row.

Postgres index types and when each is right:

Async replicas are by definition behind. How far behind depends on:

• Write volume on primary.
• Network bandwidth between primary and replica.
• Replica's hardware (CPU/IO).
• Replica running long-running queries that block apply.

Healthy lag is sub-second. Lag in minutes is a problem. Lag in hours is an outage.

-- on the primary
SELECT
  client_addr,
  state,
  pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn) AS sent_lag_bytes,
  pg_wal_lsn_diff(pg_current_wal_lsn(), write_lsn) AS write_lag_bytes,
  pg_wal_lsn_diff(pg_current_wal_lsn(), flush_lsn) AS flush_lag_bytes,
  pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes
FROM pg_stat_replication;

-- on the replica
SELECT
  pg_last_wal_receive_lsn(),
  pg_last_wal_replay_lsn(),
  EXTRACT(EPOCH FROM (NOW() - pg_last_xact_replay_timestamp())) AS lag_seconds;

The sequence:

1. Detect failure (heartbeat timeout, health check).
2. Pick the most up-to-date replica (lowest replica lag).
3. Promote it to primary.
4. Repoint application traffic.
5. Eventually rejoin the old primary as a replica.

The hard parts:

• False positives. Network blip causes false failover. Now you have two primaries. Use multiple health checks and fencing.
• Data loss window. If async replication, the new primary may be missing the last few seconds of writes.
• Connection pooling. Apps must reconnect to the new primary. Pgbouncer / RDS proxy helps.
• DNS / connection string updates. Most setups use a virtual IP or proxy that gets repointed.

Tools that automate this: Patroni (Postgres), Orchestrator (MySQL), RDS Multi-AZ (managed).

Opening a Postgres connection costs:

• TCP handshake
• TLS handshake
• Authentication
• Forking a backend process (Postgres uses one OS process per connection)
• ~1MB of memory per connection on the server

Total: 50-200ms of latency and ~1MB. For a request that needs 10ms of actual DB work, that's 95% overhead.

Pools keep N connections open and hand them out to requests on demand.

The naïve assumption: "we have 100 app servers, give each a pool of 50, so 5000 connections to Postgres." This breaks Postgres.

The reality: Postgres handles roughly 100-300 active connections well, depending on hardware. Each connection uses a backend process; too many = context-switching overhead dominates.

The math you actually want:

connections_needed = (request_rate * avg_db_time)

# If you serve 1000 req/sec, each request takes 20ms of DB time:
# 1000 * 0.020 = 20 connections actively in use at any moment
# Add headroom (2-3x) → pool size of 40-60 connections

You want the smallest pool that handles peak load with headroom. More is not better.

Transaction mode is what you want for most web apps, but it has constraints:

• No prepared statements that span transactions (some ORMs configure this poorly).
• No LISTEN/NOTIFY (session-scoped).
• No temporary tables that span transactions.
• No SET session variables (use SET LOCAL instead).

# bad pattern
with db.transaction():
    user = db.query("SELECT * FROM users WHERE id = ?", uid)
    response = requests.get(f"https://external.api/data/{user.id}")  # 5 seconds
    db.execute("UPDATE users SET ... WHERE id = ?", uid)

# good pattern
user = db.query("SELECT * FROM users WHERE id = ?", uid)
response = requests.get(f"https://external.api/data/{user.id}")
with db.transaction():
    db.execute("UPDATE users SET ... WHERE id = ?", uid)

pool_size = ((core_count * 2) + effective_spindle_count)

A typical web request budget:

The number you care about is not p50 (median) but p95, p99, or p99.9. Why:

• If you have 10 microservices each with p99 of 10ms, your overall request might call all 10. Probability that any of them is in its p99 is much higher than 1%.
• Users notice slow requests, not fast ones. The 1% that are slow are what gets complained about.
• P50 can stay flat while p99 goes through the roof — average tells you nothing.

1 - (0.99 ^ 10) ≈ 9.6%

# Built-in cProfile
import cProfile
cProfile.run('expensive_function()', sort='cumulative')

# Better: py-spy (sampling profiler, doesn't slow things down)
$ py-spy record -o profile.svg --pid <PID>
# generates a flame graph showing where time is spent

# For async / FastAPI: pyinstrument
from pyinstrument import Profiler
profiler = Profiler()
profiler.start()
expensive_function()
profiler.stop()
print(profiler.output_text())

orders = db.query("SELECT * FROM orders WHERE user_id = ?", uid)  # 1 query
for order in orders:
    order.items = db.query("SELECT * FROM items WHERE order_id = ?", order.id)  # N queries
return orders

# option 1: JOIN
SELECT o.*, i.*
FROM orders o
LEFT JOIN items i ON i.order_id = o.id
WHERE o.user_id = ?;

# option 2: IN clause
order_ids = [o.id for o in orders]
items = db.query("SELECT * FROM items WHERE order_id = ANY(?)", order_ids)
items_by_order = group_by(items, 'order_id')
for order in orders:
    order.items = items_by_order.get(order.id, [])

# option 3: ORM eager loading
orders = db.query(Order).options(joinedload(Order.items)).filter(...).all()

# option 4: DataLoader (GraphQL pattern)
# batch all .items requests in one tick into a single query

Caching is the single biggest performance lever, and the easiest to get wrong.

The hierarchy (each layer 10-100x faster than the next):

1. L1 — in-process cache. Python dict, Caffeine, lru_cache. Sub-microsecond. Doesn't survive restarts.
2. L2 — distributed cache. Redis, Memcached. Sub-millisecond. Shared across instances.
3. L3 — CDN. Cloudflare, CloudFront. Sub-50ms. Shared across all users globally.
4. L4 — database. 1-100ms. Source of truth.

def get_user(user_id):
    # L1: per-process cache, 10s TTL
    if user_id in process_cache and process_cache[user_id].fresh:
        return process_cache[user_id].value

    # L2: Redis, 5min TTL
    cached = redis.get(f"user:{user_id}")
    if cached:
        process_cache[user_id] = Entry(cached, ttl=10)
        return cached

    # L4: DB
    user = db.query("SELECT * FROM users WHERE id = ?", user_id)
    redis.setex(f"user:{user_id}", 300, serialize(user))
    process_cache[user_id] = Entry(user, ttl=10)
    return user

def update_user(user_id, ...):
    db.execute("UPDATE users ...")
    redis.delete(f"user:{user_id}")
    # L1 still has stale data for up to 10s on each instance — accept it,
    # or use a pub/sub invalidation channel

Things that aren't indexes but matter:

The default response when a request fails: try again. Often this is right. Sometimes it makes things catastrophic.

A circuit breaker watches calls to a downstream service. If the failure rate exceeds a threshold, the breaker "trips" — subsequent calls fail fast without even attempting the downstream call. After a timeout, it tries one call ("half-open"); if that succeeds, it closes again.

Three states:

• Closed — calls go through normally.
• Open — calls fail immediately. Downstream is given time to recover.
• Half-open — let one through to test. Success → close. Failure → back to open.

from circuitbreaker import circuit

@circuit(failure_threshold=5, recovery_timeout=30, expected_exception=ServiceError)
def call_payment_service(order):
    return requests.post("https://payments/charge", ..., timeout=2)

When you can't keep up, what do you do? Three options:

• Drop — refuse the work. Returns 503 / queue full. The caller decides.
• Buffer — accept it into a queue. Hope you catch up. Risk: queue grows unbounded → memory exhaustion → crash.
• Block / slow down — make the caller wait. Slows them down to your pace.

The wrong answer: unbounded queue. The right answer depends on the system, but always with explicit limits.

queue = Queue(maxsize=10000)

def produce(event):
    try:
        queue.put_nowait(event)
    except Full:
        metrics.increment("queue.dropped")
        # explicit decision: log, alert, or send to a dead-letter queue

# Redis-based, atomic via Lua script
def check_rate_limit(user_id, max_tokens=100, refill_rate=10):
    key = f"rate:{user_id}"
    now = time.time()

    # in a single Lua call:
    # 1. read current tokens + last refill time
    # 2. add tokens based on (now - last_refill) * refill_rate, capped at max
    # 3. if tokens >= 1, decrement and return success
    # 4. else return failure

    return redis_lua_eval(key, max_tokens, refill_rate, now)

TCP gives you a reliable byte stream. UDP gives you packets that may or may not arrive. For most app-level work, TCP. For real-time stuff where stale data is worse than missing data, UDP.

To open a TCP connection: SYN → SYN-ACK → ACK. Three packets, 1.5 round-trips.

If your client is in Sydney and your server is in Virginia (~200ms RTT), opening a TCP connection costs 300ms before any data flows. Add TLS handshake (1-2 more RTTs) and you're at 500-700ms before the request goes out.

This is why connection reuse matters. HTTP/1.1 keep-alive, HTTP/2 multiplexing, connection pools — all amortizing handshake cost across many requests.

1. Client → ClientHello (supported versions, cipher suites, random nonce).
2. Server → ServerHello (chosen version + cipher), Certificate, ServerKeyExchange (ECDHE public), ServerHelloDone.
3. Client verifies cert chain. If chain valid + domain matches + not expired/revoked, proceed.
4. Client → ClientKeyExchange (its ECDHE public), ChangeCipherSpec, Finished.
5. Both sides derive shared session key from ECDHE values + nonces.
6. Server → ChangeCipherSpec, Finished.
7. Application data flows encrypted with the session key.

TLS 1.3 collapses this to 1 round-trip (sometimes 0 with session resumption).

Resolution order for api.example.com:

1. Browser cache (sub-ms)
2. OS resolver cache (sub-ms)
3. Local resolver (router, ISP) — 1-50ms
4. Recursive resolver (1.1.1.1, 8.8.8.8, your VPC's resolver) — 1-50ms
5. Authoritative resolution: root → TLD → authoritative server — 50-200ms cold

TTL on records determines how long results are cached. Set too low → constant lookups. Set too high → can't change records quickly.

A CDN is just servers close to users. The mechanics:

• You point your domain at the CDN (CNAME or anycast IP).
• User's DNS lookup returns an IP near them (geographic DNS or anycast routing).
• User's request hits the closest CDN edge node.
• If cached, edge serves directly (~10-50ms).
• If not, edge fetches from origin (your server) and caches.

What CDNs cache by default:

• Static assets (JS, CSS, images) with long TTLs.
• API responses if you set Cache-Control: public, max-age=... headers.

What they don't cache:

• Anything with cookies (by default).
• Anything with Cache-Control: private or no-store.
• POST/PUT/DELETE requests.

Cache-Control: public, max-age=31536000, immutable
# Static asset that never changes (hashed filename). Cache forever.

Cache-Control: public, max-age=60, stale-while-revalidate=600
# API response. Fresh for 60s, but if requested between 60-660s, serve stale
# while fetching a fresh copy in the background. Best of both worlds.

Cache-Control: private, max-age=300
# User-specific. Browser cache only, not shared (CDN).

Cache-Control: no-store
# Don't cache anywhere. For sensitive data.

ETag: "v3-abc123"
# Validation. Browser sends If-None-Match: "v3-abc123" on next request.
# Server compares; if match, returns 304 Not Modified (no body).

Vary: Accept-Encoding, Authorization
# Cache key includes these headers. Different responses for different users.

L7 is the right default for HTTP services. L4 for non-HTTP protocols, very high throughput, or when you need to preserve client IP without modification.

Instead of waiting for the whole page to render before sending HTML, stream it as components finish. Slow data fetches don't block fast ones from showing.

React 18's Suspense + streaming = page renders progressively. The header shows immediately; the slow widget shows a skeleton; when its data arrives, it streams in.

For users on slow connections, this dramatically improves perceived performance.

Don't ship one 5MB JS file. Split by route, lazy-load heavy components, defer third-party scripts.

• Route-based splitting — each route gets its own bundle. Next.js, Remix, React Router do this automatically.
• Dynamic import — const Chart = lazy(() => import('./Chart')) for components that aren't on the critical path.
• Tree shaking — bundler removes unused exports. Works only if you import named exports, not whole modules.
• Third-party deferral — analytics, chat widgets, etc., load after the page is interactive.

These directly affect SEO ranking. Your competitors with better Vitals beat you on search.

<!-- bad: -->
<img src="/photo.jpg">

<!-- good: -->
<img src="/photo.jpg" width="800" height="600">

<!-- or with CSS: -->
<img src="/photo.jpg">

// Manual data fetching with useState
function UserProfile({ userId }) {
  const [user, setUser] = useState(null);
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState(null);

  useEffect(() => {
    setLoading(true);
    fetch(`/api/users/${userId}`)
      .then(r => r.json())
      .then(setUser)
      .catch(setError)
      .finally(() => setLoading(false));
  }, [userId]);

  if (loading) return <Spinner />;
  if (error) return <Error />;
  return <div>{user.name}</div>;
}

Every component reimplements loading, error, race conditions (what if userId changes mid-fetch?), caching (the next component fetches the same user again), refetch on focus, optimistic updates, etc. Hundreds of lines of boilerplate.

The modern answer. You declare what data you want; the library handles caching, deduplication, retries, refetch on focus, background updates, optimistic updates.

import { useQuery } from '@tanstack/react-query';

function UserProfile({ userId }) {
  const { data: user, isLoading, error } = useQuery({
    queryKey: ['user', userId],
    queryFn: () => fetch(`/api/users/${userId}`).then(r => r.json()),
    staleTime: 5 * 60 * 1000,  // 5 minutes
  });

  if (isLoading) return <Spinner />;
  if (error) return <Error />;
  return <div>{user.name}</div>;
}

Behavior you get for free:

• Two components both calling useQuery(['user', 123]) share one network request and one cache entry.
• Returning to the tab triggers a background refetch (configurable).
• Errors retry with exponential backoff.
• The cache survives unmount; navigating back to a screen shows cached data instantly while refetching in background.
• You can manually invalidate keys after mutations.

const queryClient = useQueryClient();

const likeMutation = useMutation({
  mutationFn: (postId) => fetch(`/api/posts/${postId}/like`, { method: 'POST' }),

  onMutate: async (postId) => {
    // Cancel any in-flight refetches
    await queryClient.cancelQueries({ queryKey: ['post', postId] });

    // Snapshot the previous state
    const previous = queryClient.getQueryData(['post', postId]);

    // Optimistically update
    queryClient.setQueryData(['post', postId], (old) => ({
      ...old,
      liked: true,
      likeCount: old.likeCount + 1,
    }));

    // Return rollback context
    return { previous };
  },

  onError: (err, postId, context) => {
    // Roll back on failure
    queryClient.setQueryData(['post', postId], context.previous);
  },

  onSettled: (data, err, postId) => {
    // Refetch to get the canonical state
    queryClient.invalidateQueries({ queryKey: ['post', postId] });
  },
});

You mutated data. Now what's stale?

• Invalidate specific keys: queryClient.invalidateQueries({ queryKey: ['post', postId] }). Refetches that one query.
• Invalidate by prefix: ['posts'] invalidates all queries starting with 'posts'. Refetches lists too.
• Set query data directly: if you got the new data back from the mutation, just setQueryData. No refetch needed.
• Time-based: just rely on staleTime. Eventually all caches refresh.

React Suspense lets a component "suspend" while waiting for data. The parent renders a fallback. Combined with TanStack Query's suspense mode:

const { data: user } = useSuspenseQuery({ queryKey: ['user', userId], queryFn: ... });
// data is never undefined here. While loading, the component suspends.

function UserPage() {
  return (
    <Suspense fallback={<Spinner />}>
      <UserProfile userId={123} />
    </Suspense>
  );
}

Cleaner than checking isLoading everywhere. Lets you render multiple components in parallel that each suspend independently, and the parent's fallback shows until they all resolve.

• Server needs to push data to client (chat messages, notifications, live counters).
• Latency-sensitive bidirectional communication (multiplayer games, live cursors).
• Streaming data with multiple intermediate updates (LLM token streaming, progress updates).

• One-shot request/response — use HTTP, you'll regret WebSockets.
• Infrequent updates (every few minutes) — use polling or SSE.
• You don't need bidirectional — use Server-Sent Events (SSE), which is simpler.

SSE is one-way: server → client, over standard HTTP. Auto-reconnects. Works through proxies that hate WebSockets.

// Client
const events = new EventSource('/api/notifications');
events.onmessage = (e) => console.log(e.data);

// Server
res.writeHead(200, { 'Content-Type': 'text/event-stream' });
res.write(`data: hello\n\n`);
// keep the connection open, write more events as they happen

For LLM token streaming, push notifications, server-driven UI updates: SSE is usually the right choice. Use WebSockets only when you genuinely need client-to-server too.

WebSocket connections die: networks change (wifi to cellular), proxies time out (most cloud providers kill idle TCP connections after 60s), servers restart, mobile browsers suspend tabs.

A naïve client just sees "connection closed." A robust client handles:

1. Detection. The onclose handler fires. Sometimes onerror first.
2. Backoff reconnection. Don't hammer reconnect. Exponential backoff with jitter.
3. State recovery. What did we miss while disconnected?
4. UI feedback. Show "reconnecting..." so users know.

class ResilientWebSocket {
  constructor(url) {
    this.url = url;
    this.attempts = 0;
    this.lastEventId = null;
    this.connect();
  }

  connect() {
    this.ws = new WebSocket(this.url + (this.lastEventId ? `?since=${this.lastEventId}` : ''));

    this.ws.onopen = () => {
      this.attempts = 0;  // reset backoff
      this.onStatus('connected');
    };

    this.ws.onmessage = (e) => {
      const msg = JSON.parse(e.data);
      this.lastEventId = msg.id;
      this.onMessage(msg);
    };

    this.ws.onclose = () => {
      this.onStatus('reconnecting');
      const delay = Math.min(30000, 1000 * Math.pow(2, this.attempts)) * (0.5 + Math.random());
      this.attempts++;
      setTimeout(() => this.connect(), delay);
    };

    this.ws.onerror = () => {
      // onclose will fire after; let it handle reconnection
    };
  }

  send(msg) {
    if (this.ws.readyState === WebSocket.OPEN) {
      this.ws.send(JSON.stringify(msg));
    } else {
      // queue or fail
    }
  }
}

HTTP scales horizontally trivially because every request is independent. WebSockets don't, because the connection is stateful and pinned to one server.

Problem: User A is connected to server-1. User B is connected to server-2. User A sends a message to user B. How does server-1 tell server-2?

The standard mental model:

• Unit tests — test one function/class in isolation. Mock dependencies. Hundreds, run in seconds.
• Integration tests — test multiple components together (real DB, real Redis, possibly mocked third parties). Tens, run in tens of seconds.
• End-to-end tests — test the full system through the UI or public API. Few, run in minutes.

The pyramid is upside-down (an "ice cream cone" anti-pattern) when teams have lots of slow E2E tests and few unit tests. Easy to write, hard to maintain, slow to run, flaky.

• Pure functions with non-trivial logic (parsers, calculators, transformers).
• Business logic with edge cases (pricing rules, auth checks, validation).
• Things that are hard to debug from production (date math, currency math, retry logic).

• Trivial getters/setters (no logic = no test).
• Framework code (don't test that React renders).
• Code where the test is just a copy of the implementation.

Most production bugs aren't in pure logic; they're in how components fit together. The DB query that works in isolation but deadlocks under concurrency. The cache that returns stale data because invalidation has a bug. Integration tests catch these.

# pytest fixture: real Postgres in Docker, fresh schema per test
import pytest
import asyncpg
import asyncio

@pytest.fixture(scope="session")
async def db_pool():
    # spin up a postgres container or use testcontainers-python
    pool = await asyncpg.create_pool("postgres://test:test@localhost:5433/test")
    await run_migrations(pool)
    yield pool
    await pool.close()

@pytest.fixture
async def db(db_pool):
    # transaction-per-test pattern: rollback at the end
    async with db_pool.acquire() as conn:
        async with conn.transaction():
            yield conn
            raise asyncpg.PostgresError("__rollback__")  # forces rollback

# usage
async def test_user_creation(db):
    user = await create_user(db, email="a@b.com")
    assert user.id is not None
    fetched = await get_user(db, user.id)
    assert fetched.email == "a@b.com"
    # transaction rolls back; next test sees clean DB

You have Service A that calls Service B. Service B's team changes the response format. Service A breaks in production. Whose fault?

Contract testing solves this: A and B agree on a contract (typically OpenAPI / Pact). Tests on both sides verify they meet the contract. If B's team breaks the contract, B's tests fail before merge.

• Consumer-driven contracts (Pact) — A writes a contract describing what it expects. B verifies it produces what A expects. The consumer drives the schema.
• Schema-first (OpenAPI, Protobuf) — both sides codegen from a shared schema. Schema changes go through review.

You don't know if your system handles 1000 req/sec until you've tried. Load testing in non-production:

• k6 — script in JS, scales to high concurrency, good output.
• Locust — Python-based, distributed, web UI.
• Gatling — Scala-based, very high throughput.
• Artillery — YAML config, simple cases.

// k6 script
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },   // ramp up to 100 users
    { duration: '5m', target: 100 },   // stay at 100
    { duration: '2m', target: 1000 },  // spike to 1000
    { duration: '5m', target: 1000 },  // stay
    { duration: '3m', target: 0 },     // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],   // 95% of requests under 500ms
    http_req_failed: ['rate<0.01'],     // less than 1% failures
  },
};

export default function () {
  const res = http.get('https://staging.api.example.com/products');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'has products': (r) => JSON.parse(r.body).length > 0,
  });
  sleep(1);
}

Deliberately break things in production (or production-like environments) to verify your system handles failures gracefully. Made famous by Netflix's Chaos Monkey.

• Random instance termination — kill a server. Does the load balancer route around it?
• Network partitions — cut a service off from the DB. Does it fail gracefully?
• Latency injection — make a service take 5s to respond. Do timeouts fire correctly?
• Resource exhaustion — fill up disk, max out CPU. What breaks?

Tools: AWS Fault Injection Simulator, Chaos Mesh, Gremlin, Litmus.

Most teams aren't ready for production chaos engineering. Start in staging, prove your runbooks work, then graduate to game-day exercises in production.

• Database column on user — simplest, but requires DB query per check.
• Redis — fast, allows real-time updates.
• In-process cache + periodic refresh — fastest. Refresh every 30s from a backend store. Slight delay on flag changes but no per-request cost.
• External SaaS — LaunchDarkly, Statsig, Unleash. SDKs handle caching and updates. Real-time push.

Server-side flags can be changed instantly and affect any caller. The client can't lie about the flag state.

Client-side flags (in browser/mobile) ship the flag value to the client. Faster (no round trip per check), but:

• Client can override or inspect the flag (don't put security-sensitive features behind client flags).
• Updates propagate slowly (next page load or SDK refresh).

For most user-facing features, server-side. For UI variations (button color, layout), client-side is fine.

Replace instances one at a time. Default in Kubernetes (the RollingUpdate strategy).

Pros: simple, no extra resources, gradual rollout.

Cons: rollback means another rolling deploy (slow). During the deploy, you're running two versions simultaneously — clients may hit either. Backwards-compatibility required.

Two complete environments: blue (current) and green (new). Deploy v2 to green, test, then flip traffic. Old version stays around for instant rollback.

Pros: instant rollback (just flip traffic back). Test the new version with real production conditions before cutover.

Cons: 2x resources during deploy. DB migrations are tricky (both versions hit the same DB). The cutover is sudden — all traffic flips at once.

Like blue-green but gradual. Send 1% of traffic to v2, watch metrics, ramp up.

Pros: catches problems before full rollout. Small blast radius if something's broken. Real production traffic and data.

Cons: requires sophisticated traffic management (service mesh, smart load balancer). Need good metrics to make ramp-up decisions automatically.

baseline (v1): error_rate=0.5%, p99=120ms
canary (v2):   error_rate=0.6%, p99=125ms
=> difference within tolerance. Promote.

baseline (v1): error_rate=0.5%, p99=120ms
canary (v2):   error_rate=2.1%, p99=180ms
=> significant regression. Roll back.

# at the load balancer / service mesh level
hash = sha256(user_id) % 100
if hash < canary_percentage:
    route to v2
else:
    route to v1

Send copies of production traffic to v2 without using its responses. Compare v2's behavior to v1's.

request → v1 → response (returned to client)
              ↓
              also sent to v2 → response (compared, not used)

Used for high-risk migrations: replacing a search algorithm, switching DBs, rewriting a service. You can verify v2 produces the same outputs (or measure where it differs) before any user sees it.

Heavy on infrastructure (you're running 2x compute) but invaluable for high-stakes changes.

The single most common cause of bad deploys. The discipline:

• Never deploy schema and code together. Migrations go first, code goes second.
• Migrations are forward-compatible with old code. If you can't roll back the code, you can't roll forward the migration.
• No locking migrations on big tables. ALTER TABLE ... ADD COLUMN with a default = full table rewrite = locked for minutes. Use ADD COLUMN ... NULL first, then backfill, then add the default.
• Add indexes concurrently. CREATE INDEX CONCURRENTLY doesn't lock writes.
• Drop columns last. Stop using the column in code → deploy → wait → drop column.