diff --git a/AGENTS.md b/AGENTS.md index fcd7931..c2801ac 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,7 +51,7 @@ What experienced Go engineers forget before an interview, grouped by subtopic. - **Bold** the term or number worth memorizing — at most 3 per concept, not counting tables. - **Versions**: note when behavior changed ("since Go 1.22", "Python 3.14+", "Pectra, May 2025"). - **Cross-topic**: the owner file covers a concept in full; another file may overlap briefly from its own angle and links to the owner: `see [security.md](security.md)` (rules in Topic ownership). Link to files, not anchors (the strict build validates anchors; only use one if you've checked the slug). -- Suggested size: 5–10 sections, 3–8 concepts each. Shorter is better if nothing is lost. +- Suggested size: 5–20 sections, 3–8 concepts each. Shorter is better if nothing is lost. ### What to include @@ -78,15 +78,15 @@ Each concept has one owner file that covers it in full. Other files may overlap | Concept | Owner | Referenced from | |---|---|---| -| Consistency models, replication theory, consensus, clocks, delivery semantics, sagas, CRDTs | distributed | database, system, backend | +| Consistency models, replication theory, consensus, clocks (monotonic vs wall), delivery semantics, sagas, CRDTs, CDC | distributed | database, system, backend, os, golang | | Retries, backoff, circuit breakers, backpressure, tail latency | distributed | backend, system | | Isolation levels, MVCC, indexes, WAL, DB replication setup, pooling | database | backend, system | -| Caching strategies, rate-limiting algorithms, load balancing, CDN, observability (SLI/SLO) | system | backend, devops | -| REST/gRPC/GraphQL, pagination, idempotency keys, outbox, graceful shutdown, N+1 | backend | system, database | +| Caching strategies, rate-limiting algorithms, load balancing, CDN, observability (SLI/SLO), capacity math (Little's law), multi-region and DR tiers | system | backend, devops, os, aws | +| REST/gRPC/GraphQL, pagination, idempotency keys, outbox, graceful shutdown, N+1, schema evolution, multi-tenancy, time handling | backend | system, database, distributed | | AuthN, sessions, OAuth/OIDC, JWT, XSS, CSRF, CORS, clickjacking, crypto, TLS | security | backend, frontend | | Containers, Kubernetes, cloud, IaC, deployment strategies, zero-downtime migrations | devops | backend, database | | Browser event loop, JS language semantics | javascript | frontend, typescript | -| Rendering strategies, browser pipeline, Web Vitals, HTTP caching in the browser, React | frontend | javascript | +| Rendering strategies, browser pipeline, Web Vitals, HTTP caching in the browser, browser storage, React | frontend | javascript, security | | asyncio event loop | python | — | | TCP/UDP, HTTP versions, QUIC, DNS, IP addressing, NAT, MTU, BGP/anycast | networking | backend, system, devops | | Processes, signals, scheduling, virtual memory, OOM, file descriptors, epoll/io_uring, fsync, locks | os | devops, database, golang | diff --git a/docs/topics/ai.md b/docs/topics/ai.md index 8765391..2bec062 100644 --- a/docs/topics/ai.md +++ b/docs/topics/ai.md @@ -68,6 +68,13 @@ What experienced AI engineers forget before an interview, grouped by subtopic. F - A small **draft model** proposes several tokens; the large model verifies them in one pass and keeps the accepted prefix. - Faster decode with the **same output distribution**; the gain depends on how often drafts are accepted. +### Model parallelism and GPU memory + +- Serving memory ≈ weights + KV cache + activations; a model too large for one GPU must be split. +- **Tensor parallelism** splits each layer's matrices across GPUs; it needs a fast interconnect (NVLink), so it stays within a node. +- **Pipeline parallelism** puts consecutive layers on different GPUs or nodes: less communication, but pipeline bubbles leave stages idle. +- Data parallelism replicates the whole model for throughput; **disaggregated serving** runs prefill and decode on separate GPU pools, each tuned for its bottleneck. + ### Quantization - Weights in FP16 take **2 bytes per parameter** (a 70B model ≈ 140 GB); INT8 halves that, **4-bit** (GPTQ, AWQ) quarters it, with some quality loss. @@ -95,6 +102,8 @@ What experienced AI engineers forget before an interview, grouped by subtopic. F | **IVF** | cluster, then probe `nprobe` clusters | less memory; recall depends on `nprobe` | | **PQ** | compress vectors into codes | much smaller, lower recall; often combined as IVF-PQ | +- **Filtered search**: post-filtering the top-k returns too few hits when the filter is selective, and pre-filtering breaks HNSW graph connectivity — use a filter-aware engine or partition the index (per tenant). + ### Chunking - Chunk along **document structure** (sections, paragraphs); a few hundred tokens with **10–20% overlap** is a common starting point — tune on evals. @@ -118,7 +127,13 @@ What experienced AI engineers forget before an interview, grouped by subtopic. F | RAG | facts are private or change often; citations matter | | Fine-tuning | a stable style, format, or decision gap persists after prompting; poor for keeping facts current | -## Fine-tuning +## Training and fine-tuning + +### Training pipeline + +- **Pretraining**: next-token prediction on trillions of tokens; Chinchilla-optimal is ≈ **20 tokens per parameter**, but modern models train far past it so a smaller model is cheaper to serve. +- Post-training: SFT on demonstrations, then preference tuning (RLHF, DPO) for helpfulness and safety. +- **RL with verifiable rewards** (math and code graded automatically, e.g. GRPO) is how reasoning models learn long chains of thought. ### SFT, RLHF, DPO @@ -147,8 +162,8 @@ What experienced AI engineers forget before an interview, grouped by subtopic. F ### Stop conditions -- Cap steps, tokens, and cost per run; require evidence (tests pass, a record exists) before declaring success; escalate on repeated failure. -- Open-ended self-critique loops often repeat the same mistake and burn budget. +- Cap steps, tokens, and cost per run; escalate to a human on repeated failure. +- Require evidence (tests pass, a record exists) before declaring success. ### Context management @@ -167,11 +182,6 @@ What experienced AI engineers forget before an interview, grouped by subtopic. F - **MCP** standardizes how apps discover and call tools, read resources, and fetch prompts — JSON-RPC 2.0 over stdio (local) or **Streamable HTTP** (remote, replaced HTTP+SSE in 2025). - Authorization is optional; HTTP servers that support it use **OAuth 2.1**. MCP doesn't replace per-action authorization or business validation. -### Browser automation as a fallback - -- Use it only without an API — layouts change and clicks have ambiguous effects. -- Isolate its credentials and require **approval** before irreversible submits. - ## Evaluation ### Eval sets and metrics @@ -200,7 +210,6 @@ What experienced AI engineers forget before an interview, grouped by subtopic. F - **Route** easy requests to a small, fast model and hard ones to a strong model. - **Semantic caching** reuses answers to similar queries; its key must include user permissions, source version, and freshness, or it leaks. -- Measure cost **per completed task**, not per call. ### Batch and streaming @@ -221,5 +230,5 @@ What experienced AI engineers forget before an interview, grouped by subtopic. F ### Human approval gates -- Require approval before high-impact actions (money, legal, external sends); show the action, evidence, and affected records. +- Require approval before high-impact actions (money, legal, external sends, irreversible form submits by a browser agent); show the action, evidence, and affected records. - An approval covers **one specific action**; re-check that data hasn't changed since the proposal, and record who approved what and when. diff --git a/docs/topics/algorithms.md b/docs/topics/algorithms.md index 9624560..519a118 100644 --- a/docs/topics/algorithms.md +++ b/docs/topics/algorithms.md @@ -6,10 +6,23 @@ What experienced engineers forget before an algorithms interview, grouped by sub ### Time and space bounds -- State the **input size**, then worst-case time and **auxiliary space**; producing an output of size `k` already costs Ω(k). +- Producing an output of size `k` already costs **Ω(k)**, whatever the algorithm. - Nested loops aren't automatically O(n²): if two pointers each advance at most `n` times, total work is O(n). - Recursion stack counts as space: balanced divide-and-conquer uses O(log n) frames, a linear chain O(n). +### Input size and target complexity + +At ~10⁸ simple operations per second, the constraints tell you which complexity to aim for: + +| n up to | Target | Typical technique | +|---|---|---| +| 10–11 | O(n!) | permutations | +| 20 | O(2ⁿ) | subsets, bitmask DP | +| 500 | O(n³) | interval DP, Floyd–Warshall | +| 5,000 | O(n²) | pairwise DP | +| 10⁶ | O(n log n) | sorting, heaps, binary search | +| beyond | O(n) or O(log n) | linear scan, math | + ### Amortized analysis - A dynamic array's occasional **O(n) resize** still gives **O(1) amortized** append, because capacity grows geometrically. @@ -17,8 +30,8 @@ What experienced engineers forget before an algorithms interview, grouped by sub ### Recurrence patterns -- **Binary search**: `T(n) = T(n/2) + O(1) = O(log n)`. -- **Merge sort**: `T(n) = 2T(n/2) + O(n) = O(n log n)`. +- **Master theorem** for `T(n) = aT(n/b) + f(n)`: compare `f(n)` with `n^(log_b a)` — smaller → Θ(n^(log_b a)); equal → Θ(n^(log_b a) · log n); larger → Θ(f(n)). +- Merge sort (a = b = 2, f = n) is the equal case → O(n log n); binary search (a = 1, b = 2, f = 1) → O(log n). - Recursion without memoization can revisit states exponentially often; after memoizing, cost = **states × work per state**. ## Arrays and sequences @@ -50,6 +63,12 @@ What experienced engineers forget before an algorithms interview, grouped by sub - **Sliding window maximum**: keep indices with decreasing values; pop smaller values from the back, drop the front once it leaves the window. - Every index enters and leaves once — **O(n)** total instead of O(nk). +### Maximum subarray (Kadane) + +- Scan once: `cur = max(x, cur + x)`, `best = max(best, cur)` — **O(n)** time, O(1) space. +- Initialize both with the first element, not 0, so an all-negative array returns its largest value. +- **Circular** variant: `max(best, total − minimum subarray)`, unless every value is negative (then just `best`). + ### Intervals - **Merge overlapping intervals**: sort by start, extend the last merged interval while the next start ≤ its end — O(n log n). @@ -72,7 +91,7 @@ What experienced engineers forget before an algorithms interview, grouped by sub - A **polynomial rolling hash** updates a window's hash in O(1) as it slides, so search is O(n + m) expected. - Collisions happen: **verify each match** for exact results — two moduli only make collisions rarer; also finds duplicate substrings via binary search on length. -## Sorting and selection +## Sorting, selection, and sampling ### Comparison sorts @@ -94,6 +113,12 @@ Standard array implementations; variants differ in stability and space. - **Quickselect** partitions around a random pivot and recurses into one side: O(n) expected, O(n²) worst. - A size-`k` **min-heap** keeps the largest `k` of `n` items in O(n log k) time and O(k) space. +### Random sampling and shuffling + +- **Reservoir sampling** picks `k` items from a stream of unknown length: keep the first `k`, then replace a random slot with the i-th item with probability k/i — one pass, O(k) memory. +- **Fisher–Yates** shuffle: for `i` from `n−1` down to 1, swap `a[i]` with `a[rand(0..i)]` — O(n), every permutation equally likely. +- Swapping with `a[rand(0..n−1)]` at each step looks similar but is **biased** (nⁿ outcomes can't split evenly over n! permutations). + ## Data structures ### Heap invariants @@ -101,6 +126,12 @@ Standard array implementations; variants differ in stability and space. - A binary heap is a **complete tree** in an array; the root is the min or max, siblings are unordered. - Peek O(1); insert and remove-root O(log n); **bottom-up construction is O(n)**. +### Heap patterns + +- **k-way merge**: push the head of each of `k` sorted lists into a min-heap, pop and push the next from the same list — O(n log k). +- Running median with **two heaps**: a max-heap for the lower half, a min-heap for the upper, sizes differing by at most one — O(log n) per insert, O(1) median. +- Heaps can't delete arbitrary items cheaply: **lazy deletion** marks items removed and discards them when they reach the top. + ### Hash tables and collisions - Expected O(1) lookup and update, no ordering; **collisions** are resolved by chaining or probing, and the table rehashes past a load threshold. @@ -118,8 +149,8 @@ Standard array implementations; variants differ in stability and space. ### LRU cache -- **Hash map + doubly linked list**: the map finds a node in O(1), the list keeps recency — move to front on access, evict from the tail. -- Python: `OrderedDict.move_to_end` + `popitem(last=False)`; Java: `LinkedHashMap` in access order. +- **Hash map + doubly linked list**: the map finds a node in O(1), the list keeps recency. +- Move a node to the front on every access and evict from the tail — both O(1). ### Fenwick and segment trees diff --git a/docs/topics/aws.md b/docs/topics/aws.md index 00b6184..c95b9a4 100644 --- a/docs/topics/aws.md +++ b/docs/topics/aws.md @@ -15,9 +15,16 @@ What experienced AWS developers and DevOps engineers forget before an interview, - Use **temporary role credentials** for EC2, ECS tasks, and Lambda; use OIDC federation for external CI so pipelines do not store long-lived AWS keys. - In ECS, the **task role** grants the application access to AWS APIs; the **execution role** lets the ECS agent pull images, fetch configured secrets, and send logs. +### Cross-account access + +- Within one account, an allow in the identity policy **or** the resource policy suffices (KMS key policies and role trust policies are exceptions). +- Across accounts, **both** sides must allow: the resource policy trusts the other account or principal, and that principal's identity policy grants the action. +- Third parties assume your role with an **`sts:ExternalId`** condition, preventing the confused-deputy problem. +- `aws:PrincipalOrgID` in a resource policy limits access to accounts in your organization. + ### KMS and envelope encryption -- **Envelope encryption**: `GenerateDataKey` returns a plaintext data key (encrypt locally, then discard) and the same key encrypted under the KMS key (store it with the data). KMS `Encrypt` itself only takes **4 KB**. +- `GenerateDataKey` returns a data key in plaintext (use it, then discard it) and wrapped under the KMS key (store it with the data) — the envelope pattern in [security.md](security.md); KMS `Encrypt` itself takes at most **4 KB**. - Every KMS key has a **key policy**; IAM policies grant access only if the key policy delegates to the account. ## Networking @@ -32,6 +39,19 @@ What experienced AWS developers and DevOps engineers forget before an interview, - **Security groups** are stateful and attach to network interfaces; **network ACLs** are stateless and apply at subnet boundaries, so return traffic needs an explicit ACL rule. - **Gateway VPC endpoints** route S3 and DynamoDB traffic without a NAT device or internet gateway; interface endpoints use private IPs and AWS PrivateLink for supported services. +### ALB vs NLB + +| | ALB | NLB | +|---|---|---| +| Layer | L7: HTTP(S), gRPC, WebSockets | L4: TCP, UDP, TLS | +| Routing | host, path, header, query string | port only | +| Addresses | changing IPs; use the DNS name | **static IP per AZ** (Elastic IPs optional) | +| Client IP | in `X-Forwarded-For` | preserved, or sent via PROXY protocol v2 | +| Extras | OIDC/Cognito auth, WAF, Lambda targets | TLS passthrough, PrivateLink services, very high throughput | +| Cross-zone | on by default | **off** by default; cross-AZ data is charged when on | + +Gateway Load Balancer inserts inline appliances such as third-party firewalls. + ### Connecting VPCs | Option | Topology | Note | @@ -59,7 +79,6 @@ What experienced AWS developers and DevOps engineers forget before an interview, ### Lambda limits and concurrency - Maximum timeout **15 minutes**; synchronous payloads 6 MB each way (streamed responses allow more). -- Lambda Managed Instances allow up to **90 minutes** for async and event-source invocations (Sept 2026); synchronous calls stay at 15. - **Reserved concurrency** guarantees capacity and caps a function; provisioned concurrency pre-initializes environments to reduce startup latency and incurs separate charges. ### Lambda cold starts and scaling @@ -74,11 +93,6 @@ What experienced AWS developers and DevOps engineers forget before an interview, - **HTTP API**: cheaper, JWT/Lambda/IAM authorizers and throttling, but no usage plans or request validation; integration timeout max **30 s**. - ALB in front of Lambda or containers can authenticate via OIDC/Cognito but has no throttling, usage plans, or validation; LCU pricing is usually cheaper at sustained high volume. -### ECS service mechanics - -- A **task definition** specifies containers and resources; a **service** keeps the desired number of tasks running and replaces failed ones. -- Services register tasks with a load balancer target group and roll out new task definitions with rolling or blue/green deployments. - ## Storage and databases ### S3 consistency and versions @@ -87,6 +101,12 @@ What experienced AWS developers and DevOps engineers forget before an interview, - With **versioning**, deleting an object without a version ID adds a delete marker; deleting a specific version permanently removes it. Lifecycle rules can expire noncurrent versions. - A **presigned URL** grants temporary access using the signer's permissions and expires no later than the credentials used to sign it. +### S3 access and CloudFront + +- New buckets block public access and disable ACLs (**bucket owner enforced**) by default since April 2023; objects are encrypted with SSE-S3 by default since January 2023. +- Serve a private bucket through CloudFront with **Origin Access Control**, which replaces the legacy OAI; the bucket policy allows only that distribution. +- CloudFront **signed URLs or signed cookies** gate private content at the edge; S3 presigned URLs bypass the CDN. + ### S3 performance and classes - Each **prefix** supports about **3,500 writes and 5,500 reads per second**; spread hot keys over prefixes. @@ -147,5 +167,11 @@ What experienced AWS developers and DevOps engineers forget before an interview, ### Capacity and cost guardrails - Service **quotas** are per account and Region and often stop scaling before the application does — many are soft and raised by request. -- AWS Budgets alerts follow billing-data updates, at least daily; use CloudWatch service metrics for operational alarms. Cost Explorer is for spend analysis. - EC2 **Spot** capacity can be reclaimed; stop/terminate interruption notices give about **2 minutes** on a best-effort basis (hibernation starts immediately). Checkpoint work and replace capacity proactively. + +### Cost drivers + +- **NAT gateway** data processing (~$0.045/GB in us-east-1, plus hourly) is a classic surprise; S3 and DynamoDB gateway endpoints are free and skip it. +- **Cross-AZ traffic** costs ~$0.01/GB in each direction, so chatty services and replicas spread across AZs add up. +- Internet egress and CloudWatch Logs ingestion (~$0.50/GB) are the other usual suspects. +- Discounts: Savings Plans and Reserved Instances (up to ~72% for 1–3 year commitments), Spot (up to ~90%), Graviton (roughly 20% cheaper per hour than comparable x86). diff --git a/docs/topics/backend.md b/docs/topics/backend.md index 2a198b8..4ccf095 100644 --- a/docs/topics/backend.md +++ b/docs/topics/backend.md @@ -4,11 +4,6 @@ What experienced backend engineers forget before an interview, grouped by subtop ## API design -### Resource modeling - -- Resources are nouns and HTTP methods are verbs: `POST /orders/42/items`, not `/addItemToOrder`. -- Non-CRUD actions become sub-resources or state changes: `POST /orders/42/cancellation`. - ### Status codes people confuse | Codes | Meaning | @@ -88,6 +83,13 @@ What experienced backend engineers forget before an interview, grouped by subtop - **Propagate the deadline** (gRPC does it natively) so downstream services stop work once the original caller has given up. - Retries, backoff, and circuit breakers: see [distributed.md](distributed.md). +### Schema evolution + +- **Backward compatible**: new readers read old data; **forward compatible**: old readers read new data — rolling deploys and replayed events need both. +- Protobuf: never reuse or renumber a field tag (mark removed ones `reserved`); adding fields is safe; proto3 has no `required`; unknown fields survive a round trip. +- Avro with a schema registry checks BACKWARD, FORWARD, or FULL compatibility on every schema change. +- JSON consumers must ignore unknown fields (**tolerant reader**); rename a field by adding the new one, writing both, moving readers over, then removing the old. + ### Webhooks - **Sign** each payload (HMAC over timestamp + body) and have receivers verify it with a **constant-time** compare; reject stale timestamps to stop replays. @@ -96,11 +98,6 @@ What experienced backend engineers forget before an interview, grouped by subtop ## Async work -### Queue vs direct call - -- Direct call when the caller needs the result now and the dependency is fast and available. -- Queue when work is slow, the downstream is flaky or rate-limited (**load leveling**), or several consumers need the event. - ### Long-running operations - Return **202 Accepted** with a `Location` to a status resource (`/operations/123`); the client polls it or gets a webhook when done. @@ -133,12 +130,25 @@ What experienced backend engineers forget before an interview, grouped by subtop - Never make a network call inside an open DB transaction — it holds **locks and a connection** for the whole round trip. - For "commit, then notify another service", use the outbox instead of calling inside the transaction. +### Multi-tenancy models + +- Shared tables with a `tenant_id`: cheapest, but every query must filter — enforce it with PostgreSQL **row-level security** (the table owner bypasses it unless `FORCE ROW LEVEL SECURITY`). +- Schema per tenant: more isolation, but migrations and connection pools multiply with the tenant count. +- Database per tenant: strongest isolation, per-tenant restore and data residency, the most operational work. +- Move **noisy neighbors** (the largest tenants) to their own shard or database, and rate-limit per tenant. + +### Time handling + +- Store and send instants in **UTC** (RFC 3339 with an offset, `2026-03-08T09:30:00Z`); in PostgreSQL use `timestamptz`. +- For future local-time events (meetings, schedules), store the local time plus an **IANA zone** (`Europe/Berlin`), not an offset — DST rules and offsets change. +- A DST switch skips or repeats a local hour, so a job scheduled at 02:30 local time runs never or twice; schedule jobs in UTC. + ### File uploads -- Large files go straight to object storage with a **presigned URL**, bypassing app servers; the app stores only metadata. +- Clients upload straight to object storage with a presigned URL, bypassing app servers ([system.md](system.md)). - **Multipart/resumable uploads** retry individual parts instead of the whole file. -## Lifecycle and config +## Service lifecycle ### Graceful shutdown @@ -150,11 +160,6 @@ What experienced backend engineers forget before an interview, grouped by subtop - Separate endpoints: **liveness** checks only that the process can make progress; **readiness** checks what serving needs (warm-up done, required dependencies) and fails while draining. - Keep them cheap and on an internal port; probe configuration and pitfalls: see [devops.md](devops.md). -### Configuration and secrets - -- Config that varies by environment comes from the **environment**, so one immutable artifact promotes from staging to production. -- Secrets come from a secret manager at runtime, never from the repo; **validate config at startup** and fail fast. - ## Backend security ### Mass assignment diff --git a/docs/topics/database.md b/docs/topics/database.md index 9056be4..ec27266 100644 --- a/docs/topics/database.md +++ b/docs/topics/database.md @@ -60,10 +60,11 @@ What experienced database engineers forget before an interview, grouped by subto - **Default: Read Committed**; Repeatable Read is **snapshot isolation** — no phantoms, and a concurrent update of the same row aborts (no lost update), but write skew is possible. - Serializable is **SSI**: it prevents write skew by aborting with a serialization error, so the app must retry. -### MySQL InnoDB isolation +### InnoDB isolation and MVCC - **Default: Repeatable Read** with **next-key locks** (record + gap) on locking reads, blocking many phantoms. - Plain `SELECT` reads the snapshot, but `UPDATE` sees the latest committed row — lost updates slip through without `SELECT ... FOR UPDATE`. +- Updates happen in place; older versions are rebuilt from the **undo log**, and a long transaction makes undo history grow. ## MVCC and locking @@ -74,10 +75,6 @@ What experienced database engineers forget before an interview, grouped by subto - **XID wraparound** (32-bit transaction IDs) forces aggressive anti-wraparound vacuuming. - HOT updates skip index maintenance when no indexed column changed and the page has room. -### InnoDB MVCC - -- Updates happen in place; older versions are rebuilt from the **undo log**, and a long transaction makes undo history grow. - ### Row locks, queue tables, advisory locks - `SELECT ... FOR UPDATE` locks the rows it returns; **`SKIP LOCKED`** lets workers claim different rows of a queue table without blocking. @@ -128,9 +125,14 @@ What experienced database engineers forget before an interview, grouped by subto - **`FROM/JOIN → WHERE → GROUP BY → HAVING → SELECT → DISTINCT → ORDER BY → LIMIT`** — why a `SELECT` alias works in `ORDER BY` but not in `WHERE`. -### Reading EXPLAIN ANALYZE +### Why an index isn't used -- Look for sequential scans on big tables, **estimated vs actual rows** far apart (stale statistics — run `ANALYZE`), and sorts or hashes spilling to disk. +- Low selectivity: when a filter matches a large share of rows, the planner prefers a **sequential scan**, since one sequential pass beats many random heap reads. +- `LIKE 'x%'` uses a B-tree only with C collation or `text_pattern_ops`; `LIKE '%x'` needs a trigram GIN index (`pg_trgm`). +- A function or implicit cast on the column, and `OR` across columns without an index on each, also defeat it. +- **Stale statistics** show up in `EXPLAIN ANALYZE` as estimated vs actual rows far apart — run `ANALYZE`; also watch for sorts or hashes spilling to disk. +- Scan types: Seq Scan, Index Scan, Index Only Scan, and **Bitmap Heap Scan** (collects matches from one or more indexes, then reads pages in physical order). +- `OFFSET` still reads and discards every skipped row, so deep pages get slower; keyset pagination seeks by index — API side in [backend.md](backend.md). ### Join algorithms @@ -140,10 +142,6 @@ What experienced database engineers forget before an interview, grouped by subto | **Hash join** | large equality joins without a useful index; builds on the smaller side | | **Merge join** | both inputs already sorted (e.g. from indexes) | -### Deep pagination cost - -- **OFFSET** still reads and discards every skipped row, so deep pages get slower; **keyset** pagination seeks by index — API side in [backend.md](backend.md). - ## SQL gotchas ### NULL and three-valued logic @@ -163,6 +161,12 @@ What experienced database engineers forget before an interview, grouped by subto - **Since PostgreSQL 12**, a side-effect-free CTE referenced **once** is inlined into the outer query; one referenced several times is materialized. - `MATERIALIZED` / `NOT MATERIALIZED` override the default. +### Join pitfalls + +- A condition on the right table in **`WHERE`** discards the NULL-extended rows and turns a `LEFT JOIN` into an inner join; put it in `ON`. +- Joining a parent to **two one-to-many children** multiplies rows (fan-out) and inflates `SUM`/`COUNT`; aggregate each child in a subquery first. +- `EXISTS` / `NOT EXISTS` are semi- and anti-joins: they never duplicate rows, unlike `JOIN` + `DISTINCT`. + ### Upsert - **`INSERT ... ON CONFLICT (...) DO UPDATE`** (PostgreSQL) / `ON DUPLICATE KEY UPDATE` (MySQL) replaces a racy check-then-write. @@ -175,11 +179,24 @@ What experienced database engineers forget before an interview, grouped by subto - Structures: strings, hashes, lists, sets, **sorted sets** (skip list + hash — leaderboards, rate limiters), streams, HyperLogLog. - Redis Cluster splits keys into **16,384 hash slots**; multi-key operations need the same slot, forced with hash tags (`{user42}:cart`). -### Redis persistence and licensing +### Redis persistence and replication - RDB: periodic fork + snapshot (copy-on-write), loses writes since the last one. AOF: logs every write; `appendfsync everysec` (default) loses up to **~1 s**. - Replication is **asynchronous** — failover can drop acknowledged writes. -- The 2024 license change led to the **Valkey** fork (Linux Foundation); Redis 8 (2025) added AGPLv3 as an option. + +### Redis eviction, expiry, and messaging + +- `maxmemory-policy` defaults to **`noeviction`**: at the memory limit, writes fail with an OOM error. +- Caches use `allkeys-lru` or `allkeys-lfu`; `volatile-*` policies evict only keys with a TTL, so without TTLs they behave like `noeviction`. +- Expired keys are removed **lazily** on access plus by background sampling, so memory lags behind expirations. +- Pub/sub is fire-and-forget (offline subscribers miss messages); Streams persist entries, with consumer groups, acknowledgments, and redelivery of pending entries. + +### Document stores (MongoDB) + +- **Embed** data that is read together (one read, atomic update); reference unbounded or shared data — a document is capped at **16 MB**. +- Single-document writes are atomic; multi-document transactions (replica sets since 4.0, sharded clusters since 4.2) cost more, so design for single-document atomicity. +- Write concern **`w: "majority"`** is the default since 5.0; `w: 1` acknowledges before replication, so a failover can roll back acknowledged writes. +- A query without a matching index scans the collection (`COLLSCAN`); order compound index keys by the ESR rule: Equality, Sort, Range. ### Wide-column key design (Cassandra, DynamoDB) @@ -232,9 +249,5 @@ What experienced database engineers forget before an interview, grouped by subto ### OLTP vs OLAP -- **OLTP**: row storage, many small transactional reads and writes. - **OLAP**: **column storage**, compression, vectorized scans of few columns over many rows, star schemas (facts + dimensions). - -### Materialized views - -- Store a query result for fast reads that are stale until refresh; `REFRESH MATERIALIZED VIEW CONCURRENTLY` avoids blocking reads but needs a unique index. +- Materialized views store a query result for fast reads that stay stale until refresh; `REFRESH MATERIALIZED VIEW CONCURRENTLY` avoids blocking reads but needs a unique index. diff --git a/docs/topics/devops.md b/docs/topics/devops.md index efbe81a..a75e3f0 100644 --- a/docs/topics/devops.md +++ b/docs/topics/devops.md @@ -30,6 +30,13 @@ What experienced DevOps engineers forget before an interview, grouped by subtopi - A destructive migration (dropping a column old code still reads) makes rollback impossible. - Keep each DDL short and lock-safe (`lock_timeout`, concurrent index builds) — see [database.md](database.md). +### Incident response + +- Assign roles: an **incident commander** coordinates while others handle communications and operations. +- **Mitigate first** (roll back, flip a flag, fail over); find the root cause later. +- Write a blameless postmortem: timeline, contributing factors, and action items with owners. +- Track time to detect and time to recover (MTTD, MTTR); link a runbook from every alert. + ## Containers ### How containers work @@ -48,31 +55,25 @@ What experienced DevOps engineers forget before an interview, grouped by subtopi - PID 1 gets no default signal handlers, so an app not written for it may ignore **`SIGTERM`** and be killed after the grace period. - Use the **exec form** (`CMD ["app"]`, not `CMD app`, which wraps it in a shell) and a tiny init (`tini`, `--init`) to forward signals and reap zombies. -### Container networking and storage - -- On a user-defined bridge network, containers resolve each other by name; `EXPOSE` only documents a port, `-p` publishes it. -- **Volumes** are Docker-managed and survive the container; **bind mounts** map a host path; the writable layer dies with the container. - ## Kubernetes architecture ### Control plane - The **API server** is the only component talking to **etcd**; the scheduler assigns Pods to nodes; **controllers** reconcile actual state toward desired state. - Everything is declarative and level-triggered: `kubectl apply` succeeding means the object was stored, not that the app is healthy. +- A bare Pod has no controller, so nothing recreates it after a node failure — run workloads through a Deployment, StatefulSet, DaemonSet, or Job. ### Node components - **kubelet** runs the node's Pods through the container runtime (CRI) and reports status. - **kube-proxy** or an **eBPF** CNI such as Cilium implements Service virtual IPs. -- kube-proxy modes: iptables (default), nftables (GA in 1.33), IPVS (deprecated since 1.35). -## Kubernetes workloads +### Packaging and extension -### Workload types +- **Helm** templates charts with values and tracks releases; **Kustomize** patches plain YAML with overlays, no templating. +- **CRDs + operators** extend the API: an operator's controller reconciles custom resources (databases, certificates) like built-in objects. -- **Deployment**: interchangeable stateless Pods via ReplicaSets. **StatefulSet**: stable names (`db-0`), ordered rollout, a volume per Pod. -- **DaemonSet**: one Pod per node (log agents, CNI). Job/CronJob: run to completion, once or on a schedule. -- Never run a bare Pod for anything that should come back after failure. +## Kubernetes workloads ### Rolling update tuning @@ -136,7 +137,9 @@ What experienced DevOps engineers forget before an interview, grouped by subtopi ### Autoscaling -- **HPA** scales replicas from metrics; **VPA** adjusts requests; Cluster Autoscaler/**Karpenter** add nodes when Pods can't be scheduled. +- **HPA** scales replicas from metrics; VPA adjusts requests; Cluster Autoscaler/Karpenter add nodes when Pods can't be scheduled. +- HPA utilization targets are a percentage of **requests**, so Pods without CPU requests can't scale on CPU; scale-down waits out a 5-minute stabilization window by default. +- **KEDA** scales on external events (queue length, consumer lag), down to zero; Karpenter also consolidates underused nodes. - More replicas don't help when the bottleneck is a saturated downstream database. ### Graceful Pod shutdown @@ -145,26 +148,30 @@ What experienced DevOps engineers forget before an interview, grouped by subtopi - The kubelet runs **`preStop`** first, then sends `SIGTERM` — a short `preStop` sleep covers that race. - **`terminationGracePeriodSeconds`** (default **30 s**) covers `preStop` plus shutdown; then the kubelet sends `SIGKILL`. App-side steps are in [backend.md](backend.md). -### ConfigMaps and Secrets - -- Secret values are only **base64-encoded**; enable encryption at rest (KMS), restrict access with **RBAC**, or sync from a vault (External Secrets Operator). -- Env vars from a ConfigMap don't update in a running Pod; mounted files do (eventually) — most apps still need a restart. - ### Debugging Pod states - **`Pending`**: unschedulable (resources, quota, taints, unbound volume). **`ImagePullBackOff`**: wrong image or registry credentials. **`CrashLoopBackOff`**: the process or its liveness probe keeps failing. - `kubectl describe pod` shows events; `kubectl logs --previous` shows the crashed container's last output. +## Kubernetes security + +### Pod security + +- **Pod Security Admission** enforces the privileged, baseline, or **restricted** profile per namespace via labels; PodSecurityPolicy was removed in 1.25. +- A hardened `securityContext`: `runAsNonRoot`, `readOnlyRootFilesystem`, `allowPrivilegeEscalation: false`, drop all capabilities, `seccompProfile: RuntimeDefault`. +- `privileged: true`, `hostPath` mounts, and `hostNetwork`/`hostPID` effectively hand over the node. +- Kyverno or OPA Gatekeeper enforce custom rules (allowed registries, no `latest` tags, required labels). + ### ServiceAccounts and workload identity - Each Pod runs as a **ServiceAccount**; RBAC Roles bound to it limit what it may do through the Kubernetes API. - Its tokens are projected, short-lived, and audience-bound; set `automountServiceAccountToken: false` when the app doesn't call the API. - **Workload identity** maps a ServiceAccount to a cloud role (EKS IRSA or Pod Identity, GKE Workload Identity) — no static cloud keys in Secrets. -### Packaging and extension +### ConfigMaps and Secrets -- **Helm** templates charts with values and tracks releases; **Kustomize** patches plain YAML with overlays, no templating. -- **CRDs + operators** extend the API: an operator's controller reconciles custom resources (databases, certificates) like built-in objects. +- Secret values are only **base64-encoded**; enable encryption at rest (KMS), restrict access with **RBAC**, or sync from a vault (External Secrets Operator). +- Env vars from a ConfigMap don't update in a running Pod; mounted files do (eventually) — most apps still need a restart. ## Terraform @@ -178,17 +185,23 @@ What experienced DevOps engineers forget before an interview, grouped by subtopi - **State** maps resource addresses to real objects and can hold secrets — keep it in a **remote backend with locking**, never in Git. - **Drift**: out-of-band changes show up in the next plan, which proposes reverting them. +### count vs for_each + +- `count` addresses instances by index (`aws_instance.web[2]`): removing one element shifts every later index, so Terraform **destroys and recreates** those resources. +- **`for_each`** keys instances by stable map or set keys (`aws_instance.web["api"]`), so adding or removing one touches only that one. +- `for_each` keys must be known at plan time, not computed during apply; switching between the two needs `moved` blocks. + ### Refactoring resources - **`moved`** blocks record a renamed address, so the plan doesn't destroy and recreate it. - **`import`** blocks (1.5) adopt existing resources into state. - **`removed`** blocks (1.7) drop a resource from config; the default destroys it — only `lifecycle { destroy = false }` keeps the real object. -### Modules, workspaces, licensing +### Modules, workspaces, OpenTofu - Modules package resources behind inputs and outputs. - Workspaces give one configuration several states but are **not an access boundary** — isolate environments with separate backends or root modules. -- Terraform moved to the **BSL license in 2023**; **OpenTofu** is the open-source fork with the same workflow. +- **OpenTofu** is the open-source fork of Terraform with the same workflow. ## CI/CD and GitOps diff --git a/docs/topics/distributed.md b/docs/topics/distributed.md index 591cff9..7051b63 100644 --- a/docs/topics/distributed.md +++ b/docs/topics/distributed.md @@ -109,7 +109,8 @@ Key distributed-systems building blocks and trade-offs, grouped by subtopic. ### Physical clocks and TrueTime - Clocks drift and NTP can step them backward, so wall-clock timestamps can't order events across nodes — **last-write-wins** silently loses data. -- Spanner's **TrueTime** returns an uncertainty interval; a commit **waits out** the uncertainty (~ms) before becoming visible, giving external consistency. +- Spanner's **TrueTime** returns an uncertainty interval; a commit waits out the uncertainty (~ms) before becoming visible, giving external consistency. +- Measure durations, timeouts, and leases with the **monotonic clock** (`CLOCK_MONOTONIC`, Go's `time.Since`, `performance.now()`), which never jumps; wall-clock time steps on NTP corrections, manual changes, and VM migration. ### Logical clocks @@ -138,10 +139,12 @@ Key distributed-systems building blocks and trade-offs, grouped by subtopic. - A chain of local transactions, each with a **compensating action**; eventual consistency, **no isolation** (others see intermediate states). - **Orchestration** (a central coordinator) vs choreography (services react to each other's events). -### Effectively-once processing +### Delivery guarantees -- **Effectively-once = at-least-once delivery + idempotent processing**: handlers may run repeatedly, but the effect lands once. -- Publish events atomically with the local write via the **transactional outbox**; dedupe on the consumer via an inbox — both in [backend.md](backend.md). +- **At-most-once**: ack (or commit the offset) before processing — a crash mid-processing loses the message. +- **At-least-once**: ack after processing — a crash between the effect and the ack redelivers it, so handlers see duplicates. +- Exactly-once *delivery* is impossible; an **exactly-once effect** = at-least-once + idempotent handling, or one transaction covering both the effect and the offset. +- Publish events atomically with the local write via the transactional outbox; dedupe on the consumer via an inbox — both in [backend.md](backend.md). ## Conflict resolution @@ -192,19 +195,25 @@ Key distributed-systems building blocks and trade-offs, grouped by subtopic. ## Stream processing -### Event time and watermarks +### Event time, watermarks, and windows - **Event time** (when it happened) vs **processing time** (when it arrived) — late events break processing-time windows. - A **watermark** says "no events older than T are expected", letting a window close; later stragglers go to a side output or update results. +- Windows: tumbling (fixed, non-overlapping), sliding or hopping (fixed, overlapping), session (closed by an inactivity gap). -### Window types +### Change data capture -- **Tumbling** (fixed, non-overlapping), **sliding/hopping** (fixed, overlapping), **session** (closed by an inactivity gap). +- **Log-based CDC** (Debezium) tails the WAL or binlog: every committed change in commit order, deletes included, with no dual writes in the application. +- Polling an `updated_at` column misses hard deletes and rows that commit late with an older timestamp. +- Bootstrap from a **consistent snapshot**, then stream from the log position it was taken at. +- Events are ordered per key (partition by primary key); connectors deliver at least once, so consumers must be idempotent. ### Kafka ordering and consumer groups - Order is guaranteed **only within a partition**; key by entity ID to keep one entity's events in order. -- A **consumer group** splits partitions among its members — parallelism is capped at the **partition count**. +- A consumer group splits partitions among its members — parallelism is capped at the **partition count**. +- Adding partitions changes `hash(key) % partitions`, so existing keys move and **per-key ordering breaks**; the count can't be decreased — size it up front. +- Consumer lag (latest offset − committed offset) is the health metric to alert on. ### Kafka producer semantics diff --git a/docs/topics/ethereum.md b/docs/topics/ethereum.md index 81c29be..dda14ca 100644 --- a/docs/topics/ethereum.md +++ b/docs/topics/ethereum.md @@ -4,9 +4,8 @@ What experienced Ethereum engineers forget before an interview, grouped by subto ## Accounts and transactions -### Account model +### Account nonces -- **EOA** (controlled by a secp256k1 key) vs **contract account** (controlled by its code); only EOAs originate transactions. - The **nonce** orders an EOA's transactions and prevents replay; an EIP-7702 authorization also increments it, even in someone else's transaction. - A stuck transaction blocks every later nonce until replaced by one with the same nonce and a higher fee. @@ -31,6 +30,12 @@ What experienced Ethereum engineers forget before an interview, grouped by subto - Signed → propagated to the **mempool** → a builder assembles a block (**PBS** via MEV-Boost separates building from proposing) → the proposer publishes it → validators attest. - **Finalized after 2 epochs (~12.8 min)**; before that a block can be reorged, so exchanges and bridges wait for finality or several confirmations. +### RPC block tags and reorgs + +- `latest` can be reorged; **`safe`** (the justified checkpoint) rarely is; **`finalized`** can't be reverted without slashing at least ⅓ of the stake. +- `eth_call` simulates against a chosen block without sending a transaction; `eth_estimateGas` binary-searches a working gas limit and can underestimate state-dependent paths, so add headroom. +- Indexers store block hashes and roll back on a reorg, or index only finalized blocks. + ## Gas and fees ### EIP-1559 fee market @@ -62,6 +67,20 @@ What experienced Ethereum engineers forget before an interview, grouped by subto - Memory expansion cost is **quadratic** in size; storage is the most expensive resource, priced per 32-byte slot. - `DELEGATECALL` runs another contract's code on the caller's storage; any call forwards at most 63/64 of the remaining gas (EIP-150) — see [solidity.md](solidity.md). +### Precompiles + +Built-in contracts at fixed low addresses for cryptography too expensive in bytecode; they have no code, so `extcodesize` returns 0. + +| Address | Function | +|---|---| +| `0x01` | `ecrecover` | +| `0x02`, `0x03` | SHA-256, RIPEMD-160 | +| `0x05` | modexp (RSA verification) | +| `0x06`–`0x08` | BN254 add, mul, pairing — **SNARK verifiers** | +| `0x0a` | KZG point evaluation (EIP-4844 blobs) | +| `0x0b`–`0x11` | BLS12-381 operations (Pectra, EIP-2537) | +| `0x100` | P-256 verification (Fusaka, EIP-7951) — passkeys and secure enclaves | + ### Transient storage and SELFDESTRUCT - **Transient storage** (`TSTORE`/`TLOAD`, EIP-1153, Cancun) costs 100 gas and is cleared after each transaction — cheap reentrancy locks. @@ -103,6 +122,8 @@ What experienced Ethereum engineers forget before an interview, grouped by subto - **ERC-4337**: users sign UserOperations, bundlers submit them to a singleton EntryPoint, and **paymasters** can sponsor gas — no protocol change. - **EIP-7702** lets an existing EOA delegate to smart-wallet code while keeping its address. +- 7702 risks: one authorization signature hands the EOA to the delegate's code, a prime phishing target; an authorization with `chain_id = 0` is valid on every chain. +- The delegate's storage lives in the EOA, so switching delegates can collide storage layouts — use namespaced storage (ERC-7201). ## Consensus (PoS) @@ -131,6 +152,12 @@ What experienced Ethereum engineers forget before an interview, grouped by subto - Rollups post transaction data to L1 for **data availability**, so anyone can rebuild state and challenge or exit. - Most use a **centralized sequencer** for ordering, with forced inclusion through L1 as the censorship backstop. +### ZK proof systems + +- **SNARKs** (Groth16, PLONK): proofs of a few hundred bytes, cheap to verify on L1 via the BN254 pairing precompile; many need a **trusted setup** (per circuit for Groth16, universal for PLONK). +- **STARKs**: no trusted setup and hash-based (plausibly post-quantum), but proofs run to tens or hundreds of KB, so rollups often wrap a STARK in a SNARK for L1. +- zkEVM types 1–4 trade Ethereum equivalence against proving cost: type 1 proves unmodified Ethereum blocks, type 4 compiles Solidity to a different VM. + ### Blobs and PeerDAS - **EIP-4844** blobs are cheap, temporary data (pruned after ~18 days), separate from calldata. @@ -161,5 +188,6 @@ What experienced Ethereum engineers forget before an interview, grouped by subto ### Bridge exploits -- Bridges hold large locked pools, making them prime targets. -- **Ronin** (5 of 9 validator keys compromised), **Wormhole** (signature verification bypass), **Nomad** (bad initialization let anyone forge messages). +- A bridge is only as secure as whoever attests to cross-chain messages, and its locked pool makes it a prime target. +- **Ronin** (2022): 5 of 9 validator keys compromised — a small validator set is just a multisig. +- **Wormhole** (2022): a signature-verification bypass minted unbacked tokens; **Nomad** (2022): an upgrade marked the zero root as trusted, so any message passed and copycats drained it. diff --git a/docs/topics/frontend.md b/docs/topics/frontend.md index 59e3265..b0c5115 100644 --- a/docs/topics/frontend.md +++ b/docs/topics/frontend.md @@ -53,9 +53,8 @@ What experienced frontend engineers forget before an interview, grouped by subto ## CSS -### CSS layout and specificity +### Specificity and the cascade -- **Flexbox** for one axis (a toolbar, centering); **Grid** for two axes (page layout, card grids). - **Specificity**: inline > ID > class/attribute/pseudo-class > element; cascade layers and origin rank above specificity, and `!important` inverts layer order. ### Stacking contexts @@ -66,13 +65,12 @@ What experienced frontend engineers forget before an interview, grouped by subto ### Modern CSS features - **Container queries** (`@container`) style a component by its container's size, not the viewport — reusable responsive components. -- **`:has()`** selects a parent by its children (`form:has(:invalid)`); native **nesting** and `subgrid` remove common preprocessor and wrapper hacks. -- All Baseline since 2023. +- **`:has()`** selects a parent by its children (`form:has(:invalid)`), which used to need JavaScript. ### View transitions - **View Transitions API** animates between two DOM states: `document.startViewTransition(update)` snapshots old and new, then cross-fades or morphs elements sharing a `view-transition-name`. -- Same-document transitions are Baseline since Firefox 144 (October 2025); multi-page apps opt in with `@view-transition { navigation: auto }`. +- Multi-page apps opt in to cross-document transitions with `@view-transition { navigation: auto }`. ## Performance @@ -123,6 +121,18 @@ What experienced frontend engineers forget before an interview, grouped by subto - Long `max-age` + `immutable` on **content-hashed filenames** (`app.a3f9c1.js`), with HTML served `no-cache` — each deploy is a new URL. - `ETag`/`Last-Modified` let the server answer **`304 Not Modified`** — saves the body, not the round trip. +### Browser storage + +| Storage | Size | Notes | +|---|---|---| +| Cookies | ~4 KB each | sent with every request to the domain | +| `localStorage` | ~5 MB per origin | **synchronous** (blocks the main thread), strings only | +| `sessionStorage` | ~5 MB per origin | per tab, cleared when the tab closes | +| **IndexedDB** | a share of free disk | async, structured data, available in workers | +| Cache API | a share of free disk | request/response pairs, used by service workers | + +Any XSS can read all of them except `HttpOnly` cookies — token storage in [security.md](security.md). + ### Service workers - **Service workers** implement cache-first, network-first, or stale-while-revalidate strategies and offline support. @@ -135,6 +145,18 @@ What experienced frontend engineers forget before an interview, grouped by subto - **Keys** match list items across renders; an array index as key attaches the wrong state after reorder or filter. - A component re-renders on its own state change, a **parent re-render** (unless memoized), or a consumed **context value** change. +### State snapshots and resets + +- State is a **snapshot** per render: `setCount(count + 1)` twice adds 1; the updater form `setCount(c => c + 1)` twice adds 2. +- Changing a component's **`key`** remounts it and resets its state — the clean way to reset a form when the selected item changes. +- `useRef` holds a mutable value across renders without triggering a re-render; don't read or write it during render. + +### Error boundaries + +- An error boundary catches errors thrown while **rendering** its subtree and shows a fallback UI instead of unmounting the whole app. +- It's still a **class component** (`getDerivedStateFromError`, `componentDidCatch`) or the `react-error-boundary` package; there's no hook equivalent. +- It doesn't catch errors in event handlers or async code (`setTimeout`, promises) — handle those with `try/catch` and state. + ### Fiber and concurrent rendering - **Fiber** splits rendering into interruptible units: the render phase can pause, restart, or be discarded; the **commit phase** applies DOM changes synchronously. @@ -174,22 +196,24 @@ What experienced frontend engineers forget before an interview, grouped by subto - Deriving state inside an effect + `setState` causes an extra render and a flash — **compute it during render**. - Every subscription, timer, or listener an effect creates needs a **cleanup** function. +### Data fetching in effects + +- Responses can arrive **out of order**: ignore stale ones in the effect cleanup (an `ignore` flag or `AbortController`), or a slow earlier request overwrites newer data. +- Effect-based fetching creates **waterfalls** (parent fetches → child mounts → child fetches); hoist fetching to route loaders, Server Components, or a query library. +- Strict Mode runs effects twice in development, so a fetch without cleanup fires twice there. + ### Memoization pitfalls - `React.memo` is defeated by a **new object, array, or function** literal passed as a prop on every render. - Memoize only when the skipped work costs more than the comparison. -### Controlled vs uncontrolled inputs - -- **Controlled** (`value` + `onChange`): React owns the value — needed when it drives other UI. -- **Uncontrolled** (read via ref): less code, fewer re-renders; a file input is always uncontrolled. - ## State management -### Where state lives +### Context performance -- Local state → lift to the nearest common parent → context → a **dedicated store** only when many unrelated components share it. -- Reach for context or a store when prop drilling actually hurts, not for a two-level pass. +- Every consumer re-renders when the provider's `value` changes **identity** — an inline object literal changes it on every render. +- Memoize the value, split fast- and slow-changing state into separate contexts, or use a store with selectors (`useSyncExternalStore`, Zustand) so components subscribe to slices. +- Context suits low-frequency values (theme, locale, current user), not fast-changing shared state. ### Server state vs client state diff --git a/docs/topics/golang.md b/docs/topics/golang.md index cad31a3..474b555 100644 --- a/docs/topics/golang.md +++ b/docs/topics/golang.md @@ -7,8 +7,9 @@ What experienced Go engineers forget before an interview, grouped by subtopic. ### Scheduler (GMP) - Goroutines (G) run on OS threads (M) through logical processors (P); **`GOMAXPROCS`** = number of Ps. -- Default = min(logical CPUs, affinity mask); **since Go 1.25** also capped by the cgroup CPU limit (rounded up, floor 2) and updated when it changes — before that, containers needed `automaxprocs`. +- Default = min(logical CPUs, affinity mask); since Go 1.25 also capped by the cgroup CPU limit (rounded up, floor 2) and updated when it changes — before that, containers needed `automaxprocs`. - Blocking syscall → the M is parked and the P moves to another M; network I/O goes through the netpoller and doesn't hold a thread. +- Each P has a local run queue (256 goroutines) plus a `runnext` slot; an idle P takes half of another P's queue (**work stealing**), and every 61st schedule checks the global queue for fairness. - Preemption is asynchronous (signal-based) since Go 1.14, so tight loops no longer starve the scheduler. - Goroutines start with a **2 KB** stack that grows and is copied as needed — this is why launching hundreds of thousands is normal. @@ -51,9 +52,10 @@ Only the sender should close a channel; `for range ch` ends once it's closed and ### context.Context - Carries cancellation, deadlines, and request-scoped values across API and goroutine boundaries; canceling a parent cancels all children. -- `WithCancelCause` (**1.20**) lets callers retrieve *why* a context was canceled via `context.Cause(ctx)`. +- `WithCancelCause` (1.20) lets callers retrieve *why* a context was canceled via `context.Cause(ctx)`. - `AfterFunc` and `WithoutCancel` (**1.21**): run a function on cancellation, or derive a context that keeps values but drops the parent's cancellation. - Pass as the **first parameter**, never store in a struct; use `WithValue` only for request-scoped data, not optional parameters. +- Every `WithCancel`/`WithTimeout`/`WithDeadline` returns a `cancel` func: call it (**`defer cancel()`**) or the child context and its timer live until the parent ends; `go vet` flags it (`lostcancel`). ### Patterns and goroutine leaks @@ -86,12 +88,6 @@ Only the sender should close a channel; `for range ch` ends once it's closed and - Fields are aligned to their type's alignment, so on 64-bit `struct{ a bool; b int64; c bool }` takes **24 bytes** while `{b int64; a, c bool}` takes **16** — order fields largest first in hot, numerous structs. - An empty struct (`struct{}`) is **zero bytes** — used for sets (`map[K]struct{}`) and signal channels. -### Cleanups, weak pointers, interning - -- **`runtime.AddCleanup`** (1.24) replaces `SetFinalizer`: several cleanups per object, no resurrection, and no leak when objects form a cycle. -- **`weak.Pointer`** (1.24) references an object without keeping it alive — for caches and canonicalization maps. -- **`unique.Make`** (1.23) interns comparable values, turning equality into a pointer compare. - ## Slices, maps, strings ### Slice mechanics @@ -105,8 +101,9 @@ Only the sender should close a channel; `for range ch` ends once it's closed and - Hash table; implemented as **Swiss tables since Go 1.24** (faster, but iteration order was already randomized and remains so). - Concurrent write (or write + read) is a race; the runtime detects it best-effort, and then it's a **fatal, unrecoverable** error, not a panic you can `recover`. -- Maps never shrink after deletions; `clear(m)` (**1.21**) empties one without reallocating the header. +- Maps never shrink after deletions; `clear(m)` (1.21) empties one without reallocating the header. - Map values aren't addressable: `&m[k]` and `m[k].field = x` don't compile for struct values. +- Reading a nil map returns zero values, but **writing to a nil map panics** — `make` it first (a nil slice, by contrast, works with `append`). ### Strings @@ -139,12 +136,6 @@ Only the sender should close a channel; `for range ch` ends once it's closed and - **Range-over-func** iterators (`iter.Seq`, `iter.Seq2`, **1.23**) let `range` work over a function, powering the `slices`/`maps` iterator helpers. - **Since Go 1.22**, each iteration gets its own copy of variables declared by the loop (`:=`) — the classic closure-capture bug needs `go 1.21` semantics or a reused outer variable (`for i = 0; ...`). -### Newer language features - -- 1.21: `min` and `max` built-ins. 1.22: **`for i := range 10`** over integers. -- 1.24: generic type aliases. -- 1.26: **`new(expr)`** allocates and initializes in one step (`new(42)` → `*int`). - ## Errors, defer, panic ### Error wrapping @@ -165,12 +156,14 @@ Only the sender should close a channel; `for range ch` ends once it's closed and - The default **`http.Client` has no timeout** — a hung server blocks the goroutine forever; always set `Timeout` or use a context. - Always **close `resp.Body`** (and drain it) or the connection isn't reused and leaks. - `http.Server` needs `ReadHeaderTimeout`/`ReadTimeout`/`WriteTimeout`, or slow clients (Slowloris) hold connections open. +- **`ServeMux` patterns since 1.22** take methods and wildcards: `mux.HandleFunc("GET /items/{id}", h)` with `r.PathValue("id")`. -### Routing, JSON, logging +### encoding/json gotchas -- **`ServeMux` patterns since 1.22**: methods and wildcards, `mux.HandleFunc("GET /items/{id}", h)` with `r.PathValue("id")`. -- **`encoding/json/v2`**: experimental in 1.25 (`GOEXPERIMENT=jsonv2`), standard in 1.27 — faster, and stricter defaults (e.g. case-sensitive field matching). -- **`log/slog`** (1.21) is the standard structured logger. +- Numbers decoded into `any` become **float64**, so integer IDs above 2⁵³ lose precision — use `Decoder.UseNumber` or a typed field. +- A nil slice or map encodes as `null`, an empty one as `[]` or `{}`. +- Unexported fields are silently skipped; unknown JSON fields are ignored unless `DisallowUnknownFields`; field-name matching is case-insensitive. +- `omitempty` never omits a struct value (a zero `time.Time` included); **`omitzero`** (1.24) does. ## Modules and tooling @@ -185,12 +178,6 @@ Only the sender should close a channel; `for range ch` ends once it's closed and - **`internal/`** packages are importable only from within the parent tree. - **`go.work`** (**1.18**) builds several local modules together without `replace` directives. -### Toolchains and tools - -- The `toolchain` line and **`GOTOOLCHAIN`** (1.21) let `go` download the toolchain a module requires. -- **`tool` directives** in `go.mod` (**1.24**) track dev tools, run with `go tool ` — replaces the `tools.go` hack. -- `go fix` (1.26) applies "modernizer" rewrites that adopt new idioms across a codebase. - ### Compatibility and GODEBUG - Potentially breaking changes ship behind **`GODEBUG`** settings whose defaults follow the main module's **`go` line**, so a toolchain upgrade alone keeps their old behavior (other runtime and performance changes still apply). @@ -200,7 +187,6 @@ Only the sender should close a channel; `for range ch` ends once it's closed and ### Test tooling -- Table-driven tests with `t.Run` subtests, `t.Parallel()`, `t.Cleanup()`. - Benchmarks use `b.Loop()` (**1.24**), which replaced the manual `for i := 0; i < b.N; i++` idiom and avoids some compiler over-optimization pitfalls. - Fuzzing (`func FuzzX`, **1.18**) generates inputs from a seed corpus. diff --git a/docs/topics/javascript.md b/docs/topics/javascript.md index cc071a9..d762b05 100644 --- a/docs/topics/javascript.md +++ b/docs/topics/javascript.md @@ -78,6 +78,13 @@ What experienced JavaScript engineers forget before an interview, grouped by sub | `Promise.race` | first settles (either way) | timeouts, first response wins | | `Promise.any` | first fulfills, or all reject (`AggregateError`) | first success wins | +### async/await ordering + +- The `new Promise` executor runs **synchronously**; only the `.then` callbacks are deferred. +- Code after `await` always resumes as a microtask, even when the awaited value is already resolved or isn't a promise. +- Inside `try`, **`return promise`** skips the `catch` when it rejects; write `return await promise`. +- An `async` function always returns a new promise, so a `throw` inside it becomes a rejection, not a synchronous exception. + ### Sequential vs parallel await - `for (const x of xs) await f(x)` runs **serially**: each call starts after the previous settles; start all calls first, then `await Promise.all([...])` so they overlap. @@ -88,10 +95,12 @@ What experienced JavaScript engineers forget before an interview, grouped by sub - **`AbortController`** cancels `fetch` and other abortable APIs via a shared `AbortSignal`; `AbortSignal.timeout(ms)` builds a deadline. - **`for await...of`** consumes async generators and any object with `Symbol.asyncIterator`. -### Promise helpers +### fetch semantics -- **`Promise.withResolvers()`** (ES2024) returns `{ promise, resolve, reject }`. -- **`Promise.try()`** (ES2025) runs a function and turns both its result and a synchronous throw into a promise. +- `fetch` rejects only on network failure (DNS, refused connection, CORS, abort); HTTP 4xx/5xx **resolve** with `res.ok === false`, so check it. +- The body is a stream and can be read **once**; call `res.clone()` before reading it twice. +- There is no default timeout: pass `signal: AbortSignal.timeout(ms)`. +- Cookies go only to the same origin by default (`credentials: 'same-origin'`); cross-origin requests need `'include'` plus CORS credential headers. ## Modules @@ -113,6 +122,13 @@ What experienced JavaScript engineers forget before an interview, grouped by sub - Common leaks: **detached DOM nodes** still referenced from JS, forgotten listeners and timers, unbounded caches. - A closure keeps its referenced outer variables alive, so a long-lived listener can retain a large object it barely uses. +### Hidden classes and inline caches + +- V8 gives objects created with the same properties in the same order a shared **hidden class** (shape); adding a property transitions to a new one. +- Each property-access site caches the shapes it has seen: monomorphic (one) is fastest, polymorphic up to 4, then **megamorphic** (a generic slow lookup). +- Initialize every field in the constructor in a fixed order; `delete` on a hot object can drop it into slow dictionary mode. +- Arrays track element kinds too: mixing integers, doubles, and holes (`new Array(n)`) downgrades them for good. + ### Weak references - **`WeakMap`/`WeakSet`** keys don't keep objects alive — good for per-object metadata and caches. @@ -134,6 +150,7 @@ What experienced JavaScript engineers forget before an interview, grouped by sub - An object is iterable if it has **`[Symbol.iterator]()`** returning `{ next() → { value, done } }`; `for...of`, spread, and destructuring use it. - Generators (`function*`) are lazy, pause at `yield`, and can receive values via `next(v)`; async generators power `for await`. +- **Iterator helpers** (`.map`, `.filter`, `.take`, `.drop` on iterators, ES2025) are lazy, unlike the array methods. ## Node.js runtime @@ -154,11 +171,10 @@ What experienced JavaScript engineers forget before an interview, grouped by sub - **`write()` returns `false`** once the buffer passes `highWaterMark`; stop writing until the `'drain'` event. - **`stream.pipeline()`** handles backpressure, errors, and cleanup of every stream in the chain — `.pipe()` doesn't forward errors. -### Workers, cluster, child processes +### Worker threads and cluster - **`worker_threads`**: threads in one process with separate V8 isolates, for CPU-bound JS; share memory through `SharedArrayBuffer`. - **`cluster`**: forks processes sharing one server port to use all cores — in containers, usually replaced by more replicas. -- `child_process`: runs other programs (`spawn` streams output, `exec` buffers it). ### AsyncLocalStorage @@ -179,22 +195,12 @@ What experienced JavaScript engineers forget before an interview, grouped by sub ## Recent language additions -### Array and object helpers - -- **`toSorted`, `toReversed`, `toSpliced`, `with`** (ES2023) return a new array instead of mutating. -- **`Object.groupBy`/`Map.groupBy`** (ES2024) group an iterable by a callback's key. - ### Temporal -- **`Temporal`** (Stage 4 in March 2026, **ES2026**) replaces `Date`: immutable values, explicit time zones (`ZonedDateTime`), calendar-safe arithmetic, separate types for dates, times, and instants. -- Shipped in Firefox 139 and Chrome 144; Safari has it only in Technology Preview, so it isn't Baseline yet. +- **`Temporal`** (Stage 4 in March 2026, **ES2026**) replaces `Date` with immutable values and separate types for instants, dates, times, and zoned date-times. +- Time zones are explicit (`ZonedDateTime`) and arithmetic is calendar-safe, unlike `Date`'s mutable, local-time-by-default API. ### Explicit resource management - **`using`** / `await using` (ES2026; TypeScript since 5.2) call **`[Symbol.dispose]()`** / `[Symbol.asyncDispose]()` when the block exits, even on throw — no `try/finally`. - **`DisposableStack`** collects several resources and disposes them in reverse order. - -### Sets and iterators - -- **Set methods** (`union`, `intersection`, `difference`, `isSubsetOf`, …) arrived in **ES2025**. -- **Iterator helpers** (`.map`, `.filter`, `.take`, `.drop` on iterators, ES2025) are lazy, unlike the array methods. diff --git a/docs/topics/networking.md b/docs/topics/networking.md index e04b5b5..80699bb 100644 --- a/docs/topics/networking.md +++ b/docs/topics/networking.md @@ -10,10 +10,16 @@ What experienced engineers forget about networking before an interview, grouped - A cold HTTPS request over TCP costs about **3 RTTs** before the first response byte; **HTTP/3** saves one, and resumption with 0-RTT saves more. - Every later request reuses the connection, which is why keep-alive and connection pools matter. -### Debugging tools +### Connection failure signatures -- `dig` (DNS answer, TTL), `curl -v` and `curl -w '%{time_connect} %{time_appconnect} %{time_starttransfer}'` (per-phase timing). -- `ss -tanp` (sockets and states), `traceroute`/`mtr` (path, TTL expiry), `tcpdump`/Wireshark (packets). +| Symptom | Usual cause | Check with | +|---|---|---| +| Name doesn't resolve, or resolves to an old IP | NXDOMAIN, stale cache, TTL not expired | `dig` (answer, TTL) | +| Connection refused | RST: host reachable, nothing listening on the port | `ss -tlnp` on the server | +| Connect timeout | SYN dropped: firewall, security group, or dead host; Linux retries the SYN for **~127 s**, so set connect timeouts | `traceroute`/`mtr`, `tcpdump` for SYNs | +| Connection reset by peer | peer crashed, or a middlebox dropped the flow after its idle timeout | `tcpdump`/Wireshark around the RST | +| **502** from a proxy | upstream refused, reset, or sent an invalid response | proxy error log, `ss -tanp` states | +| **504** from a proxy, or just slow | upstream didn't answer in time | `curl -w '%{time_connect} %{time_appconnect} %{time_starttransfer}'` per phase | ## TCP @@ -36,6 +42,13 @@ What experienced engineers forget about networking before an interview, grouped - Throughput ≤ window / RTT (**bandwidth-delay product**) — high-latency links need large windows. - CUBIC (Linux default) backs off on packet loss; BBR models bandwidth and RTT instead, doing better on lossy links. +### Retransmission and loss recovery + +- The retransmission timeout (**RTO**) derives from smoothed RTT; Linux floors it at **200 ms** (initial 1 s) and doubles it on each retry. +- One lost packet on a quiet connection therefore adds ≥ 200 ms — a classic cause of p99 spikes. +- **Fast retransmit**: 3 duplicate ACKs resend the missing segment without waiting for the RTO; SACK tells the sender exactly which ranges arrived. +- A lost last segment produces no duplicate ACKs; Linux's tail loss probe resends it after ~2 RTTs instead of waiting for the RTO. + ### Keepalive and half-open connections - **TCP keepalive** probes an idle connection to detect a dead peer (Linux: first probe after **2 hours** by default); HTTP keep-alive only means reusing a connection — unrelated. @@ -52,17 +65,12 @@ What experienced engineers forget about networking before an interview, grouped - TCP delivers bytes in order, so one lost packet stalls everything behind it — even data for unrelated HTTP/2 streams. - HTTP/1.1 blocks at the request level, HTTP/2 fixes that but not TCP's, and **QUIC** fixes both: streams are delivered independently, so a lost packet stalls only the streams whose data it carried. -### UDP - -- No handshake, ordering, retransmission, or congestion control — the application decides. -- Used where latency beats reliability or the protocol handles it itself: DNS, VoIP, games, **QUIC**. - -## HTTP versions +## HTTP ### HTTP/1.1 - Persistent connections (keep-alive), but **one outstanding request per connection** in practice (pipelining is unused). -- Browsers open about **6 connections per host** to parallelize — the reason for old tricks like domain sharding and sprite sheets. +- Browsers open about **6 connections per host** to parallelize. ### HTTP/2 @@ -72,6 +80,7 @@ What experienced engineers forget about networking before an interview, grouped ### HTTP/3 and QUIC +- UDP has no handshake, ordering, or retransmission, so QUIC implements them in user space; DNS, VoIP, and games use UDP directly and handle loss themselves. - QUIC runs over UDP with TLS 1.3 built in: **1-RTT** setup (0-RTT on resumption) and independent stream delivery, so no cross-stream head-of-line blocking; loss detection and congestion control stay per connection. - **Connection migration**: a connection ID survives an IP change (Wi-Fi → cellular). - Clients discover it via the `Alt-Svc` header or a DNS HTTPS record; many networks block UDP, so browsers fall back to TCP. @@ -81,6 +90,12 @@ What experienced engineers forget about networking before an interview, grouped - Starts as HTTP/1.1 **`Upgrade: websocket`** → **`101 Switching Protocols`**, then framed full-duplex messages on the same TCP connection. - Proxies and load balancers need idle timeouts above the app's ping interval, or they drop quiet connections. +### Client IP behind proxies + +- L7 proxies append the peer's address to **`X-Forwarded-For`**; clients can forge earlier entries, so trust only those your own proxies added — read from the right, skipping known proxy hops. +- L4 load balancers either preserve the source IP or pass it in a **PROXY protocol** header, which the backend must be configured to expect. +- Rate limits, geo-blocking, and audit logs depend on it: taking the leftmost entry lets any client spoof its IP. + ## DNS ### Resolution path @@ -110,6 +125,12 @@ What experienced engineers forget about networking before an interview, grouped - `/24` = 256 addresses, `/16` = 65,536; each bit less doubles the block. - Private ranges: **10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16**; 100.64.0.0/10 is carrier-grade NAT; AWS reserves **5 addresses** per subnet. +### Routing tables + +- The kernel picks the **longest prefix match**: a `/32` route beats a `/24`, which beats the default route `0.0.0.0/0` (the gateway). +- On the local link, **ARP** (IPv6: neighbor discovery) resolves the next hop's MAC address; MACs change at every hop, while IPs stay end to end unless NAT rewrites them. +- `ip route get ` shows the route, interface, and source address the kernel would use. + ### NAT - NAT rewrites source IP and port and tracks each flow in a **connection-tracking table** — unsolicited inbound connections are dropped unless a port forward or static mapping exists. diff --git a/docs/topics/os.md b/docs/topics/os.md index 8ca9aa2..47a8827 100644 --- a/docs/topics/os.md +++ b/docs/topics/os.md @@ -4,16 +4,22 @@ What experienced engineers forget about Linux and OS internals before an intervi ## Processes and threads -### Processes vs threads - -- A **process** has its own address space, file descriptors, and credentials; **threads** share the address space and descriptors of their process. -- On Linux both are **tasks** created by `clone()` with different sharing flags; the scheduler treats them alike. - ### fork, exec, copy-on-write - **`fork()`** duplicates the process lazily: pages are shared **copy-on-write** until one side writes. - **`exec()`** replaces the program image, keeping the PID and open descriptors (unless `O_CLOEXEC`). - Forking a multi-threaded process copies only the calling thread — locks held by others stay locked forever in the child. +- On Linux, processes and threads are both tasks created by `clone()` with different sharing flags; the scheduler treats them alike. + +### Process states + +| State | Meaning | +|---|---| +| R | running or runnable (on a run queue) | +| S | interruptible sleep: waiting for an event; signals wake it | +| **D** | uninterruptible sleep, usually disk or NFS I/O; even `SIGKILL` waits, and it counts toward load average | +| Z | zombie: exited, not yet reaped by its parent | +| T | stopped (`SIGSTOP`, a debugger) | ### Zombies and orphans @@ -122,6 +128,12 @@ What experienced engineers forget about Linux and OS internals before an intervi - Data is durable only after **`fsync`** (or `fdatasync`); a new file also needs its **directory** fsynced. - After an fsync error, Linux may drop the dirty pages and a retry can falsely succeed — since 2018's "fsyncgate", **PostgreSQL panics** on fsync failure instead of retrying. +### Atomic file replace + +- Write a temp file in the **same directory** → `fsync` it → `rename()` it over the target → `fsync` the directory. +- `rename` within one filesystem is **atomic** on POSIX: readers see the old or the new file, never a torn mix. +- Skipping the first `fsync` can leave an empty file after a crash (the rename persisted before the data). + ### Zero-copy I/O - **`sendfile`**/`splice` move file data to a socket inside the kernel, skipping user-space copies — how static file servers and Kafka reach high throughput. @@ -140,10 +152,12 @@ What experienced engineers forget about Linux and OS internals before an intervi - All four **Coffman conditions** must hold: mutual exclusion, hold and wait, no preemption, **circular wait**. - Break one — usually circular wait, with a global **lock ordering** — or use `trylock` with timeouts. -### Priority inversion +### Atomics, CAS, and the ABA problem -- A low-priority task holds a lock needed by a high-priority task while a medium-priority task starves it (Mars Pathfinder, 1997). -- **Priority inheritance** temporarily boosts the lock holder. +- **Compare-and-swap** writes only if the value still equals the expected one; retry loops build lock-free stacks, counters, and mutex fast paths. +- **ABA**: the value goes A → B → A between the read and the CAS, so a stale CAS succeeds (a popped stack node freed and reused); fix with version-tagged pointers or safe reclamation (hazard pointers, epochs). +- Under heavy contention, CAS loops bounce the cache line between cores and can be slower than a mutex. +- x86 is strongly ordered (TSO), ARM weakly ordered, so code that works on x86 can break on ARM without the right memory orderings ([rust.md](rust.md)). ### False sharing @@ -154,10 +168,21 @@ What experienced engineers forget about Linux and OS internals before an intervi ### Linux observability tools -- `top`/`htop` (CPU per process), `vmstat 1` (run queue, swap, context switches), `iostat -x 1` (disk utilization, await), `pidstat` (per-process CPU and I/O). - **`perf`** samples stacks with low overhead → flame graphs; **eBPF** tools (`bpftrace`, bcc) trace kernel events in production safely. - **`strace`** shows every syscall but slows the process heavily (ptrace) — avoid on hot production paths. +### Memory hierarchy latencies + +| Level | Latency | +|---|---| +| L1 cache | ~1 ns | +| L2 cache | ~4 ns | +| L3 cache | ~10–40 ns | +| DRAM | **~100 ns** | +| NVMe SSD read | ~10–100 µs | + +Sequential access and contiguous layouts (arrays over linked lists, struct-of-arrays) win through prefetching and full cache-line use. + ### CPU time breakdown - us (user code), sy (kernel), **wa** (idle waiting for I/O), **st** (stolen by the hypervisor — noisy neighbors on VMs). diff --git a/docs/topics/python.md b/docs/topics/python.md index 148c6ae..f5cace2 100644 --- a/docs/topics/python.md +++ b/docs/topics/python.md @@ -16,17 +16,17 @@ What experienced Python engineers forget before an interview, grouped by subtopi - The **free-threaded build** (no GIL) was experimental in 3.13 and is **officially supported since 3.14** (PEP 779); an incompatible C extension can force the GIL back on. - **`concurrent.interpreters`** (3.14, PEP 734) runs several interpreters in one process, each with its own GIL, talking over cross-interpreter queues. -### Specializing interpreter and JIT +### Specializing interpreter -- **3.11** added the adaptive specializing interpreter, roughly **25%** faster on average: hot bytecodes are rewritten into type-specialized versions. -- 3.13 added an experimental copy-and-patch **JIT**; 3.14's official Windows and macOS builds include it, off by default (`PYTHON_JIT=1`). +- **3.11** added the adaptive specializing interpreter, roughly **25%** faster on average. +- Hot bytecodes are rewritten in place into type-specialized versions, which fall back when the observed types change. ## Memory management ### Refcounting and the cyclic GC -- **Reference counting** frees an object the instant its count hits zero; a **generational cyclic GC** periodically frees reference cycles counting can't. -- 3.14.0–3.14.4 shipped an incremental GC; **3.14.5 reverted it** over production memory growth. +- **Reference counting** frees an object the instant its count hits zero. +- A **generational cyclic GC** periodically finds and frees reference cycles, which counting alone can't. ### Object identity and caching @@ -71,8 +71,9 @@ What experienced Python engineers forget before an interview, grouped by subtopi ### multiprocessing start methods - **`fork`** copies the parent — fast, but can inherit inconsistent state (a lock held by another thread mid-acquire). -- `spawn` starts a fresh interpreter — slower, safest; **`forkserver`** forks children from a clean single-threaded server process. -- **`forkserver` is the Linux default since 3.14** (previously `fork`). +- `spawn` starts a fresh interpreter — slower, safest; `forkserver` forks children from a clean single-threaded server process. +- **`forkserver` is the Linux default since 3.14** (previously `fork`); macOS defaults to `spawn` since 3.8, Windows always has. +- With `spawn`/`forkserver`, children **re-import the main module**, so guard entry code with `if __name__ == "__main__":`; the target, arguments, and results must pickle (no lambdas, local functions, or open sockets). ### asyncio event loop @@ -88,20 +89,29 @@ What experienced Python engineers forget before an interview, grouped by subtopi ## Gotchas -### Mutable default arguments +### Default arguments and late binding - Defaults are evaluated **once, at definition time** — `def f(x, cache=[])` shares one list across calls; default to `None` and create the object inside. - Dataclasses reject unhashable defaults (`list`, `dict`, `set`) — use **`field(default_factory=list)`**. - -### Late-binding closures - -- A closure captures the *variable*, not its value — `[lambda: i for i in range(3)]` all return `2`; bind early with `lambda i=i: i`. +- A closure captures the *variable*, not its value — `[lambda: i for i in range(3)]` all return `2`; a default binds early: `lambda i=i: i`. ### Scope and `UnboundLocalError` - **LEGB** lookup: Local, Enclosing, Global, Built-in. - Assigning to a name anywhere in a function makes it local for the **whole** body, so reading it before the assignment raises `UnboundLocalError`; use `nonlocal`/`global`. +### Numeric gotchas + +- `//` and `%` **floor toward −∞**: `-7 // 2 == -4` and `-7 % 2 == 1`; C, Go, Rust, and JavaScript truncate toward zero (`-7 % 2 == -1`). +- **`round()` rounds half to even**: `round(2.5) == 2`, `round(3.5) == 4`; `round(2.675, 2) == 2.67` because the float is slightly below 2.675. +- `int` never overflows (arbitrary precision); `float` is an IEEE 754 double, so use **`decimal.Decimal`** built from strings for money. + +### Recursion depth + +- The default limit is **1000** frames (`sys.getrecursionlimit()`); going deeper raises `RecursionError`. +- There's no **tail-call optimization**, so deep DFS or memoized recursion on large inputs needs an explicit stack. +- `sys.setrecursionlimit` helps for moderate depths; for depths around 10⁵ and beyond, rewrite iteratively. + ### Copies and aliasing - `[[0] * 3] * 3` builds three references to the **same** inner list — mutating one row mutates all. @@ -109,8 +119,8 @@ What experienced Python engineers forget before an interview, grouped by subtopi ### Mutation during iteration -- Removing items from a list inside `for` skips elements; resizing a dict raises `RuntimeError: dictionary changed size during iteration` — iterate over a copy. -- Dict insertion order is guaranteed **since 3.7**. +- Removing items from a list inside `for` skips elements. +- Resizing a dict while iterating raises `RuntimeError: dictionary changed size during iteration` — iterate over a copy. ### Import mechanics @@ -124,6 +134,12 @@ What experienced Python engineers forget before an interview, grouped by subtopi - Multiple inheritance resolves via **C3 linearization** (the MRO, `Cls.__mro__`). - **`super()`** follows the instance's MRO, not the direct parent, so every class in a mixin chain must call `super().__init__()`. +### Shared class attributes + +- A mutable class attribute (`items = []` in the class body) is **shared by every instance**. +- `self.items.append(x)` mutates the shared list; `self.items = [...]` creates an instance attribute that shadows it. +- Create per-instance state in `__init__`; dataclasses force `field(default_factory=list)` for this reason. + ### Descriptors - An object with `__get__`/`__set__`/`__delete__` on a class attribute is a **descriptor**; `property`, bound methods, `classmethod`, and `staticmethod` are all built on it. @@ -162,19 +178,7 @@ What experienced Python engineers forget before an interview, grouped by subtopi - **`__exit__` returning `True` suppresses** the exception; `contextlib.contextmanager` turns a one-`yield` generator into a context manager. - `try ... else` runs only when no exception was raised; **`except*`** (**3.11**) handles exception groups, e.g. from a `TaskGroup`. -## Recent language features - -### Structural pattern matching - -- `match` (**3.10**) destructures by shape: `case {"type": "click", "x": x}`, `case Point(x=0)`. -- A bare name in a `case` **binds**, it doesn't compare — `case RED:` matches everything; use dotted names (`Color.RED`) for constants. - -### Template strings - -- **t-strings** (`t"Hello {name}"`, **3.14**, PEP 750) produce a `Template` object with static parts and interpolated values kept separate. -- A library receiving it can escape values safely (SQL, HTML), unlike an f-string that arrives already joined. - -## Typing +## Typing and recent features ### Gradual typing and protocols @@ -187,6 +191,16 @@ What experienced Python engineers forget before an interview, grouped by subtopi - **PEP 695** syntax (`class Box[T]: ...`, `type Alias = ...`, 3.12) replaces manual `TypeVar` declarations with scoped type parameters. - Annotations are **evaluated lazily since 3.14** (**PEP 649**), making `from __future__ import annotations` largely unnecessary. +### Structural pattern matching + +- `match` (**3.10**) destructures by shape: `case {"type": "click", "x": x}`, `case Point(x=0)`. +- A bare name in a `case` **binds**, it doesn't compare — `case RED:` matches everything; use dotted names (`Color.RED`) for constants. + +### Template strings + +- **t-strings** (`t"Hello {name}"`, **3.14**, PEP 750) produce a `Template` object with static parts and interpolated values kept separate. +- A library receiving it can escape values safely (SQL, HTML), unlike an f-string that arrives already joined. + ## Tooling ### Packaging and tests diff --git a/docs/topics/rust.md b/docs/topics/rust.md index e76d685..dbf7bcd 100644 --- a/docs/topics/rust.md +++ b/docs/topics/rust.md @@ -63,6 +63,13 @@ What experienced Rust engineers forget before an interview, grouped by subtopic. - Every type parameter is implicitly `Sized`; **`?Sized`** accepts `str`, `[T]`, or `dyn Trait` behind a pointer. - `impl Trait` in argument position is an anonymous generic; in return position it hides the concrete type but stays static dispatch. +### Eq, Ord, and Hash + +- `PartialEq`/`PartialOrd` allow incomparable values (NaN ≠ NaN); **`Eq`/`Ord`** promise a total order. +- `f64` implements none of `Eq`, `Ord`, or `Hash`, so it can't key a `HashMap` or `BTreeMap` or use `.sort()`; use `sort_by(f64::total_cmp)` or an ordered wrapper. +- `Hash` must agree with `Eq` (`a == b` ⇒ equal hashes), or map lookups silently fail. +- Derived `PartialOrd`/`Ord` compare fields in **declaration order** (enums by variant order). + ### Async functions in traits - **Async fn in traits** (**1.75**) desugars to a method returning `impl Future`. @@ -80,16 +87,17 @@ What experienced Rust engineers forget before an interview, grouped by subtopic. - Every `Fn` is also `FnMut` and `FnOnce`; take the **loosest** bound your code allows (`FnOnce` if you call it once), so callers can pass more kinds of closures. - **`move`** changes *how* variables are captured (by value), not which trait is implemented — a `move` closure that only reads is still `Fn`. +- Async closures (`async || { ... }`, 1.85) can borrow from their captures across `.await`, which `|| async { ... }` couldn't; bound them with **`AsyncFn`/`AsyncFnMut`/`AsyncFnOnce`**. ### Iterators -- Iterator adapters (`map`, `filter`, …) are **lazy** and compile to the same code as a hand-written loop — a **zero-cost abstraction**; nothing runs until a consumer (`collect`, `sum`, `for`). -- `iter()` yields `&T`, `iter_mut()` `&mut T`, `into_iter()` `T` (consumes the collection). +- Iterator adapters (`map`, `filter`, …) are **lazy**: nothing runs until a consumer (`collect`, `sum`, `for`). +- They compile to the same code as a hand-written loop — a **zero-cost abstraction**. ### String vs &str -- **`String`** is an owned, growable UTF-8 buffer; **`&str`** is a borrowed slice of UTF-8 bytes — accept `&str` in parameters. - No indexing by integer (`s[0]` doesn't compile) because characters are variable-width; slicing at a non-char boundary **panics**. +- `len()` counts bytes; `chars()` yields Unicode scalar values, which still aren't user-perceived characters. ## Smart pointers and memory @@ -111,6 +119,12 @@ What experienced Rust engineers forget before an interview, grouped by subtopic. - **`AsRef`** is a cheap reference conversion for flexible parameters: `fn open>(p: P)`. - **`Borrow`** also promises identical `Eq`/`Hash`/`Ord`, which is why `HashMap::get` accepts a `&str`. +### Layout and niche optimization + +- **Niche optimization**: `Option<&T>`, `Option>`, and `Option` are the same size as the inner type — the forbidden null or zero value encodes `None`. +- The default `repr(Rust)` may reorder fields to cut padding; **`repr(C)`** fixes C field order for FFI. +- `&str`, `&[T]`, and `&dyn Trait` are two words (pointer + length or vtable); an enum is its largest variant plus a tag, unless a niche absorbs the tag. + ### Pin and Unpin - **`Pin

`** stops the pointee from moving **only if it's `!Unpin`**; most types are `Unpin` and move freely. @@ -123,6 +137,12 @@ What experienced Rust engineers forget before an interview, grouped by subtopic. - `?` returns early with `Err`, converting via **`From`**; on `Option` it returns `None`. - **`thiserror`** for libraries (typed enums callers can match); **`anyhow`** for applications (one dynamic error with context). +### Integer overflow and casts + +- Overflow **panics in debug builds and wraps in release** (unless `overflow-checks = true`), so state intent with `checked_*`, `wrapping_*`, `saturating_*`, or `overflowing_*`. +- `as` between integers truncates or reinterprets silently (`300_i32 as u8 == 44`, `-1_i32 as u32 == u32::MAX`); `try_from` fails instead. +- `as` from float to integer **saturates** (since 1.45), and NaN becomes 0. + ### Panics and unwinding - `panic!` **unwinds** by default, running destructors; `panic = "abort"` exits immediately — smaller binary, no recovery. @@ -156,6 +176,8 @@ What experienced Rust engineers forget before an interview, grouped by subtopic. - Futures are **lazy** — nothing runs until an executor (e.g. **Tokio**) polls them. - Holding a non-`Send` guard across `.await` makes the future non-`Send`, so it can't be spawned on a multi-threaded runtime. +- `tokio::spawn` needs a **`Send + 'static`** future: move owned data or `Arc` clones in, not references. +- Tokio's default runtime is multi-threaded with work stealing; `current_thread` plus a `LocalSet` runs non-`Send` futures; `JoinSet` spawns and awaits a group of tasks. ### Blocking and locks in async code @@ -169,10 +191,6 @@ What experienced Rust engineers forget before an interview, grouped by subtopic. - Dropping a Tokio `JoinHandle` **detaches** the spawned task instead — cancel with `abort()`. - **Cancel safety**: in `select!`, a losing branch is dropped mid-await — `read_line` into a buffer can lose data, `recv()` on a channel can't. -### Async closures - -- **Async closures** (`async || { ... }`, **1.85**) can borrow from their captures across `.await`, which `|| async { ... }` couldn't; bound them with **`AsyncFn`/`AsyncFnMut`/`AsyncFnOnce`**. - ## Unsafe and FFI ### What `unsafe` allows @@ -200,5 +218,5 @@ What experienced Rust engineers forget before an interview, grouped by subtopic. ### Lock files and editions - Commit **`Cargo.lock`** for binaries; since 2023, Cargo's guidance is to commit it for libraries too, as a CI baseline (dependents ignore it). -- Editions are opt-in, per-crate language changes; **edition 2024** is available since Rust 1.85. -- Edition 2024 enables **let chains** (`if let Some(x) = a && x > 0`, 1.88) and makes `impl Trait` in return position capture all in-scope lifetimes by default. +- Editions are opt-in, per-crate language changes; crates on different editions link together freely. +- **Edition 2024** enables **let chains** (`if let Some(x) = a && x > 0`, 1.88) and makes `impl Trait` in return position capture all in-scope lifetimes by default. diff --git a/docs/topics/security.md b/docs/topics/security.md index 2322010..3a879ad 100644 --- a/docs/topics/security.md +++ b/docs/topics/security.md @@ -36,6 +36,13 @@ What experienced engineers forget about application security before an interview - A proxy and the backend disagree on where a request ends (**`Content-Length` vs `Transfer-Encoding`**), so attacker bytes become the start of the next user's request. - Reject ambiguous requests at the edge, use HTTP/2 end to end, and keep proxies patched. +### Resource-exhaustion attacks + +- **ReDoS**: backtracking regex engines take exponential time on patterns like `(a+)+$` with crafted input; use linear-time engines (RE2, Go `regexp`, Rust `regex`) or match timeouts. +- Decompression bombs and XML **billion laughs** expand kilobytes into gigabytes — cap the decompressed size and disable entity expansion. +- Hash flooding sends colliding keys that make hash tables O(n) per operation; runtimes defend with randomized hash seeds (SipHash). +- Cap JSON nesting depth, GraphQL query depth and cost, and request sizes at the edge. + ## Browser-side attacks ### XSS @@ -47,6 +54,7 @@ What experienced engineers forget about application security before an interview - A **strict CSP** uses per-response nonces or hashes (`script-src 'nonce-…' 'strict-dynamic'`); host allowlists are routinely bypassed through JSONP endpoints and CDN-hosted gadgets. - **`'strict-dynamic'`** lets a trusted script load further scripts and makes supporting browsers ignore host allowlists. +- **`frame-ancestors`** blocks framing, which stops clickjacking (a hidden frame tricking the user into clicking the real site); `X-Frame-Options` is the legacy header. - Roll out with `Content-Security-Policy-Report-Only` first. ### CSRF @@ -54,15 +62,12 @@ What experienced engineers forget about application security before an interview - The browser auto-attaches cookies to a forged cross-site request. - Defend with framework **CSRF tokens** plus `SameSite=Lax`/`Strict` cookies (**Lax is Chrome's default**); XSS defeats CSRF defenses, so it's no substitute. -### Clickjacking - -- A hidden frame tricks the user into clicking the real site. -- Block framing with CSP **`frame-ancestors`** (or legacy `X-Frame-Options`). - ### CORS - CORS is a **browser read policy**, not authentication or a firewall — non-browser clients ignore it. - A credentialed response can't use `Access-Control-Allow-Origin: *`; reflecting any `Origin` with credentials is equivalent to no policy. +- **Simple requests** (GET, HEAD, or POST with a form or `text/plain` body and no custom headers) go out without a preflight: the server acts on them even though the browser hides the response, so CORS is no CSRF defense. +- Other requests are **preflighted** with `OPTIONS`; `Access-Control-Max-Age` caches the result. ### Open redirect @@ -102,6 +107,13 @@ What experienced engineers forget about application security before an interview - An attacker plants a known session ID before login, then shares the victim's session once they authenticate. - Always **issue a new session ID** on authentication and privilege change. +### Password reset and recovery + +- Reset tokens are single-use, short-lived, high-entropy, and **stored hashed** like passwords. +- Respond identically whether or not the account exists; revoke other sessions after a reset. +- Build the reset link from a configured origin, never the request's `Host` header — **host-header poisoning** sends the token to the attacker's domain. +- Recovery is often the weakest login path (SIM swap, support-desk social engineering): hold it to the same assurance as MFA. + ### Browser token storage - **`HttpOnly` cookie**: safe from XSS token theft, needs `SameSite` + CSRF tokens — the default for web apps. @@ -155,13 +167,11 @@ What experienced engineers forget about application security before an interview ## Cryptography -### Encoding vs encryption vs hashing +### Secure randomness -| | Reversible? | Needs a key? | Purpose | -|---|---|---|---| -| Encoding (Base64) | yes | no | representation | -| Encryption | yes, with the key | yes | confidentiality | -| Hashing | no | no | integrity, fingerprint | +- Tokens, session IDs, nonces, and reset codes need a **CSPRNG**: `crypto.getRandomValues`/`crypto.randomBytes`, Python `secrets`, Go `crypto/rand`. +- `Math.random`, Python `random` (Mersenne Twister, reconstructable from 624 outputs), and Go `math/rand` aren't designed for secrets. +- Use at least **128 bits** for unguessable tokens. ### AES-GCM nonces @@ -210,7 +220,7 @@ What experienced engineers forget about application security before an interview - **HSTS** forces HTTPS on later visits after one HTTPS response (preload lists cover the first visit). - **Certificate pinning** resists CA compromise but makes key rotation painful — mostly limited to mobile apps now. -## Supply chain +## Secure development ### Dependency attacks @@ -222,8 +232,6 @@ What experienced engineers forget about application security before an interview - An **SBOM** (SPDX, CycloneDX) lists what's inside an artifact, so a new CVE can be matched to affected builds. - **SLSA** levels grade build provenance; **Sigstore** (cosign) signs artifacts with short-lived keys tied to a CI identity. -## Threat modeling - ### STRIDE | STRIDE threat | Violates | @@ -236,9 +244,3 @@ What experienced engineers forget about application security before an interview | Elevation of privilege | Authorization | - STRIDE finds threats at each trust boundary; it doesn't rank them — pair it with impact/likelihood. - -### OWASP Top 10:2025 - -- Broken Access Control, Security Misconfiguration, **Software Supply Chain Failures** (new), Cryptographic Failures, Injection, Insecure Design. -- Authentication Failures, Software/Data Integrity Failures, Security Logging and Alerting Failures, **Mishandling of Exceptional Conditions** (new). -- An awareness list, not a test checklist — name the edition when citing it. diff --git a/docs/topics/solidity.md b/docs/topics/solidity.md index 60ccda9..97beab3 100644 --- a/docs/topics/solidity.md +++ b/docs/topics/solidity.md @@ -21,13 +21,12 @@ What experienced Solidity engineers forget before an interview, grouped by subto - `_` marks where the function body runs; a `return` in the body only leaves the body, so code after `_` still runs **after the return**. - Modifiers apply left to right; `_` may run several times, and if it never runs the body is skipped and return values stay at their defaults. - Legacy codegen **inlines** modifier code at every use — move the logic into an internal function to shrink bytecode; via-IR emits modifiers as functions. -- `virtual` modifiers are deprecated since 0.8.31, ahead of 0.9. ### Libraries and user-defined types - `internal` library functions are inlined into the caller (a `JUMP`); `public`/`external` ones run through **`DELEGATECALL`** into a separately deployed, linked library. -- Libraries have no state variables and can't receive ETH; `using L for T` attaches their functions to a type, and `global` (0.8.13, user-defined types only) applies it in every file. -- **User-defined value types** (`type Price is uint256;`, 0.8.8) are zero-cost wrappers that stop mixing units; `using {add as +} for Price global` binds operators (0.8.19). +- Libraries have no state variables and can't receive ETH; `using L for T` attaches their functions to a type, and `global` (user-defined types only) applies it in every file. +- **User-defined value types** (`type Price is uint256;`, 0.8.8) are zero-cost wrappers that stop mixing units; `using {add as +} for Price global` binds operators. ### Arithmetic and casts @@ -54,7 +53,6 @@ What experienced Solidity engineers forget before an interview, grouped by subto - **`transient`** state variables (0.8.28, value types only) compile to `TSTORE`/`TLOAD`: 100 gas, cleared at the end of the transaction — see [ethereum.md](ethereum.md). - They can't have initializers, and they **persist across calls within one transaction** — a lock or flag must be reset explicitly, or a later call in a batch sees it. -- **`layout at `** (0.8.29) moves a contract's storage to a custom base slot; `layout at erc7201("my.app")` (0.8.35) uses an ERC-7201 namespace's base slot. ### Mapping and delete gotchas @@ -109,11 +107,32 @@ What experienced Solidity engineers forget before an interview, grouped by subto - **`msg.sender == tx.origin`** no longer proves there is no code in the path: since EIP-7702 (Pectra, May 2025) a delegated EOA runs code and makes several calls per transaction. - `code.length == 0` doesn't prove an EOA either: it's 0 for a contract still in its constructor, and a 7702-delegated EOA has code. +### Admin keys and governance + +- **`Ownable2Step`** makes the new owner accept the transfer, so a typo in the address can't brick ownership. +- Route admin actions through a **timelock** controlled by a multisig, giving users a window to exit before a change lands. +- Flash-loan governance attacks borrow voting power for one transaction (Beanstalk, April 2022, ~$182M lost); snapshot voting power at proposal creation (`ERC20Votes` checkpoints) and add a voting delay. + ### Signature replay and malleability - Signed messages without a **nonce, chainId, and deadline** can be replayed on another chain or later. - ECDSA `s` can be flipped (`n − s`) into a second valid signature — enforce **low-`s`** and never use a signature as a unique ID. - Raw `ecrecover` returns `address(0)` on a bad signature, which matches an unset signer — use OpenZeppelin's `ECDSA`, which reverts and enforces low-`s`. +- Smart-contract wallets (Safe, ERC-4337 accounts) sign via **ERC-1271** `isValidSignature`; OpenZeppelin's `SignatureChecker` tries ECDSA, then ERC-1271, and ERC-6492 wraps signatures from wallets not yet deployed. + +### Denial of service + +- **Unbounded loops** over user-growable arrays eventually exceed the gas limit — paginate or cap the size. +- Pushing ETH to many receivers fails if one reverts (a contract without `receive`) — switch to **pull payments** (a withdraw function). +- A malicious callee can return huge data that Solidity copies into memory even when discarded — a **returnbomb**; cap the copy with an assembly `call`. + +### Randomness and block data + +- **`block.prevrandao`** (0.8.18, replaced `difficulty`) is known to the proposer, who can bias it by withholding a block — use **Chainlink VRF** or commit-reveal. +- `blockhash(n)` returns 0 for blocks older than **256**; EIP-2935 (Pectra) serves 8,191 hashes from a system contract, but the opcode is unchanged. +- Anything derived from block data can be computed by an attacker's contract in the same transaction, which reverts when the outcome is bad. + +## Integration risks ### Token integration quirks @@ -125,22 +144,15 @@ What experienced Solidity engineers forget before an interview, grouped by subto ### Oracle integration -- A DEX **spot price** can be moved within one transaction using a flash loan — use a TWAP or Chainlink; see [ethereum.md](ethereum.md). +- Never price assets off a DEX spot price — manipulation and TWAPs in [ethereum.md](ethereum.md). - Chainlink `latestRoundData`: check **staleness** (`updatedAt` against the feed's heartbeat) and `answer > 0`; on L2s also check the **sequencer uptime feed**. - Feeds have their own decimals (USD pairs use 8, ETH pairs 18) — scale before combining with token amounts. -### Denial of service +### Front-running and slippage -- **Unbounded loops** over user-growable arrays eventually exceed the gas limit — paginate or cap the size. -- Pushing ETH to many receivers fails if one reverts (a contract without `receive`) — switch to **pull payments** (a withdraw function). -- A malicious callee can return huge data that Solidity copies into memory even when discarded — a **returnbomb**; cap the copy with an assembly `call`. - -### Randomness and block data - -- **`block.prevrandao`** (0.8.18, replaced `difficulty`) is known to the proposer, who can bias it by withholding a block — use **Chainlink VRF** or commit-reveal. -- `blockhash(n)` returns 0 for blocks older than **256**; EIP-2935 (Pectra) serves 8,191 hashes from a system contract, but the opcode is unchanged. -- Anything derived from block data can be computed by an attacker's contract in the same transaction, which reverts when the outcome is bad. -- On L1, `block.timestamp` advances in fixed 12 s slots; L2 sequencers set it within looser bounds. +- Public-mempool transactions can be front-run or **sandwiched**: the attacker trades right before and after the victim's swap. +- Swap and liquidity functions take a **`minAmountOut`** and a `deadline` chosen off-chain by the user; a minimum computed on-chain from the current spot price protects nothing. +- Use commit-reveal for bids, auctions, and games where seeing a pending transaction gives an edge; MEV mechanics in [ethereum.md](ethereum.md). ### Rounding and precision @@ -156,13 +168,19 @@ What experienced Solidity engineers forget before an interview, grouped by subto - **Pack** small variables that are read and written together into one slot. - **`constant`/`immutable`** values are embedded in bytecode — no `SLOAD`. - Cache storage reads in local variables inside loops; use `calldata` parameters. -- Custom errors are cheaper than revert strings; 0.8.22 made simple loop-counter increments unchecked automatically. +- Custom errors are cheaper than revert strings; the compiler already skips overflow checks on simple loop-counter increments, so `unchecked { ++i; }` is no longer needed there. ### Code size limit - Deployed code is capped at **24,576 bytes** (EIP-170) and initcode at 49,152 (EIP-3860, Shanghai). - Workarounds: move logic into external libraries, split into several contracts, lower optimizer `runs`, or use the Diamond pattern. -- **EIP-7954**, scheduled for Glamsterdam, raises the limits to 64 KiB and 128 KiB. + +### Clones and factories + +- An **EIP-1167 minimal proxy** is 45 bytes of runtime code that `delegatecall`s a fixed implementation, so deploying many instances (vaults, wallets, pools) costs a fraction of a full deployment. +- Clones can't be upgraded and run no constructor: they need an initializer, and each call pays an extra `DELEGATECALL`. +- **`CREATE2`** sets the address to `keccak256(0xff ++ deployer ++ salt ++ keccak256(initcode))[12:]`, known before deployment (counterfactual wallets). +- Derive the salt from the caller and the init parameters, or a front-runner can take the predicted address with other arguments. ## Upgradeability @@ -198,7 +216,7 @@ What experienced Solidity engineers forget before an interview, grouped by subto - `require(cond, "msg")` and `revert("msg")` encode **`Error(string)`** (selector `0x08c379a0`). - `assert`, overflow, division by zero, and out-of-bounds access raise **`Panic(uint256)`** with codes `0x01`, `0x11`, `0x12`, `0x32`. - Before 0.8, `assert` hit the `INVALID` opcode and burned all remaining gas; now it reverts like any other error. -- **Custom errors** (0.8.4) encode a selector plus arguments; `require(cond, MyError())` works since 0.8.26 with via-IR and 0.8.27 with legacy codegen. +- **Custom errors** (0.8.4) encode a selector plus arguments; `require(cond, MyError())` accepts them too. ### try/catch limits @@ -249,6 +267,6 @@ What experienced Solidity engineers forget before an interview, grouped by subto ### Assembly gotchas - No overflow or bounds checks, and values narrower than 256 bits may carry **dirty upper bits** — mask them. -- Mark blocks **`assembly ("memory-safe")`** (0.8.13) so via-IR can still move variables to memory; the `/// @solidity memory-safe-assembly` comment is deprecated since 0.8.31. +- Mark blocks **`assembly ("memory-safe")`** (0.8.13) so via-IR can still move variables to memory. - `return` and `revert` in assembly end the whole call, not just the current function. - Storage variables expose `.slot` and `.offset`; calldata arrays expose `.offset` and `.length`. diff --git a/docs/topics/system.md b/docs/topics/system.md index 7b00174..99549c7 100644 --- a/docs/topics/system.md +++ b/docs/topics/system.md @@ -15,6 +15,7 @@ Key system design building blocks and trade-offs, grouped by subtopic. For CAP, - 1 day ≈ 86,400 s ≈ 10⁵ s, so **1M requests/day ≈ 12 QPS**; assume peak ≈ 2–3× average unless told otherwise. - 1 KB × 1M items = 1 GB; 1 KB × 1B = 1 TB. - Ballpark capacity (varies with workload): a well-indexed PostgreSQL box handles **10k+ simple queries/s**; Redis ~100k ops/s per core. +- **Little's law** L = λ × W: requests in flight = throughput × latency — 1,000 RPS × 200 ms = 200 concurrent requests, which sizes thread pools, connection pools, and queues. ## Traffic and edge @@ -79,7 +80,6 @@ Key system design building blocks and trade-offs, grouped by subtopic. For CAP, ### SQL vs NoSQL -- **SQL**: relational schema, ACID transactions, joins — the default for data needing correctness (payments, orders). - NoSQL families by access pattern: key-value, document, **wide-column** (heavy writes, time series), graph; model tables around queries, not entities. - Pick NoSQL only for a named reason (write volume, schema flexibility, access pattern). Internals in [database.md](database.md). @@ -127,7 +127,20 @@ Key system design building blocks and trade-offs, grouped by subtopic. For CAP, - Components in **series** multiply: 99.9% × 99.9% ≈ 99.8%. - Redundant components in **parallel** fail only together: 1 − 0.001² ≈ 99.9999%. - Both formulas assume **independent failures**; a shared zone or dependency wipes out most of the parallel redundancy gain. -- RPO (acceptable data loss) and RTO (acceptable recovery time) drive backup and failover design; active-passive vs active-active. + +### Multi-region and disaster recovery + +- **RPO** = acceptable data loss, **RTO** = acceptable recovery time; together they pick the tier. + +| Tier | What runs in the standby region | RTO / RPO | +|---|---|---| +| Backup and restore | nothing; restore from backups | hours / hours | +| Pilot light | replicated data, compute off | tens of minutes / minutes | +| Warm standby | a scaled-down copy | minutes / seconds | +| Active-active | full capacity everywhere | ~zero / ~zero with sync replication | + +- Async cross-region replication means RPO > 0; active-active also needs conflict handling or a home region per record, plus global routing (GeoDNS, anycast). +- A failover that was never rehearsed rarely works: run regular game days. ### SLIs, SLOs, error budgets @@ -137,17 +150,18 @@ Key system design building blocks and trade-offs, grouped by subtopic. For CAP, ### Observability -- Metrics: numeric time series (Prometheus); **golden signals** latency, traffic, errors, saturation — **RED** for services, **USE** for resources. +- Metrics: numeric time series (Prometheus); golden signals latency, traffic, errors, saturation — **RED** for services, **USE** for resources. - Logs: structured, centralized, tagged with a request ID. Traces: a request's span tree across services (OpenTelemetry). - Metrics say something is wrong, traces say where, logs say why. +- Percentiles **can't be averaged** across hosts or time windows: merge the histograms (or sketches), then take p99. +- High-cardinality labels (user ID, request ID) multiply Prometheus time series; keep them in logs and traces. ## Architecture ### Monolith vs microservices -- Monolith: simple to develop, deploy, and debug, with in-process calls and local transactions; struggles with many teams or very different scaling needs. -- Microservices: independent deploys and scaling, at the cost of network failures, distributed transactions, and operational overhead. -- Default to a **modular monolith**; extract services for a concrete reason. Boundaries mirror teams (**Conway's law**). +- Default to a **modular monolith**: in-process calls and local transactions, with module boundaries that could later become services. +- Extract a service for a concrete reason (independent scaling, deploy cadence, team ownership); boundaries mirror teams (**Conway's law**). ### Service discovery @@ -197,6 +211,13 @@ Key system design building blocks and trade-offs, grouped by subtopic. For CAP, - A **trie** (or prefix index) with precomputed **top-k suggestions** per node, built offline from query logs and cached aggressively. - Debounce on the client; for semantic matches, query an approximate nearest-neighbor index (HNSW). +### Video streaming + +- Upload to object storage → a **transcoding** pipeline splits the video into chunks and encodes them in parallel at several resolutions and bitrates. +- Package as **HLS or DASH**: 2–10 s segments plus a manifest listing the renditions, served from a CDN. +- The player picks a rendition per segment from measured bandwidth and buffer (**adaptive bitrate**), so quality drops instead of stalling. +- Pre-warm popular titles at the edge; the long tail streams from regional caches or the origin. + ### Notification system - One API → a queue per channel (push, SMS, email) → workers calling providers (APNs, FCM, Twilio) with retries. @@ -242,7 +263,6 @@ Key system design building blocks and trade-offs, grouped by subtopic. For CAP, ### Leaderboard - A Redis **sorted set**: `ZINCRBY` updates a score and `ZREVRANK` returns a user's rank in O(log n); `ZRANGE ... REV` reads the top k in O(log n + k). -- `ZRANGE ... REV` replaces `ZREVRANGE`, deprecated since Redis 6.2. - Beyond one node, shard by score range or keep per-shard top-k and merge. ### Booking and inventory contention diff --git a/docs/topics/typescript.md b/docs/topics/typescript.md index ce8cb4c..a1ff83f 100644 --- a/docs/topics/typescript.md +++ b/docs/topics/typescript.md @@ -4,10 +4,6 @@ What experienced TypeScript engineers forget before an interview, grouped by sub ## Type system semantics -### Structural typing - -- Compatibility is by **shape**, not declared name — an object satisfies an interface just by having the members; no `implements` needed. - ### Excess property checks - Only **fresh object literals** get excess-property errors: `const p: Point = { x: 1, y: 2, z: 3 }` fails. @@ -18,6 +14,18 @@ What experienced TypeScript engineers forget before an interview, grouped by sub - **`any`** turns checking off in both directions; **`unknown`** accepts anything but must be narrowed before use. - **`never`** is the empty type — the result of exhaustive narrowing and of functions that never return. +### `{}`, `object`, and index signatures + +- **`{}` means any non-nullish value** — strings and numbers included — not "empty object"; `Object` behaves the same. +- `object` is any non-primitive; `Record` is a truly empty object. +- With an index signature (`Record`), every key reads as present and typed `T` unless **`noUncheckedIndexedAccess`** is on. + +### Function assignability + +- A function returning a value is assignable to a **`void`-returning** function type, so `arr.forEach(x => out.push(x))` compiles and the result is ignored. +- A function with **fewer parameters** is assignable to one with more: callbacks may ignore arguments (`arr.map(x => x * 2)`). +- Only the contextual case is relaxed: a function declared with a `void` return type still can't return a value. + ### `interface` vs `type` - `interface` supports **declaration merging** (reopening adds members) and gives clearer error messages for object shapes. @@ -39,16 +47,13 @@ What experienced TypeScript engineers forget before an interview, grouped by sub - `typeof`, `instanceof`, `in`, equality, and truthiness checks narrow a variable inside the branch where they're provably true. - **Discriminated unions**: a shared literal tag (`kind`) lets `switch (x.kind)` narrow each case. +- **Exhaustiveness**: in the `default` branch, assign the value to a `never` variable; adding a new union member then fails to compile until handled. ### Custom type guards - A **type predicate** (`x is Fish`) or an **`asserts x is T`** function packages narrowing the compiler can't infer. - **Inferred type predicates** (5.5): a boolean-returning function that already narrows gets `x is T` automatically, e.g. `arr.filter(x => x !== undefined)`. -### Exhaustiveness with `never` - -- In a `default` branch, assign the value to a `never` variable; adding a new union member then fails to compile until handled. - ### `as` vs `satisfies` vs `!` - **`x as T`** is an unchecked assertion — it only rejects conversions between unrelated types (bypass: `as unknown as T`). @@ -79,7 +84,7 @@ What experienced TypeScript engineers forget before an interview, grouped by sub ### Branded types -- Structural typing makes `UserId` and `OrderId` both plain `string`s; **brand** them for nominal-like safety: `type UserId = string & { readonly __brand: 'UserId' }`. +- Types are compatible by **shape**, not declared name (structural typing), so `UserId` and `OrderId` are both plain `string`s; **brand** them for nominal-like safety: `type UserId = string & { readonly __brand: 'UserId' }`. - Create values only through a validating function (`asUserId(s)`); the brand has no runtime cost. ## Compilation and tooling @@ -113,7 +118,6 @@ What experienced TypeScript engineers forget before an interview, grouped by sub ### TypeScript 7 native compiler - **TypeScript 7** (2026) ships the Go port of the compiler (`tsgo` in preview): roughly **10×** faster builds with parallel checking. -- Its stable programmatic API is planned for **7.1**, so some tools (typescript-eslint, Vue/Svelte/Angular language tooling) still need the 6.x JS compiler. ## Strictness flags @@ -146,6 +150,12 @@ What experienced TypeScript engineers forget before an interview, grouped by sub ## Typing patterns +### Runtime validation at boundaries + +- Types are erased: `JSON.parse` and `res.json()` return `any`, and env vars or `as` casts are unchecked, so data can violate its declared type at runtime. +- Parse untrusted input with a schema library (Zod, Valibot, ArkType) and **derive the type from the schema** (`type User = z.infer`), so the two can't drift. +- Type unparsed input as **`unknown`**, not `any`, so it must be validated or narrowed before use. + ### Overloads - Several **overload signatures** followed by one implementation signature that callers can't see.