diff --git a/.agents/skills/quiz/SKILL.md b/.agents/skills/quiz/SKILL.md index e911fbd..04b767a 100644 --- a/.agents/skills/quiz/SKILL.md +++ b/.agents/skills/quiz/SKILL.md @@ -1,6 +1,6 @@ --- name: quiz -description: Run an interactive interview quiz from the topic briefs in docs/topics/. Use when the user asks to be quizzed or tested on a topic, e.g. "/quiz golang", "/quiz ethereum database", "/quiz all 20". +description: Run an interactive interview quiz from the topic briefs in docs/topics/. Use when the user asks to be quizzed or tested on a topic, e.g. "/quiz golang", "/quiz ethereum postgresql", "/quiz all 20". argument-hint: " [count]" --- diff --git a/.agents/skills/renovate/SKILL.md b/.agents/skills/renovate/SKILL.md index ae1fc6b..79dbef7 100644 --- a/.agents/skills/renovate/SKILL.md +++ b/.agents/skills/renovate/SKILL.md @@ -31,7 +31,7 @@ Without subagents, do the topics one after another. Wait until every topic has r Your training data is older than the latest releases. Take versions, dates, and behavior from current official sources (release notes, changelogs, spec or EIP pages, the project blog, docs via Context7), not from memory. Every change you make needs a source you have actually read in this session. 1. Read `AGENTS.md` (format rules, what to include and cut, topic ownership) and the whole brief. -2. Inventory what is versioned. List the subjects the brief depends on (e.g. ethereum: protocol forks, EIPs, client defaults; nextjs: Next.js, React; database: PostgreSQL, MySQL, Redis, PgBouncer) and every version-bound claim: `since X`, `X+`, version numbers, dates, "experimental", "deprecated", "default", "planned", "upcoming", and numbers that change between releases (limits, defaults, prices, thresholds). Fundamentals briefs (algorithms, distributed, system) have few of these; check the tools and products they name. +2. Inventory what is versioned. List the subjects the brief depends on (e.g. ethereum: protocol forks, EIPs, client defaults; nextjs: Next.js, React; postgresql: PostgreSQL, PgBouncer) and every version-bound claim: `since X`, `X+`, version numbers, dates, "experimental", "deprecated", "default", "planned", "upcoming", and numbers that change between releases (limits, defaults, prices, thresholds). Fundamentals briefs (algorithms, distributed, system) have few of these; check the tools and products they name. 3. Find the latest stable release of each subject and the newest version the brief mentions. Read the release notes for every version in between. Prereleases (beta, RC, nightly, draft EIPs, proposals) don't count as shipped. Mention one only when the brief already does, or when interviewers already ask about it, and label it with its status (e.g. "planned for Fusaka", "TypeScript 7 beta"). 4. Check the brief's claims and prose against the sources, starting with the inventory from step 2, and fix what is wrong, stale, or misleading. Keep historical version distinctions that are still useful ("before Go 1.22"). Look in particular for: - an experimental or opt-in feature that is now the default or removed; diff --git a/AGENTS.md b/AGENTS.md index 0caf4ff..5451b7c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -79,24 +79,25 @@ Keep facts that a working engineer knew once but loses without daily use: ## Topic ownership -Each concept has one owner file that covers it in full. Other files may overlap where the concept matters to them too: a bullet or a short concept stating what matters from their angle (database says the WAL is only durable after `fsync`; os explains `fsync`), with a link to the owner for the rest. Don't copy the owner's concept wholesale — copies drift apart, and the quiz would ask the same question twice. +Each concept has one owner file that covers it in full. Other files may overlap where the concept matters to them too: a bullet or a short concept stating what matters from their angle (postgresql says the WAL is only durable after `fsync`; os explains `fsync`), with a link to the owner for the rest. Don't copy the owner's concept wholesale — copies drift apart, and the quiz would ask the same question twice. | Concept | Owner | Referenced from | |---|---|---| -| Consistency models, replication theory, consensus, clocks (monotonic vs wall), delivery semantics, sagas, CRDTs, CDC | distributed | database, system, backend, os, golang | +| Consistency models, replication theory, consensus, clocks (monotonic vs wall), delivery semantics, sagas, CRDTs, CDC | distributed | postgresql, redis, system, backend, os, golang | | Retries, backoff, circuit breakers, backpressure, tail latency | distributed | backend, system | -| Isolation levels, MVCC, indexes, WAL, DB replication setup, pooling | database | backend, system | -| Caching strategies, rate-limiting algorithms, load balancing, CDN, observability (SLI/SLO), capacity math (Little's law), multi-region and DR tiers | system | backend, devops, os, aws | -| REST/gRPC/GraphQL, pagination, idempotency keys, outbox, graceful shutdown, N+1, schema evolution, multi-tenancy, time handling | backend | system, database, distributed | +| Isolation levels, MVCC, indexes, WAL, DB replication setup, pooling, SQL | postgresql | backend, system, devops | +| Redis execution model, data structures, eviction, persistence, Sentinel and Cluster, streams | redis | system, distributed | +| Caching strategies, rate-limiting algorithms, load balancing, CDN, observability (SLI/SLO), capacity math (Little's law), multi-region and DR tiers, storage engines (B-tree vs LSM), search and OLAP stores | system | backend, devops, os, aws, postgresql, redis | +| REST/gRPC/GraphQL, pagination, idempotency keys, outbox, graceful shutdown, N+1, schema evolution, multi-tenancy, time handling | backend | system, postgresql, distributed | | AuthN, sessions, OAuth/OIDC, JWT, XSS, CSRF, CORS, clickjacking, crypto, TLS | security | backend, frontend, nextjs | -| Containers, Kubernetes, cloud, IaC, deployment strategies, zero-downtime migrations | devops | backend, database | +| Containers, Kubernetes, cloud, IaC, deployment strategies, zero-downtime migrations | devops | backend, postgresql | | Browser event loop, JS language semantics | javascript | frontend, typescript | | Rendering strategies, browser pipeline, Web Vitals, HTTP caching in the browser, browser storage | frontend | javascript, security, react, nextjs | | React rendering model, hooks, client state management, Suspense and transitions, Server Components and Server Functions, hydration | react | frontend, nextjs | | Next.js App Router, rendering and caching layers, `use cache`, routing conventions, Server Actions configuration, proxy, self-hosting | nextjs | react, frontend | | asyncio event loop | python | — | | TCP/UDP, HTTP versions, QUIC, DNS, IP addressing, NAT, MTU, BGP/anycast | networking | backend, system, devops | -| Processes, signals, scheduling, virtual memory, OOM, file descriptors, epoll/io_uring, fsync, locks | os | devops, database, golang | +| Processes, signals, scheduling, virtual memory, OOM, file descriptors, epoll/io_uring, fsync, locks | os | devops, postgresql, redis, golang | | Container internals (namespaces, cgroups), Kubernetes graceful shutdown | devops | os, backend | | Solidity language, storage layout, calls and ABI, contract security, gas optimization, upgradeability | solidity | ethereum | | EVM execution, gas and fee market, transaction types, signatures and token standards, DeFi (flash loans, oracle manipulation) and MEV | ethereum | solidity | diff --git a/README.md b/README.md index ca948d0..dfac0f5 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,6 @@ Each topic in `docs/topics/` is a brief grouped by subtopic, covering what an ex ### Fundamentals - [Algorithms](docs/topics/algorithms.md) — complexity, arrays and sequences, strings, sorting, data structures, graphs, dynamic programming, greedy and backtracking, NP-hardness -- [Databases](docs/topics/database.md) — indexes, isolation, MVCC and locking, storage engines, query tuning, SQL gotchas, Redis and NoSQL, schema design, scaling - [Distributed Systems](docs/topics/distributed.md) — theory, consistency models, replication, partitioning, consensus, clocks and IDs, transactions, conflict resolution, failure handling, Kafka and stream processing - [Networking](docs/topics/networking.md) — TCP, keepalive and congestion control, HTTP/1.1 vs 2 vs 3, QUIC, DNS, IP addressing, NAT, MTU, BGP and anycast - [Operating Systems](docs/topics/os.md) — processes and signals, scheduling and cgroup throttling, virtual memory, OOM and cgroup limits, NUMA, file descriptors, epoll and io_uring, fsync, synchronization, perf tools @@ -38,6 +37,8 @@ Each topic in `docs/topics/` is a brief grouped by subtopic, covering what an ex - [AWS](docs/topics/aws.md) — IAM and KMS, VPC networking, compute, storage and databases, events, operations - [DevOps](docs/topics/devops.md) — deployment strategies, containers, Kubernetes workloads/storage/networking/operations, Terraform, CI/CD and GitOps +- [PostgreSQL](docs/topics/postgresql.md) — indexes, isolation, MVCC and VACUUM, locking and schema changes, query tuning, SQL gotchas, WAL and backups, replication, pooling and partitioning, schema design +- [Redis](docs/topics/redis.md) — execution model, transactions and scripts, data structures, expiration and eviction, persistence, replication and Sentinel, Cluster, pub/sub and streams, locks and counters ### Domains diff --git a/docs/index.md b/docs/index.md index 660d6fd..6656254 100644 --- a/docs/index.md +++ b/docs/index.md @@ -7,7 +7,6 @@ Topic briefs on what engineers forget before technical interviews, grouped by su ### Fundamentals - [Algorithms](topics/algorithms.md) — complexity, arrays and sequences, strings, sorting, data structures, graphs, dynamic programming, greedy and backtracking, NP-hardness -- [Databases](topics/database.md) — indexes, isolation, MVCC and locking, storage engines, query tuning, SQL gotchas, Redis and NoSQL, schema design, scaling - [Distributed Systems](topics/distributed.md) — theory, consistency models, replication, partitioning, consensus, clocks and IDs, transactions, conflict resolution, failure handling, Kafka and stream processing - [Networking](topics/networking.md) — TCP, keepalive and congestion control, HTTP/1.1 vs 2 vs 3, QUIC, DNS, IP addressing, NAT, MTU, BGP and anycast - [Operating Systems](topics/os.md) — processes and signals, scheduling and cgroup throttling, virtual memory, OOM and cgroup limits, NUMA, file descriptors, epoll and io_uring, fsync, synchronization, perf tools @@ -34,6 +33,8 @@ Topic briefs on what engineers forget before technical interviews, grouped by su - [AWS](topics/aws.md) — IAM and KMS, VPC networking, compute, storage and databases, events, operations - [DevOps](topics/devops.md) — deployment strategies, containers, Kubernetes workloads/storage/networking/operations, Terraform, CI/CD and GitOps +- [PostgreSQL](topics/postgresql.md) — indexes, isolation, MVCC and VACUUM, locking and schema changes, query tuning, SQL gotchas, WAL and backups, replication, pooling and partitioning, schema design +- [Redis](topics/redis.md) — execution model, transactions and scripts, data structures, expiration and eviction, persistence, replication and Sentinel, Cluster, pub/sub and streams, locks and counters ### Domains diff --git a/docs/topics/algorithms.md b/docs/topics/algorithms.md index ea03c37..a5b5efe 100644 --- a/docs/topics/algorithms.md +++ b/docs/topics/algorithms.md @@ -1,6 +1,6 @@ # Algorithms -What experienced engineers forget before an algorithms interview, grouped by subtopic. Bounds follow [MIT 6.006](https://ocw.mit.edu/courses/6-006-introduction-to-algorithms-spring-2020/), [Princeton Algorithms](https://algs4.cs.princeton.edu/), and the [USACO Guide](https://usaco.guide/). For database indexes see [database.md](database.md); for rate limiting and load balancing see [system.md](system.md); for consensus see [distributed.md](distributed.md). +What experienced engineers forget before an algorithms interview, grouped by subtopic. Bounds follow [MIT 6.006](https://ocw.mit.edu/courses/6-006-introduction-to-algorithms-spring-2020/), [Princeton Algorithms](https://algs4.cs.princeton.edu/), and the [USACO Guide](https://usaco.guide/). For database indexes see [postgresql.md](postgresql.md); for rate limiting and load balancing see [system.md](system.md); for consensus see [distributed.md](distributed.md). ## Complexity and problem solving diff --git a/docs/topics/aws.md b/docs/topics/aws.md index cf36cef..f189997 100644 --- a/docs/topics/aws.md +++ b/docs/topics/aws.md @@ -1,6 +1,6 @@ # AWS -What experienced AWS developers and DevOps engineers forget before an interview, grouped by subtopic; facts follow the AWS documentation. For database internals see [database.md](database.md); for delivery semantics and retries see [distributed.md](distributed.md); for deployment patterns and Terraform see [devops.md](devops.md). +What experienced AWS developers and DevOps engineers forget before an interview, grouped by subtopic; facts follow the AWS documentation. For PostgreSQL and Redis internals see [postgresql.md](postgresql.md) and [redis.md](redis.md); for delivery semantics and retries see [distributed.md](distributed.md); for deployment patterns and Terraform see [devops.md](devops.md). ## Identity and access diff --git a/docs/topics/backend.md b/docs/topics/backend.md index 8bef518..e080af8 100644 --- a/docs/topics/backend.md +++ b/docs/topics/backend.md @@ -1,6 +1,6 @@ # Backend -What experienced backend engineers forget before an interview, grouped by subtopic. For database internals see [database.md](database.md); for queues, sagas, and retries see [distributed.md](distributed.md); for caching and rate limiting see [system.md](system.md); for HTTP versions see [networking.md](networking.md); for auth and vulnerabilities see [security.md](security.md). +What experienced backend engineers forget before an interview, grouped by subtopic. For PostgreSQL internals see [postgresql.md](postgresql.md); for Redis see [redis.md](redis.md); for queues, sagas, and retries see [distributed.md](distributed.md); for caching and rate limiting see [system.md](system.md); for HTTP versions see [networking.md](networking.md); for auth and vulnerabilities see [security.md](security.md). ## API design @@ -123,7 +123,7 @@ What experienced backend engineers forget before an interview, grouped by subtop ### Connection pool sizing - Size the [connection pool](https://en.wikipedia.org/wiki/Connection_pool) from the database's capacity, not the app's thread count — many app instances × a large pool overwhelm the database. -- A good starting point is a small pool per instance (~2× DB cores total across instances, [HikariCP's formula](https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing)), then measure wait time; see [database.md](database.md). +- A good starting point is a small pool per instance (~2× DB cores total across instances, [HikariCP's formula](https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing)), then measure wait time; server-side pooling in [postgresql.md](postgresql.md). ### Transaction boundaries diff --git a/docs/topics/database.md b/docs/topics/database.md deleted file mode 100644 index d707a50..0000000 --- a/docs/topics/database.md +++ /dev/null @@ -1,253 +0,0 @@ -# Databases - -What experienced database engineers forget before an interview, grouped by subtopic. For SQL vs NoSQL and the scaling ladder, see [system.md](system.md); for replication theory and sharding, see [distributed.md](distributed.md). - -## Indexes - -### B+tree indexes - -- The default index is a [B+tree](https://en.wikipedia.org/wiki/B%2B_tree): balanced, sorted, O(log n), serves range scans and `ORDER BY`. -- [Clustered](https://dev.mysql.com/doc/refman/8.4/en/innodb-index-types.html) (InnoDB): rows are stored in primary-key order and secondary indexes point to the PK. [Heap](https://www.postgresql.org/docs/current/storage-page-layout.html) (PostgreSQL): rows are unordered and indexes point to a physical location ([`ctid`](https://www.postgresql.org/docs/current/ddl-system-columns.html#DDL-SYSTEM-COLUMNS-CTID)). - -### Composite indexes - -- [Leftmost-prefix rule](https://dev.mysql.com/doc/refman/8.4/en/multiple-column-indexes.html): `(a, b, c)` seeks efficiently on `a`, `a,b`, `a,b,c` — not on `b` alone. -- [Skip scan](https://dev.mysql.com/doc/refman/8.4/en/range-optimization.html#range-access-skip-scan) (MySQL 8.0.13+, [PostgreSQL 18+](https://www.postgresql.org/docs/current/indexes-multicolumn.html)) serves `b` alone by probing each distinct `a`; cheap only when `a` has few values. -- Put equality columns first, range columns last; columns after a range condition only filter inside the scanned range (unless skip scan applies). - -### Covering indexes - -- A [covering index](https://www.postgresql.org/docs/current/indexes-index-only-scans.html) holds every column the query needs, enabling an index-only scan ([`INCLUDE`](https://www.postgresql.org/docs/current/sql-createindex.html) adds payload columns in PostgreSQL). -- PostgreSQL still checks the [visibility map](https://www.postgresql.org/docs/current/storage-vm.html); recently modified pages force heap visits until [VACUUM](https://www.postgresql.org/docs/current/routine-vacuuming.html) runs. - -### Partial and expression indexes - -- [Partial](https://www.postgresql.org/docs/current/indexes-partial.html) (`WHERE deleted_at IS NULL`) and [expression](https://www.postgresql.org/docs/current/indexes-expressional.html) (`ON lower(email)`) indexes cover only what queries use. -- [Sargability](https://en.wikipedia.org/wiki/Sargable): a function or implicit cast on an indexed column (`WHERE lower(email) = ...`) blocks index use unless a matching expression index exists. - -### Specialized index types - -| Index | Use for | -|---|---| -| [GIN](https://www.postgresql.org/docs/current/gin.html) | full-text, JSONB, arrays ([inverted index](https://en.wikipedia.org/wiki/Inverted_index)) | -| [GiST](https://www.postgresql.org/docs/current/gist.html) | geometry, ranges, nearest neighbor | -| [BRIN](https://www.postgresql.org/docs/current/brin.html) | huge, naturally ordered tables (append-only time series); tiny | -| [Hash](https://www.postgresql.org/docs/current/hash-index.html) | equality only | - -### Building indexes online - -- Plain `CREATE INDEX` blocks writes for the whole build; [`CREATE INDEX CONCURRENTLY`](https://www.postgresql.org/docs/current/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY) doesn't, but is slower, can't run in a transaction, and leaves an `INVALID` index if it fails. -- Index every [foreign key](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-FK) you join or cascade-delete on — PostgreSQL doesn't create them automatically. - -## Transactions and isolation - -### Anomalies × isolation levels - -| Level | [Dirty read](https://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Dirty_reads) | [Non-repeatable read](https://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Non-repeatable_reads) | [Phantom read](https://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Phantom_reads) | -|---|---|---|---| -| [Read Uncommitted](https://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Read_uncommitted) | possible | possible | possible | -| [Read Committed](https://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Read_committed) | — | possible | possible | -| [Repeatable Read](https://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Repeatable_reads) | — | — | possible (per SQL standard) | -| [Serializable](https://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Serializable) | — | — | — | - -### Lost update and write skew - -- Lost update: two read-modify-write cycles overwrite each other. -- [Write skew](https://en.wikipedia.org/wiki/Snapshot_isolation): two transactions read overlapping data and write different rows, together breaking an invariant (both doctors go off call). - -### PostgreSQL isolation - -- Default: [Read Committed](https://www.postgresql.org/docs/current/transaction-iso.html#XACT-READ-COMMITTED); [Repeatable Read](https://www.postgresql.org/docs/current/transaction-iso.html#XACT-REPEATABLE-READ) is [snapshot isolation](https://en.wikipedia.org/wiki/Snapshot_isolation) — no phantoms, and a concurrent update of the same row aborts (no lost update), but write skew is possible. -- Serializable is [SSI](https://www.postgresql.org/docs/current/transaction-iso.html#XACT-SERIALIZABLE): it prevents write skew by aborting with a serialization error, so the app must retry. - -### InnoDB isolation and MVCC - -- Default: [Repeatable Read](https://dev.mysql.com/doc/refman/8.4/en/innodb-transaction-isolation-levels.html#isolevel_repeatable-read) with [next-key locks](https://dev.mysql.com/doc/refman/8.4/en/innodb-locking.html#innodb-next-key-locks) (record + gap) on locking reads, blocking many phantoms. -- Plain `SELECT` reads the [snapshot](https://dev.mysql.com/doc/refman/8.4/en/innodb-consistent-read.html), but `UPDATE` sees the latest committed row — lost updates slip through without [`SELECT ... FOR UPDATE`](https://dev.mysql.com/doc/refman/8.4/en/innodb-locking-reads.html). -- Updates happen in place; older versions are rebuilt from the [undo log](https://dev.mysql.com/doc/refman/8.4/en/innodb-undo-logs.html), and a long transaction makes undo history grow. - -## MVCC and locking - -### PostgreSQL MVCC - -- Plain `SELECT`s read a [snapshot](https://www.postgresql.org/docs/current/mvcc-intro.html) and neither block nor wait for row writes; writers (and `FOR UPDATE`) still lock rows against each other. -- Old row versions stay in the table, tagged [`xmin`/`xmax`](https://www.postgresql.org/docs/current/ddl-system-columns.html#DDL-SYSTEM-COLUMNS-XMIN); [VACUUM](https://www.postgresql.org/docs/current/routine-vacuuming.html) reclaims them, and a long-running transaction blocks it, causing bloat. -- [XID wraparound](https://www.postgresql.org/docs/current/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND) (32-bit transaction IDs) forces aggressive anti-wraparound vacuuming. -- [HOT updates](https://www.postgresql.org/docs/current/storage-hot.html) skip index maintenance when no indexed column changed and the page has room. - -### Row locks, queue tables, advisory locks - -- [`SELECT ... FOR UPDATE`](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-ROWS) locks the rows it returns; [`SKIP LOCKED`](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE) lets workers claim different rows of a queue table without blocking. -- PostgreSQL [advisory locks](https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS) are app-defined locks on a number, not a row — for leader election or job dedup without a table to lock. - -### Optimistic locking - -- Add a [`version` column](https://en.wikipedia.org/wiki/Optimistic_concurrency_control): `UPDATE ... SET version = version + 1 WHERE id = ? AND version = ?`; zero rows updated means someone else won — reload and retry. -- Best when conflicts are rare; pessimistic locks when they're frequent. - -### Deadlocks - -- Two transactions each hold a lock the other needs; the database [detects the cycle](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-DEADLOCKS) and aborts one. -- Prevent with a consistent lock order and short transactions. - -### DDL locks and the lock queue - -- Most `ALTER TABLE` forms take an [`ACCESS EXCLUSIVE`](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-TABLES) lock. If it waits behind a long query, every later query queues behind it — a brief outage from a "fast" migration. -- Set [`lock_timeout`](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-LOCK-TIMEOUT) (a few seconds) on migrations and retry; rollout patterns in [devops.md](devops.md). - -## Storage engines and durability - -### B-tree vs LSM-tree - -| | [B-tree](https://en.wikipedia.org/wiki/B-tree) (PostgreSQL, InnoDB) | [LSM-tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree) ([RocksDB](https://rocksdb.org/), [Cassandra](https://cassandra.apache.org/doc/latest/cassandra/architecture/storage-engine.html)) | -|---|---|---| -| Writes | in-place page updates, random I/O | append to [memtable](https://cassandra.apache.org/doc/latest/cassandra/architecture/storage-engine.html#memtables) → flush immutable [SSTables](https://cassandra.apache.org/doc/latest/cassandra/architecture/storage-engine.html#sstables) | -| Reads | one tree walk, predictable | may check several files; [bloom filters](https://en.wikipedia.org/wiki/Bloom_filter) skip most | -| Background work | page splits, [vacuum](https://www.postgresql.org/docs/current/routine-vacuuming.html) | [compaction](https://github.com/facebook/rocksdb/wiki/Compaction) | -| Fits | read-heavy, range scans | write-heavy | - -- Compaction strategy trades [read, write, and space amplification](https://smalldatum.blogspot.com/2015/11/read-write-space-amplification-pick-2_23.html) against each other. - -### Write-ahead log - -- A commit is durable once its [WAL](https://www.postgresql.org/docs/current/wal-intro.html) record is [fsynced](os.md); dirty data pages are flushed later ([checkpoints](https://www.postgresql.org/docs/current/wal-configuration.html), background writer), never before their WAL. -- Crash recovery replays WAL from the last checkpoint: PostgreSQL only redoes (uncommitted rows stay invisible by commit status); InnoDB redoes, then rolls back with [undo](https://dev.mysql.com/doc/refman/8.4/en/innodb-undo-logs.html). -- [Group commit](https://www.postgresql.org/docs/current/wal-configuration.html) batches many commits into one fsync. - -### Backups and PITR - -- Logical backups ([`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html)): portable, slow to restore at scale. Physical ([`pg_basebackup`](https://www.postgresql.org/docs/current/app-pgbasebackup.html), snapshots): fast restore, same major version. -- [PITR](https://www.postgresql.org/docs/current/continuous-archiving.html) = physical base backup + [WAL](https://www.postgresql.org/docs/current/wal-intro.html) replayed to a chosen moment. Replicas are not backups — they replicate mistakes too. - -## Query execution and tuning - -### Logical evaluation order - -`FROM/JOIN → WHERE → GROUP BY → HAVING → SELECT → DISTINCT → ORDER BY → LIMIT` — why a `SELECT` alias works in `ORDER BY` but not in `WHERE`. - -### Why an index isn't used - -- Low selectivity: when a filter matches a large share of rows, the planner prefers a [sequential scan](https://www.postgresql.org/docs/current/using-explain.html), since one sequential pass beats many random heap reads. -- `LIKE 'x%'` uses a B-tree only with C collation or [`text_pattern_ops`](https://www.postgresql.org/docs/current/indexes-opclass.html); `LIKE '%x'` needs a trigram GIN or GiST index ([`pg_trgm`](https://www.postgresql.org/docs/current/pgtrgm.html)). -- A function or implicit cast on the column, and `OR` across columns without an index on each, also defeat it. -- Stale statistics show up in [`EXPLAIN ANALYZE`](https://www.postgresql.org/docs/current/using-explain.html#USING-EXPLAIN-ANALYZE) as estimated vs actual rows far apart — run [`ANALYZE`](https://www.postgresql.org/docs/current/sql-analyze.html); also watch for sorts or hashes spilling to disk. -- Scan types: Seq Scan, Index Scan, Index Only Scan, and [Bitmap Heap Scan](https://www.postgresql.org/docs/current/indexes-bitmap-scans.html) (collects matches from one or more indexes, then reads pages in physical order). -- `OFFSET` still reads and discards every skipped row, so deep pages get slower; [keyset pagination](https://use-the-index-luke.com/no-offset) seeks by index — API side in [backend.md](backend.md). - -### Join algorithms - -| Join | Best when | -|---|---| -| [Nested loop](https://en.wikipedia.org/wiki/Nested_loop_join) | outer side small, inner side index-backed | -| [Hash join](https://en.wikipedia.org/wiki/Hash_join) | large equality joins without a useful index; builds on the smaller side | -| [Merge join](https://en.wikipedia.org/wiki/Sort-merge_join) | both inputs already sorted (e.g. from indexes) | - -## SQL gotchas - -### NULL and three-valued logic - -- Any comparison with NULL — even `NULL = NULL` — is [UNKNOWN](https://en.wikipedia.org/wiki/Null_%28SQL%29#Comparisons_with_NULL_and_the_three-valued_logic_%283VL%29), and `WHERE` drops it; use `IS [NOT] NULL` or [`IS [NOT] DISTINCT FROM`](https://www.postgresql.org/docs/current/functions-comparison.html). -- [`NOT IN`](https://www.postgresql.org/docs/current/functions-subquery.html#FUNCTIONS-SUBQUERY-NOTIN) with a NULL in the list returns no rows — use [`NOT EXISTS`](https://www.postgresql.org/docs/current/functions-subquery.html#FUNCTIONS-SUBQUERY-EXISTS). -- [`COUNT(*)`](https://www.postgresql.org/docs/current/functions-aggregate.html) counts rows; `COUNT(col)`, `SUM`, `AVG`, `MIN`/`MAX` skip NULLs, but `array_agg`/`json_agg` keep them. - -### Window functions - -- [`fn() OVER (PARTITION BY ... ORDER BY ...)`](https://www.postgresql.org/docs/current/tutorial-window.html) computes per row without collapsing rows. -- [`ROW_NUMBER`](https://www.postgresql.org/docs/current/functions-window.html) 1,2,3; `RANK` 1,1,3 (gaps); `DENSE_RANK` 1,1,2. -- The [default frame](https://www.postgresql.org/docs/current/sql-expressions.html#SYNTAX-WINDOW-FUNCTIONS) with `ORDER BY` is `RANGE ... CURRENT ROW`, which includes all peer rows with equal sort keys — use `ROWS` for a true running total. - -### CTE materialization - -- Since PostgreSQL 12, a non-recursive, side-effect-free CTE referenced once is [inlined](https://www.postgresql.org/docs/current/queries-with.html#QUERIES-WITH-CTE-MATERIALIZATION) into the outer query; one referenced several times is materialized. -- `MATERIALIZED` / `NOT MATERIALIZED` override the default. - -### Join pitfalls - -- A condition on the right table in `WHERE` (other than `IS NULL`) discards the NULL-extended rows and turns a [`LEFT JOIN`](https://www.postgresql.org/docs/current/queries-table-expressions.html#QUERIES-JOIN) into an inner join; put it in `ON`. -- Joining a parent to two one-to-many children multiplies rows (fan-out) and inflates `SUM`/`COUNT`; aggregate each child in a subquery first. -- [`EXISTS`](https://www.postgresql.org/docs/current/functions-subquery.html#FUNCTIONS-SUBQUERY-EXISTS) / `NOT EXISTS` are [semi-](https://en.wikipedia.org/wiki/Join_%28relational_algebra%29#Semijoin) and [anti-joins](https://en.wikipedia.org/wiki/Join_%28relational_algebra%29#Antijoin): they never duplicate rows, unlike `JOIN` + `DISTINCT`. - -### Upsert - -[`INSERT ... ON CONFLICT (...) DO UPDATE`](https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT) (PostgreSQL) / [`ON DUPLICATE KEY UPDATE`](https://dev.mysql.com/doc/refman/8.4/en/insert-on-duplicate.html) (MySQL) replaces a racy check-then-write. - -## Non-relational stores - -### Redis - -- Commands execute on one thread, so each command (and [Lua script](https://redis.io/docs/latest/develop/programmability/eval-intro/) or [`MULTI` block](https://redis.io/docs/latest/develop/using-commands/transactions/)) is atomic; a slow command ([`KEYS *`](https://redis.io/docs/latest/commands/keys/), big `SMEMBERS`) blocks everyone. -- Structures: strings, hashes, lists, sets, [sorted sets](https://redis.io/docs/latest/develop/data-types/sorted-sets/) ([skip list](https://en.wikipedia.org/wiki/Skip_list) + hash — leaderboards, rate limiters), [streams](https://redis.io/docs/latest/develop/data-types/streams/), [HyperLogLog](https://redis.io/docs/latest/develop/data-types/probabilistic/hyperloglogs/). -- Redis Cluster splits keys into [16,384 hash slots](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/); multi-key operations need the same slot, forced with [hash tags](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/#hash-tags) (`{user42}:cart`). - -### Redis persistence and replication - -- [RDB](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/): periodic fork + snapshot ([copy-on-write](https://en.wikipedia.org/wiki/Copy-on-write)), loses writes since the last one. [AOF](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/#append-only-file): logs every write; `appendfsync everysec` (default) loses up to ~1 s. -- Replication is [asynchronous](https://redis.io/docs/latest/operate/oss_and_stack/management/replication/) — failover can drop acknowledged writes. - -### Redis eviction, expiry, and messaging - -- [`maxmemory-policy`](https://redis.io/docs/latest/develop/reference/eviction/#eviction-policies) defaults to `noeviction`: at the memory limit, writes fail with an OOM error. -- Caches use `allkeys-lru` or `allkeys-lfu`; `volatile-*` policies evict only keys with a TTL, so without TTLs they behave like `noeviction`. -- Expired keys are removed [lazily](https://redis.io/docs/latest/commands/expire/) on access plus by background sampling, so memory lags behind expirations. -- [Pub/sub](https://redis.io/docs/latest/develop/pubsub/) is fire-and-forget (offline subscribers miss messages); [Streams](https://redis.io/docs/latest/develop/data-types/streams/) persist entries, with consumer groups, acknowledgments, and redelivery of pending entries. - -### Document stores (MongoDB) - -- [Embed](https://www.mongodb.com/docs/manual/data-modeling/embedding/) data that is read together (one read, atomic update); reference unbounded or shared data — a document is capped at [16 MB](https://www.mongodb.com/docs/manual/reference/limits/#bson-documents). -- Single-document writes are atomic; [multi-document transactions](https://www.mongodb.com/docs/manual/core/transactions/) (replica sets since 4.0, sharded clusters since 4.2) cost more, so design for single-document atomicity. -- [Write concern](https://www.mongodb.com/docs/manual/reference/write-concern/) `w: "majority"` is the default since 5.0 (except some arbiter setups); `w: 1` acknowledges before replication, so a failover can roll back acknowledged writes. -- A query without a matching index scans the collection ([`COLLSCAN`](https://www.mongodb.com/docs/manual/reference/explain-results/#collection-scan)); order compound index keys by the [ESR rule](https://www.mongodb.com/docs/manual/tutorial/equality-sort-range-guideline/): Equality, Sort, Range. - -### Wide-column key design (Cassandra, DynamoDB) - -- The [partition key](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-design.html) picks the node; the sort/clustering key orders rows inside the partition — an efficient query must supply the partition key; otherwise it's a full scan (DynamoDB [`Scan`](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html), Cassandra [`ALLOW FILTERING`](https://cassandra.apache.org/doc/latest/cassandra/developing/cql/dml.html#allow-filtering)) or a secondary index. -- Model [tables per query](https://cassandra.apache.org/doc/latest/cassandra/developing/data-modeling/index.html) (denormalized), not per entity; unbounded partitions (all events of a popular user) become hot and huge — add a time bucket to the key. -- Deletes write [tombstones](https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compaction/tombstones.html), which slow reads until [compaction](https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compaction/index.html) purges them. - -### Search engines and inverted indexes - -- An [inverted index](https://en.wikipedia.org/wiki/Inverted_index) maps each term to the list of documents containing it; scoring is [BM25](https://en.wikipedia.org/wiki/Okapi_BM25). -- Elasticsearch/OpenSearch are [near-real-time](https://www.elastic.co/docs/manage-data/data-store/near-real-time-search) (new docs searchable after a refresh, ~1 s default), and the primary shard count is fixed at index creation — resize by reindexing or [split](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-split)/shrink. -- Treat the search index as a derived store, fed by [CDC](distributed.md) from the source of truth. - -## Schema design - -### Normalization - -- [1NF](https://en.wikipedia.org/wiki/First_normal_form) atomic values → [2NF](https://en.wikipedia.org/wiki/Second_normal_form) no partial-key dependency → [3NF](https://en.wikipedia.org/wiki/Third_normal_form) no transitive dependency through non-key columns: each fact stored once. -- [BCNF](https://en.wikipedia.org/wiki/Boyce%E2%80%93Codd_normal_form) is stricter: every nontrivial dependency's determinant is a superkey; 3NF still allows it when the dependent column is part of a candidate key. -- [Denormalize](https://en.wikipedia.org/wiki/Denormalization) measured hot paths only (copied columns, counters, [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html)), accepting harder writes. - -### Surrogate keys - -- Auto-increment bigint: compact and ordered, but reveals counts and needs one sequence. -- [UUIDv4](https://www.rfc-editor.org/rfc/rfc9562#section-5.4) scatters inserts across the B-tree; [UUIDv7](https://www.rfc-editor.org/rfc/rfc9562#section-5.7) is time-ordered and native via [`uuidv7()`](https://www.postgresql.org/docs/current/functions-uuid.html) since PostgreSQL 18 — ID schemes in [distributed.md](distributed.md). - -### Constraints and soft deletes - -- [`FOREIGN KEY`](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-FK), [`CHECK`](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-CHECK-CONSTRAINTS), and [`UNIQUE`](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-UNIQUE-CONSTRAINTS) are the last line of defense — application checks alone race. -- Soft deletes (`deleted_at`) keep deleted rows for undo and audit; a [partial unique index](https://www.postgresql.org/docs/current/indexes-partial.html) (`WHERE deleted_at IS NULL`) frees the value for reuse, so an undelete can then conflict. - -## Scaling - -### Physical vs logical replication - -- Physical ([streaming](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION)): ships [WAL](https://www.postgresql.org/docs/current/wal-intro.html) bytes, exact copy, same major version. -- [Logical](https://www.postgresql.org/docs/current/logical-replication.html): row changes per table, works across versions and subsets, feeds [CDC](distributed.md). -- PostgreSQL [`synchronous_commit`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-SYNCHRONOUS-COMMIT) picks how far a commit waits (`remote_write`, `on`, `remote_apply`); sync vs async trade-offs are in [distributed.md](distributed.md). - -### Table partitioning - -- [Range](https://www.postgresql.org/docs/current/ddl-partitioning.html) (most common, by date), list, or hash; [partition pruning](https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITION-PRUNING) skips irrelevant partitions. -- Dropping an old partition is a cheap metadata change, unlike a bloating bulk `DELETE`, but takes an [`ACCESS EXCLUSIVE`](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-TABLES) lock on the parent — [`DETACH PARTITION CONCURRENTLY`](https://www.postgresql.org/docs/current/sql-altertable.html#SQL-ALTERTABLE-DETACH-PARTITION) (PostgreSQL 14+) first avoids that. -- PostgreSQL [unique constraints must include the partition key](https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITIONING-DECLARATIVE-LIMITATIONS). - -### Connection pooling - -- Each PostgreSQL connection is a [process](https://www.postgresql.org/docs/current/connect-estab.html) (a few MB), so thousands of direct connections hurt — pool them. -- [PgBouncer](https://www.pgbouncer.org/features.html) session mode keeps `SET` and [advisory locks](https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS); transaction mode scales further but loses session state; protocol-level prepared statements work in it since PgBouncer 1.21 when [`max_prepared_statements`](https://www.pgbouncer.org/config.html#max_prepared_statements) > 0 (default 0 before 1.24, now 200); SQL [`PREPARE`](https://www.postgresql.org/docs/current/sql-prepare.html) still doesn't. - -### OLTP vs OLAP - -- [OLAP](https://en.wikipedia.org/wiki/Online_analytical_processing): [column storage](https://en.wikipedia.org/wiki/Column-oriented_DBMS), compression, vectorized scans of few columns over many rows, [star schemas](https://en.wikipedia.org/wiki/Star_schema) (facts + dimensions). -- [Materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) store a query result for fast reads that stay stale until refresh; [`REFRESH MATERIALIZED VIEW CONCURRENTLY`](https://www.postgresql.org/docs/current/sql-refreshmaterializedview.html) avoids blocking reads but needs a unique index. diff --git a/docs/topics/devops.md b/docs/topics/devops.md index 66010d3..ab24da7 100644 --- a/docs/topics/devops.md +++ b/docs/topics/devops.md @@ -28,7 +28,7 @@ What experienced DevOps engineers forget before an interview, grouped by subtopi - [Expand-and-contract](https://martinfowler.com/bliki/ParallelChange.html): add the new schema (backward compatible) → deploy code that handles both → backfill → remove the old schema in a later release. - A destructive migration (dropping a column old code still reads) makes rollback impossible. -- Keep each DDL short and lock-safe ([`lock_timeout`](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-LOCK-TIMEOUT), [concurrent index builds](https://www.postgresql.org/docs/current/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY)) — see [database.md](database.md). +- Keep each DDL short and lock-safe ([`lock_timeout`](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-LOCK-TIMEOUT), [concurrent index builds](https://www.postgresql.org/docs/current/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY)) — see [postgresql.md](postgresql.md). ### Incident response diff --git a/docs/topics/postgresql.md b/docs/topics/postgresql.md new file mode 100644 index 0000000..a582c7e --- /dev/null +++ b/docs/topics/postgresql.md @@ -0,0 +1,353 @@ +# PostgreSQL + +What experienced PostgreSQL engineers forget before an interview, grouped by subtopic. For SQL vs NoSQL, storage engines, and the scaling ladder, see [system.md](system.md); for replication theory and consistency models, see [distributed.md](distributed.md). + +## Indexes + +### B-tree indexes + +- The default index is a [B+tree](https://en.wikipedia.org/wiki/B%2B_tree): balanced, sorted, O(log n), serves equality, range scans, and `ORDER BY`. +- Tables are [heaps](https://www.postgresql.org/docs/current/storage-page-layout.html): rows are unordered and every index entry points to a physical location ([`ctid`](https://www.postgresql.org/docs/current/ddl-system-columns.html#DDL-SYSTEM-COLUMNS-CTID)), which changes when the row is updated. +- [Deduplication](https://www.postgresql.org/docs/current/btree.html#BTREE-DEDUPLICATION) (PostgreSQL 13+) stores repeated keys once, shrinking indexes on low-cardinality columns. + +### Composite indexes + +- [Leftmost-prefix rule](https://www.postgresql.org/docs/current/indexes-multicolumn.html): `(a, b, c)` seeks efficiently on `a`, `a,b`, `a,b,c` — not on `b` alone. +- [Skip scan](https://www.postgresql.org/docs/current/indexes-multicolumn.html) (PostgreSQL 18+) serves `b` alone by probing each distinct `a`; cheap only when `a` has few values. +- Put equality columns first, range columns last; columns after a range condition only filter inside the scanned range (unless skip scan applies). + +### Covering indexes + +- A [covering index](https://www.postgresql.org/docs/current/indexes-index-only-scans.html) holds every column the query needs, enabling an index-only scan; [`INCLUDE`](https://www.postgresql.org/docs/current/sql-createindex.html) (PostgreSQL 11+) adds non-key payload columns. +- An index-only scan still checks the [visibility map](https://www.postgresql.org/docs/current/storage-vm.html); recently modified pages force heap visits until [VACUUM](https://www.postgresql.org/docs/current/routine-vacuuming.html) runs. + +### Partial and expression indexes + +- [Partial](https://www.postgresql.org/docs/current/indexes-partial.html) (`WHERE deleted_at IS NULL`) and [expression](https://www.postgresql.org/docs/current/indexes-expressional.html) (`ON lower(email)`) indexes cover only what queries use. +- [Sargability](https://en.wikipedia.org/wiki/Sargable): a function or implicit cast on an indexed column (`WHERE lower(email) = ...`) blocks index use unless a matching expression index exists. +- The planner uses a partial index only when it can prove the query's `WHERE` implies the index predicate; a generic prepared-statement plan, which sees `$1` instead of the value, can't prove it. + +### Specialized index types + +| Index | Use for | +|---|---| +| [GIN](https://www.postgresql.org/docs/current/gin.html) | full-text, JSONB, arrays ([inverted index](https://en.wikipedia.org/wiki/Inverted_index)); slower writes | +| [GiST](https://www.postgresql.org/docs/current/gist.html) | geometry, ranges, nearest neighbor, exclusion constraints | +| [BRIN](https://www.postgresql.org/docs/current/brin.html) | huge, naturally ordered tables (append-only time series); tiny | +| [Hash](https://www.postgresql.org/docs/current/hash-index.html) | equality only | + +### Building indexes online + +- Plain [`CREATE INDEX`](https://www.postgresql.org/docs/current/sql-createindex.html) takes a `SHARE` lock: reads continue, writes block for the whole build. +- [`CREATE INDEX CONCURRENTLY`](https://www.postgresql.org/docs/current/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY) doesn't block writes, but scans the table twice, can't run in a transaction, and leaves an `INVALID` index if it fails — drop it and retry. +- [`REINDEX CONCURRENTLY`](https://www.postgresql.org/docs/current/sql-reindex.html) (PostgreSQL 12+) rebuilds a bloated index without blocking writes. +- Index every [foreign key](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-FK) you join or cascade-delete on — PostgreSQL doesn't create them automatically. + +## Transactions and isolation + +### Isolation levels + +| Level | [Dirty read](https://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Dirty_reads) | [Non-repeatable read](https://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Non-repeatable_reads) | [Phantom read](https://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Phantom_reads) | Serialization anomaly | +|---|---|---|---|---| +| Read Uncommitted | not in PostgreSQL (runs as Read Committed) | possible | possible | possible | +| [Read Committed](https://www.postgresql.org/docs/current/transaction-iso.html#XACT-READ-COMMITTED) (default) | — | possible | possible | possible | +| [Repeatable Read](https://www.postgresql.org/docs/current/transaction-iso.html#XACT-REPEATABLE-READ) | — | — | not in PostgreSQL (the SQL standard allows it) | possible | +| [Serializable](https://www.postgresql.org/docs/current/transaction-iso.html#XACT-SERIALIZABLE) | — | — | — | — | + +### Lost update and write skew + +- Lost update: two read-modify-write cycles overwrite each other; Read Committed allows it — use an atomic `UPDATE ... SET n = n + 1`, [`SELECT ... FOR UPDATE`](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-ROWS), or a version check. +- [Write skew](https://en.wikipedia.org/wiki/Snapshot_isolation): two transactions read overlapping data and write different rows, together breaking an invariant (both doctors go off call); only Serializable or explicit locks prevent it. + +### Read Committed re-check + +- Each statement in [Read Committed](https://www.postgresql.org/docs/current/transaction-iso.html#XACT-READ-COMMITTED) takes a new snapshot, so two `SELECT`s in one transaction can disagree. +- An `UPDATE` that waits on a row locked by a concurrent writer re-evaluates its `WHERE` against the newest committed version, then updates or skips it — it never sees rows that newly started matching. + +### Repeatable Read and Serializable + +- [Repeatable Read](https://www.postgresql.org/docs/current/transaction-iso.html#XACT-REPEATABLE-READ) is [snapshot isolation](https://en.wikipedia.org/wiki/Snapshot_isolation): one snapshot from the first statement; updating a row a concurrent transaction changed fails with "could not serialize access", so no lost updates — but write skew is possible. +- Serializable is [SSI](https://www.postgresql.org/docs/current/transaction-iso.html#XACT-SERIALIZABLE): predicate (`SIReadLock`) tracking detects dangerous read-write cycles and aborts one transaction. +- Both abort with SQLSTATE [`40001`](https://www.postgresql.org/docs/current/mvcc-serialization-failure-handling.html); the app must retry the whole transaction, not just the failed statement. + +### Optimistic locking + +- Add a [`version` column](https://en.wikipedia.org/wiki/Optimistic_concurrency_control): `UPDATE ... SET version = version + 1 WHERE id = ? AND version = ?`; zero rows updated means someone else won — reload and retry. +- Best when conflicts are rare; pessimistic locks when they're frequent. + +### Errors inside a transaction + +- After any error, the [transaction is aborted](https://www.postgresql.org/docs/current/tutorial-transactions.html): every later statement fails with "current transaction is aborted" until `ROLLBACK`. +- [`SAVEPOINT`](https://www.postgresql.org/docs/current/sql-savepoint.html) + `ROLLBACK TO SAVEPOINT` recovers part of the work; psql `ON_ERROR_ROLLBACK` and pgJDBC `autosave` wrap each statement in one, costing an extra round trip. + +### Timeouts + +- [`statement_timeout`](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-STATEMENT-TIMEOUT) caps one statement; [`lock_timeout`](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-LOCK-TIMEOUT) caps the wait for a lock. +- [`idle_in_transaction_session_timeout`](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUT) kills sessions left in an open transaction, which otherwise hold locks and block VACUUM cleanup. +- [`transaction_timeout`](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-TRANSACTION-TIMEOUT) (PostgreSQL 17+) caps a whole transaction. +- All default to off; set them per role or per session rather than server-wide. + +## MVCC and VACUUM + +### Row versions + +- An `UPDATE` writes a new row version and marks the old one dead; versions carry [`xmin`/`xmax`](https://www.postgresql.org/docs/current/ddl-system-columns.html#DDL-SYSTEM-COLUMNS-XMIN), and each [snapshot](https://www.postgresql.org/docs/current/mvcc-intro.html) sees only the versions committed before it. +- Plain `SELECT`s never block or wait for row writers; writers (and `FOR UPDATE`) still lock rows against each other, and DDL's `ACCESS EXCLUSIVE` lock blocks even `SELECT`. +- The new version needs a new entry in every index unless the update is [HOT](https://www.postgresql.org/docs/current/storage-hot.html), so wide-indexed, update-heavy tables pay heavy write amplification. + +### HOT updates + +- A [HOT update](https://www.postgresql.org/docs/current/storage-hot.html) skips index maintenance when no indexed column changed and the new version fits on the same page. +- A lower [`fillfactor`](https://www.postgresql.org/docs/current/sql-createtable.html#RELOPTION-FILLFACTOR) (e.g. 90) leaves free space per page to make HOT more likely on update-heavy tables. +- Indexing a frequently updated column (`updated_at`) disables HOT for every update that touches it; since PostgreSQL 16, columns covered only by BRIN indexes don't. + +### VACUUM and autovacuum + +- [VACUUM](https://www.postgresql.org/docs/current/routine-vacuuming.html) makes dead versions' space reusable but rarely shrinks the file; [`VACUUM FULL`](https://www.postgresql.org/docs/current/sql-vacuum.html) rewrites the table under an `ACCESS EXCLUSIVE` lock ([pg_repack](https://reorg.github.io/pg_repack/) does it online). +- [Autovacuum](https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM) runs when dead rows exceed 50 + 20% of the table; since PostgreSQL 18 [`autovacuum_vacuum_max_threshold`](https://www.postgresql.org/docs/current/runtime-config-vacuum.html#GUC-AUTOVACUUM-VACUUM-MAX-THRESHOLD) caps that at 100 million rows. Large tables still need a lower per-table scale factor. +- Anything holding an old snapshot stops cleanup everywhere: long transactions, idle-in-transaction sessions, abandoned replication slots, and replicas with `hot_standby_feedback` — the result is bloat. + +### Transaction ID wraparound + +- [Transaction IDs](https://www.postgresql.org/docs/current/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND) are 32-bit, so only ~2 billion are "in the past"; VACUUM freezes old rows so they stay visible. +- When a table's oldest unfrozen XID passes [`autovacuum_freeze_max_age`](https://www.postgresql.org/docs/current/runtime-config-vacuum.html#GUC-AUTOVACUUM-FREEZE-MAX-AGE) (200 million), an anti-wraparound vacuum runs even if autovacuum is off. +- With fewer than 3 million XIDs left, the server refuses new write transactions until a manual VACUUM catches up — an outage, usually caused by something blocking vacuum for weeks. + +## Locking + +### Table lock levels + +| Statement | [Lock](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-TABLES) | Blocks | +|---|---|---| +| `SELECT` | `ACCESS SHARE` | only `ACCESS EXCLUSIVE` | +| `INSERT`, `UPDATE`, `DELETE` | `ROW EXCLUSIVE` | `SHARE` and stronger (plain `CREATE INDEX`) | +| `VACUUM`, `CREATE INDEX CONCURRENTLY`, `VALIDATE CONSTRAINT` | `SHARE UPDATE EXCLUSIVE` | itself and DDL, not reads or writes | +| `CREATE INDEX` | `SHARE` | writes | +| most `ALTER TABLE`, `DROP`, `TRUNCATE`, `VACUUM FULL` | `ACCESS EXCLUSIVE` | everything, including `SELECT` | + +### DDL locks and the lock queue + +- Most `ALTER TABLE` forms take an [`ACCESS EXCLUSIVE`](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-TABLES) lock. If it waits behind a long query, every later query queues behind it — a brief outage from a "fast" migration. +- Set [`lock_timeout`](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-LOCK-TIMEOUT) (a few seconds) on migrations and retry; rollout patterns in [devops.md](devops.md). + +### Safe schema changes + +- [`ADD COLUMN`](https://www.postgresql.org/docs/current/sql-altertable.html#SQL-ALTERTABLE-NOTES) with a constant default is metadata-only since PostgreSQL 11; a volatile default (`clock_timestamp()`) rewrites the table. +- Changing a column's type usually rewrites the table and its indexes; widening `varchar(n)` or switching to `text` doesn't. +- Add foreign keys and `CHECK`s as [`NOT VALID`](https://www.postgresql.org/docs/current/sql-altertable.html#SQL-ALTERTABLE-DESC-ADD-TABLE-CONSTRAINT), then `VALIDATE CONSTRAINT`, which scans under `SHARE UPDATE EXCLUSIVE` without blocking writes. +- `SET NOT NULL` scans the table under `ACCESS EXCLUSIVE`, skipped if a validated `CHECK (col IS NOT NULL)` already proves it; since PostgreSQL 18, `NOT NULL` itself can be added `NOT VALID`. + +### Row locks and queue tables + +- [`SELECT ... FOR UPDATE`](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-ROWS) locks the rows it returns; `FOR NO KEY UPDATE` is weaker and doesn't block inserts of rows referencing them by foreign key. +- [`SKIP LOCKED`](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE) lets workers claim different rows of a queue table without blocking; `NOWAIT` errors instead of waiting. +- Row locks live in the row header on disk, not in memory: there's no limit on rows locked and no lock escalation, but locking writes to the page. + +### Advisory locks + +- [Advisory locks](https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS) are app-defined locks on a number, not a row — for leader election or job dedup without a table to lock. +- Session-level locks (`pg_advisory_lock`) persist until unlock or disconnect; transaction-level ones (`pg_advisory_xact_lock`) release at commit. +- Session-level locks break behind a transaction-mode pooler: the next transaction may run on a different server connection. + +### Deadlocks + +- Two transactions each hold a lock the other needs; after [`deadlock_timeout`](https://www.postgresql.org/docs/current/runtime-config-locks.html#GUC-DEADLOCK-TIMEOUT) (1 s) PostgreSQL [detects the cycle](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-DEADLOCKS) and aborts one. +- Prevent with a consistent lock order (e.g. update rows sorted by ID) and short transactions. + +## Query planning and tuning + +### Reading EXPLAIN + +- [`EXPLAIN ANALYZE`](https://www.postgresql.org/docs/current/using-explain.html#USING-EXPLAIN-ANALYZE) executes the query (wrap writes in a rolled-back transaction); since PostgreSQL 18 it includes `BUFFERS` by default. +- Estimated vs actual rows far apart means stale or insufficient statistics — run [`ANALYZE`](https://www.postgresql.org/docs/current/sql-analyze.html) or add [extended statistics](https://www.postgresql.org/docs/current/planner-stats.html#PLANNER-STATS-EXTENDED) for correlated columns. +- Scan types: Seq Scan, Index Scan, Index Only Scan, and [Bitmap Heap Scan](https://www.postgresql.org/docs/current/indexes-bitmap-scans.html) (collects matches from one or more indexes, then reads pages in physical order). +- [`pg_stat_statements`](https://www.postgresql.org/docs/current/pgstatstatements.html) ranks normalized queries by total time — start tuning there. + +### Why an index isn't used + +- Low selectivity: when a filter matches a large share of rows, the planner prefers a [sequential scan](https://www.postgresql.org/docs/current/using-explain.html), since one sequential pass beats many random heap reads. +- `LIKE 'x%'` uses a B-tree only with C collation or [`text_pattern_ops`](https://www.postgresql.org/docs/current/indexes-opclass.html); `LIKE '%x'` needs a trigram GIN or GiST index ([`pg_trgm`](https://www.postgresql.org/docs/current/pgtrgm.html)). +- A function or implicit cast on the column, and `OR` across columns without an index on each, also defeat it. +- Small tables: a seq scan of a few pages is cheaper than any index. + +### Join algorithms + +| Join | Best when | +|---|---| +| [Nested loop](https://en.wikipedia.org/wiki/Nested_loop_join) | outer side small, inner side index-backed | +| [Hash join](https://en.wikipedia.org/wiki/Hash_join) | large equality joins without a useful index; builds on the smaller side | +| [Merge join](https://en.wikipedia.org/wiki/Sort-merge_join) | both inputs already sorted (e.g. from indexes) | + +### Memory per query + +- [`work_mem`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-WORK-MEM) (4 MB) applies per sort or hash node, per parallel worker; hashes get `work_mem × hash_mem_multiplier` (2.0) — one query can use several multiples, times every connection. +- A sort or hash that exceeds it spills to disk (`external merge` in `EXPLAIN ANALYZE`); raise it per session for reporting queries, not globally. + +### Prepared statements and generic plans + +- A [prepared statement](https://www.postgresql.org/docs/current/sql-prepare.html) runs its first five executions with custom plans, then switches to a cached generic plan if its estimated cost isn't much higher. +- On skewed data a generic plan can be badly wrong for rare values; [`plan_cache_mode = force_custom_plan`](https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-PLAN-CACHE-MODE) forces replanning. +- Drivers may prepare implicitly: [pgJDBC](https://jdbc.postgresql.org/documentation/server-prepare/) after 5 executions of a statement, [pgx](https://pkg.go.dev/github.com/jackc/pgx/v5#hdr-Prepared_Statements) by default. + +### Pagination and counting + +- `OFFSET` still reads and discards every skipped row, so deep pages get slower; [keyset pagination](https://use-the-index-luke.com/no-offset) seeks by index — API side in [backend.md](backend.md). +- `count(*)` scans the table or an index (MVCC means no stored row count); for a rough number, read [`pg_class.reltuples`](https://www.postgresql.org/docs/current/catalog-pg-class.html). + +## SQL gotchas + +### Logical evaluation order + +`FROM/JOIN → WHERE → GROUP BY → HAVING → SELECT → DISTINCT → ORDER BY → LIMIT` — why a `SELECT` alias works in `ORDER BY` but not in `WHERE`. + +### NULL and three-valued logic + +- Any comparison with NULL — even `NULL = NULL` — is [UNKNOWN](https://en.wikipedia.org/wiki/Null_%28SQL%29#Comparisons_with_NULL_and_the_three-valued_logic_%283VL%29), and `WHERE` drops it; use `IS [NOT] NULL` or [`IS [NOT] DISTINCT FROM`](https://www.postgresql.org/docs/current/functions-comparison.html). +- [`NOT IN`](https://www.postgresql.org/docs/current/functions-subquery.html#FUNCTIONS-SUBQUERY-NOTIN) with a NULL in the list returns no rows — use [`NOT EXISTS`](https://www.postgresql.org/docs/current/functions-subquery.html#FUNCTIONS-SUBQUERY-EXISTS). +- [`COUNT(*)`](https://www.postgresql.org/docs/current/functions-aggregate.html) counts rows; `COUNT(col)`, `SUM`, `AVG`, `MIN`/`MAX` skip NULLs, but `array_agg`/`json_agg` keep them. +- A [`UNIQUE`](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-UNIQUE-CONSTRAINTS) column accepts many NULLs; `UNIQUE NULLS NOT DISTINCT` (PostgreSQL 15+) allows only one. + +### Window functions + +- [`fn() OVER (PARTITION BY ... ORDER BY ...)`](https://www.postgresql.org/docs/current/tutorial-window.html) computes per row without collapsing rows. +- [`ROW_NUMBER`](https://www.postgresql.org/docs/current/functions-window.html) 1,2,3; `RANK` 1,1,3 (gaps); `DENSE_RANK` 1,1,2. +- The [default frame](https://www.postgresql.org/docs/current/sql-expressions.html#SYNTAX-WINDOW-FUNCTIONS) with `ORDER BY` is `RANGE ... CURRENT ROW`, which includes all peer rows with equal sort keys — use `ROWS` for a true running total. + +### CTE materialization + +- Since PostgreSQL 12, a non-recursive, side-effect-free CTE referenced once is [inlined](https://www.postgresql.org/docs/current/queries-with.html#QUERIES-WITH-CTE-MATERIALIZATION) into the outer query; one referenced several times is materialized. +- `MATERIALIZED` / `NOT MATERIALIZED` override the default. + +### Join pitfalls + +- A null-rejecting condition on the right table in `WHERE` (`r.x = 1`, unlike `IS NULL` or `COALESCE`) discards the NULL-extended rows and turns a [`LEFT JOIN`](https://www.postgresql.org/docs/current/queries-table-expressions.html#QUERIES-JOIN) into an inner join; put it in `ON`. +- Joining a parent to two one-to-many children multiplies rows (fan-out) and inflates `SUM`/`COUNT`; aggregate each child in a subquery first. +- [`EXISTS`](https://www.postgresql.org/docs/current/functions-subquery.html#FUNCTIONS-SUBQUERY-EXISTS) / `NOT EXISTS` are [semi-](https://en.wikipedia.org/wiki/Join_%28relational_algebra%29#Semijoin) and [anti-joins](https://en.wikipedia.org/wiki/Join_%28relational_algebra%29#Antijoin): they never duplicate rows, unlike `JOIN` + `DISTINCT`. + +### Upsert and MERGE + +- [`INSERT ... ON CONFLICT (...) DO UPDATE`](https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT) replaces a racy check-then-write; it needs a unique index matching the conflict target, and `EXCLUDED.col` holds the proposed row. +- `ON CONFLICT DO NOTHING ... RETURNING` returns nothing for conflicting rows, so "get or create" needs a follow-up `SELECT`. +- [`MERGE`](https://www.postgresql.org/docs/current/sql-merge.html) (PostgreSQL 15+) doesn't handle concurrent inserts like `ON CONFLICT` does — it can fail with a unique violation. +- `MERGE ... RETURNING` exists since PostgreSQL 17; [`OLD`/`NEW` in `RETURNING`](https://www.postgresql.org/docs/current/dml-returning.html) (any DML) since 18. + +## WAL and durability + +### Write-ahead log + +- A commit is durable once its [WAL](https://www.postgresql.org/docs/current/wal-intro.html) record is [fsynced](os.md); dirty data pages are flushed later ([checkpoints](https://www.postgresql.org/docs/current/wal-configuration.html), background writer), never before their WAL. +- Crash recovery only redoes WAL from the last checkpoint; uncommitted rows stay invisible by their commit status, so there's no undo phase. +- [Group commit](https://www.postgresql.org/docs/current/wal-configuration.html) batches many commits into one fsync. + +### Checkpoints + +- A checkpoint runs every [`checkpoint_timeout`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-CHECKPOINT-TIMEOUT) (5 min) or when WAL reaches [`max_wal_size`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-MAX-WAL-SIZE) (1 GB), whichever comes first. +- The first change to a page after a checkpoint logs the whole page ([`full_page_writes`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-FULL-PAGE-WRITES)) to survive torn writes, so frequent checkpoints inflate WAL volume. +- Longer intervals mean less WAL and I/O but slower crash recovery. + +### Commit durability levels + +- [`synchronous_commit`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-SYNCHRONOUS-COMMIT) picks how far a commit waits: `off` (doesn't wait for the local flush), `local`, `remote_write`, `on` (standby fsync), `remote_apply` (visible on standby); without `synchronous_standby_names`, every mode except `off` just waits for the local flush. +- `off` ([asynchronous commit](https://www.postgresql.org/docs/current/wal-async-commit.html)) can lose up to 3 × `wal_writer_delay` (600 ms by default) of commits on a crash, but never corrupts data — settable per transaction for low-value writes. +- `fsync = off` is different: a crash can corrupt the cluster. + +### Backups and PITR + +- Logical backups ([`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html)): portable across versions, slow to restore at scale. Physical ([`pg_basebackup`](https://www.postgresql.org/docs/current/app-pgbasebackup.html), snapshots): fast restore, same major version. +- [PITR](https://www.postgresql.org/docs/current/continuous-archiving.html) = physical base backup + archived [WAL](https://www.postgresql.org/docs/current/wal-intro.html) replayed to a chosen moment. Replicas are not backups — they replicate mistakes too. +- [Incremental backups](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-INCREMENTAL-BACKUP) (`pg_basebackup --incremental`, PostgreSQL 17+) copy only changed blocks; [`pg_combinebackup`](https://www.postgresql.org/docs/current/app-pgcombinebackup.html) reconstructs a full one. + +## Replication and high availability + +### Physical vs logical replication + +- Physical ([streaming](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION)): ships [WAL](https://www.postgresql.org/docs/current/wal-intro.html) bytes; an exact, read-only copy of the whole cluster on the same major version, except `UNLOGGED` tables, which are empty on standbys. +- [Logical](https://www.postgresql.org/docs/current/logical-replication.html): row changes per table, works across major versions and subsets — used for zero-downtime upgrades and [CDC](distributed.md). +- Logical replication [doesn't copy](https://www.postgresql.org/docs/current/logical-replication-restrictions.html) DDL or sequence values, and `UPDATE`/`DELETE` need a [replica identity](https://www.postgresql.org/docs/current/sql-altertable.html#SQL-ALTERTABLE-REPLICA-IDENTITY) (usually the primary key). + +### Synchronous replication + +- [`synchronous_standby_names`](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-SYNCHRONOUS-STANDBY-NAMES) lists standbys a commit waits for: `FIRST n (...)` by priority or `ANY n (...)` as a quorum. +- If the required standbys are down, commits hang rather than fail — list more candidates than `n`. +- Sync vs async trade-offs in [distributed.md](distributed.md). + +### Replication slots + +- A [replication slot](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION-SLOTS) keeps WAL (and, for logical slots, old catalog rows) until its consumer confirms it — a dead consumer fills the disk. +- Cap it with [`max_slot_wal_keep_size`](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-MAX-SLOT-WAL-KEEP-SIZE) (PostgreSQL 13+, unlimited by default) or [`idle_replication_slot_timeout`](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-IDLE-REPLICATION-SLOT-TIMEOUT) (PostgreSQL 18+, off by default); either invalidates the slot. +- Logical slots can be [synced to a standby](https://www.postgresql.org/docs/current/logicaldecoding-explanation.html#LOGICALDECODING-REPLICATION-SLOTS-SYNCHRONIZATION) (PostgreSQL 17+), so CDC survives a failover. + +### Read replicas + +- Async [hot standbys](https://www.postgresql.org/docs/current/hot-standby.html) lag, so a read after a write may miss it — route read-your-writes to the primary. +- Replaying a VACUUM that removes rows a standby query still needs cancels that query once replay has been held back for [`max_standby_streaming_delay`](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-MAX-STANDBY-STREAMING-DELAY) (30 s) — a budget for the replay lag, not a per-query grace period. +- [`hot_standby_feedback`](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-HOT-STANDBY-FEEDBACK) avoids those cancellations but holds back vacuum on the primary, causing bloat. + +### Failover + +- PostgreSQL has no built-in automatic failover; [Patroni](https://patroni.readthedocs.io/) and similar tools elect a leader through a consensus store (etcd) and fence the old primary. +- A demoted primary may have WAL the new one never got; [`pg_rewind`](https://www.postgresql.org/docs/current/app-pgrewind.html) discards it so the old primary can rejoin as a standby. +- With async replication, commits acknowledged but not yet shipped are lost on failover. + +## Architecture and scaling + +### Process model and memory + +- Each connection is a separate backend [process](https://www.postgresql.org/docs/current/connect-estab.html) (several MB), so thousands of direct connections hurt — pool them. +- [`shared_buffers`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-SHARED-BUFFERS) (~25% of RAM is a common start) caches pages, and the OS page cache holds them again; [`effective_cache_size`](https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-EFFECTIVE-CACHE-SIZE) is only a planner hint. +- Since PostgreSQL 18, sequential scans, bitmap heap scans, and vacuum read through [asynchronous I/O](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-IO-METHOD) (`io_method = worker` by default; `io_uring` on Linux). + +### Connection pooling + +- [PgBouncer](https://www.pgbouncer.org/features.html) session mode keeps `SET` and session [advisory locks](https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS); transaction mode scales further but loses session state. +- Protocol-level prepared statements work in transaction mode since PgBouncer 1.21 when [`max_prepared_statements`](https://www.pgbouncer.org/config.html#max_prepared_statements) > 0 (default 0 before 1.24, now 200); SQL [`PREPARE`](https://www.postgresql.org/docs/current/sql-prepare.html) still doesn't. +- App-side pool sizing in [backend.md](backend.md). + +### Table partitioning + +- [Range](https://www.postgresql.org/docs/current/ddl-partitioning.html) (most common, by date), list, or hash; [partition pruning](https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITION-PRUNING) skips irrelevant partitions only when the query filters on the partition key. +- Dropping an old partition is a cheap metadata change, unlike a bloating bulk `DELETE`, but takes an [`ACCESS EXCLUSIVE`](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-TABLES) lock on the parent — [`DETACH PARTITION CONCURRENTLY`](https://www.postgresql.org/docs/current/sql-altertable.html#SQL-ALTERTABLE-DETACH-PARTITION) (PostgreSQL 14+) first avoids that. +- [Unique constraints must include the partition key](https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITIONING-DECLARATIVE-LIMITATIONS), so a global unique ID across partitions isn't enforced. + +### Materialized views + +- A [materialized view](https://www.postgresql.org/docs/current/rules-materializedviews.html) stores a query result for fast reads that stays stale until refreshed — no incremental refresh built in. +- [`REFRESH MATERIALIZED VIEW CONCURRENTLY`](https://www.postgresql.org/docs/current/sql-refreshmaterializedview.html) doesn't block reads but needs a unique index and is slower. + +## Schema design + +### Normalization + +- [1NF](https://en.wikipedia.org/wiki/First_normal_form) atomic values → [2NF](https://en.wikipedia.org/wiki/Second_normal_form) no partial-key dependency → [3NF](https://en.wikipedia.org/wiki/Third_normal_form) no transitive dependency through non-key columns: each fact stored once. +- [BCNF](https://en.wikipedia.org/wiki/Boyce%E2%80%93Codd_normal_form) is stricter: every nontrivial dependency's determinant is a superkey; 3NF still allows it when the dependent column is part of a candidate key. +- [Denormalize](https://en.wikipedia.org/wiki/Denormalization) measured hot paths only (copied columns, counters, materialized views), accepting harder writes. + +### Keys and sequences + +- Prefer [`GENERATED ALWAYS AS IDENTITY`](https://www.postgresql.org/docs/current/ddl-identity-columns.html) (PostgreSQL 10+) over `serial`; use `bigint`, since `int` runs out at ~2.1 billion. +- [Sequences](https://www.postgresql.org/docs/current/functions-sequence.html) are non-transactional: `nextval` is never rolled back, so IDs have gaps. +- [UUIDv4](https://www.rfc-editor.org/rfc/rfc9562#section-5.4) scatters inserts across the B-tree; [UUIDv7](https://www.rfc-editor.org/rfc/rfc9562#section-5.7) is time-ordered and native via [`uuidv7()`](https://www.postgresql.org/docs/current/functions-uuid.html) since PostgreSQL 18 — ID schemes in [distributed.md](distributed.md). + +### Constraints and soft deletes + +- [`FOREIGN KEY`](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-FK), [`CHECK`](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-CHECK-CONSTRAINTS), and [`UNIQUE`](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-UNIQUE-CONSTRAINTS) are the last line of defense — application checks alone race. +- [Exclusion constraints](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-EXCLUSION) (`EXCLUDE USING gist (room WITH =, during WITH &&)`) forbid overlapping bookings; since PostgreSQL 18 a primary key or unique constraint can say `WITHOUT OVERLAPS`. +- Soft deletes (`deleted_at`) keep rows for undo and audit; a [partial unique index](https://www.postgresql.org/docs/current/indexes-partial.html) (`WHERE deleted_at IS NULL`) frees the value for reuse, so an undelete can then conflict. + +### JSONB + +- [`jsonb`](https://www.postgresql.org/docs/current/datatype-json.html) is parsed binary: faster to query and indexable, but drops duplicate keys (last wins), key order, and whitespace; `json` keeps the text as is. +- A [GIN index](https://www.postgresql.org/docs/current/datatype-json.html#JSON-INDEXING) with the default `jsonb_ops` supports key-existence and containment; `jsonb_path_ops` is smaller and faster but supports only containment and path queries. +- Updating one key rewrites the whole value; columns you filter or join on belong in real columns. + +### Row size and TOAST + +- Pages are 8 KB; values that push a row past ~2 KB are compressed or moved out of line by [TOAST](https://www.postgresql.org/docs/current/storage-toast.html), up to 1 GB per field. +- `SELECT *` fetches and decompresses every toasted column; select only what you need. +- [Generated columns](https://www.postgresql.org/docs/current/ddl-generated-columns.html) are virtual (computed on read) by default since PostgreSQL 18; `STORED` computes on write and takes space. + +### Common type choices + +- [`timestamptz`](https://www.postgresql.org/docs/current/datatype-datetime.html) stores an instant in UTC and stores no time zone; `timestamp` ignores zones entirely — time handling in [backend.md](backend.md). +- [`text`](https://www.postgresql.org/docs/current/datatype-character.html) and `varchar(n)` perform the same; `char(n)` pads with spaces. +- [`numeric`](https://www.postgresql.org/docs/current/datatype-numeric.html#DATATYPE-NUMERIC-DECIMAL) is exact for money; `float8` rounds, and the `money` type depends on `lc_monetary`. diff --git a/docs/topics/redis.md b/docs/topics/redis.md new file mode 100644 index 0000000..ba35c48 --- /dev/null +++ b/docs/topics/redis.md @@ -0,0 +1,210 @@ +# Redis + +What experienced Redis users forget before an interview, grouped by subtopic. For caching strategies and rate-limiting algorithms, see [system.md](system.md); for Redlock and delivery semantics, see [distributed.md](distributed.md). + +## Execution model + +### Single-threaded execution + +- Commands execute one at a time on the main thread, so each command is atomic and needs no locks. +- One slow command ([`KEYS *`](https://redis.io/docs/latest/commands/keys/), `SMEMBERS` on a huge set, `DEL` of a big key) blocks every other client for its duration. +- [I/O threads](https://redis.io/docs/latest/operate/oss_and_stack/management/config-file/) (`io-threads`, since 6.0; rewritten in 8.0) parse requests and write replies; commands still run on the main thread. Off by default. +- Scale CPU-bound workloads with more shards (Cluster), not bigger machines. + +### Slow and blocking operations + +- [`SCAN`](https://redis.io/docs/latest/commands/scan/) iterates with a cursor instead of `KEYS`; it can return a key twice and only [guarantees](https://redis.io/docs/latest/commands/scan/#scan-guarantees) keys present for the whole iteration; `COUNT` is a hint. +- [`UNLINK`](https://redis.io/docs/latest/commands/unlink/) (4.0+) frees a big value in a background thread; `DEL` frees it synchronously unless `lazyfree-lazy-user-del yes`. +- [`SLOWLOG`](https://redis.io/docs/latest/commands/slowlog/) records commands slower than 10 ms by default, counting execution time only, not network or queueing. +- Find offenders with `redis-cli --bigkeys` / `--memkeys`, and since 8.6 the [`HOTKEYS`](https://redis.io/docs/latest/commands/hotkeys/) command. + +### Pipelines, transactions, and scripts + +| | Atomic | Round trips | Can branch on read values | +|---|---|---|---| +| [Pipeline](https://redis.io/docs/latest/develop/using-commands/pipelining/) | no — other clients' commands interleave | one per batch | no | +| [`MULTI`/`EXEC`](https://redis.io/docs/latest/develop/using-commands/transactions/) | yes | one per batch when pipelined | no; `WATCH` aborts on change | +| [Lua script](https://redis.io/docs/latest/develop/programmability/eval-intro/) or [function](https://redis.io/docs/latest/develop/programmability/functions-intro/) | yes | one | yes | + +### MULTI/EXEC and WATCH + +- [Transactions](https://redis.io/docs/latest/develop/using-commands/transactions/) don't roll back: a command that fails at run time (wrong type) leaves the others applied; an error while queuing (bad syntax) aborts the whole `EXEC`. +- [`WATCH`](https://redis.io/docs/latest/commands/watch/) makes `EXEC` return null if a watched key changed — an optimistic check-and-set; retry the read-modify-write loop. +- In Cluster, every key in the transaction must hash to the same slot. + +### Lua scripts and functions + +- A [script](https://redis.io/docs/latest/develop/programmability/eval-intro/) runs without interleaving but doesn't roll back: a runtime error keeps the writes made before it. +- A long script blocks everyone: past [`busy-reply-threshold`](https://redis.io/docs/latest/develop/programmability/#maximum-execution-time) (5 s) other clients get `BUSY`, and `SCRIPT KILL` works only if it hasn't written yet — otherwise `SHUTDOWN NOSAVE`. +- `EVALSHA` runs a cached script by hash; the cache isn't persisted, so clients must handle `NOSCRIPT` by resending the source. +- [Functions](https://redis.io/docs/latest/develop/programmability/functions-intro/) (7.0+) are named libraries loaded once with `FUNCTION LOAD` and called with `FCALL`; they're persisted and replicated like data. +- Pass every key the script touches in `KEYS`, so Cluster can route it and check slots. + +## Data structures + +### Core types + +| Type | Encoding when large | Typical use | +|---|---|---| +| [String](https://redis.io/docs/latest/develop/data-types/strings/) (≤ 512 MB) | raw bytes or integer | cache values, counters (`INCR`), locks | +| [Hash](https://redis.io/docs/latest/develop/data-types/hashes/) | hash table | objects with per-field updates; per-field TTL since 7.4 | +| [List](https://redis.io/docs/latest/develop/data-types/lists/) | quicklist (linked listpacks) | queues, recent items; index access is O(n) | +| [Set](https://redis.io/docs/latest/develop/data-types/sets/) | hash table | membership, tags, intersections | +| [Sorted set](https://redis.io/docs/latest/develop/data-types/sorted-sets/) | [skip list](https://en.wikipedia.org/wiki/Skip_list) + hash table | leaderboards, sliding-window limiters, delay queues | +| [Stream](https://redis.io/docs/latest/develop/data-types/streams/) | radix tree of listpacks | append-only event log with consumer groups | + +### Compact encodings + +- Small collections use a flat [listpack](https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/memory-optimization/): hashes up to 512 fields, sets and sorted sets up to 128 members, values ≤ 64 bytes; integer-only sets up to 512 use an intset. +- Crossing a threshold converts the key to a hash table or skip list, which takes several times the memory; [`OBJECT ENCODING`](https://redis.io/docs/latest/commands/object-encoding/) shows the current form. +- Many small hashes beat millions of top-level keys: each key costs tens of bytes of overhead. + +### Sorted set details + +- [`ZADD`](https://redis.io/docs/latest/commands/zadd/), `ZRANK`, `ZREM` are O(log n); `ZSCORE` is O(1) via the hash; range by rank or score is O(log n + m). +- Scores are 64-bit floats: integers are exact only up to 2^53, so don't pack large IDs into scores. +- Members with equal scores sort lexicographically; `ZRANGE ... BYLEX` works only when every member has the same score. + +### Probabilistic and specialized types + +- [HyperLogLog](https://redis.io/docs/latest/develop/data-types/probabilistic/hyperloglogs/) counts distinct items in 12 KB with 0.81% standard error; `PFMERGE` unions them. +- [Bitmaps](https://redis.io/docs/latest/develop/data-types/bitmaps/) are bit operations on strings — one bit per user ID for daily actives. +- [Geo](https://redis.io/docs/latest/develop/data-types/geospatial/) commands store geohashes as sorted-set scores and search by radius or box. +- Since 8.0, the former Redis Stack modules ship built in: [JSON](https://redis.io/docs/latest/develop/data-types/json/), time series, Bloom and Cuckoo filters, count-min sketch, top-k, t-digest, vector sets, and the query engine. + +## Expiration and eviction + +### Key expiration + +- A [TTL](https://redis.io/docs/latest/commands/expire/) belongs to the key; [`SET`](https://redis.io/docs/latest/commands/set/) without `KEEPTTL` clears it, while `INCR`, `HSET`, or `LPUSH` keep it. +- Per-field TTLs on hashes ([`HEXPIRE`](https://redis.io/docs/latest/commands/hexpire/)) exist since 7.4; before that, expiry was whole-key only. +- [Expired keys](https://redis.io/docs/latest/commands/expire/#how-redis-expires-keys) are deleted lazily on access plus by a background sampler, so memory lags behind expirations. +- Replicas don't expire keys themselves; they wait for the primary's `DEL`, but reads of a logically expired key already return nothing. + +### Eviction policies + +- [`maxmemory`](https://redis.io/docs/latest/develop/reference/eviction/#maxmem) is 0 (unlimited) by default on 64-bit, so an unbounded cache grows until the kernel OOM killer steps in. +- [`maxmemory-policy`](https://redis.io/docs/latest/develop/reference/eviction/#eviction-policies) defaults to `noeviction`: at the limit, writes fail with an OOM error while reads continue. +- Caches use `allkeys-lru` or `allkeys-lfu`; `allkeys-lrm`/`volatile-lrm` (8.6+) evict the least recently modified keys. +- `volatile-*` policies evict only keys with a TTL, so without TTLs they behave like `noeviction`. +- Leave headroom: replication and AOF buffers aren't counted against `maxmemory`. + +### Approximated LRU and LFU + +- [LRU](https://redis.io/docs/latest/develop/reference/eviction/#apx-lru) is approximated: evict the oldest of `maxmemory-samples` (5) random keys, helped by a pool of good candidates; 10 samples is close to true LRU. +- [LFU](https://redis.io/docs/latest/develop/reference/eviction/#lfu-eviction) keeps an 8-bit logarithmic (Morris) counter per key that decays every minute (`lfu-decay-time 1`), so formerly hot keys age out. + +## Persistence + +### RDB snapshots + +- [RDB](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/) forks; the child writes a point-in-time snapshot while the parent keeps serving, sharing memory [copy-on-write](https://en.wikipedia.org/wiki/Copy-on-write). +- Under heavy writes the fork can approach 2× memory; set [`vm.overcommit_memory = 1`](https://redis.io/docs/latest/operate/oss_and_stack/management/admin/) and disable transparent huge pages — fork and overcommit in [os.md](os.md). +- Default save points: after 1 hour with ≥ 1 change, 5 min with ≥ 100, 1 min with ≥ 10,000 — everything since the last snapshot is lost on a crash. +- `stop-writes-on-bgsave-error yes` (default) rejects writes after a failed snapshot. + +### AOF + +- [AOF](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/#append-only-file) logs every write; it's off by default (`appendonly no`) and, when on, takes precedence over RDB at startup. +- `appendfsync`: `always` (slow, safest), `everysec` (default, loses up to ~1 s), `no` (the OS decides, ~30 s on Linux). +- A rewrite (`BGREWRITEAOF`) forks to compact the log; since 7.0 the AOF is [multi-part](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/#log-rewriting) — an RDB-format base file plus incremental files and a manifest. +- A pure cache can run without persistence; even with AOF, async replication still loses writes on failover. + +## Replication and high availability + +### Replication + +- [Replication](https://redis.io/docs/latest/operate/oss_and_stack/management/replication/) is asynchronous: the primary acknowledges writes before replicas have them. +- A reconnecting replica resumes with `PSYNC` (replication ID + offset) if the gap still fits the replication backlog (1 MB default); otherwise it does a full resync — a fork and an RDB transfer. +- [`WAIT`](https://redis.io/docs/latest/commands/wait/) blocks until n replicas acknowledge, and [`WAITAOF`](https://redis.io/docs/latest/commands/waitaof/) (7.2+) until they fsync; both shrink the loss window but don't make Redis strongly consistent. +- `min-replicas-to-write` (off by default) rejects writes when fewer replicas than that acknowledged within `min-replicas-max-lag` seconds, limiting what an isolated primary accepts. +- A primary with persistence off that auto-restarts comes back empty, and its replicas [sync to the empty dataset](https://redis.io/docs/latest/operate/oss_and_stack/management/replication/#safety-of-replication-when-master-has-persistence-turned-off). + +### Sentinel + +- [Sentinel](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/) monitors a primary, promotes a replica on failure, and tells clients the new primary; run at least 3 on independent hosts. +- The `quorum` sets how many Sentinels must agree the primary is down; the failover itself still needs a majority of Sentinels. +- Clients must be Sentinel-aware and ask it for the current primary, not cache the address. +- Acknowledged writes not yet replicated are lost on failover, and an old primary on the minority side keeps accepting writes unless `min-replicas-to-write` is set and it loses sight of enough replicas. + +## Cluster + +### Hash slots + +- [Redis Cluster](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/) splits keys into 16,384 slots (`CRC16(key) mod 16384`); each primary owns a range. +- [Hash tags](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/#hash-tags) hash only the part in braces, so `{user42}:cart` and `{user42}:orders` share a slot. +- Multi-key commands, transactions, and scripts need all keys in one slot, else `CROSSSLOT`; only database 0 exists. + +### Redirections and resharding + +- [`MOVED`](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/#moved-redirection): the slot lives elsewhere — the client updates its slot map. [`ASK`](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/#ask-redirection): the slot is mid-migration — retry once on the target after `ASKING`, without updating the map. +- Smart clients cache the slot map ([`CLUSTER SHARDS`](https://redis.io/docs/latest/commands/cluster-shards/)) and refresh it on `MOVED`. +- Since 8.4, [`CLUSTER MIGRATION`](https://redis.io/docs/latest/commands/cluster-migration/) moves whole slots atomically instead of key by key with `ASK` redirects. + +### Cluster failure handling + +- Nodes gossip over the cluster bus (data port + 10000); a primary unreachable for [`cluster-node-timeout`](https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/#redis-cluster-configuration-parameters) (15 s) is marked failed by a majority of primaries, and one of its replicas is promoted. +- [Not strongly consistent](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/#write-safety): async replication loses acknowledged writes, and a primary on the minority side of a partition accepts writes until the node timeout. +- With `cluster-require-full-coverage yes` (default), the whole cluster stops serving if any slot has no live owner. + +## Messaging + +### Pub/sub + +- [Pub/sub](https://redis.io/docs/latest/develop/pubsub/) is fire-and-forget: a disconnected subscriber misses messages for good. +- A slow subscriber is disconnected once its output buffer exceeds 32 MB, or 8 MB for 60 s. +- In Cluster, `PUBLISH` is broadcast to every node; [sharded pub/sub](https://redis.io/docs/latest/develop/pubsub/#sharded-pubsub) (`SPUBLISH`, 7.0+) keeps a channel on its slot's shard. +- [Keyspace notifications](https://redis.io/docs/latest/develop/pubsub/keyspace-notifications/) are off by default and travel over pub/sub, so they're lossy too. + +### Streams + +- [`XADD`](https://redis.io/docs/latest/commands/xadd/) appends entries with time-based IDs (`-`); [`XREADGROUP`](https://redis.io/docs/latest/commands/xreadgroup/) gives each entry to one consumer in a group. +- Delivered entries sit in the pending entries list until [`XACK`](https://redis.io/docs/latest/commands/xack/) — at-least-once delivery. +- A crashed consumer's entries are taken over with [`XAUTOCLAIM`](https://redis.io/docs/latest/commands/xautoclaim/) (6.2+) or `XREADGROUP ... CLAIM` (8.4+); [`XNACK`](https://redis.io/docs/latest/commands/xnack/) (8.8+) releases them explicitly. +- Streams grow until trimmed: `XADD ... MAXLEN ~ n` trims approximately and cheaply. +- Since 8.6, `XADD ... IDMP` deduplicates producer retries. + +### Lists as queues + +- `LPUSH` + [`BRPOP`](https://redis.io/docs/latest/commands/brpop/) is the simplest queue, but a worker crash after the pop loses the job. +- Reliable queue: [`BLMOVE`](https://redis.io/docs/latest/commands/blmove/) (6.2+) atomically moves the job to a per-worker processing list; remove it after success, re-queue stale ones. +- Prefer Streams when you need acknowledgments, replay, or several consumer groups; Kafka-style logs in [system.md](system.md). + +## Common patterns + +### Distributed locks + +- Acquire with [`SET lock:k NX PX 30000`](https://redis.io/docs/latest/commands/set/); the token identifies the owner. +- Release only if the token still matches: compare-and-delete in a Lua script, or [`DELEX key IFEQ token`](https://redis.io/docs/latest/commands/delex/) since 8.4. +- The TTL can expire mid-work (GC pause, slow I/O), giving two holders — protect the resource with fencing tokens; Redlock's safety debate in [distributed.md](distributed.md). +- With async replication, a failover can lose the lock key and let a second client acquire it. + +### Caching with Redis + +- Strategies (cache-aside, write-through) and stampede fixes are in [system.md](system.md); give every key a TTL with jitter. +- [Client-side caching](https://redis.io/docs/latest/develop/reference/client-side-caching/) (6.0+): with `CLIENT TRACKING`, Redis sends invalidation messages for keys a client has read, so it can keep a local copy. +- Store objects as hashes when you update single fields; as serialized strings when you always read the whole object. + +### Counters and rate limiters + +- `INCR` then `EXPIRE` as two calls can leave a counter without a TTL if the client dies between them — wrap them in `MULTI` or a Lua script. +- Since 8.8, [`INCREX`](https://redis.io/docs/latest/commands/increx/) increments with bounds and an expiry in one command; with `EX ... ENX` the TTL is set only on creation, giving a fixed-window limiter. +- Sliding-window log: a sorted set of timestamps — `ZREMRANGEBYSCORE` old entries, `ZADD` now, `ZCARD`; algorithm trade-offs in [system.md](system.md). + +## Operations + +### Memory + +- Redis uses [jemalloc](https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/memory-optimization/); `mem_fragmentation_ratio` (RSS ÷ used) below 1 means swapping; a high value can be fragmentation, but confirm with `allocator_frag_ratio` and absolute bytes before enabling `activedefrag`. +- [`MEMORY USAGE`](https://redis.io/docs/latest/commands/memory-usage/) reports one key's footprint; `INFO memory` the totals. +- Freed memory isn't always returned to the OS right away, so RSS can stay high after deletes. + +### Security defaults + +- The shipped `redis.conf` binds to `127.0.0.1`, and [protected mode](https://redis.io/docs/latest/operate/oss_and_stack/management/security/#protected-mode) refuses remote clients when no password is set; exposing it without auth invites remote takeover (`CONFIG SET dir` + `SAVE`). +- [ACLs](https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/) (6.0+) give users per-command and per-key-pattern permissions; TLS is built in since 6.0. + +### Licensing and Valkey + +- Redis 7.4 (2024) moved from BSD to RSALv2/SSPLv1; the Linux Foundation forked 7.2.4 as [Valkey](https://valkey.io/) (BSD), which several clouds now offer as managed Redis. +- [Redis 8.0](https://github.com/redis/redis/releases/tag/8.0.0) added AGPLv3 as a third license option and renamed Community Edition to Redis Open Source. diff --git a/docs/topics/system.md b/docs/topics/system.md index 8327540..b46bb03 100644 --- a/docs/topics/system.md +++ b/docs/topics/system.md @@ -81,7 +81,25 @@ Key system design building blocks and trade-offs, grouped by subtopic. For CAP, ### SQL vs NoSQL - [NoSQL](https://en.wikipedia.org/wiki/NoSQL) families by access pattern: [key-value](https://en.wikipedia.org/wiki/Key%E2%80%93value_database), [document](https://en.wikipedia.org/wiki/Document-oriented_database), [wide-column](https://en.wikipedia.org/wiki/Wide-column_store) (heavy writes, time series), [graph](https://en.wikipedia.org/wiki/Graph_database); model tables around queries, not entities. -- Pick NoSQL only for a named reason (write volume, schema flexibility, access pattern). Internals in [database.md](database.md). +- Pick NoSQL only for a named reason (write volume, schema flexibility, access pattern). Product internals in [postgresql.md](postgresql.md) and [redis.md](redis.md). + +### B-tree vs LSM-tree + +| | [B-tree](https://en.wikipedia.org/wiki/B-tree) (PostgreSQL, MySQL) | [LSM-tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree) ([RocksDB](https://rocksdb.org/), Cassandra) | +|---|---|---| +| Writes | in-place page updates, random I/O | append to an in-memory memtable → flush immutable sorted files ([SSTables](https://github.com/facebook/rocksdb/wiki/Rocksdb-BlockBasedTable-Format)) | +| Reads | one tree walk, predictable | may check several files; [bloom filters](https://en.wikipedia.org/wiki/Bloom_filter) skip most | +| Background work | page splits, vacuum | [compaction](https://github.com/facebook/rocksdb/wiki/Compaction) | +| Fits | read-heavy, range scans | write-heavy | + +- Compaction strategy trades [read, write, and space amplification](https://smalldatum.blogspot.com/2015/11/read-write-space-amplification-pick-2_23.html) against each other. + +### Search and analytics stores + +- An [inverted index](https://en.wikipedia.org/wiki/Inverted_index) maps each term to the documents containing it; relevance scoring is usually [BM25](https://en.wikipedia.org/wiki/Okapi_BM25). +- Elasticsearch/OpenSearch are [near-real-time](https://www.elastic.co/docs/manage-data/data-store/near-real-time-search) (new docs searchable after a refresh, ~1 s default), and the primary shard count is fixed at index creation — resize by reindexing or [split](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-split)/shrink. +- [OLAP](https://en.wikipedia.org/wiki/Online_analytical_processing) stores use [column storage](https://en.wikipedia.org/wiki/Column-oriented_DBMS), compression, and vectorized scans of few columns over many rows, with [star schemas](https://en.wikipedia.org/wiki/Star_schema) (facts + dimensions). +- Treat search indexes and warehouses as derived stores, fed by [CDC](distributed.md) from the source of truth. ### Scaling ladder @@ -262,7 +280,7 @@ Key system design building blocks and trade-offs, grouped by subtopic. For CAP, ### Leaderboard -- A Redis [sorted set](https://redis.io/docs/latest/develop/data-types/sorted-sets/): [`ZINCRBY`](https://redis.io/docs/latest/commands/zincrby/) updates a score and [`ZREVRANK`](https://redis.io/docs/latest/commands/zrevrank/) returns a user's rank in O(log n); [`ZRANGE ... REV`](https://redis.io/docs/latest/commands/zrange/) reads the top k in O(log n + k). +- A Redis [sorted set](https://redis.io/docs/latest/develop/data-types/sorted-sets/): [`ZINCRBY`](https://redis.io/docs/latest/commands/zincrby/) updates a score and [`ZREVRANK`](https://redis.io/docs/latest/commands/zrevrank/) returns a user's rank in O(log n); [`ZRANGE ... REV`](https://redis.io/docs/latest/commands/zrange/) reads the top k in O(log n + k); internals in [redis.md](redis.md). - Beyond one node, shard by score range or keep per-shard top-k and merge. ### Booking and inventory contention diff --git a/zensical.toml b/zensical.toml index 5fd88c6..2329d5d 100644 --- a/zensical.toml +++ b/zensical.toml @@ -9,7 +9,6 @@ nav = [ { "Home" = "index.md" }, { "Fundamentals" = [ { "Algorithms" = "topics/algorithms.md" }, - { "Databases" = "topics/database.md" }, { "Distributed Systems" = "topics/distributed.md" }, { "Networking" = "topics/networking.md" }, { "Operating Systems" = "topics/os.md" }, @@ -33,6 +32,8 @@ nav = [ { "Infrastructure" = [ { "AWS" = "topics/aws.md" }, { "DevOps" = "topics/devops.md" }, + { "PostgreSQL" = "topics/postgresql.md" }, + { "Redis" = "topics/redis.md" }, ] }, { "Domains" = [ { "AI Engineering" = "topics/ai.md" },