Raqueeb

Systems · 2026

KVStore

Durable, crash-tested key-value database from scratch in Go — stdlib only

Complete · source available

Overview

  • What: a durable, concurrent key-value database written from scratch in Go, with a custom binary TCP protocol and no external dependencies.
  • Hardest problem: making an acknowledged write survive kill -9, which means getting the write-ahead log, group commit and recovery right.
  • How it is validated: crash tests that SIGKILL real server processes, chaos tests, differential tests against a reference map, fuzzing and the race detector.

KVStore is a durable, concurrent key-value store with a sharded in-memory engine, a write-ahead log with group commit and CRC32C checksums, snapshots, TTL expiry, memory-bounded eviction and asynchronous replication. It uses no Redis, RocksDB or embedded database. Its central claim, that acknowledged writes survive kill -9, is tested by SIGKILLing real server processes.

Architecture

KVStore write path A client sends a binary TCP frame to a shard router. The shard holds the map, LRU and TTL state and appends to a write-ahead log with group commit and CRC32C checksums, which is fsynced. The log also feeds an asynchronous replica; the shard can write a snapshot. client binary TCP frame shard router shard n map · LRU · TTL WAL group commit · CRC32C fsync snapshot replica (async) tooling kvctl kvbench wal verify
fig. 01 — kvstore write path, with the kvctl, kvbench and wal verify tools · acknowledged writes survive kill -9

Highlights

  • Binary protocol with a 16-byte little-endian header, explicit framing, pipelining via request_id, and hard bounds validated before allocation; the decoder is fuzzed (FuzzDecodeFrame asserts no panics, no over-allocation, no out-of-bounds slices).
  • Write-ahead log with CRC32C records, segment rotation, group commit and three fsync policies (always | everysec | no); recovery = snapshot + WAL replay, and it distinguishes a torn tail from real corruption.
  • Three interchangeable concurrency engines (sharded | global | actor), three expiry mechanisms (sampled sweeper, hierarchical timing wheel, lazy), and eviction policies allkeys-lru | volatile-lru | noeviction (sampled LRU, K=5).
  • Async replication with partial and full resync, lag metrics and epoch fencing; replicas are read-only and apply the primary's expiry deletes rather than their own clock. Manual promotion only: the README states up front there is no leader election, and why (ADR-014).
  • Test suites: test/crash (SIGKILL at random moments, 24 scenarios), test/chaos (forged 4 GiB length fields, TLS handshakes to the KV port, clients that never read), test/differential (random op sequences vs a 20-line reference map, failures auto-shrunk), cluster tests, race detector, fuzzing.
  • Tooling: kvctl client plus an offline WAL inspector (wal verify | stats | dump | replay), and kvbench, an open/closed-loop load generator with a log-linear latency histogram. Builds on Linux, macOS and Windows; docs include DESIGN, PROTOCOL, WAL, DECISIONS (ADRs) and BENCHMARKS.

What I learned

Recovery has to tell a torn tail (an interrupted last write) apart from real corruption, otherwise either good data is thrown away or damage is silently accepted. The differential test against a 20-line reference map, with auto-shrunk failures, is a check I trust more than hand-written cases. Writing the no-leader-election position down as an ADR forced me to be precise about what replication does and does not promise.

Limits

Replication is asynchronous with manual promotion only; there is no leader election (ADR-014).