Engines

The eight embedded Go key/value engines kvbench measures, one page each: what it is, what it is best at with real numbers, and what to watch out for.

Eight embedded key/value engines, all pure Go, all added with go get, all measured through the identical harness. Each page below is a short card: the shape, the best workload with real numbers, the weak spot, and when to reach for it.

At a glance

Throughput is operations per second on the Apple M4, 100,000 keys of 1 KB, 8 concurrent clients, disk flush off. Space is on-disk bytes per byte of data after fresh writes.

Engine	Shape	Random reads	Fresh writes	Ordered scan	Space	Durable writes
tamnd/kv	hash-log	6,848,000	83,000	no	5.0x	740
badger	LSM	594,000	239,000	yes	22x	16,000
pebble	LSM	856,000	97,000	yes	0.26x	980
bbolt	B+tree	865,000	38,000	yes	2.3x	110
buntdb	in-memory B-tree	3,572,000	230,000	yes	1.0x	250
pogreb	hash-log	4,008,000	190,000	no	2.0x	360
goleveldb	LSM	1,032,000	92,000	yes	0.15x	1,100
sqlite	B-tree	45,000	29,000	yes	4.5x	17,000

No engine wins every column. The hash-logs own reads and lose on scans and update churn; the LSMs own writes and disk footprint; the B-trees own ordered scans and durable batching. Pick by the column your workload lives in, then read that engine's page.

What "shape" means

hash-log appends every write to one file with an in-memory index. Fastest point reads, no ordered scan, update churn.
LSM buffers writes and merges them in the background. Cheap writes, small on disk, occasional merge latency.
B+tree keeps keys sorted in pages. Great ordered scans, slower random writes.

The start-here pages explain the shapes in full.

tamnd/kv tamnd/kv is a single-file embedded store with a hash-log core and full transactions. It reads faster than any engine measured, and is weak on update-heavy workloads. badger badger is a pure-Go LSM key/value store with a separate value log. Fast writes, the lowest tail on mixed workloads, group-commit durability, and a large disk footprint. pebble pebble is the pure-Go LSM that backs CockroachDB. It scales writes across cores, compresses data to a quarter of its size, and is the safest default for a server workload. bbolt bbolt is a single-file copy-on-write B+tree, the storage under etcd. The simplest engine here, the fastest ordered scanner, and slow at random and durable writes. buntdb buntdb is an in-memory B-tree with an append-only file for durability. Fast at almost everything, as long as the whole dataset fits in RAM. pogreb pogreb is a pure-Go hash-log built for fast point reads on datasets larger than memory. Like tamnd/kv it has no ordered scan, but it holds up better under updates. goleveldb goleveldb is a pure-Go port of LevelDB. No standout strength, no glaring weakness, and the smallest disk footprint of any engine measured. sqlite sqlite (the pure-Go modernc build) is a full SQL database used as a key/value store. Slowest at raw key/value ops, but it brings real SQL and group-commit durability.