columnar append-only log engine · written in rust

A database for
events that only
move forward.

KeplorDB is an embeddable engine purpose-built for high-throughput, time-ordered event ingestion. No server process. No SQL parser. No background threads. You open the engine, append events, query columns.

cargo add keplordb read the docs →

src/main.rs rust · 18 lines

use keplordb::{Engine, EngineConfig, LogEvent};

let engine = Engine::open(EngineConfig {
    data_dir: "/var/log/keplor".into(),
    wal_max_events: 100_000,
})?;

// append — WAL-durable, ~900 ns per event
let mut e = LogEvent::new(ts_ns());
e.set_model("gpt-4o");
e.set_cost_nanodollars(5_000_000);
e.set_input_tokens(1000);
engine.append(&e)?;

// aggregate — SIMD column scan, ~54 µs / 100K rows
let totals = engine.aggregate(&QueryFilter::default())?;

embeddable · 0 server deps

throughput · 973K ev/s

aggregate · 54 µs

license · Apache-2.0

[01] capabilities

Existing options are either too heavy, or too general. KeplorDB is a library — not a server.

a. storage

Columnar segments

Queries touch only the columns they need. Aggregations scan contiguous arrays.

Zero-copy reads

Columns are read directly from mmap'd segment files via zerocopy — no deserialization.

Bloom filter skip

Per-segment bloom on the primary dimension. Skip entire files on mismatch.

b. throughput

AVX2 SIMD aggregation

Vectorized sum, count and filter using 256-bit registers with scalar fallback.

Arena-backed interning

Zero String allocations on the write hot path. Hash and equality resolved through a contiguous arena.

Batch append

Bulk column writes via zerocopy::IntoBytes — single write_all per column.

c. operations

WAL durability

Every event persisted to disk before returning. Configurable fsync interval, default every 64 events.

Segment-level GC

Deleting old data is rm segment_file. No compaction, no write amplification.

Embeddable

Engine::open() in your Rust binary. No TCP, no SQL, no background threads.

[02] benchmarks

Numbers from cargo bench, not slides.

Measured with Criterion across 100,000 events in 2 segments. Appends are WAL-durable; aggregates scan real, on-disk column data via mmap + SIMD.

Run cargo bench locally to reproduce on your hardware. Throughput scales linearly with batch size and CPU vector width.

hardware · intel i5-1135G7 · single thread

operation	latency	throughput
write path
batch append · 1024 ev	1.05 ms	973K ev/s
batch append · 256 ev	291 µs	878K ev/s
single append × 256	304 µs	843K ev/s
in-memory push ceiling	428 µs	2.33M ev/s
read path
aggregate · no filter	54 µs	—
aggregate · time range ½	18.5 µs	—
aggregate · user filter	1.80 ms	—
query_recent · 50	1.80 ms	—
query_recent · 1000	1.98 ms	—

[03] architecture

One write path. One read path. No threads in the middle.

write path append() ─→ WAL (disk + mem) ─→ Segment .kseg │ │ └── fsync every 64 ev │ ▼ mmap reader segment file layout ┌──────────────────────────────┐ │ header 256 B │ ├──────────────────────────────┤ │ ts_ns i64 × N │ ← binary-searchable │ metric i64 × N │ ← SIMD SUM │ counters u32 × N × 5 │ │ latencies u32 × N × 2 │ │ status · flags u16 × N │ │ dim indices u16/u8 × N×5 │ ← interned refs ├──────────────────────────────┤ │ bloom filter 128 B │ │ intern table zstd │ │ variable data zstd │ └──────────────────────────────┘

write hot path: Arena-backed InternTable with hashbrown::HashTable. Hash and equality resolved through a contiguous Vec<u8> — zero String allocations per event.
read hot path: Zero-copy column access via zerocopy::FromBytes. Segment-level time skip using min_ts/max_ts before mmap open. Lazy intern decompression.
durability: WAL-backed. On crash, Engine::open() replays the log. Max data loss = one sync interval (default: 63 events).
garbage collection: Time-partitioned segments. Dropping old data is rm — no compaction, no write amplification, no read pause.

[04] use cases

Anything append-only and time-ordered. One schema, three shapes.

src/examples/llm.rs per-request llm traces, tokens, cost, latency

[05] schema

Fixed, flat, fast. One LogEvent, ten columns.

field	type	description
id	String	Unique event identifier.
ts_ns	i64	Nanosecond timestamp. Sorted, binary-searchable per segment.
metric	i64	Primary signed metric — cost, duration, value.
counters[0..5]	u32	Five unsigned counters — tokens, bytes, retries.
latency_ms	u32	Primary latency in milliseconds.
status	u16	Status code — HTTP, gRPC, application.
flags	u16	16 boolean bitflags.
dims[0..5]	String	Five indexed, filterable dimensions. Interned per segment.
labels[0..3]	String	Three free-form string labels.
payload	String	JSON metadata — opaque to the engine.

[06] install

Cargo.toml keplordb = { git = "https://github.com/themankindproject/keplordb" }

shell $ cargo add keplordb --git https://github.com/themankindproject/keplordb

v0.1.0 · apache-2.0 · single-crate

Append events.
Query columns.
Nothing else.

cargo add keplordb view docs ↗

A database for events that only move forward.