Brokers are stateful
Every broker owns local JBOD disks, a partition directory tree, and years of partition-reassignment scar tissue. Scaling up means moving terabytes of data between machines.
Single binary · Go 1.27 · Apache Kafka® wire compatible
Kafka's protocol.
Kafka's protocol.
Object storage's economics.
Kimistore is a lightweight streaming agent that speaks the Apache Kafka binary protocol and keeps your entire log in S3. No ZooKeeper. No KRaft quorum. No replicated broker disks. Point your existing producers and consumers at it and go.
- 0 ZooKeeper nodes
- 1 binary to deploy
- 27 MB compiled size
- AGPL-3.0 licensed
$ kcat -P -b localhost:19092 -t orders
% Reached end of topic orders (Produced 1 new message)
$ kcat -C -b localhost:19092 -t orders -o beginning -e -q
order-1042 paid 99.00Computestateless agent
Hot tierlocal WAL, 64 MB segments
Cold tierS3 / RustFS / any S3 API
Coordinationbuilt-in lite coordinator
librdkafkaconfluent-kafkaJava clientkafka-gokcatPythonGo.NETlibrdkafkaconfluent-kafkaJava clientkafka-gokcatPythonGo.NET
Running Kafka is a distributed systems project.
Using it shouldn't be.
A production Kafka deployment is a quorum, a controller, partitioned storage across every broker, replication, and a metadata log — plus the disk estate that comes with it. Most teams don't need that. Most teams need a durable, ordered, replayable log they can hand to existing clients.
The cost scales with data
Replication triples your storage bill to survive a disk failure, and you're paying it for the full retention window — including the 99% of data nobody reads.
Operations are a full-time job
ZooKeeper or KRaft, partition leaders, ISR shrink events, under-replicated partitions, consumer group rebalances. Kafka is a product. It's also a platform you now operate.
Kimistore inverts the model.
The agent holds no state worth keeping. The WAL is a write buffer, not a system of record. Sealed segments live in S3, consumer group offsets live in S3, and the metadata checkpoint lives in S3. Kill the agent, start a new one on a different machine, and it picks up exactly where the last one left off. That is what makes it scale like serverless and cost like object storage.
Four pillars, no platform tax
Protocol compatibility
18 Kafka API keys are implemented — Produce, Fetch, ListOffsets, Metadata, the full consumer group lifecycle, topic admin, and SASL PLAIN. MessageSet v0/v1 and RecordBatch v2, with GZIP, Snappy, and LZ4 decoded so offset accounting stays exact.
Your existing clients don't need to know anything changed.
Tiered storage that actually tiers
Writes append to a local WAL and acknowledge immediately. A rolling 64 MB segment is sealed and handed to an eight-worker upload pool, with a reconciliation loop as a safety net. Reads check the hot WAL first and transparently fall through to ranged object-store reads using a sparse segment index.
A group coordinator built in
JoinGroup, SyncGroup, Heartbeat, LeaveGroup, and leader-driven assignment with KIP-62 background heartbeats and dynamic rebalancing. Offsets are buffered and flushed to S3, so a restart resumes from the right place instead of reprocessing the world.
Operable in production
Prometheus metrics on :9091 covering request counts and latency
histograms, network throughput, connections, uploader health, and topic/partition
cardinality. Plus SASL/PLAIN auth, time- and size-based retention, and metadata
checkpointing.
One binary, three layers, two storage tiers
Protocol on the front, a storage engine underneath, and a coordinator in the middle. The write path never blocks on the network.
Write
Append & acknowledge
Records land in the partition's local WAL and the producer gets its ack. No network round trip to object storage on the hot path.
Seal
Roll at 64 MB
The active segment rolls, the seal event is pushed onto a buffered channel, and eight upload workers pick it up in parallel.
Serve
Hot reads, cold fallback
Consumers tail the WAL for live data and transparently fall back to range reads against S3 for history. One API, two tiers.
Kafka is now the centre of the Grafana write path.
Grafana Mimir 3.0 made ingest storage architecture stable and the recommended way to run: Kafka sits between distributors and ingesters, and the write path ends at Kafka. Grafana Tempo's microservices mode uses a Kafka-compatible queue as the durable write-ahead log behind every write. Both documents point at the same door — the Mimir docs say its use of the Kafka protocol is deliberately limited precisely so you can bring a compatible system instead of Apache Kafka.
That is the workload Kimistore was built for: a write path that must be durably acknowledged, replayed by several independent consumers, and retained for a window measured in hours or days — on storage that costs pennies per gigabyte.
0
coordination services — no ZooKeeper, no KRaft quorum
1×
durable copy on object storage, not three broker replicas
acks=all
the durability contract both Grafana services expect
3
consumer groups in a Tempo deployment, each with its own offsets
Write
Distributor → Kimistore
The distributor hashes each series into a topic partition, produces with
acks=all, and does not reply to the client — be that an OTel collector
or a Prometheus remote-write sender — until the broker confirms. Tempo does the
same and documents it as "the Kafka protocol's strongest acknowledgment mode".
Persist
Group-committed fsync
The ack is a real durability promise: the batch is fsynced to the local WAL before the offset is returned, with concurrent producers sharing a single flush. Sealed segments are offloaded to object storage in the background, and flushed on shutdown, so an ephemeral local disk loses nothing.
Read
Ingesters consume, independently
Every Mimir ingester runs its own consumer group, so a zone can be
restarted or rolled back independently. Tempo runs three — block-builder,
live-store and metrics-generator — each tracking its own
offsets at its own pace. One lagging group never blocks another.
Grafana Mimir
3.0+
Ingest storage architecture
Mimir's classic architecture keeps ingesters heavily stateful: they combine in-memory data with local write-ahead logs and sit on both the write and the read path, so heavy queries disrupt live writes. Ingest storage architecture decouples the two, and the write path ends at Kafka — a push succeeds once the broker confirms, without an ingester quorum.
- One consumer group per ingester zone, so HA on the read path and independent rollback
- Partition assignment derived from the instance ID — the agent serves plain
Metadatafor it - Strong read consistency makes the query-frontend read partition offsets from Kafka and gate ingesters on them: a hard
ListOffsetsrequirement, answered from durable state consume-from-position-at-startupresolves to earliest/latest/a timestamp through the same API- No transactions and no idempotency required — the agent implements neither, and is not asked to
In this architecture the broker is a durable hand-off point, not a queryable store — the samples are written again downstream into per-tenant TSDB blocks. A node that can be torn down and recreated is therefore no longer a liability.
Grafana Tempo
microservices mode
Kafka as the durable write-ahead log
In microservices mode Tempo uses a Kafka-compatible queue as the WAL between
distributors and every downstream consumer: block-builders, live-stores and
metrics-generators. Because durability is centralised, Tempo does not have to
replicate data across instances on the write path — it runs with a
replication factor of 1. Monolithic mode
(target: all) pushes in-process and never touches Kafka.
- Distributors write with
acks=alland wait for backend confirmation before answering the client - One topic for all tenants, hashed by trace ID to an active partition
- Three independent consumer groups, each with its own offsets and its own pace
- The partition ring maps Tempo partitions to Kafka partitions, typically 1:1
- Kafka retention defines the replay budget — and lag is tracked per group via
tempo_ingest_group_partition_lag
Tempo's design already runs the write path at replication factor 1. Here that single copy costs object storage rather than three provisioned broker disks.
| What Grafana requires of a Kafka-compatible backend | Who needs it | Kimistore |
|---|---|---|
| Durable acknowledgement before the client is answered | Mimir distributor, Tempo distributor | Yes group-committed fsync for acks>=1 |
| Metadata for topic and leader discovery | Both distributors and ingesters | Yes Metadata v0–v6, configurable advertised listener |
| Several independent consumer groups | Mimir per ingester zone; Tempo block-builder, live-store, metrics-generator |
Yes per-group offsets, persisted to object storage and rehydrated |
| Accurate partition offsets for read consistency and startup positioning | Strong read consistency, consume-from-position-at-startup |
Yes real log start and log end, correct across restarts |
| Replay window measured in hours or days | Block-builder cycle, live-store startup replay | Yes time and size retention on the object-store tier |
| No transactions, no idempotent writes | Both, by design | Neither implemented or required |
| Multi-broker replication across failure domains | Optional for them, never required | Not implemented — by design |
Sources: Mimir ingest storage architecture · Tempo’s Kafka component · Tempo: configure a Kafka-compatible backend
Why the economics work out
A replicated Kafka ingest path pays for every write three times, plus the broker disks, the controller quorum and the capacity plan. In both Grafana architectures Kafka is a durable hand-off point rather than a queryable store: the data is written again into per-tenant TSDB blocks by the ingesters or block-builders, and read from there. Kimistore keeps exactly one durable copy, in object storage, and exists only to get samples durably acknowledged and replayable. That is the layer where a single-node design is not a compromise but the whole point.
Where Kimistore fits
A straight answer, including where Kimistore is the wrong tool.
| Dimension | Apache Kafka | Managed Kafka | Kimistore |
|---|---|---|---|
| Client compatibility | Native | Native | Yes subset of the protocol |
| Coordination | KRaft / ZooKeeper | Vendor-managed | None built-in lite coordinator |
| Deployment unit | Broker cluster | Managed service | One binary |
| Storage system of record | Local JBOD disks | Vendor hardware | S3 / object storage |
| Replication | 3× by default | 3× by default | None — single node |
| Transactions | Yes | Yes | Not supported |
| Best for | Multi-region, high-scale event backbone | Teams that want zero ops at any cost | Single-region streaming, stream-to-lake ingestion, microservices, edge pipelines, dev/staging, cost-sensitive retention |
Clear architectural scope.
Kimistore is intentionally designed as a single-node streaming agent backed by S3, eliminating the operational complexity of cluster replication, leader elections, and ISR management. If a workload requires multi-broker active clustering across failure domains with distributed transactions, upstream Kafka is the specialized solution.
What the agent actually implements
Negotiated through ApiVersions one ApiVersions table, generated from the same map the request dispatcher refuses from — so the broker can never advertise a version it then mis-parses.
Data plane
- Produce0v0–v3✓
- Fetch1v0–v5✓
- ListOffsets2v0–v2✓
- Metadata3v0–v6✓
- ApiVersions18v0✓
Consumer groups
- OffsetCommit8v0✓
- OffsetFetch9v0–v1✓
- FindCoordinator10v0✓
- JoinGroup11v0–v1✓
- SyncGroup14v0✓
- Heartbeat12v0✓
- LeaveGroup13v0✓
- DescribeGroups15v0✓
- ListGroups16v0✓
Admin & security
- CreateTopics19v0✓
- DeleteTopics20v0✓
- SaslHandshake17v0–v1✓
- SaslAuthenticate36v0✓
Message formats
- ✓ MessageSet v0 / v1 (legacy)
- ✓ RecordBatch v2 (Kafka 0.11+)
- ✓ GZIP, Snappy, LZ4 decompression
- ✓ Record-level offset rewriting on fetch, with CRC repair
- ✓ Kafka record headers preserved end to end, which Mimir’s ingest path depends on
- ✓ Log position restored from object storage, so a restart with a wiped local disk continues the log
- ✕ Transactions / idempotent producer
- ✕ Replication, partition leaders, ISR
Running in under a minute
Requires Go 1.27+, an S3-compatible bucket, and credentials. kcat is the fastest way to see it work.
# From a checkout of github.com/kimistore/agent
$ go build -o agent ./cmd/agent
$ ls -lh agent
-rwxr-xr-x 1 andy staff 27M agent# Point it at a bucket. Any S3 API store works.
$ export S3_BUCKET=kimistore
$ export AWS_REGION=us-east-1
$ export AWS_ACCESS_KEY_ID=...
$ export AWS_SECRET_ACCESS_KEY=...
$ ./agent
Starting Kimistore Agent...
Metrics listening on :9091
Listening on :19092 # Kafka protocol# Produce
$ kcat -P -b localhost:19092 -t orders
order-1042 paid 99.00
# Consume from the beginning, then exit
$ kcat -C -b localhost:19092 -t orders -o beginning -e
# Consumer group — open two terminals and watch them split the partitions
$ kcat -b localhost:19092 -G payments orders# Required
S3_BUCKET=kimistore # default: kimistore
AWS_REGION=us-east-1 # default: us-east-1
# Optional — S3-compatible stores
S3_ENDPOINT=http://localhost:9000 # enables path-style addressing
# Optional — SASL PLAIN. When set, all requests
# require authentication before dispatch.
SASL_USERNAME=admin
SASL_PASSWORD=secret
# Optional — retention. Inert until set.
KIMISTORE_RETENTION_MS=604800000 # 7 days
KIMISTORE_RETENTION_BYTES=-1 # unlimited
KIMISTORE_RETENTION_CHECK_MS=300000 # sweep every 5 min
# Prometheus metrics
curl localhost:9091/metrics
Python
from confluent_kafka import Consumer
c = Consumer({
'bootstrap.servers': 'localhost:19092',
'group.id': 'payments',
'auto.offset.reset': 'earliest',
})
c.subscribe(['orders'])
for msg in c:
print(msg.value())
Java
Properties p = new Properties();
p.put("bootstrap.servers", "localhost:19092");
p.put("group.id", "payments");
p.put("key.deserializer",
StringDeserializer.class);
p.put("value.deserializer",
StringDeserializer.class);
new KafkaConsumer<String, String>(p)
.subscribe(List.of("orders"));Beyond the Grafana stack
Dev & staging parity
One binary, one bucket. Your integration tests get a real Kafka wire protocol instead of a mock that drifts from reality.
Edge & intermittent
A laptop, a Pi, a spot instance, or a local RustFS. The agent holds nothing durable, so it can be torn down and recreated freely.
Cost-sensitive retention
Long retention windows on data that is rarely read are exactly the workload object storage wins at. The cold tier costs pennies.
Proof of concept
Validate a streaming design against real clients in an afternoon, without a capacity plan, a quorum, or a procurement conversation.
Engineered for durability and operational simplicity
Built from the ground up to deliver Kafka wire-protocol compatibility with object storage economics. Every guarantee is verified by comprehensive durability, crash-recovery, and protocol test suites.
27 MB
single static binary, zero dependencies
Built-in guarantees & capabilities
- Produce, Fetch, ListOffsets, Metadata with Kafka V0–V3 protocol support
- Acks-aware WAL durability: group-committed fsync before client ack (
acks>=1), fire-and-forget foracks=0 - Automated crash recovery with torn-tail truncation, ensuring partitions recover cleanly after unclean shutdowns
- Fetch long-polling up to
maxWaitMs(capped at 1s) to eliminate client tight polling loops - Lite group coordinator with background session reaper evicting crashed members past
session.timeout.ms - Bounded group rebalancing preventing hung followers on coordinator or leader disconnects
- S3-backed consumer offset persistence with rehydration across restarts and retry on shutdown
- Consumer-aware retention with partition-scoped S3 listing protecting active group offsets
- Transparent tiered storage: hot local WAL with 64 MB rolling segments and parallel S3 uploader pool
- MessageSet v0/v1 and RecordBatch v2 with exact record-level offset accounting across GZIP, Snappy, and LZ4
- CreateTopics, DeleteTopics, ListGroups, DescribeGroups, and SASL PLAIN authentication
- Partition ownership and routing: each agent claims the partitions it writes in S3, publishes where it can be reached, and Metadata routes clients to the agent that owns each partition
- HA consumer groups: the coordinator is picked by rendezvous hashing over the live agent set and every group request is fenced, so only the elected agent acts on a group
- Comprehensive Prometheus metrics on port 9091 covering request latencies, WAL recoveries, retention, ownership claims and cluster routing
Architectural scope & non-goals
- No replication or ISR quorums: durability comes from object storage, not replicas. One writer per partition, fenced by a compare-and-swap claim in S3, so a second agent cannot silently corrupt a partition
- Standard Kafka at-least-once streaming: no 2PC distributed transactions or transactional producer IDs
- Built-in SASL PLAIN authentication: topic-level ACL authorization is not enforced
- Dynamic consumer rebalance assignment: delegates partition assignment to group leader without cooperative sticky protocols
Questions people actually ask
Is this a Kafka replacement?
Kimistore is not a drop-in cluster substitute for multi-datacenter Kafka deployments requiring cross-broker replication, ISR quorums, or distributed transactions. Instead, it is a specialized streaming engine engineered for single-region streaming, stream-to-lake ingestion, microservices, and edge workloads. By combining the Kafka wire protocol with S3 tiered storage in a single binary, Kimistore eliminates cluster operational complexity, ZooKeeper/KRaft maintenance, and expensive broker EBS disks while maintaining seamless compatibility with standard Kafka client libraries.
What happens if the agent dies?
Sealed segments are already safe in S3. For active writes with acks=1 (or higher), Kimistore uses group-commit fsync to guarantee records are committed to stable disk before acknowledging producers. If the process crashes during an unaligned write, automatic torn-tail truncation on startup detects CRC mismatches and rolls the active segment back to the last intact record, preventing partition corruption. Consumer group offsets are continuously persisted with S3-backed rehydration across restarts and bounded shutdown flush retries. Restarting the agent with the same bucket and storage directory restores all topic metadata, consumer offsets, and high watermarks seamlessly.
What happens if a consumer crashes?
A background reaper evicts members that stop heartbeating past their session.timeout.ms, advances the group generation, and re-elects a leader if the leader was the one that died. The remaining consumers rebalance and pick up the orphaned partitions without operator action. Rebalancing waits are bounded too, so a leader that dies mid-rebalance cannot leave followers hanging. After a restart, members restored from a checkpoint are treated as dead, since their connections died with the previous process.
Will a caught-up consumer spin the CPU?
No. Fetch implements long polling. When a consumer is caught up and asks for data, the request parks on an append signal for up to maxWaitMs rather than returning empty immediately, so there is no tight re-poll loop and no wasted round trips. The wait is capped at one second, and clients that set minBytes=0 still get an immediate answer, which is what that setting means.
Does acks=1 hurt throughput?
Yes, and it is worth being precise about why. Honouring acks=1 means one device flush per batch for a single sequential producer, which is a hard floor for any durable broker. Kimistore uses group commit: concurrent producers that arrive while a flush is in flight are all covered by that one flush rather than each paying their own, so parallel producers stay fast. Producers that genuinely do not need durability should use acks=0, which skips the flush entirely. The repository ships benchmarks for both paths so you can measure the trade-off on your own hardware.
Will my existing clients just work?
For the covered API surface, yes. Kafka clients negotiate capabilities through ApiVersions, and Kimistore advertises exactly what it implements. If your application depends on transactions, idempotence, or ACLs, it will get a clear error rather than silent corruption — but it will get an error.
Can I use RustFS or another S3-compatible store?
Yes. Set S3_ENDPOINT (or AWS_ENDPOINT_URL) and the client switches to path-style addressing, which is what most self-hosted S3 API implementations expect. This is what makes local development and air-gapped environments practical.
How is this licensed?
AGPL-3.0. If you modify it and serve it over a network, you are required to offer your source. That is a deliberate choice for infrastructure software, and the license file in the repository is the authoritative text.
How fast is it?
The write path acknowledges after a local WAL append and never blocks on object storage, so producer latency is bounded by local disk rather than S3 round trips. A parallel multi-partition benchmark and a load-test harness ship with the repository — run them against your own hardware and bucket rather than trusting a number on a marketing page.
Point a Kafka client at it and see what happens.
No cluster to plan, no quorum to babysit. A 27 MB binary and a bucket.