r/apachekafka • u/PlusDistribution1915 • 7h ago
Tool Using Kafka as glue between services: what we had to build around it, and the broker we ended up writing
Hi Kafka community,
I'm a software engineer at Smartness (B2B travel tech SaaS).
Disclaimer:
- If you use Kafka as a firehose, this post probably won't interest you.
- If you use Kafka as glue between services, it might.
We were (and partially still are) Kafka users, not to manage tons of events or data, but to handle coordination and messaging between our products and microservices. So basically we used Kafka in a way it wasn't designed for, like many of us, I believe.
During the last year we built a new message queue (QueenMQ). At first it was a tiny one, only to solve head-of-line blocking in Kafka for our slow pipelines. Over the year it evolved into a distributed transactional message broker. I like to call it the ACID message broker.
Why am I here? Because using it instead of Kafka totally changed how we develop applications: we can offload to it things we would otherwise have to build into every application. It's built to make our lives easier, and every feature it has is one we use in production. It now powers 5/6 of our products.
It's a bit of a paradigm shift, and it covers a lot, so I made a map to help people quickly understand what I mean: https://queenmq.com/map/
Main pillars:
- Almost infinite (tested between 10 and 10 million) dynamic, named partitions (like 'order-123')
- Transactions that span queues, partitions, the embedded KV and timers
- ACK and NACK, with retries, as a proper message queue
- The features you usually need to build connected systems: DLQ, conflation, routing
- Tested with our own Jepsen test suite
All of this runs on a super simple HTTP protocol (curl is a valid client). It's also compatible with Kafka clients, although with limited features: we added this because we wanted to switch our small, simple services to the new broker without rewriting them.
Clients are dumb, and the broker is etcd-shaped: one Raft group per tenant. That allows things like one ordered stream across every queue of a tenant (read: super simple stream joins). The price, obviously, is the ceiling: the system doesn't scale by adding more than 3 or 5 nodes. But we tested it up to 3M msg/s with decent latency (around 15-25 ms e2e p99, fsync to two of three disks), so that ceiling will never be a bottleneck for us (on the same machines, Kafka made 8M msg/s at around 10ms e2e p99, no fsync).
Inside our company it was also hard to make people understand why to use this instead of Kafka, so I'd really like feedback from the Kafka community.
Our goal is to bring the project to CNCF one day, so honest criticism helps more than anything.
Repo is here: https://github.com/queen-mq/queen














