Skip to content

Projects

Valka

A distributed task queue in Rust that only needs S3.

In progressLive demoSource
  • Rust
  • Tokio
  • gRPC
  • S3
  • TLA+
  • The Valka site: the headline beside a terminal showing a three-node cluster after one node was killed
The Valka site: the headline beside a terminal showing a three-node cluster after one node was killed

Valka is a task queue: you hand it jobs, it hands them to workers, and it makes sure none of them get lost. Most queues need a database, a message broker and a cache to do that. Valka needs one thing, an S3 bucket, and nothing else.

Every job is saved to the bucket before Valka says OK, so a server can die at any moment and another one picks up its work. Failed jobs retry on their own, jobs that keep failing are set aside for a person to look at, and jobs can be scheduled for later. There are SDKs for Rust, TypeScript, Go and Python.

I built it to see how small a reliable queue could be. It's tested by breaking it on purpose: killing servers mid-job, failing storage calls, and checking that nothing acknowledged ever goes missing. Still in progress. Next up is publishing the SDKs and a Helm chart.