Helios
Distributed Job Scheduler
A multi-node cron scheduler that elects a single leader over etcd and dispatches jobs through Kafka, guaranteeing that a scheduled job is claimed and executed exactly once — even while nodes crash or get partitioned.
- 1
- active leader at any instant
- Once
- execution per scheduled job
- 5s
- dispatch & outbox poll loop
- CRUD
- REST API for jobs & cluster state
Overview
Helios stores cron-defined jobs, elects one leader via etcd, and dispatches due work through Kafka using a transactional outbox. Follower nodes run the worker loop and execute HTTP jobs from their JSON payloads. The same deployable runs as multiple instances on Docker Compose, with role behaviour decided at runtime.
The problem
Distributed schedulers fail in the gaps: a leader that thinks it is still in charge after a partition, two workers grabbing the same job, or a crash between writing job state and publishing the dispatch message. Each of those gaps turns into a duplicate fire, a missed job, or a corrupt record.
Helios treats leader state, dispatch, and job claiming as three separate problems, and solves each with a mechanism that is correct under failure rather than only on the happy path.
Architecture
Coordination layer
etcd holds the leadership key. Nodes campaign with a CAS transaction backed by a lease and renew it through heartbeats; losing the lease means losing the right to schedule.
Control plane
The elected leader runs the dispatch loop — reading due jobs, writing outbox rows, and publishing them to the jobs topic. Only the leader ever schedules.
Message broker
Kafka decouples dispatch from execution so workers can scale and recover independently of the scheduler.
Execution layer
Workers claim jobs from PostgreSQL with SKIP LOCKED polling and execute them, updating job state transactionally on completion.
Design decisions
etcd compare-and-swap leader election
Leadership is claimed with an etcd CAS transaction on a single key backed by a lease. A heartbeat keeps the lease alive, and a node that loses the lease stops dispatching immediately — so there is at most one active leader at any instant.
Fencing tokens against stale writes
Every leadership epoch carries a monotonically increasing token. Writes stamped with a token older than the last accepted one are rejected before they can reach the data layer, so a demoted or partitioned leader can never corrupt state.
Kafka dispatch + SKIP LOCKED claiming
The leader publishes due jobs to a Kafka topic; workers consume independently and claim their work with SELECT … FOR UPDATE SKIP LOCKED. Two workers can never lock and execute the same pending job.
Transactional outbox & dead-letter handling
Job-state changes and their dispatch records are written in a single database transaction to an outbox table, giving atomic DB-then-broker delivery. Jobs stranded after a crash are recovered asynchronously, and tasks stuck in repeated-failure loops are quarantined in a dead-letter queue.
Failure & concurrency
Split-brain during a network partition
CAS election plus lease expiry means a partitioned node cannot keep acting as leader; the surviving majority elects a new one and stale writes are fenced out by epoch tokens.
Duplicate execution
Because claims use FOR UPDATE SKIP LOCKED, concurrent workers skip rows already being processed instead of racing for them.
Crash between DB write and broker publish
The outbox row is committed atomically with the job state, so dispatch is retried from durable state rather than lost.
Poison jobs
Repeatedly failing jobs leave the hot path and land in a dead-letter queue instead of starving healthy work.
Testing & verification
- Runs as a multi-node environment via Docker Compose, exercising real leader election and worker coordination.
- Spring Boot context test verifies the application wires up cleanly.
- Outbox and job status tables are inspected to confirm dispatch state stays flat under load.