Building reliable systems for the real world.

Distributed Job Scheduler

A multi-node cron scheduler that elects a single leader over etcd and dispatches jobs through Kafka, guaranteeing that a scheduled job is claimed and executed exactly once — even while nodes crash or get partitioned.

Key decisions

• 01

  etcd compare-and-swap leader election

  Leadership is claimed with an etcd CAS transaction on a single key backed by a lease. A heartbeat keeps the lease alive, and a node that loses the lease stops dispatching immediately — so there is at most one active leader at any instant.

• 02

  Fencing tokens against stale writes

  Every leadership epoch carries a monotonically increasing token. Writes stamped with a token older than the last accepted one are rejected before they can reach the data layer, so a demoted or partitioned leader can never corrupt state.

• 03

  Kafka dispatch + SKIP LOCKED claiming

  The leader publishes due jobs to a Kafka topic; workers consume independently and claim their work with SELECT … FOR UPDATE SKIP LOCKED. Two workers can never lock and execute the same pending job.

• 04

  Transactional outbox & dead-letter handling

  Job-state changes and their dispatch records are written in a single database transaction to an outbox table, giving atomic DB-then-broker delivery. Jobs stranded after a crash are recovered asynchronously, and tasks stuck in repeated-failure loops are quarantined in a dead-letter queue.

Financial Ledger API

An ACID-compliant double-entry ledger REST API that transfers funds with exactly-once semantics, staying correct under hundreds of concurrent threads and deadlock-free by construction.

Key decisions

• 01

  Pessimistic write locks

  Both participating accounts are locked with PESSIMISTIC_WRITE for the critical section. This blocks conflicting writers at the database level and gives deterministic, predictable latency instead of optimistic retry storms.

• 02

  Deterministic UUID lock ordering

  Locks are always acquired lower-UUID first. If every transfer follows the same order, a circular wait — and therefore a deadlock — becomes structurally impossible.

• 03

  Idempotency keys

  Header-supplied keys are persisted under a unique index. Duplicate requests short-circuit on an O(1) indexed lookup before any lock is taken, so a network retry can never double-spend.

• 04

  Immutable append-only ledger

  Balances are derived by summing append-only DEBIT/CREDIT entries. The ledger is the single source of truth, every mutation is auditable, and there is no separate balance row that can drift out of sync.

• 05

  LOCK → CHECK → ACT in one transaction

  Validation, locking, and the debit/credit pair all commit inside a single @Transactional(timeout = 10) boundary. Any failure rolls the whole transfer back within a ten-second guardrail.

Distributed Rate Limiter

A Redis-backed rate limiter that smooths bursty traffic with a sliding-window weighted average and enforces limits before a request ever reaches application logic.

Key decisions

• 01

  Sliding-window weighted average

  weighted_avg = (elapsed × current + (60 − elapsed) × previous) / 60 blends the current and previous minute, so the limit transitions smoothly instead of snapping open at the minute boundary.

• 02

  Redis-backed shared state

  Counters live in Redis and are keyed per client IP, so the limit holds across every instance of the service rather than only within a single process.

• 03

  Enforcement before application logic

  A custom servlet filter registered with highest precedence rate-checks every incoming request before it reaches a controller and returns HTTP 429 on violation.

• 04

  Five-minute key expiry

  Keys expire automatically, so stale per-IP counters never pile up and Redis memory stays bounded without a cleanup job.