By Super Washington BangaPost-mortem
Ledger-Core v1.0Redis idempotency keys — tuning the reconciliation retry storm
Post-mortem on a retry storm that pushed 12 duplicate outbox deliveries through Redis: the TTL mismatch, the idempotency-key fix, and the monitoring gap that hid it for three hours.
What Worked
- Idempotency keys keyed on the outbox message id — a natural, stable key
- A replay harness that replayed the storm from the real event log
What Failed
- Retry TTL outlived the idempotency key TTL, so duplicates slipped through
- The rate limiter counted key hits before the idempotency check ran
- No alert fired on duplicate-key rejections for three hours
Key Lessons
- Key TTL must outlive the longest retry window
- Check idempotency before rate limiting, not after
- Rejections of duplicates are a signal worth alerting on
What happened
A bad deploy retried a batch of outbox deliveries 12 times in 20 minutes. The retry TTL had been set longer than the idempotency key TTL, so keys expired mid-storm and duplicates reached the ledger worker. The rate limiter rejected most of them — but it ran before the idempotency check, so the rejection reason was wrong and no alert fired for three hours.
The fix
Two ordering rules, now encoded in tests:
- Idempotency key TTL always outlives the longest retry window.
- The pipeline checks idempotency before rate limiting, so every rejection carries the true cause.