How we moved API rate limiting to a token bucket

01 / THE OLD SETUP

Per-minute limits have a sharp edge

Fixed windows divide time into neat boxes. If a client has sixty requests available from 12:00:00 through 12:00:59, it can spend all sixty near the end of that window and another sixty just after the next one begins. From the limiter's point of view, both minutes are valid. From the API's point of view, 120 jobs may have arrived almost at once.

The opposite case felt just as odd. A client could use its allowance early, wait nearly a full minute, and still receive a 429 while work had already cleared. People naturally asked why they were receiving 429s. Our best answer never felt as solid as it should have.

We briefly considered making the minute smaller. Ten second windows make the cliff happen six times as often. Sliding window approaches are more accurate, but they store and inspect more history than we needed. This motivated the tocket bucket refill system we now have in place.

02 / THE NEW MODEL

The bucket approach

The replacement is a token bucket algorithm. An account starts with ten 'tokens' (tokens meaning rate limit currency rather than its use in AI now). An accepted request spends one and the bucket earns one token per second until it is full again. That lets a quiet client build up enough room for a short burst without letting the burst continue forever.

There is no refill timer ticking in the background. Redis stores the balance and the timestamp of the last decision. When the next request arrives, we calculate how many milliseconds have passed, add the corresponding fractional tokens, cap the result at ten, and then ask whether one full token is available.

This changes the conversation with callers. Capacity returns steadily, based on real elapsed time. An idle account comes back to a full bucket. A busy one gets a predictable pace after its opening burst. There is still a limit, of course, but it no longer has a trapdoor hidden at the 59th second.

03 / ADMISSION

The whole decision happens once

A token balance is about half the story in synchronous image processing. A request keeps compute occupied until the output is ready. So the limiter also checks how many jobs are already in flight for the account and across the service.

Those first three steps run in one Redis Lua script. That detail matters once several API instances are serving the same account. A read followed by a separate write leaves a small window where two workers can see the same token and both claim it. The script turns refill, checks, token spend, and counter increments into one ordered event.

The current defaults allow three active requests per account. Once a request is admitted, the handler releases both counters in a finally block, including error paths. Limiter keys expire after two minutes of inactivity as a last bit of housekeeping—not as the primary release mechanism.

04 / THE CLIENT CONTRACT

429 and 503 should mean different things

If an account has no token or has filled its three active slots, the API returns HTTP 429. The response includes Retry-After, and the structured error repeats the delay. If the service-wide pool is full, the response is HTTP 503 with its own retry timing. One means “your lane is full”; the other means “the road is full.”

async function processWithRetry(job) {
  const response = await send(job)

  if (response.status === 429 || response.status === 503) {
    const seconds = Number(response.headers.get("Retry-After") ?? 1)
    await wait(seconds * 1000 + Math.random() * 250)
    return processWithRetry(job)
  }

  return response
}

Real client code should also cap its attempts and keep durable job state. The important part is to honor the server's timing, add a little jitter, and retry from a bounded queue. A pool of three workers per account fits the present concurrency ceiling. Spawning fifty local workers only creates 47 tiny coordination problems.

05 / FAILURE POLICY

What if the limiter itself disappears?

There is no universally correct answer here. We chose to fail open when Redis is unavailable and record the limiter error. Image processing remains available during a control-plane failure, though the protection is temporarily weaker. For our current service, preserving the main API path was the less damaging failure mode.

That choice only works if it is visible. Redis health, limiter errors, active inference, and request latency have to live in the same operational picture. “Fail open” cannot mean “pretend nothing happened.” We also kept a log-only mode so a new policy can be observed against real traffic before it begins rejecting requests.

The token bucket system just made much more sense in our situation. Bursts have a place to go, capacity returns at a rate callers can reason about, and concurrency is considered before work enters the expensive part of the system. That's a much better contract than “wait until the clock says a new minute.”