I came across an article about agentic system design this week. One sentence stuck with me:
"These failures are ordinary, and you can plan for them."
We learned that lesson the hard way, in production.
We run an AI-powered design pipeline on a Windows machine. A worker picks up tasks and executes them through a model CLI using a subscription plan. Separate operator processes handle the generated images and uploads.
On October 2, we hit the subscription's session limit.
Our worker treated the limit message as a regular error and retried the same task every second. By the end of the day, we'd accumulated roughly 900 attempts.
About two hours later, every new process launch started failing with 0xC0000142 (STATUS_DLL_INIT_FAILED). The machine only recovered after a reboot.
Initially, we blamed the model CLI. But the same command worked before and after the incident. Looking at the logs, our retry loop became the more likely explanation. We haven't proven a direct causal link, though.
We also had other infrastructure problems during that production round (28 screens, ~42 generations):
- An upload folder was deleted, losing 7 results.
- Two operators resumed after sleep and uploaded images to the wrong screens.
- Operators logged successful uploads even when delivery had failed.
At least 8 generations (~19%) had to be redone because of infrastructure issues, not model quality.
The biggest lesson: A usage limit is a state, not an ordinary error.
Our worker now handles limits separately:
result = run_model_cli(task)
switch classify(result):
case LIMIT:
sleep_until(reset_time)
case ERROR:
backoff = next_step(backoff)
requeue(task, after=backoff)
We also added:
- PID + heartbeat-based operator locks.
- Upload verification instead of trusting success logs.
- Exponential backoff and retry limits.
- A plan to introduce execution budgets, job leases and better run tracking.
We considered switching agent frameworks, but realized the problem wasn't the framework. It was missing reliability mechanisms in our existing system.
Our next production target is zero infrastructure-caused regenerations. The fixes haven't been validated in another production round yet.
With HTTP APIs, you usually get a 429 and sometimes a Retry-After header.
But with CLI tools running on subscription plans, you often just get a text message saying you've reached a usage limit.
How do you reliably distinguish a subscription limit from an actual execution failure?
And how do you determine when the limit resets?
Do you parse CLI output, track usage yourself, or use another approach?
I'd especially like to hear from people running autonomous workers with subscription-based AI CLIs in production.