Building Production AI Agents That Actually Work

The failure below is one of mine, on this site, with the commit that fixed it. Where a claim is a recommendation rather than a measurement, I say so.

Most writing about AI agents argues about prompts. Almost none of it is about the thing that actually takes agents down, which is the plumbing around the model call.

Here is a failure from my own site. It is small, it is boring, and it is exactly the shape of the ones that hurt.

An agent's reliability is decided by the timeout of the thing that calls it, not by the prompt inside it.

What actually broke on my own system?

A queued job that calls a language model wrote nothing, three times, and left no error a caller could see. The job was GeneratePostSeoJob, which asks a model to write SEO fields for a blog post.

The cause was two timeouts that disagreed. Laravel Horizon gives a worker 60 seconds. The service that makes the model call allows a single request 120. Neither number was wrong on its own. Together they guaranteed that any post long enough to take the model past a minute would be killed mid-request.

The production failed_jobs table recorded it plainly: Illuminate\Queue\TimeoutExceededException: App\Jobs\GeneratePostSeoJob has timed out, at 10:24:47 on 19 September 2026. The fix landed at 10:47:39 the same morning.

Continue Reading

Cutting LLM API Costs 60%: A Production RAG Post-Mortem

11 min

AI Agents Ate My Boilerplate: What Actually Works in Enterprise Banking

11 min

Why is a mismatched timeout worse than a crash?

Because a crash tells you. This did not.

The worker killed the request while the model was still answering. No response arrived, so no parsing failed, no validation tripped, and no handler ran. The job simply stopped. Horizon retried it twice more, hit the same wall twice more, and only then wrote one TimeoutExceededException with no mention of the model, the post, or the 120-second call it interrupted.

A timeout that fires inside a call you cannot see does not produce an error. It produces a gap, and gaps do not page anyone.

The rule I wrote into the code as a comment, so the next person cannot miss it: any queued job that reaches the model must declare a $timeout larger than the model call's own ceiling. Both model-calling jobs in this repo now sit at 150 against a 120-second call.

How do you size the timeout for a job that calls a model?

Work outwards from the slowest thing, not inwards from the platform's ceiling. The model call is the innermost budget. Every layer wrapping it needs headroom, and the outermost layer is usually a platform limit you do not control.

An earlier version of this post claimed Convex actions time out at 30 seconds and budgeted 25 seconds against it. That is wrong, and I have corrected it. The documented limits are 30 minutes and 10 minutes. The lesson survives the correction, which is that you should never guess a platform limit. Read it, write it down next to your own budget, and make the gap between them deliberate.

What should an agent do when the model returns nothing usable?

Fall back, every time, on a path that cannot itself call the model. Three different failures collapse into the same answer in my own pipeline, and none of them raise to the caller.

The model returns null, the response does not decode as JSON, or processing it throws. In all three cases PostAiService::analyzeContent() logs and returns fallbackAnalysis(), a heuristic that reads the post and produces a summary without any model at all.

That is the part most agent tutorials skip. A fallback that calls a cheaper model is still a fallback that can fail. Somewhere at the bottom of the ladder there has to be a rung made of plain code.

Decide the ladder before the incident. During one you will pick whichever rung is nearest, and that is how a system ends up with no rung 4.

How do you force a model to return a shape you can trust?

Ask for a schema and refuse anything else. In TypeScript that is generateObject from the Vercel AI SDK with a Zod schema, so the SDK retries or fails on invalid output instead of handing you a broken object.

const ToolCallSchema = z.object({
  tool: z.enum(['search_orders', 'create_ticket', 'escalate']),
  params: z.record(z.unknown()),
  confidence: z.number().min(0).max(1),
});

A schema is necessary and not sufficient. Valid JSON can still be wrong, and the interesting checks are the ones your schema cannot express.

My own pipeline has one worth copying. The model is asked for a summary of at most 60 words and overshoots that budget often enough to matter. The frontend truncates on sentence boundaries, so an overshooting summary loses its final clause, which is the part carrying the outcome and therefore the part worth quoting. So the same word clamp the heuristic fallback uses is applied to the model's output too.

Schema validation asks whether the shape is right. Business validation asks whether the value is usable. A refund larger than the order passes every schema you can write.

Which retries help, and which ones just cost money?

Retry the failures that are about the network and the provider. Never retry the ones that are about your own request, because they will fail identically and bill you for the privilege.

That last row is the one I learned the hard way. Horizon retried my job twice after the first kill. All three attempts burned the same model call and none of them could have finished.

What should every model call record?

Input tokens, output tokens, the model name, the duration, and whether it succeeded. Without those five you are guessing, and Langfuse gives you all of them as a distributed trace across a multi-step agent run.

The reason to trace rather than log is that agent steps are separate model calls. One user request becomes intent classification, then tool selection, then response synthesis. A log line per step tells you each step happened. A trace tells you which step spent the time and which step spent the money.

Cost attribution is the payoff. Tag every trace with the feature that triggered it and you can answer questions about a feature's unit economics rather than about a bill. I built the same instinct into a client's retrieval system, and the measured result is in cutting LLM API costs 60%. Caching moved four times more money than model routing did, and I only know that because every call was traced first.

How do you know an agent got better rather than just different?

You run evals, and you accept that this is the least mature part of the stack. Three kinds are worth having, and I am describing what I recommend rather than a suite I can point you at.

Deterministic checks answer whether the output matched the schema and carried the required fields. A model-as-judge grades outputs against written criteria using a cheap model, which is imprecise but catches large regressions. Human spot-checks on a small sample of real traffic catch what both of the others miss, which is usually tone and context.

An eval suite that only asks "did it return something" is a smoke test wearing a lab coat. The useful question is whether the answer got better, and that needs a graded dataset you maintain by hand.

What breaks first when an agent meets real users?

Not the prompt. This is the order I now expect, with what each one looks like from outside.

Four of those six cost nothing but attention. The expensive ones are the fallback ladder and the eval dataset, and both are cheaper than the incident that teaches you to build them.

How does this fit a Laravel or Next.js codebase?

The same way any other unreliable dependency does. The model call is an HTTP request to somebody else's machine, and it deserves the treatment you already give those.

On this site the model work runs as queued jobs behind Horizon, which is the same queue that pushes cache purges to the frontend. I wrote that protocol up in the cache handshake, and the surrounding architecture, including what has no retry at all and why, in what yabasha.dev actually runs.

The constraints get sharper the more regulated the client. What survives contact with a bank's review is a narrower set than what survives a demo, and I wrote about that gap in what actually works in enterprise banking.

What does the Composable AI Stack actually give you?

Less than an earlier version of this post claimed. I read the repository row by row before republishing, and three of its eight claims were aspirational. Here is what is in the tree today.

The import path in the older table was wrong too. The generated template uses the @acme scope, so it is runWithGuardrails from @acme/ai, not from a @yabasha/cas subpath. The CLI on npm is @yabasha/cas; the packages it scaffolds are not.

The guardrail, tracing and eval core is real and worth taking. The agent loop, the retry layer and the fallback chains are a roadmap I published as if it had already shipped, which is exactly the habit this post argues against.

bun add -g @yabasha/cas
cas init my-agent-system

What would I check before calling an agent production-ready?

Start at the outside and work in. The prompt is the last thing on this list for a reason.

• Every layer that wraps a model call declares a timeout, and each one is larger than the layer inside it.
• The platform's own execution limit is written down next to your budget, read from the vendor's docs rather than remembered.
• Every model output passes a schema, and then passes a business rule the schema cannot express.
• The fallback ladder ends in code that cannot call a model.
• Retries fire on 429 and 5xx, and nowhere else.
• Every call records tokens, model, duration and outcome, and every trace carries the feature that caused it.
• A graded dataset exists, and something runs against it before you ship.

Demos fail on the prompt. Production fails on the timeout, the schema, and the missing rung at the bottom of the ladder.

Measured with

Numbers read from this repository and from production on 2026-09-21.

• Laravel 12 on PHP 8.4, Horizon on Redis, one VPS
• config/horizon.php: tries 3, timeout 60, memory 128
• OpenRouterService::HTTP_TIMEOUT_SECONDS 120; both model-calling jobs declare $timeout 150
• The incident: TimeoutExceededException on GeneratePostSeoJob, 2026-09-19 10:24:47, fixed 10:47:39
• Convex execution limits quoted from the vendor's limits page, fetched 2026-09-21
• @yabasha/cas 0.2.0 on the npm registry; composable-ai-stack public, last pushed 2026-05-12, cloned and read on 2026-09-21

Questions: @bayyash