Five items work in development. Fifty items reach production, the request runs for a minute, and the platform closes the connection without returning the completed work.
In one translation feature, a 50-item job made model calls that took roughly 3 seconds each. Depending on batching, the full request ran for 60–100 seconds. It crossed the deployed function's duration limit even though every individual call succeeded.
Provider limits change by plan and runtime. The durable lesson is not a particular timeout number. It is that one HTTP request should not own a variable amount of external work.
Turn the request into a job
The robust API accepts the work, records it, and returns quickly:
POST /api/translations/jobs
202 Accepted
{ "jobId": "job_123", "status": "queued" }
A worker processes bounded units. The UI polls or subscribes to job progress:
{
"status": "running",
"total": 50,
"completed": 31,
"failed": 1
}
Each unit needs an idempotency key, attempt count, last error, and durable output. A retry should repeat only incomplete work, not bill for all 50 calls again. Cap concurrency to the model provider's rate and token limits; “parallel” is not permission to create an unbounded Promise.all().
When client-driven batches are enough
For a small internal tool, the browser can request one batch at a time and display progress. That is simpler than deploying a queue, and a failed request loses one batch rather than the whole run.
Be honest about the tradeoff: closing the tab pauses the job, network loss interrupts it, and two tabs can start duplicates unless the server still owns idempotency and progress. Client orchestration is acceptable when those semantics are acceptable. It is not a replacement for durable background work.
Define the unit of recovery
Do not store only completed: 31. Record which 31 items finished and the input version each result belongs to. Otherwise a retry after the source changes can combine outputs from different dictionaries or prompts.
Useful job states are queued, running, completed, completed_with_errors, failed, and cancelled. Make cancellation cooperative: stop claiming new units, but retain the results already paid for.
Test the production-sized case
Seed 50 items, inject a failure after item 17, retry, and verify that the first 16 are not repeated. Close the client halfway through and confirm whether the documented behavior is pause or continuation. Then run against the deployed runtime, not only the local server.
The original timeout was not an unusually slow model call. It was a workload whose duration grew with input size trapped inside a request whose lifetime did not.