Using Pub/Sub to scale an expensive video pipeline

A player records a football drill on their phone and uploads it for feedback. That’s the experience we’re building with AIM Football. ^[1] Behind it, multiple videos need processing, and there’s substantial post-processing to finish before the result is useful.

We use Pub/Sub queues to help manage that work. They let us separate incoming videos from the expensive processing and control how much work starts at once. I’ve simplified the examples here and left out the deployment details.

I find the interesting decisions start after you’ve chosen a queue. How much work should a worker take? What happens when the next stage is already busy? How much of a video should you have to process again after a failure?

Let the queue hold the burst

An upload arriving doesn’t have to mean another expensive process starts immediately. In this pattern, the video stays in storage. A small message identifies the job and points to the input, so a worker can pick it up when there’s capacity.

Accepting the job needs care too. If the app records a job but publishing its message fails, there must be a way to schedule that work again. Otherwise, someone can have an upload marked as accepted with nothing actually waiting to process it.

For queue-like work, workers share a subscription. Separate subscriptions receive their own deliveries of the same published message. That distinction matters: adding a subscription for every worker can multiply the work you meant to distribute. ^[2]

Subscriber flow control limits how many messages a worker takes on before it stops asking for more.^[3] For video, I’d set that limit from the processing footprint. A tiny message can point to a large video, so the size of the message says very little about the memory its job will need.

This gives a burst somewhere to wait. If work keeps arriving faster than it finishes, the queue will keep growing.^[3] At that point, you need more capacity, less work per video, or a limit on what you accept. The waiting time belongs in that decision alongside the bill.

Post-processing needs room to finish

The AI stage gets a lot of attention because it’s the part you can demonstrate. In a video product, its output can still need substantial work before anyone should see it. Depending on the product, that might mean combining intermediate results or preparing feedback for the app.

Imagine the AI stage finishing videos faster than post-processing can handle them. Adding more capacity to that first stage creates a bigger pile of unfinished work. The player still waits for the same last step.

I’d give stages with different resource needs their own concurrency limits. Heavy post-processing shouldn’t inherit a limit chosen for another kind of work. A queue between stages lets the next stage accept work at its own pace; it also makes the buildup visible.

Splitting work has a cost. You have more handoffs to track, and a failure between stages needs a recovery path. I wouldn’t separate every small operation. I’d separate work when it needs different capacity or when restarting it independently would avoid repeating an expensive step.

Be careful about when a message is finished

Acknowledging a message tells the messaging service that the subscriber has dealt with it. If you acknowledge first and the worker then crashes, normal redelivery won’t recover that work. ^[4]

I think of the acknowledgement as a decision about who is responsible next. When the subscriber does the processing, it should save the required output and progress before acknowledging. If it only hands work to another job system, that handoff must be durable and recoverable before the message is acknowledged. Starting a background process isn’t enough.

Long video jobs make the timing awkward. With pull subscribers, lease management can extend the acknowledgement deadline while work continues. Those extensions have configured limits, and they don’t remove the need to handle duplicate deliveries. ^[5]A job that routinely outlasts those limits needs a different work boundary or a durable handoff.

The next stage needs the same care. Saving a result and scheduling the follow-up work must survive a crash between those actions. A completed AI step with no post-processing scheduled still leaves the person waiting.

A retry still costs money

Pub/Sub’s default delivery is at least once, so the same message can arrive again. ^[2]With expensive video work, treating every delivery as a fresh job can mean paying to process the same input again.

A stable job identity helps a worker recognise completed work. The result needs to be idempotent: repeating an operation must not create a second, conflicting outcome. That protects the result, but avoiding two costly attempts running at once also needs coordination over who owns the active job.

I’d keep valid intermediate outputs where they make recovery cheaper. If a later stage fails, it should be possible to retry that stage without automatically rerunning everything before it. Those saved outputs need to match the input and processing version; an old result isn’t always reusable.

Some failures are worth another attempt. A temporarily unavailable dependency may recover, and backoff leaves time between retries. Pub/Sub supports an exponential backoff policy for failed deliveries.^[6] A corrupt or unsupported video needs a different response. Repeating the same processing won’t repair the input.

I’d put a deliberate limit on application retries and make persistent failures visible for review. Dead-letter handling can help isolate repeated delivery failures, though Pub/Sub’s delivery-attempt threshold is approximate.^[7] Moving a message there should lead to an alert and a decision about the job, not leave its status saying “processing” forever.

Measure the wait all the way to the result

Queue depth helps, but it doesn’t tell the whole story. A handful of long videos can represent more work than a much larger batch of short ones. I want to know how long the oldest unfinished job has been waiting, where time is going, and whether retries are consuming the capacity meant for new work.

For cost, I’d look at spend per successfully completed video, including post-processing and failed attempts. For speed, I’d measure from the accepted upload to the result being available. Comparing similar video lengths makes those numbers more useful than one average across everything.

A player should be able to leave the app and come back to the same job status. If processing has failed, the app should say whether it will try again or needs a new recording. That status has to come from a lasting record of the job, since an empty queue doesn’t tell you whether the final result was saved.

Before increasing concurrency, I’d test a mixed batch of videos and interrupt a worker partway through. I’d want to see unfinished work recover while other videos keep moving, without competing attempts publishing conflicting results. Then I could judge whether starting more work at once actually helps people get their feedback sooner.