File Upload and Processing System Design

The problem

File upload looks simple while the file is small and the work after it is light.

When the system has to deal with:

• large video
• transcoding
• OCR
• image analysis
• report generation

The implementation that “receives on the server and replies when it finishes” stops being viable very quickly.

Mental model

There are two different flows here:

1. upload the file
2. process the file

Mixing both inside the same request is usually the root of the problem.

Healthy designs usually separate:

• upload to storage
• confirmation that the file arrived
• work queue
• processing worker
• status updates

In one simple sentence:

The user does not need to wait for processing to finish for the system to keep working.

Breaking the problem down

Take the large file out of the main backend

The browser can ask the backend for a temporary signed URL and upload directly to object storage.

In simple language:

• the backend authorizes
• the browser sends directly to storage
• the main server does not become a tunnel for a huge file

This also reduces the risk of burning CPU, memory, and bandwidth in a service that should focus on authorization and state.

Decouple processing from the request

When the file arrives, the heavy work should not depend on the original request still being alive.

The normal move is to publish an event or job to:

• a queue
• a worker
• a processing pipeline

Give the user a way to follow status

The client cannot stay stuck waiting on the original request.

So the system needs to expose something like:

• PENDING
• PROCESSING
• DONE
• FAILED

That status can be observed through:

• polling
• webhook
• real-time channel

What matters is honest visibility into progress, not an infinite spinner with no explanation.

Think about large files and poor connections

Very large files often require multipart upload.

That helps because it:

• allows resume after failure
• avoids starting from zero again
• behaves better on unstable networks

Security does not come at the end

Not every uploaded file should be processed.

Before the heavy work starts, systems usually add:

• type validation
• maximum size
• signature checking
• security scanning

Depending on the scenario, a quarantine stage may also exist before the file is released for internal use or download.

Simple example

A mature answer could sound like this:

“I would separate upload from processing. The client asks the backend for a signed URL and uploads directly to object storage. When the upload finishes, storage or the backend publishes an event to a queue. Workers process the file asynchronously and update a status record. The frontend follows that status through polling. If the file is very large, I use multipart upload so the transfer can resume. I also validate type and size before starting processing.”

That answer covers:

• the file path
• request decoupling
• queueing
• status return
• basic security

Common mistakes

• Making the full upload go through the backend without a reason.
• Processing inside the request.
• Forgetting resumable uploads for large files.
• Ignoring file validation and security.
• Returning results without an expiration or reprocessing policy.

How a senior thinks about it

People with more experience usually ask this early:

Does the user need to wait for processing, or only know that the system received the file and will keep working?

When the answer is the second one, the design becomes much clearer.

Because a big part of the complexity stops being trapped inside the initial request.

What the interviewer wants to see

In this scenario, the interviewer is checking whether you:

• separate upload from processing
• know why object storage shows up early
• use queue and worker with clear purpose
• think about user-facing status
• remember validation and security

A mature upload system does not keep the user staring at an endless bar. It splits the flow into stages the system and the user can both follow.