Private File Upload Architecture for GDPR: Signed Download Retention in US/EU Apps

Short answer: For a beginner building a private file workflow, keep object storage private, let the application issue short-lived signed upload and download URLs, and make the deletion deadline a database state transition rather than a property hidden in a bucket.

That architecture gives a browser a narrow, temporary capability instead of a storage credential. The application remains responsible for identity, authorization, retention, and deletion. Object storage handles bytes. This split is especially useful when a document will later enter a Python RAG or agent pipeline: ingestion can be evaluated independently from upload mechanics and prompt cost.

The interesting failure is not usually a failed upload. It is a file that uploaded successfully, lost its database owner, and then survived past the deadline because nobody could prove who should delete it.

Start with the deletion contract, not the bucket

The contract is simple: every stored document has an owner, a region, a purpose, a current state, and a deletion deadline. A download is allowed only while the record says it is allowed. A deletion request must reach the object, extracted text, vector index, cache, thumbnail, and any other copy that the product created.

That is the contract.

The bucket is only one participant. This framing is useful for a Python team moving from notebook-to-prod because it keeps retrieval evaluation honest: a document cannot remain eligible for search merely because a background cleanup worker forgot one table.

For a concrete timeline, imagine an upload intent at 09:00, a 30-minute signed upload capability, and a deletion deadline at 09:00 the next day. The browser may finish at 09:04, while the parser starts at 09:06 and the user revokes access at 10:00. At 09:00 on the following day, the deletion worker must stop new downloads immediately, remove the object, enqueue index cleanup, and leave an audit record even if one downstream operation needs a retry. A lifecycle rule may eventually remove bytes, but it cannot explain which authorization decision allowed a retrieval at 08:59 or whether the vector index was also cleared. That evidence belongs in the application record and its event trail.

What breaks when a signed URL becomes the whole design?

The first failure mode is confusing possession with permission. A signed URL is a temporary capability; it does not know that a user changed tenant, that a file was revoked, or that the retention deadline passed. The application must make those decisions before issuing it.

The second is treating a browser callback as proof that the object is valid. An upload can be incomplete, oversized, mislabeled, or a valid container with an unsafe payload. Use an opaque server-generated object key, check the stored object's metadata and size, validate its type, and apply scanning rules appropriate to the product. OWASP recommends an extension allow-list, file-type validation rather than trust in a supplied Content-Type, size limits, restricted upload permissions, and malware scanning where needed.

The third is deleting only the original bytes. For a RAG or agent feature, extracted text and embeddings are copies with their own retention behavior. If the source disappears but its chunk remains retrievable, the system has not fulfilled the user's deletion expectation.

Don't hide that work behind a single “delete” button.

How can beginners protect private files with signed upload and download URLs in object storage?

Use two paths with different jobs. An authenticated application endpoint creates an upload intent. It checks the user and tenant, generates an opaque object key, records the expected size and media type, and returns a short-lived signed upload URL. The browser sends bytes directly to the private bucket. A completion request then verifies the object and moves the row from pending to ready.

Downloads repeat the authorization step. The application looks up the file by its database ID, confirms that the requester can read it and that the file is still within its retention window, then returns a short-lived signed download URL. The browser never gets a long-lived access key, and a copied URL should expire quickly.

Keep the user-facing filename in the database. It is useful for display, but it should not become the object key. Server-generated names reduce collisions and make path traversal less relevant. OWASP also recommends an extension allow-list, file-type validation rather than trust in a supplied Content-Type, size limits, restricted upload permissions, and malware scanning where the product needs it.

Signed URLs are capabilities, not a compliance program. A GDPR review still has to cover the purpose and legal basis for processing, processor terms, regional placement, retention, erasure requests, access records, and backups. A US/EU application should make the region a deliberate policy value on the file record, not an assumption buried in a storage client.

A Python test for the retention boundary

The simple design is to put an expiry hint in a storage object and trust a lifecycle rule to remove it. That can be useful as a backstop, but it is a poor source of truth for an application that must answer “why is this document still here?” The chosen design records expires_at, deleted_at, owner_id, region, and an explicit state in SQL. A scheduled worker finds due rows, deletes the object, and records the result.

The state transition should be idempotent. A worker may crash after deleting the object but before committing its row update; the next run must treat the missing object as a converged outcome, not recreate it. Conversely, a database row marked deleted should never cause a download URL to be issued.

Here is the small part I would put under an evaluation harness before connecting a real storage client. It tests the policy boundary, not a vendor API. I've found this boundary is more valuable than a happy-path upload test because it makes a revocation decision observable and cheap to run on every change.

from datetime import datetime, timezone


def can_issue_download(file_record: dict, now: datetime) -> bool:
    """Return true only for a ready, non-expired, non-deleted file."""
    return (
        file_record["state"] == "ready"
        and file_record["deleted_at"] is None
        and file_record["expires_at"] > now
    )


now = datetime.now(timezone.utc)
record = {
    "state": "ready",
    "deleted_at": None,
    "expires_at": now.replace(year=now.year + 1),
}

assert can_issue_download(record, now) is True
record["deleted_at"] = now
assert can_issue_download(record, now) is False

The real worker needs a lease or an equivalent database lock so two workers do not process the same row indefinitely. It also needs a retry policy with a bounded attempt count, an audit event, and an alert for records that remain due after the service-level deletion target. A 404 from object storage can mean the byte object is already gone; it is not automatically proof that the database update was applied. In a longer-running job, I would keep the original deadline, attempt number, worker lease, provider response class, and downstream acknowledgements together, because otherwise an operator cannot distinguish an already-completed delete from a retry that is still waiting on an index consumer, and the next automated action may issue a download URL from stale state.

That distinction matters for AI systems. If a deleted document remains in an embedding index, the storage deletion is incomplete from the user's perspective. The deletion job should enqueue removal from extracted text, vector records, search caches, and thumbnails, then record which downstream stages acknowledged the request. I keep those stages visible in the eval fixture because a retrieval answer from an erased source is a data-lifecycle failure, even if the bucket is empty.

Measure before copying this pattern: time from deadline to object deletion, time to index deletion, orphan count, repeated completion calls, expired-link rejection, and the percentage of upload intents that never become ready. Your mileage may vary by storage provider and region; a small test from the actual US and EU deployment locations resolves latency and cross-region behavior better than a generic benchmark.

Which boundaries make a private file workflow easier to operate?

The application is the control plane. It owns authentication, tenant checks, retention policy, and the audit trail. Storage is the data plane. It owns durable bytes, private access, and the provider's lifecycle machinery. A queue connects deletion and downstream cleanup without making a browser request wait for every consumer.

Do not let the browser choose an arbitrary bucket, prefix, or retention period. The server should derive the object key from an internal file ID and return only the fields the client needs. A signed upload should be limited by method, object key, expiration, and any available content constraints. A signed download should be generated only after the current authorization check.

Secrets need their own boundary too. Storage credentials belong in a secrets manager or deployment secret store, with least privilege and rotation. They do not belong in JavaScript, notebooks committed to a repository, or a prompt used by an agent. The OWASP Secrets Management Cheat Sheet is a good baseline for ownership, access, and rotation decisions.

There is a practical notebook-to-prod checkpoint here: run the same policy tests against a fake storage adapter, then run a small integration suite against the chosen provider. Test an interrupted upload, a duplicate completion call, a revoked user, a file at the deadline, and a worker retry. I would expect the revoked case to produce no signed URL and a 403 at the application boundary, while a previously issued URL should be short-lived rather than treated as a permanent entitlement.

What should you compare before choosing a storage boundary for US and EU users?

Compare capabilities that affect the deletion promise, not just upload throughput. Ask where data and replicas live, how lifecycle rules behave, whether versioning or immutable retention can preserve bytes after an application deletion request, how access events are exposed, and how credentials are scoped. Then check the client experience: direct browser upload, CORS configuration, resumable transfers, and the provider's behavior after a lost response.

The catch is that a common abstraction can hide features you eventually need. It is not suitable when the product requires public anonymous URLs, static-site hosting, object lock, automatic cross-region replication, or a provider-specific migration tool. Stick with a direct provider or a dedicated data platform when those are hard requirements. The portability gained from one interface is worth less than a missing deletion guarantee.

Cost deserves a place in the comparison, but it should not lead it. Include storage, requests, egress, scanning, indexing, and retry traffic in a testable estimate. A low transfer price does not fix an orphaned vector record or an audit trail that cannot explain an erasure.

A release gate for signed document retention

Before launch, make the retention contract executable. Every file should have an owner, region, state, creation time, and deletion deadline. Every download should pass a current authorization check. Every deletion should produce an auditable event and a retryable downstream job. Every RAG ingestion record should point back to the file ID and parser version.

Then exercise the ugly paths. Upload a disallowed type. Send a body larger than the declared limit. Repeat completion after a timeout. Delete an account while an indexing job is queued. Ask for a download after expiry. Check the bucket, database, search index, cache, and backup policy separately.

This is enough architecture for a beginner to ship browser-direct transfers without turning the Python service into a byte proxy. It also leaves room to replace the storage implementation later, provided the application keeps ownership and deletion state explicit.

References: Further reading

• https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html
• https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html