Dev.to Security 🔐 Cybersecurity 👁 0 📖 5 min read

Uploaded User Documents Explained — Private File Storage, Access, and Throughput

Support teams collect screen recordings, call audio, and scanned documents. The simplest design for a startup app is to keep those bytes out of the application process: upload directly to private object storage, save onl

Support teams collect screen recordings, call audio, and scanned documents. The simplest design for a startup app is to keep those bytes out of the application process: upload directly to private object storage, save only metadata in the database, and mint a short-lived signed URL when an authorized agent needs a file. That split improves large-file throughput because the app authorizes a transfer instead of becoming the transfer.

Short answer: use a private bucket, direct multipart uploads, and signed download URLs; add a queue and lifecycle policy before adding a custom file service.

The before-and-after mental model

The fragile version is browser -> app -> storage. Every byte consumes an application connection, a worker, and often a second copy in memory. A busy support shift can turn one 2 GB recording into a pile of slow requests and expensive retries.

The healthier path is browser -> storage for upload and agent -> signed URL -> storage for download. The app still owns authorization. It simply hands the client a capability that expires. I treat the object key as an opaque identifier such as tenant/incident/uuid, never as a user-supplied filename.

That boundary also makes observability clearer. Record upload intent, object size, checksum, tenant, and final status in the app; record transfer latency and bytes at the storage edge. A missing object is then a state mismatch to investigate, not a mysterious timeout inside an API worker.

How should a startup app keep user-uploaded documents private with signed URLs?

Start with an authorization check against the database, then issue the narrowest URL possible. A 15-minute read URL for one object is a useful default; shorter is better for sensitive material, while a long-running export may need a separate workflow. Never put a permanent public URL in a ticket, email, or browser history.

Here is the shape of a tiny TypeScript handler. The storage client is intentionally abstract, so the same contract can sit over any S3-compatible service or a self-hosted gateway.

type DocumentRecord = {
  tenantId: string;
  objectKey: string;
  contentType: string;
  bytes: number;
};

async function createDownloadLink(
  actorId: string,
  documentId: string,
): Promise<string> {
  const doc = await db.documents.findVisibleToActor(actorId, documentId);
  if (!doc) throw new Error("document_not_found");

  return storage.signGetUrl({
    key: doc.objectKey,
    expiresInSeconds: 15 * 60,
    responseHeaders: {
      "Content-Disposition": `attachment; filename="${documentId}.bin"`,
      "Content-Type": doc.contentType,
    },
  });
}

The response headers matter. Content-Disposition: attachment tells a browser to download rather than render inline, while the server still controls the filename. Validate the stored content type and size before signing; do not copy either value blindly from an upload form. For previews, issue a different endpoint and a different policy.

Uploads need the same discipline. Create an intent row first, let the client send parts directly, then mark the row complete only after a storage-side existence and size check. A retry must reuse the intent and object key, not create a second document. Your mileage may vary on part size and concurrency; measure with the largest recording your support workflow actually receives.

What fails first when large-file throughput becomes the constraint?

Most incidents are boring state problems. A client loses Wi-Fi after part 7, a browser retries with a new key, or a worker marks success before the final object is visible to the reader. Model uploads as a small state machine: created, uploading, complete, aborted, and expired. Include an idempotency token and a last-seen timestamp. In practice, that means the support UI can safely refresh without starting over: it asks for the intent by token, discovers which parts already exist, and resumes only the missing work. A nightly job can mark stale intents expired and abort their multipart sessions. I keep this cleanup separate from the request path because a cleanup timeout should never block an agent who is trying to attach evidence to a ticket. That one separation has saved more debugging time than another layer of abstraction would have.

Then watch the signals that explain a slow queue: p50 and p95 upload duration by size band, aborted multipart sessions, signed-link issuance failures, bytes per tenant, and the age of incomplete intents. Alert on a rising error ratio or a growing incomplete-age percentile, not on a single slow 2 GB transfer. I prefer a log line with documentId, tenantId, partNumber, and requestId; it turns a support ticket into a traceable lookup.

There is a security trap here. A signed URL is a bearer capability. Redact it from logs, keep it out of analytics query strings, and avoid forwarding it to third-party image or preview services. If a link leaks, the only practical containment is a short expiry and the ability to revoke the underlying object or authorization record.

The trade-offs that should change your choice

Direct storage is not suitable when every byte must pass through a malware scanner or a regional policy gateway before persistence. In that case, use a quarantine bucket and an asynchronous promotion step, accepting extra latency and storage I/O. Stick with an application proxy when you need synchronous content transformation, but budget enough workers and egress capacity for the worst upload window.

Decision Direct private object storage Application proxy
Large-file throughput Usually higher; app handles metadata Limited by app connections and workers
Access control Database check plus expiring capability Centralized in the app request
Inspection before save Requires quarantine and promotion Easy to run inline
Operational cost More states to reconcile More compute and egress pressure

The catch is operational ownership. You must test expiry, partial uploads, deletion, and tenant isolation. A storage service with an S3-compatible API can reduce migration work, but compatibility is not identical behavior: verify multipart limits, checksum semantics, range reads, and lifecycle rules in a staging bucket. No single interface removes those decisions.

Choose the direct path when the primary axis is large-file throughput and your app can tolerate asynchronous scanning. Keep the bucket private, expose only signed capabilities, and make the database the source of authorization. Choose a proxy or quarantine-first flow when policy requires inspection before any durable object is available.

Keep it private.

I would ship the smallest version with one upload-intent table, one private bucket, one signed-download endpoint, and dashboards for the five signals above. Add resumable cleanup and retention automation next. The design stays understandable under pressure, which is exactly what a support system needs.

References

📰 Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.