AWS for Newbies — Episode 2

AWS S3, Deduplication & the Presigned URL Production Flow

An Uncle–Nephew Chronicle

A ~1 hour read. Continues directly from Episode 1 (IAM + EC2). Every term is defined the moment it shows up.

👦 Nephew: Uncle, I've been thinking about what you asked last time — where does the uploaded file actually go if my server is disposable. I looked it up. It's S3, right?

👨‍🦳 Uncle: Right. And today we're not doing the toy version. We're doing the version a real production app needs: files organized by type, no duplicate files silently eating your storage bill, file size actually enforced instead of "hoped for," and a presigned URL flow with a real expiry policy — not the 2-minute throwaway example from the basics. By the end of this, you'll have the full Node.js code for it, not just diagrams.

👦 Nephew: Let's go. Start from the beginning — what is S3, really?

Part 1 — S3: Storage That Isn't "A Folder on a Server"

👨‍🦳 Uncle: S3 stands for Simple Storage Service. It is object storage — a place to store files (images, PDFs, videos, backups, logs — anything as raw bytes) completely separate from any server. Unlike EC2's disk (EBS), which is tied to one specific virtual machine, S3 exists independently. Ten different servers, or zero servers, can all read and write to the same S3 storage. That's exactly the property we needed after last episode's problem — files that survive even if the server that received them is gone.

In S3's vocabulary:

• A bucket is the top-level container — think of it as one storage account/warehouse. Bucket names must be globally unique across all of AWS, not just your account.
• An object is a single stored file, along with its metadata (size, content type, upload date, permissions).
• A key is the object's full path-like name inside the bucket, e.g. images/profile-abc123.jpg.

👦 Nephew: So a "key" is basically the file path?

👨‍🦳 Uncle: Functionally, yes — but here's a fact that surprises almost everyone: S3 doesn't actually have real folders. It's a flat structure of objects, each with a long key string. When you see images/profile.jpg displayed with a little folder icon in the AWS console, that's the console being helpful — it's just splitting the key string on the / character and drawing a folder illustration for you. Underneath, there is no such thing as an "images folder" object. It's purely a naming convention, called a prefix.

👦 Nephew: Wait, that actually matters for what I want to do — separating PDFs, text files, and images into their own "folders."

👨‍🦳 Uncle: It matters a lot, and it's good news — it means organizing by file type costs you nothing extra. You just design your key naming convention deliberately, and S3 will happily group them for you visually and let you list/filter by prefix efficiently.

Part 2 — Designing the Key Structure (Folder-Wise Organization)

👨‍🦳 Uncle: Let's design it properly instead of winging it. A clean, type-separated key convention looks like this:

documents/{hash}.pdf
images/{hash}.jpg
text-files/{hash}.txt

Or, if you also want per-user isolation (very common in real apps):

users/{userId}/images/{hash}.jpg
users/{userId}/documents/{hash}.pdf

👦 Nephew: Why the hash instead of the original filename, like resume.pdf?

👨‍🦳 Uncle: Three solid reasons. First — two different users might both upload a file called resume.pdf; using the raw filename risks silent overwrites unless you're careful. Second — filenames can contain characters that misbehave in URLs. Third, and most important for today's topic: the hash is how we detect duplicate files. Which brings us to the real meat of today's lesson.

Part 3 — Deduplication: Don't Store the Same File Twice

👨‍🦳 Uncle: Imagine 500 users all upload the exact same company logo, or the same PDF brochure gets re-uploaded across 50 different form submissions. Without deduplication, you're paying S3 storage costs for 500 identical copies of the same bytes. We fix this with content hashing.

👦 Nephew: Meaning?

👨‍🦳 Uncle: SHA-256 is a cryptographic hash function — an algorithm that takes any input (in our case, a file's raw bytes) and produces a fixed-length, 64-character string (called a hash or digest) that is essentially a unique fingerprint of that exact content. Two important properties matter to us:

1. The same file content always produces the exact same hash, no matter who uploads it or what they named it.
2. Even a single-bit difference in the file produces a completely different hash. So it's not "similar files get similar hashes" — it's "identical content, identical hash; anything else, unrelated-looking hash."

The chance of two genuinely different files accidentally producing the same SHA-256 hash is astronomically small — small enough that the entire software industry (including Git itself) relies on this property daily.

👦 Nephew: So the plan is: compute the hash, and if we've seen that hash before, don't store the file again?

👨‍🦳 Uncle: Exactly. Let's build it.

3.1 — Computing a SHA-256 hash in Node.js

Node has hashing built into its standard library — no extra package needed. Here's the core building block, written to handle large files efficiently by streaming the file instead of loading the whole thing into memory at once:

const crypto = require("crypto");
const fs = require("fs");

function hashFileStream(filePath) {
  return new Promise((resolve, reject) => {
    const hash = crypto.createHash("sha256");
    const stream = fs.createReadStream(filePath);

    stream.on("data", (chunk) => hash.update(chunk));
    stream.on("end", () => resolve(hash.digest("hex")));
    stream.on("error", reject);
  });
}

// Usage:
// const fileHash = await hashFileStream("/tmp/uploaded-file.pdf");
// e.g. "3f786850e387550fdab836ed7e6dc881de23001b"

👦 Nephew: Why stream it instead of just crypto.createHash('sha256').update(buffer).digest('hex') on the whole file at once?

👨‍🦳 Uncle: Because if someone uploads a 200 MB video file, loading the entire thing into memory just to hash it can spike your server's RAM and slow everything else down — especially if ten uploads happen at once. Streaming reads the file in small chunks, feeds each chunk into the hash calculation, and never holds the whole file in memory. Small files barely notice the difference; large files, it's the difference between a smooth server and a crashed one.

3.2 — Checking for duplicates before storing

Now, the hash alone is only useful if you remember which hashes you've already stored. That's a job for your database, not S3 itself.

CREATE TABLE files (
  id            SERIAL PRIMARY KEY,
  sha256_hash   VARCHAR(64) NOT NULL UNIQUE,
  s3_key        TEXT NOT NULL,
  file_type     VARCHAR(20) NOT NULL,   -- 'pdf' | 'image' | 'text'
  size_bytes    BIGINT NOT NULL,
  uploaded_by   INTEGER REFERENCES users(id),
  created_at    TIMESTAMP DEFAULT now()
);

Notice sha256_hash has a UNIQUE constraint. That single line is doing a lot of work — even if two upload requests race each other at the exact same millisecond, the database itself will reject the second insert of the same hash, so you can't accidentally create a duplicate even under concurrent load.

The check-then-act flow in code:

async function findOrRegisterFile(fileHash, fileType, sizeBytes, extension, userId) {
  // 1. Have we already stored this exact content?
  const existing = await db.query(
    "SELECT s3_key FROM files WHERE sha256_hash = $1",
    [fileHash]
  );

  if (existing.rows.length > 0) {
    // Duplicate! Don't upload again — just reuse the existing object.
    return { isDuplicate: true, s3Key: existing.rows[0].s3_key };
  }

  // 2. New file — decide its key, based on type, using the hash itself
  const folder = { pdf: "documents", image: "images", text: "text-files" }[fileType];
  const s3Key = `${folder}/${fileHash}.${extension}`;

  await db.query(
    `INSERT INTO files (sha256_hash, s3_key, file_type, size_bytes, uploaded_by)
     VALUES ($1, $2, $3, $4, $5)`,
    [fileHash, s3Key, fileType, sizeBytes, userId]
  );

  return { isDuplicate: false, s3Key };
}

👦 Nephew: So if it's a duplicate, we just... don't touch S3 at all? We just point the new "upload" record at the old object?

👨‍🦳 Uncle: Exactly right. The user experience looks identical — "your file uploaded successfully" — but behind the scenes, you saved storage cost, saved upload bandwidth, and saved processing time (if you were going to compress/resize it), all because the bytes were already sitting in S3 from someone else's earlier upload.

Part 4 — Enforcing File Size (At Every Layer, Not Just One)

👨‍🦳 Uncle: Here's a mistake I see constantly: a developer checks file size once, on the frontend, in JavaScript, and calls it done. That check is trivially bypassed — anyone can call your API directly with curl or Postman, skipping your frontend entirely. Real file-size enforcement is layered, the same "reject bad traffic as early as possible" principle from our security-groups discussion.

Layer 1 — Client-side (UX only, not security):
Reject obviously oversized files before even starting an upload, so the user gets instant feedback instead of waiting for a slow upload to fail.

Layer 2 — Backend validation, before generating any upload permission:

const MAX_SIZE_BYTES = 5 * 1024 * 1024; // 5 MB

function validateFileSize(sizeBytes) {
  if (sizeBytes > MAX_SIZE_BYTES) {
    const err = new Error("File exceeds the 5MB limit");
    err.statusCode = 413; // "Payload Too Large" — the correct HTTP status for this
    throw err;
  }
}

Layer 3 — Enforced by S3 itself, at the moment of upload, via the presigned request's conditions. This is the layer most beginners don't know exists — and it's the one that actually matters for direct-to-S3 uploads, because your backend never sees the file bytes in that flow, so layer 2 alone can be lied to. We'll wire this up properly in Part 6.

Layer 4 — Load balancer / API Gateway request size limits, as a blunt outer boundary against absurdly oversized requests hitting your infrastructure at all.

👦 Nephew: So the frontend check is basically just "be nice to the user," and the real enforcement happens server-side and inside the S3 request itself.

👨‍🦳 Uncle: Correctly understood.

Part 5 — Permissions: Locking S3 Down Properly

👨‍🦳 Uncle: Now let's get the access model right, because this is where careless setups leak private user files to the entire internet — a mistake that's made the news more than once.

5.1 — Block Public Access (keep it ON)

When you create the bucket, AWS shows a setting called "Block all public access." Leave it enabled. With this on, no object in the bucket can be made public by accident — not through a misconfigured bucket policy, not through an ACL mistake, nothing. Any such attempt is silently denied. Public exposure should always be an intentional, narrow exception (which we handle via presigned URLs, or a CDN in front — future episode), never the default state of the bucket.

5.2 — IAM Policies: deciding what your backend/app can do

Access to S3 is controlled through IAM policies — JSON documents describing which actions are allowed on which resources. The key actions you'll use:

A properly scoped policy for your backend's role looks like this — notice it's restricted to one specific bucket, not "all S3 buckets everywhere":

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": "arn:aws:s3:::my-app-uploads/*"
    }
  ]
}

That Resource line, with the /* at the end, means "any object key inside the my-app-uploads bucket" — not the bucket's own settings, not other buckets. This is the Principle of Least Privilege again, applied to a service instead of a person.

5.3 — Attach this via an IAM Role, not access keys

If your backend runs on EC2 (from Episode 1) or Lambda, attach this permission through an IAM Role, exactly like we discussed last time — never paste an Access Key and Secret Key into your .env file for this. The role gives your running server temporary, auto-rotating credentials behind the scenes, with zero secrets to leak.

Part 6 — The Node.js Setup

👨‍🦳 Uncle: Let's install what we need. AWS's modern JavaScript SDK is modular — you install just the pieces you need, not one giant package.

npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

Set up the client once, and reuse it everywhere:

// s3Client.js
const { S3Client } = require("@aws-sdk/client-s3");

const s3 = new S3Client({
  region: "ap-south-1", // keep this the same region as your EC2/RDS — lower latency, lower cost
});

module.exports = s3;

Notice — no access key, no secret key in this code. If this runs on an EC2 instance with the IAM role from Part 5 attached, the SDK automatically discovers and uses those temporary credentials. This is the payoff of setting up roles properly back in Episode 1.

Part 7 — The Full End-to-End Upload Flow

👨‍🦳 Uncle: Now let's assemble everything — hashing, deduplication, size limits, folder-wise keys, and permissions — into one coherent request flow. This is the production pattern, not the toy version.

1. Client picks a file → computes size (and optionally a hash) locally
2. Client calls  POST /api/uploads/request-url  { fileName, fileType, sizeBytes, sha256Hash }
3. Backend:
     a. Validates the user is authenticated & rate-limit not exceeded
     b. Validates sizeBytes <= 5MB          (Layer 2 size check)
     c. Checks sha256Hash against the database
          → if it exists already: respond immediately, "already uploaded",
            return the existing s3Key. No S3 call needed at all.
          → if new: build the S3 key using the folder-by-type convention
            and the hash, e.g.  images/9f86d0...jpg
     d. Generates a PRESIGNED URL for that exact key, with:
          - a Content-Length-Range condition (enforces size at the S3 level)
          - a Content-Type condition (enforces file type at the S3 level)
          - an expiry time appropriate to the use case (see Part 8)
     e. Registers a row in the database (status: "pending")
4. Client uploads the raw file bytes DIRECTLY to S3, using that presigned URL
     — the backend server never touches the file bytes.
5. Client notifies backend: "upload finished" (with the s3Key)
6. Backend calls HeadObject on that S3 key, to CONFIRM the object genuinely
   exists and matches the expected size — this defeats a client that lied
   about having uploaded something.
7. Backend marks the database row "confirmed", and the file is now live.

👦 Nephew: Why does the backend need to double-check in step 6? Didn't we already validate everything?

👨‍🦳 Uncle: Because a presigned URL, once issued, is a bit like handing someone a signed blank cheque with a spending limit written on it — you've limited what they can do with it, but you still want proof of what actually happened before you update your own records. Never trust the client's word that an upload succeeded — verify against S3 directly.

7.1 — Code: requesting the upload URL

const { PutObjectCommand } = require("@aws-sdk/client-s3");
const { getSignedUrl } = require("@aws-sdk/s3-request-presigner");
const s3 = require("./s3Client");

const MAX_SIZE_BYTES = 5 * 1024 * 1024; // 5MB
const ALLOWED_TYPES = {
  pdf: { contentType: "application/pdf", folder: "documents" },
  image: { contentType: "image/jpeg", folder: "images" }, // extend for png etc.
  text: { contentType: "text/plain", folder: "text-files" },
};

async function requestUploadUrl({ fileType, sizeBytes, sha256Hash, extension, userId }) {
  if (sizeBytes > MAX_SIZE_BYTES) {
    const err = new Error("File exceeds 5MB limit");
    err.statusCode = 413;
    throw err;
  }

  const typeConfig = ALLOWED_TYPES[fileType];
  if (!typeConfig) {
    const err = new Error("Unsupported file type");
    err.statusCode = 400;
    throw err;
  }

  // Dedup check
  const existing = await db.query(
    "SELECT s3_key FROM files WHERE sha256_hash = $1", [sha256Hash]
  );
  if (existing.rows.length > 0) {
    return { duplicate: true, s3Key: existing.rows[0].s3_key };
  }

  const s3Key = `${typeConfig.folder}/${sha256Hash}.${extension}`;

  const command = new PutObjectCommand({
    Bucket: "my-app-uploads",
    Key: s3Key,
    ContentType: typeConfig.contentType,
    ContentLength: sizeBytes, // enforced as part of the signature
  });

  // Short expiry — this is a normal user-initiated upload,
  // it should happen within a couple of minutes.
  const uploadUrl = await getSignedUrl(s3, command, { expiresIn: 120 });

  await db.query(
    `INSERT INTO files (sha256_hash, s3_key, file_type, size_bytes, uploaded_by, status)
     VALUES ($1, $2, $3, $4, $5, 'pending')`,
    [sha256Hash, s3Key, fileType, sizeBytes, userId]
  );

  return { duplicate: false, uploadUrl, s3Key };
}

7.2 — Code: confirming after upload

const { HeadObjectCommand } = require("@aws-sdk/client-s3");

async function confirmUpload(s3Key, expectedSizeBytes) {
  const head = await s3.send(
    new HeadObjectCommand({ Bucket: "my-app-uploads", Key: s3Key })
  );

  if (head.ContentLength !== expectedSizeBytes) {
    throw new Error("Uploaded file size mismatch — possible tampering");
  }

  await db.query(
    "UPDATE files SET status = 'confirmed' WHERE s3_key = $1", [s3Key]
  );

  return { confirmed: true };
}

👦 Nephew: This is the cleanest way I've seen this explained. Okay — now the thing I actually asked about: the 5-hour expiry.

Part 8 — Presigned URLs: Choosing the Right Expiry for Production

👨‍🦳 Uncle: Good, let's slow down here, because "just set expiresIn to a big number" is the wrong way to think about it. The expiry time should match why the URL exists.

8.1 — Why expiry length varies by use case

👦 Nephew: So a 5-hour expiry isn't inherently risky — it depends on what it's for.

👨‍🦳 Uncle: Correct — but it does widen the window during which that specific URL, if leaked, remains usable. So when you deliberately choose a long expiry, you compensate with tighter guardrails elsewhere, which we've already built:

• Content-Length-Range / Content-Type conditions baked into the signed request, so even a leaked URL can't be abused to upload an oversized file or a disguised file type.
• One object key per presigned URL — the signature is tied to one specific S3 key, not a whole folder, so a leaked URL can't be used to overwrite arbitrary other files.
• Server-side confirmation (HeadObject), so even if someone uploads something using a leaked URL, your system won't treat it as trusted, confirmed data without matching your expected metadata.
• Short-lived by default, long only when justified — don't reach for 5 hours out of laziness; reach for it because the use case (e.g. a large video upload) genuinely needs that window.

8.2 — Code: generating a 5-hour presigned URL for a large-file production case

async function requestLargeUploadUrl({ fileType, sizeBytes, sha256Hash, extension, userId }) {
  const MAX_LARGE_FILE_BYTES = 500 * 1024 * 1024; // 500MB, for e.g. video

  if (sizeBytes > MAX_LARGE_FILE_BYTES) {
    const err = new Error("File exceeds the maximum allowed size");
    err.statusCode = 413;
    throw err;
  }

  const s3Key = `videos/${userId}/${sha256Hash}.${extension}`;

  const command = new PutObjectCommand({
    Bucket: "my-app-uploads",
    Key: s3Key,
    ContentType: "video/mp4",
    ContentLength: sizeBytes,
  });

  // 5 hours = 5 * 60 * 60 seconds
  const uploadUrl = await getSignedUrl(s3, command, { expiresIn: 5 * 60 * 60 });

  await db.query(
    `INSERT INTO files (sha256_hash, s3_key, file_type, size_bytes, uploaded_by, status, expires_at)
     VALUES ($1, $2, $3, $4, $5, 'pending', NOW() + INTERVAL '5 hours')`,
    [sha256Hash, s3Key, fileType, sizeBytes, userId]
  );

  return { uploadUrl, s3Key, expiresInSeconds: 5 * 60 * 60 };
}

👦 Nephew: What's that expires_at column doing in the database — isn't the S3 URL's own expiry enough?

👨‍🦳 Uncle: Sharp question. The S3-side expiry stops the URL from working after 5 hours — but it doesn't clean up your own database. If the upload never happens, you'd otherwise have a pending row sitting forever, pointing at an object that will never exist. Tracking expires_at yourself lets a background job periodically clean up abandoned pending uploads — good hygiene for a production system.

Part 9 — Wiring In the Rest of the Protection (Recap From Last Time, Now Complete)

👨‍🦳 Uncle: Let's connect this to the earlier security layering, so you see the whole chain end to end for a production deployment:

Internet
   ↓
AWS WAF               → blocks malicious/abnormal traffic patterns, rate-limits by IP
   ↓
Load Balancer          → max request size, connection limits, timeouts
   ↓
Application rate limit → e.g. max 20 upload-requests/min per user
   ↓
Backend validates size, type, dedup hash → issues presigned URL (scoped, size-capped, time-capped)
   ↓
Client uploads DIRECTLY to S3            → backend never touches the raw bytes
   ↓
Backend confirms via HeadObject           → verifies before trusting the upload

Nothing in this chain relies on a single point of trust. Even if one layer is bypassed, the next one still holds.

Part 10 — A Few More Production Essentials (Quick Hits)

👨‍🦳 Uncle: A handful of settings that separate a "working demo" from a "production-grade" S3 setup — worth knowing even if we don't deep-dive each one today:

• Server-side encryption (SSE-S3 or SSE-KMS) — S3 encrypts objects at rest automatically. Enable it at the bucket level so you don't have to remember it per-upload.
• Versioning — keeps prior versions of an object if it's ever overwritten, protecting against accidental deletes or bad overwrites.
• CORS configuration — since the browser is uploading directly to S3 (a different origin than your app's domain), S3's bucket needs a CORS (Cross-Origin Resource Sharing) policy explicitly allowing your frontend's domain to make that PUT request. Without this, browsers block the upload even with a valid presigned URL.
• Lifecycle rules — automatically move old, rarely-accessed objects to cheaper storage tiers (like S3 Glacier) after a set number of days, or delete genuinely temporary files automatically.
• Deny insecure transport — a bucket policy statement that rejects any request not made over HTTPS, so nothing is ever transmitted in plaintext.

Part 11 — Recap

👨‍🦳 Uncle: Let's replay it:

1. S3 is object storage, fully decoupled from any server — the fix for the "disposable server" problem from Episode 1.
2. "Folders" in S3 are an illusion — really just prefixes inside a flat key namespace. We used that to organize uploads by type: documents/, images/, text-files/.
3. SHA-256 gives every file a content fingerprint. We hash the file, check it against a UNIQUE-constrained database column, and skip storing (and re-uploading) anything we've already got.
4. File size is enforced at every layer — client (UX only), backend (before issuing any URL), and S3 itself (via signed Content-Length-Range conditions) — never trusting just one.
5. Permissions stay locked down: Block Public Access ON, tightly scoped IAM policies, delivered via an IAM Role rather than hardcoded keys.
6. The full flow: request URL → backend validates + dedups + signs → client uploads directly to S3 → backend confirms via HeadObject before trusting anything.
7. Presigned URL expiry should match the use case — short (minutes) for quick interactive uploads, longer (hours, like 5 hours) specifically for large files needing a long transfer window — always paired with size/type conditions baked into the signature itself, never an open-ended, unrestricted permission slip.

Part 12 — Glossary (New Terms From Today)

Part 13 — What's Next

👦 Nephew: Uncle, this actually feels like a real production system now — not a tutorial toy. What's after this?

👨‍🦳 Uncle: Think about it yourself again, same as last time. You've now got files landing safely in S3, deduplicated, size-checked, permission-locked. But — a user in Chennai and a user in Delhi both request the same image. Where is that image physically served from, and how fast does each of them get it?

👦 Nephew: ...Straight from the S3 bucket in Mumbai, I'd guess? Which means the Delhi user gets it faster than someone further away?

👨‍🦳 Uncle: Exactly the gap we'll close next — CloudFront, AWS's CDN (Content Delivery Network), which caches your S3 content at edge locations around the world so nobody's request has to travel all the way back to Mumbai every time. And while we're there, we'll also cover what happens after upload — using Lambda to automatically process, resize, or validate a file the moment it lands in S3, without you running a single always-on server for that job.

👦 Nephew: Episode 3, then.

👨‍🦳 Uncle: Episode 3.

End of Episode 2. Next up — Episode 3: CloudFront (CDN) & Lambda-Powered Post-Upload Processing.