Why "aws s3 sync" Was Re-Uploading My Entire Site on Every Push (And How We Fixed Our CI/CD Pipeline)

When I moved this site off Amplify and onto a globally replicated S3S3Simple Storage ServiceCloudAmazon Web Services scalable object storage offering 99.999999999% (11 9s) of data durability with fine-grained IAM policies and versioning. Multi-Region Access Point (MRAPMRAPMulti-Region Access PointCloudAn Amazon S3 global endpoint that dynamically routes client requests across multiple AWS regions using AWS Global Accelerator for lowest latency.) behind CloudFrontCloudFrontCloudFront, I thought the CI/CDCI/CDContinuous Integration & Continuous DeliveryDevOpsDevOps automation pipeline practice that automatically compiles, tests, and deploys software changes directly to staging or production environments. pipeline was about as simple and done as a static site pipeline gets: build Astro in one stage, run aws s3 sync . s3://my-bucket/ --delete in the next, invalidate CloudFront, and call it a day.

And on paper, it worked. Every push to main built cleanly and showed up live a minute or two later.

Wait a minute, though: why was a one-line typo fix taking a minute or two to build and deploy, and why was S3 Cross-Region Replication chewing through nearly 60 MB of traffic on every single commit?

Once I actually looked under the hood of our GitLab CI pipeline today, I found two classic infrastructure traps hiding in plain sight.

---

Trap #1: Caching node_modules/ When before_script Runs npm ci

Like most GitLab CI templates for Node projects, my .gitlab-ci.yml originally cached node_modules/ between builds and ran npm ci at the start of the build stage:

cache:
  paths:
    - node_modules/

build:
  stage: build
  before_script:
    - npm ci
  script:
    - npm run build

Spot the contradiction?

By design, npm ci deletes the entire node_modules/ directory before installing a clean slate from package-lock.json. That means caching node_modules/ in GitLab CI was doing nothing except wasting time zipping and unzipping an archive that npm ci immediately nuked on line one.

Worse yet, both Astro’s image optimization cache (node_modules/.astro/assets/, holding 284 responsive WebP variants of my photography gallery) and our custom Satori + Resvg Open Graph card cache (node_modules/.cache/og-images/, holding 140 generated 1200×630 PNGs) lived inside node_modules/.

Locally, warm builds finished in 1.6 seconds because both caches hit 100%. In CI, npm ci wiped both caches on every run, forcing the runner to re-encode 284 WebP photos and rasterize 140 Open Graph images from scratch every single time.

The Fix: Move Build Caches Out of node_modules/

The fix was to separate the package manager cache from the build artifact cache and move both outside node_modules/:

1. In astro.config.mjs, we pointed Astro’s cache directory to ./.cache/astro:

   export default defineConfig({
     cacheDir: './.cache/astro',
     // ...
   });

2. In our Open Graph generator (generateOgImage.ts), we moved the SHA-256 content-addressed PNG cache to ./.cache/og-images/.
3. In .gitlab-ci.yml, we cached .cache/ and .npm/ keyed on package-lock.json, and switched npm ci to pull from the local .npm tarball cache offline:

cache:
  key:
    files:
      - package-lock.json
  paths:
    - .cache/
    - .npm/

build:
  image: mcr.microsoft.com/playwright:v1.64.1-noble
  stage: build
  before_script:
    - npm ci --cache .npm --prefer-offline
  script:
    - mkdir dist
    - npx playwright install chromium
    - npm run build

Now npm ci can blow away node_modules/ all it wants—.cache/astro and .cache/og-images sit safely outside the blast radius, dropping warm build times from ~60 seconds down to ~1.6 seconds.

---

Trap #2: Why aws s3 sync Re-Uploads Identical Files in CI

With the build stage sorted, I looked at the deploytos3 stage:

deploytos3:
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  stage: deploytos3
  script:
    - cd ./dist
    - aws s3 sync . s3://example-bucket-primary/ --delete
    - aws cloudfront create-invalidation --distribution-id EXXXXXXXXXXXXX --paths '/*'

There’s a widespread assumption that aws s3 sync hashes your files and only uploads what changed. It doesn’t.

By default, aws s3 sync compares only two things: file size and last-modified timestamp (mtime). If a local file has the exact same size as the object in S3 but its local mtime is newer than the S3 object’s timestamp, aws s3 sync assumes the file was modified and uploads it again.

In a static site CI pipeline, npm run build generates a brand-new dist/ folder on every run. Every single HTML file, RSS feed, Open Graph PNG, and WebP photo gets a brand-new mtime of “right now.” So aws s3 sync dutifully re-uploaded all 592 files (58 MB) on every push.

And because my primary Tokyo bucket (ap-northeast-1) replicates via S3 Cross-Region Replication to secondary buckets across Europe, the US, Asia-Pacific, and South America behind our S3 Multi-Region Access Point, every single redundant PUT was multiplied across the globe—followed by a blanket /* CloudFront invalidation that dumped the edge cache for the whole site.

---

Writing an MD5/ETag-Aware Deploy Script (And a Container Gotcha)

To make deploys genuinely incremental without relying on local filesystem timestamps, we wrote a custom s3-deploy.sh script.

Instead of trusting mtime, the script queries S3 once via aws s3api list-objects-v2 to grab the Key and ETag (which for non-multipart objects is simply the hex MD5 digest) of every object currently in the bucket, hashes the local files in dist/, and only uploads files that are new or whose MD5 actually changed.

The First Attempt: python3: command not found

My first draft used a short inline python3 snippet to parse the JSON response from list-objects-v2 and compute the MD5 diff. It ran great on macOS—and immediately faceplanted in GitLab CI:

./scripts/s3-deploy.sh: line 27: python3: command not found

Even though AWS CLI v2 is written in Python, the registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest container ships a self-contained embedded runtime for aws and strips out a standalone python3 binary (as well as jq). What is inside the container? Standard POSIX/GNU coreutils: bash, awk, find, md5sum, sort, and tr.

And honestly, when you have aws --query and awk, you don’t need Python at all.

The Final Zero-Dependency POSIX Script

By asking the AWS CLI to format its output as tab-separated text (--query 'Contents[].[Key, ETag, Size]' --output text), we can feed the remote inventory and a single batch find ... -exec md5sum {} + pass directly into awk:

#!/usr/bin/env bash
set -euo pipefail

BUCKET="${S3_BUCKET:-example-bucket-primary}"
DIST_ID="${CLOUDFRONT_DISTRIBUTION_ID:-EXXXXXXXXXXXXX}"
DIST_DIR="${1:-./dist}"

if [ ! -d "$DIST_DIR" ]; then
  echo "Error: build output directory '$DIST_DIR' does not exist." >&2
  exit 1
fi

cd "$DIST_DIR"

echo "==> Fetching remote object inventory from s3://${BUCKET} ..."
REMOTE_TSV="$(mktemp)"
# Output tab-separated lines: Key \t ETag \t Size
aws s3api list-objects-v2 \
  --bucket "$BUCKET" \
  --query 'Contents[].[Key, ETag, Size]' \
  --output text | tr -d '"' > "$REMOTE_TSV"
echo "==> Remote inventory complete."

LOCAL_TSV="$(mktemp)"
# Compute MD5 for all local files in a single batch (Linux md5sum + macOS md5 -r fallback)
if command -v md5sum >/dev/null 2>&1; then
  find . -type f -exec md5sum {} + | sed 's|  \./|\t|' > "$LOCAL_TSV"
else
  find . -type f -exec md5 -r {} + | sed 's| \./|\t|' > "$LOCAL_TSV"
fi

PLAN_FILE="$(mktemp)"

awk -F'\t' '
  NR == FNR {
    if (NF >= 2 && $1 != "None") {
      remote_etag[$1] = $2
      remote_size[$1] = $3
    }
    next
  }
  {
    md5 = $1
    key = $2
    local_seen[key] = 1
    if (!(key in remote_etag)) {
      print "UPLOAD\t" key
    } else if (remote_etag[key] != md5) {
      # Skip multipart ETag false-positives on immutable _astro/ assets
      if (index(remote_etag[key], "-") > 0 && index(key, "_astro/") == 1) {
        next
      }
      print "UPLOAD\t" key
    }
  }
  END {
    for (k in remote_etag) {
      if (!(k in local_seen)) {
        print "DELETE\t" k
      }
    }
  }
' "$REMOTE_TSV" "$LOCAL_TSV" | sort -k1,1r -k2,2 > "$PLAN_FILE"

rm -f "$REMOTE_TSV" "$LOCAL_TSV"

Once awk emits the UPLOAD and DELETE plan, the second half of the script applies the changes and collects only the mutable URL paths that actually changed for CloudFront:

UPLOAD_COUNT=0
DELETE_COUNT=0
CHANGED_PATHS=()

while IFS=$'\t' read -r action key; do
  [ -z "${action:-}" ] && continue
  if [ "$action" = "UPLOAD" ]; then
    UPLOAD_COUNT=$((UPLOAD_COUNT + 1))
    if [[ "$key" == _astro/* ]]; then
      aws s3 cp "$key" "s3://${BUCKET}/${key}" \
        --cache-control "public, max-age=31536000, immutable" \
        --only-show-errors
    else
      aws s3 cp "$key" "s3://${BUCKET}/${key}" \
        --only-show-errors
      if [[ "$key" == */index.html ]]; then
        CHANGED_PATHS+=("/${key%index.html}")
        CHANGED_PATHS+=("/${key}")
      elif [[ "$key" == "index.html" ]]; then
        CHANGED_PATHS+=("/")
        CHANGED_PATHS+=("/index.html")
      else
        CHANGED_PATHS+=("/${key}")
      fi
    fi
    echo "  [uploaded] $key"
  elif [ "$action" = "DELETE" ]; then
    DELETE_COUNT=$((DELETE_COUNT + 1))
    aws s3 rm "s3://${BUCKET}/${key}" --only-show-errors
    CHANGED_PATHS+=("/${key}")
    echo "  [deleted]  $key"
  fi
done < "$PLAN_FILE"

rm -f "$PLAN_FILE"

echo "==> Sync summary: ${UPLOAD_COUNT} uploaded, ${DELETE_COUNT} deleted."

if [ "${#CHANGED_PATHS[@]}" -eq 0 ]; then
  echo "==> No mutable routes changed; skipping CloudFront invalidation."
elif [ "${#CHANGED_PATHS[@]}" -gt 25 ]; then
  echo "==> ${#CHANGED_PATHS[@]} paths changed (>25); invalidating /* on CloudFront (${DIST_ID})..."
  aws cloudfront create-invalidation --distribution-id "$DIST_ID" --paths "/*"
else
  echo "==> Invalidating ${#CHANGED_PATHS[@]} changed path(s) on CloudFront (${DIST_ID})..."
  aws cloudfront create-invalidation --distribution-id "$DIST_ID" --paths "${CHANGED_PATHS[@]}"
fi

---

What This Buys Us

1. Zero Redundant S3 Uploads & Replication: When I fix a typo in a single blog post, the deploy stage now uploads only the 4–5 files whose MD5 actually changed (the post HTML, the blog index, the RSS/Atom/LLM feeds) instead of pushing all 592 files and 58 MB across every replicated S3 region.
2. Long-Lived Immutable Caching on _astro/*: Content-hashed bundles and WebP photos in _astro/* now get Cache-Control: public, max-age=31536000, immutable headers on upload, and never trigger a CloudFront invalidation since their filenames are already content-addressed.
3. Surgical CloudFront Invalidations: If fewer than 25 mutable routes changed, we invalidate only those exact paths so every other page stays warm at CloudFront edge locations worldwide (and if a site-wide layout tweak touches more than 25 paths, it falls back to /* so we stay well inside CloudFront’s free monthly invalidation tier).

Sometimes the best infrastructure improvements aren’t adding new services—they’re noticing where your defaults are quietly doing 100× more work than necessary.