www.awsgood.com
uploads/ prefix carrying ~2,800 PUT/s plus a lifecycle COPY burst crossed the documented per-prefix ceiling (3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD per second) and started returning 503 SlowDown at 02:47 UTC.PutObject latency dropped from 1,900 ms to 120 ms in about 40 minutes.The page came from a Lambda that uploads user photos. Error text, verbatim from CloudWatch Logs:
An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4):
Please reduce your request rate.
Bucket was media-uploads-prod in us-east-1, 4.1 TB across 31 million objects, and every single key sat directly under uploads/ — uploads/9f2c1a7e-....jpg, uploads/1b8dd0f4-....png. No date folders, no shards, nothing.
Peak traffic on a normal Tuesday was around 600 PUT/s. That night a marketing push landed and the ingest fleet scaled to 800 concurrent Lambda workers, sustaining roughly 2,800 PUT/s for 19 minutes. Under 3,500/s, so it should have been fine. It wasn't, and the reason is in the next section.
First check was the cheapest: is the request rate itself the problem, or is something else broken?
aws s3api put-object
--bucket media-uploads-prod
--key healthcheck/probe.txt
--body /dev/null
--region us-east-1
That succeeded on the first try. So credentials, bucket policy, and the endpoint were fine — this was a rate signal, not a permissions signal. Second check: is everything hammering the same prefix?
```bash
aws s3api list-objects-v2
--bucket media-uploads-prod
--delimiter '/'
--max-keys 100
--query 'CommonPrefixes[].Prefix'
--output text
Empty output, `KeyCount` 100. A single delimiter level returned no common prefixes at all, which confirmed there was exactly one flat namespace. If you run that and get one line back, you have the same layout.
Third check: enable and read S3 request metrics. We were lucky — an `EntireBucket` filter had been turned on months earlier for an unrelated reason, so 1-minute metrics existed.
```bash
aws cloudwatch get-metric-statistics
--namespace AWS/S3
--metric-name 5xxErrors
--dimensions Name=BucketName,Value=media-uploads-prod Name=FilterId,Value=EntireBucket
--start-time 2026-01-14T01:30:00Z
--end-time 2026-01-14T03:30:00Z
--period 60
--statistics Sum
--region us-east-1
`5xxErrors` went from 0 to 1,840/minute starting at 02:47 and stopped climbing at 03:06. Without that filter you get nothing from CloudWatch — S3 request metrics are opt-in and billed per metric. Check the pricing page for your region before assuming they're free.
The last piece came from server access logs for the 02:00 hour:
```bash
aws s3 cp s3://media-logs-prod/s3/2026-01-14-02-00-00-0000/ . --recursive --quiet
zcat *.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -5
48921 REST.PUT.OBJECT
11044 REST.COPY.OBJECT_GET
8802 REST.GET.OBJECT
331 REST.HEAD.OBJECT
11,044 COPY operations in one hour, on the same prefix, from a lifecycle rule that transitioned objects older than 30 days to Standard-IA. That rule had been running for eight months and had never mattered, because 600 PUT/s plus 200 COPY/s is nowhere near the ceiling. Add 2,800 PUT/s during a marketing push and the shared per-prefix budget is gone.
That's the hard-won lesson: the throttle wasn't caused by one workload, it was caused by two workloads that never knew about each other. Any single-prefix design accumulates these silently.
## The fix: date shards plus a hash shard
We couldn't rename 31 million existing objects cheaply — S3 has no rename, so it's a copy-plus-delete at roughly one request pair per object, and we'd pay for the storage twice during the window. So we left the historical data where it was. Reads on the old prefix were ~400 GET/s, well under 5,500/s, so the read path never had a problem.
New writes go through a key builder in the upload service:
```python
import hashlib
from datetime import datetime, timezone
def build_key(asset_id: str, ext: str, shards: int = 16, when: datetime | None = None) -> str:
now = when or datetime.now(timezone.utc)
shard = int(hashlib.md5(asset_id.encode()).hexdigest()[:8], 16) % shards
return f"uploads/{now:%Y/%m/%d/%H}/s{shard:02d}/{asset_id}.{ext}"
Resulting key: `uploads/2026/01/14/03/s07/9f2c1a7e.jpg`. Counts under this layout at current volume: about 11,000 keys per hourly shard, 180,000 per day. That's the order of magnitude where nothing collides.
Retry config matters as much as the key layout. The default `legacy` retry mode backs off on a fixed curve and lets every worker retry in near-lockstep — a self-inflicted thundering herd:
```python
from botocore.config import Config
cfg = Config(
retries={"max_attempts": 10, "mode": "adaptive"},
max_pool_connections=50,
connect_timeout=3,
read_timeout=10,
)
And the lifecycle rule moved off the hot prefix entirely. New rule scopes to `archive/`, with a transition window at 45 days and a separate `ExpiredObjectDeleteMarker` cleanup. Lifecycle rules are evaluated against a prefix filter, so pointing one at `uploads/` and another at `archive/` is two rules, not one — worth writing down somewhere.
## Detours: three things that didn't help
We burned about 90 minutes on these before the access logs gave us the answer.
**Putting CloudFront in front of the bucket.** Useless for this incident. CloudFront absorbs read traffic; our failure was entirely the write path. It did help `GET` latency for European users a few weeks later, but that's a different project.
**Bumping `max_attempts` without changing the retry mode.** We went from 3 to 10 attempts in legacy mode and the error rate went *up* for about four minutes, because all 800 workers were retrying on the same schedule. The `adaptive` mode plus jitter is what actually calmed it. If you only change one setting, change `mode`.
**Turning off the lifecycle rule entirely.** This worked, and it's also why we nearly stopped there. The rule was doing real cost work — Standard-IA on cold photos — and disabling it for the night just deferred the COPY burst to the next day. The correct fix was scoping it to a different prefix, not killing it.
The pattern across all three: we kept treating the symptom (errors) instead of the shared resource (one prefix's request budget).
## The generalized playbook, and when not to shard
Next time a bucket starts throwing `SlowDown`, this is the order I run things in.
1. Confirm it's throttling, not auth: `aws s3api put-object --bucket <b> --key healthcheck/probe.txt --body /dev/null`. A success means rate, not policy.
2. Check the namespace shape: `aws s3api list-objects-v2 --bucket <b> --delimiter '/' --max-keys 100 --query 'CommonPrefixes[].Prefix'`. One entry or none means flat.
3. Pull `5xxErrors` and `PutRequests` at 1-minute resolution, if a request-metrics filter exists. If it doesn't, that's your first prevention task.
4. Count operations by type from server access logs for the peak hour — `awk '{print $7}'` on field 7. COPY and PUT share the same budget; this is where hidden workloads show up.
5. Only then change the key layout. Sharding into an already-broken bucket is fine; sharding before you know what generated the load is not.
| Layout | Keys per prefix | Hot-spot risk | Listing ergonomics | Migration cost |
|---|---|---|---|---|
| Flat `uploads/<uuid>` | 31M | High past ~3,500 PUT/s | Poor — `ls` returns nothing useful | None (current) |
| Date only `uploads/2026/01/14/` | ~180k/day | Medium — hourly bursts still collide | Good | New writes only |
| Date + hash `.../03/s07/` | ~11k/hour | Low | Good, shard is invisible to humans | New writes only |
| Random first char `uploads/7/...` | ~120k per char | Low | Poor — destroys any natural order | New writes only |
### When not to shard prefixes at all
Don't add hash shards if you're using the prefix as a Hive-style partition for Athena, Spectrum, or partition projection. `s3://bucket/events/year=2026/month=01/` is a partition layout, and a `s07` segment in the middle breaks the `LOCATION` path for every tool that reads it. Likewise: under about 1,000 PUT/s sustained, sharding buys you nothing and costs you browsability — a support engineer running `aws s3 ls` at 3 a.m. cares about that. And if your workload is sequential-by-time reads (log replay, replaying a day of events), keep the timestamp-first layout; it's far more valuable than shard headroom you'll never use.
## Prevention: alarms, Storage Lens, and a naming rule
The alarm we should have had on day one:
```bash
aws cloudwatch put-metric-alarm
--alarm-name s3-media-uploads-5xx
--namespace AWS/S3
--metric-name 5xxErrors
--dimensions Name=BucketName,Value=media-uploads-prod Name=FilterId,Value=EntireBucket
--statistic Sum --period 60 --evaluation-periods 2
--threshold 10 --comparison-operator GreaterThanThreshold
--alarm-actions arn:aws:sns:us-east-1:111122223333:ops-pager
--region us-east-1
Two consecutive minutes over 10 errors is early enough to catch a ramp before a client notices. Storage Lens advanced metrics give you prefix-level breakdowns if you need to find the hot prefix across a hundred buckets without turning on access logging everywhere — it's an extra charge, so scope it to the accounts that matter.
The naming rule is the cheapest control: no production bucket accepts a key that doesn't match `^<domain>/d{4}/d{2}/d{2}/d{2}/sd{2}/`. Enforce it in a shared upload helper, not in code review — people forget. We run this alarm set and the Storage Lens review for a handful of buckets through awsgood.com's managed services, mainly because the quarterly check is the first thing to slip when nothing is on fire.
Start with the request-metrics filter. It takes 30 seconds to enable and it's the difference between a 40-minute diagnosis and a three-hour one.
## FAQ
**Does S3 still enforce a per-prefix request rate?**
AWS documents 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD per second per prefix, and while S3 has gotten much better at absorbing bursts, sustained load near those numbers plus a background job will still produce `503 SlowDown`. Treat the numbers as a budget you share across every writer, not a hard wall you can probe.
**Should I hash-shard every new bucket?**
No. Shard when sustained writes approach four figures per second, or when a background process (lifecycle, replication, batch re-processing) shares the prefix with live traffic. Below that, a `yyyy/mm/dd` layout gives you lifecycle targeting and human-readable listing for free.
## Sources
- [Best practices design patterns: optimizing Amazon S3 performance](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html)
- [Amazon S3 error responses — SlowDown](https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html)
- [Managing your storage lifecycle](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html)
-