r/aws • u/CriticalPhen • 9h ago
article Went to bed with a $50 budget alert. Woke up to a $4,180 AWS bill from a key I forgot existed
This happened last week and I'm still a bit shaken, so writing it down.
I went to sleep around midnight. The budget alert email for my $50 budget arrived at 6:40 in the morning, when the bill was already over $3,000. By the time I logged in and found the cause it was $4,180. GPU instances in 4 regions I've never used, all launched within about 20 minutes of each other around 1am.
The key was from an IAM user I made in 2024 for a tiny CLI tool I published to npm. I forgot that user existed. The package never had a .npmignore and no "files" field in package.json, so npm publish included everything in the folder. Including the .env with the access key. It's been sitting in a public tarball on the npm registry for over a year. I only ever checked GitHub for leaked keys, and that repo on GitHub is clean.
What I keep coming back to is the alert. I thought $50 was my safety net. But billing data runs hours behind, so the alert is really telling you what happened earlier in the night. Someone who launches a dozen GPU instances at 1am has a few hours of free compute before anything even emails you.
Support was decent so far. They asked me to secure the account, delete the key and confirm nothing else is running, and they're reviewing the charges.
What I changed:
- Deleted every IAM user with long lived keys and moved my own CLI access to IAM Identity Center, so there are no access keys on my laptop anymore
- Set up a budget action that applies a deny policy on EC2 when spend crosses $100, so it actually stops things instead of only sending an email
- The few app secrets I still need live in Infisical now and get pulled in at runtime, and every package I publish has a "files" field so only the build goes out
- Ran npm pack --dry-run on all my old packages to see what each one ships
Is there anything closer to a real hard stop on AWS than budget actions? And has anyone actually checked what their old npm packages contain? I'd never once looked