I have a story to share!
I've had the unfortunate luck to participate in resolving one of the scariest and also professionally most insightful security incident I've ever seen, and next to the lack of sleep what I hated most: how will I talk about this with folks? Because there is so much learning in this, it worths discussing. Then my VP of Eng said: okay, we need to go and be as transparent with this as much as we can, then he wrote this piece.
Of course a serious forensics investigation is still ongoing, so there are details we cannot explicitly share, but still, it feels good to work in a team where transparency is really the default approach, and I can openly discuss most of the things that happened with fellow folks in the industry. And if you are reading this: don't want to scare you, but probably your company is up next. I'm telling everyone around me in the last few months that 2027 will be the loudest year in cybersecurity, as both researchers and attackers are using AI to find new opportunities. I was just not expecting this to happen in 2026 :'D
I'm working as an Engineering Director, I have more than 15 years of hands-on experience, so I've seen a lot of different situation when đ© hit the fans. Given this, when it's Friday late afternoon and a member of our Security team is calling me on the phone saying "we need you ASAP", I could have just hoped my weekend is not ruined. But that was Friday the 13th, so I shouldn't have had my hopes up high.
We triggered a SEV-1 incident after reviewing an alert from one of our internal tools, and it felt like a massive adrenalin rush when my VP called it: initiate shutdown protocol. I think that was the point I realized I won't stand up from my desk, and it's gonna be a long night.
Then we started rotating and revoking credentials everywhere, including almost a thousand service account keys and all our GitHub app tokens. Our Platform team started to rebuild the infrastructure across three data centers.
My learnings so far:
- Perfect security doesnât exist. So you NEED to be fast and prepared. I know a lot of folks will come at me in the comments that if this can happen with us, we don't have good practices, or we don't understand security, but trust me: you can never cover a 100% of your systems. That's why you need the ability to detect early, make decisions quickly, hold your team together, and move forward fast.
- You need incident commanders. It's a hell of a ride, and I somehow love this role, but this was going on for day and night, so I obviously wasn't the only one. But this role was much needed. We had to ensure decisions being made quickly, people are going to sleep when they are tired, we have other team members to jump back in to their places etc.
- Everyone helped. Of course that can be tricky as it can lead to extra noise, but really, everyone was doing whatever they could to help. One of a non-engineering team member started to send food delivery vouchers to folks working, without any hesitation.
- Attackers probably prepared. they signed up for free test accounts with throwaway emails. What they did was fitting into normal patterns of using the product, so nothing suspicious.
- If there's a broken window, someone will find it. The day before the breach, they ran scripts from a new account trying to grab credentials and see other customersâ data. Every request was refused. They got in through a forgotten internal bot account that had no 2FA. (That sucks. This is a good reminder. If you read this: check your internal stuff, change passwords, set up 2FA, rotate secrets.)
- Trust no-one. During the incident, we treated the whole platform as compromised. We revoked all account keys and every GitHub token, then rebuilt everything from scratch. If someone was messaging me on Slack, I asked them a question only they could know the answer.
- If you block, you'll be in the dark. Instead of just blocking the attacker, we âtarpittedâ them: their requests got slowed way down so we could watch what they tried and study the logs.
- It's not only your attacker who has AI: you have it, too. Yea, AI-driven attacks sucks, but it's not like someone is having a super weapon you don't, your LLM can help you a lot during an incident. Especially when you have to analyze tons of data to provide forensics reports and understand what happened exactly. Despite this, what AI-driven tools can do is quite scary, if you are not able to catch early, it's not gonna end well, not even on the most protected system.
It was a rough few days, but luckily we managed to solve this, and to be honest, during the incident and afterwards we hardened so much things that would have been months of continuous work, so I'll guess there's always a silver lining.
Anyway, thanks for reading through. It really was a hell of a ride, and feel free to ask, if I can, I'm happy to answer.