r/ExperiencedDevs • • 2h ago

AI/LLM Scariest Friday the 13th as incident commander in a recent AI-driven attack

9 Upvotes

I have a story to share!

I've had the unfortunate luck to participate in resolving one of the scariest and also professionally most insightful security incident I've ever seen, and next to the lack of sleep what I hated most: how will I talk about this with folks? Because there is so much learning in this, it worths discussing. Then my VP of Eng said: okay, we need to go and be as transparent with this as much as we can, then he wrote this piece.

Of course a serious forensics investigation is still ongoing, so there are details we cannot explicitly share, but still, it feels good to work in a team where transparency is really the default approach, and I can openly discuss most of the things that happened with fellow folks in the industry. And if you are reading this: don't want to scare you, but probably your company is up next. I'm telling everyone around me in the last few months that 2027 will be the loudest year in cybersecurity, as both researchers and attackers are using AI to find new opportunities. I was just not expecting this to happen in 2026 :'D

I'm working as an Engineering Director, I have more than 15 years of hands-on experience, so I've seen a lot of different situation when đŸ’© hit the fans. Given this, when it's Friday late afternoon and a member of our Security team is calling me on the phone saying "we need you ASAP", I could have just hoped my weekend is not ruined. But that was Friday the 13th, so I shouldn't have had my hopes up high.

We triggered a SEV-1 incident after reviewing an alert from one of our internal tools, and it felt like a massive adrenalin rush when my VP called it: initiate shutdown protocol. I think that was the point I realized I won't stand up from my desk, and it's gonna be a long night.

Then we started rotating and revoking credentials everywhere, including almost a thousand service account keys and all our GitHub app tokens. Our Platform team started to rebuild the infrastructure across three data centers.

My learnings so far:

  • Perfect security doesn’t exist. So you NEED to be fast and prepared. I know a lot of folks will come at me in the comments that if this can happen with us, we don't have good practices, or we don't understand security, but trust me: you can never cover a 100% of your systems. That's why you need the ability to detect early, make decisions quickly, hold your team together, and move forward fast.
  • You need incident commanders. It's a hell of a ride, and I somehow love this role, but this was going on for day and night, so I obviously wasn't the only one. But this role was much needed. We had to ensure decisions being made quickly, people are going to sleep when they are tired, we have other team members to jump back in to their places etc.
  • Everyone helped. Of course that can be tricky as it can lead to extra noise, but really, everyone was doing whatever they could to help. One of a non-engineering team member started to send food delivery vouchers to folks working, without any hesitation.
  • Attackers probably prepared. they signed up for free test accounts with throwaway emails. What they did was fitting into normal patterns of using the product, so nothing suspicious.
  • If there's a broken window, someone will find it. The day before the breach, they ran scripts from a new account trying to grab credentials and see other customers’ data. Every request was refused. They got in through a forgotten internal bot account that had no 2FA. (That sucks. This is a good reminder. If you read this: check your internal stuff, change passwords, set up 2FA, rotate secrets.)
  • Trust no-one. During the incident, we treated the whole platform as compromised. We revoked all account keys and every GitHub token, then rebuilt everything from scratch. If someone was messaging me on Slack, I asked them a question only they could know the answer.
  • If you block, you'll be in the dark. Instead of just blocking the attacker, we “tarpitted” them: their requests got slowed way down so we could watch what they tried and study the logs.
  • It's not only your attacker who has AI: you have it, too. Yea, AI-driven attacks sucks, but it's not like someone is having a super weapon you don't, your LLM can help you a lot during an incident. Especially when you have to analyze tons of data to provide forensics reports and understand what happened exactly. Despite this, what AI-driven tools can do is quite scary, if you are not able to catch early, it's not gonna end well, not even on the most protected system.

It was a rough few days, but luckily we managed to solve this, and to be honest, during the incident and afterwards we hardened so much things that would have been months of continuous work, so I'll guess there's always a silver lining.

Anyway, thanks for reading through. It really was a hell of a ride, and feel free to ask, if I can, I'm happy to answer.


r/ExperiencedDevs • • 8h ago

AI/LLM In a world where AI does replace all of us, what job do you think engineers will flock to next?

0 Upvotes

Opinions are like assholes, everyone’s got one and we should keep em to ourselves. That’s how I feel about the “whether ai will replace software engineering question”.

Instead, I’m curious, if the worst outcome does come to bear, then where do you think experienced software engineering skills will be most applicable?

My bet is on finance, assuming that doesn’t also get replaced. And if that gets replaced, perhaps the electrician trade.


r/ExperiencedDevs • • 13h ago

Career/Workplace Job ads requiring cover letter to address many person specifications but impose word limits

0 Upvotes

I have around 10 years of experience and am looking for another senior position. The problem is that every job ad has at least 10 "person specification points" or more, and they ask that the cover letter address them, BUT then impose a small word limit on it. You're then rejected because you didn't have enough experience?

For example, there's one that looks interesting but has 15 separate criteria points, but "Your covering letter should be no more than 1 page referencing how you meet the criteria set out in the job description". At least for this one I can make the font and margins small to fit more in.


r/ExperiencedDevs • • 23h ago

Career/Workplace How are your teams proposing architecture changes these days?

80 Upvotes

In the age of LLMs, since implementation has gotten faster, I'm seeing fewer processes form around what a prototype should look like or who needs to sign off on architecture changes.

Literally had my manager over the span of 4 days build a whole new system that takes an idea from a FAANG research paper. That's fine with me except there was no discussion on how we'd adopt it, what a prototype version would be, what to onboard and how it affects the current work in progress. He got a bunch of Staff level people on board and is pushing it through while others weren't even aware something like this is happening. I'm Staff+ and brought up quality concerns (the whole system is based on ML models) and was ignored.

It seems like the decision to adopt/abandon should at least involve the same rigor now that getting to v0.1 is easier.

How are your teams handling this? Is it common for your managers to completely overhaul systems without consulting the leads?

I feel like I'm going insane. I don't want to gatekeep ideas but rethinking current WIP in favor of building on the new thing seems ill-advised. For most of this work there are no tickets, no decision logs, nothing. If AI can do it, it seems like it should be easier to have a lightweight process instead of no process?


r/ExperiencedDevs • • 7h ago

Career/Workplace How is your org handling AI peer-review?

61 Upvotes

When a developer creates a pull request that's mostly AI-generated, how do you handle peer reviews?

At my org, pull requests are mostly AI generated. Most of my peers handle peer reviews by putting the pull request back into AI.

I was doing the same for a while, but now I carefully review every single PR regardless of the source. It's not fun, it's not pretty, and it takes me a long time, but until the org says that human PRs are not necessary, that's what I'm going to do.

I leave a LOT of comments - I'm hoping this will shed light on the actual code quality of AI-generated code. Some teams aren't using agentic instructions, so maybe a PR or three littered with comments will encourage them. Even the fully-curated agentic code still suffers from quality issues, even from frontier models, just not as many.

I assume some orgs have simply abandoned peer-reviews, while others encourage it, and even those are probably being waved through most of the time (or just put back through AI).

What's your experience?