r/devops • u/ToddGergey • 5h ago
Ops / Incidents We reinstalled thousands of Linux and Mac build machines over one weekend after a security incident
Hey all, I work at Bitrise. We had a security incident on Nov 13 and our preliminary review is up by our VP of Engineering that’s worth reading for everyone running CI: https://bitrise.io/blog/post/the-bitrise-security-incident-preliminary-review-and-recommended-steps.
We decided not to trust any machine that had been running during the attack, so we wiped them instead of trying to clean them. That was thousands of Linux and Mac servers across three data centers. Each one got new keys before it went back into service, and nothing came back on its own. The team that owned a service had to go through its logs and get a green light from security first.
Credentials got the same treatment. We disabled all 919 of our cloud service account keys and revoked everything our GitHub apps had issued, instead of trying to work out which ones the attacker had seen.
We also published details on the attacker that you can find in our public advisory: https://bitrise.io/security-advisory-november-2026
Bitrise is secure now. We thought it would be interesting to you how an attacker could decoy themselves and cause so much work to restore normal operations.
Our team will be in the comments if you want to ask questions.