r/devops • • 5h ago

Ops / Incidents We reinstalled thousands of Linux and Mac build machines over one weekend after a security incident

78 Upvotes

Hey all, I work at Bitrise. We had a security incident on Nov 13 and our preliminary review is up by our VP of Engineering that’s worth reading for everyone running CI: https://bitrise.io/blog/post/the-bitrise-security-incident-preliminary-review-and-recommended-steps.

We decided not to trust any machine that had been running during the attack, so we wiped them instead of trying to clean them. That was thousands of Linux and Mac servers across three data centers. Each one got new keys before it went back into service, and nothing came back on its own. The team that owned a service had to go through its logs and get a green light from security first.

Credentials got the same treatment. We disabled all 919 of our cloud service account keys and revoked everything our GitHub apps had issued, instead of trying to work out which ones the attacker had seen.

We also published details on the attacker that you can find in our public advisory: https://bitrise.io/security-advisory-november-2026

Bitrise is secure now. We thought it would be interesting to you how an attacker could decoy themselves and cause so much work to restore normal operations.

Our team will be in the comments if you want to ask questions.


r/devops • • 8h ago

Discussion GitHub vs azure DevOps what's your opinion?

16 Upvotes

Currently in my company we run both for different historical reasons and mostly using GH for new projects since it seems the correct path for the future, but it seems that ADO is also being added with new functionalities such as advanced security and yesterday GitHub copilot app support...

Do you think Ado will remain be maintained and new features arriving? Before I thought it will be deprecated sooner or later but now I'm not that sure

What do you think? And what key differenciators do you see?


r/devops • • 7h ago

Discussion Main use-case of cloud sandboxes and how it works?

0 Upvotes

Hey guys, I've been seeing a shift in cloud sandboxes being mentioned a lot lately in agent architecture discussions. I'm trying to understand the difference in them compared to say, just using either a standard Docker container or a basic cloud VM.

My very basic level understanding is that these cloud platforms give agents isolated environments to run code without using your local machine. Additionally I've seen a post on Ramp Inspect where they boot a separate remote sandbox for coding sessions with pre-warmed repo snapshots. However, I still don't see what they do that regular Docker containers don't.

Feel free to correct any misconceptions I have here lol, just trying to make sure how it all works. Thanks!


r/devops • • 22h ago

Discussion Solo dev, 4 self-hosted Windows runners on one laptop, merge trains take ~2 hours. How would you speed this up?

0 Upvotes

solo dev building a windows desktop app, most of the coding is done by claude code sessions (earlier post about that). the bottleneck now is testing. the sessions spend more time waiting on CI than writing code, so i'm looking for ideas

The setup:

  • the app drives blender, unity 2022.3 and the vrchat sdk, so the real tests need windows
  • 4 self-hosted github actions runners, all on my one laptop (ryzen 9 7945hx 16 cores, rtx 4090 laptop 16gb, 64gb ram, 8tb nvme), one job per runner. the real blender/unity jobs can only use 3 of them, and up to 2 local test runs share the same laptop too
  • self-hosted because github's hosted windows minutes cost double and i'm on a tight budget, so the laptop is the free option
  • changes get batched into merge trains, 1-8 changes each, about 3 on average

What a train looks like:

  • pipeline first: 16 real tool jobs (mostly blender, plus unity and a vrm validator) and fake-tool pytest in 4 shards, about 45-80 min
  • then the full windows UI suite, ~25,600 tests in 3 shards, about 65 min. these run back to back on purpose because the timing tests flake when they share the machine with blender jobs, so that's about an hour of every train just waiting
  • a train takes about 2 hours start to finish (83-145 min over the last 8)
  • 7 of the last 8 landed, 4 clean first try. fix and retest loops took anywhere from 30 min to 2 hours
  • laptop averages 55-60% cpu while all this runs and peaks at 100%, ram peaks around 80%

How much testing:

  • every fix starts with a failing test, a review session checks it, the author runs the affected tests locally, then the train runs everything again in CI plus a local gate
  • so the plain python tests end up running about 3 times. the real blender/unity tests did catch the only real failure in those 8 trains though

What i've tried:

  • dev drive, only ~4% faster on my test so i didn't keep it
  • sharding by recorded test duration, and splitting local runs by run time so they stop timing out
  • allowing one re-run of only the failed jobs for flaky infra stuff
  • tags so only affected tests run, built but not measured yet

Why one train at a time: changes often touch the same files, and the timing tests got flaky when all 4 runners were busy. so i'm not sure more runners even helps or if the real answer is a second machine

What i'm asking:

  1. am i over testing? how do you cut duplicate runs without losing coverage
  2. how do you pick tests by changed area without missing stuff that imports across areas
  3. is there a safe way to run 2 trains when changes overlap files, or is one at a time just right
  4. is a second machine or a cloud windows runner worth it for a ~2 hour cycle, or is there a better service for this
  5. how do you keep timing sensitive tests reliable when CI jobs share one pc
  6. could the plain python and fake-tool tests run on linux, or does that risk missing windows-only bugs

at the current pace my next release is about 3 weeks out and the one after that is a couple months, so anything that shortens the 2 hour loop helps a lot. thanks!