r/softwaretesting • • 18h ago

Lessons from building a self-hosted Patchright + Camoufox + rrweb runner with an 800MB proxy byte-budget guard

3 Upvotes

I built an open-source Playwright/Crawlee studio (MIT) over the last few months and wanted to share a few engineering patterns that actually held up in production runs:

  1. Patchright + Camoufox separation: Keeping Patchright isolated from playwright-extra avoids CDP-level leaks while deriving the UA from the installed browser version to avoid version drift.
  2. Live rrweb bridge without leaking secrets: Streaming DOM snapshots to a local dashboard (:4310) in 25-event batches every 250ms with maskAllInputs enabled, plus an in-memory FIFO secret redactor (up to 500 secrets) before anything hits logs or telemetry.
  3. Metered proxy Byte-Budget Guard: Enforcing an 800MB cap on a 1GB residential rail with automatic failover to static datacenter pools, plus an ultra_lite asset cache that blocks heavy media while keeping GA4/analytics beacons firing.
  4. Read-only Target Preflight: Classifies targets as open / cf_interstitial / site_tick / dead before spinning up workers. Strictly detection and reporting — never attempts to solve CAPTCHAs.
  5. Live Inbox OTP verification: Spins up temporary inboxes per session to test full signup and email verification flows end-to-end.

Happy to share the MIT GitHub repo in the comments if anyone wants to inspect the code or tests.