r/sysadmin • • 9h ago

Inside Microsoft’s giant Outlook outage

854 Upvotes

https://www.theverge.com/tech/1035364/microsoft-outlook-outage-notepad

Microsoft engineers had a running bridge call the entire time, and at one point it hit the maximum of 1,000 participants. That shows you how many teams were involved with trying to find the root cause of this huge outage, which started at the peak window of traffic in the US.


r/sysadmin • • 21h ago

General Discussion java4shell RCE in b64 decoder CVE 9.2

391 Upvotes

https://calif.io/java4shell/

Well here we go again...


r/sysadmin • • 9h ago

Rant I am completely worn the f**k out, and I don't know where to go from here.

223 Upvotes

31 years old, been in IT since 19 when I broke (and fixed) my first Exchange environment during a CU upgrade. This post might out me to my colleagues, but frankly I don't care anymore. (and if any of you do happen to read this, I know you understand, feel free to talk to me alone about it.)

I've been at my current employer (an MSP) for 8 years. We develop and sell software integrations for an ERP system while also offering traditional break/fix IT services (servers, clients, user support) rather than a modern per-device/user model.

Our compensation is a pretty mediocre base salary for my years in the sector, paired with a bonus based on billables. Because of the sheer number of hours I've put in, it has generally given me a very good income—easily double what most of my friends in other fields make.

For the past 6 years, we’ve also been helping the ERP vendor upgrade customers to newer versions. While straightforward and good for billables, it is completely mind-numbing and destructive to both my psyche and my social life, as these upgrades almost always happen on late Friday afternoons, evenings, and weekends.

On top of that, I'm the go-to guy for anything above entry-level infrastructure. I've been implementing and teaching the other guys things like:

  • Hyper-V Failover Clusters with SANs and/or HCI solutions
  • Secure backup architectures with Veeam (multiple copies, test restores, immutability, etc.)
  • Network segregation with VLANs and securing LAN/WLAN access with 802.1X
  • Entra/AD Sync integrations, client deployment/management with Intune and Autopilot, WHfB, Conditional Access, and more

Meanwhile, I'm still expected to jump into everyday L1 support tickets, handle the ERP upgrades, and deal with whatever else gets thrown my way.

We don't have a written daily billable target, though you're expected to bill at least 8 hours a day. Over the past few years, I've been billing between 2,800 and 3,200 hours a year—including 250–400 hours of overtime on Friday nights and weekends on top of the normal 9-to-5, not instead of it.

When I started here 8 years ago, I loved IT and problem-solving. I loved tinkering with my homelab and gaming—I used to take a week off for every WoW expansion. This job has completely killed the passion I had for computers and gaming.

A couple of years ago, I got into cars instead and bought two project cars. Diving headfirst into them gave me an outwhere I could use my hands, build cool shit, have fun and make new friends. Now I haven't turned a wrench in 6 months.

That made me realize I couldn't keep going like this, so I decided to slow down—work my 8 hours and limit overtime as much as possible. It took barely a month before talk started going around that I "don't deserve my salary" because I don't want to work overtime anymore.

Meanwhile, my favorite colleague—who is also a close friend, a few years younger than me, someone who always gave 100%, was great with clients, and genuinely liked by everyone—just left for a new job because our CEO treated him like absolute garbage. I told him to take it and never look back, and he's now at another MSP that is actually properly managed.

As for our CEO: a textbook narcissist who only cares about himself and the company's bottom line. The kind of person who can look you in the eye, appear to care, and say all the right things in the moment, only to turn around and do the exact opposite a week later.

Over the last few years, I've started taking much better care of myself. I've lost around 40 kg, going from obesity to being in decent shape, and I feel better physically than I have in 20 years. I exercise every day and get 6–8 hours of sleep.

Outside of work, I'm genuinely happy and most days are good. The second I set foot inside these walls, though, my mood and well-being nosedive. I can't concentrate, and all I can think about is walking right the fuck out and never coming back.

Unfortunately, I need to tough it out here until at least the end of January, but man, I am sick of this shit. I am kind of worried about switching jobs because of the current state of IT with LLMs and such, but with my experience and enough references to a fill a few A4 pages from important people and companies in the region it should hopefully be easy enough.

I dont know why I opted to write this, but I guess I needed to get it off my chest.


r/sysadmin • • 3h ago

Remote Code Execution in NGINX HTTP/2: CVE-2026-123633 and CVE-2026-123007

122 Upvotes

r/sysadmin • • 23h ago

How do you handle users refusing to put in Tickets

121 Upvotes

There are always exceptions to the rule or policy, but how do you handle users that circumvent ticketing just to play games ?


r/sysadmin • • 10h ago

General Discussion I feel like a fraud

114 Upvotes

I work for a small company, I don't get paid very much but honestly... I don't really do much. I patch workstations and servers, set up a new hire once every few months, but for the most part like over half my day is me acting like I'm busy or watching YouTube or something. I'm more than happy to help someone when they call or walk into my office but there's just not much to do here. Im bored. Complacent. And like 80% of the shit I do anyone else could do if they just googled it.

Edit: Thank you everyone I appreciate all the feedback, I was drinking a lot last night alone and kind of got in my head about some stuff.


r/sysadmin • • 1h ago

Is fundamental knowledge of hardware being lost?

67 Upvotes

Today I had a conversation with a mid senior sysadmin about why we cant simply jam a mix of DDR5 ram into our hosts. When he ended up being holistically wrong his response was to tell me he didn't need to know anything about the hardware running our data center.

This guy is one of our T2 hypervisor admins and all he does is troubleshoot VM issues and host issues. The idea that he wouldn't need to know anything about the underlying technology running his cluster blew my mind so I asked others. Turns out many people I work with either dont care about the upstream hardware or simply refuse to learn it.

Is this normal or because people I work with are just siloing themselves.


r/sysadmin • • 10h ago

General Discussion Java4Shell (CVE-2026-114938) is confirmed and patched. Raspberry Pi/ARM is safe, here's a 10-second check for x86 boxes

59 Upvotes

The "unconfirmed OpenJDK RCE" from yesterday is now confirmed by OpenJDK and patched by Oracle. Short version for self-hosters:

What: heap overflow in Java's hardware-accelerated Base64 decoder. Calif showed pre-auth RCE with default configs on Keycloak, Jira, Confluence, Artifactory, Jenkins and PingFederate.

You need ALL of these to be exposed:

- Java 17.0.6+, 21, 25 or 27 (any vendor: Temurin, Corretto, Zulu...). Java 11 and normal Java 8 are fine.

- x86-64 CPU with AVX-512 VBMI (Intel Ice Lake Xeon+, AMD Zen 4+). ARM / Raspberry Pi is NOT affected.

- Untrusted Base64 reaching the decoder (Basic auth, JWTs, uploads...).

Check your CPU (all 4 flags = affected):

grep -m1 '^flags' /proc/cpuinfo | tr ' ' '\n' | grep -E '^(avx512f|avx512bw|avx512vbmi|bmi2)$'

Check Java inside containers, not just the host:

docker exec -it CONTAINER java -version

Fix: Oracle builds ending in .1 (17.0.21.1, 21.0.13.1, 25.0.5.1, 27.0.1.1); other vendors are rolling out. Until then, -XX:-UseBASE64Intrinsics stopped the overflow in every tested version (needs a restart/recreate). Don't use -XX:DisableIntrinsic=_base64_decodeBlock on its own, it stops the JVM from starting.

No exploitation in the wild reported yet, and VulnCheck says full RCE needs per-app work, but crashing a JVM is easy.

Full write-up with version table: https://arelvox.com/java4shell-cve-2026-114938/


r/sysadmin • • 8h ago Bravo

Ignite 2026: Microsoft Introduces Vulnerability Shielding to Protect Unpatched Systems

56 Upvotes

There's often a gap between discovering a vulnerability and deploying its fix. During this time, affected systems remain exposed to potential attacks.

Now, you can protect your systems while waiting for the fix.

With Vulnerability Shielding in Microsoft Defender, announced at Ignite 2026, Microsoft introduces a way to protect vulnerable systems until permanent patches are deployed.

Here's what it brings:

  • Proactive protection: Microsoft Defender provides proactive protection using the Risk Reduction Agent in Microsoft Perception. It identifies vulnerabilities and recommends shielding plans for administrators to review and approve.
  • Protection during active attacks: For ongoing attacks, Microsoft Defender uses Attack Disruption to automatically apply shielding controls. These controls help stop attackers from exploiting additional paths.
  • Automatic shield removal: Shields are automatically retired once the permanent patch is fully deployed. 
  • Admin control: Admins can review the expected impact, approve protection plans, exclude devices, and roll back shields when needed. 

The feature is currently in public preview, with initial support for server-side remote code execution (RCE) vulnerabilities.

It'll be interesting to see how effectively this works in production environments, especially when balancing security restrictions with normal business operations.


r/sysadmin • • 4h ago

Project Zero has cut its disclosure deadline from 90 days to 30

53 Upvotes

That 90-day window was set based on how long it reasonably took a human researcher to go from bug to working exploit. Google is saying that assumption no longer holds, because of AI-assisted exploit development.

Vendor patch cadences, internal vulnerability SLAs and the common "we'll fix it next release" habit were all built on the assumption that 90 days was generous enough. It wasn't generous, but it was something teams could work within.

30 days is going to break a lot of patching processes that were already running close to the old deadline, and that needed to happen. Some might say that 30 days is even too long.

What do you think?


r/sysadmin • • 3h ago

RCE in Nginx with http2 enabled - CVSS 9.2

44 Upvotes

https://www.wiz.io/blog/nginx-http2-rce-vulnerabilities

CVSS 4.0 score of 9.2. The flaws are in NGINX itself. Websites and APIs can be exposed even when their own application code is secure. Both vulnerabilities require HTTP/2 to be enabled on the affected NGINX server. Additional conditions are described below. CVE-2026-123007 affects versions dating back to NGINX 1.9.5 in 2015.

I am tired. But I think none of our Nginx has http2 enabled.

On a side note, given all these recent memory related CVEs, I am seriously considering learning Rust now. I was pretty content with my Golang until now. And I can finally see what is that borrow checker every cool kids talk about all the time.


r/sysadmin • • 8h ago

UK sysadmin pay vs cloud/platform in 2026, the jump is about £30k

40 Upvotes

Quick disclosure up front: I work at Archer from hackajob, and this is from our own platform data, 4,423 UK DevOps/Cloud roles posted in 2026, client base skews finance and enterprise. Not linking anything, happy to drop the source in the comments if people want it.

Sysadmin sat around £53k, the lowest band in the set. The biggest single jump was sysadmin to Platform Engineer, about +£30k, and a lot of it builds on what sysadmins already do day to day. Cloud Engineer and DevOps Engineer landed around £75k, so even the first step up is roughly +£20k.

The skills that seemed to bridge it: a cloud platform (AWS was in ~32% of postings, Azure a strong second at ~25%) and automation/scripting, with AI/automation experience paying about +£20k over the median.

If you’re a UK sysadmin, that platform jump looks like the most reachable one. You’re probably already doing part of the job.

Anyone here made that move? Does the salary match what you're seeing where you are?


r/sysadmin • • 1h ago

General Discussion No availability for warranty replacements

36 Upvotes

Well, the ai supply chain strain finally caught up to warrantied items.
Recently had a drive failure in one of backup disk arrays. HPE is going to replace the drive, I waited for 3 days, and they responded that they have no stock.

After asking for an ETA, voicing the concern of lost resiliency. The response was.
"We have escalated the matter to our Supply Chain team, and they are actively working on it. We will share the ETA with you as soon as we receive an update."

Reaching out to our VAR about just buying a drive. There's no stock in the US. This is stupid.


r/sysadmin • • 13h ago

Does Telnet Thin Client Hardware Still Exist

29 Upvotes

I've got a server in a retail building that manages inventory, sales, etc. that every client connects to with a telnet software client on a Windows PC, except for a couple really old thin clients. It would be so much easier, safer, and cheaper if I could put an IP in a hardware thin client and plug it in to the network and be done.

I can't find any! Please tell me where to get some. (I'm In Canada)


r/sysadmin • • 17h ago

Microsoft Intune can now fully manage Windows Server (preview). Notes from Microsoft’s guide, including a couple of gotchas

26 Upvotes

Microsoft has opened a preview that lets Windows Server enrol into Intune as a proper MDM device, not just through Defender security settings management. I went through the Learn guide and pulled out what's actually in it.

Scope
- Windows Server vNext Insider builds only (29621.1000 or later), Desktop Experience. No Server Core, no 2022 or 2025
- Non-production and test tenants only. Microsoft recommends against trying it in your production Intune
- Needs a user. Userless, bulk and Autopilot enrolment aren't supported
- No extra licence during the preview. Auto-enrolment still needs Entra ID P1 or P2

What works
- Compliance policies, and compliance as a Conditional Access signal
- Settings catalog (about 60 groups) plus custom OMA-URI profiles
- Endpoint security: Antivirus, Firewall, Disk encryption, LAPS, ASR
- Win32 apps and Company Portal
- Update rings with deadlines and maintenance windows (quality, feature and driver update policies aren't supported)
- Remote actions: sync, restart, BitLocker and LAPS rotation, diagnostics, Defender scan and update, retire, wipe

What doesn't
- Firewall, antivirus and antispyware compliance checks (configuring them is fine)
- EPM, Resource Explorer, ConfigMgr co-management
- Moving servers from Defender security settings management to Intune

Gotchas

  1. Anything assigned to All devices, to users or to dynamic groups will land on servers as soon as they enrol. A separate server group doesn't exclude them. Microsoft added operatingSystemSKU filter values for the server editions, but scripts and policy sets don't support filters.
  2. If you enrol via "Add work or school account", the enrolment restriction needs Personally owned devices allowed as well as Server devices.
  3. Enrol an Azure VM and anyone with the right Intune role can manage it, including wiping it, without Azure RBAC or JIT access. Azure Update Manager and Azure Policy can also conflict with Intune on the same VM.
  4. If a CA policy requiring MFA covers servers, they can silently stop syncing until someone clicks Sync and signs in.

Microsoft's position: Intune is for "endpoint-like servers that have an associated user identity" (jump boxes, PAWs, branch and dev servers). Arc stays the recommendation for production workloads, and the two can run on the same server as long as they don't manage the same settings.

Full write-up with screenshots if it's useful: https://azureblog.co.uk/posts/intune-windows-server-preview/

Is anyone testing this yet? I'm curious how the settings catalog behaves on the Insider builds in practice.


r/sysadmin • • 3h ago

Java4Shell: how are you finding which systems are exposed and deciding what to patch first?

22 Upvotes

Java4Shell came out this week and Oracle already has a patch, but the harder part seems to be finding where vulnerable Java is actually running. Some apps ship their own copy of Java, so they need an update from that vendor rather than a regular JDK patch.

For those working through it:

  • How are you finding the affected installs, especially the bundled ones?
  • What are you patching first?
  • What are you telling management while it's still in progress?

r/sysadmin • • 10h ago

Memberof is back? For a price

22 Upvotes

Just found this

https://learn.microsoft.com/en-gb/entra/identity/users/groups-dynamic-rule-advanced

i haven’t seen any announcement. Obviously a preview feature and requires Entra ID Governance plus limited to twice a day but cool it’s back for people who never migrated away.


r/sysadmin • • 22h ago

Anyone else see their Attack Surface Reduction rule detections in Defender flatline after 11/12?

21 Upvotes

Did my morning check yesterday and today and my detections went from a few hundred a day down to single digits.

Noisy rules like "Block credential stealing from the Windows local security authority subsystem (lsass.exe)" catching run of the mill stuff like svchost and lenovo controller plugins as well as more dodgy ones like "Block all Office applications from creating child processes" seem like they've had the rug pulled out from under them.

Hoping Microsoft didn't bungle Defender in the process of bringing their infra back up.


r/sysadmin • • 4h ago

General Discussion user naming convention : how do you handle people with no first name?

17 Upvotes

looking for advice. I consult for a couple of clients in the architectural field in Canada.

They have recently been hiring people from Southeast Asia that appear to have no first name. When the hiring information is sent over to be setup as accounts, how do you handle this? some of them have accept to supply a "preferred" first names, other have refused.

these clients naming conventions require either first.last or firstinitial+lastname.


r/sysadmin • • 5h ago

Retirement Idea - Dude (Server) Farm... as opposed to Dude Ranch

17 Upvotes

In the past, it was common for people to pay a ranch to allow them to sleep there and learn how to do some of the jobs that it requires to keep a ranch running while working through their mid-life crises. City Slickers was a comical expression of this phenomenon.

As a computer dude nearing retirement it dawned on me last night that I could start a Dude Farm, but instead of livestock, I'll be teaching everyone about server maintenance, upgrade windows, proper backup and restore procedures, etc.

What could go wrong?


r/sysadmin • • 11h ago

Question Java4Shell: how are you finding the JVMs you didn't know you had?

17 Upvotes

Anyone found a decent way to inventory bundled JREs on Windows and appliances?


r/sysadmin • • 5h ago

Question Anyone else get Grok AI forced on their Copilot?

12 Upvotes

SpaceXAI is firmly set to "no users" for the tenant, yet some people have "GrokAI" as a model they can select in Copilot... idk how to get rid of it or audit it. Anyone else seeing this?


r/sysadmin • • 16h ago

General Discussion KB5132043 on ARM64 breaks apps

11 Upvotes

Saw this being talked about in another thread but thought I'd just pull it out for discussion directly on this instead of hijacking the other thread.

I'm on an ARM64 device (Surface Laptop 7).

After receiving hotpatch KB5132043 I started experiencing weird app issues like apps not running or constantly crashing (1Password, Office apps). Event viewer calls out Hotpatch-Loader and the apps exe but that's about it.

Only solution right now was to remove KB5132043 which let everything work again.

Anyone else seeing this on ARM64 devices only? Would love to know if and when this is addressed. Haven't seen anything mentioned by MS yet.


r/sysadmin • • 5h ago

General Discussion Reminder to go through your bot and service accounts, we learned this the hard way

10 Upvotes

Hey all! I work at Bitrise.

We had a security incident last week and our VP of Engineering just published our preliminary review today worth reading by everyone running CI: https://bitrise.io/blog/post/the-bitrise-security-incident-preliminary-review-and-recommended-steps#how-we-contained-it

The way in was an internal testing bot account. The attacker had its real username and password (we’re still investigating how), and it was set up to sign in to Apple’s Developer Portal and it had no 2FA. Anyone with the password could just log in. If you’ve got accounts like that, built around a third-party login where MFA is a pain, they’re worth a second look. We disabled every bot account that night and went through their credentials and 2FA settings. Every other credential got rotated too, whether or not we had evidence it was touched.

We’re back to normal now, Bitrise is safe to use. Sharing all of this information to help other teams. For more, look at our public advisory (https://bitrise.io/security-advisory-november-2026).

If you have questions about the incident and how we addressed events, we’re here to answer your questions.