r/SysAdminBlogs • • 5h ago

Terraform Modules: 5 practices that make infrastructure code easier to maintain

2 Upvotes

When working with Terraform in real infrastructure environments, writing working configuration is only one part of the job.

The bigger challenge is keeping infrastructure code reusable, maintainable, consistent, and easy for other engineers to understand.

For example, imagine multiple engineers managing VPCs, EC2 instances, and storage resources across different environments.

Without proper module design and version management, configurations can become difficult to maintain and changes can introduce unexpected issues.

Here are five practices worth following when building Terraform modules:

1. Keep modules small and focused

Separate networking, compute, and storage responsibilities instead of building one oversized module.

2. Use descriptive variable names

Avoid unclear names like v1 or x. Use meaningful names such as instance_type and ami_id.

3. Document modules properly

Include a README with usage examples, input variables, outputs, and requirements.

4. Use Git version control

Track module changes, maintain stable releases, and make rollbacks easier.

5. Manage access to shared modules

Use private repositories, access controls, and branch protection to manage infrastructure code securely.

I recently put together a practical guide covering these five Terraform module practices, with examples.

Full article:
https://datadevblog.com/terraform-modules-best-practices/

Discussion question:

For those managing Terraform in production, what causes the biggest maintenance problems in your modules?

Is it poor module design, version management, documentation, or something else?

Would love to hear your experiences.


r/SysAdminBlogs • • 10h ago

We're building a supercomputer

Thumbnail
golem.de
2 Upvotes

r/SysAdminBlogs • • 13h ago

A Linux time travel tool

Thumbnail
github.com
2 Upvotes

About a month ago I asked here if a tool to see ps, ss or lsof output at a past moment was worth building:

https://www.reddit.com/r/linuxadmin/comments/1wyhwfe/is_a_linux_time_travel_tool_worth_building/

I got some fair pushback (sar, atop, below, Prometheus, zabbix) and a few people who were curious. I looked at all of them. They cover a lot, but none of them could tell me which process had which connections open, or what lsof showed for a process at 02:45 last night. So I built it, mostly to find out if it could be done without hurting the server.

I wanted to name it "command line time travel" but then a though about the consequences of such a name and It's called clifr instead (command line flight recorder) and has two parts:

  • clifrd runs as a service and records processes, process starts and exits (through eBPF, so a process that lives 5 ms still shows up), TCP connections with the process that owns them, files held open, mounts, kernel messages and the usual system counters. It keeps 72 hours in /var/lib/clifr, 2 GB max.
  • clifr rebuilds the output of the normal commands for a past moment or a time range. It doesn't run the command, it rebuilds the output from the recording.clifr --at "02:45" "ps aux" clifr --since "02:00" --until "03:00" "ss -tanp dst 10.0.4.21 and dport = :5432" clifr --since "1 hour ago" events # starts, exits, OOM kills, mounts clifr diff --at "14:00" --at "14:20" "ss -ltnp"

There are about 40 commands so far: ps, top, pidstat, lsof, ss, netstat, free, vmstat, iostat, mpstat, df, ip, dmesg, systemctl list-units, lsblk, and more and all are 100% accurate.

The first thing people asked last time was the cost, so here are the numbers:

Quiet serverMixed loadBusy server
CPU0.35% of one core0.5-0.6% of one core1.9% of one core
RAM~90 MiB~80-120 MiB~290 MiB (335 max)
Disk~`20 MiB/day~100 MiB/day~4 GiB/day
Networknone, no ports opennonenone

Disk use depends heavily on how busy the server is.

clifr is not a monitoring tool and has no alerts. It's not a security audit tool either: it doesn't record file reads or writes, and root can stop it. It's for the "what was going on at 02:45" question, after the fact.

Status: it's a prototype, version 0.1.0. It hasn't run on a real production server yet (a friend starts one this week), and all the load has been synthetic. AGPL-3.0, no telemetry, nothing leaves the box.

So if you have a test or dev box (RHEL 8/9, Rocky, Alma), I'd really appreciate it if you tried it and told me what breaks or what looks wrong. The most useful thing for me is comparing what clifr shows with what you know was really happening.

I appreciate all your comments even if they are harsh


r/SysAdminBlogs • • 10h ago

How I Added 250GB of Storage to a Linux VPS for €4/Month Using S3 and ext4

Post image
0 Upvotes

I recently ran into a shortage of disk space on my VPS. I currently rent a VPS for €7 a month with this configuration: 4 vCPUs, 8 GB of RAM, and a 120 GB NVMe drive. It has 50 GB of free space left. But I need to run a Docker container whose image is 76 GB. I could upgrade my current VPS to get a 240 GB disk, but that would cost €30 a month. That’s steep. I noticed that my hosting provider offers 250 GB of S3 storage for just €4 a month, accessible over its internal network. So I started wondering how I could turn it into a filesystem and have Docker store files there.

First, I tried rclone. It lets you create a FUSE filesystem from your S3 storage. To speed things up, you can cache frequently used files on a local SSD using the --vfs-cache-max-size flag. Unfortunately, Docker uses overlay2, which requires filesystem operations that rclone mount does not guarantee: atomic renames, correct chmod and chown behavior, hard links, and extended attributes. In short, a full POSIX filesystem is needed.

And I found a solution: s3backer. It has 600+ stars on GitHub and provides a full-fledged ext4 filesystem. I set a 20 GB cache on my local SSD using the --blockCacheSize option, mounted it at /mnt/s3, and reconfigured Docker to store images, containers, and volumes there. As a result, the most frequently used files are cached locally instead of being pulled over the network from S3. What do you think? Expanding my virtual machine’s disk space by 250 GB for €4 a month? Pretty great, if you ask me!

By the way, here’s a summary table of the available options:

SolutionHow it worksProsCons
s3backer + ext4S3 as a block deviceFull POSIX filesystem, Docker compatibility, cachingComplex setup, latency, risk of filesystem corruption
rclone mountFUSE over S3Simple setup, local cache, support for multiple cloud providersIncomplete POSIX compatibility, some operations are unsupported
s3fsFUSE over S3Simple mounting, familiar file operationsSlow metadata operations, POSIX limitations
JuiceFSDistributed filesystem on top of S3POSIX, caching, high performanceRequires separate metadata storage
S3QLFilesystem on top of S3POSIX, compression, deduplication, encryptionExtra overhead, complex recovery

Of course, s3backer is far from the only way to turn S3 into a filesystem. There are at least two other interesting projects: JuiceFS and S3QL. Both run on top of S3, support POSIX operations and local caching, but they seemed like overkill for my use case.

JuiceFS is a fairly powerful distributed filesystem. It supports multiple servers working concurrently, caching, POSIX, and much more. But there’s a catch: in addition to S3 itself, it requires a separate metadata store. This could be Redis, PostgreSQL, MySQL, or even a local SQLite database. The last option doesn’t require a separate service, but you still have to keep the metadata database safe and back it up. Overall, it’s an interesting solution, especially if you have a server cluster. But I only have one VPS, and I didn’t want to set up extra infrastructure just to store Docker images.

S3QL is another interesting option. It supports POSIX, local caching, compression, deduplication, and encryption. It doesn’t need a separate database server: metadata is stored in SQLite and synchronized with the object store. But these extra features come at the cost of added complexity. Deduplication requires computing hashes, compression uses CPU time, and recovering the filesystem after a failure may require checking and restoring metadata with specialized tools. These aren’t critical drawbacks, but I didn’t see any particular advantages for my use case, either.

The key difference lies in the approach: JuiceFS and S3QL implement filesystems themselves using FUSE, while s3backer provides virtual block storage on top of which you can create a standard ext4 filesystem. As a result, Docker works with the familiar Linux filesystem, and s3backer translates block operations into requests to S3. No separate metadata databases, no deduplication, and no extra layers of file management.

Of course, this setup has its downsides too. Random reads and writes can be heavily affected by S3 latency, and losing the connection to the storage can bring the containers to a halt. A serious failure could also corrupt the filesystem. So I wouldn’t use this setup for critical databases. But for storing large Docker images that are downloaded once and then mostly read, it seemed perfectly reasonable-especially when S3 is available over the hosting provider’s local network. In short, I wanted an extra 250 GB for €4 a month without building a distributed storage system. From that perspective, s3backer seemed like the best fit.


r/SysAdminBlogs • • 12h ago

Every service desk demo this quarter was AI, so we built a test from our own queue

1 Upvotes

Four service desk demos in six weeks and every one led with AI. We're about 3,500 people across three regions, so leadership wants a number before anything gets budget.

The one with the boldest claim was Atera's Robin, pitched as an AI technician that resolves up to 92% of tier 1 and complex tier 2 incidents. What sets it apart from the others is that it runs the fix itself, checks the result, and closes the ticket with notes on what it did. Hardware still comes back to us, and the approval rules for what it can touch are ours to write.

So we picked three categories from last quarter, password lockouts, print queues, and VPN client errors, and set one rule. A ticket only counts as resolved if no tech touched it and it wasn't reopened. That's the test every vendor gets on our data.


r/SysAdminBlogs • • 23h ago

Entra Privilege Identity Management (PIM) - Role activation and more now possible at MyAccess

Thumbnail
blog.sonnes.cloud
7 Upvotes

From a feature request to Microsoft Ignite announcement – almost a year in the making!

Back in November 2025, I sent a feature suggestion directly to the Microsoft Entra engineering team: Why not bring Privileged Identity Management (PIM) role activation into the My Access portal?

An idea partly inspired by my findings around the hidden PIM URLs and undocumented UI experiences:

https://blog.sonnes.cloud/the-hidden-microsoft-entra-pim-urls-microsoft-never-documented-even-when-restrict-access-to-microsoft-entra-admin-center-is-enabled/

Fast forward almost a year… and here we are! 😎

I've had the opportunity to participate in the Private Preview, test the capabilities, share feedback and work closely with the team. Now it's officially announced in the Microsoft Ignite 2026 Book of News!

In my latest blog post, I share the story, hands-on testing, technical findings and a complete walkthrough, covering:
> PIM activation for Entra ID, Azure RBAC and Groups
> Just-in-Time (JIT) privileged access
> Remote Resource Access, Azure Bastion and Cloud Shell
> Requirements, limitations and findings not previously covered!

It's incredibly rewarding to see an idea evolve into an actual Microsoft product capability.

Sometimes, it's worth sending that email. You never know where it might lead! 😉

#Microsoft #EntraID #PIM #IdentitySecurity #MSIgnite #MicrosoftMVP #MVPBuzz #Community


r/SysAdminBlogs • • 21h ago

Global Secure Access - Device Access is here!

Thumbnail
blog.sonnes.cloud
4 Upvotes

Another Global Secure Access update - Device Access is here!

Bringing pre-logon connectivity back to Windows devices using device identity instead of user identity like we know from the old days!

Think on password changes, first-time sign-ins, domain controller connectivity, Group Policy processing and on-premises device management - all before a user signs in to Windows!

I've been following and testing this feature since its earlier preview in June 2026, and it's great to finally see it reach Public Preview at Microsoft Ignite.

In my latest blog post, I cover a lot - check it out!

#Microsoft #EntraID #GlobalSecureAccess #GSA #ZeroTrust #MSecurity #MSIgnite #MVP #MVPBuzz


r/SysAdminBlogs • • 1d ago

Docker bypasses UFW: why it happens and how to fix it for good

2 Upvotes

UFW was set to allow only SSH, and a container published on 8080 was still reachable from the Internet. If you run Docker on a server with UFW, some of your containers may be more exposed than you think, so I went through the whole thing on a clean server.

What I found:

  • The DOCKER-USER rule with --dport 8080 that many guides show does nothing. DNAT has already rewritten the port to the container's port by the time the packet gets there. You need to match the original port with conntrack instead (the exact rule is in the post).
  • Rules added with the iptables command are gone after a reboot. Putting them in /etc/ufw/after.rules keeps them.
  • On the default bridge, IPv6 goes through docker-proxy, so UFW does apply there.
  • For anything that doesn't need to be public, binding to 127.0.0.1:8080:80 is still the simplest fix.

Full write-up with every command and output: https://staunch.sh/blog/docker-ufw-debian-13/


r/SysAdminBlogs • • 1d ago

GitHub Backup Tool - update 1.4.0.1 is out

Thumbnail
blog.sonnes.cloud
2 Upvotes

No failure email doesn’t mean your backup is succeeded! 😅

I discovered my GitHub backup was failing without sending an email notification, as if a fun error from a GitHub org. im in

That got me to update mt GitHub Backup Tool to 1.4.0.1, bringing the following fixes:

> Repository owner filtering with -owner
> Discovery and branch enumeration that respect your filters
> Better error logging and a fix for missing failure emails when configured
and more...

And this is a strong reminder to check your backups - and test your restores!

Read more in my blog post here with why and the fix, and update your tool also - everyone has already downloaded my tool here from my GitHub over 29k times - a HUGE thanks you all! ❤️

#Microsoft #GitHub #Backup #OpenSource #CSharp #MVP #MVPBuzz #Community


r/SysAdminBlogs • • 23h ago

Microsoft vs. Facebook outages: how do you test whether your recovery path is independent?

1 Upvotes

Microsoft’s November 13 incident report describes a network change that exposed a fault from an earlier capacity upgrade. In Facebook’s 2021 outage, a backbone failure led to the withdrawal of routes to its authoritative DNS servers, and engineers had trouble accessing recovery systems. Different failures, same question about shared dependencies.

If your primary network became unreachable, could you still resolve the names you need, authenticate, reach your management tools, and roll back the change? For those of you who’ve tested this end to end, what dependency surprised you?

I wrote up the comparison here: https://lippi.hu/microsoft-vs-facebook-leallas-en


r/SysAdminBlogs • • 1d ago

Logon Duration vs. Analyze Logon Duration: different numbers, different measurements

Thumbnail reddit.com
1 Upvotes

r/SysAdminBlogs • • 1d ago

Automating network discovery and remediation, what's your stack..

1 Upvotes

Trying to automate device discovery across a few sites and have scripts fire automatically when alerts hit. Some go Ansible plus NetBox, others use RMM platforms like Atera or NinjaOne that cover discovery and scripting. Leaning toward a platform, not sure yet how much of it works on our Linux boxes.


r/SysAdminBlogs • • 1d ago

Entra ID - PIM for App roles is here!

Thumbnail
gallery
1 Upvotes

In case you haven't spotted it yet in your Entra ID Portal this morning, but you can now use Privileged Identity Management (PIM) for App Roles on your applications!

With this new feature, you can manage your App roles on your applications like the same way you already handle Entra and Azure roles. It can be made eligible and activated just-in-time.

Keep in mind that is in Public Preview - read all in details in my blog post here, as I was a part of the private preview too.

Check out the full guide to use it and how it looks here: https://blog.sonnes.cloud/entra-id-pim-for-app-roles/

N.B: Requires an E7/ID Governance license!

#Microsoft #EntraID #PIM #IdentityGovernance #MSIgnite #AppRoles #MVP #MVPBuzz #Commuity


r/SysAdminBlogs • • 1d ago

The Hidden Cost of Protocol Fragmentation: Why Enterprise Java Needs Its "JDBC Moment" for File Access

Post image
0 Upvotes

r/SysAdminBlogs • • 1d ago

Linux Root Filesystem 100% Full Overnight: How to Find and Fix the Cause(Interview Scenario 2)

Thumbnail
datadevblog.com
1 Upvotes

Yesterday, everything was working normally. Root filesystem utilization was around 65%.

This morning, you receive an alert:

/ filesystem utilization: 100%

Applications are reporting errors, users are experiencing issues, and you need to investigate before the situation gets worse.

Where would you start?

In this troubleshooting scenario, I walk through the investigation step by step:

  • Verify filesystem and inode utilization using df -h and df -i.
  • Identify which directories are consuming the most disk space using du.
  • Drill down to find unusually large files.
  • Investigate deleted-but-open files using lsof +L1.
  • Understand when to truncate an active log file instead of deleting it.
  • Configure logrotate to prevent recurring disk-space issues.

I have also included example command outputs, possible root causes, common troubleshooting mistakes, and follow-up Linux interview questions.

One interesting question covered in the article:

What would you investigate if df -h reports 100% disk usage, but du shows significantly less space being used?

I'm building a series of practical Linux troubleshooting scenarios focused on production support and interview preparation.


r/SysAdminBlogs • • 2d ago

GSAClientDiagnosticsToolkit v1.0.1 is here: small tweaks, new health checks and a few fixes

Thumbnail
gallery
7 Upvotes

GSAClientDiagnosticsToolkit v1.0.1 is out!

Just a few days after introducing my PowerShell diagnostics module for Microsoft Global Secure Access (GSA) to the community, the first update is here - and it's based straight on your feedback!

What's new:

  • 3 new checks: Hyper-V external virtual switch detection, Group Policy NRPT conflict detection (with automatic SYSVOL source-GPO lookup), and GSA Edge Diagnostics status (reading the client's own cached per-channel health result)
  • 2 smarter checks: the auth certificate check now cross-checks the exact CertCommonName the client expects, and the Secure DNS check now also covers Firefox
  • 2 real fixes: a default log path crash on non-writable folders, and a false failure in the Private DNS resolver check for tenants only using Enterprise Applications.

And a big thank you to everyone who tested, reported issues and gave input for this release 🙏

Get it:
🔗 PowerShell Gallery: https://www.powershellgallery.com/packages/GSAClientDiagnosticsToolkit
🔗 GitHub Releases: https://github.com/michaelmsonne/GSAClientDiagnosticsToolkit/releases
🔗 Full write-up with all the details: https://blog.sonnes.cloud/global-secure-access-client-diagnostics-toolkit-1-0-1-is-here/

Open source, free to use, and built from real troubleshooting scenarios - keep the feedback coming!

#Microsoft #PowerShell #Community #GlobalSecureAccess #GSA #Entra #Identity #Security #ZTNA #MVP #MVPBuzz


r/SysAdminBlogs • • 2d ago

Onboarding VDC Vault Customers End to End with PowerShell and the VSPC API

Thumbnail
bcthomas.com
4 Upvotes

I've just finished the third and last post in a series on automating Veeam Data Cloud Vault through the VSPC API.

The first two posts went through the individual API calls. Part 1 covered creating the VDC tenant, mapping it to a company and creating the vault. Part 2 covered registering VBR, assigning the vault, and creating the Vault repository. Doing that through the UI is fine for a couple of customers, but it gets repetitive quickly.

This post wraps those calls into a set of PowerShell functions, plus Invoke-VspcVaultOnboarding, which takes a CSV of customers and runs the whole chain for each row. It supports -WhatIf, and it's safe to rerun, so a failed row can be fixed and rerun without duplicating anything. I've been running it against VSPC 9.3 and VBR 13.1 in my lab. The post also covers a few quirks I hit along the way, like the folder browse failing straight after a new vault assignment.

Post: https://bcthomas.com/2026/11/onboarding-vdc-vault-customers-end-to-end-with-powershell-and-the-vspc-api/

Code: https://github.com/comnam90/bcthomas.com-scripts/tree/master/Powershell/Functions/VspcVault

Happy to answer questions if you're working with the VSPC API.


r/SysAdminBlogs • • 2d ago

A monthly roundup of IBM i resources worth bookmarking

1 Upvotes

A good IBM i upgrade guide is easy to bookmark and just as easy to forget about until you need it. Same goes for the webinar you meant to watch or the firmware update you wanted to look into.

Once a month, we gather those kinds of resources in an email we call the Relion Recap. We include things you can use on the job, like PTF and firmware updates, technical articles, and webinar recordings. There’s no sales material in it.

What’s one IBM i resource you’ve actually used lately that you’d pass along to someone else? If you have a link, drop it in the comments so others can use it too.

Get the monthly roundup


r/SysAdminBlogs • • 2d ago

Standardizing Cisco UC DRS backup configuration across separate managed networks with PowerShell

3 Upvotes

Configuring a Cisco UC backup schedule is familiar work. The repetition gets tedious when you manage separate networks, each with its own local SFTP server (Windows -- OpenSSH), publisher addresses, credentials, and cluster topology.

I put together a PowerShell workflow to make that process more consistent. It uses reusable site profiles and includes:

  • Windows OpenSSH/SFTP setup with preview and explicit apply.
  • DRS configuration workflows for CUCM, Unity Connection, CER, and UCCX.
  • Node and feature registration checks, including CUCM environments with IM&P.
  • Optional Windows Credential Manager storage and per-run JSONL logs.

I wrote up my use case (MSP/Separated UC Deployments), the manual comparison, and the operating limits on my blog. The post includes the script, README, example inputs, and expandable code previews.

v1.6.1 is a testing candidate. Offline checks have passed; native Windows and live appliance qualification remain pending. The script configures backup destinations and schedules—it does not establish that a backup or restore will succeed.

Development and review included AI assistance to translate python scripts (deprecated) into an easy to use PowerShell script.

Full write-up and downloads on NOC Thoughts - https://nocthoughts.com/2026/11/14/standardizing-cisco-uc-backup-configuration-with-powershell.html


r/SysAdminBlogs • • 3d ago

Dev and Ops Facing AI: an asymmetric transformation

1 Upvotes

I wrote a short essay on how AI agents affect the nature of Dev and Ops work differently:

- for Devs, they change a core activity (writing the code)
- for Ops, they mostly add another layer to the automation and orchestration already in place.

By Ops I mean infrastructure and reliability work, closer to SRE or platform engineering than traditional sysadmin.

I’m currently Ops, previously Dev, and I’d like to hear how this matches what you see in practice, thanks !

https://cl3don.substack.com/p/dev-and-ops-facing-ai


r/SysAdminBlogs • • 4d ago

Script to collect the evidence for your M365 outage SLA claim

90 Upvotes

According to Service Health, EX1550543 ran 30 hours, which puts November at 95.8% uptime. That’s the 50% tier, but only on the Exchange Online share of your license, roughly $2 per user.

Microsoft won’t apply it automatically, and claims without the impact window and affected users per location get rejected.

The article covers how to file the claim directly or through a CSP, what Microsoft needs, and includes a script that collects the evidence and writes a claim summary. The deadline for the claim is December 31.

https://lazyadmin.nl/office-365/claim-microsoft-365-sla-credit/


r/SysAdminBlogs • • 4d ago

Linux High Load Average but Low CPU? A Practical NFS Troubleshooting Scenario

0 Upvotes

Hi everyone!

I've published a Linux troubleshooting walkthrough covering a scenario that can be confusing during production incidents.

The problem:

A Linux server has:

  • Increasing load average
  • Low CPU utilization
  • Slow application response
  • Multiple processes stuck in D state
  • An application dependent on an NFS mount

The article walks through the investigation, including:

  • Checking load average and CPU activity using uptime and vmstat
  • Identifying blocked processes and checking kernel wait channels
  • Investigating NFS mounts and network connectivity
  • Finding the root cause
  • Applying the resolution and discussing preventive measures

I have also included related interview questions at the end.

Read the complete walkthrough:

https://datadevblog.com/linux-high-load-average-low-cpu-troubleshooting/

I'd appreciate feedback from fellow sysadmins. If you've encountered a similar issue, I'd be interested to know how you diagnosed it and what tools helped you.


r/SysAdminBlogs • • 4d ago

Registering VBR and Provisioning Vault Repositories with the VSPC 9.2 API

Thumbnail
bcthomas.com
2 Upvotes

If you're onboarding customers to VDC Vault through VSPC, I've written up the second half of the API flow.

Part 1 covered creating the Vault tenant and storage vault through the VSPC 9.2 API. This one picks up from there. It registers the customer's VBR server with VDC, assigns the vault to it, creates the folder, and creates the repository inside VBR. There are curl and PowerShell examples for each call.

All of these endpoints are still in Preview Operations, so there are rough edges. A few that caught me out:

- Every POST comes back as a 202, so you have to poll the async action, and the result is only kept for about 5 minutes.

- A folder creation reported failed, but the folder had been created anyway.

- Lots of errors that aren't server faults come back as HTTP 520.

- A VBR 12.3 server can still be registered and have vaults assigned. It only fails when you browse folders, and the error doesn't mention the version.

https://bcthomas.com/2026/11/registering-vbr-and-provisioning-vault-repositories-with-the-vspc-9.2-api/


r/SysAdminBlogs • • 4d ago

No syslog during the outage = a stalled support case. I built a free syslog server to fix that part

3 Upvotes

I work escalations for network gear. The cases that stall the longest usually look the same: the outage happened, the device rebooted or rolled its buffer, and there's no syslog from the window that matters. Either nobody had a syslog server, or the one they had dropped messages and nobody knew. At that point the best anyone can do is "set up logging and wait for it to happen again", and the case just sits.

A lot of the time it's not that people don't know syslog. It's that the box available is a Windows server or a laptop, and standing up something reliable there takes longer than it should. Woodshed is my answer to that: MSI, pkg, deb, rpm or a portable zip. Listens on UDP and TCP 514 straight away. Plain text files per device per day, a disk budget it won't exceed, drop/refused counters shown in the UI, and an export zip with a manifest, SHA-256s and the gaps when it wasn't running, so the person on the other end of the case can trust it.

Most of the code was written with AI assistance. I did the design and ran the lab testing; the security review was a separate AI reviewer with read-only access to the code, and I went through every finding, decided the fixes and had them re-tested.

All the testing so far is lab traffic generated by my own sender (RFC 5424, RFC 3164 and bare lines, 8 source IPs). No real devices yet beyond a friend's homelab, which is in progress. It doesn't depend on any vendor's format and always keeps the raw line exactly as received, so any device that speaks syslog should work, but I haven't proven that.

What it's been through:

- release bench every tag: 10 min at 10k msg/s UDP on a 4 vCPU GitHub runner, 6,000,000 sent / 6,000,000 stored / 0 missing

- Ubuntu 24.04 (deb) and AlmaLinux 9 (rpm) VMs, 8 senders over a real network: 0 loss to 40k msg/s TCP at ~1/3 of a core and <50 MiB RSS. UDP loss above that was before the VM's NIC (interface counters), every datagram that arrived was stored

- reboot mid-stream: service back on its own, 2 UDP messages lost, export shows the 17-19 s gap as "Woodshed wasn't running"

- 24 h soak at 3,000 msg/s on the Linux VMs, 259.2M messages each: TCP exactly once, UDP lost 3-4 per million, all of it upstream of the VM

- security review before the first RC: 0 critical, 1 high (memory exhaustion from spoofed sources), 7 medium. fixed and re-tested twice

- CI install/upgrade/uninstall on Windows, macOS and Linux, x64 and ARM. checksums + `gh attestation verify` on releases

Limits, plainly:

- unsigned installers for now. SmartScreen ("Windows protected your PC", More info > Run anyway) and Gatekeeper (Privacy & Security > Open Anyway) ask once

- 1.0.0

- use the sender allowlist on untrusted segments. in my lab an open 514 on a shared /16 collected a stranger's syslog within minutes

- not a SIEM, no TLS syslog/alerts/forwarding yet

- Windows and macOS don't expose kernel UDP drops, the UI says so instead of showing 0

What I'd like to know:

  1. Which devices or formats break the parser? Redacted raw line please.
  2. What would you need to see before you'd drop this at a customer site during an outage?
  3. Windows folks, especially with endpoint protection: did it get flagged, did the firewall rules behave?

Repo: https://github.com/gmccollu/woodshed

Site: https://gmccollu.github.io/woodshed/

Releases: https://github.com/gmccollu/woodshed/releases