r/kubernetes • • 17d ago

Periodic Monthly: Who is hiring?

23 Upvotes

This monthly post can be used to share Kubernetes-related job openings within your company. Please include:

  • Name of the company
  • Location requirements (or lack thereof)
  • At least one of: a link to a job posting/application page or contact details

If you are interested in a job, please contact the poster directly.

Common reasons for comment removal:

  • Not meeting the above requirements
  • Recruiter post / recruiter listings
  • Negative, inflammatory, or abrasive tone

r/kubernetes • • 8h ago

Periodic Weekly: Show off your new tools and projects thread

1 Upvotes

Share any new Kubernetes tools, UIs, or related projects!


r/kubernetes • • 7h ago

I built a game to learn kubernetes - Night Shift

Thumbnail play.snowops.net
32 Upvotes

I have been working on multiple projects and one thing I felt was the biggest challenge for adoption. Setup time. If something takes up more than 5 minutes, you lose 90% of your potential users.

For this, i created a browser game which helps you learn real world kubernetes incidents within your browser. It's free and no setup required.

All the results are from logs of my another project - Snowops Labs. So, everything you will get is real.

It teaches you how the real engineering team works and gives you real incidents to solve and learn.

I'd genuinely like to hear what experienced engineers think. Especially if you've spent time troubleshooting production incidents.

TL;DR: An attempt to make practical DevOps troubleshooting more interactive and game-like. Give it a try and tell me what you think.


r/kubernetes • • 3h ago

Pod ran out of PIDs, every subsystem failed with a different error, and the liveness probe kept passing

6 Upvotes

Writing this up because the symptoms looked like four unrelated outages and the actual cause was one number.

We run a service in a pod that shells out a lot (az, kubectl, some bash pipelines). One morning it started throwing errors from everywhere at once:

  • DB client: connection reset by peer
  • shell runner: fork/exec /usr/bin/bash: resource temporarily unavailable
  • a fetcher: “config not initialized”
  • a cloud integration marked INVALID with az login failed and 500+ retries behind it

Different errors, different components, same minute. And the liveness probe was green the whole time, so nothing restarted it.

resource temporarily unavailable is EAGAIN. On fork it means the kernel wouldn’t give the container another PID. The pod had hit its pids cgroup limit. Everything that needed to spawn a process was failing, each in its own words. The health endpoint is a plain HTTP handler that never forks, so it answered 200 right through the outage.

What was eating the PIDs: subprocesses that never exited. Some execs had no timeout at all. The one that did have a timeout killed the bash -c wrapper on expiry, but not the az and kubectl it had started, which got reparented and kept running. kubectl in particular was hanging forever on auth for clusters where our service principal was missing a role. A few leaked per scan, a scan every hour, for weeks.

How we confirmed it from inside the pod:

cat /sys/fs/cgroup/pids.current /sys/fs/cgroup/pids.max # cgroup v2

cat /sys/fs/cgroup/pids/pids.current /sys/fs/cgroup/pids/pids.max # cgroup v1

ps -eo pid,etime,cmd | sort -k2 -r | head # oldest processes first

current was at max, and the oldest entries were weeks-old kubectl processes.

What we changed:

  1. Every exec gets a context with a deadline, and cancellation kills the whole process group, not just the direct child (in Go: Setpgid on the child, signal the negative pid). This was the actual bug.
  2. Explicit podPidsLimit on the kubelet (--pod-max-pids), low enough that a leaking pod dies before the node does. We were on the default, which is high enough to take the node down first.
  3. Alert on container_processes from cAdvisor trending up against the limit. It had been climbing in a straight line for weeks and nobody was looking at it.
  4. Considering a liveness probe that actually forks something, since an HTTP 200 told us nothing here.

The lesson I keep coming back to: when several unrelated things break in the same minute, stop reading the individual errors and look for what they share. Here it was the kernel saying no to fork.

Curious what people use for the liveness side. Is “probe that execs something” a thing, or do you just alert on the pids metric and leave liveness alone?


r/kubernetes • • 1d ago

Moved my homelab to Argo CD

Post image
412 Upvotes

My homelab used to run on helm installs from my laptop, and the repo kept drifting from the cluster.

Now Argo CD syncs everything from Git. If you change something directly, it gets reverted. Nothing was reinstalled, it took over what was running.

A few things stay outside on purpose: Cilium, since Argo CD runs on its network, and Secrets, since the repo is public.

Repo: https://github.com/prateekjaindev/homelab

Detailed write-up: https://blog.prateekjain.dev/gitops-for-a-3-node-talos-kubernetes-homelab-with-argo-cd-336c076e93cc?source=friends_link&sk=ebda84ecdc3e2458fead2638374be51e


r/kubernetes • • 1d ago

EKS 1.31 extended support ends Nov 26, then AWS auto-upgrades the control plane. 1.34 standard support ends Dec 2

8 Upvotes

Disclosure: I'm the founder of Expiry Radar, a commercial tool that tracks dates like these. Everything below works only with aws cli.

Two EKS dates coming up:

- 1.31: extended support ends Nov 26, 2026. After that AWS upgrades the control plane to the next version on its own schedule, so add-ons and node groups that aren't ready can break.

- 1.34: standard support ends Dec 2, 2026. From then on the cluster bills extended support, $0.60 per cluster-hour instead of $0.10 (about $365/month more per cluster).

Check your clusters and what blocks the upgrade (per region):

for c in $(aws eks list-clusters --query 'clusters[]' --output text); do echo "$c $(aws eks describe-cluster --name "$c" --query cluster.version --output text)"; done

aws eks list-insights --cluster-name <name>

aws eks describe-cluster-versions --query 'clusterVersions[].[clusterVersion,endOfStandardSupportDate,endOfExtendedSupportDate]' --output table

`list-insights` is the useful one: it lists deprecated APIs still in use and add-on or node version skew before you upgrade.

Source for the dates: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

How are you handling upgrade from 1.31 to 1.32, in place or blue/green clusters?


r/kubernetes • • 1d ago

Kubernetes Podcast episode 273: Pod Autoscaling and Resource Management, with Adrian Moisey and Jakub Pawełczak

Post image
6 Upvotes

Check out our latest episode: Pod Autoscaling and Resource Management, with Adrian Moisey and Jakub Pawełczak!

https://kubernetespodcast.com/episode/273-pod-autoscaling/


r/kubernetes • • 1d ago

Catching a dependency that phones home, with Falco rules scoped to the pods of one change

2 Upvotes

Hi all, just published this tutorial on scoping Falco rules to the pods of one change, so you catch things like a dependency that phones home before merge. Thought it might be useful if you already run Falco in production.

Some problems only show in what code does at runtime, like a new dependency opening a connection to the internet or a handler that shells out, and neither review nor tests look for them. Scoping the rules to one change's pods means only that change alerts. Tuning notes, like the outbound macro and sidecars, are in the write-up. The Falco side works with any pod label or per-PR namespace. Signadot runs just the changed service next to shared ones so you don't deploy the whole stack per PR.

Hope it's useful, and happy to answer any questions. Full transparency, I work for Signadot.

https://www.signadot.com/docs/tutorials/falco-runtime-security


r/kubernetes • • 1d ago

automated WAF

5 Upvotes

What is the best automated WAF (ML/AI). Currently on OSS Traefik - the options are limited to Coraza wasm plugin.

I am looking into open-appsec with Istio.

Please share experience and recommendations.


r/kubernetes • • 1d ago

Periodic Weekly: Questions and advice

5 Upvotes

Have any questions about Kubernetes, related tooling, or how to adopt or use Kubernetes? Ask away!


r/kubernetes • • 2d ago

How are you handling multi-cluster Kubernetes networking and traffic routing in a hybrid cloud environment?

48 Upvotes

Hey folks,

We're running multiple Kubernetes clusters across on-prem (OpenShift) and GCP (GKE), and I'm curious how others are handling networking and traffic routing in a similar setup.

Some of our services run in multiple clusters, while others are only deployed in specific clusters.

A few things I'm wondering about:

  1. How do you handle ingress and route traffic to the right cluster?
  2. How do services communicate across clusters?
  3. Are you using global load balancers, API gateways, service mesh, or something else?
  4. How do you handle failover when one cluster goes down?

Would love to hear what your setup looks like, what tools you're using, and any pain points you've run into.


r/kubernetes • • 1d ago

Do we have enough hands-on practice for real-world DevOps incidents?

21 Upvotes

We have no shortage of Kubernetes tutorials, cloud certifications, courses, and documentation. You can learn almost any DevOps tool online today.

But I'm curious about something: how do we get better at troubleshooting when things don't go as expected?

In a real engineering environment, these situations can take hours to diagnose. Yet most learning resources tend to focus on setting things up correctly rather than dealing with things when they go wrong.

I'm curious how other engineers approach this.

Do you create your own broken environments to practice? Do you rely on production incidents and experience? Or do you feel there's a gap in the way we currently learn troubleshooting?

I've been exploring this space and working on an open-source project called SnowOps Labs, which provides hands-on scenarios where engineers can reproduce failures, investigate incidents, and practice fixing them.

I'd particularly like some outside perspectives on the concept and the implementation.

Project: https://github.com/snowops-oss/snowopslabs

A few questions I'd love to hear your thoughts on:

  1. Is this a problem worth solving, or am I overestimating the gap?
  2. What would make a hands-on incident simulation platform genuinely useful?
  3. Would you use something like this for personal learning, interview preparation, or team training?
  4. What existing tools or platforms do you think already solve this problem well?

Feel free to be critical. I'd much rather hear why this isn't useful than get generic encouragement.

Interested to hear how others think about this.

Not looking for stars or promotion. Just genuinely interested in hearing what other engineers think, including criticism.


r/kubernetes • • 2d ago

IBM core-dump-handler removed?

1 Upvotes

Did IBM remove core-dump-handler? The GitHub page is returning a 404: https://github.com/IBM/core-dump-handler


r/kubernetes • • 2d ago

How do you keep Kubernetes rightsizing and autoscaling configs from becoming a permanent manual job?

19 Upvotes

When running different types of clusters (prod, staging, dev), how do you keep resource allocation and autoscaling settings aligned with workload requirements as they change, to avoid unnecessary increases in the cloud bill?

Over the lifetime of a product, new services get added in a cluster, traffic patterns would change, requirements evolve, etc. CPU/memory requests could get padded, limits and replica counts can become outdated, and the node types originally chosen may no longer fit the current workload mix particularly well.

I'm particularly interested in continuous pod rightsizing alongside tools like Karpenter and KEDA. As a simple example, cpu/mem requests could remain significantly higher than workloads actually need over time. This usually leaves the cluster running far more nodes than necessary (even for smaller workloads with a few nodes, even 1 or 2 of overprovisioned and unutilized nodes could cause a significant dent in the final bill). This can be overprovisioned intentionally, but over time it does become a bottleneck in a sense that there's not enough time to address everything.

How do you keep those settings appropriate as workloads evolve? How are you handling this in practice, and lately with AI?

- Do you review the full cluster regularly, leave it to application teams to right size (where applicable), or just accept the overprovisioned capacity?

- How do you distinguish necessary headroom from avoidable waste?

- Are you using any tools that recommend specific changes and explain the expected savings and risks, ideally across the major clouds? I know this can get very workload-specific, but still would like to hear general thoughts.

- Does anyone use automated tools that continuously adjust pod resources, scaling settings, or node selection and placement? Can you share your experience with such tools? Does any of these run internally within own clusters, or do they all offer managed control plane?

Thanks


r/kubernetes • • 2d ago

Running ZAP as an in-cluster Job against one microservice change instead of shared staging

0 Upvotes

Hey folks, we just published a tutorial on running ZAP as an in-cluster Job against one microservice change. Might be helpful if your security scans only ever see shared staging.

ZAP's official image runs as a Job and scans only the changed service, so a finding belongs to that change rather than to whatever else shipped to staging. The scan itself works against any in-cluster URL. Signadot is what runs the changed service next to the shared ones without copying the stack. The tradeoffs, like real attack payloads reaching shared databases, are covered in the write-up.

Hope it helps. Happy to answer any questions. Full transparency, I work for Signadot.

https://www.signadot.com/docs/tutorials/zap-security-testing


r/kubernetes • • 1d ago

So who else came home from Kubecon with the flu

0 Upvotes

Least favorite thing I brought home. Feelsbadman.


r/kubernetes • • 3d ago

Hows everyone handling secret zero outside kubernetes?

41 Upvotes

vault or openbao handing out short lived postgres logins is ok and nothing sits in a k8s secret anymore. but the app still has to log into vault first

inside the cluster thats mostly solved, vault checks the pods service account token against the TokenReview api and theres even a patch on pgsql-hackers so postgres cert auth can match spiffe ids. outside kubernetes its murkier. cloud instance identity covers vms on aws or gcp, spiffe setups like spire or hush security go wider and a lot of the rest ends up with an approle secret id baked in somewhere which is the same problem one step back

whats ideal to use for that first login outside the cluster?


r/kubernetes • • 3d ago

How do you handle prod promotion with Argo CD + Image Updater when CI is triggered by a git tag?

Thumbnail reddit.com
3 Upvotes

r/kubernetes • • 3d ago

Anyone running kubernetes-sigs/agent-sandbox in prod or lab for coding agents? Looking for real-world feedback on isolation, warm pools & latency

29 Upvotes

I’m currently evaluating kubernetes-sigs/agent-sandbox to run autonomous coding agents (like Claude Code, custom harness ,etc) on our own Kubernetes cluster, and I’d love to get feedback from anyone who has deployed it beyond a quick local test.

​Context & Current Setup:

  • ​Cluster: AWS EKS
  • ​Workload: Ephemeral dev/test environments where agents clone repos, run builds/tests, and execute arbitrary shell commands via MCP or tool calls
  • ​Isolation: Evaluating gVisor vs. Kata Containers vs recently introduced AWS nitro sandboxes

​Questions for folks actually running this:

  1. ​Cold-start latency & SandboxWarmPool: What kind of pod startup and readiness times are you seeing in practice? Are warm pools reliable enough to make interactive/agentic loops feel snappy, or are you seeing churn issues?
  2. ​Runtime performance hit: If you’re pairing agent-sandbox with gVisor or Kata, how noticeable is the performance hits
  3. ​Egress & network isolation: How are you locking down outbound network access (e.g., allowing git clone and package registries while blocking internal VPC/metadata endpoints)?
  4. ​Workspace caching: Are you doing anything clever to cache dependencies across ephemeral sandbox runs without compromising isolation between sessions?
  5. ​Hidden gotchas: Any rough edges around CRD lifecycle cleanup, orphaned PVCs/volumes, or resource quotas when an agent loop goes rogue?

​If you looked at agent-sandbox and decided to go a different route—like self-hosting E2B, rolling custom Firecracker microVMs, or just using hardened Kubernetes Jobs—I’d also love to hear what pushed you away from it.


r/kubernetes • • 4d ago

Kubefactory - Container Management Gaming Interface

Post image
515 Upvotes

A first-person factory game that views and manages your Kubernetes clusters and Docker containers.

In-browser (sandbox) demo: https://dascondor.github.io/kubefactory/
GitHub Repo: https://github.com/DasCondor/kubefactory

Edit: Thank you all for the support & feedback, if I haven't answered your question in the comments I am sorry - I will try and get with you over this coming week.

Different versions of Kubefactory 1.0.1-release vs 2.0-beta & Coming 2.1;

1.0.1 — Original release

  • Windows desktop
  • Browser demo
  • Mobile/tablet access through the browser

2.0.0-beta.2 — Available now

  • Windows, Linux, and macOS builds
  • Browser demo with phone and touch support
  • Native Kubernetes and Docker connectivity
  • Model library, LiveView, storage inspection, Krew, Porter, Bob, Gauge, and Foreman
  • VPN registry, exit-IP checks, direct pipes, VPN sidecars, and gateway views
  • Service pipelines and Jellyfin + VPN media stack
  • Text editor with Standard, Vim, and Nano modes
  • High-graphics browser fix, updated docs, screenshots, tests, and release attestations
  • Windows signing workflow wired in, pending activation

2.1 — Coming next

Further polish, fixes, testing, and features building on the 2.0 beta.

Note: beta.2 is a pre-release, and its packages are not yet code-signed.


r/kubernetes • • 3d ago

Prune=false on every Namespace with Argo CD auto-prune, is this what people actually do?

2 Upvotes

Sorry first, english is not my first language and I write this with a translator.

I never worked in a big SRE or platform team, so I want to check if what I did is normal. I run a small k3s cluster at home, plus EKS at work and a small AKS for a client. A read-only audit tool I made for my own clusters (kubegap) flagged that my Argo CD apps with automated prune also track their own Namespace.

So I tested it with throwaway apps on Argo CD 3.2.0. I removed namespace.yaml from kustomization.yaml and pushed. 2 min 48 s later the Namespace was pruned, and a Secret I had created by hand inside it was gone too.

Removing an app's directory under an ApplicationSet git generator is a different path. The Application itself gets deleted and the finalizer cascades. Prune=false does nothing there, Delete=false does.

What I changed: every Namespace manifest now has the sync-options annotation Prune=false,Delete=false (8 of them). The istio app owns 23 CRDs, so I turned automated prune off for that one. And I added an alert for apps that stay OutOfSync for 30 minutes, because a skipped prune just leaves the app OutOfSync and nobody looks.

Questions for people who run this in production:

Do you keep Namespaces inside the app at all, or use CreateNamespace=true or a separate app just for namespaces?

Is per-resource Prune=false normal, or do teams just turn automated prune off?

I read that a force sync can still delete a namespace even with Prune=false. Do you guard against that somehow?


r/kubernetes • • 3d ago

Does GKE's node kernel have the SCTP `sctp_init_cause` padding fix, or is it still shipping unpatched?

Thumbnail reddit.com
3 Upvotes

I've been checking which Linux kernels still carry an SCTP bug that panics the host: `sctp_init_cause()` checks buffer room for an error cause but not for the up-to-3 bytes of 4-byte alignment padding the write adds, so a crafted INIT (listener) or INIT-ACK (outbound association) can walk an skb one byte past its end and hit `skb_over_panic()`. It's fixed upstream in `ed6c53b27be4` (mainline 7.4-rc2, Cc: stable), introduced back in 2018.

As of 14 Nov I've read the guard in the shipped source of every maintained stable branch and several distro kernels, and none of them had the fix yet. That includes the current Ubuntu kernels (jammy 5.15, noble 6.8, resolute 7.0). GKE Standard supports SCTP on Ubuntu nodes with Dataplane V2, and Google documents it as on by default from `1.32.2-gke.1297000`. Since GKE's Ubuntu node images are built from Ubuntu kernels, my working assumption is that most GKE Ubuntu nodes are running the unpatched code. I haven't confirmed a specific GKE node-image build, which is where I could use help.

If you run SCTP workloads on GKE (telecom CNFs especially), could you check a non-production node and report back:

- `uname -r` on the node, and whether `ed6c53b27be4` is backported (the node's kernel source/changelog, not the version string, since backports move independently)

- whether `lsmod | grep '^sctp '` shows the module loaded, i.e. a workload is actually using it

- the GKE version and node image

All read-only. Please don't send anything at a listener or a live service; I'm after kernel/config state, not a crash.

If it turns out a GKE configuration is unpatched and reachable, that belongs in a report to Google's Vulnerability Reward Program, not in this thread. I'll help put that together rather than post specifics here.

Mechanism and the full affected-version matrix (no exploit) are written up here:

Here

Mainly I want to nail down one thing: is a default-config GKE Ubuntu node running the fixed kernel yet, or not?


r/kubernetes • • 4d ago

Metrics, logs and alerts for my 3-node Talos homelab

10 Upvotes

My 3-node Talos homelab had no monitoring, so I only found out something was wrong when an app stopped working.

So I added VictoriaMetrics for metrics and VictoriaLogs for logs from every node and pod. Both show up in Grafana, and alerts go to my phone on Telegram.

Healthchecks[.]io also watches the alerting itself, so I hear about it if the alerts stop. The whole stack uses about 1.3 GiB of RAM across the three nodes.

Repo: https://github.com/prateekjaindev/homelab

Write-up: https://blog.prateekjain.dev/metrics-logs-and-alerts-for-a-3-node-talos-homelab-with-victoriametrics-0f4a91b9d44e?sk=751eedebb5d54cc15561d59abb7e52d8