r/mffpc • u/wack_overflow • 21h ago
I built this! (MATX) Compact inference homelab build - 21.6L - 96GB VRAM
Managed to basically max out this Mechanic Master C34 case. It JUST fits in the bottom of my IKEA FJÄLLBO shelf/workbench (highly recommend, btw) with the handle on.
Topline numbers:
- Case size: 13.46 × 7.28 × 13.46 inches (21.6L)
- 96 GB VRAM, 128 GB DDR4 RAM
- Running 2x Qwen3.8 Flash Next at 262k context, 50-70 t/s
Runs like a champ. Both RTX A6000s set to 300 W power limits, seeing around ~76°C under load even in this tight case. Blowers FTW.
Full hardware list:
- ASRock Rack X570D4U motherboard
- AMD Ryzen 7 5700X cpu
- WD_BLACK SN850X 2 TB NVMe ssd
- NEMIX RAM 128 GB (4 × 32 GB), DDR4-3200 rated, running at 2666 MT/s
- 2 × NVIDIA RTX A6000 48 GB, ECC enabled
- NVIDIA 2-slot NVLink bridge
- Mechanic Master C34 case with mesh aluminum side panel and dust filter
- Corsair SF1000 1000 W SFX PSU
- Noctua NH-D12L chromax black CPU cooler
- 5 × Arctic P12 Pro PST 120 mm case fans
Managed to buy both A6000s on eBay a few months ago for a bit less than they’re going for now. The first one had a faulty vapor chamber, but Northwest Repair got it sorted and it’s finally home. Total cost was around $11k.
Been running the machine with just the one card for a while, hence the dust. Yes, I’m messy. No pretty cable management going on over here.
Highest single-day electrical usage so far is ~7.6 kwh (tho I'm not running inference 24/7 atm). Not super loud either. I have the GPUs configured to go to 100% fan as soon as they start working to keep the temps as low as possible. It does heat the room up a bit when they're running for a few hours straight... It'll keep my heating bill down this winter for sure.
Software side is mostly in place. Ubuntu Server w llama.cpp behind llama-swap. Sharing with a handful of friends & family for now, running OWUI on a vps, or tailscale (for those wanting direct api access). Also running a cline kanban fork I've built out for agent orchestration.
I'll let the AI tell you about itself. Go off short king...
Current lineup:
- Qwen3.8-Flash-Next, GSQ-RCO IQ3_S: The big boy, main daily driver for near-frontier output. 125B main-model parameters with ~6B active per token, plus its massive n-gram embedding table accessed through mmap/lazy loading. Split across both A6000s, reasoning enabled, two configured 262K context slots. Seeing around 63–67 tokens/s for a single request, or about 51 tokens/s per request with two going in parallel. Warm concurrent tests managed 84–98 output tokens/s combined, including request overhead. With Gemma still resident, the whole setup was using about 82 GiB VRAM. Pretty darn happy with that.
- Gemma 4 E4B, HauhauCS Aggressive Q8_K_P: Kept resident for fast responses/summaries/etc and quick vision (was that a delivery on the doorbell camera?), with a 32K context and reasoning disabled. This is the little guy that stays available to mostly automated services needing quick responses, while the bigger models swap around to serve users.
Also have a few toys installed:
- Z-Image Turbo Q8_0, through stable-diffusion.cpp: roughly 10.5 GB of model memory in VRAM, with a tested 512×512 image in ~4.7 seconds at eight steps.
- FLUX.2-dev Q4_K_S for image editing: using CPU parameter offload, with about 34 GB of parameters stored in system RAM. A tested 512×512 edit took ~70 seconds, or ~76 seconds including startup.
- Wan2.2-TI2V-5B for text/image-to-video: an earlier single-A6000 test produced 81 frames at 832×480/16 fps—about five seconds of video—in ~112 seconds, using roughly 32 GB peak VRAM. Still more experimental than the text setup.
- EmbeddingGemma 2, Nomic Embed Text v1.5, and multilingual E5 Small for embedding/RAG/routing duties.
Those image/video numbers are from earlier single-card testing; I haven’t rerun the whole lot on the finished build. The big models swap rather than all living in VRAM together.
Also, “two 262K slots” is configured capacity—I’ve tested concurrent Flash conversations around 10K tokens each, not two completely full 262K histories. The text numbers above include reasoning tokens.
---
Thanks robot. Back to work now.
Anyway. Glad the incredibly expensive card was saved and I managed to get it all together. Shout out (again) to captain save the day Northwest Repair on youtube for sending it back all fixed up.
Cheers all.