r/StableDiffusion • • 2d ago

Discussion State of the Sub - 1M Members Reached!

131 Upvotes

Howdy! This has been an interesting ride to get to this point over the years. I remember joining really early on when Stable Diffusion was still around releasing models. Then, picked up as a mod, followed by inviting the few great mods we currently have. We even do have some OG mods who peek in every now and then.

I’d like to thank u/mcmonkey4eva and u/HughWattmate9001 for sticking around and trying to keep up with our queue with me. Also, I'd like to thank those who have contributed to the community and kept interesting conversations.

I don't think I've really put an emphases on member count for the sub, but u/HughWattmate9001 pointed out that we were reaching 1M. So, I provided a screenshot below of how this year has gone for the community and mods:

Past 12 Months Insights

I know we have gone through many different waves/phases both in user post types and different moderating styles over the years. The few of us left as mods have tried to be consistent (when available) in moderating depending on members reports more so than any controlling methods you see in other subs. I hope we continue to remain and grow as the open source sub that we consistently try to be.

Feel free to express how you feel about the sub. Good or bad. Talk about the sub community, 1girls, mods, memes, improvements, events, and/or whatever relevant.

I know I am not as active as I was the first few years with u/mcmonkey4eva and u/HughWattmate9001 passing me in moderation actions. However, I hope I've personally done well enough for my part in keeping this sub up for everyone. Even if I don't respond to each one, I'll more than likely read all the comments throughout the week.

Happy Holidays everyone and I hope yall stick around.


r/StableDiffusion • • 1h ago

Discussion I compared 62 samplers and 16 schedulers for Minimax H3 Image and rated the image quality so you don't have to 😉

61 Upvotes

Here's a sampler/scheduler comparison table for image generation with Minimax H3 Image. The numbers are the time in seconds it took to generate the image on my 5090. I have removed rows and columns with only red cells and obviously it reads like Red < Orange < Yellow < Green.

You're welcome!

PS. If you don't like it, don't appreciate it or think I'm wasting my time... Then... Don't waste your time, just move along 😉


r/StableDiffusion • • 2h ago

News [Papers] DWD - Decoding Without a Decoder

59 Upvotes

"Latent diffusion models rely on a decoder to map denoised latents to pixels, adding complexity and inference cost. We introduce Decoding Without a Decoder (DWD), which lets a finetuned diffusion transformer produce final pixels directly through a linear head. Unlike previous methods, where the denoising output leads to final pixels, DWD separates the two roles: the original denoising head continues to drive sampling, then the new DWD head predicts the pixels at the final step.

We extend pretrained latent models with DWD efficiently by finetuning only the last few transformer blocks, training the DWD head with pixel quality losses while anchoring the denoising head to the pretrained model to preserve the sample trajectory. Across latent image and video generators, DWD removes the decoder and reduces inference cost while maintaining or even improving pixel quality. On Wan video generation, it visibly reduces detail distortion and flickering, lowering warping error by 23%. Remarkably, the same separation also improves pixel diffusion models that do not even use a decoder: a final-step DWD head produces substantially better texture detail in text-to-image generation, and improves ImageNet FID from 1.55 to 1.43, a new state of the art among pixel flow models."

Site: DWD: Decoding Without a Decoder

Paper: [2611.22287] Decoding Without a Decoder: Direct Pixel Prediction from Diffusion Transformers


r/StableDiffusion • • 1h ago

Resource - Update Famegrid Spice Moss Image LoRA

Thumbnail
gallery
37 Upvotes

r/StableDiffusion • • 2h ago

Question - Help Why does AI models always zoom to most described object on the scene no matter what you say on the prompt to zoom out of that object?

16 Upvotes

r/StableDiffusion • • 7h ago

Workflow Included Sharing my experience with H3-Image

30 Upvotes

I decided to try squeeze out the quality showcased in the official examples for H3-Image.
While I mostly solved the grunge type artefacts thanks to the existing community efforts, I struggled with grid artefacts. Eventually, I realised that there is no working around the nature of the VAE and large resolutions are necessary to avoid the grid artefacts.
Workflow: https://gofile.io/d/Slgntt7J
I hit a wall with clarity and sharpness of details, so I'm sharing my workflow with the hope that the community can take it further. The prompts are taken from the official examples.

Aspect ratio 21:9. A realistic fantasy-creature facial close-up: a beast king combining a lion's mane with an ape-like facial structure faces the camera, angled slightly left. Clearly show its broad gray nose bridge, deep-set eye sockets, and amber eyes. Its lips are slightly parted, revealing a few small ivory-colored teeth. Short, dense gray-white fur covers its face, surrounded by an enormous brown and silvery-white mane spreading toward the right and bottom of the frame. The fur has natural curls, layers, and fluffy volume, with fine side lighting on individual strands. Only a small portion of a dark-gold ornament appears at the lower-right edge; the face and mane remain the focus. Use a dark, blurred background, realistic transitions between skin and fur, moist eye reflections, and cinematic creature materials. Output one complete artwork with no website borders, artist signatures, or watermarks.
Aspect ratio 16:9. A realistic post-apocalyptic first-person shooter scene. At the lower edge, the player's hands hold a rifle and optic aimed down an abandoned woodland road. In the left midground, a companion wearing green tactical clothing, a backpack, and a helmet fires forward, with a small orange-yellow muzzle flash. On the right, another fighter in a hood and worn brown coat crouches halfway while holding a long gun. Tall pines, gray shrubs, and thin fog surround the distant road; a rusty yellow bus sits in the left background. The ground has rich details of weeds, stones, and soil. Low sunlight at dawn or dusk filters through the trees, contrasting the cool gray-green environment with brief gunfire flashes. Equipment structures should be realistic. Tense atmosphere with no gore and no HUD.
Reddit breaks the formatting, so refer to the official example for the prompt. It's the 1st example in showcase/real-world.md in the official GitHub repo.

r/StableDiffusion • • 4h ago

Discussion What's the best "Director" node for Minimax H3?

18 Upvotes

I've been looking for a director node for Minimax H3, but so far I haven't found one simple enough to use. It would be nice to have a director node similar to those designed for LTX. Have you found anything simple? What nodes do you use as directors, if any?


r/StableDiffusion • • 16h ago

News [Release] Otaku TTI 1.0: A featherweight, sub-gigabyte anime diffusion model that runs on almost any hardware (<1.2 GB VRAM)

Post image
145 Upvotes

Hey everyone!

I wanted to share **Otaku TTI 1.0**, a lightweight Latent Diffusion model built specifically for anime character art that I designed and trained entirely from scratch on a consumer

laptop GPU (RTX 4050 with 6GB VRAM).

While the AI space has moved toward massive 10GB+ models that require expensive cloud servers or high-end GPUs, I wanted to build something fast, compact, and accessible to

anyone even on older hardware, laptops, or handhelds like the Steam Deck.

### Highlights & Specs:

* **Architecture:** 166M Parameter Latent UNet conditioned on CLIP embeddings via cross-attention.

* **Microscopic Footprint:** Downloads at just **318 MB** (FP16).

* **Ultra-Low Memory:** Peaks at **under 1.2 GB VRAM** during inference (also runs cleanly on pure CPU in 3-5 seconds).

* **Blazing Fast:** Generates in **~190-220ms per image** on entry-level consumer GPUs using DPM-Solver++.

* **Trained from Scratch:** Pretrained from step 0 and fine-tuned across 173,000+ steps on curated Danbooru anime illustrations.

* **License:** Creative Commons Attribution-NonCommercial 4.0 (CC-BY-NC 4.0) — free for personal, hobby, and research use.

---

### Links:

* **GitHub Repository:** https://github.com/OtakuTTI/Otaku-TTI-1.0

* **Hugging Face Model:** https://huggingface.co/OtakuTTI/Otaku-TTI-1.0

---

### Quickstart:

You can clone the repo and start generating in under a minute:

```bash

git clone https://github.com/OtakuTTI/Otaku-TTI-1.0.git

cd Otaku-TTI-1.0

pip install -r requirements.txt

# Generate 4 images

python3 generate.py -p "1girl, hatsune miku, aqua hair, twintails, solo" -n 4
```


r/StableDiffusion • • 3h ago

Resource - Update I made ComfyUI nodes for LongCat-Video-Avatar 1.6: one photo + one voice = lip-synced talking video (free, MIT)

13 Upvotes

the 3 clips are one run each: one photo + one ElevenLabs voice line, 480p, seed 42. no lora, no face swap, no editing. sound on.

built it on top of rookiestar28's 1.5 pack (MIT, full credit to them). what's different in 1.6: the official INT8 model, 8 steps, the 1.6 LoRA switch (mix_steps), no loudness norm on the audio, and it picks SageAttention > FlashAttention 2 > PyTorch by itself. long videos work window by window (93 frames, about 3.7 s each). the model files download on first run (about 25 GB), you only add the umt5 text encoder.

honest numbers: I only tested on a 5090 32GB + 64GB RAM, Windows, 480p. a 3.8 s clip takes 5.6 to 7.7 min depending on attention and mix_steps, each extra 3.2 s of audio adds one window (1.5 to 2.5 min). I haven't tried cards under 32GB. with block streaming the model itself used about 13GB at 480p, but that's not the same as "runs on 12GB".

what helped: calm presenter = write "hands relaxed" in the prompt, clean speech with no music, short 4 s tests first, new seed instead of rewriting the prompt.


r/StableDiffusion • • 1h ago

Workflow Included Krea2 vs Minimax H3 vs Z-Image Turbo.

Thumbnail
gallery
8 Upvotes

Решил провести небольшой тест и сравнить генерации одних и тех же промптов на моделях Krea2, Minimax H3 и Z-Image Turbo. Все тесты делал с фиксированным сидом, чтобы условия были абсолютно равными.

Моё субъективное мнение и вердикт:

Если честно, я в восторге от того, как отрабатывает Minimax. На мой взгляд, по совокупности факторов - это топ. Понимает анатомию, прорабатывает кожу, свет и тени. Она абсолютно точно может выдавать коммерческий результат. В некоторых генерациях она выдаёт картинку даже лучше, чем Z-Image, особенно когда дело касается сложного освещения или групповых сцен. При этом Z-Image, как и всегда, продолжает оставаться моей рабочей лошадкой. Она всё так же является топовой моделью для реалистичных изображений. Я ни в коем случае не собираюсь с нее слезать, но теперь буду использовать её в связке с Minimax.

Раньше я использовал связку Z-Image для генерации изображений и Minimax H3 для видео. Но теперь я однозначно буду работать с Minimax и для картинок тоже. Это кайф, когда можно находиться в рамках одной экосистемы. Krea2 на этом фоне выглядит неплохо, но как-то более "пластиково", что ли, ну и в совокупности выдаёт результаты более проигрышные.

Очень интересно узнать, какие настройки сэмплера и планировщика вы используете для ZIT и Minimax? Вдруг ваши настройки пайплайна выдают еще более крутые результаты.

Кому интересно, можете сами потестить на этом же пайплайне: https://gist.githubusercontent.com/taliratra/aca4cafae7b956f74dab8c4adebae43d/raw/68f5867cfa8ad90d7c745936435da4e87464f55b/gistfile1.txt


r/StableDiffusion • • 6h ago

News Nightmare Mobile: free offline local AI image and video generation on Android, ComfyUI-style node editor on the Snapdragon NPU. MiniMax H3 video with sound, MOSS-Image, FLUX.3, FLUX.2, Z-Image, Qwen Image, Wan 2.2, SDXL, LoRA, ControlNet, inpainting, img2img. No cloud

19 Upvotes

Nightmare Mobile is an Android app I have been building: image and video generation as a node graph, like ComfyUI, but the models run on the phone's NPU (the Hexagon chip in Snapdragons). No server, no account, nothing uploaded.

The video is one Galaxy S25 Ultra. The race shows real render times at each model's default size:

\- SD 1.5: 3 to 7 s (512x512 to 512x768)

\- SDXL (Juggernaut, Illustrious, a DMD2 merge): 6 to 47 s at 1024x1024

\- NitroFusion 2 and FLUX.2 Klein 4B: about 24 s at 1024x1024

\- Anima Turbo 42 s, FLUX.2 Klein 9B 51 s (768)

\- MOSS-Image 1 to 2 min, Z-Image Turbo 1 min 38

\- Qwen Image 2.1, Krea 2 Turbo, Ideogram 4: 4 to 5 min

What else it does:

\- Video: Wan 2.2 (a 5 s clip at 832x480 in under 3 min), Wan 2.2 14B on a 12 GB phone, MiniMax H3 clips with speech and sound, motion transfer from a video

\- Inpaint and outpaint, tap to select (SAM 2.1), ControlNet (canny, depth, openpose), IP-Adapter, hires fix, ADetailer-style Fix faces and hands, upscalers

\- Image editing by instruction: FLUX.2 Klein, Qwen Image, Fibo 2.0 Edit

\- LoRAs: search and download from CivitAI and Hugging Face inside the app, or \`<lora:name:0.8>\` in a pasted prompt

\- Your own checkpoints: a FLUX.2, Z-Image, Qwen Image, Krea 2, Anima or MOSS-Image fine-tune from CivitAI (safetensors or GGUF) runs as it is, no conversion. SD 1.5 and SDXL checkpoints need converting for the NPU first (npuforge, also free)

\- Saved PNGs carry A1111 parameters (CivitAI reads them) plus the whole flow; drop one back in and the flow comes back

\- A local API that also speaks A1111's /sdapi/v1/txt2img, so other apps can use the phone as their backend

Current limits:

\- Snapdragon only for the NPU. 8 Gen 3 and newer runs nearly everything; 8 Gen 2 and the 8s chips run the DiT models but slower; 8 Gen 1 gets SD 1.5 plus FLUX.2 Klein 4B and NitroFusion 2 on the GPU. MediaTek, Exynos and Tensor phones can only run an imported SD 1.5 or SDXL model on the CPU/GPU for now

\- The big ones want 12 GB of RAM (Ideogram 4, Krea 2, Fibo, MiniMax H3)

\- The times above are a flagship. Older chips are slower

How it is built: the SD 1.5 / SDXL NPU server started as a fork of xororz's Local Dream, and the DiT models run on leejet's stable-diffusion.cpp with happyyzy's Hexagon work. On top of that I brought most of the models and all of the video to the phone (85 engine patches, all public), sped up the NPU's attention and copy kernels, got the DiT engine running on the 8 Gen 2 and 8s chips (and found the 8s chips' matrix unit returns wrong results), and added a GPU route for older Snapdragons. The README's "What we built" has the list; full credits are in the guide.

It is free and CC BY-NC (no ads, no account, nothing gated).

\- APK: https://github.com/AbrahamPaulJ/nightmare-mobile/releases/latest

\- Repo link: https://github.com/AbrahamPaulJ/nightmare-mobile

Happy to answer questions about getting diffusion models onto a phone NPU, and bug reports are very welcome.


r/StableDiffusion • • 22h ago

Question - Help Help required recreating this Artsyle NSFW

Thumbnail
gallery
270 Upvotes

Hey guy, can anyone help me to replicate this art style.
i have tried finding spider verse and arcane style but not getting desired result.
i am primarily looking for ada wong's art style.
these photos are created by neogalaxy.453

Edit: I use illustrious and anima models only.


r/StableDiffusion • • 22h ago

News Thank dog for Claude Code

Post image
248 Upvotes

Zluda is history. It won't do wonders, but especially open sourced stuff like ComfyUI-Nodes with SageAttention, Flash-Attention-Variants, bitsandbytes, 3D- or Video-Nodes with .cu-files can be ported now by Claude Code.

Some Reddit user already ported an entire CUDA backend to AMD’s ROCm platform in just 30 minutes—without a translation layer.

All in all this year has been nice to Team Red, and i wonder what's coming for Christmas.


r/StableDiffusion • • 16h ago

Resource - Update sprite32: a 32x32 text-to-sprite diffusion model trained from scratch, open weights (Apache-2.0)

81 Upvotes

I trained a little text-to-sprite model from scratch. 32x32, transparent background, open weights.

Some handpicked samples

It’s a 143M parameter UNet trained on 660k sprites, icons and emoji. Training took about 20 hours on my 5090. It works directly on the pixels (no VAE) and the output is RGBA, so the background is actually transparent.

Also, it’s completely vibe coded in 5 or 6 sprints over the last year. I put a lot of work into it, but Claude wrote the code.

There are two versions:

  • 1.0 is the full model. It’s a little better and you can turn the guidance up or down, but it uses about 9 GB of VRAM (mostly the 4B text encoder) and takes about 0.9s for a single sprite on a 5090
  • 1.0 fast is the one to start with. It’s about 5x faster (0.16s per sprite on a 5090), fits in under 3 GB, and runs fine on a CPU. It’s a 2.3 GB download instead of 8.7 GB. The quality is close.

Try it here: sprite32.com. It runs on a spare PC in my house, on CPU until my GPU shows up, so be nice to it.

Weights: huggingface.co/danemarguglio/sprite32
Code: github.com/danemarguglio/sprite32


r/StableDiffusion • • 1d ago

News He has finally spoken

Thumbnail
gallery
429 Upvotes

r/StableDiffusion • • 23h ago

Workflow Included Retrolab: real time film emulation for ComfyUI images

179 Upvotes

I made this node to give ai images a more natural and filmic look

Bring photographic texture, film grain, expressive light behavior simulating bloom, halation, solarization, light leaks, soft focus, vignette, tonal reversal and optical finishing to ai images directly inside ComfyUI, use after decode as a post-process node to achieve a more natural and realistic vintage look.

It's a single node <1mp without any dependencies written on Python and JavaScript, published in my GitHub and registered in Comfy: zuanfilm-retrolab

Workflow: Zuanfilm_Retrolab.json

Works real-time 512px Live preview even on a toaster with 8gb ram and no Vram card using Pytorch CPU, when a fast GPU is detected switch to Nvidia CUDA acceleration

Node was tested and optimized with images from Moss, Qwen Image 2.1, Minimax H3 image, Krea2, Flux2Klein and also 60Mp real photos, use float32 internal precision for photographic calculations & normal ComfyUI image-tensor output behavior

Supported UX languages and on hover tips: English, Simplified Chinese, Japanese, Spanish, Portuguese, French, Russian, German, Hindi & Arabic

I included a selectable color/tonal response from 50-look emulsions inspired on cinema, instant, slide and 35mm including Kodak VISION3 50D, Polaroid SX-70 Color, Fujifilm FP-100C, Fujichrome Velvia 50,Kodak Portra 160 and Fujicolor Superia 800

The Anti-AI slider is a custom 0–100% correction (default 0%) intended to bring common AI-render excesses toward a more natural photographic balance. Its smooth global response gently compresses contrast, reduces red/orange chroma, softens the black toe, opens midtones and rolls highlights more gradually

White Balance is controlled with a Resolve-inspired two-axis gain wheel. Retrolab converts the input sRGB values to linear RGB, applies restrained channel gains, normalizes the gain against Rec.709 luminance, and then converts back to sRGB

I tested and calibrated color and speed with 60 Mp real photos taken by me

Bloom use Soft-knee highlight extraction and multi-scale optical glow

Exposure it ranges from -100 to +100, with 0 as an exact bypass. Negative values retain the established linear darkening response in linear-light RGB

The halation engine combines an annular kernel with an outward scattering tail. Color picker selects the normalized linear-RGB emission tint and does not change the Amount control

I included procedural texture with Gaussian, Value fBm, Gradient fBm-Perlin-style, Simplex fBm, Ridged fBm fractal response, Worley Microcells, Multi scale Domain Warp, Silver Halide Clumps, Emulsion Mottle, Film Scratches, Dust & Fibers and Paper Foxing

For film grain and procedural texture I included Blend modes: Additive, Multiply, Screen, Overlay, Soft Light, Hard Light, Color Dodge, Color Burn,Lighten, Darken, Difference and Exclusion

Apache License, Version 2.0


r/StableDiffusion • • 10h ago

Discussion PAN 2 (OPEN SOURCE I2V)

15 Upvotes

What Happened to PAN 2 (Open-Source I2V Model)?

Does anyone have any updates on PAN 2? It was supposed to be an interesting open-source Image-to-Video model, but lately, there seems to be almost no news or discussion about it.

Is the project still in development, has it been delayed, or has development been abandoned? Does anyone know if the developers have shared any updates or announcements recently?

I'm really curious about what happened to this project. Any information would be appreciated!


r/StableDiffusion • • 1d ago

News If you think Jensen supports Open Source AI, you are wrong

Post image
231 Upvotes

He was cheating not only on his family but average consumers too.


r/StableDiffusion • • 2h ago

Discussion 5070 vs 5060ti 16gb

3 Upvotes

Wanted to do a poll. Whats general consensus on these 2 cards for general AI use without going into details. Relatively similarly priced with 5070 being some more. No other card options please.

With alot of models optimized for low VRAM, offloading and compartmentalization, does the extra VRAM really make a big enough difference?

5060Ti 16gb - VRAM is king!

Or

5070 12gb - Less VRAM but faster

*im using a 5080 myself that does fine what i ise it for. This is just a general question for average enthusiast.*


r/StableDiffusion • • 7h ago

Question - Help RelayVSR runs out of RAM when rendering longer videos – Is there a way to process videos in chunks automatically?

8 Upvotes

I'm using RelayVSR through ComfyUI workflows, and I can render a 10-second video in about 16 minutes.

However, when I try to process longer videos, I run into issues during the decoding stage. System RAM usage keeps increasing, the process gets progressively slower, and eventually I run into problems.

I have two questions:

  1. Is there any way to prevent this excessive RAM usage during decoding, or manage memory more efficiently?
  2. If not, is there a workflow that can automatically split a longer video (for example, a 4-minute video) into 10-second segments, process each segment separately, and then merge everything back into a single video?

I can do this manually, but I'd like to automate the process as much as possible to reduce the workload.

Any suggestions, workflows, or tips would be greatly appreciated!

System
32gb system ram
3060 ti 8gb vram
ryzen 5 5600


r/StableDiffusion • • 37m ago

Resource - Update I made PromptMaker, a free, open-source local studio for writing SD prompts (Ollama, Forge and ComfyUI, works on 8 GB)

2 Upvotes

I've been building PromptMaker for a while and figured it's ready to share. It's a desktop app for Windows (macOS and Linux should work too) that helps you write prompts and then sends them straight to Forge or ComfyUI.

What it does:

  • Builds prompts from 800 presets, or from a plain words idea, refined in a chat with a local LLM (Ollama, LM Studio, llama.cpp, KoboldCpp or anything OpenAI-compatible)
  • Formats each prompt for the model you're using: SDXL, Pony, Illustrious, FLUX.2, Qwen-Image, Anima and more
  • Turns images back into prompts, and has A/B testing, an auto generate, check and refine loop, wildcards, inpainting and regional prompts
  • Face Swap with IP-Adapter, Wallpaper mode, a Civitai model browser, and a Video maker that uses Wan2GP

Everything runs locally by default. The AI features are optional, and it never downloads models without asking. I use it every day on an 8 GB card.

Install: grab the repo and double-click install.bat, then run.bat.

GitHub: github.com/Kisahime/promptmaker

It's MIT licensed. Feedback and bug reports are very welcome!


r/StableDiffusion • • 40m ago

Question - Help MMH3 Image does a poor job of recreating certain faces?

2 Upvotes

I've been experimenting with a bunch of reference images and found that the model struggles with certain faces. Anyone else experience this? Or am I just prompting wrong? Or maybe some of my reference images just aren't hi-res enough?


r/StableDiffusion • • 56m ago

Question - Help AI Voice assistance

2 Upvotes

Hey geniuses,

I'm looking for some help/advice. I'm trying to create an animated video and I'm struggling to get the voices I want for the characters.

For one of the characters I need it to sound like a child but Elevenlabs is a pain in the arse when trying to create this. I want the voice to be like Peppa Pigs (but with a different regional accent).

Other than getting a real child actor does anyone know how I could get an AI voice that doesn't just sound like an adult pretending to be young?

Many thanks. Love everyone's work.


r/StableDiffusion • • 12h ago

News Manus AI becomes a Platinum Sponsor of the Godot Foundation

12 Upvotes

Godot (cross-platform 2D and 3D game engine) has announced that Manus AI is now a Platinum Sponsor, providing financial support for the continued development of the open-source engine.

They just announced it a while ago on their X and their blog:

We're happy to welcome ManusAI as a Platinum Sponsor of the Godot Foundation! Thank you for supporting open-source game development.

Sources:

https://x.com/godotengine/status/2122760835881976198
https://godotengine.org/article/partners_manus_sponsorship_2026_announcement/

So far Godot has had quite restrictive AI policies, and they clarified that in June of this year.

Sources:

https://godotengine.org/article/contribution-policy-2026/
https://contributing.godotengine.org/en/latest/development/contribution_rules.html

But this is an interesting move, as role of AI in game development is growing.


r/StableDiffusion • • 5h ago

Resource - Update PAN 2 now runs in Forge Neo, with a timeline editor that writes its JSON prompts for you

3 Upvotes

I made two Forge Neo extensions for PAN 2, the 29B world model from IFM (MBZUAI): one runs the model from the normal txt2img and img2img tabs, and the other gives you a timeline to build PAN 2's structured prompts instead of writing the JSON by hand.

8 seconds, one description, one continuous shot: the skater drops into the bowl, carves, airs above the coping and rolls out while the camera rises. The dog leaves the frame during the air and comes back at the end as the same dog.

1. PAN 2 for Forge Neo

Same checkpoint list, same Generate button, and the clip lands as an MP4 in the usual result area.

  • Text to video, and image to video (your img2img picture becomes the first frame)
  • 24 fps, up to 15 seconds from the Frames slider
  • A pan2 preset with the official recipe: Euler, Simple, 50 steps, CFG 2.5, Shift 9
  • The released negative prompt is used automatically when yours is empty
  • Loads speedyrulz's INT8 ConvRot and NVFP4 files as they are, with the Qwen3.5 text encoder and the Wan 2.2 VAE
  • JSON prompts reach the model exactly as written, even with Dynamic Prompts on
  • Changes no Forge Neo file

2. Regional Boxes: a timeline instead of hand-written JSON

PAN 2 was trained on structured descriptions: the style, the light, who and what is in the scene, and then what happens second by second, with the camera move for each part. Plain sentences work noticeably worse, and writing that JSON by hand is where most people give up.

So my Regional Boxes extension has a PAN 2 mode. Pick the pan2 preset and its panel turns into a timeline of the clip:

  • Fill in the style, lighting, perspective and first frame in plain fields
  • Add the subjects (people, animals, things that move), each with a name and a look
  • Split the clip into events by dragging lines, and pick the camera moves for each event from PAN 2's own list (static, pan, tilt, track, dolly, zoom, roll...)
  • Drag on a subject's lane to say what it does and when
  • Press Write to prompt and you get PAN 2's JSON. Read from prompt loads one back to edit, and fits its times to your clip length if they differ
  • Before you generate, it lists what PAN 2's prompt guide asks for that your description is still missing: an empty field, an action outside its event, a description that is too long

Built on the timeline, not typed: three events, three camera moves and two subjects' actions, then Write to prompt and Generate.

The honest part

  • It's slow. It's a 29B model at 50 steps. On an RTX 6000 Ada (48 GB) with INT8: 1248x704 runs at about 17 s per step, so a 5-second clip takes about 15 minutes. At 640x384 it's about 5 s per step (a 5-second clip in roughly 4-5 minutes), which is good for testing a description before the full render. I haven't tested on 24 GB cards yet, so reports are very welcome
  • No audio. That's the model, not the extension
  • Not there yet: the official prompt rewriter, GGUF, full-precision single files, faster sampling
  • It's a first version (0.1.0)

Links

Credit to speedyrulz for the ComfyUI port and the INT8 / NVFP4 files, and to Haoming02 for Forge Neo. IFM recommends running PAN 2 behind their separate guardrail, which is not part of the extension.