Recently I benchmarked the Qwen3-VL-235B, and the result was very disappointing, as it showed only marginal improvements over the Qwen3-VL-2B.
The unusually high failure rate tipped me to investigate the results. Initially I suspected the benchmark or provider integration, but checking the saved responses were all pointing to the model direct output.
As Qwen3-VL-235B from our provider did not had a native json_schema, I had to use prompt to generate json_object in response, and surprisingly there were alarming amount of failures (For 100 samples per tasks):
| Task | Failed responses | Invalid JSON | Valid JSON, invalid schema or values | Main issue |
|---|
| Grounding (Box) | 53 | 47 | 6 | All coordinates placed under x1, often with a missing closing bracket |
| Grounding (Point) | 50 | 6 | 44 | Coordinate arrays under x, or an array replacing the point object |
| Detection | 26 | 14 | 12 | Malformed coordinate objects, wrong types and out-of-range values |
| Attributes | 5 | 0 | 5 | Values outside the provided vocabulary |
Classification, VQA and captioning had no invalid responses. Probably due to simpler task schema.
For example, instead of separate numeric x and y fields, one response was:
{"status": "found", "point": {"x": [0.55, 0.24]}}
This goes beyond a missing bracket. Making the JSON parseable still leaves the wrong structure, and automatically assigning those array elements to different fields would mean interpreting the answer.
And just to be clear most of these failures weren’t caused by the token cap as almost all ended normally, and failed point responses used just 20–28 output tokens against a 4,096-token limit.
The Workaround
The practical approach was correction retries: send validation errors back to the model and let it regenerate, keeping the original schema. No ground-truth answers or scores are provided as feedback:
task = Ground(PydanticAIBackend(model, retries=2), target="the leftmost cat")
results = task('cat-garden.png')
That allows up to two additional attempts after an invalid response. A fresh run on the same samples produced:
| Task | Metric | No retries | With retries |
|---|
| Classification | Top-1 Acc | 96% | 96% |
| Detection | F1@0.5 | 16.67% | 28.45% |
| Grounding (Box) | Acc@0.5 | 31% | 81% |
| Grounding (Point) | Point-in-box Acc | 41% | 85% |
| Attributes | Acc | 73.86% | 78.02% |
| VQA | VQA Acc | 68.4% | 64.9% |
| Captioning | ROUGE-L | 0.3841 | 0.3837 |
Failures remain included in the scores. The retry run had zero final invalid responses across 700 samples, but requests averaged 1.2 per sample overall, rising to 1.55 for box grounding and 1.47 for point grounding.
The Fix
Probably moving to a provider that support json_schema at inference would be the best choice. Sure I can use PydanticAI json_schema but it is not constrained decoding. I did a test run and it helped but then it was not a clear win, but rather a trade-off:
Native JSON Schema was 10% cheaper overall, but 47% slower than the normal prompted run, with Detection driving most of the slowdown:
| Configuration | Input tokens / sample | Output tokens / sample | Requests / sample | Estimated cost / 1,000 samples | Mean latency (s) |
|---|
| Normal | 812 | 65 | 1 | $0.604 | 1.441 |
| Up to 2 retries | 1,058 | 80 | 1.196 | $0.775 | 1.812 |
| Native JSON Schema | 442 | 116 | 1 | $0.543 | 2.121 |
Compared with normal:
- Retries: cost +28%, latency +26%, total tokens +30%.
- JSON Schema: cost −10%, latency +47%, total tokens −36%.
- Compared with retries, JSON Schema cost 30% less, but took 17% longer.
The detection difference was substantial:
| Detection | Normal | JSON Schema | Change |
|---|
| Output tokens / sample | 162 | 499 | +208% |
| Estimated cost / 1,000 samples | $1.169 | $1.728 | +48% |
| Mean latency (s) | 2.915 | 7.554 | +159% |
Closing
There’s clearly useful visual capability in Qwen3-VL-235B, particularly in grounding. But unreliable structured output can become a serious pain when you need usable results in an application: extra requests, validation logic, latency and cost.
If you are interested in comparison between variants of Qwen3-VL models, check [Revised] Qwen3-VL: 2B vs 235B on standard vision tasks, if you want a comparison between 2B and 235B.
Code, runs, benchmarks all available at https://github.com/Tenkei/vlm-tasks