TL;DR: QWED verifies LLM outputs against developer-written code specifications, not natural language. If you let the LLM generate both the answer AND the spec, you’ve just verified a hallucination.
How QWED actually works
The golden rule
QWED does NOT solve: Natural language → Formal specification translation.
QWED DOES solve: “Is this LLM output correct given my known ground truth?”
Examples by engine
Math engine
SQL engine
Code engine
Logic engine
When QWED does not work
When QWED works best
Summary
“Math verifies the proof, not the premise. If you hallucinate the constraints, you’ve verified a hallucination.” — Valid criticism we agree with. That’s why specs come from you, not the LLM.