Skip to main content
TL;DR: QWED verifies LLM outputs against developer-written code specifications, not natural language. If you let the LLM generate both the answer AND the spec, you’ve just verified a hallucination.

How QWED actually works

The golden rule

QWED does NOT solve: Natural language → Formal specification translation. QWED DOES solve: “Is this LLM output correct given my known ground truth?”

Examples by engine

Math engine

SQL engine

Code engine

Logic engine


When QWED does not work


When QWED works best


Summary


“Math verifies the proof, not the premise. If you hallucinate the constraints, you’ve verified a hallucination.” — Valid criticism we agree with. That’s why specs come from you, not the LLM.