Claude Sonnet 5.5 vs GPT-6.1 Sol

How it was run

Not run: it needs an OpenAI API key, which this project doesn't hold. The runner, suite and scoring are ready; adding the key and an OpenAI provider to the runner is the remaining step. The pairing is provisional until we confirm comparable tiers.

Test suite version 1.0. Scores are automatic checks on each reply; the protocol, runner and raw outputs are kept with our research notes.

What we’ll run

No results yet. When this runs, both models get these four tasks with identical prompts:

• Coding: Write a small parsing function

  11 automatic checks. Small, well-specified functions are the everyday coding job. Hidden tests decide the score, including invalid input the function must reject.

• Research: Answer only from a source

  6 automatic checks. Research with AI fails when the model fills gaps from its own guesses. One question has no answer in the source, and the right reply is to say so.

• Writing: Rewrite a notice in plain English

  7 automatic checks. Plain-English rewriting is one of the most common writing jobs. The checks test that every fact survives, the word limit holds and the jargon goes.

• Reasoning: Solve a scheduling puzzle

  2 automatic checks. A small logic puzzle with exactly one solution, checked by brute force when we wrote it. The score depends on the answer, not the working.