Claude Sonnet 5.5 vs Gemini 3.8 Flash

How it was run

Not run: it needs a Gemini API key, which this project doesn't hold. Note that Google's unpaid tier lets people review inputs, so a paid key would be the right choice for any private test material. The pairing is provisional.

Test suite version 1.0. Scores are automatic checks on each reply; the protocol, runner and raw outputs are kept with our research notes.

What we’ll run

No results yet. When this runs, both models get these four tasks with identical prompts:

• Coding: Write a small parsing function

  11 automatic checks. Small, well-specified functions are the everyday coding job. Hidden tests decide the score, including invalid input the function must reject.

• Research: Answer only from a source

  6 automatic checks. Research with AI fails when the model fills gaps from its own guesses. One question has no answer in the source, and the right reply is to say so.

• Writing: Rewrite a notice in plain English

  7 automatic checks. Plain-English rewriting is one of the most common writing jobs. The checks test that every fact survives, the word limit holds and the jargon goes.

• Reasoning: Solve a scheduling puzzle

  2 automatic checks. A small logic puzzle with exactly one solution, checked by brute force when we wrote it. The score depends on the answer, not the working.