RigSight

The AI journey, part 4 of 6

Picking an AI model blind, and not picking my favourite

Six AI models answered the same questions about my PC, hidden behind letters A to F. Gemini 3.8 Flash won my vote. Claude Haiku 5.5 got the job.

Blind votes on 19 questions. F, Gemini 3.8 Flash, won my vote. E, Claude Haiku 5.5, got the job.

The short version

I voted blind on six models' answers to real questions about my PC. Gemini 3.8 Flash won 12 of 19. After stricter rules it got slower and only partly better, so Claude Haiku 5.5, cheaper and better at following them, got the job.

Picking the AI model for RigSight AI sounds like a quick decision. Look at a leaderboard, take the top one. But leaderboards test general knowledge and coding. I needed something narrower, a model that reads look-ups about one PC and explains them to a person who just had a game crash. The only way to know was to try it on exactly that.

Six models, six letters

On 21 October I picked six models from different companies, from very cheap to expensive, and gave them all the same setup. They had the same instructions, the same look-ups into my PC's real records, the same questions. Things like why a game crashed, whether my GPU runs hotter than it used to, where my time went last week.

Then I hid the names. Each model became a letter, A to F, and the answers went on one page side by side. I read them and voted for the best answer to each question without knowing who wrote it. If I knew which company made which answer, I'd have had opinions before reading a word.

F was clearly my favourite. Its answers were warm, well organised and easy to read.

I only looked up who was behind each letter after voting.

  • A was Qwen3.8 Flash
  • B was DeepSeek V4.1 Flash
  • C was GPT-5.6 Luna
  • D was Claude Sonnet 5.5
  • E was Claude Haiku 5.5
  • F was Gemini 3.8 Flash

Cost per question

The price per question varied a lot, because it depends on the model and on how much it reads. The most expensive model in the test, Claude Sonnet 5.5, cost about 7.3 cents for one question. Gemini 3.8 Flash cost about 1.5 cents. Claude Haiku 5.5 cost about 0.4 cents.

That matters because the helper looks things up. A good answer can involve several look-ups and a fair amount of reading, so the cost of each question adds up quickly. The most expensive model was out on price alone, whatever its answers were like.

Then I tightened the rules

Reading dozens of answers side by side showed me patterns I didn't like in all of them, so I wrote stricter rules for how every answer must be put together.

  • Say "likely" for a likely cause, and only call something certain when the records prove it.
  • Check counts before saying them. "Three crashes" has to be three.
  • No describing words the data can't back up. If it says "much hotter", the numbers have to show it.
  • Facts from the web that change over time get a note on where to check them.

I also decided the helper speaks as RigSight. It says "we found", never "RigSight found", and when it uses something from the web it blends it into the answer instead of labelling it.

Then I ran the test again. Gemini 3.8 Flash, my favourite, followed the new rules only partly, and it got slower. Some answers took 43 seconds. Claude Haiku 5.5, the cheapest of the good ones, followed them closely.

Choosing the one that follows the rules

So Claude Haiku 5.5 got the job. Not because I liked its writing more. I didn't, at first. But in RigSight, following the rules matters more than the writing. An answer that's a little plainer but never claims more than the records show is worth more to someone with a crashing PC than a beautiful answer that guesses. And 43 seconds is a long time to wait for an answer about your own PC.

A bug that looked like the model's fault

One more thing from that week. With Haiku, some answers came back completely empty. No error, just nothing.

The model thinks before it writes, and that thinking counts toward the length limit for an answer. I had set the limit at a size that's plenty for the answer itself. On harder questions, the thinking used it all up, and the answer was cut off before its first word. Raising the limit well above what an answer needs fixed it.

It's a good reminder that when an AI gives a strange result, the first place to look is often the code around it, not the model. The next post is all about that.