Millan Kaul

less than 1 minute read

Written by - Millan Kaul

I tested two models today. One answer was only slightly better and much more expensive.

Image shwoing Measure Value not hype
The best AI answer is not always the best business decision.

What I did today:

I ran the same small dataset through two model options and estimated the cost per response. Then I compared that cost with the quality improvement I could actually see.

What surprised me:

I had a strong opinion about quality, but no opinion about whether that quality was worth paying for.

My QA takeaway:

  1. An eval score without cost context is incomplete.
  2. The best AI answer is not always the best business decision.

Would you pay 10x more for a 5% quality lift?

#AIEvals #AIProduct #LLMOps

You may also enjoy

Better answers took longer - day 09

less than 1 minute read

The better answer was also the slower answer. That is a product decision, not just a model decision.

The ‘best’ model lost - day 08

less than 1 minute read

The model everyone calls ‘best’ lost my tiny test today. Learn why a cheaper, faster model might outperform state-of-the-art models for specific tasks.

I made AI Judge - Day 06

less than 1 minute read

Evaluating the logic behind having an LLM judge another LLM’s output. Establishing a helpfulness rubric and confronting definitions.