Millan Kaul

less than 1 minute read

Written by - Millan Kaul

The better answer was also the slower answer. That is a product decision, not just a model decision.

Image showing Latency is part of testing
Quality includes how long users wait for it.

What I did today:

I timed five requests to two model options. I wrote down response quality and time-to-first-answer.

What surprised me:

The quality difference was small, but the waiting-time difference was obvious.

My QA takeaway:

  1. Latency is part of AI quality because users experience the whole system, not just the final text.
  2. Quality includes how long users wait for it.

How much delay would your users tolerate for a better answer?

#AIEvals #LLMOps #UserExperience

You may also enjoy

The ‘best’ model lost - day 08

less than 1 minute read

The model everyone calls ‘best’ lost my tiny test today. Learn why a cheaper, faster model might outperform state-of-the-art models for specific tasks.

I made AI Judge - Day 06

less than 1 minute read

Evaluating the logic behind having an LLM judge another LLM’s output. Establishing a helpfulness rubric and confronting definitions.