Millan Kaul

less than 1 minute read

Written by - Millan Kaul

You do not need 1,000 test cases to begin AI evals. I started with five.

Image showing five examples are enough to start AI evaluation
Your first eval dataset can fit on a sticky note.

What I did today:

I wrote down five examples based on realistic user requests: an easy one, an ambiguous one, an edge case, a failure case, and a sensitive request. That became my tiny eval dataset.

What surprised me:

Writing the examples clarified the product behavior before I even ran the model.

My QA takeaway:

  1. A small, representative dataset is more valuable than a large random one.
  2. Your first eval dataset can fit on a sticky note.

What would be in your first five AI test cases?

#AIEvals #TestAutomation #AIProduct

You may also enjoy

I tested a longer prompt - day 03

less than 1 minute read

Increasing the complexity of prompt inputs to see how the LLM judge responds and analyzing its limits when prompt length grows.

My first AI eval was boring - day 02

less than 1 minute read

Setting up a basic LLM evaluation only to find the initial results predictable, but discovering why simple tests are the best place to start.