How to test the ai llm judge - day 07

Written by - Millan Kaul

𝗜 𝗳𝗼𝘂𝗻𝗱 𝗼𝘂𝘁 𝗺𝘆 𝗔𝗜 𝗷𝘂𝗱𝗴𝗲 𝘄𝗮𝘀 𝘁𝗼𝗼 𝗴𝗲𝗻𝗲𝗿𝗼𝘂𝘀.

Your evaluator needs evaluation too.

What I did today:

I gave my judge one clearly strong answer and one obviously weak answer. Then I checked whether it scored them differently and explained why.

What surprised me:

The judge liked the weak answer more than I expected because my rubric did not say what a failure looked like.

My QA takeaway:

1. An LLM judge is also a system under test.
2. Your evaluator needs evaluation too.

Have you ever tested the tool that tests your AI?

#LLMTesting #AIEvals #TestStrategy