I removed the right document - day 13

Written by - Millan Kaul

I removed one document from my RAG test. The model still sounded completely sure.

Sometimes the model is not the problem. Retrieval is.

What I did today:

I asked the same question with and without the relevant source document in the retrieved context. Then I compared whether the assistant admitted uncertainty or invented an answer.

What surprised me:

The biggest issue was not generation quality. It was that retrieval failed silently.

My QA takeaway:

1. Test retrieval and generation separately, or you will debug the wrong layer.
2. Sometimes the model is not the problem. Retrieval is.

Do your RAG tests tell you whether the right context was retrieved?

#RAG #AIEvals #TestAutomation