[jcmoro]

What

• Two-number baseline: the E1–E3 report now gives two numbers, which together are the baseline:
  • End-to-end, out of 10, next to the retrieval ceiling. With the current embeddings, q06 and q10 have no acceptable SKU among the candidates, so the ceiling is 8/10.
  • Selector only, counting only the questions whose acceptable SKU was among the candidates.
• Separate embeddings PR: the plan records the embeddings model change as its own PR (PR 4b, after PR 4), to be decided with data.

Why

The current embedding model is paraphrase-multilingual-MiniLM-L12-v2. It is already multilingual, yet it leaves q06 and q10 out of the top 10. Brand names lead it astray ("Bota Glaciar", "Tienda Refugio"), and both questions need an inference to reach the right product. The dataset is unchanged.

Without the second number, a retrieval miss would count as a selector error.

Free in-memory check (no Claude calls):

Model	Dims	Questions with an acceptable SKU in the top 10
MiniLM-L12 (current)	384	8/10
multilingual-e5-small	384	9/10 (loses q05)
multilingual-e5-base	768	8/10
multilingual-e5-large	1024	10/10 (q05 only just, at rank 10)

Proof

• baseline_lines test: failed first (ImportError), now passes.
• make lint: clean.
• make test: 273 passed, 4 deselected (the eval tests).
• E1–E3 have not run yet: they're waiting for API credit.

🤖 Generated with Claude Code