feat(assistant): LangGraph agent with rule-based recommendation (PR 3b)

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help install run up down stop sh test unit-test integration-test e2e-test lint format build migrate migration migrate-test reindex
+.PHONY: help install run up down stop sh test unit-test eval integration-test e2e-test lint format build migrate migration migrate-test reindex
 
 DC = docker compose
 SERVICE = api
@@ -30,6 +30,9 @@ test: ## Run the full test suite
 unit-test: ## Run unit tests only
 	$(DC) run --rm $(SERVICE) pytest tests/unit
 
+eval: ## Run the evals against the real Claude API (reads the key from .env)
+	$(DC) run --rm -v $(CURDIR)/.env:/app/.env:ro $(SERVICE) pytest -m eval -s -p no:cacheprovider tests/evals
+
 integration-test: ## Run integration tests
 	$(DC) run --rm $(SERVICE) pytest tests/integration
 

README.md

@@ -4,13 +4,14 @@
 
 **Generative AI on FastAPI, done the clean way.**
 
-A reference **Generative AI backend** in Python: a **RAG** assistant that answers from a product catalog, built with **FastAPI**, **Domain-Driven Design (DDD)** and the **Hexagonal (Ports & Adapters)** pattern. The LLM, the embeddings and the vector store are adapters behind ports — swap providers, fake them in tests, and keep them out of your domain.
+A reference **Generative AI backend** in Python: a **RAG** assistant that answers from a product catalog, with an optional **LangGraph agent** that recommends products by rules, not by the model's word. Built with **FastAPI**, **Domain-Driven Design (DDD)** and the **Hexagonal (Ports & Adapters)** pattern. The LLM, the embeddings and the vector store are adapters behind ports — swap providers, fake them in tests, and keep them out of your domain.
 
 [![CI](https://github.com/jcmoro/python-fastapi/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/jcmoro/python-fastapi/actions/workflows/ci.yml)
-![Tests](https://img.shields.io/badge/tests-192%20passing-2e7d32?style=flat)
+![Tests](https://img.shields.io/badge/tests-272%20passing-2e7d32?style=flat)
 ![Architecture](https://img.shields.io/badge/architecture-DDD%20%2B%20Hexagonal-2e7d32?style=flat)
 ![mypy](https://img.shields.io/badge/mypy-strict-1565c0?style=flat)
-![GenAI](https://img.shields.io/badge/GenAI-RAG%20assistant-6a1b9a?style=flat)
+![GenAI](https://img.shields.io/badge/GenAI-RAG%20%2B%20agent-6a1b9a?style=flat)
+![LangGraph](https://img.shields.io/badge/agent-LangGraph-1c3c3c?style=flat)
 ![pgvector](https://img.shields.io/badge/vectors-pgvector-336791?style=flat&logo=postgresql&logoColor=white)
 ![Claude](https://img.shields.io/badge/LLM-Claude-D97757?style=flat&logo=anthropic&logoColor=white)
 ![Python](https://img.shields.io/badge/Python-3.12-3776AB?style=flat&logo=python&logoColor=white)
@@ -40,7 +41,10 @@ Most GenAI demos wire the SDK straight into the route handler. This one treats t
 - **Retrieval is a port too.** `EmbeddingClient` and `DocumentSearch` are ports; embeddings run locally with a multilingual model (no API key, no cost) and vectors live in the existing Postgres with `pgvector`.
 - **Indexing is explicit and incremental.** `make reindex` re-embeds only products whose content or embedding model changed (a `content_hash` per document), in batches, and deletes documents only after confirming their product no longer exists. `catalog` doesn't know the assistant exists.
 - **Tests never call a real model.** Unit and e2e tests use fakes at the port boundary (`FakeLLMClient`, `FakeEmbeddingClient`, `FakeDocumentSearch`). The real LLM adapter is tested against a mocked HTTP transport, and the vector adapters against a real pgvector database.
-- **Failures have a contract.** If the LLM is unavailable (connection, timeout, 429, 5xx), the answer still comes back — **200** with the retrieved products and `fallback_used: true`. Configuration errors (bad key, unknown model) are not disguised as outages.
+- **Failures have a contract.** If the LLM is unavailable (connection, timeout, 429, 5xx), or its answer is cut off by the token limit or refused, the answer still comes back — **200** with the retrieved products and `fallback_used: true`. Configuration errors (bad key, unknown model) are not disguised as outages.
+- **The model doesn't pick the product.** In agent mode, `recommend_candidate` asks Claude two yes/no rubric questions per candidate (3 samples each) and a pure `select()` function filters, ranks and applies thresholds. `recommendation` comes only from that selection, never from the model's text.
+- **The agent is bounded.** At most `AGENT_MAX_TOOL_CALLS` tool calls per question; past that it answers in RAG mode with what it retrieved. A transient failure mid-run keeps the recommendation and sources gathered so far.
+- **Evals are opt-in and honest.** `make eval` runs the selector and the full agent against a fixed 20-product dataset in its own database, reports results as counts, and stops if the API rate-limits too much. They never run in `make test` or CI.
 - **Latency is bounded.** `LLM_TIMEOUT_S` applies per attempt and `LLM_MAX_RETRIES` defaults to 0, so the worst case is `timeout * (retries + 1)`.
 - **Every answer reports its cost.** Responses include the model used, input/output tokens and latency.
 
@@ -76,13 +80,18 @@ Example response (values are illustrative):
     "input_tokens": 52,
     "output_tokens": 87,
     "latency_ms": 1240
-  }
+  },
+  "recommendation": null
 }
 ```
 
+`recommendation` is always present. It is `null` in `rag` mode, and in `agent` mode when the agent did not run the selector. When the selector ran, it is an object: `{"product_id", "sku", "score", "reason"}`, where `reason` is `chosen`, `below_threshold` or `no_assessments`, and `product_id`, `sku` and `score` are non-null only when `reason` is `chosen`.
+
+Agent mode is off by default. Turn it on with `ASSISTANT_MODE=agent`: the agent then has three tools, `semantic_search`, `get_product_by_sku` and `recommend_candidate`.
+
 | Status | When |
 |--------|------|
-| `200`  | The model answered from the retrieved products. Also when the LLM is unavailable (connection, timeout, 429, 5xx): then `fallback_used` is `true`, `usage` is `null` and the text lists the retrieved products. |
+| `200`  | The model answered from the retrieved products. Also when the LLM is unavailable (connection, timeout, 429, 5xx) or its answer is cut off by the token limit or refused: then `fallback_used` is `true`, `usage` is `null` and the text lists the retrieved products. |
 | `422`  | Empty or blank question, more than 1,000 characters, or unknown fields. |
 | `500`  | Misconfiguration: missing or invalid API key, unknown model, invalid request. |
 
@@ -98,7 +107,7 @@ Delivered as small, shippable PRs. Full plan: [`docs/plan-assistant-genai.md`](d
 |------|-------|--------|
 | 1 | `LLMClient` port, Anthropic adapter, `POST /assistant/ask`, failure contract | ✅ Done |
 | 2 | **RAG** with `pgvector` in the existing Postgres: local multilingual embeddings behind a port, incremental `make reindex`, top-k retrieval, `sources` in the answer, deterministic fallback when the LLM fails | ✅ Done |
-| 3 | **Agent** with LangGraph: catalog tools, step limits, token usage summed across steps, offline evals | ⏳ Planned |
+| 3 | **Agent** with LangGraph: candidate selection by rubric and pure `select()` (3a), agent with catalog tools, step limit and summed token usage (3b), evals E1–E4 | 🔄 3a done, 3b in review |
 | 4 | **Deploy**: Kubernetes manifests, local `kind` cluster, optional cloud target | ⏳ Planned |
 
 ---
@@ -178,6 +187,7 @@ src/
         assessment_usage.py            # API calls and tokens per assessment
         rubric.py                      # The two yes/no rubric questions, defined once
         selection.py                   # select(): filter, rank, tie-break, threshold — pure
+        agent.py                       # AgentResult, stop reasons, AgentUnavailableError (partial state)
         exceptions.py                  # InvalidQuestionError, LLMUnavailableError, InvalidAssessmentError
         ports/
           llm_client.py                # LLMClient ABC — documents which failures it raises
@@ -186,17 +196,22 @@ src/
           document_index.py            # Hashes, upsert, delete
           catalog_reader.py            # Products from the catalog, no duplicates
           candidate_assessor.py        # One candidate per call; probabilities in [0, 1]
+          catalog_agent.py             # Step limit, model_incomplete, partial state on failure
       application/
         dtos.py                        # Ask/Answer/Source/Usage/IndexReport/Candidate/Recommendation DTOs
         use_cases/
-          ask_assistant.py             # Embed → retrieve → generate, fallback without generation
+          ask_assistant.py             # Embed → retrieve → generate, or the agent; fallback without generation
           index_catalog.py             # Incremental reindex + orphan deletion (before upsert)
           recommend_candidate.py       # Top-N, deadline, cancellation, select(), one log line
       infrastructure/                  # The only place anthropic, fastembed and pgvector live.
         llm/
           anthropic_errors.py          # Shared transient-vs-configuration classification
           anthropic_llm_client.py      # Timeout, retries, error classification
           claude_candidate_assessor.py # Rubric via structured output; 3 samples; per-call concurrency limit
+        agent/
+          anthropic_agent_model.py     # One messages.create per turn; keeps every content block
+          langgraph_catalog_agent.py   # Graph: model → tools → model, RAG answer past the limit
+          tools.py                     # Tool definitions, dispatch, question embedded once per run
         embeddings/
           fastembed_embedding_client.py
         persistence/
@@ -258,6 +273,7 @@ make test               # Full test suite (never calls a real model)
 make unit-test          # Unit tests only (no I/O)
 make integration-test   # Integration tests (real postgres, mocked LLM transport)
 make e2e-test           # E2E tests (HTTP via TestClient/AsyncClient)
+make eval               # Evals E1–E4 against the real Claude API (reads the key from .env; costs money)
 make lint               # ruff + pylint + mypy
 make format             # auto-format
 make sh                 # Shell into a fresh api container
@@ -267,6 +283,8 @@ make migration m="add foo"  # Generate a new Alembic migration via autogenerate
 
 LLM settings (all in `.env.example`): `ANTHROPIC_API_KEY`, `LLM_MODEL` (default `claude-haiku-5-5`), `LLM_TIMEOUT_S`, `LLM_MAX_RETRIES`, `LLM_MAX_TOKENS`. Retrieval: `EMBEDDING_MODEL` (default `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`) and `RAG_TOP_K` (default 5, max 40). The embedding model (~0.24 GB) is **baked into the Docker image at build time**; at runtime it is only loaded from disk (`EMBEDDING_LOCAL_FILES_ONLY=true`, never downloaded) and warmed up when the app starts, so computing embeddings needs no network (the LLM call still does).
 
+Assistant mode: `ASSISTANT_MODE` (`rag` by default, or `agent`) and `AGENT_MAX_TOOL_CALLS` (default 5). Candidate selector: `SELECTOR_TOP_N` (10), `SELECTOR_SAMPLES` (3), `SELECTOR_MAX_CONCURRENCY` (10), `SELECTOR_TIMEOUT_S` (10), `SELECTOR_MAX_TOKENS` (1024), `SELECTOR_REQUIREMENTS_THRESHOLD` (0.5) and `SELECTOR_MIN_SCORE` (0.5).
+
 API is served on `http://localhost:8000` with auto-generated docs:
 
 - `/docs` — Swagger UI (interactive)
@@ -281,6 +299,7 @@ API is served on `http://localhost:8000` with auto-generated docs:
 |--------------|------------------------|-------------------------------------------------------------|
 | Unit         | `tests/unit/`          | Pure domain and application logic. Mocks at the port boundary via `FakeProductRepository`, `FakeLLMClient`, `FakeEmbeddingClient`, `FakeDocumentIndex` and `FakeDocumentSearch`. Covers incremental indexing and orphan deletion. **No I/O.** |
 | Integration  | `tests/integration/`   | Real `SqlAlchemyProductRepository` against `db-test` (transaction-rollback per test). Real `AnthropicLLMClient` on an `httpx2.MockTransport`: error classification and retry bound. Real pgvector adapters: upsert, hashes, delete, nearest-first search. Catalog reader across pages. |
-| E2E          | `tests/e2e/`           | Full HTTP via `httpx.AsyncClient` + `ASGITransport`. The DB session, `LLMClient`, `EmbeddingClient` and `DocumentSearch` are overridden; covers sources, fallback and that no indexing endpoint exists. Each domain error is checked against its status code. |
+| E2E          | `tests/e2e/`           | Full HTTP via `httpx.AsyncClient` + `ASGITransport`. The DB session, `LLMClient`, `EmbeddingClient` and `DocumentSearch` are overridden; covers sources, fallback and that no indexing endpoint exists. Each domain error is checked against its status code. In agent mode the agent is a `FakeCatalogAgent`, and `rag` mode is checked never to build the agent. |
+| Evals        | `tests/evals/`         | Opt-in (`eval` marker, `make eval`), against the real Claude API. E1: selector accuracy; E2: same outcome with shuffled candidates; E3: same outcome on a rerun; E4: full agent end to end. The scoring rule's own tests are not marked and run in `make test`. |
 
 ---

docs/plan-assistant-genai.md

@@ -315,20 +315,38 @@ rúbrica.
 - Infraestructura — `infrastructure/agent/langgraph_catalog_agent.py`: un grafo pequeño con las
   herramientas `semantic_search` (usa `DocumentSearch`), `get_product_by_sku` (lee el read model) y
   `recommend_candidate` (más abajo), y después un nodo de respuesta final.
-  - El LLM del agente va por `langchain-anthropic`, que es un segundo camino hacia Anthropic fuera de
-    `AnthropicLLMClient`. El adaptador del agente **traduce sus errores a `LLMUnavailableError` con
-    la misma clasificación** que el PR 1 (transitorios frente a configuración) y aplica **el mismo
-    timeout**.
+  - **LangGraph solo orquesta.** El nodo del modelo llama al SDK de Anthropic directamente
+    (tool use) a través de nuestro adaptador en `infrastructure/agent/`, reutilizando **el único
+    cliente** del asistente y **la clasificación de errores del PR 1**. No se usa `langchain-anthropic`:
+    su última versión (1.7.6) exige `anthropic<2.0.0` y el proyecto usa 2.3.0.
+  - **Cada herramienta se define una vez** (esquema + handler); de esa definición se derivan tanto las
+    definiciones de herramientas que se envían a la API como el nodo de herramientas del grafo.
 - Límites — **como mucho 5 llamadas a herramientas por ejecución** (`recommend_candidate` cuenta
-  como una). Al llegar al límite, el agente responde en modo RAG con lo recuperado y registra la
-  razón `max_steps`. Lista blanca de herramientas, el mismo timeout que el PR 1 y el mismo fallback
+  como una), con **un contador en el estado del grafo y una arista condicional** que, al llegar al
+  límite, pasa a responder en modo RAG con lo recuperado y registra la razón `max_steps`.
+  `recursion_limit` queda solo como red de seguridad. Lista blanca de herramientas, el mismo timeout que el PR 1 y el mismo fallback
   que el PR 2, con el uso de tokens sumado entre pasos.
 - `AskAssistant` delega en el agente según un setting (`assistant_mode = "rag" | "agent"`), **`rag`
-  por defecto**: el agente es opt-in hasta que las evals demuestren que es mejor. Los e2e del PR 2
-  no cambian.
+  por defecto**: el agente es opt-in hasta que las evals demuestren que es mejor. El comportamiento en
+  `rag` no cambia; los e2e del PR 2 solo ganan la clave `"recommendation": None` en el diccionario
+  esperado, porque el contrato ganó un campo aditivo.
 - Dependencias — los proveedores de `RecommendCandidate` y del evaluador se añaden aquí (no en 3a),
   reutilizando **el único cliente de Anthropic del asistente**, que se cierra en el lifespan.
-- `pyproject.toml` — añadir `langgraph` y `langchain-anthropic`.
+- `pyproject.toml` — añadir solo `langgraph` (>= 1.2.18). Comprobado con el resolver: con todas las
+  dependencias del proyecto, `anthropic` se queda en 2.3.0 y no entra ningún otro paquete de
+  Anthropic.
+- **Contrato** — `AnswerResponse` gana un campo opcional y nullable `recommendation:
+  {product_id, sku, score, reason} | null`. Es aditivo, así que no rompe a los clientes, y el esquema
+  es el mismo en los dos modos. Sus valores salen de la `Selection` del selector, **nunca del texto
+  del modelo**.
+  - `null` cuando **el selector no se ejecutó**: modo `rag`, o el agente respondió sin recomendar.
+  - **Objeto** cuando se ejecutó. `reason` ∈ `chosen` | `below_threshold` | `no_assessments`, y
+    `product_id`, `sku` y `score` son no nulos **si y solo si** `reason == "chosen"`. Lo impone un
+    validador de Pydantic, con test.
+  - Un e2e por modo.
+- **Razón de parada del agente** — `answered` | `max_steps`. Es un resultado del agente, no de la
+  selección, así que **no va en `recommendation` ni en la API**: va en el DTO del caso de uso (para
+  las evals) y en el log.
 - Tests — unitarios con un agente falso; un test de integración del cableado del grafo con un modelo
   falso guionizado.
 
@@ -340,9 +358,23 @@ SKU y precio). Por ahora, sin pregunta de seguimiento.
 
 ### Evals (marker `eval`, fuera del CI por defecto)
 
-Necesitan una `ANTHROPIC_API_KEY` real, que el usuario pone en `.env`. Su valor nunca aparece en logs,
-salidas de comandos ni documentos. **Antes de la primera ejecución real se informa del número de
-llamadas** que hará.
+Necesitan la `ANTHROPIC_API_KEY` real, que el usuario ya ha puesto en `.env`. Su valor nunca aparece
+en logs, salidas de comandos, documentos ni mensajes de commit. **Antes de la primera ejecución real
+se informa del número de llamadas y del modelo**, y se espera.
+
+**Dataset** (lo propone Claude Code y lo revisa el usuario antes de cualquier ejecución real):
+
+- Un catálogo ficticio de material de montaña (cualquier dominio salvo viajes), unos 20 productos, y
+  unas 10 preguntas.
+- Las preguntas describen **el uso**, sin reutilizar la redacción de las descripciones de producto.
+- Incluye productos casi idénticos que solo se diferencian en un requisito explícito (color,
+  talla), preguntas cuya respuesta correcta es **no recomendar**, y al menos una pregunta
+  **ambigua**.
+- Cada pregunta tiene un **conjunto de SKUs aceptables**, no uno solo.
+- Versionado en `tests/evals/` y cargado de forma determinista en **su propia base de datos**,
+  nunca en los datos del usuario.
+- Los resultados se informan como **recuentos** («8/10»), no como porcentajes. Las evals comparan
+  `recommendation.sku` con los SKUs aceptables.
 
 - Dataset de preguntas con el SKU esperado.
 - Precisión y cobertura (con qué frecuencia recomienda) para varios valores de `selector_min_score`

docs/step-17-assistant-recommend.md

@@ -197,7 +197,7 @@ Verde de las correcciones: `10 passed` (CR1×2, CR2×3, CR3×2, CR4×2, CR6). CR
 | A8 | ✅ | `test_a8_a_reused_sku_does_not_break_the_reindex` (rojo por el orden → verde); `test_a8_sku_is_unique_in_the_index` contra Postgres real (`IntegrityError`) |
 | A9 | ✅ | `test_a9_one_line_per_selection_with_ids_only`: una sola línea con `provider`, `model`, `candidates`, `assessed`, `api_calls`, tokens, `cost_usd`, `latency_ms`, `reason` y `product_id`; no aparecen SKU, nombre ni texto de la pregunta |
 | A10 | ✅ | `grep` sin resultados: el dominio y la aplicación de `assistant` no importan `anthropic`, `sqlalchemy`, `fastapi`, `pgvector`, `fastembed` ni `httpx`. `catalog` sin cambios. Los e2e del PR 2 sin cambios. Ningún `type: ignore`, `noqa` ni `pylint: disable` nuevo. |
-| A11 | ✅ local | `make lint`: ruff OK, pylint 10.00/10, mypy sin errores (80 ficheros). `make test`: **192 passed** (antes 137), sin avisos de excepciones sin recoger. _CI: pendiente de la PR._ |
+| A11 | ✅ | `make lint`: ruff OK, pylint 10.00/10, mypy sin errores (80 ficheros). `make test`: **192 passed** (antes 137), sin avisos de excepciones sin recoger. **CI de la PR #9:** run `40599241763` sobre `6bbbd65`, en verde (login, build, migraciones, lint y tests en ambos jobs). |
 | A12 | ✅ | Plan con la firma real de `select` y las correcciones CR1–CR6; README con el badge de 192 tests y el layout de los ficheros nuevos |
 
 ### Tests nuevos o ampliados