diff --git a/Makefile b/Makefile index f86c00d..8c164f0 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help install run up down stop sh test unit-test integration-test e2e-test lint format build migrate migration migrate-test reindex +.PHONY: help install run up down stop sh test unit-test eval integration-test e2e-test lint format build migrate migration migrate-test reindex DC = docker compose SERVICE = api @@ -30,6 +30,9 @@ test: ## Run the full test suite unit-test: ## Run unit tests only $(DC) run --rm $(SERVICE) pytest tests/unit +eval: ## Run the evals against the real Claude API (reads the key from .env) + $(DC) run --rm -v $(CURDIR)/.env:/app/.env:ro $(SERVICE) pytest -m eval -s -p no:cacheprovider tests/evals + integration-test: ## Run integration tests $(DC) run --rm $(SERVICE) pytest tests/integration diff --git a/README.md b/README.md index a09aad9..008d876 100644 --- a/README.md +++ b/README.md @@ -4,13 +4,14 @@ **Generative AI on FastAPI, done the clean way.** -A reference **Generative AI backend** in Python: a **RAG** assistant that answers from a product catalog, built with **FastAPI**, **Domain-Driven Design (DDD)** and the **Hexagonal (Ports & Adapters)** pattern. The LLM, the embeddings and the vector store are adapters behind ports — swap providers, fake them in tests, and keep them out of your domain. +A reference **Generative AI backend** in Python: a **RAG** assistant that answers from a product catalog, with an optional **LangGraph agent** that recommends products by rules, not by the model's word. Built with **FastAPI**, **Domain-Driven Design (DDD)** and the **Hexagonal (Ports & Adapters)** pattern. The LLM, the embeddings and the vector store are adapters behind ports — swap providers, fake them in tests, and keep them out of your domain. [![CI](https://github.com/jcmoro/python-fastapi/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/jcmoro/python-fastapi/actions/workflows/ci.yml) -![Tests](https://img.shields.io/badge/tests-192%20passing-2e7d32?style=flat) +![Tests](https://img.shields.io/badge/tests-272%20passing-2e7d32?style=flat) ![Architecture](https://img.shields.io/badge/architecture-DDD%20%2B%20Hexagonal-2e7d32?style=flat) ![mypy](https://img.shields.io/badge/mypy-strict-1565c0?style=flat) -![GenAI](https://img.shields.io/badge/GenAI-RAG%20assistant-6a1b9a?style=flat) +![GenAI](https://img.shields.io/badge/GenAI-RAG%20%2B%20agent-6a1b9a?style=flat) +![LangGraph](https://img.shields.io/badge/agent-LangGraph-1c3c3c?style=flat) ![pgvector](https://img.shields.io/badge/vectors-pgvector-336791?style=flat&logo=postgresql&logoColor=white) ![Claude](https://img.shields.io/badge/LLM-Claude-D97757?style=flat&logo=anthropic&logoColor=white) ![Python](https://img.shields.io/badge/Python-3.12-3776AB?style=flat&logo=python&logoColor=white) @@ -40,7 +41,10 @@ Most GenAI demos wire the SDK straight into the route handler. This one treats t - **Retrieval is a port too.** `EmbeddingClient` and `DocumentSearch` are ports; embeddings run locally with a multilingual model (no API key, no cost) and vectors live in the existing Postgres with `pgvector`. - **Indexing is explicit and incremental.** `make reindex` re-embeds only products whose content or embedding model changed (a `content_hash` per document), in batches, and deletes documents only after confirming their product no longer exists. `catalog` doesn't know the assistant exists. - **Tests never call a real model.** Unit and e2e tests use fakes at the port boundary (`FakeLLMClient`, `FakeEmbeddingClient`, `FakeDocumentSearch`). The real LLM adapter is tested against a mocked HTTP transport, and the vector adapters against a real pgvector database. -- **Failures have a contract.** If the LLM is unavailable (connection, timeout, 429, 5xx), the answer still comes back — **200** with the retrieved products and `fallback_used: true`. Configuration errors (bad key, unknown model) are not disguised as outages. +- **Failures have a contract.** If the LLM is unavailable (connection, timeout, 429, 5xx), or its answer is cut off by the token limit or refused, the answer still comes back — **200** with the retrieved products and `fallback_used: true`. Configuration errors (bad key, unknown model) are not disguised as outages. +- **The model doesn't pick the product.** In agent mode, `recommend_candidate` asks Claude two yes/no rubric questions per candidate (3 samples each) and a pure `select()` function filters, ranks and applies thresholds. `recommendation` comes only from that selection, never from the model's text. +- **The agent is bounded.** At most `AGENT_MAX_TOOL_CALLS` tool calls per question; past that it answers in RAG mode with what it retrieved. A transient failure mid-run keeps the recommendation and sources gathered so far. +- **Evals are opt-in and honest.** `make eval` runs the selector and the full agent against a fixed 20-product dataset in its own database, reports results as counts, and stops if the API rate-limits too much. They never run in `make test` or CI. - **Latency is bounded.** `LLM_TIMEOUT_S` applies per attempt and `LLM_MAX_RETRIES` defaults to 0, so the worst case is `timeout * (retries + 1)`. - **Every answer reports its cost.** Responses include the model used, input/output tokens and latency. @@ -76,13 +80,18 @@ Example response (values are illustrative): "input_tokens": 52, "output_tokens": 87, "latency_ms": 1240 - } + }, + "recommendation": null } ``` +`recommendation` is always present. It is `null` in `rag` mode, and in `agent` mode when the agent did not run the selector. When the selector ran, it is an object: `{"product_id", "sku", "score", "reason"}`, where `reason` is `chosen`, `below_threshold` or `no_assessments`, and `product_id`, `sku` and `score` are non-null only when `reason` is `chosen`. + +Agent mode is off by default. Turn it on with `ASSISTANT_MODE=agent`: the agent then has three tools, `semantic_search`, `get_product_by_sku` and `recommend_candidate`. + | Status | When | |--------|------| -| `200` | The model answered from the retrieved products. Also when the LLM is unavailable (connection, timeout, 429, 5xx): then `fallback_used` is `true`, `usage` is `null` and the text lists the retrieved products. | +| `200` | The model answered from the retrieved products. Also when the LLM is unavailable (connection, timeout, 429, 5xx) or its answer is cut off by the token limit or refused: then `fallback_used` is `true`, `usage` is `null` and the text lists the retrieved products. | | `422` | Empty or blank question, more than 1,000 characters, or unknown fields. | | `500` | Misconfiguration: missing or invalid API key, unknown model, invalid request. | @@ -98,7 +107,7 @@ Delivered as small, shippable PRs. Full plan: [`docs/plan-assistant-genai.md`](d |------|-------|--------| | 1 | `LLMClient` port, Anthropic adapter, `POST /assistant/ask`, failure contract | ✅ Done | | 2 | **RAG** with `pgvector` in the existing Postgres: local multilingual embeddings behind a port, incremental `make reindex`, top-k retrieval, `sources` in the answer, deterministic fallback when the LLM fails | ✅ Done | -| 3 | **Agent** with LangGraph: catalog tools, step limits, token usage summed across steps, offline evals | ⏳ Planned | +| 3 | **Agent** with LangGraph: candidate selection by rubric and pure `select()` (3a), agent with catalog tools, step limit and summed token usage (3b), evals E1–E4 | 🔄 3a done, 3b in review | | 4 | **Deploy**: Kubernetes manifests, local `kind` cluster, optional cloud target | ⏳ Planned | --- @@ -178,6 +187,7 @@ src/ assessment_usage.py # API calls and tokens per assessment rubric.py # The two yes/no rubric questions, defined once selection.py # select(): filter, rank, tie-break, threshold — pure + agent.py # AgentResult, stop reasons, AgentUnavailableError (partial state) exceptions.py # InvalidQuestionError, LLMUnavailableError, InvalidAssessmentError ports/ llm_client.py # LLMClient ABC — documents which failures it raises @@ -186,10 +196,11 @@ src/ document_index.py # Hashes, upsert, delete catalog_reader.py # Products from the catalog, no duplicates candidate_assessor.py # One candidate per call; probabilities in [0, 1] + catalog_agent.py # Step limit, model_incomplete, partial state on failure application/ dtos.py # Ask/Answer/Source/Usage/IndexReport/Candidate/Recommendation DTOs use_cases/ - ask_assistant.py # Embed → retrieve → generate, fallback without generation + ask_assistant.py # Embed → retrieve → generate, or the agent; fallback without generation index_catalog.py # Incremental reindex + orphan deletion (before upsert) recommend_candidate.py # Top-N, deadline, cancellation, select(), one log line infrastructure/ # The only place anthropic, fastembed and pgvector live. @@ -197,6 +208,10 @@ src/ anthropic_errors.py # Shared transient-vs-configuration classification anthropic_llm_client.py # Timeout, retries, error classification claude_candidate_assessor.py # Rubric via structured output; 3 samples; per-call concurrency limit + agent/ + anthropic_agent_model.py # One messages.create per turn; keeps every content block + langgraph_catalog_agent.py # Graph: model → tools → model, RAG answer past the limit + tools.py # Tool definitions, dispatch, question embedded once per run embeddings/ fastembed_embedding_client.py persistence/ @@ -258,6 +273,7 @@ make test # Full test suite (never calls a real model) make unit-test # Unit tests only (no I/O) make integration-test # Integration tests (real postgres, mocked LLM transport) make e2e-test # E2E tests (HTTP via TestClient/AsyncClient) +make eval # Evals E1–E4 against the real Claude API (reads the key from .env; costs money) make lint # ruff + pylint + mypy make format # auto-format make sh # Shell into a fresh api container @@ -267,6 +283,8 @@ make migration m="add foo" # Generate a new Alembic migration via autogenerate LLM settings (all in `.env.example`): `ANTHROPIC_API_KEY`, `LLM_MODEL` (default `claude-haiku-5-5`), `LLM_TIMEOUT_S`, `LLM_MAX_RETRIES`, `LLM_MAX_TOKENS`. Retrieval: `EMBEDDING_MODEL` (default `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`) and `RAG_TOP_K` (default 5, max 40). The embedding model (~0.24 GB) is **baked into the Docker image at build time**; at runtime it is only loaded from disk (`EMBEDDING_LOCAL_FILES_ONLY=true`, never downloaded) and warmed up when the app starts, so computing embeddings needs no network (the LLM call still does). +Assistant mode: `ASSISTANT_MODE` (`rag` by default, or `agent`) and `AGENT_MAX_TOOL_CALLS` (default 5). Candidate selector: `SELECTOR_TOP_N` (10), `SELECTOR_SAMPLES` (3), `SELECTOR_MAX_CONCURRENCY` (10), `SELECTOR_TIMEOUT_S` (10), `SELECTOR_MAX_TOKENS` (1024), `SELECTOR_REQUIREMENTS_THRESHOLD` (0.5) and `SELECTOR_MIN_SCORE` (0.5). + API is served on `http://localhost:8000` with auto-generated docs: - `/docs` — Swagger UI (interactive) @@ -281,6 +299,7 @@ API is served on `http://localhost:8000` with auto-generated docs: |--------------|------------------------|-------------------------------------------------------------| | Unit | `tests/unit/` | Pure domain and application logic. Mocks at the port boundary via `FakeProductRepository`, `FakeLLMClient`, `FakeEmbeddingClient`, `FakeDocumentIndex` and `FakeDocumentSearch`. Covers incremental indexing and orphan deletion. **No I/O.** | | Integration | `tests/integration/` | Real `SqlAlchemyProductRepository` against `db-test` (transaction-rollback per test). Real `AnthropicLLMClient` on an `httpx2.MockTransport`: error classification and retry bound. Real pgvector adapters: upsert, hashes, delete, nearest-first search. Catalog reader across pages. | -| E2E | `tests/e2e/` | Full HTTP via `httpx.AsyncClient` + `ASGITransport`. The DB session, `LLMClient`, `EmbeddingClient` and `DocumentSearch` are overridden; covers sources, fallback and that no indexing endpoint exists. Each domain error is checked against its status code. | +| E2E | `tests/e2e/` | Full HTTP via `httpx.AsyncClient` + `ASGITransport`. The DB session, `LLMClient`, `EmbeddingClient` and `DocumentSearch` are overridden; covers sources, fallback and that no indexing endpoint exists. Each domain error is checked against its status code. In agent mode the agent is a `FakeCatalogAgent`, and `rag` mode is checked never to build the agent. | +| Evals | `tests/evals/` | Opt-in (`eval` marker, `make eval`), against the real Claude API. E1: selector accuracy; E2: same outcome with shuffled candidates; E3: same outcome on a rerun; E4: full agent end to end. The scoring rule's own tests are not marked and run in `make test`. | --- diff --git a/docs/plan-assistant-genai.md b/docs/plan-assistant-genai.md index fa572d0..1dfca15 100644 --- a/docs/plan-assistant-genai.md +++ b/docs/plan-assistant-genai.md @@ -315,20 +315,38 @@ rúbrica. - Infraestructura — `infrastructure/agent/langgraph_catalog_agent.py`: un grafo pequeño con las herramientas `semantic_search` (usa `DocumentSearch`), `get_product_by_sku` (lee el read model) y `recommend_candidate` (más abajo), y después un nodo de respuesta final. - - El LLM del agente va por `langchain-anthropic`, que es un segundo camino hacia Anthropic fuera de - `AnthropicLLMClient`. El adaptador del agente **traduce sus errores a `LLMUnavailableError` con - la misma clasificación** que el PR 1 (transitorios frente a configuración) y aplica **el mismo - timeout**. + - **LangGraph solo orquesta.** El nodo del modelo llama al SDK de Anthropic directamente + (tool use) a través de nuestro adaptador en `infrastructure/agent/`, reutilizando **el único + cliente** del asistente y **la clasificación de errores del PR 1**. No se usa `langchain-anthropic`: + su última versión (1.7.6) exige `anthropic<2.0.0` y el proyecto usa 2.3.0. + - **Cada herramienta se define una vez** (esquema + handler); de esa definición se derivan tanto las + definiciones de herramientas que se envían a la API como el nodo de herramientas del grafo. - Límites — **como mucho 5 llamadas a herramientas por ejecución** (`recommend_candidate` cuenta - como una). Al llegar al límite, el agente responde en modo RAG con lo recuperado y registra la - razón `max_steps`. Lista blanca de herramientas, el mismo timeout que el PR 1 y el mismo fallback + como una), con **un contador en el estado del grafo y una arista condicional** que, al llegar al + límite, pasa a responder en modo RAG con lo recuperado y registra la razón `max_steps`. + `recursion_limit` queda solo como red de seguridad. Lista blanca de herramientas, el mismo timeout que el PR 1 y el mismo fallback que el PR 2, con el uso de tokens sumado entre pasos. - `AskAssistant` delega en el agente según un setting (`assistant_mode = "rag" | "agent"`), **`rag` - por defecto**: el agente es opt-in hasta que las evals demuestren que es mejor. Los e2e del PR 2 - no cambian. + por defecto**: el agente es opt-in hasta que las evals demuestren que es mejor. El comportamiento en + `rag` no cambia; los e2e del PR 2 solo ganan la clave `"recommendation": None` en el diccionario + esperado, porque el contrato ganó un campo aditivo. - Dependencias — los proveedores de `RecommendCandidate` y del evaluador se añaden aquí (no en 3a), reutilizando **el único cliente de Anthropic del asistente**, que se cierra en el lifespan. -- `pyproject.toml` — añadir `langgraph` y `langchain-anthropic`. +- `pyproject.toml` — añadir solo `langgraph` (>= 1.2.18). Comprobado con el resolver: con todas las + dependencias del proyecto, `anthropic` se queda en 2.3.0 y no entra ningún otro paquete de + Anthropic. +- **Contrato** — `AnswerResponse` gana un campo opcional y nullable `recommendation: + {product_id, sku, score, reason} | null`. Es aditivo, así que no rompe a los clientes, y el esquema + es el mismo en los dos modos. Sus valores salen de la `Selection` del selector, **nunca del texto + del modelo**. + - `null` cuando **el selector no se ejecutó**: modo `rag`, o el agente respondió sin recomendar. + - **Objeto** cuando se ejecutó. `reason` ∈ `chosen` | `below_threshold` | `no_assessments`, y + `product_id`, `sku` y `score` son no nulos **si y solo si** `reason == "chosen"`. Lo impone un + validador de Pydantic, con test. + - Un e2e por modo. +- **Razón de parada del agente** — `answered` | `max_steps`. Es un resultado del agente, no de la + selección, así que **no va en `recommendation` ni en la API**: va en el DTO del caso de uso (para + las evals) y en el log. - Tests — unitarios con un agente falso; un test de integración del cableado del grafo con un modelo falso guionizado. @@ -340,9 +358,23 @@ SKU y precio). Por ahora, sin pregunta de seguimiento. ### Evals (marker `eval`, fuera del CI por defecto) -Necesitan una `ANTHROPIC_API_KEY` real, que el usuario pone en `.env`. Su valor nunca aparece en logs, -salidas de comandos ni documentos. **Antes de la primera ejecución real se informa del número de -llamadas** que hará. +Necesitan la `ANTHROPIC_API_KEY` real, que el usuario ya ha puesto en `.env`. Su valor nunca aparece +en logs, salidas de comandos, documentos ni mensajes de commit. **Antes de la primera ejecución real +se informa del número de llamadas y del modelo**, y se espera. + +**Dataset** (lo propone Claude Code y lo revisa el usuario antes de cualquier ejecución real): + +- Un catálogo ficticio de material de montaña (cualquier dominio salvo viajes), unos 20 productos, y + unas 10 preguntas. +- Las preguntas describen **el uso**, sin reutilizar la redacción de las descripciones de producto. +- Incluye productos casi idénticos que solo se diferencian en un requisito explícito (color, + talla), preguntas cuya respuesta correcta es **no recomendar**, y al menos una pregunta + **ambigua**. +- Cada pregunta tiene un **conjunto de SKUs aceptables**, no uno solo. +- Versionado en `tests/evals/` y cargado de forma determinista en **su propia base de datos**, + nunca en los datos del usuario. +- Los resultados se informan como **recuentos** («8/10»), no como porcentajes. Las evals comparan + `recommendation.sku` con los SKUs aceptables. - Dataset de preguntas con el SKU esperado. - Precisión y cobertura (con qué frecuencia recomienda) para varios valores de `selector_min_score` diff --git a/docs/step-17-assistant-recommend.md b/docs/step-17-assistant-recommend.md index d42e527..0356e44 100644 --- a/docs/step-17-assistant-recommend.md +++ b/docs/step-17-assistant-recommend.md @@ -197,7 +197,7 @@ Verde de las correcciones: `10 passed` (CR1×2, CR2×3, CR3×2, CR4×2, CR6). CR | A8 | ✅ | `test_a8_a_reused_sku_does_not_break_the_reindex` (rojo por el orden → verde); `test_a8_sku_is_unique_in_the_index` contra Postgres real (`IntegrityError`) | | A9 | ✅ | `test_a9_one_line_per_selection_with_ids_only`: una sola línea con `provider`, `model`, `candidates`, `assessed`, `api_calls`, tokens, `cost_usd`, `latency_ms`, `reason` y `product_id`; no aparecen SKU, nombre ni texto de la pregunta | | A10 | ✅ | `grep` sin resultados: el dominio y la aplicación de `assistant` no importan `anthropic`, `sqlalchemy`, `fastapi`, `pgvector`, `fastembed` ni `httpx`. `catalog` sin cambios. Los e2e del PR 2 sin cambios. Ningún `type: ignore`, `noqa` ni `pylint: disable` nuevo. | -| A11 | ✅ local | `make lint`: ruff OK, pylint 10.00/10, mypy sin errores (80 ficheros). `make test`: **192 passed** (antes 137), sin avisos de excepciones sin recoger. _CI: pendiente de la PR._ | +| A11 | ✅ | `make lint`: ruff OK, pylint 10.00/10, mypy sin errores (80 ficheros). `make test`: **192 passed** (antes 137), sin avisos de excepciones sin recoger. **CI de la PR #9:** run `40599241763` sobre `6bbbd65`, en verde (login, build, migraciones, lint y tests en ambos jobs). | | A12 | ✅ | Plan con la firma real de `select` y las correcciones CR1–CR6; README con el badge de 192 tests y el layout de los ficheros nuevos | ### Tests nuevos o ampliados diff --git a/docs/step-18-assistant-agent.md b/docs/step-18-assistant-agent.md index 872fc53..23d592b 100644 --- a/docs/step-18-assistant-agent.md +++ b/docs/step-18-assistant-agent.md @@ -22,8 +22,11 @@ por defecto, todo sigue igual que en el PR 2. 2. Como mucho 5 llamadas a herramientas por ejecución (`recommend_candidate` cuenta como una). Al llegar al límite, el agente responde en modo RAG con lo recuperado y registra la razón `max_steps`. -3. El LLM del agente va por `langchain-anthropic`. Sus errores se traducen a `LLMUnavailableError` - con la clasificación del PR 1, y se aplica el mismo timeout. +3. LangGraph solo orquesta. El nodo del modelo llama al SDK de Anthropic directamente (tool use) a + través de nuestro adaptador, con el único cliente del asistente y la clasificación de errores del + PR 1. Cada herramienta se define una vez (esquema + handler). +3b. `AnswerResponse` gana `recommendation: {product_id, sku, score, reason} | null`, que sale de la + `Selection`, nunca del texto del modelo. 4. `assistant_mode` (`rag` | `agent`), con `rag` por defecto. 5. Evals con el marker `eval`, fuera del CI. @@ -32,9 +35,15 @@ por defecto, todo sigue igual que en el PR 2. | Tema | Decisión | |---|---| | Límite de pasos | 5 llamadas a herramientas. Al llegar, respuesta en modo RAG y razón `max_steps` en el log. | -| `assistant_mode` | `rag` por defecto; los e2e del PR 2 no cambian. | +| `assistant_mode` | `rag` por defecto; el comportamiento en `rag` no cambia. En el JSON, `recommendation` está siempre presente y vale `null` en `rag`: omitir la clave crearía un tercer estado. Por eso un e2e del PR 2 gana esa clave en el diccionario esperado. | | Sin recomendación | «No hay una recomendación clara» y el top-k con nombre, SKU y precio, sin pregunta de seguimiento. | -| LLM del agente | Errores de `langchain-anthropic` traducidos con la misma clasificación que el PR 1, con el mismo timeout. | +| LLM del agente | **Sin `langchain-anthropic`**: su última versión (1.7.6) exige `anthropic<2.0.0`. El nodo del modelo usa el SDK de Anthropic a través de nuestro adaptador, con el único cliente y la clasificación del PR 1. | +| Herramientas | Cada una se define **una vez** (esquema + handler); de ahí salen las definiciones que se envían a la API y el nodo de herramientas. | +| Límite de pasos (mecanismo) | Contador en el estado del grafo y arista condicional hacia el modo RAG; `recursion_limit` solo como red de seguridad. | +| Dependencias | Solo `langgraph` (>= 1.2.18). Resolver en seco con todas las dependencias del proyecto: `anthropic` 2.3.0, `langchain-core` 1.7.0 y `langchain-protocol`, sin `langchain-anthropic` y sin conflictos. | +| Contrato | `recommendation: {product_id, sku, score, reason} \| null` en `AnswerResponse`: opcional y aditivo. **`null` si el selector no se ejecutó** (modo `rag`, o el agente respondió sin recomendar). **Objeto si se ejecutó**: `reason` ∈ `chosen` \| `below_threshold` \| `no_assessments`, y `product_id`, `sku` y `score` son no nulos **si y solo si** `reason == "chosen"`, con un validador de Pydantic. Sale de la `Selection`, nunca del texto. | +| Razón de parada del agente | `answered` \| `max_steps`. Es un resultado del agente, no de la selección: va en el DTO del caso de uso y en el log, **no en la API** por ahora. | +| Dataset de evals | Material de montaña (cualquier dominio salvo viajes), ~20 productos y ~10 preguntas que describen el uso. Incluye productos casi idénticos que difieren en un requisito explícito, casos de no recomendar y al menos una pregunta ambigua. Conjuntos de SKUs aceptables. En `tests/evals/`, cargado de forma determinista en su propia BD. Resultados en recuentos. El usuario lo revisa antes de cualquier ejecución real. | | Clave para las evals | El usuario pone `ANTHROPIC_API_KEY` en `.env`. **Su valor nunca aparece en logs, salidas de comandos ni documentos.** | | Primera ejecución real de evals | **Antes de lanzarla se informa al usuario del número de llamadas y del modelo** que usará. | | Cliente de Anthropic | Los proveedores de `RecommendCandidate` y de `ClaudeCandidateAssessor` se crean aquí y reutilizan el único cliente de Anthropic del asistente, que se cierra en el lifespan (decidido en la code review del 3a). | @@ -45,14 +54,165 @@ por defecto, todo sigue igual que en el PR 2. |---|---|---| | B1 | Sin recomendación, la respuesta dice que no hay una clara y lista el top-k con nombre, SKU y precio | Test del agente | | B2 | Al llegar a la 5.ª llamada a herramienta se corta, responde en modo RAG y el log registra `max_steps` | Test del grafo con un modelo falso guionizado | -| B3 | `assistant_mode=rag` es el valor por defecto, y los e2e del PR 2 pasan **sin modificarse** | `git diff` vacío en esos tests y suite en verde | -| B4 | Los errores del LLM del agente se clasifican como en el PR 1: los transitorios llevan al fallback y los de configuración se propagan | Test del adaptador del agente con errores simulados | -| B5 | El dominio y la aplicación de `assistant` no importan `langgraph` ni `langchain` | `grep` | +| B3 | `assistant_mode=rag` es el valor por defecto y el comportamiento en `rag` no cambia. Los e2e del PR 2 solo ganan la clave `"recommendation": None` en el diccionario esperado de `test_c7_returns_200_with_answer_sources_and_usage`, porque el contrato ganó un campo aditivo | `git diff` de esos tests limitado a esa línea, y suite en verde | +| B4 | Los errores del LLM del agente se clasifican como en el PR 1: los transitorios llevan al fallback y los de configuración se propagan | Test del adaptador del agente con errores simulados sobre `httpx2.MockTransport` | +| B5 | El dominio y la aplicación de `assistant` no importan `langgraph` ni `langchain`, y `langchain-anthropic` no está entre las dependencias | `grep` y `pip show` | +| B10 | `recommendation` es `null` si el selector no se ejecutó, y un objeto sacado de la `Selection` si se ejecutó. El texto del modelo nunca altera el campo. | Un e2e por modo (`rag`: `null`; `agent`: objeto) y un test con un modelo que «dice» otro SKU en el texto | +| B14 | Invariante: `product_id`, `sku` y `score` no nulos **si y solo si** `reason == "chosen"` | Test del validador de Pydantic: los cuatro casos inválidos dan error | +| B15 | La razón de parada del agente (`answered` \| `max_steps`) está en el DTO del caso de uso y en el log, y **no** en la respuesta HTTP | Test del DTO, test con `caplog` y un e2e que comprueba que el campo no está en el JSON | +| B11 | Cada herramienta se define una vez, y las definiciones de la API y el nodo de herramientas salen de ahí | Test que compara ambas con la definición única | +| B12 | El límite de pasos lo aplica el contador del estado y la arista condicional, no `recursion_limit` | Test del grafo | +| B13 | El dataset cumple sus reglas (dominio, uso en lugar de descripción, casi idénticos, no recomendar, ambigua, conjuntos aceptables) y se carga de forma determinista en su propia BD | Revisión del usuario y test de carga | | B6 | Las evals existen con el marker `eval` y el CI por defecto no las ejecuta | Recolección de pytest y CI de la PR | | B7 | Antes de la primera ejecución real se informa del número de llamadas, y el valor de la clave no aparece en ninguna salida | Mensaje al usuario antes de ejecutar, y revisión de las salidas | | B8 | Los quality gates y el CI pasan | `make lint`, `make test` y CI de la PR | | B9 | El plan y el README coinciden con el código | Revisión, anclas y badges | +## Plan de evals + +**Dataset** — `tests/evals/dataset.json`, versión 2, aprobado por el usuario con dos cambios: + +- 20 productos de material de montaña con **SKUs opacos** (`MT-0001`…`MT-0020`), para que el + modelo no pueda leer la talla ni el color en el código. +- 10 preguntas: 7 `recommend`, 2 `no_recommendation` y 1 `ambiguous`, cada una con su conjunto de + SKUs aceptables. +- Se carga de forma determinista en su propia base de datos (`catalog_eval`, en el servidor de + `db-test`), nunca en los datos del usuario. + +**Modelos** — las evaluaciones y el agente usan `claude-haiku-5-5` (`LLM_MODEL` por defecto; `.env` +no lo cambia). Los embeddings usan `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`: +comprobado que es **multilingüe** (~50 idiomas), y ni `.env` ni la imagen lo cambian, así que el +dataset en español no necesita otro modelo. + +**Regla de puntuación** (decidida por el usuario): + +| Tipo de pregunta | Cuenta como acierto | Cuenta como fallo | +|---|---|---| +| `recommend` | `reason = chosen` y SKU en el conjunto aceptable | otro SKU, `below_threshold`, `null` o `no_assessments` | +| con `accept_no_recommendation = true` (`no_recommendation`, `ambiguous`) | `reason = below_threshold`, `recommendation = null` (solo en E4), o, en la ambigua, `chosen` con un SKU aceptable | `chosen` con un SKU no aceptable, o `no_assessments` | + +Cada resultado registra **qué ocurrió** (`chosen:`, `below_threshold`, `null` o +`no_assessments`). `no_assessments` siempre cuenta como fallo: es un fallo del modelo, no un juicio, +y se informa aparte. Los resultados se dan en **recuentos** (por ejemplo «7/10: recommend 5/7, +no_recommendation 2/2, ambiguous 0/1»), nunca en porcentajes. + +| Eval | Qué mide y cómo | Llamadas reales | +|---|---|---| +| **E1 · Selector** | `RecommendCandidate` con `ClaudeCandidateAssessor` real sobre cada pregunta: top 10 de la recuperación × 3 muestras. Precisión según la regla. Con las mismas evaluaciones se recalcula `select` (función pura, **sin llamadas nuevas**) para la tabla de umbrales: `requirements_threshold` y `min_score` ∈ {0,33; 0,67; 1,0}, con precisión y cobertura (cuántas veces recomienda) en recuentos. Con 3 muestras, 0,5 equivale a 0,67. | 300 | +| **E2 · Invariancia al orden** | E1 otra vez con los candidatos barajados; cuántas preguntas repiten el mismo resultado que en E1 | 300 | +| **E3 · Repetición** | E1 otra vez en el mismo orden; cuántas preguntas repiten el resultado de E1 | 300 | +| **E4 · Agente completo** | `AskAssistant` en modo `agent`, extremo a extremo, comparando `recommendation` con el conjunto aceptable. También registra la razón de parada, las llamadas a herramientas y la latencia. | ~330 | + +Además, en cada eval: latencia por selección o por pregunta, tokens, y llamadas contestadas frente a +llamadas hechas. El coste de Claude se **estima** con los tokens y el precio de lista, porque la API +no lo devuelve. + +**Hallazgo sobre E2 y E3, para decidir sobre E3.** Con este diseño, **barajar los candidatos no +cambia lo que ve el modelo**: + +- Cada llamada evalúa un solo candidato. +- `RecommendCandidate` reordena la entrada por `retrieval_score` antes de evaluar. + +Así que E2 manda exactamente las mismas 300 peticiones que una repetición, y E2 y E3 miden lo +mismo: la variabilidad del modelo entre ejecuciones. La invariancia al orden **del código** ya la +demuestran los tests unitarios sin coste (A2 y la entrada barajada en `RecommendCandidate`). La +invariancia al orden **del modelo** está garantizada por construcción, porque nunca ve dos +candidatos a la vez. + +**Propuesta:** quedarse con una sola reejecución. Llamarla «E2 · Estabilidad (orden barajado)» +cubre ambas preguntas con 300 llamadas y deja el total en unas **930**. + +**Decisión del usuario:** se ejecutan E2 y E3. Total previsto: ~1.230 llamadas. + +### Harness + +- **Ficheros:** `tests/evals/` (`harness.py`, `conftest.py`, `test_eval_selector.py` con E1–E3, + `test_eval_agent.py` con E4). Llevan el marcador `eval`, que `pytest` excluye por defecto, y se + lanzan con `make eval`. +- **Clave:** `make eval` monta `.env` en solo lectura. La clave se lee del fichero y no aparece en + ningún comando. +- **Base de datos:** se crea `catalog_eval` en el servidor de `db-test` y se migra con `alembic`. + Después se cargan los 20 productos con IDs `uuid5` fijos y se indexan con los embeddings locales. +- **Informe:** + - una línea por eval que dice qué mide; + - aciertos en recuentos por tipo de pregunta; + - tokens y coste estimado; + - llamadas contestadas frente a hechas, y los 429. + - E1 añade la tabla de umbrales, recalculada con `select` sobre las mismas evaluaciones. Usa + fracciones exactas (1/3, 2/3, 1) para que «0,67» signifique «2 de 3». +- **Parada por 429:** la ejecución se detiene con un informe parcial si los 429 superan el 5 % de + las peticiones. +- **Tests de la regla de puntuación:** `test_eval_scoring.py` no lleva el marcador y corre en + `make test`. +- **Techo de recuperación** (comprobado sin llamadas a la API): el SKU aceptable no está en el + top 10 de la recuperación en q06 (`MT-0020`) ni en q10 (`MT-0017`). En q09 faltan 2 de sus 4. + Por eso E1–E3 llegan como mucho a 8/10, y el informe lo muestra en la columna `reachable`. + +## Correcciones tras la code review (3b) + +La revisión (`/code-review`, lanzada por el usuario) dio 10 hallazgos y se comprobó cada uno contra el +código. Estas son las decisiones del usuario: + +| # | Qué | Por qué | Criterio | +|---|---|---|---| +| RB1 (#1) | El agente solo se construye en modo `agent` | En `rag` se construían el evaluador, el selector y el grafo para nada, y un fallo de configuración del agente rompía `rag` | En modo `rag` no se llama a `get_catalog_agent` | +| RB2 (#3) | Si el agente falla después de ejecutar el selector, el fallback conserva `recommendation`, reutiliza las `sources` ya recuperadas y el error lleva el uso acumulado hasta el fallo (como CR4) | B10: `recommendation` es un objeto siempre que el selector se ejecutó. Además se evita una recuperación repetida. | Test: el selector elige X, el siguiente turno da 529 → respuesta con `fallback_used=true`, `recommendation` = X y el uso registrado | +| RB3 (#4) | `find_by_sku(sku, embedding)` devuelve la **similitud coseno real** contra el embedding de la pregunta, en la misma consulta. El producto entra en `sources`, y si ya estaba se conserva la entrada existente. **La pregunta se embebe una vez por petición** y se reutiliza en las herramientas. | `score` mantiene un único significado en todas las `sources` sin inventar valores | Tests del adaptador y de la herramienta | +| RB4 (#5) | Si el modelo termina con `max_tokens` o `refusal`: razón de parada nueva, **`model_incomplete`**, con el `stop_reason` original. Se responde con el **fallback RAG determinista** (`fallback_used=true`). Queda en el DTO y en el log, no en la API. **No** se usa `LLMUnavailableError`, que significa fallo transitorio. **El modo `rag` se comporta igual.** | Hoy, en los dos modos, una respuesta cortada o una negativa salen como una respuesta normal con 200 (comprobado: `AnthropicLLMClient` ignora `stop_reason`) | Tests en los dos modos: `max_tokens` → fallback; `refusal` → fallback; y la razón queda en el DTO | +| RB5 (#8) | `usage.model` es el modelo que devuelve la API, también en modo `agent` | En `rag` ya era así; en `agent` era el nombre configurado | Test del agente | +| RB6 (#9) | Una sola llamada a `messages.create` con `omit` cuando no hay herramientas, en un **commit de refactor aparte** | Elimina la duplicación del `if/else` | Los tests existentes siguen en verde | +| Aplazado (#7) | Herramientas del mismo turno en paralelo | Se decide cuando las evals midan la latencia. Si se hace, **el límite de concurrencia de llamadas debe compartirse por petición**, para que dos `recommend_candidate` en paralelo no dupliquen el presupuesto. | — | + +**Sin cambios:** +- **#2 (overrides de `get_catalog_agent`):** es el mismo patrón que `get_llm_client`, y deja de importar con RB1. +- **#6 (cerrar el cliente con varias apps en el mismo proceso):** no ocurre en producción, y en los tests secuenciales las cachés se limpian al cerrar. +- **#10 (docstrings en los puertos):** falso positivo. El `CLAUDE.md` del repo los permite en los puertos. + +**Ajuste de B3:** el comportamiento de `rag` solo cambia en una cosa (RB4). Una respuesta cortada por +`max_tokens` o una negativa ya no salen como una respuesta normal, sino como el fallback determinista. + +### Cómo se implementaron (commit 3) + +- **RB1:** el nuevo proveedor `get_mode_catalog_agent` depende de `get_assistant_mode` y solo llama a + `get_catalog_agent()` en modo `agent`. En `rag` devuelve `None`. `get_ask_assistant_use_case` + depende de él. Por eso los e2e del modo `agent` sobrescriben `get_mode_catalog_agent` en lugar de + `get_catalog_agent` y `get_assistant_mode`. +- **RB2:** el error `AgentUnavailableError(LLMUnavailableError)` lleva `sources`, `recommendation` y + `usage`. + - Vive en `domain/agent.py`, junto a `AgentResult`. En `domain/exceptions.py` creaba un ciclo de + imports: `agent → selection → candidate → exceptions → agent`. + - Los nodos `model` y `rag_answer` envuelven el fallo transitorio con el estado de ese momento, y + `run()` construye el error con el uso acumulado. + - El log es la misma línea `agent_run`, con `stop_reason=unavailable`. Ese valor solo aparece en + el log. + - `AskAssistant` reutiliza las `sources` del error y solo recupera si vienen vacías, por ejemplo + cuando falla el primer turno. +- **RB3:** + - La pregunta se embebe una vez al principio de `run()` y viaja en `ToolContext.question_embedding`. + - `find_by_sku` calcula la distancia `<=>` en la misma consulta. + - `IndexedProduct` desaparece: el puerto devuelve `RetrievedDocument`. +- **RB4:** + - `INCOMPLETE_STOP_REASONS = {"max_tokens", "refusal"}`, en `domain/completion.py`. + - `Completion` y `ModelTurn` exponen el `stop_reason` de la API. + - Un turno incompleto no ejecuta sus herramientas. + - `AnswerDTO.model_stop_reason` y el campo `model_stop_reason` del log guardan la razón. + - El texto OpenAPI de `fallback_used` y el README describen el nuevo caso. +- **RB5:** el estado del grafo guarda el `model` del último turno. Si ningún turno respondió, se usa + el nombre configurado. + +Evidencia (rojo → verde): + +- **Rojo:** con el código sin modificar fallaban las 17 pruebas RB. + - Las de `test_ask_assistant.py`, `test_agent_tools.py` y `test_langgraph_catalog_agent.py`, en la + recolección: `ImportError: cannot import name 'AgentUnavailableError'` y + `TypeError: ToolContext.__init__() got an unexpected keyword argument 'question_embedding'`. + - `test_rb1_rag_mode_never_builds_the_agent`, con `assert 1 == 0` en `currsize`. + - Las de los adaptadores: `'ModelTurn' object has no attribute 'stop_reason'`, + `'Completion' object has no attribute 'stop_reason'` y + `find_by_sku() takes 2 positional arguments but 3 were given`. +- **Verde:** pasan las 17 pruebas RB, `make lint` (ruff, pylint 10.00/10 y mypy sin incidencias) y + `make test` (252 pruebas). + ## Evidencia _Se completa al terminar._ diff --git a/main.py b/main.py index 084b639..6412817 100644 --- a/main.py +++ b/main.py @@ -1,5 +1,5 @@ -from collections.abc import AsyncIterator -from contextlib import asynccontextmanager +from collections.abc import AsyncIterator, Callable +from contextlib import AbstractAsyncContextManager, asynccontextmanager from fastapi import FastAPI @@ -10,10 +10,17 @@ from shared.infrastructure.settings import get_settings -@asynccontextmanager -async def _warm_up_models(_application: FastAPI) -> AsyncIterator[None]: - await assistant_lifespan.warm_up_embeddings() - yield +def _lifespan(*, warm_up_models: bool) -> Callable[[FastAPI], AbstractAsyncContextManager[None]]: + @asynccontextmanager + async def lifespan(_application: FastAPI) -> AsyncIterator[None]: + if warm_up_models: + await assistant_lifespan.warm_up_embeddings() + try: + yield + finally: + await assistant_lifespan.close_anthropic_client() + + return lifespan def create_app(*, warm_up_models: bool = False) -> FastAPI: @@ -26,7 +33,7 @@ def create_app(*, warm_up_models: bool = False) -> FastAPI: docs_url="/docs", redoc_url="/redoc", openapi_url="/openapi.json", - lifespan=_warm_up_models if warm_up_models else None, + lifespan=_lifespan(warm_up_models=warm_up_models), ) @application.get( diff --git a/pyproject.toml b/pyproject.toml index 985c896..66dd8c4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,6 +19,7 @@ dependencies = [ "anthropic>=2.3", "pgvector>=0.5.1", "fastembed>=0.9.1", + "langgraph>=1.2.18", ] [project.optional-dependencies] @@ -43,7 +44,8 @@ testpaths = ["tests"] asyncio_mode = "auto" asyncio_default_fixture_loop_scope = "session" asyncio_default_test_loop_scope = "session" -addopts = "-q --strict-markers" +addopts = "-q --strict-markers -m 'not eval'" +markers = ["eval: evaluations against the real Claude API; run with make eval"] pythonpath = ["."] [tool.ruff] diff --git a/src/contexts/assistant/application/dtos.py b/src/contexts/assistant/application/dtos.py index 8608805..c046c4d 100644 --- a/src/contexts/assistant/application/dtos.py +++ b/src/contexts/assistant/application/dtos.py @@ -3,6 +3,7 @@ from dataclasses import dataclass from uuid import UUID +from contexts.assistant.domain.agent import AgentRecommendation, AgentStopReason from contexts.assistant.domain.candidate import Candidate from contexts.assistant.domain.completion import Completion from contexts.assistant.domain.retrieved_document import RetrievedDocument @@ -33,12 +34,32 @@ def from_retrieved(cls, document: RetrievedDocument) -> SourceDTO: return cls(product_id=document.product_id, content=document.content, score=document.score) +@dataclass(frozen=True, slots=True) +class RecommendationSummaryDTO: + product_id: UUID | None + sku: str | None + score: float | None + reason: SelectionReason + + @classmethod + def from_agent(cls, recommendation: AgentRecommendation) -> RecommendationSummaryDTO: + return cls( + product_id=recommendation.product_id, + sku=recommendation.sku, + score=recommendation.score, + reason=recommendation.reason, + ) + + @dataclass(frozen=True, slots=True) class AnswerDTO: text: str sources: tuple[SourceDTO, ...] fallback_used: bool usage: UsageDTO | None + recommendation: RecommendationSummaryDTO | None = None + stop_reason: AgentStopReason | None = None + model_stop_reason: str | None = None @classmethod def from_completion(cls, completion: Completion, sources: tuple[SourceDTO, ...]) -> AnswerDTO: @@ -55,8 +76,24 @@ def from_completion(cls, completion: Completion, sources: tuple[SourceDTO, ...]) ) @classmethod - def fallback(cls, text: str, sources: tuple[SourceDTO, ...]) -> AnswerDTO: - return cls(text=text, sources=sources, fallback_used=True, usage=None) + def fallback( + cls, + text: str, + sources: tuple[SourceDTO, ...], + *, + recommendation: RecommendationSummaryDTO | None = None, + stop_reason: AgentStopReason | None = None, + model_stop_reason: str | None = None, + ) -> AnswerDTO: + return cls( + text=text, + sources=sources, + fallback_used=True, + usage=None, + recommendation=recommendation, + stop_reason=stop_reason, + model_stop_reason=model_stop_reason, + ) @dataclass(frozen=True, slots=True) @@ -90,6 +127,7 @@ def from_candidate(cls, candidate: Candidate) -> CandidateDTO: @dataclass(frozen=True, slots=True) class RecommendationDTO: product_id: UUID | None + sku: str | None score: float | None reason: SelectionReason fallback_used: bool diff --git a/src/contexts/assistant/application/use_cases/ask_assistant.py b/src/contexts/assistant/application/use_cases/ask_assistant.py index 3fca6ce..5b10f3c 100644 --- a/src/contexts/assistant/application/use_cases/ask_assistant.py +++ b/src/contexts/assistant/application/use_cases/ask_assistant.py @@ -1,9 +1,25 @@ -from contexts.assistant.application.dtos import AnswerDTO, AskCommand, SourceDTO +from collections.abc import Sequence + +from contexts.assistant.application.dtos import ( + AnswerDTO, + AskCommand, + RecommendationSummaryDTO, + SourceDTO, + UsageDTO, +) +from contexts.assistant.domain.agent import ( + AgentRecommendation, + AgentStopReason, + AgentUnavailableError, +) +from contexts.assistant.domain.completion import INCOMPLETE_STOP_REASONS from contexts.assistant.domain.exceptions import LLMUnavailableError +from contexts.assistant.domain.ports.catalog_agent import CatalogAgent from contexts.assistant.domain.ports.document_search import DocumentSearch from contexts.assistant.domain.ports.embedding_client import EmbeddingClient from contexts.assistant.domain.ports.llm_client import LLMClient from contexts.assistant.domain.question import Question +from contexts.assistant.domain.retrieved_document import RetrievedDocument SYSTEM_PROMPT = ( "You are the assistant of a product catalog. " @@ -26,39 +42,95 @@ def __init__( llm_client: LLMClient, embedding_client: EmbeddingClient, document_search: DocumentSearch, + *, max_tokens: int, top_k: int, + agent: CatalogAgent | None = None, ) -> None: self._llm_client = llm_client self._embedding_client = embedding_client self._document_search = document_search self._max_tokens = max_tokens self._top_k = top_k + self._agent = agent async def execute(self, command: AskCommand) -> AnswerDTO: question = Question.create(command.question) - [embedding] = await self._embedding_client.embed([question.text]) - documents = await self._document_search.search(embedding, self._top_k) - sources = tuple(SourceDTO.from_retrieved(document) for document in documents) + if self._agent is not None: + return await self._ask_agent(self._agent, question.text) + + sources = await self._retrieve(question.text) try: completion = await self._llm_client.complete( system=SYSTEM_PROMPT, - prompt=_build_prompt(question.text, sources), + prompt=build_rag_prompt(question.text, sources), max_tokens=self._max_tokens, ) except LLMUnavailableError: return AnswerDTO.fallback(_fallback_text(sources), sources) + if completion.stop_reason in INCOMPLETE_STOP_REASONS: + return AnswerDTO.fallback( + _fallback_text(sources), sources, model_stop_reason=completion.stop_reason + ) return AnswerDTO.from_completion(completion, sources) + async def _ask_agent(self, agent: CatalogAgent, question: str) -> AnswerDTO: + try: + result = await agent.run(question) + except AgentUnavailableError as error: + sources = _sources(error.sources) or await self._retrieve(question) + return AnswerDTO.fallback( + _fallback_text(sources), sources, recommendation=_summary(error.recommendation) + ) + except LLMUnavailableError: + sources = await self._retrieve(question) + return AnswerDTO.fallback(_fallback_text(sources), sources) + if result.stop_reason is AgentStopReason.MODEL_INCOMPLETE: + sources = _sources(result.sources) or await self._retrieve(question) + return AnswerDTO.fallback( + _fallback_text(sources), + sources, + recommendation=_summary(result.recommendation), + stop_reason=result.stop_reason, + model_stop_reason=result.model_stop_reason, + ) + return AnswerDTO( + text=result.text, + sources=_sources(result.sources), + fallback_used=False, + usage=UsageDTO( + model=result.usage.model, + input_tokens=result.usage.input_tokens, + output_tokens=result.usage.output_tokens, + latency_ms=result.usage.latency_ms, + ), + recommendation=_summary(result.recommendation), + stop_reason=result.stop_reason, + ) + + async def _retrieve(self, question: str) -> tuple[SourceDTO, ...]: + [embedding] = await self._embedding_client.embed([question]) + return _sources(await self._document_search.search(embedding, self._top_k)) + -def _build_prompt(question: str, sources: tuple[SourceDTO, ...]) -> str: +def build_rag_prompt(question: str, sources: tuple[SourceDTO, ...]) -> str: if not sources: return question products = "\n\n".join(source.content for source in sources) return f"Catalog products:\n\n{products}\n\nQuestion: {question}" +def _sources(documents: Sequence[RetrievedDocument]) -> tuple[SourceDTO, ...]: + return tuple(SourceDTO.from_retrieved(document) for document in documents) + + +def _summary(recommendation: AgentRecommendation | None) -> RecommendationSummaryDTO | None: + if recommendation is None: + return None + return RecommendationSummaryDTO.from_agent(recommendation) + + def _fallback_text(sources: tuple[SourceDTO, ...]) -> str: if not sources: return FALLBACK_WITHOUT_SOURCES diff --git a/src/contexts/assistant/application/use_cases/recommend_candidate.py b/src/contexts/assistant/application/use_cases/recommend_candidate.py index bdae713..7416fe3 100644 --- a/src/contexts/assistant/application/use_cases/recommend_candidate.py +++ b/src/contexts/assistant/application/use_cases/recommend_candidate.py @@ -34,8 +34,10 @@ async def execute(self, question: str, candidates: Sequence[Candidate]) -> Recom assessments = await self._assess_all(question, top) selection = select(top, assessments, self._requirements_threshold, self._min_score) self._log(top, assessments, selection, started) + chosen = next((c for c in top if c.product_id == selection.product_id), None) return RecommendationDTO( product_id=selection.product_id, + sku=chosen.sku if chosen is not None else None, score=selection.score, reason=selection.reason, fallback_used=selection.fallback_used, diff --git a/src/contexts/assistant/domain/agent.py b/src/contexts/assistant/domain/agent.py new file mode 100644 index 0000000..0f699c2 --- /dev/null +++ b/src/contexts/assistant/domain/agent.py @@ -0,0 +1,54 @@ +from dataclasses import dataclass +from enum import Enum +from uuid import UUID + +from contexts.assistant.domain.exceptions import LLMUnavailableError +from contexts.assistant.domain.retrieved_document import RetrievedDocument +from contexts.assistant.domain.selection import SelectionReason + + +class AgentStopReason(str, Enum): + ANSWERED = "answered" + MAX_STEPS = "max_steps" + MODEL_INCOMPLETE = "model_incomplete" + + +@dataclass(frozen=True, slots=True) +class AgentRecommendation: + product_id: UUID | None + sku: str | None + score: float | None + reason: SelectionReason + + +@dataclass(frozen=True, slots=True) +class AgentUsage: + model: str + tool_calls: int + input_tokens: int + output_tokens: int + latency_ms: int + + +@dataclass(frozen=True, slots=True) +class AgentResult: + text: str + sources: tuple[RetrievedDocument, ...] + recommendation: AgentRecommendation | None + stop_reason: AgentStopReason + usage: AgentUsage + model_stop_reason: str | None = None + + +class AgentUnavailableError(LLMUnavailableError): + def __init__( + self, + *, + sources: tuple[RetrievedDocument, ...], + recommendation: AgentRecommendation | None, + usage: AgentUsage, + ) -> None: + super().__init__("the language model is unavailable") + self.sources = sources + self.recommendation = recommendation + self.usage = usage diff --git a/src/contexts/assistant/domain/completion.py b/src/contexts/assistant/domain/completion.py index 72a3149..45b31bb 100644 --- a/src/contexts/assistant/domain/completion.py +++ b/src/contexts/assistant/domain/completion.py @@ -1,5 +1,7 @@ from dataclasses import dataclass +INCOMPLETE_STOP_REASONS = frozenset({"max_tokens", "refusal"}) + @dataclass(frozen=True, slots=True) class Completion: @@ -8,3 +10,4 @@ class Completion: input_tokens: int output_tokens: int latency_ms: int + stop_reason: str | None = None diff --git a/src/contexts/assistant/domain/ports/catalog_agent.py b/src/contexts/assistant/domain/ports/catalog_agent.py new file mode 100644 index 0000000..f05a435 --- /dev/null +++ b/src/contexts/assistant/domain/ports/catalog_agent.py @@ -0,0 +1,16 @@ +from abc import ABC, abstractmethod + +from contexts.assistant.domain.agent import AgentResult + + +class CatalogAgent(ABC): + @abstractmethod + async def run(self, question: str) -> AgentResult: + """Makes at most the configured number of tool calls; past that limit it answers in RAG + mode with what was retrieved (``stop_reason`` = ``max_steps``). ``recommendation`` comes + only from the candidate selector, never from the model's text. A turn cut off by + ``max_tokens`` or refused ends the run with ``stop_reason`` = ``model_incomplete`` and the + model's own reason in ``model_stop_reason``. Raises ``AgentUnavailableError`` for transient + model failures, carrying the sources, recommendation and usage gathered until then; + configuration errors propagate. + """ diff --git a/src/contexts/assistant/domain/ports/document_search.py b/src/contexts/assistant/domain/ports/document_search.py index 6d93f5b..640ca62 100644 --- a/src/contexts/assistant/domain/ports/document_search.py +++ b/src/contexts/assistant/domain/ports/document_search.py @@ -9,3 +9,9 @@ async def search(self, embedding: list[float], k: int) -> list[RetrievedDocument """Returns at most ``k`` documents, most similar first. ``score`` is the cosine similarity (higher is closer). An empty index returns an empty list. """ + + @abstractmethod + async def find_by_sku(self, sku: str, embedding: list[float]) -> RetrievedDocument | None: + """Exact SKU match. ``score`` is the cosine similarity to ``embedding``, as in ``search``. + ``None`` if the SKU is not indexed or its catalog fields are missing. + """ diff --git a/src/contexts/assistant/domain/retrieved_document.py b/src/contexts/assistant/domain/retrieved_document.py index ffd28f2..cc63523 100644 --- a/src/contexts/assistant/domain/retrieved_document.py +++ b/src/contexts/assistant/domain/retrieved_document.py @@ -7,3 +7,7 @@ class RetrievedDocument: product_id: UUID content: str score: float + sku: str | None = None + name: str | None = None + price_cents: int | None = None + currency: str | None = None diff --git a/src/contexts/assistant/infrastructure/agent/__init__.py b/src/contexts/assistant/infrastructure/agent/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/contexts/assistant/infrastructure/agent/anthropic_agent_model.py b/src/contexts/assistant/infrastructure/agent/anthropic_agent_model.py new file mode 100644 index 0000000..cd28745 --- /dev/null +++ b/src/contexts/assistant/infrastructure/agent/anthropic_agent_model.py @@ -0,0 +1,98 @@ +from collections.abc import Mapping, Sequence +from dataclasses import dataclass + +from anthropic import AsyncAnthropic, omit +from anthropic.types import ( + ContentBlockParam, + MessageParam, + RedactedThinkingBlock, + TextBlock, + ThinkingBlock, + ToolParam, + ToolUseBlock, +) + +from contexts.assistant.infrastructure.llm.anthropic_errors import transient_errors_as_unavailable + + +@dataclass(frozen=True, slots=True) +class ToolCall: + id: str + name: str + input: Mapping[str, object] + + +@dataclass(frozen=True, slots=True) +class ModelTurn: + assistant_content: list[ContentBlockParam] + text: str + tool_calls: tuple[ToolCall, ...] + model: str + input_tokens: int + output_tokens: int + stop_reason: str | None + + +class AnthropicAgentModel: + def __init__( + self, + *, + client: AsyncAnthropic, + model: str, + timeout_s: float, + max_retries: int, + max_tokens: int, + ) -> None: + self._client = client.with_options(timeout=timeout_s, max_retries=max_retries) + self._model = model + self._max_tokens = max_tokens + + @property + def model_name(self) -> str: + return self._model + + async def respond( + self, + *, + system: str, + messages: Sequence[MessageParam], + tools: Sequence[ToolParam] | None, + ) -> ModelTurn: + with transient_errors_as_unavailable(): + message = await self._client.messages.create( + model=self._model, + max_tokens=self._max_tokens, + system=system, + messages=list(messages), + tools=list(tools) if tools else omit, + ) + + content: list[ContentBlockParam] = [] + calls: list[ToolCall] = [] + texts: list[str] = [] + for block in message.content: + if isinstance(block, TextBlock): + texts.append(block.text) + content.append({"type": "text", "text": block.text}) + elif isinstance(block, ToolUseBlock): + tool_input = block.input if isinstance(block.input, dict) else {} + calls.append(ToolCall(id=block.id, name=block.name, input=tool_input)) + content.append( + {"type": "tool_use", "id": block.id, "name": block.name, "input": tool_input} + ) + elif isinstance(block, ThinkingBlock): + content.append( + {"type": "thinking", "thinking": block.thinking, "signature": block.signature} + ) + elif isinstance(block, RedactedThinkingBlock): + content.append({"type": "redacted_thinking", "data": block.data}) + + return ModelTurn( + assistant_content=content, + text="".join(texts), + tool_calls=tuple(calls), + model=message.model, + input_tokens=message.usage.input_tokens, + output_tokens=message.usage.output_tokens, + stop_reason=message.stop_reason, + ) diff --git a/src/contexts/assistant/infrastructure/agent/langgraph_catalog_agent.py b/src/contexts/assistant/infrastructure/agent/langgraph_catalog_agent.py new file mode 100644 index 0000000..bd71a13 --- /dev/null +++ b/src/contexts/assistant/infrastructure/agent/langgraph_catalog_agent.py @@ -0,0 +1,293 @@ +import logging +import time +from collections.abc import Sequence +from typing import Protocol, TypedDict + +from anthropic.types import MessageParam, ToolParam, ToolResultBlockParam +from langgraph.graph import END, START, StateGraph +from langgraph.graph.state import CompiledStateGraph + +from contexts.assistant.application.dtos import SourceDTO +from contexts.assistant.application.use_cases.ask_assistant import SYSTEM_PROMPT, build_rag_prompt +from contexts.assistant.domain.agent import ( + AgentRecommendation, + AgentResult, + AgentStopReason, + AgentUnavailableError, + AgentUsage, +) +from contexts.assistant.domain.completion import INCOMPLETE_STOP_REASONS +from contexts.assistant.domain.exceptions import LLMUnavailableError +from contexts.assistant.domain.ports.catalog_agent import CatalogAgent +from contexts.assistant.domain.ports.embedding_client import EmbeddingClient +from contexts.assistant.domain.retrieved_document import RetrievedDocument +from contexts.assistant.infrastructure.agent.anthropic_agent_model import ModelTurn, ToolCall +from contexts.assistant.infrastructure.agent.tools import ( + ToolContext, + ToolSpec, + api_tool_definitions, + dispatch, +) + +logger = logging.getLogger(__name__) + +AGENT_SYSTEM_PROMPT = ( + "You are the assistant of a product catalog. Use the tools to look products up. " + "When the user asks which product to choose for a use, call recommend_candidate and " + "report its result as given: never pick a product yourself. If it reports no clear " + "recommendation, say so and list the candidates with name, SKU and price. " + "Answer concisely." +) + +ROUTE_TOOLS = "tools" +ROUTE_RAG = "rag_answer" + + +class AgentModel(Protocol): + @property + def model_name(self) -> str: ... + + async def respond( + self, + *, + system: str, + messages: Sequence[MessageParam], + tools: Sequence[ToolParam] | None, + ) -> ModelTurn: ... + + +class _State(TypedDict): + question: str + question_embedding: list[float] + messages: list[MessageParam] + pending: list[ToolCall] + tool_calls: int + retrieved: list[RetrievedDocument] + recommendation: AgentRecommendation | None + text: str + stop_reason: AgentStopReason | None + model_stop_reason: str | None + model: str + input_tokens: int + output_tokens: int + + +class _ModelFailedError(Exception): + def __init__(self, state: _State) -> None: + super().__init__("the language model is unavailable") + self.state = state + + +class LangGraphCatalogAgent(CatalogAgent): + def __init__( + self, + *, + model: AgentModel, + tools: Sequence[ToolSpec], + embedding_client: EmbeddingClient, + max_tool_calls: int, + ) -> None: + self._model = model + self._tools = list(tools) + self._embedding_client = embedding_client + self._definitions = api_tool_definitions(self._tools) + self._max_tool_calls = max_tool_calls + self._graph = self._build() + + async def run(self, question: str) -> AgentResult: + started = time.perf_counter() + [embedding] = await self._embedding_client.embed([question]) + initial: _State = { + "question": question, + "question_embedding": embedding, + "messages": [{"role": "user", "content": question}], + "pending": [], + "tool_calls": 0, + "retrieved": [], + "recommendation": None, + "text": "", + "stop_reason": None, + "model_stop_reason": None, + "model": self._model.model_name, + "input_tokens": 0, + "output_tokens": 0, + } + try: + final = await self._graph.ainvoke( + initial, config={"recursion_limit": 2 * self._max_tool_calls + 6} + ) + except _ModelFailedError as failure: + partial = failure.state + error = AgentUnavailableError( + sources=tuple(partial["retrieved"]), + recommendation=partial["recommendation"], + usage=_usage( + model=partial["model"], + tool_calls=partial["tool_calls"], + input_tokens=partial["input_tokens"], + output_tokens=partial["output_tokens"], + started=started, + ), + ) + _log("unavailable", None, error.usage, error.recommendation) + raise error from failure.__cause__ + result = AgentResult( + text=final["text"], + sources=tuple(final["retrieved"]), + recommendation=final["recommendation"], + stop_reason=final["stop_reason"], + usage=_usage( + model=final["model"], + tool_calls=final["tool_calls"], + input_tokens=final["input_tokens"], + output_tokens=final["output_tokens"], + started=started, + ), + model_stop_reason=final["model_stop_reason"], + ) + _log( + result.stop_reason.value, + result.model_stop_reason, + result.usage, + result.recommendation, + ) + return result + + def _build(self) -> CompiledStateGraph[_State, None, _State, _State]: + graph = StateGraph(_State) + graph.add_node("model", self._model_node) + graph.add_node(ROUTE_TOOLS, self._tools_node) + graph.add_node(ROUTE_RAG, self._rag_node) + graph.add_edge(START, "model") + graph.add_conditional_edges( + "model", self._route, {ROUTE_TOOLS: ROUTE_TOOLS, ROUTE_RAG: ROUTE_RAG, END: END} + ) + graph.add_edge(ROUTE_TOOLS, "model") + graph.add_edge(ROUTE_RAG, END) + return graph.compile() + + def _route(self, state: _State) -> str: + if not state["pending"]: + return END + if state["tool_calls"] + len(state["pending"]) > self._max_tool_calls: + return ROUTE_RAG + return ROUTE_TOOLS + + async def _respond( + self, + state: _State, + *, + system: str, + messages: Sequence[MessageParam], + tools: Sequence[ToolParam] | None, + ) -> ModelTurn: + try: + return await self._model.respond(system=system, messages=messages, tools=tools) + except LLMUnavailableError as error: + raise _ModelFailedError(state) from error + + async def _model_node(self, state: _State) -> dict[str, object]: + turn = await self._respond( + state, system=AGENT_SYSTEM_PROMPT, messages=state["messages"], tools=self._definitions + ) + incomplete = turn.stop_reason in INCOMPLETE_STOP_REASONS + assistant: MessageParam = {"role": "assistant", "content": turn.assistant_content} + update: dict[str, object] = { + "messages": [*state["messages"], assistant], + "pending": [] if incomplete else list(turn.tool_calls), + "model": turn.model, + "input_tokens": state["input_tokens"] + turn.input_tokens, + "output_tokens": state["output_tokens"] + turn.output_tokens, + } + if incomplete: + update["text"] = turn.text + update["stop_reason"] = AgentStopReason.MODEL_INCOMPLETE + update["model_stop_reason"] = turn.stop_reason + elif not turn.tool_calls: + update["text"] = turn.text + update["stop_reason"] = AgentStopReason.ANSWERED + return update + + async def _tools_node(self, state: _State) -> dict[str, object]: + context = ToolContext( + question=state["question"], question_embedding=state["question_embedding"] + ) + retrieved = list(state["retrieved"]) + recommendation = state["recommendation"] + results: list[ToolResultBlockParam] = [] + for call in state["pending"]: + outcome = await dispatch(self._tools, call.name, call.input, context) + results.append( + { + "type": "tool_result", + "tool_use_id": call.id, + "content": outcome.text, + "is_error": outcome.is_error, + } + ) + known = {document.product_id for document in retrieved} + retrieved += [d for d in outcome.retrieved if d.product_id not in known] + if outcome.recommendation is not None: + recommendation = outcome.recommendation + tool_message: MessageParam = {"role": "user", "content": results} + return { + "messages": [*state["messages"], tool_message], + "pending": [], + "tool_calls": state["tool_calls"] + len(state["pending"]), + "retrieved": retrieved, + "recommendation": recommendation, + } + + async def _rag_node(self, state: _State) -> dict[str, object]: + sources = tuple(SourceDTO.from_retrieved(document) for document in state["retrieved"]) + turn = await self._respond( + state, + system=SYSTEM_PROMPT, + messages=[{"role": "user", "content": build_rag_prompt(state["question"], sources)}], + tools=None, + ) + incomplete = turn.stop_reason in INCOMPLETE_STOP_REASONS + return { + "text": turn.text, + "stop_reason": ( + AgentStopReason.MODEL_INCOMPLETE if incomplete else AgentStopReason.MAX_STEPS + ), + "model_stop_reason": turn.stop_reason if incomplete else None, + "pending": [], + "model": turn.model, + "input_tokens": state["input_tokens"] + turn.input_tokens, + "output_tokens": state["output_tokens"] + turn.output_tokens, + } + + +def _usage( + *, model: str, tool_calls: int, input_tokens: int, output_tokens: int, started: float +) -> AgentUsage: + return AgentUsage( + model=model, + tool_calls=tool_calls, + input_tokens=input_tokens, + output_tokens=output_tokens, + latency_ms=round((time.perf_counter() - started) * 1000), + ) + + +def _log( + stop_reason: str, + model_stop_reason: str | None, + usage: AgentUsage, + recommendation: AgentRecommendation | None, +) -> None: + logger.info( + "agent_run stop_reason=%s model_stop_reason=%s tool_calls=%d model=%s input_tokens=%d " + "output_tokens=%d latency_ms=%d recommendation_reason=%s product_id=%s", + stop_reason, + model_stop_reason, + usage.tool_calls, + usage.model, + usage.input_tokens, + usage.output_tokens, + usage.latency_ms, + recommendation.reason.value if recommendation is not None else None, + recommendation.product_id if recommendation is not None else None, + ) diff --git a/src/contexts/assistant/infrastructure/agent/tools.py b/src/contexts/assistant/infrastructure/agent/tools.py new file mode 100644 index 0000000..3cf15ab --- /dev/null +++ b/src/contexts/assistant/infrastructure/agent/tools.py @@ -0,0 +1,161 @@ +from collections.abc import Awaitable, Callable, Mapping, Sequence +from dataclasses import dataclass + +from anthropic.types import ToolParam + +from contexts.assistant.application.dtos import CandidateDTO +from contexts.assistant.application.use_cases.recommend_candidate import RecommendCandidate +from contexts.assistant.domain.agent import AgentRecommendation +from contexts.assistant.domain.candidate import Candidate +from contexts.assistant.domain.ports.document_search import DocumentSearch +from contexts.assistant.domain.ports.embedding_client import EmbeddingClient +from contexts.assistant.domain.retrieved_document import RetrievedDocument +from contexts.assistant.domain.selection import SelectionReason + + +@dataclass(frozen=True, slots=True) +class ToolContext: + question: str + question_embedding: list[float] + + +@dataclass(frozen=True, slots=True) +class ToolOutcome: + text: str + is_error: bool = False + retrieved: tuple[RetrievedDocument, ...] = () + recommendation: AgentRecommendation | None = None + + +ToolHandler = Callable[[Mapping[str, object], ToolContext], Awaitable[ToolOutcome]] + + +@dataclass(frozen=True, slots=True) +class ToolSpec: + name: str + description: str + input_schema: dict[str, object] + handler: ToolHandler + + +def api_tool_definitions(tools: Sequence[ToolSpec]) -> list[ToolParam]: + return [ + {"name": tool.name, "description": tool.description, "input_schema": tool.input_schema} + for tool in tools + ] + + +async def dispatch( + tools: Sequence[ToolSpec], name: str, tool_input: Mapping[str, object], context: ToolContext +) -> ToolOutcome: + for tool in tools: + if tool.name == name: + return await tool.handler(tool_input, context) + return ToolOutcome(text=f"unknown tool: {name}", is_error=True) + + +def build_tools( + *, + embedding_client: EmbeddingClient, + document_search: DocumentSearch, + recommend_candidate: RecommendCandidate, + rag_top_k: int, + selector_top_n: int, +) -> list[ToolSpec]: + async def semantic_search(tool_input: Mapping[str, object], _: ToolContext) -> ToolOutcome: + query = tool_input.get("query") + if not isinstance(query, str) or not query.strip(): + return ToolOutcome(text="semantic_search needs a non-empty 'query'", is_error=True) + [embedding] = await embedding_client.embed([query]) + hits = await document_search.search(embedding, rag_top_k) + if not hits: + return ToolOutcome(text="No catalog products match.") + return ToolOutcome(text="\n".join(_line(hit) for hit in hits), retrieved=tuple(hits)) + + async def get_product_by_sku( + tool_input: Mapping[str, object], context: ToolContext + ) -> ToolOutcome: + sku = tool_input.get("sku") + if not isinstance(sku, str) or not sku.strip(): + return ToolOutcome(text="get_product_by_sku needs a non-empty 'sku'", is_error=True) + product = await document_search.find_by_sku(sku.strip(), context.question_embedding) + if product is None: + return ToolOutcome(text=f"No catalog product has SKU {sku.strip()}.") + return ToolOutcome(text=product.content, retrieved=(product,)) + + async def recommend(_: Mapping[str, object], context: ToolContext) -> ToolOutcome: + hits = await document_search.search(context.question_embedding, selector_top_n) + candidates = [candidate for hit in hits if (candidate := _candidate(hit)) is not None] + result = await recommend_candidate.execute(context.question, candidates) + recommendation = AgentRecommendation( + product_id=result.product_id, sku=result.sku, score=result.score, reason=result.reason + ) + if result.reason is SelectionReason.CHOSEN: + chosen = next(c for c in result.candidates if c.product_id == result.product_id) + text = f"Recommended: {_summary(chosen)}." + else: + listed = "\n".join(f"- {_summary(c)}" for c in result.candidates) + text = f"No clear recommendation ({result.reason.value}). Candidates:\n{listed}" + return ToolOutcome(text=text, retrieved=tuple(hits), recommendation=recommendation) + + return [ + ToolSpec( + name="semantic_search", + description="Search the product catalog by meaning. Returns the closest products.", + input_schema={ + "type": "object", + "properties": {"query": {"type": "string", "description": "What to look for."}}, + "required": ["query"], + "additionalProperties": False, + }, + handler=semantic_search, + ), + ToolSpec( + name="get_product_by_sku", + description="Get one catalog product by its exact SKU.", + input_schema={ + "type": "object", + "properties": {"sku": {"type": "string", "description": "Exact product SKU."}}, + "required": ["sku"], + "additionalProperties": False, + }, + handler=get_product_by_sku, + ), + ToolSpec( + name="recommend_candidate", + description=( + "Recommend one product for the user's question, or report that there is no " + "clear recommendation. The choice is made by rules, not by you: report it as is." + ), + input_schema={"type": "object", "properties": {}, "additionalProperties": False}, + handler=recommend, + ), + ] + + +def _candidate(hit: RetrievedDocument) -> Candidate | None: + if hit.sku is None or hit.name is None or hit.price_cents is None or hit.currency is None: + return None + return Candidate( + product_id=hit.product_id, + sku=hit.sku, + name=hit.name, + price_cents=hit.price_cents, + currency=hit.currency, + content=hit.content, + retrieval_score=hit.score, + ) + + +def _line(hit: RetrievedDocument) -> str: + return "- " + hit.content.replace("\n", " | ") + + +def _summary(candidate: CandidateDTO) -> str: + price = _price(candidate.price_cents, candidate.currency) + return f"{candidate.name} | SKU {candidate.sku} | {price}" + + +def _price(price_cents: int, currency: str) -> str: + units, cents = divmod(price_cents, 100) + return f"{units}.{cents:02d} {currency}" diff --git a/src/contexts/assistant/infrastructure/llm/anthropic_llm_client.py b/src/contexts/assistant/infrastructure/llm/anthropic_llm_client.py index 148a444..f8dc40a 100644 --- a/src/contexts/assistant/infrastructure/llm/anthropic_llm_client.py +++ b/src/contexts/assistant/infrastructure/llm/anthropic_llm_client.py @@ -32,4 +32,5 @@ async def complete(self, *, system: str, prompt: str, max_tokens: int) -> Comple input_tokens=message.usage.input_tokens, output_tokens=message.usage.output_tokens, latency_ms=latency_ms, + stop_reason=message.stop_reason, ) diff --git a/src/contexts/assistant/infrastructure/persistence/sqlalchemy_document_search.py b/src/contexts/assistant/infrastructure/persistence/sqlalchemy_document_search.py index 5f29dd3..53ff03e 100644 --- a/src/contexts/assistant/infrastructure/persistence/sqlalchemy_document_search.py +++ b/src/contexts/assistant/infrastructure/persistence/sqlalchemy_document_search.py @@ -11,15 +11,61 @@ def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None: self._session_factory = session_factory async def search(self, embedding: list[float], k: int) -> list[RetrievedDocument]: - distance = ProductDocumentModel.embedding.op("<=>", return_type=Float)(embedding) + model = ProductDocumentModel + distance = model.embedding.op("<=>", return_type=Float)(embedding) stmt = ( - select(ProductDocumentModel.product_id, ProductDocumentModel.content, distance) + select( + model.product_id, + model.content, + distance, + model.sku, + model.name, + model.price_cents, + model.currency, + ) .order_by(distance) .limit(k) ) async with self._session_factory() as session: rows = (await session.execute(stmt)).all() return [ - RetrievedDocument(product_id=product_id, content=content, score=1.0 - float(dist)) - for product_id, content, dist in rows + RetrievedDocument( + product_id=product_id, + content=content, + score=1.0 - float(dist), + sku=sku, + name=name, + price_cents=price_cents, + currency=currency, + ) + for product_id, content, dist, sku, name, price_cents, currency in rows ] + + async def find_by_sku(self, sku: str, embedding: list[float]) -> RetrievedDocument | None: + model = ProductDocumentModel + distance = model.embedding.op("<=>", return_type=Float)(embedding) + stmt = select( + model.product_id, + model.content, + distance, + model.sku, + model.name, + model.price_cents, + model.currency, + ).where(model.sku == sku) + async with self._session_factory() as session: + row = (await session.execute(stmt)).one_or_none() + if row is None: + return None + product_id, content, dist, found_sku, name, price_cents, currency = row + if found_sku is None or name is None or price_cents is None or currency is None: + return None + return RetrievedDocument( + product_id=product_id, + content=content, + score=1.0 - float(dist), + sku=found_sku, + name=name, + price_cents=price_cents, + currency=currency, + ) diff --git a/src/contexts/assistant/ui/dependencies.py b/src/contexts/assistant/ui/dependencies.py index a15c6f8..734df18 100644 --- a/src/contexts/assistant/ui/dependencies.py +++ b/src/contexts/assistant/ui/dependencies.py @@ -7,11 +7,17 @@ from contexts.assistant.application.use_cases.ask_assistant import AskAssistant from contexts.assistant.application.use_cases.index_catalog import IndexCatalog +from contexts.assistant.application.use_cases.recommend_candidate import RecommendCandidate +from contexts.assistant.domain.ports.candidate_assessor import CandidateAssessor +from contexts.assistant.domain.ports.catalog_agent import CatalogAgent from contexts.assistant.domain.ports.catalog_reader import CatalogReader from contexts.assistant.domain.ports.document_index import DocumentIndex from contexts.assistant.domain.ports.document_search import DocumentSearch from contexts.assistant.domain.ports.embedding_client import EmbeddingClient from contexts.assistant.domain.ports.llm_client import LLMClient +from contexts.assistant.infrastructure.agent.anthropic_agent_model import AnthropicAgentModel +from contexts.assistant.infrastructure.agent.langgraph_catalog_agent import LangGraphCatalogAgent +from contexts.assistant.infrastructure.agent.tools import build_tools from contexts.assistant.infrastructure.catalog.list_products_catalog_reader import ( ListProductsCatalogReader, ) @@ -19,6 +25,9 @@ FastEmbedEmbeddingClient, ) from contexts.assistant.infrastructure.llm.anthropic_llm_client import AnthropicLLMClient +from contexts.assistant.infrastructure.llm.claude_candidate_assessor import ( + ClaudeCandidateAssessor, +) from contexts.assistant.infrastructure.persistence.sqlalchemy_document_index import ( SqlAlchemyDocumentIndex, ) @@ -35,12 +44,17 @@ from shared.infrastructure.settings import get_settings +@lru_cache +def get_anthropic_client() -> AsyncAnthropic: + api_key = get_settings().anthropic_api_key + return AsyncAnthropic(api_key=api_key.get_secret_value() if api_key else None) + + @lru_cache def get_llm_client() -> LLMClient: settings = get_settings() - api_key = settings.anthropic_api_key return AnthropicLLMClient( - client=AsyncAnthropic(api_key=api_key.get_secret_value() if api_key else None), + client=get_anthropic_client(), model=settings.llm_model, timeout_s=settings.llm_timeout_s, max_retries=settings.llm_max_retries, @@ -74,10 +88,71 @@ def get_catalog_reader( return ListProductsCatalogReader(list_products, get_product) +@lru_cache +def get_candidate_assessor() -> CandidateAssessor: + settings = get_settings() + return ClaudeCandidateAssessor( + client=get_anthropic_client(), + model=settings.llm_model, + timeout_s=settings.llm_timeout_s, + max_retries=settings.llm_max_retries, + samples=settings.selector_samples, + max_tokens=settings.selector_max_tokens, + max_concurrency=settings.selector_max_concurrency, + ) + + +def get_recommend_candidate_use_case() -> RecommendCandidate: + settings = get_settings() + return RecommendCandidate( + get_candidate_assessor(), + top_n=settings.selector_top_n, + timeout_s=settings.selector_timeout_s, + requirements_threshold=settings.selector_requirements_threshold, + min_score=settings.selector_min_score, + ) + + +@lru_cache +def get_catalog_agent() -> CatalogAgent: + settings = get_settings() + model = AnthropicAgentModel( + client=get_anthropic_client(), + model=settings.llm_model, + timeout_s=settings.llm_timeout_s, + max_retries=settings.llm_max_retries, + max_tokens=settings.llm_max_tokens, + ) + tools = build_tools( + embedding_client=get_embedding_client(), + document_search=get_document_search(), + recommend_candidate=get_recommend_candidate_use_case(), + rag_top_k=settings.rag_top_k, + selector_top_n=settings.selector_top_n, + ) + return LangGraphCatalogAgent( + model=model, + tools=tools, + embedding_client=get_embedding_client(), + max_tool_calls=settings.agent_max_tool_calls, + ) + + +def get_assistant_mode() -> str: + return get_settings().assistant_mode + + +def get_mode_catalog_agent( + mode: Annotated[str, Depends(get_assistant_mode)], +) -> CatalogAgent | None: + return get_catalog_agent() if mode == "agent" else None + + def get_ask_assistant_use_case( llm_client: Annotated[LLMClient, Depends(get_llm_client)], embedding_client: Annotated[EmbeddingClient, Depends(get_embedding_client)], document_search: Annotated[DocumentSearch, Depends(get_document_search)], + agent: Annotated[CatalogAgent | None, Depends(get_mode_catalog_agent)], ) -> AskAssistant: settings = get_settings() return AskAssistant( @@ -86,9 +161,16 @@ def get_ask_assistant_use_case( document_search, max_tokens=settings.llm_max_tokens, top_k=settings.rag_top_k, + agent=agent, ) +def clear_anthropic_dependants() -> None: + for provider in (get_llm_client, get_candidate_assessor, get_catalog_agent): + provider.cache_clear() + get_anthropic_client.cache_clear() + + def get_index_catalog_use_case( catalog_reader: Annotated[CatalogReader, Depends(get_catalog_reader)], embedding_client: Annotated[EmbeddingClient, Depends(get_embedding_client)], diff --git a/src/contexts/assistant/ui/lifespan.py b/src/contexts/assistant/ui/lifespan.py index 0dff8a5..d97ca10 100644 --- a/src/contexts/assistant/ui/lifespan.py +++ b/src/contexts/assistant/ui/lifespan.py @@ -1,7 +1,17 @@ -from contexts.assistant.ui.dependencies import get_embedding_client +from contexts.assistant.ui.dependencies import ( + clear_anthropic_dependants, + get_anthropic_client, + get_embedding_client, +) WARM_UP_TEXT = "warm-up" async def warm_up_embeddings() -> None: await get_embedding_client().embed([WARM_UP_TEXT]) + + +async def close_anthropic_client() -> None: + if get_anthropic_client.cache_info().currsize: + await get_anthropic_client().close() + clear_anthropic_dependants() diff --git a/src/contexts/assistant/ui/schemas.py b/src/contexts/assistant/ui/schemas.py index d77f88f..fe8bb3d 100644 --- a/src/contexts/assistant/ui/schemas.py +++ b/src/contexts/assistant/ui/schemas.py @@ -1,10 +1,16 @@ from __future__ import annotations +from typing import Literal from uuid import UUID -from pydantic import BaseModel, ConfigDict, Field +from pydantic import BaseModel, ConfigDict, Field, model_validator -from contexts.assistant.application.dtos import AnswerDTO, SourceDTO, UsageDTO +from contexts.assistant.application.dtos import ( + AnswerDTO, + RecommendationSummaryDTO, + SourceDTO, + UsageDTO, +) class AskRequest(BaseModel): @@ -45,6 +51,37 @@ def from_dto(cls, dto: SourceDTO) -> SourceResponse: return cls(product_id=dto.product_id, content=dto.content, score=dto.score) +class RecommendationResponse(BaseModel): + product_id: UUID | None = Field( + ..., description="Recommended catalog product. Null unless reason is 'chosen'." + ) + sku: str | None = Field(..., description="SKU of the recommended product, or null.") + score: float | None = Field( + ..., description="Probability that it suits the asked use (0-1), or null." + ) + reason: Literal["chosen", "below_threshold", "no_assessments"] = Field( + ..., + description="Outcome of the candidate selector: 'chosen' (a product is recommended), " + "'below_threshold' (no candidate was good enough) or 'no_assessments' (no candidate " + "could be assessed).", + ) + + @model_validator(mode="after") + def _product_fields_only_when_chosen(self) -> RecommendationResponse: + filled = [self.product_id is not None, self.sku is not None, self.score is not None] + if self.reason == "chosen" and not all(filled): + raise ValueError("product_id, sku and score are required when reason is 'chosen'") + if self.reason != "chosen" and any(filled): + raise ValueError("product_id, sku and score must be null unless reason is 'chosen'") + return self + + @classmethod + def from_dto(cls, dto: RecommendationSummaryDTO) -> RecommendationResponse: + return cls( + product_id=dto.product_id, sku=dto.sku, score=dto.score, reason=dto.reason.value + ) + + class AnswerResponse(BaseModel): text: str = Field(..., description="Answer generated by the assistant, or the fallback text.") sources: list[SourceResponse] = Field( @@ -52,11 +89,16 @@ class AnswerResponse(BaseModel): ) fallback_used: bool = Field( ..., - description="True when the LLM was unavailable and the answer lists the retrieved " - "products without generation.", + description="True when the LLM was unavailable, or its answer was cut off by the token " + "limit or refused, and the answer lists the retrieved products without generation.", ) usage: UsageResponse | None = Field( - ..., description="LLM usage. Null when the fallback was used (no LLM call succeeded)." + ..., description="LLM usage. Null when the fallback was used." + ) + recommendation: RecommendationResponse | None = Field( + default=None, + description="Result of the candidate selector. Null when the selector did not run " + "(rag mode, or the agent answered without recommending).", ) @classmethod @@ -66,4 +108,9 @@ def from_dto(cls, dto: AnswerDTO) -> AnswerResponse: sources=[SourceResponse.from_dto(source) for source in dto.sources], fallback_used=dto.fallback_used, usage=UsageResponse.from_dto(dto.usage) if dto.usage is not None else None, + recommendation=( + RecommendationResponse.from_dto(dto.recommendation) + if dto.recommendation is not None + else None + ), ) diff --git a/src/shared/infrastructure/settings.py b/src/shared/infrastructure/settings.py index a193332..c506901 100644 --- a/src/shared/infrastructure/settings.py +++ b/src/shared/infrastructure/settings.py @@ -64,6 +64,13 @@ class Settings(BaseSettings): description="Only load the model from the cache, never download it. " "The Docker image bakes the model in and sets this to true.", ) + assistant_mode: Literal["rag", "agent"] = Field( + default="rag", + description="'rag' answers with retrieval + one LLM call; 'agent' runs the tool agent.", + ) + agent_max_tool_calls: int = Field( + default=5, ge=1, description="Tool calls per agent run before falling back to RAG mode." + ) selector_provider: Literal["claude"] = Field( default="claude", description="Candidate assessor used by the recommendation tool.", diff --git a/tests/e2e/test_assistant.py b/tests/e2e/test_assistant.py index 135ee48..4ef689a 100644 --- a/tests/e2e/test_assistant.py +++ b/tests/e2e/test_assistant.py @@ -74,6 +74,7 @@ async def test_c7_returns_200_with_answer_sources_and_usage( "text": "The blue widget costs 19.99 EUR.", "sources": [_source_json(WIDGET)], "fallback_used": False, + "recommendation": None, "usage": { "model": "claude-test", "input_tokens": 50, diff --git a/tests/e2e/test_assistant_agent_mode.py b/tests/e2e/test_assistant_agent_mode.py new file mode 100644 index 0000000..6beff15 --- /dev/null +++ b/tests/e2e/test_assistant_agent_mode.py @@ -0,0 +1,136 @@ +from collections.abc import AsyncIterator, Callable +from uuid import uuid4 + +import pytest_asyncio +from httpx import ASGITransport, AsyncClient +from main import create_app +from tests.fakes.catalog_agent import FakeCatalogAgent +from tests.fakes.documents import FakeDocumentSearch +from tests.fakes.embedding_client import FakeEmbeddingClient +from tests.fakes.llm_client import FakeLLMClient + +from contexts.assistant.domain.agent import ( + AgentRecommendation, + AgentResult, + AgentStopReason, + AgentUsage, +) +from contexts.assistant.domain.retrieved_document import RetrievedDocument +from contexts.assistant.domain.selection import SelectionReason +from contexts.assistant.ui.dependencies import ( + get_document_search, + get_embedding_client, + get_llm_client, + get_mode_catalog_agent, +) + +BOOT = RetrievedDocument( + product_id=uuid4(), content="SKU: BOOT-1\nName: Winter boot", score=0.9, sku="BOOT-1" +) +USAGE = AgentUsage( + model="claude-test", tool_calls=1, input_tokens=50, output_tokens=9, latency_ms=10 +) + +ClientFactory = Callable[[FakeCatalogAgent], AsyncClient] + + +def _result(recommendation: AgentRecommendation | None) -> AgentResult: + return AgentResult( + text="Answer.", + sources=(BOOT,), + recommendation=recommendation, + stop_reason=AgentStopReason.ANSWERED, + usage=USAGE, + ) + + +@pytest_asyncio.fixture +async def agent_client() -> AsyncIterator[ClientFactory]: + clients: list[AsyncClient] = [] + + def _factory(agent: FakeCatalogAgent) -> AsyncClient: + application = create_app() + llm, embeddings, search = FakeLLMClient(), FakeEmbeddingClient(), FakeDocumentSearch() + application.dependency_overrides[get_mode_catalog_agent] = lambda: agent + application.dependency_overrides[get_llm_client] = lambda: llm + application.dependency_overrides[get_embedding_client] = lambda: embeddings + application.dependency_overrides[get_document_search] = lambda: search + client = AsyncClient(transport=ASGITransport(app=application), base_url="http://testserver") + clients.append(client) + return client + + yield _factory + for client in clients: + await client.aclose() + + +QUESTION = {"question": "Which boot for winter hiking?"} + + +class TestAgentMode: + async def test_b10_a_chosen_product_comes_back_as_an_object( + self, agent_client: ClientFactory + ) -> None: + agent = FakeCatalogAgent( + _result( + AgentRecommendation( + product_id=BOOT.product_id, + sku="BOOT-1", + score=1.0, + reason=SelectionReason.CHOSEN, + ) + ) + ) + + response = await agent_client(agent).post("/assistant/ask", json=QUESTION) + + assert response.status_code == 200 + assert response.json()["recommendation"] == { + "product_id": str(BOOT.product_id), + "sku": "BOOT-1", + "score": 1.0, + "reason": "chosen", + } + + async def test_b10_selector_ran_without_choosing(self, agent_client: ClientFactory) -> None: + agent = FakeCatalogAgent( + _result( + AgentRecommendation( + product_id=None, sku=None, score=None, reason=SelectionReason.BELOW_THRESHOLD + ) + ) + ) + + response = await agent_client(agent).post("/assistant/ask", json=QUESTION) + + assert response.json()["recommendation"] == { + "product_id": None, + "sku": None, + "score": None, + "reason": "below_threshold", + } + + async def test_b10_agent_answered_without_recommending( + self, agent_client: ClientFactory + ) -> None: + response = await agent_client(FakeCatalogAgent(_result(None))).post( + "/assistant/ask", json=QUESTION + ) + + assert response.json()["recommendation"] is None + + async def test_b15_stop_reason_is_not_part_of_the_http_response( + self, agent_client: ClientFactory + ) -> None: + response = await agent_client(FakeCatalogAgent(_result(None))).post( + "/assistant/ask", json=QUESTION + ) + + assert "stop_reason" not in response.json() + assert set(response.json()) == { + "text", + "sources", + "fallback_used", + "usage", + "recommendation", + } diff --git a/tests/e2e/test_assistant_modes.py b/tests/e2e/test_assistant_modes.py new file mode 100644 index 0000000..cbc51ea --- /dev/null +++ b/tests/e2e/test_assistant_modes.py @@ -0,0 +1,29 @@ +from httpx import ASGITransport, AsyncClient +from main import create_app +from tests.fakes.documents import FakeDocumentSearch +from tests.fakes.embedding_client import FakeEmbeddingClient +from tests.fakes.llm_client import FakeLLMClient + +from contexts.assistant.ui.dependencies import ( + get_catalog_agent, + get_document_search, + get_embedding_client, + get_llm_client, +) + + +async def test_rb1_rag_mode_never_builds_the_agent() -> None: + get_catalog_agent.cache_clear() + application = create_app() + llm, embeddings, search = FakeLLMClient(), FakeEmbeddingClient(), FakeDocumentSearch() + application.dependency_overrides[get_llm_client] = lambda: llm + application.dependency_overrides[get_embedding_client] = lambda: embeddings + application.dependency_overrides[get_document_search] = lambda: search + + async with AsyncClient( + transport=ASGITransport(app=application), base_url="http://testserver" + ) as client: + response = await client.post("/assistant/ask", json={"question": "What is a widget?"}) + + assert response.status_code == 200 + assert get_catalog_agent.cache_info().currsize == 0 diff --git a/tests/e2e/test_lifespan.py b/tests/e2e/test_lifespan.py index 02d2427..f14c084 100644 --- a/tests/e2e/test_lifespan.py +++ b/tests/e2e/test_lifespan.py @@ -3,7 +3,7 @@ from main import create_app from tests.fakes.embedding_client import FakeEmbeddingClient -from contexts.assistant.ui import lifespan +from contexts.assistant.ui import dependencies, lifespan @pytest.fixture @@ -21,3 +21,14 @@ def test_r6_app_loads_the_embedding_model_on_startup(embeddings: FakeEmbeddingCl def test_r6_warm_up_is_off_unless_requested(embeddings: FakeEmbeddingClient) -> None: with TestClient(create_app()): assert embeddings.calls == [] + + +def test_the_shared_anthropic_client_is_closed_on_shutdown() -> None: + dependencies.get_anthropic_client.cache_clear() + client = dependencies.get_anthropic_client() + + with TestClient(create_app()): + assert client.is_closed() is False + + assert client.is_closed() is True + assert dependencies.get_anthropic_client() is not client diff --git a/tests/evals/conftest.py b/tests/evals/conftest.py new file mode 100644 index 0000000..2a2bd31 --- /dev/null +++ b/tests/evals/conftest.py @@ -0,0 +1,134 @@ +import asyncio +import os +from collections.abc import AsyncIterator + +import anthropic +import pytest_asyncio +from sqlalchemy import delete, make_url, text +from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine +from tests.evals.harness import ( + EVAL_DATABASE, + LOADED_AT, + ApiCalls, + ClientFactory, + Dataset, + EvalCatalog, + load_dataset, +) + +from contexts.assistant.application.use_cases.index_catalog import IndexCatalog +from contexts.assistant.domain.ports.embedding_client import EmbeddingClient +from contexts.assistant.infrastructure.catalog.list_products_catalog_reader import ( + ListProductsCatalogReader, +) +from contexts.assistant.infrastructure.embeddings.fastembed_embedding_client import ( + FastEmbedEmbeddingClient, +) +from contexts.assistant.infrastructure.persistence.models import ProductDocumentModel +from contexts.assistant.infrastructure.persistence.sqlalchemy_document_index import ( + SqlAlchemyDocumentIndex, +) +from contexts.assistant.infrastructure.persistence.sqlalchemy_document_search import ( + SqlAlchemyDocumentSearch, +) +from contexts.catalog.application.use_cases.get_product import GetProduct +from contexts.catalog.application.use_cases.list_products import ListProducts +from contexts.catalog.domain.product import Product +from contexts.catalog.infrastructure.persistence.models import ProductModel +from contexts.catalog.infrastructure.persistence.sqlalchemy_product_repository import ( + SqlAlchemyProductRepository, +) +from shared.infrastructure.settings import Settings, get_settings + + +@pytest_asyncio.fixture(scope="session") +async def settings() -> Settings: + return get_settings() + + +@pytest_asyncio.fixture(scope="session") +async def eval_catalog(settings: Settings) -> AsyncIterator[EvalCatalog]: + admin_url = settings.test_database_url + eval_url = make_url(admin_url).set(database=EVAL_DATABASE).render_as_string(False) + await _ensure_database(admin_url) + await _migrate(eval_url) + + dataset = load_dataset() + embeddings = FastEmbedEmbeddingClient( + model_name=settings.embedding_model, + cache_dir=settings.embedding_cache_dir, + local_files_only=settings.embedding_local_files_only, + ) + engine = create_async_engine(eval_url) + factory = async_sessionmaker(engine, expire_on_commit=False) + indexed = await _load(factory, dataset, embeddings) + yield EvalCatalog( + dataset=dataset, + embeddings=embeddings, + search=SqlAlchemyDocumentSearch(factory), + indexed=indexed, + ) + await engine.dispose() + + +@pytest_asyncio.fixture(scope="session") +async def anthropic_clients() -> AsyncIterator[ClientFactory]: + clients: list[anthropic.AsyncAnthropic] = [] + api_key = get_settings().anthropic_api_key + + def _factory() -> tuple[anthropic.AsyncAnthropic, ApiCalls]: + calls = ApiCalls() + client = anthropic.AsyncAnthropic( + api_key=api_key.get_secret_value() if api_key else None, + http_client=anthropic.DefaultAsyncHttpxClient( + event_hooks={"request": [calls.on_request], "response": [calls.on_response]} + ), + ) + clients.append(client) + return client, calls + + yield _factory + for client in clients: + await client.close() + + +async def _ensure_database(admin_url: str) -> None: + engine = create_async_engine(admin_url, isolation_level="AUTOCOMMIT") + async with engine.connect() as connection: + found = await connection.scalar( + text("SELECT 1 FROM pg_database WHERE datname = :name"), {"name": EVAL_DATABASE} + ) + if found is None: + await connection.execute(text(f'CREATE DATABASE "{EVAL_DATABASE}"')) + await engine.dispose() + + +async def _migrate(url: str) -> None: + process = await asyncio.create_subprocess_exec( + "alembic", "upgrade", "head", env={**os.environ, "DATABASE_URL": url} + ) + assert await process.wait() == 0, "alembic upgrade failed on the eval database" + + +async def _load( + factory: async_sessionmaker[AsyncSession], dataset: Dataset, embeddings: EmbeddingClient +) -> int: + async with factory() as session: + await session.execute(delete(ProductDocumentModel)) + await session.execute(delete(ProductModel)) + repository = SqlAlchemyProductRepository(session) + for item in dataset.products: + await repository.add( + Product.create( + sku=item.sku, + name=item.name, + price_cents=item.price_cents, + currency=item.currency, + product_id=item.product_id, + created_at=LOADED_AT, + ) + ) + reader = ListProductsCatalogReader(ListProducts(repository), GetProduct(repository)) + report = await IndexCatalog(reader, embeddings, SqlAlchemyDocumentIndex(session)).execute() + await session.commit() + return report.indexed diff --git a/tests/evals/dataset.json b/tests/evals/dataset.json new file mode 100644 index 0000000..95ae3e0 --- /dev/null +++ b/tests/evals/dataset.json @@ -0,0 +1,38 @@ +{ + "version": 2, + "domain": "material de montaña", + "products": [ + {"sku": "MT-0001", "name": "Bota de invierno Glaciar, impermeable y aislada, talla 42", "price_cents": 18900, "currency": "EUR"}, + {"sku": "MT-0002", "name": "Bota de invierno Glaciar, impermeable y aislada, talla 44", "price_cents": 18900, "currency": "EUR"}, + {"sku": "MT-0003", "name": "Zapatilla de trail Sendero, ligera y transpirable, talla 42", "price_cents": 11900, "currency": "EUR"}, + {"sku": "MT-0004", "name": "Sandalia de río Remanso con suela antideslizante, talla 42", "price_cents": 4900, "currency": "EUR"}, + {"sku": "MT-0005", "name": "Chaqueta impermeable Tormenta 3 capas, roja, talla M", "price_cents": 24900, "currency": "EUR"}, + {"sku": "MT-0006", "name": "Chaqueta impermeable Tormenta 3 capas, azul, talla M", "price_cents": 24900, "currency": "EUR"}, + {"sku": "MT-0007", "name": "Chaqueta impermeable Tormenta 3 capas, azul, talla L", "price_cents": 24900, "currency": "EUR"}, + {"sku": "MT-0008", "name": "Plumífero Cumbre relleno de pluma 800 cuin", "price_cents": 29900, "currency": "EUR"}, + {"sku": "MT-0009", "name": "Forro polar Brezo de 100 g/m²", "price_cents": 4500, "currency": "EUR"}, + {"sku": "MT-0010", "name": "Saco de dormir Escarcha, temperatura de confort -10 °C", "price_cents": 34900, "currency": "EUR"}, + {"sku": "MT-0011", "name": "Saco de dormir Brisa, temperatura de confort +10 °C", "price_cents": 9900, "currency": "EUR"}, + {"sku": "MT-0012", "name": "Tienda de campaña Refugio para 2 personas, 1,6 kg", "price_cents": 27900, "currency": "EUR"}, + {"sku": "MT-0013", "name": "Tienda familiar Campamento para 4 personas, 6,8 kg", "price_cents": 31900, "currency": "EUR"}, + {"sku": "MT-0014", "name": "Linterna frontal Faro de 400 lúmenes, recargable por USB", "price_cents": 4900, "currency": "EUR"}, + {"sku": "MT-0015", "name": "Par de bastones de trekking plegables de carbono", "price_cents": 8900, "currency": "EUR"}, + {"sku": "MT-0016", "name": "Mochila de día Collado de 30 litros con funda de lluvia", "price_cents": 9900, "currency": "EUR"}, + {"sku": "MT-0017", "name": "Mochila de travesía Macizo de 65 litros", "price_cents": 21900, "currency": "EUR"}, + {"sku": "MT-0018", "name": "Hornillo de gas compacto Llama", "price_cents": 3900, "currency": "EUR"}, + {"sku": "MT-0019", "name": "Filtro de agua por gravedad Manantial de 4 litros", "price_cents": 7900, "currency": "EUR"}, + {"sku": "MT-0020", "name": "Crampones de acero Hielo de 12 puntas", "price_cents": 16900, "currency": "EUR"} + ], + "questions": [ + {"id": "q01", "kind": "recommend", "text": "Voy a caminar por nieve varios días en enero y no quiero que se me congelen los pies. Calzo un 44. ¿Cuál me recomiendas?", "acceptable_skus": ["MT-0002"], "accept_no_recommendation": false}, + {"id": "q02", "kind": "recommend", "text": "Quiero algo para no empaparme en una ruta de tres días con lluvia constante. La quiero en azul y uso una M. ¿Qué me llevo?", "acceptable_skus": ["MT-0006"], "accept_no_recommendation": false}, + {"id": "q03", "kind": "recommend", "text": "Vamos a acampar con mis dos hijos y mi pareja un fin de semana. ¿Dónde dormimos todos?", "acceptable_skus": ["MT-0013"], "accept_no_recommendation": false}, + {"id": "q04", "kind": "recommend", "text": "Pasaré noches bajo cero en un vivac de alta montaña. ¿Con qué duermo sin pasar frío?", "acceptable_skus": ["MT-0010"], "accept_no_recommendation": false}, + {"id": "q05", "kind": "recommend", "text": "Necesito ver el camino si se me hace de noche bajando del pico, con las manos libres.", "acceptable_skus": ["MT-0014"], "accept_no_recommendation": false}, + {"id": "q06", "kind": "recommend", "text": "Voy a subir por un glaciar con pendientes heladas. ¿Qué me pongo en las botas para no resbalar?", "acceptable_skus": ["MT-0020"], "accept_no_recommendation": false}, + {"id": "q07", "kind": "no_recommendation", "text": "Quiero una cámara de fotos resistente al agua para hacer fotos bajo el mar. ¿Cuál me recomiendas?", "acceptable_skus": [], "accept_no_recommendation": true}, + {"id": "q08", "kind": "no_recommendation", "text": "Busco una chaqueta impermeable en color verde. ¿Cuál me llevo?", "acceptable_skus": [], "accept_no_recommendation": true}, + {"id": "q09", "kind": "ambiguous", "text": "Voy a salir al monte este fin de semana. ¿Qué me recomiendas?", "acceptable_skus": ["MT-0016", "MT-0015", "MT-0014", "MT-0003"], "accept_no_recommendation": true}, + {"id": "q10", "kind": "recommend", "text": "Haré una travesía de una semana durmiendo en refugios y necesito cargar con todo a la espalda.", "acceptable_skus": ["MT-0017"], "accept_no_recommendation": false} + ] +} diff --git a/tests/evals/harness.py b/tests/evals/harness.py new file mode 100644 index 0000000..6238e38 --- /dev/null +++ b/tests/evals/harness.py @@ -0,0 +1,294 @@ +import json +import random +from collections import Counter +from collections.abc import Callable, Sequence +from dataclasses import dataclass, field +from datetime import datetime, timezone +from itertools import product as grid +from pathlib import Path +from statistics import median +from uuid import UUID, uuid5 + +import anthropic +import httpx2 + +from contexts.assistant.domain.agent import AgentResult, AgentUnavailableError +from contexts.assistant.domain.candidate import Assessment, Candidate +from contexts.assistant.domain.ports.candidate_assessor import CandidateAssessor +from contexts.assistant.domain.ports.catalog_agent import CatalogAgent +from contexts.assistant.domain.ports.document_search import DocumentSearch +from contexts.assistant.domain.ports.embedding_client import EmbeddingClient +from contexts.assistant.domain.retrieved_document import RetrievedDocument +from contexts.assistant.domain.selection import SelectionReason, select + +DATASET_PATH = Path(__file__).with_name("dataset.json") +EVAL_DATABASE = "catalog_eval" +PRODUCT_NAMESPACE = UUID("6f1c2a52-5d0e-4c8e-9a51-0e7f3b1d9c40") +LOADED_AT = datetime(2026, 1, 1, tzinfo=timezone.utc) +RATE_LIMIT_ABORT_SHARE = 0.05 +THRESHOLD_GRID = (1 / 3, 2 / 3, 1.0) +LIST_PRICE_USD_PER_MTOK = {"claude-haiku-5-5": (0.10, 0.50)} + +NULL = "null" +CHOSEN_PREFIX = "chosen:" + + +@dataclass(frozen=True, slots=True) +class EvalProduct: + sku: str + name: str + price_cents: int + currency: str + + @property + def product_id(self) -> UUID: + return uuid5(PRODUCT_NAMESPACE, self.sku) + + +@dataclass(frozen=True, slots=True) +class EvalQuestion: + id: str + kind: str + text: str + acceptable_skus: frozenset[str] + accept_no_recommendation: bool + + +@dataclass(frozen=True, slots=True) +class Dataset: + version: int + products: tuple[EvalProduct, ...] + questions: tuple[EvalQuestion, ...] + + +@dataclass(frozen=True, slots=True) +class EvalCatalog: + dataset: Dataset + embeddings: EmbeddingClient + search: DocumentSearch + indexed: int + + +def load_dataset(path: Path = DATASET_PATH) -> Dataset: + raw = json.loads(path.read_text(encoding="utf-8")) + return Dataset( + version=raw["version"], + products=tuple( + EvalProduct( + sku=item["sku"], + name=item["name"], + price_cents=item["price_cents"], + currency=item["currency"], + ) + for item in raw["products"] + ), + questions=tuple( + EvalQuestion( + id=item["id"], + kind=item["kind"], + text=item["text"], + acceptable_skus=frozenset(item["acceptable_skus"]), + accept_no_recommendation=item["accept_no_recommendation"], + ) + for item in raw["questions"] + ), + ) + + +def outcome_of(reason: SelectionReason | None, sku: str | None) -> str: + if reason is None: + return NULL + if reason is SelectionReason.CHOSEN: + return f"{CHOSEN_PREFIX}{sku}" + return reason.value + + +def is_correct(question: EvalQuestion, outcome: str) -> bool: + if outcome.startswith(CHOSEN_PREFIX): + return outcome.removeprefix(CHOSEN_PREFIX) in question.acceptable_skus + if outcome in (SelectionReason.BELOW_THRESHOLD.value, NULL): + return question.accept_no_recommendation + return False + + +def accuracy_line(questions: Sequence[EvalQuestion], outcomes: Sequence[str]) -> str: + hits: Counter[str] = Counter() + totals: Counter[str] = Counter() + for question, outcome in zip(questions, outcomes, strict=True): + totals[question.kind] += 1 + hits[question.kind] += is_correct(question, outcome) + by_kind = ", ".join(f"{kind} {hits[kind]}/{totals[kind]}" for kind in totals) + return f"{sum(hits.values())}/{len(questions)}: {by_kind}" + + +def candidate_of(hit: RetrievedDocument) -> Candidate | None: + if hit.sku is None or hit.name is None or hit.price_cents is None or hit.currency is None: + return None + return Candidate( + product_id=hit.product_id, + sku=hit.sku, + name=hit.name, + price_cents=hit.price_cents, + currency=hit.currency, + content=hit.content, + retrieval_score=hit.score, + ) + + +def shuffled(candidates: Sequence[Candidate], seed: str) -> list[Candidate]: + copy = list(candidates) + random.Random(seed).shuffle(copy) + return copy + + +@dataclass(frozen=True, slots=True) +class ThresholdRow: + requirements_threshold: float + min_score: float + correct: int + recommended: int + + +@dataclass(frozen=True, slots=True) +class SelectorRun: + question: EvalQuestion + outcome: str + latency_ms: int + candidates: tuple[Candidate, ...] + assessments: tuple[Assessment, ...] + unassessed: int + + @property + def correct(self) -> bool: + return is_correct(self.question, self.outcome) + + @property + def reachable(self) -> bool: + return self.question.accept_no_recommendation or any( + candidate.sku in self.question.acceptable_skus for candidate in self.candidates + ) + + +def threshold_table(runs: Sequence[SelectorRun]) -> list[ThresholdRow]: + rows = [] + for requirements_threshold, min_score in grid(THRESHOLD_GRID, THRESHOLD_GRID): + outcomes = [] + for run in runs: + selection = select(run.candidates, run.assessments, requirements_threshold, min_score) + chosen = next( + (c.sku for c in run.candidates if c.product_id == selection.product_id), None + ) + outcomes.append((run.question, outcome_of(selection.reason, chosen))) + rows.append( + ThresholdRow( + requirements_threshold=requirements_threshold, + min_score=min_score, + correct=sum(is_correct(question, outcome) for question, outcome in outcomes), + recommended=sum(outcome.startswith(CHOSEN_PREFIX) for _, outcome in outcomes), + ) + ) + return rows + + +@dataclass +class ApiCalls: + requests: int = 0 + statuses: Counter[int] = field(default_factory=Counter) + + async def on_request(self, _request: httpx2.Request) -> None: + self.requests += 1 + + async def on_response(self, response: httpx2.Response) -> None: + self.statuses[response.status_code] += 1 + + @property + def answered(self) -> int: + return self.statuses[200] + + @property + def rate_limited(self) -> int: + return self.statuses[429] + + def too_many_rate_limits(self) -> bool: + return self.rate_limited > RATE_LIMIT_ABORT_SHARE * self.requests + + def line(self) -> str: + others = {code: n for code, n in sorted(self.statuses.items()) if code not in (200, 429)} + return ( + f"api calls: {self.answered} answered of {self.requests} made, " + f"429={self.rate_limited}, other statuses={others or 'none'}" + ) + + +ClientFactory = Callable[[], tuple[anthropic.AsyncAnthropic, ApiCalls]] + + +class RecordingAssessor(CandidateAssessor): + def __init__(self, inner: CandidateAssessor) -> None: + self._inner = inner + self.assessments: list[Assessment] = [] + + @property + def provider(self) -> str: + return self._inner.provider + + @property + def model_name(self) -> str: + return self._inner.model_name + + async def assess(self, question: str, candidate: Candidate) -> Assessment: + assessment = await self._inner.assess(question, candidate) + self.assessments.append(assessment) + return assessment + + def take(self) -> tuple[Assessment, ...]: + taken = tuple(self.assessments) + self.assessments.clear() + return taken + + +@dataclass(frozen=True, slots=True) +class Tokens: + input: int + output: int + + def __add__(self, other: "Tokens") -> "Tokens": + return Tokens(self.input + other.input, self.output + other.output) + + def line(self, model: str) -> str: + prices = LIST_PRICE_USD_PER_MTOK.get(model) + cost = ( + f"~{(self.input * prices[0] + self.output * prices[1]) / 1_000_000:.4f} USD " + "(estimated at list price)" + if prices is not None + else "cost unknown for this model" + ) + return f"tokens: input={self.input} output={self.output}, {cost}" + + +def assessment_tokens(assessments: Sequence[Assessment]) -> Tokens: + return Tokens( + input=sum(a.usage.input_tokens for a in assessments), + output=sum(a.usage.output_tokens for a in assessments), + ) + + +class RecordingAgent(CatalogAgent): + def __init__(self, inner: CatalogAgent) -> None: + self._inner = inner + self.result: AgentResult | None = None + self.error: AgentUnavailableError | None = None + + async def run(self, question: str) -> AgentResult: + self.result, self.error = None, None + try: + self.result = await self._inner.run(question) + except AgentUnavailableError as error: + self.error = error + raise + return self.result + + +def latency_line(latencies_ms: Sequence[int]) -> str: + ordered = sorted(latencies_ms) + return f"latency_ms: median={median(ordered):.0f} max={ordered[-1]} total={sum(ordered)}" diff --git a/tests/evals/test_eval_agent.py b/tests/evals/test_eval_agent.py new file mode 100644 index 0000000..c71444d --- /dev/null +++ b/tests/evals/test_eval_agent.py @@ -0,0 +1,131 @@ +import time + +import pytest +from tests.evals.harness import ( + ClientFactory, + EvalCatalog, + RecordingAgent, + RecordingAssessor, + Tokens, + accuracy_line, + assessment_tokens, + is_correct, + latency_line, + outcome_of, +) + +from contexts.assistant.application.dtos import AskCommand +from contexts.assistant.application.use_cases.ask_assistant import AskAssistant +from contexts.assistant.application.use_cases.recommend_candidate import RecommendCandidate +from contexts.assistant.infrastructure.agent.anthropic_agent_model import AnthropicAgentModel +from contexts.assistant.infrastructure.agent.langgraph_catalog_agent import LangGraphCatalogAgent +from contexts.assistant.infrastructure.agent.tools import build_tools +from contexts.assistant.infrastructure.llm.anthropic_llm_client import AnthropicLLMClient +from contexts.assistant.infrastructure.llm.claude_candidate_assessor import ( + ClaudeCandidateAssessor, +) +from shared.infrastructure.settings import Settings + +pytestmark = pytest.mark.eval + +E4_MEASURES = ( + "E4 · Full agent — measures how often AskAssistant in agent mode, end to end, returns an " + "acceptable recommendation, by the scoring rule." +) + + +async def test_e4_full_agent( + eval_catalog: EvalCatalog, anthropic_clients: ClientFactory, settings: Settings +) -> None: + client, calls = anthropic_clients() + recorder = RecordingAssessor( + ClaudeCandidateAssessor( + client=client, + model=settings.llm_model, + timeout_s=settings.llm_timeout_s, + max_retries=settings.llm_max_retries, + samples=settings.selector_samples, + max_tokens=settings.selector_max_tokens, + max_concurrency=settings.selector_max_concurrency, + ) + ) + tools = build_tools( + embedding_client=eval_catalog.embeddings, + document_search=eval_catalog.search, + recommend_candidate=RecommendCandidate( + recorder, + top_n=settings.selector_top_n, + timeout_s=settings.selector_timeout_s, + requirements_threshold=settings.selector_requirements_threshold, + min_score=settings.selector_min_score, + ), + rag_top_k=settings.rag_top_k, + selector_top_n=settings.selector_top_n, + ) + agent = RecordingAgent( + LangGraphCatalogAgent( + model=AnthropicAgentModel( + client=client, + model=settings.llm_model, + timeout_s=settings.llm_timeout_s, + max_retries=settings.llm_max_retries, + max_tokens=settings.llm_max_tokens, + ), + tools=tools, + embedding_client=eval_catalog.embeddings, + max_tool_calls=settings.agent_max_tool_calls, + ) + ) + use_case = AskAssistant( + AnthropicLLMClient( + client=client, + model=settings.llm_model, + timeout_s=settings.llm_timeout_s, + max_retries=settings.llm_max_retries, + ), + eval_catalog.embeddings, + eval_catalog.search, + max_tokens=settings.llm_max_tokens, + top_k=settings.rag_top_k, + agent=agent, + ) + + questions = eval_catalog.dataset.questions + outcomes: list[str] = [] + latencies: list[int] = [] + tokens = Tokens(0, 0) + lines: list[str] = [] + for question in questions: + started = time.perf_counter() + answer = await use_case.execute(AskCommand(question=question.text)) + latencies.append(round((time.perf_counter() - started) * 1000)) + recommendation = answer.recommendation + outcome = outcome_of( + recommendation.reason if recommendation is not None else None, + recommendation.sku if recommendation is not None else None, + ) + outcomes.append(outcome) + usage = agent.result.usage if agent.result is not None else None + if usage is None and agent.error is not None: + usage = agent.error.usage + if usage is not None: + tokens += Tokens(usage.input_tokens, usage.output_tokens) + tokens += assessment_tokens(recorder.take()) + verdict = "ok" if is_correct(question, outcome) else "WRONG" + stop = answer.stop_reason.value if answer.stop_reason is not None else "unavailable" + lines.append( + f" {question.id} {question.kind:<17} {outcome:<22} {verdict:<5} stop={stop} " + f"model_stop={answer.model_stop_reason} " + f"tool_calls={usage.tool_calls if usage is not None else 0} " + f"fallback={answer.fallback_used} latency_ms={latencies[-1]}" + ) + if calls.too_many_rate_limits(): + print(f"\n{E4_MEASURES}", *lines, f" {calls.line()}", sep="\n") + pytest.fail(f"stopped after {question.id}: too many 429s. {calls.line()}") + + print(f"\n{E4_MEASURES}") + print(f" result {accuracy_line(questions, outcomes)}") + print(*lines, sep="\n") + print(f" {latency_line(latencies)} (per question)") + print(f" {tokens.line(settings.llm_model)}") + print(f" {calls.line()}") diff --git a/tests/evals/test_eval_scoring.py b/tests/evals/test_eval_scoring.py new file mode 100644 index 0000000..88b1ce9 --- /dev/null +++ b/tests/evals/test_eval_scoring.py @@ -0,0 +1,153 @@ +from dataclasses import replace +from uuid import uuid4 + +import pytest +from tests.evals.harness import ( + ApiCalls, + EvalQuestion, + SelectorRun, + accuracy_line, + is_correct, + load_dataset, + outcome_of, + shuffled, + threshold_table, +) + +from contexts.assistant.domain.candidate import Assessment, Candidate, RubricAnswers +from contexts.assistant.domain.selection import SelectionReason + +RECOMMEND = EvalQuestion( + id="q01", + kind="recommend", + text="?", + acceptable_skus=frozenset({"MT-0002"}), + accept_no_recommendation=False, +) +NO_RECOMMENDATION = EvalQuestion( + id="q07", + kind="no_recommendation", + text="?", + acceptable_skus=frozenset(), + accept_no_recommendation=True, +) +AMBIGUOUS = EvalQuestion( + id="q09", + kind="ambiguous", + text="?", + acceptable_skus=frozenset({"MT-0016", "MT-0015"}), + accept_no_recommendation=True, +) + + +class TestScoringRule: + @pytest.mark.parametrize( + ("question", "outcome", "correct"), + [ + (RECOMMEND, "chosen:MT-0002", True), + (RECOMMEND, "chosen:MT-0001", False), + (RECOMMEND, "below_threshold", False), + (RECOMMEND, "null", False), + (RECOMMEND, "no_assessments", False), + (NO_RECOMMENDATION, "below_threshold", True), + (NO_RECOMMENDATION, "null", True), + (NO_RECOMMENDATION, "chosen:MT-0002", False), + (NO_RECOMMENDATION, "no_assessments", False), + (AMBIGUOUS, "chosen:MT-0015", True), + (AMBIGUOUS, "chosen:MT-0001", False), + (AMBIGUOUS, "below_threshold", True), + ], + ) + def test_follows_the_agreed_rule( + self, question: EvalQuestion, outcome: str, correct: bool + ) -> None: + assert is_correct(question, outcome) is correct + + def test_outcome_records_what_happened(self) -> None: + assert outcome_of(None, None) == "null" + assert outcome_of(SelectionReason.CHOSEN, "MT-0002") == "chosen:MT-0002" + assert outcome_of(SelectionReason.BELOW_THRESHOLD, None) == "below_threshold" + + def test_accuracy_is_reported_in_counts_by_kind(self) -> None: + line = accuracy_line( + [RECOMMEND, NO_RECOMMENDATION, AMBIGUOUS], + ["chosen:MT-0002", "chosen:MT-0002", "below_threshold"], + ) + + assert line == "2/3: recommend 1/1, no_recommendation 0/1, ambiguous 1/1" + + +class TestHarness: + def test_dataset_loads_with_unique_opaque_skus(self) -> None: + dataset = load_dataset() + + skus = [product.sku for product in dataset.products] + assert len(skus) == len(set(skus)) == 20 + assert all(sku.startswith("MT-") for sku in skus) + assert len(dataset.questions) == 10 + known = set(skus) + assert all(q.acceptable_skus <= known for q in dataset.questions) + + def test_product_ids_are_deterministic(self) -> None: + assert [p.product_id for p in load_dataset().products] == [ + p.product_id for p in load_dataset().products + ] + + def test_shuffle_is_seeded_by_question(self) -> None: + candidates = [_candidate(f"MT-{n:04d}", 0.5) for n in range(10)] + + assert shuffled(candidates, "q01") == shuffled(candidates, "q01") + assert sorted(shuffled(candidates, "q01"), key=lambda c: c.sku) == candidates + + def test_threshold_table_recomputes_selection_without_calls(self) -> None: + good, weak = _candidate("MT-0002", 0.9), _candidate("MT-0001", 0.8) + run = SelectorRun( + question=RECOMMEND, + outcome="chosen:MT-0002", + latency_ms=1, + candidates=(good, weak), + assessments=( + Assessment(good.product_id, RubricAnswers(1.0, 2 / 3)), + Assessment(weak.product_id, RubricAnswers(1.0, 1 / 3)), + ), + unassessed=0, + ) + + rows = {(r.requirements_threshold, r.min_score): r for r in threshold_table([run])} + + assert (rows[(2 / 3, 2 / 3)].correct, rows[(2 / 3, 2 / 3)].recommended) == (1, 1) + assert (rows[(2 / 3, 1.0)].correct, rows[(2 / 3, 1.0)].recommended) == (0, 0) + + def test_an_acceptable_sku_outside_the_candidates_is_unreachable(self) -> None: + run = SelectorRun( + question=RECOMMEND, + outcome="below_threshold", + latency_ms=1, + candidates=(_candidate("MT-0001", 0.9),), + assessments=(), + unassessed=0, + ) + + assert run.reachable is False + assert replace(run, question=NO_RECOMMENDATION).reachable is True + + async def test_api_calls_flag_many_rate_limits(self) -> None: + calls = ApiCalls(requests=20) + calls.statuses[200] = 19 + calls.statuses[429] = 1 + assert calls.too_many_rate_limits() is False + + calls.statuses[429] = 2 + assert calls.too_many_rate_limits() is True + + +def _candidate(sku: str, score: float) -> Candidate: + return Candidate( + product_id=uuid4(), + sku=sku, + name=sku, + price_cents=1000, + currency="EUR", + content=sku, + retrieval_score=score, + ) diff --git a/tests/evals/test_eval_selector.py b/tests/evals/test_eval_selector.py new file mode 100644 index 0000000..67ed14b --- /dev/null +++ b/tests/evals/test_eval_selector.py @@ -0,0 +1,169 @@ +import time +from dataclasses import dataclass + +import pytest +import pytest_asyncio +from tests.evals.harness import ( + ApiCalls, + ClientFactory, + EvalCatalog, + RecordingAssessor, + SelectorRun, + Tokens, + accuracy_line, + assessment_tokens, + candidate_of, + latency_line, + outcome_of, + shuffled, + threshold_table, +) + +from contexts.assistant.application.use_cases.recommend_candidate import RecommendCandidate +from contexts.assistant.domain.selection import SelectionReason +from contexts.assistant.infrastructure.llm.claude_candidate_assessor import ( + ClaudeCandidateAssessor, +) +from shared.infrastructure.settings import Settings + +pytestmark = pytest.mark.eval + +E1_MEASURES = ( + "E1 · Selector — measures how often the real selector (Claude assessor, top 10 candidates " + "x 3 samples) ends in an acceptable outcome, by the scoring rule." +) +E2_MEASURES = ( + "E2 · Shuffled order — measures how often the selector repeats E1's outcome when the " + "candidates arrive shuffled." +) +E3_MEASURES = ( + "E3 · Repeat — measures how often the selector repeats E1's outcome on an identical rerun " + "(model run-to-run variability)." +) + + +@dataclass(frozen=True, slots=True) +class SelectorReport: + runs: tuple[SelectorRun, ...] + calls: ApiCalls + model: str + + +async def _run_selector( + title: str, + catalog: EvalCatalog, + clients: ClientFactory, + settings: Settings, + *, + shuffle: bool, +) -> SelectorReport: + client, calls = clients() + recorder = RecordingAssessor( + ClaudeCandidateAssessor( + client=client, + model=settings.llm_model, + timeout_s=settings.llm_timeout_s, + max_retries=settings.llm_max_retries, + samples=settings.selector_samples, + max_tokens=settings.selector_max_tokens, + max_concurrency=settings.selector_max_concurrency, + ) + ) + recommend = RecommendCandidate( + recorder, + top_n=settings.selector_top_n, + timeout_s=settings.selector_timeout_s, + requirements_threshold=settings.selector_requirements_threshold, + min_score=settings.selector_min_score, + ) + runs: list[SelectorRun] = [] + for question in catalog.dataset.questions: + [embedding] = await catalog.embeddings.embed([question.text]) + hits = await catalog.search.search(embedding, settings.selector_top_n) + candidates = [c for hit in hits if (c := candidate_of(hit)) is not None] + if shuffle: + candidates = shuffled(candidates, question.id) + started = time.perf_counter() + result = await recommend.execute(question.text, candidates) + runs.append( + SelectorRun( + question=question, + outcome=outcome_of(result.reason, result.sku), + latency_ms=round((time.perf_counter() - started) * 1000), + candidates=tuple(candidates), + assessments=recorder.take(), + unassessed=len(result.candidates) - result.assessed, + ) + ) + if calls.too_many_rate_limits(): + report = SelectorReport(runs=tuple(runs), calls=calls, model=settings.llm_model) + _print(title, report) + pytest.fail(f"stopped after {question.id}: too many 429s. {calls.line()}") + return SelectorReport(runs=tuple(runs), calls=calls, model=settings.llm_model) + + +def _print(title: str, report: SelectorReport, baseline: SelectorReport | None = None) -> None: + runs = report.runs + print(f"\n{title}") + print(f" result {accuracy_line([r.question for r in runs], [r.outcome for r in runs])}") + if baseline is not None: + same = sum( + run.outcome == base.outcome for run, base in zip(runs, baseline.runs, strict=True) + ) + print(f" same outcome as E1: {same}/{len(runs)}") + reachable = sum(run.reachable for run in runs) + print(f" retrieval ceiling: an acceptable outcome is reachable in {reachable}/{len(runs)}") + for run in runs: + verdict = "ok" if run.correct else "WRONG" + print( + f" {run.question.id} {run.question.kind:<17} {run.outcome:<22} {verdict:<5} " + f"reachable={'yes' if run.reachable else 'no'} unassessed={run.unassessed} " + f"latency_ms={run.latency_ms}" + ) + no_assessments = sum(r.outcome == SelectionReason.NO_ASSESSMENTS.value for r in runs) + print(f" no_assessments (model failure, not a judgement): {no_assessments}") + print(f" unassessed candidates: {sum(r.unassessed for r in runs)}") + print(f" {latency_line([r.latency_ms for r in runs])} (per selection)") + tokens = sum((assessment_tokens(r.assessments) for r in runs), Tokens(0, 0)) + print(f" {tokens.line(report.model)}") + print(f" {report.calls.line()}") + + +@pytest_asyncio.fixture(scope="session") +async def e1( + eval_catalog: EvalCatalog, anthropic_clients: ClientFactory, settings: Settings +) -> SelectorReport: + return await _run_selector( + E1_MEASURES, eval_catalog, anthropic_clients, settings, shuffle=False + ) + + +async def test_e1_selector(e1: SelectorReport) -> None: + _print(E1_MEASURES, e1) + print(" thresholds recomputed from the same assessments (no new calls):") + for row in threshold_table(e1.runs): + print( + f" requirements_threshold={row.requirements_threshold:.2f} " + f"min_score={row.min_score:.2f}: correct {row.correct}/{len(e1.runs)}, " + f"recommends {row.recommended}/{len(e1.runs)}" + ) + + +async def test_e2_shuffled_order( + e1: SelectorReport, + eval_catalog: EvalCatalog, + anthropic_clients: ClientFactory, + settings: Settings, +) -> None: + e2 = await _run_selector(E2_MEASURES, eval_catalog, anthropic_clients, settings, shuffle=True) + _print(E2_MEASURES, e2, baseline=e1) + + +async def test_e3_repeat( + e1: SelectorReport, + eval_catalog: EvalCatalog, + anthropic_clients: ClientFactory, + settings: Settings, +) -> None: + e3 = await _run_selector(E3_MEASURES, eval_catalog, anthropic_clients, settings, shuffle=False) + _print(E3_MEASURES, e3, baseline=e1) diff --git a/tests/fakes/candidate_assessor.py b/tests/fakes/candidate_assessor.py index 696184c..dcfad9f 100644 --- a/tests/fakes/candidate_assessor.py +++ b/tests/fakes/candidate_assessor.py @@ -24,6 +24,7 @@ def __init__( self._delays = delays or {} self._default = default self.assessed: list[UUID] = [] + self.questions: list[str] = [] self.in_flight = 0 self.max_in_flight = 0 self.cancelled = 0 @@ -38,6 +39,7 @@ def model_name(self) -> str: async def assess(self, question: str, candidate: Candidate) -> Assessment: self.assessed.append(candidate.product_id) + self.questions.append(question) self.in_flight += 1 self.max_in_flight = max(self.max_in_flight, self.in_flight) try: diff --git a/tests/fakes/catalog_agent.py b/tests/fakes/catalog_agent.py new file mode 100644 index 0000000..b041af7 --- /dev/null +++ b/tests/fakes/catalog_agent.py @@ -0,0 +1,25 @@ +from contexts.assistant.domain.agent import AgentResult +from contexts.assistant.domain.exceptions import LLMUnavailableError +from contexts.assistant.domain.ports.catalog_agent import CatalogAgent + + +class FakeCatalogAgent(CatalogAgent): + def __init__( + self, + result: AgentResult | None = None, + *, + fail: bool = False, + error: LLMUnavailableError | None = None, + ) -> None: + self._result = result + self._fail = fail + self._error = error + self.questions: list[str] = [] + + async def run(self, question: str) -> AgentResult: + self.questions.append(question) + if self._error is not None: + raise self._error + if self._fail or self._result is None: + raise LLMUnavailableError("the language model is unavailable") + return self._result diff --git a/tests/fakes/documents.py b/tests/fakes/documents.py index 615f7f9..dcefa99 100644 --- a/tests/fakes/documents.py +++ b/tests/fakes/documents.py @@ -64,10 +64,18 @@ async def delete(self, product_ids: list[UUID]) -> None: class FakeDocumentSearch(DocumentSearch): - def __init__(self, results: list[RetrievedDocument] | None = None) -> None: + def __init__( + self, + results: list[RetrievedDocument] | None = None, + products: list[RetrievedDocument] | None = None, + ) -> None: self._results = list(results or []) + self._products = {product.sku: product for product in products or [] if product.sku} self.calls: list[tuple[list[float], int]] = [] async def search(self, embedding: list[float], k: int) -> list[RetrievedDocument]: self.calls.append((embedding, k)) return self._results[:k] + + async def find_by_sku(self, sku: str, embedding: list[float]) -> RetrievedDocument | None: + return self._products.get(sku) diff --git a/tests/integration/contexts/assistant/infrastructure/test_anthropic_agent_model.py b/tests/integration/contexts/assistant/infrastructure/test_anthropic_agent_model.py new file mode 100644 index 0000000..33b24e7 --- /dev/null +++ b/tests/integration/contexts/assistant/infrastructure/test_anthropic_agent_model.py @@ -0,0 +1,153 @@ +import json +from collections.abc import Callable + +import anthropic +import httpx2 +import pytest + +from contexts.assistant.domain.exceptions import LLMUnavailableError +from contexts.assistant.infrastructure.agent.anthropic_agent_model import ( + AnthropicAgentModel, + ModelTurn, +) + +Handler = Callable[[httpx2.Request], httpx2.Response] +TOOLS: list[anthropic.types.ToolParam] = [ + { + "name": "semantic_search", + "description": "Search.", + "input_schema": {"type": "object", "properties": {"query": {"type": "string"}}}, + } +] + + +def _response(content: list[dict[str, object]], stop_reason: str = "end_turn") -> httpx2.Response: + return httpx2.Response( + 200, + json={ + "id": "msg_test", + "type": "message", + "role": "assistant", + "model": "claude-test", + "content": content, + "stop_reason": stop_reason, + "stop_sequence": None, + "usage": {"input_tokens": 30, "output_tokens": 12}, + }, + ) + + +def _error(status_code: int) -> httpx2.Response: + return httpx2.Response( + status_code, json={"type": "error", "error": {"type": "api_error", "message": "boom"}} + ) + + +class Recording: + def __init__(self, response: httpx2.Response) -> None: + self._response = response + self.bodies: list[dict[str, object]] = [] + + def __call__(self, request: httpx2.Request) -> httpx2.Response: + self.bodies.append(json.loads(request.content)) + return self._response + + +def _model(handler: Handler) -> AnthropicAgentModel: + client = anthropic.AsyncAnthropic( + api_key="test-key", + max_retries=0, + http_client=anthropic.DefaultAsyncHttpxClient(transport=httpx2.MockTransport(handler)), + ) + return AnthropicAgentModel( + client=client, model="claude-test", timeout_s=1.0, max_retries=0, max_tokens=512 + ) + + +async def _respond(model: AnthropicAgentModel) -> ModelTurn: + return await model.respond( + system="system", messages=[{"role": "user", "content": "hi"}], tools=TOOLS + ) + + +class TestTurns: + async def test_tool_use_turn_exposes_the_calls_and_keeps_every_block(self) -> None: + handler = Recording( + _response( + [ + {"type": "thinking", "thinking": "", "signature": "sig-1"}, + {"type": "text", "text": "Let me search."}, + { + "type": "tool_use", + "id": "toolu_1", + "name": "semantic_search", + "input": {"query": "boots"}, + }, + ], + stop_reason="tool_use", + ) + ) + + turn = await _respond(_model(handler)) + + assert [(c.id, c.name, c.input) for c in turn.tool_calls] == [ + ("toolu_1", "semantic_search", {"query": "boots"}) + ] + assert [block["type"] for block in turn.assistant_content] == [ + "thinking", + "text", + "tool_use", + ] + assert turn.assistant_content[0] == { + "type": "thinking", + "thinking": "", + "signature": "sig-1", + } + assert (turn.input_tokens, turn.output_tokens, turn.model) == (30, 12, "claude-test") + assert turn.stop_reason == "tool_use" + + async def test_final_turn_returns_the_text(self) -> None: + turn = await _respond(_model(Recording(_response([{"type": "text", "text": "Done."}])))) + + assert turn.tool_calls == () + assert turn.text == "Done." + + async def test_rb4_reports_a_cut_off_turn(self) -> None: + turn = await _respond( + _model(Recording(_response([{"type": "text", "text": "Half"}], "max_tokens"))) + ) + + assert turn.stop_reason == "max_tokens" + + async def test_sends_the_tools_and_no_sampling_parameters(self) -> None: + handler = Recording(_response([{"type": "text", "text": "ok"}])) + + await _respond(_model(handler)) + + body = handler.bodies[0] + assert body["tools"] == TOOLS + assert "temperature" not in body and "top_p" not in body + + +class TestErrors: + @pytest.mark.parametrize("status_code", [429, 500, 529]) + async def test_b4_transient_status_raises_llm_unavailable(self, status_code: int) -> None: + with pytest.raises(LLMUnavailableError): + await _respond(_model(lambda _r: _error(status_code))) + + async def test_b4_connection_failure_raises_llm_unavailable(self) -> None: + def handler(_request: httpx2.Request) -> httpx2.Response: + raise httpx2.ConnectError("refused") + + with pytest.raises(LLMUnavailableError): + await _respond(_model(handler)) + + @pytest.mark.parametrize( + ("status_code", "sdk_error"), + [(400, anthropic.BadRequestError), (401, anthropic.AuthenticationError)], + ) + async def test_b4_configuration_error_propagates( + self, status_code: int, sdk_error: type[anthropic.APIStatusError] + ) -> None: + with pytest.raises(sdk_error): + await _respond(_model(lambda _r: _error(status_code))) diff --git a/tests/integration/contexts/assistant/infrastructure/test_anthropic_llm_client.py b/tests/integration/contexts/assistant/infrastructure/test_anthropic_llm_client.py index 486bc38..39dcd87 100644 --- a/tests/integration/contexts/assistant/infrastructure/test_anthropic_llm_client.py +++ b/tests/integration/contexts/assistant/infrastructure/test_anthropic_llm_client.py @@ -80,3 +80,26 @@ def handler(request: httpx2.Request) -> httpx2.Response: await _complete(adapter) assert len(attempts) == 1 + + +class TestStopReason: + @pytest.mark.parametrize("stop_reason", ["end_turn", "max_tokens", "refusal"]) + async def test_rb4_reports_the_model_stop_reason(self, stop_reason: str) -> None: + def handler(_request: httpx2.Request) -> httpx2.Response: + return httpx2.Response( + 200, + json={ + "id": "msg_test", + "type": "message", + "role": "assistant", + "model": "claude-test", + "content": [{"type": "text", "text": "Answer"}], + "stop_reason": stop_reason, + "stop_sequence": None, + "usage": {"input_tokens": 5, "output_tokens": 5}, + }, + ) + + completion = await _adapter(handler).complete(system="s", prompt="p", max_tokens=16) + + assert completion.stop_reason == stop_reason diff --git a/tests/integration/contexts/assistant/infrastructure/test_sqlalchemy_documents.py b/tests/integration/contexts/assistant/infrastructure/test_sqlalchemy_documents.py index 2b76ad2..7bd17fa 100644 --- a/tests/integration/contexts/assistant/infrastructure/test_sqlalchemy_documents.py +++ b/tests/integration/contexts/assistant/infrastructure/test_sqlalchemy_documents.py @@ -171,3 +171,50 @@ async def test_a8_sku_is_unique_in_the_index(self, index: SqlAlchemyDocumentInde with pytest.raises(IntegrityError): await index.upsert([_embedded("second", _vector((1, 1.0)), sku="SKU-DUP")]) + + +class TestCatalogFieldsOnRead: + async def test_search_results_carry_the_catalog_fields( + self, index: SqlAlchemyDocumentIndex, search: SqlAlchemyDocumentSearch + ) -> None: + item = _embedded("boot", _vector((0, 1.0)), sku="BOOT-1") + await index.upsert([item]) + + [hit] = await search.search(_vector((0, 1.0)), k=1) + + assert (hit.sku, hit.name, hit.price_cents, hit.currency) == ( + "BOOT-1", + "Name boot", + 1999, + "EUR", + ) + + async def test_find_by_sku_returns_the_indexed_product( + self, index: SqlAlchemyDocumentIndex, search: SqlAlchemyDocumentSearch + ) -> None: + item = _embedded("tent", _vector((0, 1.0)), sku="TENT-2") + await index.upsert([item]) + + product = await search.find_by_sku("TENT-2", _vector((0, 1.0))) + + assert product is not None + assert (product.product_id, product.name, product.content) == ( + item.document.product_id, + "Name tent", + "tent", + ) + + async def test_rb3_find_by_sku_returns_the_real_cosine_similarity( + self, index: SqlAlchemyDocumentIndex, search: SqlAlchemyDocumentSearch + ) -> None: + await index.upsert([_embedded("tent", _vector((0, 1.0), (1, 1.0)), sku="TENT-3")]) + + product = await search.find_by_sku("TENT-3", _vector((0, 1.0))) + + assert product is not None + assert product.score == pytest.approx(0.7071, abs=1e-3) + + async def test_find_by_sku_returns_none_for_an_unknown_sku( + self, search: SqlAlchemyDocumentSearch + ) -> None: + assert await search.find_by_sku("NOPE", _vector((0, 1.0))) is None diff --git a/tests/unit/contexts/assistant/application/test_ask_assistant.py b/tests/unit/contexts/assistant/application/test_ask_assistant.py index 9d3d7a5..27197fa 100644 --- a/tests/unit/contexts/assistant/application/test_ask_assistant.py +++ b/tests/unit/contexts/assistant/application/test_ask_assistant.py @@ -1,20 +1,36 @@ +from dataclasses import replace from uuid import uuid4 import pytest +from tests.fakes.catalog_agent import FakeCatalogAgent from tests.fakes.documents import FakeDocumentSearch from tests.fakes.embedding_client import FakeEmbeddingClient from tests.fakes.llm_client import FakeLLMClient -from contexts.assistant.application.dtos import AnswerDTO, AskCommand, SourceDTO, UsageDTO +from contexts.assistant.application.dtos import ( + AnswerDTO, + AskCommand, + RecommendationSummaryDTO, + SourceDTO, + UsageDTO, +) from contexts.assistant.application.use_cases.ask_assistant import ( FALLBACK_WITH_SOURCES, FALLBACK_WITHOUT_SOURCES, SYSTEM_PROMPT, AskAssistant, ) +from contexts.assistant.domain.agent import ( + AgentRecommendation, + AgentResult, + AgentStopReason, + AgentUnavailableError, + AgentUsage, +) from contexts.assistant.domain.completion import Completion from contexts.assistant.domain.exceptions import InvalidQuestionError from contexts.assistant.domain.retrieved_document import RetrievedDocument +from contexts.assistant.domain.selection import SelectionReason QUESTION_VECTOR = [0.0, 1.0, 0.0] WIDGET = RetrievedDocument( @@ -161,3 +177,144 @@ async def test_c8_llm_unavailable_with_empty_index_returns_fixed_text( assert result.fallback_used is True assert result.sources == () assert result.text == FALLBACK_WITHOUT_SOURCES + + +AGENT_RESULT = AgentResult( + text="Take the winter boot.", + sources=(WIDGET,), + recommendation=AgentRecommendation( + product_id=WIDGET.product_id, sku="W-1", score=1.0, reason=SelectionReason.CHOSEN + ), + stop_reason=AgentStopReason.ANSWERED, + usage=AgentUsage( + model="claude-test", tool_calls=2, input_tokens=300, output_tokens=40, latency_ms=900 + ), +) + + +class TestAgentMode: + async def test_delegates_to_the_agent_and_maps_its_result( + self, embeddings: FakeEmbeddingClient, search: FakeDocumentSearch + ) -> None: + agent = FakeCatalogAgent(AGENT_RESULT) + use_case = AskAssistant( + FakeLLMClient(), embeddings, search, max_tokens=256, top_k=5, agent=agent + ) + + result = await use_case.execute(AskCommand(question=" Which boot for winter? ")) + + assert agent.questions == ["Which boot for winter?"] + assert result.text == "Take the winter boot." + assert result.sources == (SourceDTO.from_retrieved(WIDGET),) + assert result.fallback_used is False + assert result.usage == UsageDTO( + model="claude-test", input_tokens=300, output_tokens=40, latency_ms=900 + ) + assert result.recommendation == RecommendationSummaryDTO( + product_id=WIDGET.product_id, sku="W-1", score=1.0, reason=SelectionReason.CHOSEN + ) + assert result.stop_reason is AgentStopReason.ANSWERED + + async def test_agent_unavailable_falls_back_to_retrieval_without_generation( + self, embeddings: FakeEmbeddingClient, search: FakeDocumentSearch + ) -> None: + use_case = AskAssistant( + FakeLLMClient(), + embeddings, + search, + max_tokens=256, + top_k=5, + agent=FakeCatalogAgent(fail=True), + ) + + result = await use_case.execute(AskCommand(question="What is a widget?")) + + assert result.fallback_used is True + assert result.usage is None + assert result.recommendation is None + assert result.sources == ( + SourceDTO.from_retrieved(WIDGET), + SourceDTO.from_retrieved(GADGET), + ) + + async def test_rag_mode_has_no_recommendation_and_no_stop_reason( + self, use_case: AskAssistant + ) -> None: + result = await use_case.execute(AskCommand(question="What is a widget?")) + + assert result.recommendation is None + assert result.stop_reason is None + + +class TestReviewFixes: + async def test_rb2_agent_failure_after_selection_keeps_recommendation_and_sources( + self, embeddings: FakeEmbeddingClient, search: FakeDocumentSearch + ) -> None: + partial = AgentUnavailableError( + sources=(GADGET,), + recommendation=AGENT_RESULT.recommendation, + usage=AGENT_RESULT.usage, + ) + use_case = AskAssistant( + FakeLLMClient(), + embeddings, + search, + max_tokens=256, + top_k=5, + agent=FakeCatalogAgent(error=partial), + ) + + result = await use_case.execute(AskCommand(question="Which boot for winter?")) + + assert result.fallback_used is True + expected = AGENT_RESULT.recommendation + assert expected is not None + assert result.sources == (SourceDTO.from_retrieved(GADGET),) + assert result.recommendation == RecommendationSummaryDTO.from_agent(expected) + assert embeddings.calls == [] + + @pytest.mark.parametrize("stop_reason", ["max_tokens", "refusal"]) + async def test_rb4_rag_mode_incomplete_answer_falls_back( + self, embeddings: FakeEmbeddingClient, search: FakeDocumentSearch, stop_reason: str + ) -> None: + completion = Completion( + text="Half an ans", + model="claude-test", + input_tokens=10, + output_tokens=256, + latency_ms=5, + stop_reason=stop_reason, + ) + use_case = _use_case(FakeLLMClient(completion), embeddings, search) + + result = await use_case.execute(AskCommand(question="What is a widget?")) + + assert result.fallback_used is True + assert result.model_stop_reason == stop_reason + assert result.text.startswith(FALLBACK_WITH_SOURCES) + + async def test_rb4_agent_mode_incomplete_answer_falls_back( + self, embeddings: FakeEmbeddingClient, search: FakeDocumentSearch + ) -> None: + incomplete = replace( + AGENT_RESULT, + text="Half an ans", + stop_reason=AgentStopReason.MODEL_INCOMPLETE, + model_stop_reason="max_tokens", + ) + use_case = AskAssistant( + FakeLLMClient(), + embeddings, + search, + max_tokens=256, + top_k=5, + agent=FakeCatalogAgent(incomplete), + ) + + result = await use_case.execute(AskCommand(question="Which boot for winter?")) + + assert result.fallback_used is True + assert result.stop_reason is AgentStopReason.MODEL_INCOMPLETE + assert result.model_stop_reason == "max_tokens" + assert result.text.startswith(FALLBACK_WITH_SOURCES) + assert result.recommendation is not None diff --git a/tests/unit/contexts/assistant/application/test_recommend_candidate.py b/tests/unit/contexts/assistant/application/test_recommend_candidate.py index 7b8ec54..498b67f 100644 --- a/tests/unit/contexts/assistant/application/test_recommend_candidate.py +++ b/tests/unit/contexts/assistant/application/test_recommend_candidate.py @@ -77,6 +77,7 @@ async def test_a6_a_transient_failure_skips_only_that_candidate(self) -> None: assert result.product_id == ok.product_id assert result.reason is SelectionReason.CHOSEN + assert result.sku == ok.sku async def test_a6_all_failing_falls_back_to_retrieval_order(self) -> None: candidates = _candidates(3) @@ -87,6 +88,7 @@ async def test_a6_all_failing_falls_back_to_retrieval_order(self) -> None: assert result.product_id is None assert result.fallback_used is True assert result.reason is SelectionReason.NO_ASSESSMENTS + assert result.sku is None assert [c.product_id for c in result.candidates] == [c.product_id for c in candidates] async def test_a6_a_configuration_error_propagates(self) -> None: diff --git a/tests/unit/contexts/assistant/infrastructure/test_agent_tools.py b/tests/unit/contexts/assistant/infrastructure/test_agent_tools.py new file mode 100644 index 0000000..cd3049e --- /dev/null +++ b/tests/unit/contexts/assistant/infrastructure/test_agent_tools.py @@ -0,0 +1,156 @@ +from uuid import uuid4 + +from tests.fakes.candidate_assessor import FakeCandidateAssessor +from tests.fakes.documents import FakeDocumentSearch +from tests.fakes.embedding_client import FakeEmbeddingClient + +from contexts.assistant.application.use_cases.recommend_candidate import RecommendCandidate +from contexts.assistant.domain.candidate import RubricAnswers +from contexts.assistant.domain.retrieved_document import RetrievedDocument +from contexts.assistant.domain.selection import SelectionReason +from contexts.assistant.infrastructure.agent.tools import ( + ToolContext, + ToolSpec, + api_tool_definitions, + build_tools, + dispatch, +) + +QUESTION = "Which of these do you recommend for winter hiking?" +YES = RubricAnswers(meets_requirements=1.0, suitable_for_use=1.0) +NO = RubricAnswers(meets_requirements=1.0, suitable_for_use=0.0) + + +def _doc(sku: str, name: str, price_cents: int, score: float) -> RetrievedDocument: + return RetrievedDocument( + product_id=uuid4(), + content=f"SKU: {sku}\nName: {name}", + score=score, + sku=sku, + name=name, + price_cents=price_cents, + currency="EUR", + ) + + +BOOT = _doc("BOOT-1", "Winter boot", 12900, 0.9) +SANDAL = _doc("SAND-1", "Summer sandal", 3900, 0.4) + + +def _tools( + assessor: FakeCandidateAssessor, embeddings: FakeEmbeddingClient | None = None +) -> list[ToolSpec]: + search = FakeDocumentSearch([BOOT, SANDAL], products=[BOOT]) + recommend = RecommendCandidate( + assessor, top_n=10, timeout_s=5.0, requirements_threshold=0.5, min_score=0.5 + ) + return build_tools( + embedding_client=embeddings or FakeEmbeddingClient(), + document_search=search, + recommend_candidate=recommend, + rag_top_k=5, + selector_top_n=10, + ) + + +CONTEXT = ToolContext(question=QUESTION, question_embedding=[0.0, 1.0, 0.0]) + + +class TestSingleDefinition: + def test_b11_api_definitions_come_from_the_tool_specs(self) -> None: + tools = _tools(FakeCandidateAssessor(default=YES)) + + definitions = api_tool_definitions(tools) + + assert [d["name"] for d in definitions] == [ + "semantic_search", + "get_product_by_sku", + "recommend_candidate", + ] + for spec, definition in zip(tools, definitions, strict=True): + assert definition["description"] == spec.description + assert definition["input_schema"] == spec.input_schema + + async def test_b11_dispatch_uses_the_same_specs_and_rejects_unknown_tools(self) -> None: + tools = _tools(FakeCandidateAssessor(default=YES)) + + outcome = await dispatch(tools, "delete_everything", {}, CONTEXT) + + assert outcome.is_error is True + assert "unknown tool" in outcome.text + + +class TestHandlers: + async def test_semantic_search_lists_the_hits_and_records_them(self) -> None: + outcome = await dispatch( + _tools(FakeCandidateAssessor()), "semantic_search", {"query": "warm boots"}, CONTEXT + ) + + assert "BOOT-1" in outcome.text and "SAND-1" in outcome.text + assert [d.sku for d in outcome.retrieved] == ["BOOT-1", "SAND-1"] + + async def test_get_product_by_sku_finds_an_indexed_product(self) -> None: + outcome = await dispatch( + _tools(FakeCandidateAssessor()), "get_product_by_sku", {"sku": "BOOT-1"}, CONTEXT + ) + + assert "Winter boot" in outcome.text + assert outcome.is_error is False + + async def test_rb3_get_product_by_sku_adds_the_product_with_its_real_score(self) -> None: + outcome = await dispatch( + _tools(FakeCandidateAssessor()), "get_product_by_sku", {"sku": "BOOT-1"}, CONTEXT + ) + + assert [(d.sku, d.score) for d in outcome.retrieved] == [("BOOT-1", BOOT.score)] + + async def test_rb3_recommend_candidate_reuses_the_question_embedding(self) -> None: + embeddings = FakeEmbeddingClient() + + await dispatch( + _tools(FakeCandidateAssessor(default=YES), embeddings), + "recommend_candidate", + {}, + CONTEXT, + ) + + assert embeddings.calls == [] + + async def test_get_product_by_sku_reports_an_unknown_sku(self) -> None: + outcome = await dispatch( + _tools(FakeCandidateAssessor()), "get_product_by_sku", {"sku": "NOPE"}, CONTEXT + ) + + assert "NOPE" in outcome.text + assert outcome.retrieved == () + + async def test_recommend_candidate_reports_the_selection(self) -> None: + assessor = FakeCandidateAssessor(outcomes={BOOT.product_id: YES, SANDAL.product_id: NO}) + + outcome = await dispatch(_tools(assessor), "recommend_candidate", {}, CONTEXT) + + assert outcome.recommendation is not None + assert outcome.recommendation.reason is SelectionReason.CHOSEN + assert outcome.recommendation.sku == "BOOT-1" + assert "BOOT-1" in outcome.text + + async def test_b1_without_a_recommendation_it_says_so_and_lists_name_sku_and_price( + self, + ) -> None: + assessor = FakeCandidateAssessor(default=NO) + + outcome = await dispatch(_tools(assessor), "recommend_candidate", {}, CONTEXT) + + assert outcome.recommendation is not None + assert outcome.recommendation.reason is SelectionReason.BELOW_THRESHOLD + assert outcome.recommendation.sku is None + assert "no clear recommendation" in outcome.text.lower() + for expected in ("Winter boot", "BOOT-1", "129.00 EUR", "Summer sandal", "39.00 EUR"): + assert expected in outcome.text + + async def test_recommend_candidate_uses_the_original_question(self) -> None: + assessor = FakeCandidateAssessor(default=YES) + + await dispatch(_tools(assessor), "recommend_candidate", {"question": "ignored"}, CONTEXT) + + assert assessor.questions == [QUESTION, QUESTION] diff --git a/tests/unit/contexts/assistant/infrastructure/test_langgraph_catalog_agent.py b/tests/unit/contexts/assistant/infrastructure/test_langgraph_catalog_agent.py new file mode 100644 index 0000000..1a78331 --- /dev/null +++ b/tests/unit/contexts/assistant/infrastructure/test_langgraph_catalog_agent.py @@ -0,0 +1,281 @@ +import json +import logging +from collections.abc import Sequence +from uuid import uuid4 + +import pytest +from anthropic.types import MessageParam, ToolParam +from tests.fakes.candidate_assessor import FakeCandidateAssessor +from tests.fakes.documents import FakeDocumentSearch +from tests.fakes.embedding_client import FakeEmbeddingClient + +from contexts.assistant.application.use_cases.recommend_candidate import RecommendCandidate +from contexts.assistant.domain.agent import AgentStopReason, AgentUnavailableError +from contexts.assistant.domain.candidate import RubricAnswers +from contexts.assistant.domain.exceptions import LLMUnavailableError +from contexts.assistant.domain.retrieved_document import RetrievedDocument +from contexts.assistant.domain.selection import SelectionReason +from contexts.assistant.infrastructure.agent.anthropic_agent_model import ModelTurn, ToolCall +from contexts.assistant.infrastructure.agent.langgraph_catalog_agent import ( + LangGraphCatalogAgent, +) +from contexts.assistant.infrastructure.agent.tools import build_tools + +QUESTION = "Which of these do you recommend for winter hiking?" +BOOT = RetrievedDocument( + product_id=uuid4(), + content="SKU: BOOT-1\nName: Winter boot\nPrice: 129.00 EUR", + score=0.9, + sku="BOOT-1", + name="Winter boot", + price_cents=12900, + currency="EUR", +) +SANDAL = RetrievedDocument( + product_id=uuid4(), + content="SKU: SAND-1\nName: Summer sandal\nPrice: 39.00 EUR", + score=0.4, + sku="SAND-1", + name="Summer sandal", + price_cents=3900, + currency="EUR", +) + + +def _turn( + *calls: ToolCall, text: str = "", stop_reason: str = "", model: str = "claude-test" +) -> ModelTurn: + content: list[dict[str, object]] = [{"type": "thinking", "thinking": "", "signature": "s"}] + if text: + content.append({"type": "text", "text": text}) + content += [ + {"type": "tool_use", "id": c.id, "name": c.name, "input": dict(c.input)} for c in calls + ] + return ModelTurn( + assistant_content=content, + text=text, + tool_calls=calls, + model=model, + input_tokens=10, + output_tokens=5, + stop_reason=stop_reason or ("tool_use" if calls else "end_turn"), + ) + + +class ScriptedModel: + def __init__(self, turns: list[ModelTurn], rag_text: str = "RAG answer") -> None: + self._turns = list(turns) + self._rag_text = rag_text + self.calls: list[tuple[list[MessageParam], bool]] = [] + + @property + def model_name(self) -> str: + return "claude-test" + + async def respond( + self, + *, + system: str, + messages: Sequence[MessageParam], + tools: Sequence[ToolParam] | None, + ) -> ModelTurn: + self.calls.append((list(messages), bool(tools))) + if not tools: + return _turn(text=self._rag_text) + return self._turns.pop(0) if self._turns else _turn(_search_call(len(self.calls))) + + +def _search_call(n: int) -> ToolCall: + return ToolCall(id=f"toolu_{n}", name="semantic_search", input={"query": f"boots {n}"}) + + +def _agent( + model: ScriptedModel, + max_tool_calls: int = 5, + embeddings: FakeEmbeddingClient | None = None, +) -> LangGraphCatalogAgent: + embedding_client = embeddings or FakeEmbeddingClient() + assessor = FakeCandidateAssessor( + outcomes={ + BOOT.product_id: RubricAnswers(meets_requirements=1, suitable_for_use=1), + SANDAL.product_id: RubricAnswers(meets_requirements=1, suitable_for_use=0), + } + ) + tools = build_tools( + embedding_client=embedding_client, + document_search=FakeDocumentSearch([BOOT, SANDAL], products=[BOOT]), + recommend_candidate=RecommendCandidate( + assessor, top_n=10, timeout_s=5, requirements_threshold=0.5, min_score=0.5 + ), + rag_top_k=5, + selector_top_n=10, + ) + return LangGraphCatalogAgent( + model=model, + tools=tools, + embedding_client=embedding_client, + max_tool_calls=max_tool_calls, + ) + + +class TestStepLimit: + async def test_b2_b12_five_tool_calls_then_rag_mode( + self, caplog: pytest.LogCaptureFixture + ) -> None: + model = ScriptedModel([]) + caplog.set_level(logging.INFO, logger="contexts.assistant") + + result = await _agent(model).run(QUESTION) + + assert result.stop_reason is AgentStopReason.MAX_STEPS + assert result.usage.tool_calls == 5 + assert result.text == "RAG answer" + rag_messages, rag_with_tools = model.calls[-1] + assert rag_with_tools is False + assert "BOOT-1" in str(rag_messages) + assert any("stop_reason=max_steps" in r.getMessage() for r in caplog.records) + + async def test_b12_a_batch_that_would_pass_the_limit_is_not_executed(self) -> None: + model = ScriptedModel([_turn(_search_call(1), _search_call(2), _search_call(3))]) + + result = await _agent(model, max_tool_calls=2).run(QUESTION) + + assert result.usage.tool_calls == 0 + assert result.stop_reason is AgentStopReason.MAX_STEPS + + +class TestAnswered: + async def test_b10_recommendation_comes_from_the_selection_not_the_text(self) -> None: + model = ScriptedModel( + [ + _turn(ToolCall(id="toolu_1", name="recommend_candidate", input={})), + _turn(text="I recommend SAND-1."), + ] + ) + + result = await _agent(model).run(QUESTION) + + assert result.stop_reason is AgentStopReason.ANSWERED + assert result.text == "I recommend SAND-1." + assert result.recommendation is not None + assert result.recommendation.reason is SelectionReason.CHOSEN + assert result.recommendation.sku == "BOOT-1" + + async def test_answering_without_recommending_leaves_recommendation_null(self) -> None: + model = ScriptedModel([_turn(text="Boots keep you warm.")]) + + result = await _agent(model).run(QUESTION) + + assert result.recommendation is None + assert result.usage.tool_calls == 0 + assert result.stop_reason is AgentStopReason.ANSWERED + + async def test_tool_results_go_back_with_their_ids_after_the_assistant_blocks(self) -> None: + model = ScriptedModel([_turn(_search_call(1)), _turn(text="done")]) + + await _agent(model).run(QUESTION) + + second_messages = json.loads(json.dumps(model.calls[1][0])) + assistant, user = second_messages[-2], second_messages[-1] + assert assistant["role"] == "assistant" + assert [block["type"] for block in assistant["content"]] == ["thinking", "tool_use"] + assert user["content"][0]["tool_use_id"] == "toolu_1" + + async def test_usage_is_summed_across_steps(self) -> None: + model = ScriptedModel([_turn(_search_call(1)), _turn(text="done")]) + + result = await _agent(model).run(QUESTION) + + assert (result.usage.input_tokens, result.usage.output_tokens) == (20, 10) + + +class TestErrors: + async def test_a_transient_model_error_propagates(self) -> None: + class Failing(ScriptedModel): + async def respond( + self, + *, + system: str, + messages: Sequence[MessageParam], + tools: Sequence[ToolParam] | None, + ) -> ModelTurn: + raise LLMUnavailableError("down") + + with pytest.raises(LLMUnavailableError): + await _agent(Failing([])).run(QUESTION) + + +class TestReviewFixes: + async def test_rb2_a_model_failure_after_selection_carries_the_partial_state(self) -> None: + class FailsOnSecondTurn(ScriptedModel): + async def respond( + self, + *, + system: str, + messages: Sequence[MessageParam], + tools: Sequence[ToolParam] | None, + ) -> ModelTurn: + if self.calls: + raise LLMUnavailableError("529") + self.calls.append((list(messages), bool(tools))) + return _turn(ToolCall(id="toolu_1", name="recommend_candidate", input={})) + + with pytest.raises(AgentUnavailableError) as raised: + await _agent(FailsOnSecondTurn([])).run(QUESTION) + + partial = raised.value + assert partial.recommendation is not None + assert partial.recommendation.sku == "BOOT-1" + assert BOOT.product_id in {document.product_id for document in partial.sources} + assert (partial.usage.input_tokens, partial.usage.tool_calls) == (10, 1) + + async def test_rb3_the_question_is_embedded_once_per_run(self) -> None: + embeddings = FakeEmbeddingClient() + model = ScriptedModel( + [ + _turn( + ToolCall(id="toolu_1", name="recommend_candidate", input={}), + ToolCall(id="toolu_2", name="get_product_by_sku", input={"sku": "BOOT-1"}), + ), + _turn(text="done"), + ] + ) + + await _agent(model, embeddings=embeddings).run(QUESTION) + + assert embeddings.calls == [[QUESTION]] + + async def test_rb4_an_incomplete_turn_stops_without_running_its_tools(self) -> None: + model = ScriptedModel([_turn(_search_call(1), stop_reason="max_tokens")]) + + result = await _agent(model).run(QUESTION) + + assert result.stop_reason is AgentStopReason.MODEL_INCOMPLETE + assert result.model_stop_reason == "max_tokens" + assert result.usage.tool_calls == 0 + + async def test_rb4_a_refused_rag_answer_is_model_incomplete(self) -> None: + class RefusesRag(ScriptedModel): + async def respond( + self, + *, + system: str, + messages: Sequence[MessageParam], + tools: Sequence[ToolParam] | None, + ) -> ModelTurn: + self.calls.append((list(messages), bool(tools))) + if not tools: + return _turn(text="", stop_reason="refusal") + return _turn(_search_call(len(self.calls))) + + result = await _agent(RefusesRag([]), max_tool_calls=1).run(QUESTION) + + assert result.stop_reason is AgentStopReason.MODEL_INCOMPLETE + assert result.model_stop_reason == "refusal" + + async def test_rb5_usage_model_is_the_model_the_api_returned(self) -> None: + model = ScriptedModel([_turn(text="done", model="claude-test-20261001")]) + + result = await _agent(model).run(QUESTION) + + assert result.usage.model == "claude-test-20261001" diff --git a/tests/unit/contexts/assistant/ui/test_schemas.py b/tests/unit/contexts/assistant/ui/test_schemas.py new file mode 100644 index 0000000..c133088 --- /dev/null +++ b/tests/unit/contexts/assistant/ui/test_schemas.py @@ -0,0 +1,46 @@ +from uuid import uuid4 + +import pytest +from pydantic import ValidationError + +from contexts.assistant.ui.schemas import RecommendationResponse + + +class TestRecommendationInvariant: + def test_b14_chosen_with_every_field_is_valid(self) -> None: + response = RecommendationResponse( + product_id=uuid4(), sku="BOOT-1", score=0.67, reason="chosen" + ) + + assert response.reason == "chosen" + + @pytest.mark.parametrize("reason", ["below_threshold", "no_assessments"]) + def test_b14_no_choice_with_every_field_null_is_valid(self, reason: str) -> None: + response = RecommendationResponse(product_id=None, sku=None, score=None, reason=reason) + + assert response.product_id is None + + @pytest.mark.parametrize( + "missing", [{"product_id": None}, {"sku": None}, {"score": None}] + ) + def test_b14_chosen_requires_every_field(self, missing: dict[str, None]) -> None: + fields: dict[str, object] = { + "product_id": uuid4(), + "sku": "BOOT-1", + "score": 1.0, + "reason": "chosen", + } + fields.update(missing) + + with pytest.raises(ValidationError, match="chosen"): + RecommendationResponse.model_validate(fields) + + def test_b14_no_choice_must_not_carry_a_product(self) -> None: + with pytest.raises(ValidationError, match="chosen"): + RecommendationResponse( + product_id=uuid4(), sku=None, score=None, reason="below_threshold" + ) + + def test_reason_is_a_closed_code(self) -> None: + with pytest.raises(ValidationError): + RecommendationResponse(product_id=None, sku=None, score=None, reason="max_steps")