diff --git a/README.md b/README.md index 5b21933..13dfc9c 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +현재 구현: [정확한 근거 필드·불확실 상태 지침](docs/changes/guarded-question-reference-rules.md). 신규47/관련532·scoped정적검사·독립 코드 검토를 통과했습니다. 실제1요청에서 모델 완료/명제3개 SDK수용·JSON 재검증을 확인했으며, 필수3명제는 AI 개발판정에서 보존을 확인했으며, 전체9,515개·정적 검사/빌드/OpenAPI도 통과했습니다. 일반 품질·앱 인수와 구분합니다. + 최신 실제 관측: [질문별 근거 스트림 인수](docs/research/guarded-question-acceptance.md). 요청1회/HTTP200·모델 완료·최종 사용량2,387/2,569를 확인했지만 주체/서술어 참조 누락과 불확실 상태 불일치로 SDK가 거부했습니다. 재호출/자동 보정 없이 원래 결과를 보존하며 의미·앱 인수는 남습니다. 현재 구현: [질문별 근거 스트림 연결](docs/changes/guarded-question-evidence.md). 신규33/관련287·새전체9,468개(Python8,648+JS820)·정적 검사/빌드/OpenAPI·독립 검토 C0I0M0를 통과했다. 기존472실행 파일과 직전69개 증거를 보존했으며 제공자 호출0·앱/worker/main 활성화0이다. 실제 모델 의미와 versioned worker/앱 인수는 남는다. diff --git a/docs/README.md b/docs/README.md index b1d49a7..7e452b6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,3 +1,5 @@ +현재 구현: [정확한 근거 필드·불확실 상태 지침](changes/guarded-question-reference-rules.md). 신규47/관련532·scoped정적검사·독립 코드 검토를 통과했습니다. 실제1요청에서 모델 완료/명제3개 SDK수용·JSON 재검증을 확인했으며, 필수3명제는 AI 개발판정에서 보존을 확인했으며, 전체9,515개·정적 검사/빌드/OpenAPI도 통과했습니다. 일반 품질·앱 인수와 구분합니다. + 최신 실제 관측: [질문별 근거 스트림 인수](research/guarded-question-acceptance.md). 요청1회/HTTP200·모델 완료·최종 사용량2,387/2,569를 확인했지만 주체/서술어 참조 누락과 불확실 상태 불일치로 SDK가 거부했습니다. 재호출/자동 보정 없이 원래 결과를 보존하며 의미·앱 인수는 남습니다. 현재 구현: [질문별 근거 스트림 연결](changes/guarded-question-evidence.md). 신규33/관련287·새전체9,468개(Python8,648+JS820)·정적 검사/빌드/OpenAPI·독립 검토 C0I0M0를 통과했다. 기존472실행 파일과 직전69개 증거를 보존했으며 제공자 호출0·앱/worker/main 활성화0이다. 실제 모델 의미·합산 예산/권한/복구·앱 인수는 후속이다. Sites v129에 요약을 게시했고 이전v128을 보존했다. diff --git a/docs/changes/guarded-question-reference-rules.md b/docs/changes/guarded-question-reference-rules.md new file mode 100644 index 0000000..9539c6e --- /dev/null +++ b/docs/changes/guarded-question-reference-rules.md @@ -0,0 +1,34 @@ +# 질문별 근거 참조·불확실 상태 규칙 보완 + +2026-10-26 KST. [실제1회 거부](../research/guarded-question-acceptance.md)에서 정확한 인용17회가 있어도 subject_refs/predicate_refs를 비우고 source_statement를 제출한 두 명제를 확인했다. 검증기를 완화하지 않고 새 guarded-v2 SDK의 지침을 명확히 한다. 이전 응답·SDK·모든 비용을 보존한다. + +## Global Constraints + +- Only new `packages/agents/src/issuedigest_agents/guarded_question_evidence_v2.py` and its new test file may change execution code. Existing474 tracked non-Markdown files (including both previous generators, source validators, transport, worker, exports, dependencies) remain byte-identical. Documentation may change. No automatic repair/filling/downgrade, retries/fallback, app/worker activation, DB/server writes or main merge. Implementer makes no actual provider/network calls. +- New version-local SDK mirrors the existing async guarded SDK's strict source/hash/usage/receipt/error/cancellation guarantees, with public `GuardedQuestionEvidenceGenerationV2`, `GuardedQuestionEvidenceV2Error`, `run_guarded_question_evidence_v2(*, client: AsyncOpenAI, model: str, source: Source, question: str)` and `validate_guarded_question_evidence_v2(source, question, result)`. Reuse existing pure helpers rather than clone their internals. A narrow version-local request/receipt orchestration duplicate is acceptable to preserve frozen historical execution. New receipt profile exact `guarded-question-evidence-v2`; transport_profile remains `guarded-responses-tool-v1`, input_format remains `question-evidence-source-v2`, status unreviewed. Do not impersonate v1 receipt or silently migrate it. +- `REFERENCE_RULES_SUFFIX` exact bytes (one leading ASCII space, no trailing newline): ` Field rules: fill subject_refs and predicate_refs with exact spans when supported. If subject_refs or predicate_refs is empty, epistemic_status must be uncertain. Empty quantity metric_refs or relative-time anchor_refs also requires uncertain. Broad supporting_refs cannot replace these fields; never invent missing references.` +- `GUARDED_QUESTION_V2_INSTRUCTIONS = QUESTION_EVIDENCE_V2_INSTRUCTIONS + REFERENCE_RULES_SUFFIX`. Whole prompt/quotes/question/tool name `fill_question_evidence_v2` and existing schema remain unchanged. Reuse v2 input preflight then additionally enforce `_request_envelope(prompt, tools, GUARDED_QUESTION_V2_INSTRUCTIONS, 4000) <= 20000` before native transport. Stored instructions hash and saved validation use the new exact instructions, not the old ones. No changed token/output/deadline/stream limits, no new provider settings or model default. +- Same actual-type/raw-required guards, original source hash before/after await/construction/validation, strict budgets, source proposition validator, one borrowed native call and lifecycle, sanitized observational errors, cancellation identity, known observed/final usage and close confirmation as guarded v1. Preserve valid over-budget terminal observations before rejecting output. Unknown ordinary failures claim no invented final usage/closure. Reject missing/forged profiles/hashes/original DTO/raw required fields. Native failure carries original reason/observations unchanged; source rejection after completed transport retains known usage. Missing references with source_statement must still fail; uncertain with legitimately empty fields may structurally pass without certifying semantic coverage. Broad support alone never fills field references. + +## Task 1: Versioned exact field-rule generator + +- [x] Implement only the new two Python files plus this checkbox using TDD. + +Read test-driven-development. RED before implementation, GREEN then focused Ruff/format/mypy and related frozen guarded-v1/native/source tests. Real AsyncOpenAI+httpx2.MockTransport SSE integration asserts exact328-byte suffix, new instructions wire/hash, unchanged full prompt/tool/schema/options, one request and borrowed client remains open. Successful nonempty source-attributed proposition + JSON save/revalidate; empty insufficient accepted but unreviewed. Source_statement with missing subject or predicate, missing quantity metric, unanchored relative time rejects; same structurally uncertain data may pass without filling/replacing any field. Old/new receipt/hash substitutions fail. New instructions envelope checked before dispatch (case old envelope fits but new exceeds; boundary <=20000). Native error/cancel/observed usage, source mutation and output rejection retain prior guarantees; do not duplicate all native event matrix. Strict unchecked DTO/raw omission tests can reuse existing fixtures/guards. Module import isolation must preserve frozen reload tests. No private source/real network/model calls; public fixtures only. Commit owned files after scoped tests; write report with RED/GREEN/selfreview/static results. Controller runs fullQA/reviews; do not spawn subagents. + +## Controller gates + +- [x] Independent task/whole-branch review C0I0M0, full9,515 checks, historical byte preservation and docs/privacy checks. +- [x] New instructions measured against old full development input; preregister source/criteria/rubric/settings/wire/hash, consume at most1 real request in a new private namespace after code review. No retries/repair/limit increase; partial/invalid output not semantic success. Preserve all old paid namespaces. +- [x] Compare completion/structural acceptance/JSON saved validation and P5/P9/P10 meaning separately, other9/total12 unchanged. Independent evidence review; no general/holdout/human truth claims. +- [x] Draft PR109, Sitesv131/previousv130/7exact docs snapshots, ownQA1/volume1/SDD cleanup; no app activation. + +## 판단과 영향 + +새 버전 지침에서 실제 빈 필드 조건을 명시하고 기존 검증기·SDK는 유지한다. 과거 실행 재생을 보존하지만 버전별 요청 조율 코드가 일부 중복되며 실제 모델의 규칙 준수를 보장하지 않는다. 지침328bytes를 요청 예산에 포함하므로 상한 근처 입력은 기존 경로보다 먼저 차단될 수 있다. + +## 검증 결과 + +신규47/관련532·새전체9,515개(Python8,695+JS820)·정적 검사/빌드/OpenAPI·실패/skip/warning0·태스크/전체 코드 검토 C0I0M0다. [실제1회](../research/guarded-question-reference-acceptance.md)에서 모델 완료/명제3개 SDK수용·현재 원문/JSON 왕복·최종usage2,444/2,781을 확인했다. 독립 AI 개발판정은 P5/P9/P10 explicit3·다른9미평가, 증거 검토 C0I0M0다. 이전474실행/원문·응답·실패·사용량은 보존하며 앱/worker/main 활성화는 하지 않았다. + +최종 인계: [Draft PR109](https://github.com/DigestWorks/IssueDigest/pull/109)·Sitesv131 public/native succeeded, 이전v130/변경7문서 정확 사본(70325c2)을 보존했다. Site source `ae602f7f73f4314d026e28e9c70e1b649f882686`, saved `appgprj_6ac612dd42708191a7a0c8377be64d48~appgver_fef73bb1d6748191b31a9c00d425ad76`, deployment `appgdep_6adeb6fd9a408191933065b216cac3fc`다. HTML116/로컬대상3,335/공개텍스트350·비공개60+12항목 제외를 확인했고 사용자 탭 이동/새브라우저QA0이다. 자체QA1/전용volume1을 정리했으며 실제 요청은 이번 최초1회뿐이다. 이후 인계는 저장소 원본을 따르고 main/앱/worker 활성화는 없다. diff --git a/docs/planning/implementation-plan.md b/docs/planning/implementation-plan.md index f699d97..d416ee6 100644 --- a/docs/planning/implementation-plan.md +++ b/docs/planning/implementation-plan.md @@ -1,3 +1,7 @@ +## M3 진행 · 명시적 근거 필드·불확실 상태 + +[guarded-v2](../changes/guarded-question-reference-rules.md)는 실제 빈 필드 조건을328byte로 명시하고 이전 SDK/검증기는 보존한다. [단일 실제 인수](../research/guarded-question-reference-acceptance.md)에서 명제3개의 SDK수용/원문 검증/JSON왕복을 확인했다. 필수3명제는 비맹검 AI개발판정 explicit3/다른9미평가·독립 증거 검토 C0I0M0이며 새전체9,515개(Python8,695+JS820)·정적 검사/빌드/OpenAPI·실패/skip/warning0를 통과했다. 실제 worker/합산 예산/권한/복구·앱 인수는 별도로 남는다. + ## M3 실제 인수 · 생성 스트림의 원문 계약 거부 [단일 실제 요청](../research/guarded-question-acceptance.md)은 모델 완료/최종 사용량까지 수신했지만 주체·서술어 참조 누락과 불확실 상태 불일치로 SDK가 거부했다. 인용 위치17회 exact를 의미 성공으로 세지 않으며 재호출/보정하지 않는다. 출력 지침·참조/uncertainty를 먼저 보완하고 실제 구조/의미를 인수한 뒤 versioned worker·합산 예산/권한/복구·앱 연결로 진행한다. 기존 앱·worker·SDK는 유지한다. diff --git a/docs/planning/progress.md b/docs/planning/progress.md index 32189f3..186b324 100644 --- a/docs/planning/progress.md +++ b/docs/planning/progress.md @@ -1,3 +1,9 @@ +## 현재 구현·실제 관측 · 정확한 근거 필드·불확실 상태 · 2026-10-26 + +[새 SDK 계약](../changes/guarded-question-reference-rules.md)·[실제 인수](../research/guarded-question-reference-acceptance.md). 신규47/관련532·scoped정적검사·태스크/전체 코드 검토 C0I0M0. 같은 원문/Q2·새328byte 지침/envelope19,907을 사전 고정해 실제1요청/HTTP200/completed·30.925초·명제3개 SDK수용/JSON저장 재검증을 확인했다. 최종usage2,444/2,781(추론포함)을 보존했고 재호출/보정0이다. 독립 AI개발판정은 필수3명제 explicit3/다른9미평가이며 증거검토 C0I0M0다. 일반/사람/holdout 품질로 확대하지 않는다. 새전체9,515개(Python8,695+JS820)·정적 검사/빌드/OpenAPI·실패/skip/warning0를 통과했다. Sites/PR 최종 인계는 아래 기록을 따른다. 기존474실행/과거 결과·비용·앱/worker/main은 유지한다. + +최종 인계: [Draft PR109](https://github.com/DigestWorks/IssueDigest/pull/109)·Sitesv131 public/native succeeded, 이전v130/변경7문서 정확 사본(70325c2)을 보존했다. Site source `ae602f7f73f4314d026e28e9c70e1b649f882686`, saved `appgprj_6ac612dd42708191a7a0c8377be64d48~appgver_fef73bb1d6748191b31a9c00d425ad76`, deployment `appgdep_6adeb6fd9a408191933065b216cac3fc`다. HTML116/로컬대상3,335/공개텍스트350·비공개60+12항목 제외를 확인했고 사용자 탭 이동/새브라우저QA0이다. 자체QA1/전용volume1을 정리했으며 실제 요청은 이번 최초1회뿐이다. 이후 인계는 저장소 원본을 따르고 main/앱/worker 활성화는 없다. + ## 실제 관측 · 질문별 근거 스트림 · 2026-10-25 [실제 인수 기록](../research/guarded-question-acceptance.md). 같은 전체 원문·Q2·3필수/12기준과 새SDK 모델/settings/body를 사전 고정하고1회 호출했다. HTTP200/completed·31.842초·최종usage입력2,387/출력2,569를 받았으나 source contract가 주체/서술어 근거 누락+uncertain 불일치를 거부해 저장 결과를 만들지 못했고 의미 인수는 진행하지 않았다. 필수3명제는 SDK 거부로 미평가이며 의미0/3으로 세지 않는다. 기존474실행·SDK36/실제58/진단66 증거·원문/기준/실패/비용은 유지한다. 전체9,468QA는 코드 불변의 구조 증거로 재사용하며 신규전체QA/서버자원/앱활성화0이다. 다음은 정확참조·uncertainty 보완이며 독립 증거 검토 C0I0M0·사전 고정/보존·문서/비공개자료 검사를 통과했고 PR·Sites 인계를 진행한다. diff --git a/docs/planning/tasks.md b/docs/planning/tasks.md index 02c58df..3bc0174 100644 --- a/docs/planning/tasks.md +++ b/docs/planning/tasks.md @@ -1,3 +1,12 @@ +## 진행 · 정확한 근거 필드·불확실 상태 · 2026-10-26 + +- [x] [새 guarded-v2 지침/영수증](../changes/guarded-question-reference-rules.md) 고정·328byte/기존 입력 envelope19,907. +- [x] TDD47/관련532·독립 코드 검토 C0I0M0·전체9,515/정적 검사/빌드/OpenAPI·이전474실행/51+36증거 보존. +- [x] 사전 고정 후 실제1회·completed/SDK수용/JSON왕복·최종usage2,444/2,781·재호출0. +- [x] 필수3명제 explicit3(AI 비맹검 개발판정)/다른9미평가·독립 증거 검토 C0I0M0. +- [x] Draft PR109·Sitesv131/이전v130/7문서 정확 사본·자체QA/volume/SDD 정리·인계. +- [ ] 실제 구조/의미 이후 versioned worker·합산 예산/권한/복구·앱 인수. + ## 실제 관측 · 질문별 근거 스트림 · 2026-10-25 - [x] [원문/Q2/기준/단일 요청 사전 고정](../research/guarded-question-acceptance.md)·474실행/이전SDK36/과거 실제58·진단66 증거 보존. diff --git a/docs/research/guarded-question-reference-acceptance.md b/docs/research/guarded-question-reference-acceptance.md new file mode 100644 index 0000000..c053462 --- /dev/null +++ b/docs/research/guarded-question-reference-acceptance.md @@ -0,0 +1,41 @@ +# 명시적 근거 필드 지침 · 실제 인수 + +2026-10-26 KST. [guarded-v2 계약](../changes/guarded-question-reference-rules.md)의 구현·코드 검토 후 같은 개발 자료로 최대1회 실제 인수를 진행한다. 이전 [guarded-v1 거부](guarded-question-acceptance.md)와 입력·필수 기준/루브릭·모델/상한은 유지하고 정확328byte field-rule 지침만 새 SDK에 추가한다. 기존 응답·원문·기준·사용량/소진 잠금은 변경하지 않는다. + +## 응답 전 고정 + +원래 source/criteria/rubric 파일을 bytes 그대로 복사한다. 선택 질문Q2·필수P5/P9/P10, 전체12기준/다른9개 미평가는 유지한다. 모델 `gpt-6.1-sol`, reasoning low/service_tier default, 출력4,000/전체20,000·60초 operation/별도1초 cooperative close·무재시도/strict tool1개/no-store를 유지한다. envelope는 기존 보수적 UTF-8/schema·출력 상한 비교 값으로 실제 provider 토큰 usage와 다르다. 새 지침으로 정확한 기존 입력의 envelope는19,907이며 추가 지침 때문에 입력을 줄이거나 상한을 높이지 않는다. + +실제 송신 전 현재 실행 파일·protocol·runner·정확 wire·원문/질문/기준·schema/지침·예산 hash를 새 namespace에 고정하고 소진 잠금을 먼저 기록한다. 실패·거부·타임아웃 후 재호출/수리·모델/자료/한도 교체는 없다. actual API 호출은 최대1회이며 합성 MockTransport의 횟수와 구분한다. old paid runner는 실행하지 않는다. 원문/응답/SSE는 읽은 부분만 private로 보존하고 키/헤더를 기록하지 않는다. + +## 판정 + +HTTP/모델 terminal 완료·strict DTO·현재 원문/질문과 새 SDK 저장 영수증의 JSON 왕복·의미를 구분한다. JSON 왕복은 DB/앱 저장이 아니다. source_statement에 필요한 subject_refs/predicate_refs, quantity metric_refs/relative anchor가 비면 기존 source 검증기는 그대로 거부한다. uncertain으로 유효하게 제출해도 의미 완전성·사실 확인 성공으로 올리지 않는다. 원래 결과를 자동 다운그레이드하거나 보완하지 않는다. + +원래 Q2필수3명제의 신청 개수/각각 상한(P5), 명시 법률상 기관/광역자치단체 신청 자격(P9), 공공기관의 상위기관 승인 선행조건(P10)을 명제별 좁은 필드와 직접 인용으로만 판단한다. explicit/equivalent/partial/not_represented/contradicted/undetermined를 유지하며 부분 보존은 완전 성공이 아니다. SDK 거부면3개도 미평가, 다른 질문9개는 미평가를 유지한다. 빈 부족 응답이나 단어/인용 위치 일치만 의미 성공으로 부르지 않는다. 이미 본 개발 자료·비맹검1회이며 실제 모델 개선율/holdout·사람/중립성/일반 품질을 인증하지 않는다. + +최종 usage 미관측은0/무료가 아니며 관측 비용을 보존한다. HTTP 닫힘은 제공자 계산 취소/과금 절감 증거가 아니다. 앱/worker/main·DB/운영 데이터는 변경하지 않는다. 완료/구조/의미와 versioned worker의 합산 예산·권한·복구 인수는 별도다. + +## 결과 + +실제 요청1회/HTTP200·response.completed·30.925초, 새 `guarded-question-evidence-v2` SDK가 명제3개를 수용했다. 현재 원문/정확 질문의 결과를 JSON 파일에 저장한 뒤 새 validator로 재구성해 같은 영수증을 확인했다. DB/앱 저장이나 일반 의미 성공을 뜻하지 않는다. 재호출·수리·자동 필드/상태 보완·모델/자료/한도 변경0이다. + +최종 관측 usage는 입력2,444/출력2,781/합계5,225·요청1이다. 추론1,476은 출력에 포함되며 추가 가산하지 않는다. raw cached read0/cache write2,308을 보존했고 최종 청구 금액·캐시 TTL/비용 절감은 미확인이다. 읽은 SSE413,639bytes/HTTP1,670chunks를 보존했으며 local body 닫힘·반환 시 borrowed client 유지도 확인했다. HTTP chunk/event/token 수를 혼용하지 않는다. + +원래 인자3,668UTF-8bytes는 native JSON duplicate-key/nonfinite·strict DTO/canonical·현재 원문 계약 검사를 통과했다. 인용 참조20회는 모두 exact이며 반복을 포함한 occurrence 수다. 이것을 독립 인용20개나 의미 보존20/20으로 해석하지 않는다. 이전 raw 응답을 보정한 결과가 아니며 신규 요청의 원래 출력을 그대로 검사했다. 단일 개발 관측이므로 규칙 준수 개선율/재현율·기존 요청과 인과 효과를 확정하지 않는다. + +독립 AI 개발 검토의 첫 판정을 원래 주석과 별도로 보존했고, 조율자가 아래 근거를 사전 기준과 대조해 같은 판정에 동의했다. SDK 수용만을 근거로 삼지 않았다. + +| 필수 명제 | 개발 판정 | 좁은 필드·관계의 근거 | +| --- | --- | --- | +| P5 | explicit | 신청 주체/행위와 수량·상한·분배 관계가 각 범주별 독립 신청 상한을 보존한다. 범주를 함께 묶은 참조 span도 별도 신청 의미를 유지한다. | +| P9 | explicit | 자격 facet이 두 신청 주체 종류와 명시 법률상 한정을 보존한다. | +| P10 | explicit | 주체/행위와 선행조건 facet이 상위기관 승인을 신청 전 조건으로 보존한다. | + +선택3개는 explicit3개이며 다른 질문9개는 미평가·전체12기준을 유지한다. 출처 정보의 좁은 의미 보존에 대한 비맹검 AI 개발 판정이고 사람 정답/holdout/일반 정확도·진실·중립성 인증이 아니다. 결과 status는 계속 unreviewed다. 같은3명제에 대한 동의를 독립 표본6개나 전체12개 성공으로 세지 않는다. + +## 증거 검토 + +독립 검토 C0/I0/M0. 단일 소진 요청·사전 고정12파일/정확 body와328byte 차이·현재476실행 파일/이전474실행·51/36과거 증거 보존·raw 인자와 새 저장 결과 동일성·현재 원문/JSON 재검증을 확인했다. 자동 보정·추가 제공자 호출0이다. 실제 제공자 측 request audit·청구서·앱 저장/worker 복구 인수는 별도다. 새전체9,515개(Python8,695+JS820)·정적 검사/빌드/OpenAPI·실패/skip/warning0를 통과했다. PR/Sites 결과는 최종 인계에 기록한다. + +최종 인계: [Draft PR109](https://github.com/DigestWorks/IssueDigest/pull/109)·Sitesv131 public/native succeeded, 이전v130/변경7문서 정확 사본(70325c2)을 보존했다. Site source `ae602f7f73f4314d026e28e9c70e1b649f882686`, saved `appgprj_6ac612dd42708191a7a0c8377be64d48~appgver_fef73bb1d6748191b31a9c00d425ad76`, deployment `appgdep_6adeb6fd9a408191933065b216cac3fc`다. HTML116/로컬대상3,335/공개텍스트350·비공개60+12항목 제외를 확인했고 사용자 탭 이동/새브라우저QA0이다. 자체QA1/전용volume1을 정리했으며 실제 요청은 이번 최초1회뿐이다. 이후 인계는 저장소 원본을 따르고 main/앱/worker 활성화는 없다. diff --git a/packages/agents/src/issuedigest_agents/guarded_question_evidence_v2.py b/packages/agents/src/issuedigest_agents/guarded_question_evidence_v2.py new file mode 100644 index 0000000..6c5f647 --- /dev/null +++ b/packages/agents/src/issuedigest_agents/guarded_question_evidence_v2.py @@ -0,0 +1,172 @@ +"""Borrowed native transport for unreviewed source-specific evidence receipts. + +Structural consistency does not establish meaning, access approval, or provider cancellation. +""" + +from typing import Any, Literal, Self + +from issuedigest_domain.sources import Sha256 +from openai import AsyncOpenAI +from pydantic import BaseModel, model_validator +from pydantic_ai import ToolOutput + +from .guarded_question_evidence import _budget, _hash +from .guarded_responses_stream import ( + GUARDED_RESPONSES_PROFILE, + GuardedResponsesError, + GuardedResponsesToolResult, + run_guarded_responses_tool, +) +from .numeric_evidence import KnownUsage, Source, _StrictModel +from .numeric_selection_evaluation import _read +from .question_evidence_generation_v2 import ( + QUESTION_EVIDENCE_V2_INSTRUCTIONS, +) +from .question_evidence_generation_v2 import ( + _preflight as _base_preflight, +) +from .question_evidence_propositions import ( + QuestionEvidenceInput, + QuestionEvidencePropositions, + _guard_actual, + _original, + _source, + build_question_evidence_propositions, + validate_question_evidence_propositions, +) +from .role_bounded_runtime import _request_envelope + +REFERENCE_RULES_SUFFIX = ( + " Field rules: fill subject_refs and predicate_refs with exact spans when supported." + " If subject_refs or predicate_refs is empty, epistemic_status must be uncertain." + " Empty quantity metric_refs or relative-time anchor_refs also requires uncertain." + " Broad supporting_refs cannot replace these fields; never invent missing references." +) +GUARDED_QUESTION_V2_INSTRUCTIONS = QUESTION_EVIDENCE_V2_INSTRUCTIONS + REFERENCE_RULES_SUFFIX + + +class GuardedQuestionEvidenceV2Error(GuardedResponsesError): + """Sanitized failure preserving native observations without partial evidence.""" + + +class GuardedQuestionEvidenceGenerationV2(_StrictModel): + profile: Literal["guarded-question-evidence-v2"] + transport_profile: Literal["guarded-responses-tool-v1"] + input_format: Literal["question-evidence-source-v2"] + status: Literal["unreviewed"] + evidence: QuestionEvidencePropositions + prompt_sha256: Sha256 + instructions_sha256: Sha256 + usage: KnownUsage + + @model_validator(mode="after") + def bounded_usage(self) -> Self: + _budget(self.usage) + return self + + +def _preflight(source: Source, question: str) -> tuple[str, list[ToolOutput[BaseModel]]]: + prompt, tools = _base_preflight(source, question) + if _request_envelope(prompt, tools, GUARDED_QUESTION_V2_INSTRUCTIONS, 4000) > 20000: + raise ValueError("Invalid guarded question v2 request envelope") + return prompt, tools + + +def validate_guarded_question_evidence_v2( + source: Source, + question: str, + result: GuardedQuestionEvidenceGenerationV2 | dict[str, Any], +) -> GuardedQuestionEvidenceGenerationV2: + """Rebuild fresh evidence and receipt from original actual types and current source.""" + try: + _guard_actual(source, Source) + _original(result, GuardedQuestionEvidenceGenerationV2) + prompt, _ = _preflight(source, question) + checked = _read(GuardedQuestionEvidenceGenerationV2, result) + evidence = validate_question_evidence_propositions(source, question, checked.evidence) + _budget(checked.usage) + if ( + evidence != checked.evidence + or checked.prompt_sha256 != _hash(prompt) + or checked.instructions_sha256 != _hash(GUARDED_QUESTION_V2_INSTRUCTIONS) + ): + raise ValueError("Changed question evidence receipt") + return checked + except Exception: + raise GuardedQuestionEvidenceV2Error(reason="input") from None + + +async def run_guarded_question_evidence_v2( + *, client: AsyncOpenAI, model: str, source: Source, question: str +) -> GuardedQuestionEvidenceGenerationV2: + """Make one native request; caller owns the client and all application acceptance.""" + try: + _guard_actual(source, Source) + prompt, tools = _preflight(source, question) + source_hash, _ = _source(source) + except Exception: + raise GuardedQuestionEvidenceV2Error(reason="input") from None + terminal = False + known = None + try: + result = await run_guarded_responses_tool( + client=client, + model=model, + prompt=prompt, + instructions=GUARDED_QUESTION_V2_INSTRUCTIONS, + tool=tools[0], + ) + if ( + type(result) is not GuardedResponsesToolResult + or type(result.profile) is not str + or result.profile != GUARDED_RESPONSES_PROFILE + or result.usage_complete is not True + ): + raise ValueError("Invalid native completion") + _guard_actual(result.usage, KnownUsage) + KnownUsage.model_validate(result.usage.__dict__, strict=True) + known, terminal = result.usage, True + _budget(result.usage) + if type(result.output) is not QuestionEvidenceInput or _source(source)[0] != source_hash: + raise ValueError("Changed source or output") + evidence = build_question_evidence_propositions(source, question, result.output) + if evidence.source_sha256 != source_hash or _source(source)[0] != source_hash: + raise ValueError("Changed source during evidence construction") + receipt = GuardedQuestionEvidenceGenerationV2( + profile="guarded-question-evidence-v2", + transport_profile="guarded-responses-tool-v1", + input_format="question-evidence-source-v2", + status="unreviewed", + evidence=evidence, + prompt_sha256=_hash(prompt), + instructions_sha256=_hash(GUARDED_QUESTION_V2_INSTRUCTIONS), + usage=known, + ) + checked = validate_guarded_question_evidence_v2(source, question, receipt) + if _source(source)[0] != source_hash: + raise ValueError("Changed source during receipt validation") + return checked + except GuardedResponsesError as error: + if not terminal: + raise GuardedQuestionEvidenceV2Error( + reason=error.reason, + known_usage=error.known_usage, + may_have_started=error.may_have_started, + usage_complete=error.usage_complete, + stream_closed=error.stream_closed, + ) from None + raise GuardedQuestionEvidenceV2Error( + reason="output", + known_usage=known, + may_have_started=True, + usage_complete=True, + stream_closed=True, + ) from None + except Exception: + raise GuardedQuestionEvidenceV2Error( + reason="output", + known_usage=known, + may_have_started=True, + usage_complete=terminal, + stream_closed=terminal, + ) from None diff --git a/packages/agents/tests/test_guarded_question_evidence_v2.py b/packages/agents/tests/test_guarded_question_evidence_v2.py new file mode 100644 index 0000000..4149d57 --- /dev/null +++ b/packages/agents/tests/test_guarded_question_evidence_v2.py @@ -0,0 +1,449 @@ +"""Synthetic source-specific native SDK integration; no app or provider activation.""" + +import asyncio +import importlib +import importlib.util +import json +from copy import deepcopy +from hashlib import sha256 + +import httpx2 as httpx +import pytest +from openai import AsyncOpenAI +from pydantic_ai.profiles.openai import OpenAIJsonSchemaTransformer +from pydantic_ai.tools import GenerateToolJsonSchema +from test_guarded_responses_stream import Body, events +from test_guarded_responses_stream import ( + restore_introduced_agent_modules as restore_introduced_agent_modules, +) +from test_numeric_evidence import document +from test_question_evidence_propositions import QUESTION, TEXT, packet + +EMPTY = dict( + propositions=[], coverage=dict(linked_proposition_ids=[], unresolved_reason="insufficient") +) +SUFFIX = ( + " Field rules: fill subject_refs and predicate_refs with exact spans when supported." + " If subject_refs or predicate_refs is empty, epistemic_status must be uncertain." + " Empty quantity metric_refs or relative-time anchor_refs also requires uncertain." + " Broad supporting_refs cannot replace these fields; never invent missing references." +) +USAGE = dict(input_tokens=10, output_tokens=5, requests=1) + + +def api(): + name = "issuedigest_agents.guarded_question_evidence_v2" + assert importlib.util.find_spec(name) is not None, "Missing guarded question SDK" + return importlib.import_module(name) + + +def rows(data): + result = events(json.dumps(data)) + for row in result: + for item in [row.get("item"), *row.get("response", {}).get("output", [])]: + if item is not None: + item["name"] = "fill_question_evidence_v2" + return result + + +async def exercise(src, data=None, *, stream=None, mutate=None, cancel=False, options=None): + mod = api() + body = stream or Body(rows(packet(src) if data is None else data)) + requests = [] + + def serve(request): + requests.append(request) + if mutate: + mutate(src) + return httpx.Response(200, headers={"content-type": "text/event-stream"}, stream=body) + + http = httpx.AsyncClient(transport=httpx.MockTransport(serve)) + client = AsyncOpenAI(api_key="synthetic", http_client=http, max_retries=0) + kwargs = dict(client=client, model=" model ", source=src, question=QUESTION) + kwargs.update(options or {}) + try: + try: + if cancel: + task = asyncio.create_task(mod.run_guarded_question_evidence_v2(**kwargs)) + await body.blocked.wait() + task.cancel() + got = await task + else: + got = await mod.run_guarded_question_evidence_v2(**kwargs) + except (Exception, asyncio.CancelledError) as error: + got = error + assert not client.is_closed() and not http.is_closed + return got, body, requests + finally: + await client.close() + + +def run(src, data=None, **kwargs): + return asyncio.run(exercise(src, data, **kwargs)) + + +def failure(got, reason, *, known=None, started=True, complete=False, closed=True): + assert isinstance(got, api().GuardedQuestionEvidenceV2Error) + assert str(got) == "Guarded Responses request failed" + assert got.reason == reason + assert got.known_usage.model_dump() == known if known else got.known_usage is None + assert (got.may_have_started, got.usage_complete, got.stream_closed) == ( + started, + complete, + closed, + ) + assert got.__cause__ is None and got.__suppress_context__ + + +def test_complete_receipt_exact_wire_and_fresh_json_roundtrip(): + mod, src = api(), document(TEXT, " e\u0301\nwhole extra quote ") + got, body, requests = run(src) + assert type(got) is mod.GuardedQuestionEvidenceGenerationV2 + assert got.profile == "guarded-question-evidence-v2" + assert got.transport_profile == "guarded-responses-tool-v1" + assert got.status == got.evidence.status == "unreviewed" + assert got.evidence.model_dump(mode="json")["propositions"] == packet(src)["propositions"] + assert got.usage.model_dump() == USAGE + assert body.closed and len(requests) == 1 + wire = json.loads(requests[0].content) + v2 = importlib.import_module("issuedigest_agents.question_evidence_generation_v2") + prompt, tools = v2._preflight(src, QUESTION) + schema = tools[0].output.model_json_schema(schema_generator=GenerateToolJsonSchema) + assert wire == dict( + model=" model ", + input=prompt, + instructions=v2.QUESTION_EVIDENCE_V2_INSTRUCTIONS + SUFFIX, + stream=True, + store=False, + background=False, + parallel_tool_calls=False, + reasoning={"effort": "low"}, + service_tier="default", + max_output_tokens=4000, + tool_choice={"type": "function", "name": "fill_question_evidence_v2"}, + tools=[ + dict( + type="function", + name="fill_question_evidence_v2", + strict=True, + parameters=OpenAIJsonSchemaTransformer(schema, strict=True).walk(), + ) + ], + ) + assert json.loads(prompt)["question"] == QUESTION + assert [q["text"] for q in json.loads(prompt)["quotes"]] == [ + TEXT, + " e\u0301\nwhole extra quote ", + ] + assert len(SUFFIX.encode()) == 328 + assert mod.REFERENCE_RULES_SUFFIX == SUFFIX + assert ( + got.instructions_sha256 + == sha256((v2.QUESTION_EVIDENCE_V2_INSTRUCTIONS + SUFFIX).encode()).hexdigest() + ) + assert got.prompt_sha256 == sha256(prompt.encode()).hexdigest() + for saved in (got, json.loads(got.model_dump_json())): + checked = mod.validate_guarded_question_evidence_v2(src, QUESTION, saved) + assert checked == got and checked is not got and checked.evidence is not got.evidence + + +def test_empty_insufficient_is_structural_only(): + got, _, _ = run(document(TEXT), EMPTY) + assert got.evidence.propositions == () + assert got.evidence.coverage.unresolved_reason == "insufficient" + assert got.status == "unreviewed" + + +@pytest.mark.parametrize("kind", ["incomplete", "json", "whitespace", "transport", "cancel"]) +def test_native_failures_retain_observations(kind): + src = document(TEXT) + stream = rows(EMPTY) + known, complete, reason = None, False, kind + if kind == "incomplete": + stream[-1]["type"] = "response.incomplete" + stream[-1]["response"]["status"] = "incomplete" + known, complete, reason = USAGE, True, "protocol" + elif kind == "json": + stream = rows(EMPTY) + for row in stream: + if "delta" in row: + row["delta"] = "{" + if "arguments" in row: + row["arguments"] = "{" + for item in [row.get("item"), *row.get("response", {}).get("output", [])]: + if item and item["status"] == "completed": + item["arguments"] = "{" + known, complete, reason = USAGE, True, "output" + elif kind == "whitespace": + stream[2]["delta"] = " " * 300 + elif kind in ("transport", "cancel"): + stream = stream[:2] + body = Body(stream, read_error=kind == "transport", wait=kind == "cancel") + got, body, requests = run(src, stream=body, cancel=kind == "cancel") + assert body.closed and len(requests) == 1 + if kind == "cancel": + native = importlib.import_module("issuedigest_agents.guarded_responses_stream") + assert type(got) is native.GuardedResponsesCancelled + assert got.reason == "cancelled" and got.stream_closed + else: + failure(got, reason, known=known, complete=complete) + + +@pytest.mark.parametrize("field", ["quote", "title", "bill_number"]) +def test_source_mutation_during_await_rejects_with_terminal_usage(field): + src = document(TEXT) + + def mutate(source): + target = source[1].citations[0].selector if field == "quote" else source[2] + object.__setattr__(target, field, "changed source") + + got, _, requests = run(src, mutate=mutate) + failure(got, "output", known=USAGE, complete=True) + assert len(requests) == 1 + + +@pytest.mark.parametrize("kind", ["source", "question", "budget", "client", "model"]) +def test_invalid_preflight_sends_zero_requests(kind): + src, options = document(TEXT), {} + if kind == "source": + src = list(src) + elif kind == "question": + options["question"] = True + elif kind == "budget": + src = document("x" * 22000) + elif kind == "client": + options["client"] = object() + else: + options["model"] = " " + got, _, requests = run(src, EMPTY, options=options) + failure(got, "input", started=False, closed=False) + assert requests == [] + + +@pytest.mark.parametrize( + "kind", + [ + "profile", + "transport", + "hash", + "missing", + "extra", + "usage", + "constructed", + "question", + "source", + ], +) +def test_saved_tampering_and_original_types_reject(kind): + mod, src = api(), document(TEXT) + got, _, _ = run(src) + raw = deepcopy(got.model_dump(mode="json")) + question = QUESTION + if kind == "profile": + raw["profile"] = "question-evidence-generation-v2" + elif kind == "transport": + raw["transport_profile"] = "other" + elif kind == "hash": + raw["prompt_sha256"] = "0" * 64 + elif kind == "missing": + del raw["evidence"]["coverage"]["unresolved_reason"] + elif kind == "extra": + raw["extra"] = "secret" + elif kind == "usage": + raw["usage"]["requests"] = True + elif kind == "constructed": + raw = got.model_copy(update={"usage": got.usage.model_dump()}) + elif kind == "question": + question += " " + else: + object.__setattr__(src[2], "title", "changed metadata") + with pytest.raises(mod.GuardedQuestionEvidenceV2Error): + mod.validate_guarded_question_evidence_v2(src, question, raw) + + +@pytest.mark.parametrize( + "kind", ["output", "profile", "complete", "usage_type", "usage_budget", "unknown"] +) +def test_unchecked_native_boundary_never_manufactures_receipt(monkeypatch, kind): + mod, src = api(), document(TEXT) + native = importlib.import_module("issuedigest_agents.guarded_responses_stream") + contracts = importlib.import_module("issuedigest_agents.numeric_evidence") + usage = contracts.KnownUsage(**USAGE) + output = mod.QuestionEvidenceInput.model_validate_json(json.dumps(packet(src)), strict=True) + if kind == "output": + output = output.model_dump() + if kind == "usage_type": + usage = usage.model_copy(update={"requests": True}) + if kind == "usage_budget": + usage = usage.model_copy(update={"output_tokens": 4001}) + result = native.GuardedResponsesToolResult( + "wrong" if kind == "profile" else native.GUARDED_RESPONSES_PROFILE, + output, + usage, + 1 if kind == "complete" else True, + ) + + async def forged(**kwargs): + if kind == "unknown": + raise RuntimeError("sensitive raw detail") + return result + + monkeypatch.setattr(mod, "run_guarded_responses_tool", forged) + got, _, requests = run(src) + known = usage.model_dump() if kind in ("output", "usage_budget") else None + failure( + got, + "output", + known=known, + complete=kind in ("output", "usage_budget"), + closed=kind in ("output", "usage_budget"), + ) + assert requests == [] + + +def test_cancellation_is_reraised_by_identity_and_no_app_imports(monkeypatch): + import sys + + mod, src = api(), document(TEXT) + native = importlib.import_module("issuedigest_agents.guarded_responses_stream") + original = native.GuardedResponsesCancelled(reason="cancelled", may_have_started=True) + before = set(sys.modules) + + async def cancelled(**kwargs): + raise original + + monkeypatch.setattr(mod, "run_guarded_responses_tool", cancelled) + got, _, _ = run(src) + assert got is original + introduced = set(sys.modules) - before + assert not any( + name.startswith(("issuedigest_api", "issuedigest_worker", "dbos")) for name in introduced + ) + + +@pytest.mark.parametrize("actual", [False, True]) +def test_every_required_saved_field_is_required_even_null_empty(actual): + from pydantic import BaseModel + + mod, src = api(), document(TEXT) + got, _, _ = run(src) + + def paths(value, prefix=()): + if isinstance(value, dict): + for key, child in value.items(): + yield (*prefix, key) + yield from paths(child, (*prefix, key)) + elif isinstance(value, list): + for index, child in enumerate(value): + yield from paths(child, (*prefix, index)) + + for path in paths(got.model_dump(mode="json")): + changed = deepcopy(got) if actual else got.model_dump(mode="json") + parent = changed + for key in path[:-1]: + parent = getattr(parent, key) if isinstance(parent, BaseModel) else parent[key] + del (parent.__dict__ if isinstance(parent, BaseModel) else parent)[path[-1]] + with pytest.raises(mod.GuardedQuestionEvidenceV2Error): + mod.validate_guarded_question_evidence_v2(src, QUESTION, changed) + + +@pytest.mark.parametrize("field", ["subject", "predicate", "metric", "anchor"]) +@pytest.mark.parametrize("status", ["source_statement", "uncertain"]) +def test_empty_field_rules_remain_strict_without_repair(field, status): + src = document(TEXT) + data = packet(src) + prop = data["propositions"][0] + prop["epistemic_status"] = status + if field in ("subject", "predicate"): + prop[field + "_refs"] = [] + elif field == "metric": + prop["facets"][0]["value"]["metric_refs"] = [] + else: + prop["facets"][4]["value"]["anchor_refs"] = [] + got, _, requests = run(src, data) + assert len(requests) == 1 + if status == "source_statement": + failure(got, "output", known=USAGE, complete=True) + else: + assert got.evidence.model_dump(mode="json")["propositions"] == data["propositions"] + assert got.status == "unreviewed" + + +@pytest.mark.parametrize("field", ["profile", "instructions_sha256"]) +def test_historical_receipt_substitutions_reject_both_directions(field): + mod, src = api(), document(TEXT) + old = importlib.import_module("issuedigest_agents.guarded_question_evidence") + got, _, _ = run(src) + raw = got.model_dump(mode="json") + historical = dict( + raw, + profile="guarded-question-evidence-v1", + instructions_sha256=sha256(old.QUESTION_EVIDENCE_V2_INSTRUCTIONS.encode()).hexdigest(), + ) + changed = dict(raw) + changed[field] = historical[field] + with pytest.raises(mod.GuardedQuestionEvidenceV2Error): + mod.validate_guarded_question_evidence_v2(src, QUESTION, changed) + historical[field] = raw[field] + with pytest.raises(old.GuardedQuestionEvidenceError): + old.validate_guarded_question_evidence(src, QUESTION, historical) + + +def test_new_envelope_exact_boundary_and_old_only_fit_sends_nothing(): + mod = api() + v2 = importlib.import_module("issuedigest_agents.question_evidence_generation_v2") + envelope = importlib.import_module("issuedigest_agents.role_bounded_runtime")._request_envelope + src = document("x") + prompt, tools = v2._preflight(src, QUESTION) + remaining = 20000 - envelope(prompt, tools, v2.QUESTION_EVIDENCE_V2_INSTRUCTIONS + SUFFIX, 4000) + src = document("x" * (1 + remaining)) + prompt, tools = v2._preflight(src, QUESTION) + assert envelope(prompt, tools, v2.QUESTION_EVIDENCE_V2_INSTRUCTIONS + SUFFIX, 4000) == 20000 + got, _, requests = run(src, EMPTY) + assert type(got) is mod.GuardedQuestionEvidenceGenerationV2 and len(requests) == 1 + over = document("x" * (2 + remaining)) + prompt, tools = v2._preflight(over, QUESTION) + assert envelope(prompt, tools, v2.QUESTION_EVIDENCE_V2_INSTRUCTIONS, 4000) <= 20000 + got, _, requests = run(over, EMPTY) + failure(got, "input", started=False, closed=False) + assert requests == [] + + +@pytest.mark.parametrize("stage", ["construction", "validation"]) +def test_source_mutation_after_completion_retains_original_usage(monkeypatch, stage): + mod, src = api(), document(TEXT) + name = ( + "build_question_evidence_propositions" + if stage == "construction" + else "validate_guarded_question_evidence_v2" + ) + original = getattr(mod, name) + + def changed(*args): + result = original(*args) + object.__setattr__(src[2], "title", "changed after original check") + return result + + monkeypatch.setattr(mod, name, changed) + got, _, requests = run(src) + failure(got, "output", known=USAGE, complete=True) + assert len(requests) == 1 + + +def test_native_observations_are_forwarded_without_inventing_completion(monkeypatch): + mod, src = api(), document(TEXT) + usage = mod.KnownUsage(**USAGE) + + async def observed(**kwargs): + raise mod.GuardedResponsesError( + reason="transport", + known_usage=usage, + may_have_started=True, + usage_complete=False, + stream_closed=False, + ) + + monkeypatch.setattr(mod, "run_guarded_responses_tool", observed) + got, _, requests = run(src) + failure(got, "transport", known=USAGE, complete=False, closed=False) + assert requests == []