Skip to content

About

🔎 Agentic Programming Lab 2: where LLMs fail, the no-letter-E story experiment (CoT helps at 20 words, fakes verification at 50+)

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

🔎 Finding LLM Failure Modes: the "No Letter E" Story

Agentic Programming · Lab 2: Find Where the Model Makes Mistakes Hadar Wolf & Nehory

We asked a model to write a story that is exactly N words long and never uses the letter "e". Both rules are easy to check automatically, and they pull against each other: "e" is the most common letter in English, so avoiding it makes it harder to hit an exact word count.

TL;DR

Chain-of-thought helps the model on easy versions of the task (20 words: 0% → 80%). On harder versions (50 or 100 words), chain-of-thought still fails, and it also writes "verified ✓" next to answers that break both rules. Instead of admitting failure, it claims a check it never really did.

Setup

  • Model: claude-haiku-4-5, tools and extended thinking disabled, with a fresh claude -p process for every run
  • Conditions: 20 / 50 / 100 words × with or without chain-of-thought, 5 runs each (30 runs in total)
  • Scoring: a run passes only if the word count is exact and there is no "e". Scored automatically by check_story.py, then checked by hand

Results

Words Chain-of-thought Pass Typical failure
20 no 0/5 wrong length, plus a missed "e"
20 yes 4/5 the draft → count → check loop works
50 no 0/5 length off by 2–7 words; "e" missed (e.g. "The")
50 yes 0/5 says "verified ✓" but breaks both rules
100 no 0/5 length off by 3–8 words; "e" missed
100 yes 0/5 says "verified ✓" but breaks both rules

Key finding: the model's own reasoning says "Recount: 50 words ✓" and "No E's found ✓" when both are false. Under length pressure it keeps reaching for "the", the most common word in English.

Files

File What it is
run_experiment.sh runs every condition 5 times and writes results/ and results.csv
check_story.py scores one response: exact length and no forbidden letter (PASS or FAIL <reason>)
results.csv the verdict and the reason for every run
results/ the raw model output for every run
Lab2_Report.pdf the one-page report (.docx)

Reproduce

Requires the Claude Code CLI, logged in.

./run_experiment.sh            # about 30 short claude -p calls
cat results/wc20_cot_run1.txt | python3 check_story.py --target-words 20

Part of my Agentic Programming labs (HUJI, AICODE 67723): Lab 1 · CTF Arena · Lab 2 · LLM failure modes · Lab 3 · Coin Dash with several agents · Lab 4 · Traffic monitor spec · Lab 5 · Chess: testing new features · Lab 6 · Binary search explainer video

About

🔎 Agentic Programming Lab 2: where LLMs fail, the no-letter-E story experiment (CoT helps at 20 words, fakes verification at 50+)

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages