jsonvote

jsonvote turns several noisy LLM answers into one decision, or a clear "I don't know". You give it any function that sends a prompt to a model and returns the reply. It calls that function n times, pulls the JSON out of each reply, checks it against your rules, and counts the votes. If enough replies agree, you get the winning answer with its agreement ratio, per-field agreement and confidence. If they do not, it abstains and tells you why. It is for developers who turn model output into actions: ticket triage, intent routing, field extraction, moderation labels, LLM-as-judge. It works with any provider, local models included, and it suits servers where the n parameter is missing or unreliable. The core is pure standard-library Python.

What it does NOT do

• It does not call any model. You bring the sampler. jsonvote has no HTTP code, no API keys and no model names.
• It does not fix mistakes the model makes every time. If the model is consistently wrong, the wrong answer wins the vote.
• It does not save money. n samples cost n calls. Early stop and concurrency reduce waste and waiting, not the basic cost.
• Agreement is not a calibrated probability. An agreement of 0.8 does not mean the answer is 80% likely to be right. confidence is not calibrated either: it is the model's own self-report, averaged.
• It does not understand meaning. Two answers count together only when their JSON is identical (key order aside). "Billing" and "billing", or 1 and 1.0, are different votes unless your validator or key normalises them.
• It does not retry. A failed sample counts against the quorum. If you want retries, put them inside your sampler.
• It does not time out or pace calls. A sampler that hangs makes the vote hang. Set a timeout in your model client. concurrency caps how many calls run at once; it does not rate-limit them.
• It does not force valid JSON. It finds and checks JSON after the fact. For guaranteed formats, use constrained decoding and let jsonvote do the voting.

Install

pip install git+https://github.com/prashant7799/jsonvote

Optional extras:

pip install "jsonvote[pydantic] @ git+https://github.com/prashant7799/jsonvote"    # Pydantic models as validators
pip install "jsonvote[jsonschema] @ git+https://github.com/prashant7799/jsonvote"  # JSON Schema validators
pip install "jsonvote[repair] @ git+https://github.com/prashant7799/jsonvote"      # repair broken JSON with json_repair

Python 3.10 or newer. A PyPI release will come later.

Quickstart (60 seconds)

This runs as is. The sampler is a fake that replays five replies, including a fenced block, chatty text and a reply with no JSON. The same code is in examples/quickstart.py.

import itertools

from jsonvote import vote

# Fake sampler for the demo. In real use, it calls your model.
replies = itertools.cycle([
    '```json\n{"team": "billing", "confidence": 0.9}\n```',
    'Sure! {"team": "billing", "confidence": 0.8}',
    '{"team": "shipping", "confidence": 0.6}',
    "Sorry, I cannot help with that.",  # no JSON: a failed sample
    '{"team": "billing", "confidence": 0.7}',
])


def sampler(prompt: str) -> str:
    return next(replies)


ticket = "I was charged twice for order #1042."
prompt = (
    'Which team should handle this ticket? Reply in JSON with "team" and '
    f'"confidence". Ticket: {ticket}'
)

result = vote(
    sampler, prompt, n=5, key="team",
    min_valid=0.6, min_agreement=0.5,
    count_failures=True, abstain_on_tie=True,
    confidence_field="confidence",
)

if result.abstained:
    print("abstain:", result.reason, "-", result.detail)
else:
    print(result.value, result.agreement, round(result.confidence, 2))
    print(result.detail)
    print(result.counts)

Output:

billing 0.6 0.8
'billing' won with 3 of 5 votes
(('billing', 3), ('shipping', 1))

Why: 4 of 5 replies had usable JSON, so the valid ratio is 0.8 (above min_valid 0.6). Three said billing. The failed reply still counts in the total, so agreement is 3 / 5 = 0.6 (above min_agreement 0.5). Confidence is the mean of the three billing votes: 0.8.

To use a real model, replace sampler with a function that calls it. See Recipes.

Concepts

                 prompt (or a list of paraphrases, taking turns)
                                  |
          +-----------+-----------+-----------+-----------+
          v           v           v           v           v
      sampler()   sampler()   sampler()   sampler()   sampler()     n calls
          |           |           |           |           |
          v           v           v           v           v
      find JSON candidates: whole reply, fenced blocks, objects, arrays
          |           |           |           |           |
      first candidate that passes the validator AND has the key
          |           |           |           |           |
       "billing"  "billing"  "shipping"    FAILED     "billing"       votes
          \___________\___________|___________/___________/
                                  v
             valid ratio = 4/5     agreement = 3/5 (failures count)
                                  v
             quorum rules: min_valid, min_agreement, ties
                                  v
              VoteResult: value="billing", or abstained + reason

Sampler. Any function sampler(prompt) -> str. It usually calls a model with a temperature above 0. It may also return an already-parsed object, such as a dict or a Pydantic model. Then jsonvote skips extraction. If the sampler raises an Exception, that sample fails and the error is recorded; it does not escape. KeyboardInterrupt still stops the run.

Extraction. For each reply, jsonvote lists JSON candidates in this order:

1. the whole reply, if it is clean JSON (then it is the only candidate);
2. every fenced code block that holds valid JSON;
3. every JSON object found by scanning the rest of the text;
4. every JSON array found by that scan.

It keeps the first candidate that passes your validator and has your key. So a reply that repeats an example object before the real answer still works, as long as the example fails your validator or lacks your key. The parts of a JSON value are never separate candidates: a list of objects is one candidate, not several, and a cut-off object gives no candidate at all, even if a complete object sits inside it. If your model wraps its answer in a list, use a key function such as lambda r: r[0]["team"].

Key. What to vote on.

• key="team": the value of one top-level field.
• key=lambda r: r["team"].strip().lower(): any function of the record. Use it to normalise or to vote on a pair of fields.
• key=None (the default): the whole object. Key order does not matter.

A missing key makes the sample fail. A null value is a real vote for None. The string "none" is a real vote too.

Ballots. Votes are compared by their canonical JSON text. So {"a": 1, "b": 2} equals {"b": 2, "a": 1}, but "Billing" and "billing" differ, and so do 1 and true, and 1 and 1.0. Normalise in the validator or the key when you need to.

Validator. A function that checks one candidate. Raise an exception or return False to reject it. Return None or True to accept it unchanged. Return any other value to accept it and use that value instead (for example a cleaned dict or a Pydantic model). Ready-made helpers: field_in, all_of, from_pydantic, from_jsonschema.

Quorum. Both thresholds are ratios between 0 and 1, so they mean the same thing for any n.

• valid_ratio = valid samples / samples drawn. It must be at least min_valid.
• agreement = votes for the leader / denominator. It must be at least min_agreement.
• With count_failures=True (the default), the denominator is every sample drawn. A failure acts like a vote against every answer. One valid answer out of three cannot win.
• With count_failures=False, the denominator is only the valid samples. min_valid still applies.

Ties. With abstain_on_tie=True (the default), a tie for first place abstains. With False, the answer that appeared first wins.

Abstain reasons. result.reason is one of:

Reason	Meaning
no_valid	No sample gave a usable answer.
too_few_valid	The valid ratio is below min_valid.
tie	Two or more answers share the top count.
low_agreement	The leader's agreement is below min_agreement.
quorum_unreachable	Early stop: no answer can reach the quorum, even if every remaining sample agrees.

result.detail gives the same information as one sentence.

Confidence. If you name a confidence_field, jsonvote reads it from each sample. It accepts numbers, numeric strings and percent strings ("80%" becomes 0.8). It clamps values to 0..1, so a bare 85 becomes 1.0: ask the model for 0..1. Booleans, words like "high", NaN and missing values are left out of the average, not counted as 0. result.confidence is the mean over the winning votes only.

Per-field agreement. result.field_agreement maps each top-level field to the share of the denominator that agrees on that field's most common value. It shows which fields are stable and which are noisy, even when you vote on one key. The confidence field is left out.

Paraphrases. Pass a list of prompts and they take turns: sample i uses prompt i % len(prompts). n is the total number of calls. All votes go into one pool. There is no vote of votes, which can hide a real majority.

Early stop. With early_stop=True, jsonvote stops calling the sampler once the remaining samples cannot change the outcome. It stops with a win when the leader passes every rule even in its worst case. It stops with quorum_unreachable when no answer can pass even in its best case. The decision is always the one a full run would reach. A property test checks this on random inputs. agreement and valid_ratio are then measured over the samples actually drawn.

Concurrency. concurrency=k runs up to k calls at once: threads in vote(), asyncio tasks in avote(). New calls start only as old ones finish, so early stop can still save calls. After an early stop, avote() cancels running calls. vote() cannot cancel a running thread, so it ignores those replies. With concurrency > 1 in vote(), your sampler must be safe to call from several threads. In avote(), only an async def sampler runs calls at the same time; a plain function blocks the event loop, so its calls run one by one.

API reference

vote(sampler, prompt, *, ...) -> VoteResult

vote(
    sampler,                 # callable(prompt) -> str, or an already-parsed object
    prompt,                  # str, or a list of paraphrases that take turns
    *,
    n=5,                     # total number of sampler calls
    key=None,                # field name, function of the record, or None for the whole object
    validator=None,          # see "Validator" above
    min_valid=0.5,           # minimum valid samples / samples drawn
    min_agreement=0.5,       # minimum votes for the leader / denominator
    count_failures=True,     # failures stay in the agreement denominator
    abstain_on_tie=True,     # a tie for first place abstains
    confidence_field=None,   # field holding the model's own confidence
    concurrency=1,           # calls at once (threads)
    early_stop=False,        # stop once the outcome is settled
    repair=False,            # also try to repair broken JSON (needs the [repair] extra)
)

It raises ValueError or TypeError for bad arguments (for example n=0 or min_agreement=1.5). It raises ImportError with an install hint if repair=True and json_repair is missing. It never raises because of a bad reply: junk, very deep nesting and numbers with thousands of digits all become failed samples.

avote(sampler, prompt, *, ...) -> VoteResult

The async twin, with the same arguments. Await it. The sampler may be an async def function or a plain one (a plain one runs its calls one by one).

VoteResult (frozen dataclass)

Attribute	Meaning
abstained	True if the vote refused to decide. Check this first.
decided	not abstained
value	The winning value. None when abstained, and also when the model voted null.
reason	A Reason code when abstained, else None.
detail	One sentence that explains the outcome.
agreement	Leader's votes / denominator.
valid_ratio	Valid samples / samples drawn.
confidence	Mean confidence of the winning votes, or None.
counts	((value, count), ...), most votes first, then first seen.
record	The full JSON object of the first winning sample. None when abstained.
field_agreement	Read-only mapping: field name -> share that agrees on its most common value.
samples	Every Sample drawn, in order. Hidden from repr().
n_planned	The n you asked for.
stopped_early	True if early stop ended the run.
num_samples, num_valid, num_failed	Counts of samples drawn, valid and failed.

Sample (frozen dataclass)

index, prompt_index, ok, vote, record, confidence, error, raw. The error string starts with a code: sampler_error, empty_reply, no_json, invalid or missing_key. raw is the reply exactly as the sampler returned it, for auditing.

Reason

Constants: Reason.NO_VALID, Reason.TOO_FEW_VALID, Reason.TIE, Reason.LOW_AGREEMENT, Reason.QUORUM_UNREACHABLE. They are plain strings.

Extraction

• extract_json(text, validator=None, *, repair=False): the first candidate that passes validator, or None.
• iter_json_candidates(text, *, repair=False): a lazy generator of every candidate, in order, without duplicates. Plain scalars are never candidates, and neither are the parts of a value already found.

Validators

• field_in(name, choices, *, ignore_case=False): accept only if obj[name] is one of choices. With ignore_case=True, text is compared without case or outer spaces, and the record gets the spelling from choices.
• all_of(*validators): run validators in order. Each one sees the previous one's output.
• from_pydantic(Model): validate with a Pydantic v2 model. The record becomes the model instance, and key="field" reads its fields.
• from_jsonschema(schema): validate with a JSON Schema. The schema is checked once, up front.

Helpers

• coerce_confidence(value) -> float | None: the confidence rules above, as a function.
• ballot(value) -> str: the canonical text used to compare votes.

Logging

jsonvote never prints. It logs to the jsonvote logger: one DEBUG line per sample, one INFO line per result. To see them:

import logging
logging.basicConfig(level=logging.DEBUG)

Examples

All examples run offline with fake models and print the same output every time. A test checks that the outputs below still match.

Ticket triage with a label check

examples/ticket_triage.py routes four tickets. It uses field_in(..., ignore_case=True), so "Billing" counts as "billing" and the made-up label "sales" is rejected.

I was charged twice for order #1042.
  -> billing  (agreement 0.80, confidence 0.86)
My parcel says delivered but it is not here.
  -> shipping  (agreement 1.00, confidence 0.89)
The app crashes when I open my invoice.
  -> ask a human (tie): tie: 2 answers have 2 votes each
hello??
  -> ask a human (too_few_valid): only 1 of 5 samples were valid (0.20 is below min_valid 0.6)

Invoice extraction and per-field agreement

examples/invoice_fields.py extracts four fields five times. A validator normalises 1180, 1180.0 and "1,180.00" to one value. The due date is noisy. A whole-object vote abstains. A vote on the invoice number succeeds, and field_agreement points at the field to check by hand.

1) Vote on the whole object
   abstained=True reason=low_agreement agreement=0.40
2) Vote on the invoice number, then read per-field agreement
   invoice_number=INV-2041 agreement=1.00
   invoice_number  1.00  ok
   total           1.00  ok
   currency        1.00  ok
   due_date        0.40  check by hand
   record: {'invoice_number': 'INV-2041', 'total': 1180.0, 'currency': 'EUR', 'due_date': '2026-11-30'}

Intent routing with paraphrases and early stop

examples/intent_paraphrases.py uses three wordings of the same prompt, n=9 and min_agreement=0.6. The easy message settles after 6 calls, because 6 of 9 already reaches 0.6. The unclear one stops when even the best case cannot reach the quorum.

Please stop my subscription at the end of the month.
  -> cancel  (used 6 of 9 calls, counts {'cancel': 6})
I paid for the yearly plan but want my money back.
  -> refund  (used 8 of 9 calls, counts {'refund': 6, 'cancel': 2})
Is there something better than what I have now?
  -> quorum_unreachable  (used 8 of 9 calls, counts {'upgrade': 4, 'other': 4})

Async with a concurrency cap

examples/async_concurrency.py runs avote() with an async fake model and concurrency=3.

decision: go, agreement 1.00
calls: 6, at most 3 at the same time

Simulation: coverage against accuracy

examples/coverage_simulation.py is a toy, not a benchmark of any real model. A fake model answers 1,000 four-way questions. On 80% of them each call is right with probability 0.6. On 20% ("traps") it prefers one wrong answer. 10% of replies have no JSON. Calls are independent, which flatters voting.

 n   calls   answered   accuracy when answered   wrong answers on traps
 1    1000     90.0%              54.9%                  122
 3    3000     68.0%              67.9%                  109
 5    5000     61.5%              75.3%                  113
 7    7000     61.2%              79.4%                  113

Read it like this. More samples make the answers it gives more accurate, and it answers fewer questions. The trap errors do not go away: a consistent mistake wins the vote.

Recipe for a real model

This needs a running model server, so CI does not run it. It was tested on 23 Oct 2026 against a small open-weight model served locally by llama.cpp's llama-server.

Any server that speaks the common chat-completions HTTP format (for example llama-server from llama.cpp, running on your own machine). Read the address and model name from the environment. If your server needs a key, read it from the environment too. Never hard-code keys.

import json
import os
import urllib.request

from jsonvote import vote

BASE_URL = os.environ.get("LLM_BASE_URL", "http://127.0.0.1:8080/v1")


def sampler(prompt: str) -> str:
    body = {
        "model": os.environ.get("LLM_MODEL", "local"),
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.7,  # above 0, or every sample is the same
    }
    headers = {"Content-Type": "application/json"}
    if os.environ.get("LLM_API_KEY"):
        headers["Authorization"] = "Bearer " + os.environ["LLM_API_KEY"]
    request = urllib.request.Request(
        BASE_URL + "/chat/completions", data=json.dumps(body).encode("utf-8"), headers=headers
    )
    with urllib.request.urlopen(request, timeout=60) as response:
        reply = json.loads(response.read())
    return reply["choices"][0]["message"]["content"] or ""


# 3 calls at a time, so early stop can skip the last calls when the first ones agree.
result = vote(sampler, "...", n=5, key="team", concurrency=3, early_stop=True)

Validation libraries as the sampler. A library that validates and retries a single call (for example with Pydantic models) works as a sampler too: it returns an object, and jsonvote votes across calls. If it gives up and raises, that sample counts as a failure.

Tuning

• Use a temperature above 0. At 0, most models repeat themselves, and you pay n times for one answer.
• Use an odd n. For questions with two possible answers and no failures, it rules out ties.
• Start with n=5, min_agreement=0.6. Raise them when a wrong answer costs more than a human review.
• Track the abstain rate. It is a cheap uncertainty signal. A sudden rise often means the prompt or the input changed.

Cost and waiting time, where one call takes time t:

Setting	Calls	Waiting time
n=5	5	about 5 t
n=5, concurrency=5	5	about t (the slowest call)
n=5, early_stop=True, all samples agree	3	about 3 t
n=9, concurrency=3, early_stop=True, all agree	5 to 7	about 2 t

How correctness is tested

• Fake samplers only. Tests replay scripted replies and raise scripted errors. No test needs a network or a model, and a fixture makes any socket connection fail during tests.
• One test per rule. One valid answer out of three abstains. "none" and null are real votes. Ties abstain. Thresholds work as ratios at every n. Confidence is coerced, clamped, and left out when missing. Paraphrase votes are pooled (with a case where a vote of votes would pick the other answer). Validators, keys, sampler errors and KeyboardInterrupt each have tests.
• A second, plain implementation of the rules. A short reference version written with collections.Counter must give the same decision, reason and value as vote() on random replies and random thresholds.
• Early stop, by brute force. For short runs, a test tries every possible set of remaining replies and checks that an early win or an early abstain is never wrong.
• Property tests with Hypothesis. On random reply sequences and random rules:
  • early stop never changes the decision of a full run, also with threads and with asyncio when calls finish out of order;
  • every decision satisfies min_valid, min_agreement and the tie rule;
  • counting failures is never more lenient than not counting them;
  • the order of samples does not change the outcome when ties abstain;
  • vote() and avote() agree;
  • a JSON object placed in random prose (prose without brackets of its own), fenced or not, is found again exactly.
• Hostile replies. Random text, JSON-like junk, long runs of brackets, deep nesting and integers with thousands of digits must become failed samples, never a crash. Every error starts with one of the documented codes. A timing test checks that 100 KB of brackets is scanned quickly.
• Concurrency. Threaded and async runs give the same counts as in-order runs, and tests check that calls really overlap.
• Docs. A test runs every example and compares its output with this README.
• CI. GitHub Actions runs ruff check and pytest on Python 3.10, 3.11 and 3.12.

Run them yourself:

pip install -e ".[dev]"
pytest -q
ruff check .

Limitations and assumptions

• Votes help only when errors are independent. Paraphrases and temperature help a little. They cannot make a model's blind spots go away.
• Exact matching. Votes match on canonical JSON. Free text such as summaries rarely matches exactly; vote on a label or a normalised field instead.
• Clamped confidence. A bare 85 becomes 1.0, not 0.85. Ask the model for 0..1 or a percent string.
• Cut-off replies fail. If a reply is cut off (for example by max_tokens), nothing inside the broken JSON counts, not even a complete inner object. The sample fails. repair=True with a validator can recover some of these.
• Strict JSON. Single quotes, trailing commas and raw line breaks inside strings are not JSON, so such replies fail unless you use repair=True.
• Lists are one answer. A list of objects is one candidate. With key="team" it fails, because a list has no fields. Unwrap it in a key function if your model answers with lists.
• Degenerate replies. The scan is fast on normal text and on long runs of brackets. Its worst case is many small broken pieces, such as 100 KB of [x[x[x..., which takes about 2 seconds on a laptop. If part of a reply is nested thousands of levels deep, the scan stops there. Cap the reply length (max_tokens) in your client.
• NaN and Infinity. Python's json module accepts these non-standard values, so jsonvote does too.
• Repair can invent structure. repair=True may turn broken text into a plausible object. Use it together with a validator.
• Threads and early stop. In vote() with concurrency > 1, calls already running when the vote settles keep running in the background. Their replies are ignored, but you still pay for them.
• Early stop measures what it drew. After an early stop, agreement, valid_ratio and field_agreement are computed over the samples actually drawn, not over n.
• One process. There is no caching, no persistence and no distributed run.

Related tools

jsonvote applies a well-known technique (self-consistency). It claims no novelty. Its focus is the abstain logic, the quorum rules and the tests.

• A plain collections.Counter loop. This is the simplest case: a few lines and no dependency. You write extraction, failure counting, ties, thresholds and early stop yourself. If you only need most_common(1), you do not need jsonvote.
• Majority-vote helpers inside LLM frameworks and client wrappers. These pick the most common answer, and they are handy if you already use that framework. Check whether they offer abstain, quorum and failure counting; jsonvote is built around those.
• Validate-and-retry libraries. These check and retry one call. They do not vote, but they work well as a jsonvote sampler (see the recipe).
• JSON repair libraries. These are much better at repairing broken JSON than jsonvote. jsonvote can use one through the optional [repair] extra.
• Constrained decoding. This guarantees the output matches a schema. It guarantees the format, not the answer, so combine it with jsonvote.

References and citation

• Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023. arXiv:2203.11171.
• C. K. Chow. On Optimum Recognition Error and Reject Tradeoff. IEEE Transactions on Information Theory, 16(1), 1970. The classic paper on balancing errors against abstaining.

If jsonvote helps your work, you can cite it like this:

@software{lohar_jsonvote_2026,
  author  = {Lohar, Prashant},
  title   = {jsonvote: self-consistency voting for LLM JSON output, with abstain},
  year    = {2026},
  version = {0.1.0},
  url     = {https://github.com/prashant7799/jsonvote}
}

Licence

MIT. See LICENSE. Copyright (c) 2026 Prashant Lohar.