[chivector]

Problem

Multi-field exact deduplication can silently remove different records before any hash collision occurs. With input_keys=["question", "answer"], these two rows produce exactly the same delimiter-joined string:

{"question": "x\nanswer:\ny", "answer": "z"}
{"question": "x", "answer": "y\nanswer:\nz"}

Consequently HashDeduplicateFilter keeps only the first row with MD5, SHA-256, and xxh3 alike. Text such as field labels and newlines can legitimately occur in question/answer datasets.

Change

Encode the ordered string representations of the selected values as a JSON array. JSON escaping preserves field boundaries even when values contain delimiters, quotes, backslashes, NULs, or Unicode.

The single-field paths (input_key and a one-element input_keys) stay unchanged. Multi-field values retain their existing string-conversion semantics, and genuine duplicates still keep the first row in its original order. Hashes are local to each run; no persisted fingerprint format is changed.

Validation

Added 33 CPU cases covering all three hash algorithms, embedded field markers (including unusual column labels), exact duplicates, row indices/order, empty and escaped text, Unicode, non-string values, and both single-field entry points.

• Original production code with the new tests: 9 failed, 24 passed; all nine failures remove a distinct row.
• Fixed code, new tests plus existing single-key and short-text deduplication regressions: 46 passed.
• The same 46 cases pass on NumPy 1.26.4 / pandas 2.2.3 / SciPy 1.13.1 (within the project's NumPy constraint), and NumPy 2.3.5 / pandas 3.0.1 / SciPy 1.18.1.
• New test file passes Ruff lint/format; git diff --check passes.

python -m pytest test/cpu_only/test_hash_dedup_field_boundaries.py test/cpu_only/test_dedup_single_input_keys.py test/cpu_only/test_minhash_short_texts.py -q

Validated on Windows with Python 3.12.14 and the normal repository conftest. The full model/API test suite was not run.