Skip to content

docs: document compatibility with WOE-transformed features - #84

Merged
orgoca merged 2 commits into
orgoca:mainfrom
bh-official:docs/woe-compatibility
Oct 27, 2026
Merged

orgoca merged 2 commits into
orgoca:mainfrom
bh-official:docs/woe-compatibility

Conversation

@bh-official

Copy link
Copy Markdown
Contributor

Closes #18

What

The docs said nothing about WOE-transformed inputs. This adds docs/howto/woe.md, linked from the nav, the docs index, the README list and the changelog. It covers the interactions the issue lists:

  • what the artifact holds (thresholds in WOE units, not the WOE table) and how to record the table version in metadata
  • bands and within-band AUC on a plateaued latent
  • a reason dictionary keyed by the WOE column but written against the underlying attribute, with the example from the issue
  • monotone constraints, whose sign is in WOE units
  • a missing bin versus NaN
  • an honest list of what is not tested

Per "Claims are tests", every statement on the page is pinned by tests/test_woe_inputs.py (11 tests, seeded synthetic data, four WOE columns of 3-4 bins each). Where I did not measure something, the page says so instead of guessing, e.g. whether WOE inputs change the certified band count compared with raw features.

How I checked

pytest                       270 passed, 7 skipped  (before)
pytest                       281 passed, 7 skipped  (after)
ruff check src tests benchmarks   All checks passed
black --check src tests benchmarks   75 files unchanged
mkdocs build --strict        builds

I also broke the code under test to confirm the new tests can fail: flipping the adverse/favorable sign in format_reasons, disabling the monotone check in build_artifact, and imputing 0.0 instead of the baseline for missing inputs each make a test in the new file fail.

mypy src/compileml reports 58 errors in my environment (missing stubs for scipy and sklearn); the count is the same with and without this change. cobc is not installed here, so the COBOL parity test skips.

Add docs/howto/woe.md covering what the artifact holds for WOE columns,
band construction on a plateaued latent, reason dictionaries written against
the underlying attribute, the sign of monotone constraints in WOE units, and
how a missing bin differs from NaN. It ends with what has not been tested.

tests/test_woe_inputs.py pins each statement on seeded synthetic WOE data:
parity with the source model for sklearn, XGBoost and LightGBM, thresholds
between bin values, one band per bin combination, within-band AUC, reasons,
baseline behaviour, constraint signs, missing-bin handling and metadata.

Closes orgoca#18
@orgoca

orgoca commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Thank you @bh_official!!

This is a great first contribution: every claim pinned by a test, the
mutation checks, and an honest "not yet tested" section are exactly the bar the
project tries to hold.

One change before merging, in "Monotone directions": "Declaring the wrong sign
makes the build raise" holds only when the declaration disagrees with how the
trees were trained, which is what the test checks (trained −1, declared +1). If
the same wrong sign is used for training and for the declaration, the build passes:
the model is constrained against the data and the cost shows up as lost
discrimination, not as an error. Could you say that explicitly? Something like:
"If the declaration disagrees with the trees, the build raises. A wrong sign used
for both training and the declaration is not caught: it constrains the model
against the data, and you see it as lost discrimination."

Two optional suggestions:

  • compile_selected takes the per-column median as the baseline; for a WOE
    column that is a bin's value except at an exact tie. A sentence would cover
    users who don't call build_artifact directly.
  • In test_a_missing_bin_is_an_ordinary_woe_value_and_nan_is_not_that_bin, the
    "missing" rows are drawn at random, so nothing forces the trees to split them
    from the baseline bin. It passes everywhere today, but a future scikit-learn
    could turn the != into a tie; drawing them from riskier rows would make the
    split certain.

I approved the CI run: all 20 checks pass, including the test matrix on three
operating systems and Python 3.10–3.13, the cross-platform determinism build, and
the COBOL parity test you couldn't run locally.

…is not caught

A review pointed out that the build only raises when the declared monotone
direction disagrees with how the trees were trained. If the same wrong sign
is used for both, the build passes and the cost shows up as lost
discrimination, so say that explicitly.

Also describe the baseline compile_selected uses (the per-column median of the
development rows, which for a WOE column is a bin's value except at an exact
tie), and draw the missing-bin test rows from the riskier rows so the trees are
certain to split them off the baseline bin.
@bh-official

Copy link
Copy Markdown
Contributor Author

Thanks, all three are in the new commit.

  • "Monotone directions" now says it explicitly: if the declaration disagrees with the trees the build raises, and a wrong sign used for both training and the declaration is not caught, it constrains the model against the data and shows up as lost discrimination.
  • The baseline bullet now says that compile_selected uses the per-column median of the development rows, which for a WOE column is a bin's value except at an exact tie. For build_artifact it still says to pass the bin you mean.
  • In the missing-bin test, the "missing" rows are now drawn from the bad rows, so the trees have to split them off the baseline bin. I checked it by making decide impute NaN with the missing-bin value (0.9) instead of the baseline: the test fails, and it passes again with the source restored. Full suite passes.

@orgoca

orgoca commented Oct 27, 2026

Copy link
Copy Markdown
Owner

All three in, and the mutation check on the new test is exactly right.
Merging; thank you @bh-official! Your page ships in the next release, credited by name in
the changelog and the release notes.
Honor to have you as part of the small but very cool CompileML community!

@orgoca
orgoca merged commit 92ad3c2 into orgoca:main Oct 27, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document compatibility with WOE-transformed features

2 participants