Skip to content

fix(datafusion): report an unconvertible column statistic as absent - #9973

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fix/df-stats-no-panic
Open

jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fix/df-stats-no-panic

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

stats_set_to_df (vortex-datafusion/src/convert/stats.rs:30-66) ran each statistic through two
vortex_expects. Both sides of that conversion come from the file — the stored value out of the
footer's stats set, the dtype off the column — so both can legitimately fail on a file we did not
write:

  • Stat::dtype returns None where the statistic does not apply:
    vortex-array/src/expr/stats/mod.rs:173,175 return None for Max/Min of a DType::Null
    column, and Sum::return_dtype does the same for string, binary, list and struct columns. That hit
    vortex_expect("must have a valid dtype").
  • Scalar::try_new rejects a stored value that does not match the column dtype, hitting
    vortex_expect("Stat::Minsomehow had an incompatibleDType").

Either one panicked inside DataFusion planning instead of returning an error or simply forgoing the
statistic.

Fix

Forgoing it is what the end of the same chain already chose: try_to_df().ok() drops a scalar
DataFusion cannot represent. This extends that to the whole conversion, which also lets the three
near-identical blocks collapse into one helper — and is what the TODO(connor) above them
("There's a lot that can go wrong here, should probably handle this more gracefully") asked for, so
that comment goes too.

The statistic's Precision is preserved rather than flattened: an exact statistic stays
Precision::Exact, and only the unconvertible ones become Absent. That distinction is easy to lose
here, since StatsSet::get returns Vortex's Precision and not an Option.

Tests

cargo test --release -p vortex-datafusion: 341 passed, 335 before. Four cases cover the
unconvertible shapes — Min/Max of a null column, Sum of a string column, and a stored value
that disagrees with the column dtype — and two more pin that a convertible statistic keeps its exact
or inexact precision.

Restoring the vortex_expects fails all four unconvertible cases, on both panic sites
(vortex-error/src/lib.rs:666 for the Option and :659 for the Result), and leaves the
precision and distinct_count cases green.

AI assistance

Written with agentic AI assistance; my first version returned Option instead of Precision and
would have silently flattened every statistic's precision — the compiler caught it, and the two
precision cases above exist so a reviewer does not have to.

`stats_set_to_df` ran the stored statistic and the column dtype through two
`vortex_expect`s per statistic. Both sides come from the file -- the value out of
the footer's stats set, the dtype off the column -- so both can legitimately
fail on a file we did not write:

- `Stat::dtype` returns `None` where the statistic does not apply, which is
  `Min`/`Max` of a null column and `Sum` of a string, list or struct column;
- `Scalar::try_new` rejects a stored value that does not match the column dtype.

Either one panicked inside DataFusion planning rather than returning an error or
simply forgoing the statistic.

Forgoing it is what the end of the same chain already did: `try_to_df().ok()`
drops a scalar DataFusion cannot represent. This extends that to the whole
conversion and collapses the three near-identical blocks into one helper, which
is also what the `TODO(connor)` above them asked for.

The statistic's `Precision` is preserved, so an exact statistic stays exact and
only the unconvertible ones become `Absent`.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant