Skip to content

About

Rust reader for DATASUS .dbc/.dbf files: core library, CLI (Parquet/CSV) and Python bindings (datasuspy)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

datasus-rs

CI crates.io PyPI DOI License: MIT

Reader for DATASUS .dbc files (and plain .dbf) in Rust, shared by three front ends. Site: https://strategicprojects.github.io/datasus-rs/ (API reference under /api).

Crate What it is
crates/datasus-core The reader: PKWare DCL decompression + dBase parsing. No dependencies.
crates/datasus-arrow Core → Apache Arrow record batches.
crates/datasus-cli datasus command: convert to Parquet/CSV, inspect, decompress.
crates/datasus-py datasuspy Python package (PyO3 + maturin).

The core is a port of the C reader of the datasusr R package and is tested to return the same values for the same options (see Validation). Catalog, FTP download and cache are not part of this project; they live in datasusr.

CLI

brew install strategicprojects/tap/datasus     # Homebrew (macOS, Linux)
cargo install datasus-cli                      # or with Cargo
datasus convert RDPE2401.dbc                       # -> RDPE2401.parquet (zstd)
datasus convert *.dbc --out-dir parquet/           # many files
datasus convert RDPE2401.dbc -o rd.csv --select uf_zi,val_tot --n-max 1000
datasus schema RDPE2401.dbc                        # fields and inferred types
datasus decompress RDPE2401.dbc                    # -> RDPE2401.dbf

convert options mirror read_datasus_dbc(): --select, --n-max, --no-trim, --encoding, --no-guess-types, --col-type NAME=TYPE, --parse-dates, --keep-names. Outputs are written to a .part file and renamed on success; existing outputs need --force.

Python

pip install datasuspy

From source: cd crates/datasus-py && maturin develop --release inside a virtualenv.

import datasuspy

table = datasuspy.read_dbc("RDPE2401.dbc", select=["uf_zi", "val_tot"])  # pyarrow.Table
df = table.to_pandas()

import polars as pl
df = pl.from_arrow(datasuspy.read_dbc("RDPE2401.dbc"))

# Streaming, without pyarrow: any Arrow PyCapsule consumer works
reader = datasuspy.open_dbc("RDPE2401.dbc", batch_size=100_000)

Rust

cargo add datasus-core        # the reader, no dependencies
cargo add datasus-arrow       # Arrow record batches
use datasus_core::{open, ReadOptions};

let table = open("RDPE2401.dbc", &ReadOptions::default())?;
let columns = table.collect();           // Vec<ColumnData>

To fill your own buffers without an intermediate copy, implement datasus_core::Sink and call Table::read_rows.

Behaviour (same as datasusr 0.1.1)

  • Records flagged as deleted (*) are dropped; n_max counts live records.
  • Character fields are never null: a blank field is an empty string.
  • Numeric, date and logical fields are always trimmed before parsing; trim_ws only affects character fields. A value ends at the first NUL.
  • N/F fields with no decimals are integers unless a value does not fit a 32-bit integer (then the column is double). i32::MIN is not accepted as an integer (it is R's NA).
  • D fields are text unless parse_dates is set; invalid dates are null.
  • A header declaring more records than the file holds is a warning.
  • Decompression stops at the record count declared in the header and never goes past 4 GiB (decode_with_limit changes the bound), so a crafted file cannot expand without limit.

Known differences from the C reader:

  • Whitespace is always the six ASCII characters of the "C" locale. The C reader uses isspace(), which on macOS under a UTF-8 locale also trims 0xA0 (NBSP).
  • Latin-1 text is decoded as strict ISO-8859-1. R on macOS and Windows converts latin1 strings as Windows-1252, so bytes 0x80–0x9F come out differently there.
  • With encoding = "utf-8", invalid sequences become U+FFFD.
  • A numeric field holding 0x1A or nan(...) is null: only decimal numbers are numbers. The C reader reads them as numbers because strtod() accepts them. This does not occur in DATASUS files.

Validation

cargo test                                   # unit + ported datasusr tests
cargo build --release -p datasus-cli

# Oracle: compare against datasusr's C reader (needs R with datasusr + arrow)
Rscript tests/oracle/compare.R <path to datasus binary> \
  <datasusr repo>/tests/testthat/helper-dbf.R  [real .dbc files...]

cd crates/datasus-py && maturin develop --release && pytest tests

License

MIT. blast.rs is a translation of Mark Adler's blast.c (zlib license).

About

Rust reader for DATASUS .dbc/.dbf files: core library, CLI (Parquet/CSV) and Python bindings (datasuspy)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages