Reader for DATASUS .dbc files (and plain .dbf) in Rust, shared by three
front ends. Site: https://strategicprojects.github.io/datasus-rs/ (API
reference under /api).
| Crate | What it is |
|---|---|
crates/datasus-core |
The reader: PKWare DCL decompression + dBase parsing. No dependencies. |
crates/datasus-arrow |
Core → Apache Arrow record batches. |
crates/datasus-cli |
datasus command: convert to Parquet/CSV, inspect, decompress. |
crates/datasus-py |
datasuspy Python package (PyO3 + maturin). |
The core is a port of the C reader of the datasusr R package and is tested to return the same values for the same options (see Validation). Catalog, FTP download and cache are not part of this project; they live in datasusr.
brew install strategicprojects/tap/datasus # Homebrew (macOS, Linux)
cargo install datasus-cli # or with Cargodatasus convert RDPE2401.dbc # -> RDPE2401.parquet (zstd)
datasus convert *.dbc --out-dir parquet/ # many files
datasus convert RDPE2401.dbc -o rd.csv --select uf_zi,val_tot --n-max 1000
datasus schema RDPE2401.dbc # fields and inferred types
datasus decompress RDPE2401.dbc # -> RDPE2401.dbfconvert options mirror read_datasus_dbc(): --select, --n-max,
--no-trim, --encoding, --no-guess-types, --col-type NAME=TYPE,
--parse-dates, --keep-names. Outputs are written to a .part file and
renamed on success; existing outputs need --force.
pip install datasuspyFrom source: cd crates/datasus-py && maturin develop --release inside a
virtualenv.
import datasuspy
table = datasuspy.read_dbc("RDPE2401.dbc", select=["uf_zi", "val_tot"]) # pyarrow.Table
df = table.to_pandas()
import polars as pl
df = pl.from_arrow(datasuspy.read_dbc("RDPE2401.dbc"))
# Streaming, without pyarrow: any Arrow PyCapsule consumer works
reader = datasuspy.open_dbc("RDPE2401.dbc", batch_size=100_000)cargo add datasus-core # the reader, no dependencies
cargo add datasus-arrow # Arrow record batchesuse datasus_core::{open, ReadOptions};
let table = open("RDPE2401.dbc", &ReadOptions::default())?;
let columns = table.collect(); // Vec<ColumnData>To fill your own buffers without an intermediate copy, implement
datasus_core::Sink and call Table::read_rows.
- Records flagged as deleted (
*) are dropped;n_maxcounts live records. - Character fields are never null: a blank field is an empty string.
- Numeric, date and logical fields are always trimmed before parsing;
trim_wsonly affects character fields. A value ends at the first NUL. N/Ffields with no decimals are integers unless a value does not fit a 32-bit integer (then the column is double).i32::MINis not accepted as an integer (it is R'sNA).Dfields are text unlessparse_datesis set; invalid dates are null.- A header declaring more records than the file holds is a warning.
- Decompression stops at the record count declared in the header and never
goes past 4 GiB (
decode_with_limitchanges the bound), so a crafted file cannot expand without limit.
Known differences from the C reader:
- Whitespace is always the six ASCII characters of the "C" locale. The C
reader uses
isspace(), which on macOS under a UTF-8 locale also trims0xA0(NBSP). - Latin-1 text is decoded as strict ISO-8859-1. R on macOS and Windows
converts
latin1strings as Windows-1252, so bytes0x80–0x9Fcome out differently there. - With
encoding = "utf-8", invalid sequences become U+FFFD. - A numeric field holding
0x1Aornan(...)is null: only decimal numbers are numbers. The C reader reads them as numbers becausestrtod()accepts them. This does not occur in DATASUS files.
cargo test # unit + ported datasusr tests
cargo build --release -p datasus-cli
# Oracle: compare against datasusr's C reader (needs R with datasusr + arrow)
Rscript tests/oracle/compare.R <path to datasus binary> \
<datasusr repo>/tests/testthat/helper-dbf.R [real .dbc files...]
cd crates/datasus-py && maturin develop --release && pytest testsMIT. blast.rs is a translation of Mark Adler's blast.c (zlib license).