Read and write data on any S3-compatible object store (AWS S3, Garage, MinIO, Ceph, …) with the same nine verbs in R, Python, Rust and Go: the same defaults, the same signing, the same retries and the same integrity checks. The endpoint and the credentials come from the environment; the format comes from the file extension; every transfer prints one line.
Website: https://strategicprojects.github.io/s3pe/ · Português
| Edition | Install | Docs |
|---|---|---|
| R | install.packages("s3pe") · pak::pak("StrategicProjects/s3pe/r") |
pkgdown |
| Python | pip install s3pe ("s3pe[pandas]" for tables) |
site · API |
| Rust | cargo add s3pe · cargo install s3pe (the s3pe command) |
site · docs.rs |
| Go | go get github.com/StrategicProjects/s3pe |
site · pkg.go.dev |
library(s3pe) # R
use_endpoint("https://s3.example.org"); use_bucket("my-bucket")
meta <- read_s3("data/manifest.json") # list, from JSON
sales <- read_s3("series/sales.csv.gz") # tibble
write_s3(sales, "reports/sales.parquet")
list_s3("data/")import s3pe # Python
meta = s3pe.read_s3("s3://my-bucket/data/manifest.json") # dict
sales = s3pe.read_s3("series/sales.csv.gz", bucket="my-bucket") # DataFrame
s3pe.write_s3(sales, "s3://my-bucket/reports/sales.parquet")
s3pe.list_s3("s3://my-bucket/data/")let meta = s3pe::read_s3("s3://my-bucket/data/manifest.json")?; // Rust: Vec<u8>
s3pe::upload_s3("sales.parquet", "s3://my-bucket/reports/sales.parquet", None)?;
for o in s3pe::list_s3("s3://my-bucket/data/")? { println!("{} {}", o.size, o.key); }meta, err := s3pe.ReadS3(ctx, "s3://my-bucket/data/manifest.json") // Go: []byte
_, err = s3pe.UploadS3(ctx, "sales.parquet", "s3://my-bucket/reports/sales.parquet", "")
objects, err := s3pe.ListS3(ctx, "s3://my-bucket/data/")s3pe ls s3://my-bucket/data/ # command line
s3pe get s3://my-bucket/data/regions.geojson
s3pe put sales.parquet s3://my-bucket/reports/| Verb | Does |
|---|---|
read_s3 / write_s3 |
move an object in memory: decoded by extension in R and Python, bytes in Rust and Go |
download_s3 / upload_s3 |
move a file, streaming, written as .s3pe-part and renamed when complete |
list_s3 |
every object under a prefix, all pages |
exists_s3 / head_s3 |
whether an object exists; its size, type, date and ETag |
delete_s3 |
delete (idempotent) |
buckets_s3 |
the buckets the credentials can see |
copy_s3 |
server-side copy |
presign_s3 |
a URL that works without credentials for a while (GET or PUT) |
sync_up_s3 / sync_down_s3 |
mirror a directory and a prefix by ETag, a manifest last |
write_s3/upload_s3 take metadata and headers (so SSE is one header
away where the store supports it); delete_s3 takes many keys at once;
list_s3(delimiter = "/") lists one level. Files above 64 MiB go up as a
multipart upload, parts in parallel, aborted cleanly on failure. No ACLs
and no versioning, by design (DESIGN.md D19). Every edition also has a
Client (R: s3_config()) for explicit settings.
Argument, then environment, then default:
| Setting | Environment | Default |
|---|---|---|
| endpoint | S3PE_ENDPOINT, then AWS_ENDPOINT_URL |
required (https://host[:port]) |
| region | AWS_REGION, then AWS_DEFAULT_REGION |
us-east-1 |
| credentials | AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN |
required |
| bucket | S3PE_BUCKET |
none: write s3://bucket/key |
| quiet | S3PE_QUIET=1 |
one log line per transfer |
In R the usual place is ~/.Renviron (or use_endpoint() / use_bucket()
for the session); in Python and the CLI any .env loader or the shell.
Never put the key in a script or a repository.
- Signing: AWS Signature V4, header style, over the real SHA-256 of the
payload. No
aws-chunked(which some stores reject), no unsigned payloads. The four signers reproduce the AWS documentation vector. - Retries: connection failures and HTTP 429/500/502/503/504, 3 tries, 1 s then 2 s. Never on other 4xx.
- Typed errors: not found, access denied, other HTTP (with the S3 code),
transport, configuration, integrity.
exists_s3/head_s3turn 404 intoFALSE/None/nil. - Integrity: ETag vs MD5 after uploads;
Content-Lengthand MD5 after downloads. - Tests: each edition runs against a local fake S3 that verifies every
signature, and against a real store when
S3PE_ENDPOINTandS3PE_TEST_BUCKETare set.
Design decisions are recorded in DESIGN.md; changes in CHANGELOG.md.
r/ R package (CRAN) rust/ Rust crate and CLI (crates.io)
python/ Python package, PyO3 on the crate *.go Go module (this directory)
site/ landing pages (GitHub Pages) DESIGN.md decisions shared by the four
André Leite, Marcos Wasiliew, Hugo Vasconcelos, Carlos Amorim, Diogo Bezerra and Júlia Nascimento Barreto. MIT license. Cite with CITATION.cff.
s3pe is developed in Recife, Brazil.