Package and access data on Amazon S3 in a squash manner
-
Updated
Aug 6, 2023 - Rust
#
small-files
Here are 11 public repositories matching this topic...
A file uploader specialized for uploading many small files onto HDFS
-
Updated
Apr 18, 2018 - Java
PySpark script to aggregate small parquet files in a prefix into larger files. Designed to be run on AWS Glue
-
Updated
May 13, 2026 - Python
Production-grade Zstd dictionary compression microservice for small JSON APIs. Solves frame overhead & schema drift with trained dictionaries, versioned registries, and dual-signal drift monitoring. 75%+ savings on disjoint holdouts.
python compression microservice json-api data-compression zstd zstandard small-files dictionary-compression drift-detection api-optimization schemadrift
-
Updated
Oct 20, 2026 - Python
Read-only "small-file tax" analyzer for Delta Lake. Reads only the _delta_log to reconstruct the per-partition file-size distribution, then prices the wasted compute and S3 requests in dollars using real Spark internals (openCostInBytes, maxPartitionBytes) and emits a compaction plan with ROI and payback plus the OPTIMIZE SQL. Never runs OPTIMIZE.
python cli spark s3 optimize data-engineering zero-dependency roi cost-optimization small-files finops compaction delta-lake lakehouse data-observability delta-transaction-log
-
Updated
Jul 8, 2026 - Python
Command Line Hangman/Word Guessing game.
-
Updated
Sep 4, 2016 - C
Read-only Delta Lake table health scanner with safe fix plans for Databricks, Python, and PySpark.
python spark optimize pyspark data-engineering vacuum databricks small-files delta-lake table-maintenance
-
Updated
May 12, 2026 - Python
File merge action to merge files in HDFS or local filesystem
-
Updated
Apr 12, 2022 - Java
antfs: a distributed blob storage system in Rust, purpose-built for massive small files. Inspired by Ambry / SeaweedFS / CubeFS — append-only log + index for O(1) small-file I/O, built-in Raft control plane, REST API, admin dashboard.
rust distributed-systems storage-engine raft distributed-storage object-storage blob-storage seaweedfs small-files ambry cubefs
-
Updated
Oct 22, 2026 - Rust
Fast multithreaded archiver for massive numbers of small files, optimized for cloud object storage
compression cpp tar cloud-storage multithreading archive small-files multimodal-data ai-infrastructure
-
Updated
Oct 25, 2026 - C++
Add this topic to your repo
To associate your repository with the small-files topic, visit your repo's landing page and select "manage topics."