Using Zstd Frames to Egress Partial Parquet Files

Short summary

Data engineers waste 99.9% of egress bandwidth fetching entire multi-GB Parquet files when they only need metadata. HuskHoard uses seekable Zstd frames and jump tables to enable precise byte-range access, cutting a $9,216/month schema-sync bill to $9. The system mirrors Parquet's internal architecture at the archive level, enabling single-seek HTTP range requests across S3, GCS, and LTO tape.

• •Traditional full-file egress for schema discovery costs $9,216/month on 100TB; partial-read approach drops to $9 (99.9% savings)
• •HuskHoard uses concatenated Zstd frames with jump tables mapping uncompressed to compressed byte offsets for precise seeking
• •Architecture mirrors Parquet's row groups and column chunks at archive level, enabling single HTTP Range request per query

Generated with AI, which can make mistakes.