Granica Crunch overview

Granica Crunch is a continuous, policy-driven data lakehouse optimization platform that automates compaction, compression, vacuuming, partition lifecycle management, and query acceleration across Hive, Apache Iceberg, Delta Lake tables, and raw files and objects — at a scale that in-house tools and generic schedulers cannot reach.

The problem

Data platform teams managing thousands of tables face millions of optimization decisions every day: which files to compact, how aggressively to compress, when to expire snapshots, how to sort data for real query patterns, and how to safely delete PII to meet compliance deadlines.

These decisions are either left undone — causing runaway storage costs, scan bloat, and query degradation — or handled by brittle in-house pipelines that break under scale. DAG based schedulers such as Airflow treat all jobs equally, have no awareness of SLAs or resource contention that are unqiue to table mainainence activities, and turn cascading failures into retry spirals and wasteful orphaned files.

How Granica Crunch helps

Crunch combines a central scheduling engine with per-file optimization tracking, intelligent compute sizing, SLA-aware job dispatch, and bin-packing across resource pools. It manages the full optimization lifecycle for hundreds of thousands of tables processing tens of millions of partitions per day.

• 20–50% better data reduction on top of standard ZSTD, achieved by Granica's Rust-based compression engine that adapts to each file's unique structure
• 10–20% query speed improvement on production workloads through data fingerprinting and layout optimization — no manual tuning required. This leads to direct cost savings on compute as well.

Key features

Intelligent scheduling at scale

Crunch's scheduler has full visibility into resource pools, job SLAs, and per-partition optimization state. When a job fails, Crunch retries only the minimum necessary work and prevents the retry spirals that plague general-purpose systems.

• File-level optimization tracking — Crunch knows the state of every file, not just every table
• Intelligent batch decomposition — large tables are broken into right-sized work units matched to available compute, preventing OOM failures and wasted retries
• Bin-packing and right-sizing — jobs are packed against the resource pool to maximize utilization; compute is sized per job based on data volume and SLA
• SLA-based scheduling — jobs run at the cheapest time that still meets the agreed service window

Superior compression

Crunch's compression engine delivers 20–50% better data reduction on top of standard ZSTD. On a TPC-DS store_sales table (418 GiB, 301,949 files, 1,823 partitions), Crunch achieved 36.4% data reduction versus 14.2% for Databricks ZSTD — producing 266 GiB of output versus 359 GiB on matched hardware.

Query acceleration

Crunch profiles actual query and data patterns to select the optimal layout strategy — sort order, Z-order, file sizing, row group sizing — for each table's real access patterns. This delivers query speed improvements without any manual tuning.

Multi-format and multi-catalog support

Crunch manages all major open table formats natively:

• Delta Lake — compaction, compression, VACUUM, transaction log cleanup
• Apache Iceberg — snapshot expiration, orphan file deletion, metadata compaction
• Hive / Parquet — compaction and compression

It integrates with every major data catalog: Hive Metastore, AWS Glue, Unity Catalog, and Polaris. Critically, Crunch detects shared files across table formats — preventing accidental data loss during Hive-to-Iceberg or Hive-to-Delta migrations that format-specific tools cannot detect.

Compliance and lifecycle management

Crunch treats compliance as a first-class workflow:

• Partition-level deletion — delete entire partitions on a defined schedule or on-demand for GDPR "right to be forgotten" requests
• Row-level PII deletion — row-level deletes using the native table format's delete mechanism for tables where PII is not partition-isolated (GA coming soon)
• Audit trail — full audit log for compliance reporting

Production safety

Crunch was designed with production safety as the primary constraint:

• Read-before-write validation — validates table state and catalog metadata before issuing any write
• Atomic commits — all results are committed via the native Delta transaction log or Iceberg commit protocol; no partial writes are visible to readers
• File-level rollback — if a job is interrupted, exactly the affected files are identified and cleaned up
• Cross-catalog shared-file detection — before running vacuum or orphan deletion, Crunch checks all connected catalogs for shared file references
• Non-destructive by default — original files remain in storage until the configured retention window expires, providing a recovery window

No vendor lock-in

Crunch works exclusively with open table formats using their native commit protocols. It does not introduce any proprietary file format, metadata extension, or catalog dependency. Tables remain fully readable and writable by any engine — Spark, Trino, Flink, DuckDB, Snowflake, Athena — if Crunch is turned off.

See also