How crunching works

Once you've deployed Granica into your cloud environment, it's time to get crunching. "Crunching" is our euphemism for data processing in the context of compression optimization, where we "crunch" the data down to its purest information-rich state.

Crunch is initiated by setting a policy on a table through the Granica Console or directly through the Granica API. Once a policy is active, Crunch runs continuously in the background without any further manual steps.

Production lakehouse data

Production lakehouse data is the columnar data your teams are generating and working with every day. Crunch offers two mechanisms to compress and optimize that data:

Crunch lexicon

• Crunched buckets — those under active management, processing and monitoring by Crunch
• Crunched objects — objects evaluated by Crunch for compression optimization
• Vanilla buckets — those which have not been crunched
• Vanilla objects — those which have not been crunched
• Ingested objects — crunched and reduced (background mode only)
• Analyzed objects — crunched and analyzed to generate optimal recipes (runtime mode only)

Background crunch write workflow

This is the primary mode the Crunch works. It silently optimize tables at rest and carry out maintainece activities on the background without users even noticing. Once your data is crunched you'll see immediate savings in your lakehouse storage costs.

1. An administrator sets a Crunch policy on a table via the Granica Console or API. The Controller retrieves vanilla objects.

2. The Crunch scheduler determines when to process the table based on the configured SLA and policy.

3. The Controller sends vanilla objects to a load balanced Compression optimizer.

4. The Compression optimizer validates policy eligibility, optimizes the compression and encoding, and commits the optimized files using the table format's native commit protocol (Iceberg or Delta Lake snapshot commit, or direct object replacement for Hive) — initiating a reduction in your monthly cloud storage bill.

Runtime crunch write workflow

In some rare cases, customers may choose to integrate Crunch into their existing ETL engine. In this mode Crunch has two main components:

1. An ML-powered adaptive compression control system which analyzes your existing columnar files to create compression optimization recipes.

2. A runtime optimizer JAR which integrates into your data platform and is invoked transparently by any applications utilizing an open source Apache Parquet writer, without any code changes.

1. An administrator routes vanilla objects to a Compression recipe generator, which analyzes the unique characteristics and structure of the columnar files.

2. The Compression recipe generator updates the Compression recipe store with new or updated recipes.

3. Spark-based applications initiate columnar writes using standard commands. The Granica runtime JAR intercepts the write and applies the best available recipe.

4. The JAR writes data out in standard, lakehouse-native format (typically Parquet).

5. When needed — for example, when a schema change is detected — the runtime JAR notifies administrator to re-analyze newly created files and update the recipe.

See also