Table Compaction

Table compaction rewrites many small files into fewer, optimally sized files. Streaming ingestion, frequent micro-batches, and row-level updates all produce large numbers of small files, which inflate metadata, increase the number of requests against object storage, and slow down every query that scans the table.

Crunch compacts Hive, Apache Iceberg, and Delta Lake tables continuously, using the table format's native commit protocol so that readers never see a partial result.

Why small files hurt

• Metadata bloat — every file adds an entry to the manifest or transaction log, which each query planner must read and evaluate.
• Request overhead — thousands of small reads cost more and are slower than a few large sequential reads.
• Poor compression — compression works on a per-file basis, so small files leave significant reduction on the table.
• Scheduler pressure — query engines spawn one task per file, so small files create task overhead that dominates the actual work.

How Crunch compacts

1. Data-driven strategy — append-only vs. upsert, streaming vs. batch, with deletion files or vectors vs. without, Hive vs. Delta or Iceberg — each of these variations requires a different compaction strategy. Crunch abstracts away all these differences and picks the right strategy for you automatically.
2. File-level tracking — Crunch maintains the optimization state of every file in every partition, so it recompacts only when needed.
3. Batch decomposition — large tables are split into right-sized work units that match the available compute, which prevents out-of-memory failures and wasted retries.
4. Target file sizing — Crunch selects a target file size and row group size based on the table's format, schema, and observed query patterns.
5. Atomic commit — results are committed as an Iceberg snapshot, a Delta transaction, or a direct object replacement for Hive. No partial writes are visible to readers.
6. Production job awareness — Crunch is aware of your production pipelines and jobs, which allows it to schedule rewrites without conflicting with them.

Configure compaction

Compaction is enabled by setting a policy on a table through the Granica Console or the Granica API. A policy controls:

• Which tables or partitions are in scope
• The target file size
• The SLA window in which compaction jobs must complete

Once a policy is active, compaction runs continuously in the background with no further manual steps.

Safety

• Original files remain in storage until the configured retention window expires, which provides a recovery window.
• Table state and catalog metadata are validated before any write is issued.
• If a job is interrupted, exactly the affected files are identified and cleaned up.

See also