Data Ingestion
Bring tables, buckets, and objects under Crunch management and optimize data as it lands.
Ingestion is how data comes under Crunch management. Crunch discovers the tables and objects in your staging bucket, evaluates them against your policies, and optimizes them as they land.
Discovery
Crunch offers an optional "Discovery" feature. Once a catalog and an object store are connected, Crunch enumerates the objects and compares them against the objects referenced by your tables to identify buckets and prefixes where raw data such as JSON and CSV exists. It then records file types and total sizes.
Nothing is modified during discovery.
See Connect catalogs and Connect object stores to set up the connections.
Ingestion features
- Sources — Crunch supports common raw data types, including JSON, CSV, CSV-gzip, and Parquet. We are adding more file types.
- Targets — Crunch supports writing to Iceberg tables, Delta tables, Hive tables, and raw Parquet files.
- Transformation — Crunch supports basic transformation functions such as type casting and column mapping. We are adding more transformation features.
- Optimization — Crunch ingestion uses the same underlying engine that Crunch compaction uses, which produces optimal Parquet files that are smaller and faster to read. An internal benchmark study using the TPC-DS dataset shows that Crunch ingestion produces 30% to 70% smaller files than Databricks Auto Loader does.
- Performance — Crunch ingestion uses a streaming engine and an optimized writer that provide exceptional performance. The same internal benchmark shows that Crunch, on the same hardware configuration, ingests 4 to 5 times faster than Auto Loader at only 15% of the cost.
Throughput and scheduling
Ingestion jobs are dispatched by the same scheduler that handles compaction and garbage collection, so the initial backfill of a large table competes for resources on your terms. The scheduler sizes compute per job from data volume and SLA, packs jobs against the resource pool to maximize utilization, and runs work at the cheapest time that still meets the agreed service window.
For the first ingestion of a very large table, start with a subset of partitions to validate, then widen the policy scope.
See also
Query Acceleration
Optimize data layout — sort order, clustering, file and row group sizing, stats, and metadata — against real query patterns.
Security certifications
Learn about Granica's commitment to information security and compliance.