Data lakehouse compatibility
Understand how compression optimization works with your lakehouse.
Ensuring seamless compatibility with leading data lakehouse environments is critical for any compression-related solution. Granica Crunch compression optimization can operate in two distinct modes. Runtime Crunch optimizes net-new files in real-time as they are written, while background Crunch optimizes already written columnar files. Each mode also has different lakehouse compatibility, as outlined below.
Runtime Crunch compatibility
Runtime Crunch utilizes a patched open source Parquet writer. The patch enables dynamic loading of compression optimization recipes to create deeply optimized columnar files as they are written. As a result, runtime Crunch is fully compatible with all applications and sytems which utilize the open source Parquet reader to read Parquet files. You can learn more from reading how crunching works.
The following pre-requisites are required for integration of the runtime Crunch SDK, which includes the patched writer, into your Java Runtime Environment (JRE) for your applications/systems.
SDK Pre-requisites
- Apache Spark 2.x+
- Python 3.7+ (for PySpark users)
- Java 8+ (for Scala/Java users)
tip
Runtime Crunch optimization is available to select design partner customers - request Early Access.
Background Crunch compatibility
Granica Crunch background mode has been thoroughly tested for compatibility with key data lakehouse platforms and query engines with the following compatibility results:
| Compatibility Matrix | Result |
|---|---|
| Spark (2.4.8+) | Fully supported |
| Hadoop (2.9+) | Fully supported |
| Hive (2.0+) | Fully supported |
| Presto (0.245) | Fully supported |
| Trino (356+) | Fully supported |
| Customer-specific test xx | Fully supported |
| Read/write operations | YYY% success |
| Query execution | YYY% success |
| Schema preservation | YYY% success |
Granica is committed to flexibility — support for other platforms can be added based on evolving customer needs, and we welcome specific requests to expand compatibility further.