# Granica APIs V1 Overview [#overview] The Granica REST API (v1) lets you programmatically manage all aspects of the Granica platform, including table onboarding, compaction scheduling, catalog connections, query optimization, and object maintenance. **Base path:** `/api/v1` All requests require valid authentication credentials. Responses use standard HTTP status codes; `422` indicates a validation error with detail in the response body. *** Health [#health] Health Check [#health-check] `GET /api/v1/health` Health check endpoint. Returns `200` when the API is operational. *** Configuration [#configuration] Get Config [#get-config] `GET /api/v1/config` Get customer configuration. | Response | Description | | -------- | ------------------------------------------- | | `200` | Returns the current customer configuration. | Update Config [#update-config] `PUT /api/v1/config` Update customer configuration. **Request body:** `CustomerConfigUpdate` | Response | Description | | -------- | ---------------------------------- | | `200` | Returns the updated configuration. | | `422` | Validation error. | *** Tables [#tables] List Tables [#list-tables] `GET /api/v1/tables` List all tables with their schedules. Supports filtering by `status` and `name`. Use `name` to resolve a table's ID by its unique name (e.g. after a `409` on `POST /tables`). | Parameter | In | Required | Description | | --------- | ----- | -------- | ----------------------------------------------------------------- | | `offset` | query | No | Pagination offset. Default: `0`. | | `limit` | query | No | Results per page. Default: `100`. | | `status` | query | No | Filter by onboarding status (e.g. `pending`, `active`, `failed`). | | `name` | query | No | Filter by exact table name. | Create Table [#create-table] `POST /api/v1/tables` Register a table for onboarding. Schedule is added separately after onboarding completes. Partition pattern, date filter, cluster and enumerated columns are auto-discovered during onboarding. **Request body:** `TableCreate` ```json { "name": "orders", "uri": "s3://bucket/data/orders", "format": "delta", "priority": "P2" } ``` | Response | Description | | -------- | --------------------------- | | `201` | Table created successfully. | | `422` | Validation error. | Get Table [#get-table] `GET /api/v1/tables/{table_id}` Get a table by ID, including its schedule. Update Table [#update-table] `PUT /api/v1/tables/{table_id}` Update table configuration. **Request body:** `TableUpdate` ```json { "priority": "P1", "target_file_size_mb": 512 } ``` Delete Table [#delete-table] `DELETE /api/v1/tables/{table_id}` Delete a table and its schedules. Returns `204` on success. Get Inventory History [#get-inventory-history] `GET /api/v1/tables/{table_id}/inventory-history` Append-only audit of `inventory_report_uri` changes for a table, ordered newest-first. Rows are retained after the table is deleted, so this may return audit history for removed tables. | Parameter | In | Required | Description | | --------- | ----- | -------- | -------------------------------- | | `offset` | query | No | Pagination offset. Default: `0`. | | `limit` | query | No | Results per page. Default: `50`. | *** Schedules [#schedules] Create Schedule [#create-schedule] `POST /api/v1/tables/{table_id}/schedule` Create a schedule for a table. Only works when `onboarding_status` is `active`. **Request body:** `ScheduleCreate` **Daily with lookback range (T-15 to T-2):** ```json { "schedule_type": "daily", "lookback_start": 15, "lookback_end": 2, "run_time_utc": "02:00" } ``` **Backfill:** ```json { "schedule_type": "backfill", "backfill_col": "event_date", "backfill_start_date": "2025-01-01", "backfill_end_date": "2025-12-31" } ``` Get Schedule [#get-schedule] `GET /api/v1/tables/{table_id}/schedule` Get the schedule for a table. | Parameter | In | Required | Description | | ---------- | ----- | -------- | ------------------------------------------------------------------ | | `job_type` | query | No | Filter by job type: `crunch`, `vacuum`, or `partition_expiration`. | Update Schedule [#update-schedule] `PUT /api/v1/tables/{table_id}/schedule` Update the schedule for a table. **Request body:** `ScheduleUpdate` ```json { "lookback_start": 7, "lookback_end": 1, "run_time_utc": "03:00", "enabled": false } ``` Delete Schedule [#delete-schedule] `DELETE /api/v1/tables/{table_id}/schedule` Delete the schedule for a table. Returns `204` on success. | Parameter | In | Required | Description | | ---------- | ----- | -------- | ------------------- | | `job_type` | query | No | Filter by job type. | *** Onboarding [#onboarding] Get Onboarding Status [#get-onboarding-status] `GET /api/v1/onboarding/{table_id}/status` Get the onboarding status for a table. Retry Onboarding [#retry-onboarding] `POST /api/v1/onboarding/{table_id}/retry` Retry onboarding for a failed table. Only tables with `onboarding_status='failed'` can be retried. Resets status to `retry` for the onboarding scheduler to pick up. *** Slots [#slots] Get Slot Info [#get-slot-info] `GET /api/v1/slots/{run_time_utc}` Get slot capacity information for a specific time. Time format: `HH:MM` (e.g. `02:30`, `14:00`). *** Crunch [#crunch] Trigger Crunch [#trigger-crunch] `POST /api/v1/tables/{table_id}/crunch` Directly trigger a crunch job, bypassing the schedule. **Request body:** `CrunchTrigger` Two modes (mutually exclusive): **Lookback mode** (relative to today): ```json { "lookback_start": 15, "lookback_end": 2, "priority": "P2" } ``` **Backfill mode** (absolute dates): ```json { "backfill_start_date": "2026-01-01", "backfill_end_date": "2026-01-15", "priority": "P2", "cluster_columns": "user_id,event_type" } ``` | Parameter | In | Required | Description | | ------------------- | ----- | -------- | ------------------------------------------------------------- | | `skip_recipe_check` | query | No | Skip recipe completion check (for testing). Default: `false`. | List Table Crunch Jobs [#list-table-crunch-jobs] `GET /api/v1/tables/{table_id}/crunch` List crunch jobs for a specific table. | Parameter | In | Required | Description | | ---------- | ----- | -------- | ------------------------------------------- | | `offset` | query | No | Pagination offset. Default: `0`. | | `limit` | query | No | Results per page. Default: `100`. | | `status` | query | No | Filter by crunch job status. | | `job_type` | query | No | Filter by job type: `crunch` or `optimize`. | List All Crunch Jobs [#list-all-crunch-jobs] `GET /api/v1/crunch` List all crunch jobs with optional filters. | Parameter | In | Required | Description | | ---------- | ----- | -------- | ---------------------------- | | `offset` | query | No | Default: `0`. | | `limit` | query | No | Default: `100`. | | `table_id` | query | No | Filter by table. | | `status` | query | No | Filter by crunch job status. | | `job_type` | query | No | Filter by job type. | Get Crunch Job With Metrics [#get-crunch-job-with-metrics] `GET /api/v1/crunch/{crunch_id}/metrics` Get a crunch job with aggregated metrics including DRR (Data Reduction Rate), bytes before/after, and optimization metrics. Cancel Crunch Job [#cancel-crunch-job] `POST /api/v1/crunch/{crunch_id}/cancel` Cancel a scheduled or in-flight crunch job. *** Vacuum [#vacuum] Trigger Vacuum [#trigger-vacuum] `POST /api/v1/tables/{table_id}/vacuum` Trigger a vacuum operation on a Delta or Iceberg table. **Request body:** `VacuumTrigger` List Table Vacuum Jobs [#list-table-vacuum-jobs] `GET /api/v1/tables/{table_id}/vacuum` List vacuum jobs for a table. | Parameter | In | Required | Description | | --------- | ----- | -------- | ----------------- | | `offset` | query | No | Default: `0`. | | `limit` | query | No | Default: `100`. | | `status` | query | No | Filter by status. | Trigger Hard Delete [#trigger-hard-delete] `POST /api/v1/vacuum/hard-delete` Hard-delete files for all expired `PENDING_DELETE` records on demand. | Parameter | In | Required | Description | | ---------- | ----- | -------- | ------------------------------------------------- | | `dry_run` | query | No | Preview without deleting. Default: `false`. | | `table_id` | query | No | Scope to a specific table. | | `limit` | query | No | Max records per call. Default: `500`, cap `5000`. | *** Partition Expiration [#partition-expiration] Trigger Partition Expiration [#trigger-partition-expiration] `POST /api/v1/tables/{table_id}/partition-expiration` Trigger partition expiration (DELETE-only) on a Delta table. Files are tombstoned in the table log; physical reclamation is handled by the vacuum pipeline. **Request body:** `PartitionExpirationTrigger` List Partition Expiration Jobs [#list-partition-expiration-jobs] `GET /api/v1/tables/{table_id}/partition-expiration` List partition expiration jobs for a table. | Parameter | In | Required | Description | | --------- | ----- | -------- | ----------------- | | `offset` | query | No | Default: `0`. | | `limit` | query | No | Default: `100`. | | `status` | query | No | Filter by status. | Validate Partition Expiration Policy [#validate-partition-expiration-policy] `POST /api/v1/tables/{table_id}/partition-expiration/validate` Preflight a partition expiration policy before persisting. Side-effect-free — no DB writes. **Request body:** `PartitionExpirationValidateRequest` *** Pending Deletions [#pending-deletions] List Pending Deletions [#list-pending-deletions] `GET /api/v1/pending-deletions` List pending deletions with optional filters. | Parameter | In | Required | Description | | ------------- | ----- | -------- | ---------------------- | | `offset` | query | No | Default: `0`. | | `limit` | query | No | Default: `20`. | | `status` | query | No | Filter by status. | | `table_id` | query | No | Filter by table. | | `action_type` | query | No | Filter by action type. | | `vacuum_mode` | query | No | Filter by vacuum mode. | Get Pending Deletion Stats [#get-pending-deletion-stats] `GET /api/v1/pending-deletions/stats` Aggregated pending deletion counts by status. | Parameter | In | Required | Description | | ---------- | ----- | -------- | -------------------------- | | `table_id` | query | No | Scope to a specific table. | Get Pending Deletion [#get-pending-deletion] `GET /api/v1/pending-deletions/{deletion_id}` Get a single pending deletion record. Recover Pending Deletion [#recover-pending-deletion] `POST /api/v1/pending-deletions/{deletion_id}/recover` Recover a pending deletion to prevent hard-delete. Only `PENDING_DELETE` records can be recovered. **Request body:** `RecoverRequest` *** Catalog Connections [#catalog-connections] Test Connection [#test-connection] `POST /api/v1/catalog-connections/test` Test a catalog connection without persisting. Validates credentials by listing catalogs. **Request body:** `ConnectionTest` Create Connection [#create-connection] `POST /api/v1/catalog-connections` Create a new catalog connection. **Request body:** `ConnectionCreate` List Connections [#list-connections] `GET /api/v1/catalog-connections` List catalog connections. | Parameter | In | Required | Description | | ----------- | ----- | -------- | ------------------------ | | `offset` | query | No | Default: `0`. | | `limit` | query | No | Default: `100`. | | `is_active` | query | No | Filter by active status. | Get Connection [#get-connection] `GET /api/v1/catalog-connections/{connection_id}` Get a single catalog connection. Update Connection [#update-connection] `PUT /api/v1/catalog-connections/{connection_id}` Update a catalog connection (partial update). **Request body:** `ConnectionUpdate` Delete Connection [#delete-connection] `DELETE /api/v1/catalog-connections/{connection_id}` Delete a catalog connection and all its synced metadata. | Parameter | In | Required | Description | | --------- | ----- | -------- | ---------------------------------------------------- | | `confirm` | query | No | Must be `true` to proceed. Deletion is irreversible. | Trigger Sync [#trigger-sync] `POST /api/v1/catalog-connections/{connection_id}/sync` Trigger a catalog metadata sync. Runs in the background. | Parameter | In | Required | Description | | --------- | ----- | -------- | -------------------------------------------------------------------------------------------------- | | `catalog` | query | No | Limit sync to a specific catalog. | | `mode` | query | No | `full` (walk all tables, detect deletions) or `incremental` (Unity Catalog only). Default: `full`. | List Catalogs [#list-catalogs] `GET /api/v1/catalog-connections/{connection_id}/catalogs` List catalogs for a connection. List Schemas [#list-schemas] `GET /api/v1/catalog-connections/{connection_id}/catalogs/{catalog_name}/schemas` List schemas within a catalog. List Tables (Catalog) [#list-tables-catalog] `GET /api/v1/catalog-connections/{connection_id}/tables` List tables with filtering, sorting, and pagination. | Parameter | In | Required | Description | | ---------------- | ----- | -------- | --------------------------------- | | `catalog` | query | No | Filter by catalog. | | `schema` | query | No | Filter by schema. | | `prefix` | query | No | Table name prefix filter. | | `search` | query | No | Search in full table name. | | `format` | query | No | Filter by table format. | | `table_type` | query | No | Filter by table type. | | `min_size_bytes` | query | No | Minimum table size filter. | | `max_size_bytes` | query | No | Maximum table size filter. | | `has_partitions` | query | No | Filter by partitioning. | | `sort_by` | query | No | Sort field. Default: `full_name`. | | `sort_order` | query | No | `asc` or `desc`. Default: `asc`. | | `offset` | query | No | Default: `0`. | | `limit` | query | No | Default: `100`. | Get Table Detail (Catalog) [#get-table-detail-catalog] `GET /api/v1/catalog-connections/{connection_id}/tables/{full_name}` Get full table detail including JSONB fields. *** Feature Flags [#feature-flags] Get Feature Flag [#get-feature-flag] `GET /api/v1/feature-flags/{key}` Return the enabled state of a single feature flag. Returns `{"key": key, "enabled": false}` when the flag is missing — treat absence as disabled. *** Optimus (Query Optimization) [#optimus-query-optimization] Create Optimus Run [#create-optimus-run] `POST /api/v1/optimus/runs` Create and trigger an Optimus recommendation run. **Request body:** `OptimusRunCreate` Retry Optimus Run [#retry-optimus-run] `POST /api/v1/optimus/runs/{submission_id}/retry` Retry a failed Optimus submission. Returns `422` unless the effective status is `failed`. Sync Optimus Submission Now [#sync-optimus-submission-now] `POST /api/v1/optimus/runs/{submission_id}/sync-now` Fire an out-of-cycle Optimus run immediately, regardless of the configured `daily_sync_time_utc`. Delete Optimus Submission [#delete-optimus-submission] `DELETE /api/v1/optimus/runs/{submission_id}` Hard-delete a submission. Cascades to runs, recommendations, and per-day predicate combos. Get Optimus Run [#get-optimus-run] `GET /api/v1/optimus/runs/{run_id}` Get Optimus submission dispatch metadata and latest-run summary. Update Optimus Submission [#update-optimus-submission] `PATCH /api/v1/optimus/runs/{run_id}` Partially update an Optimus submission. Use to update `name`, `lookback_days`, `daily_sync_time_utc`, `query_logs_uri`, or `active` (pause/resume toggle). **Request body:** `OptimusSubmissionUpdate` List Optimus Recommendations [#list-optimus-recommendations] `GET /api/v1/optimus/runs/{run_id}/recommendations` List per-table recommendations for the submission's latest run. | Parameter | In | Required | Description | | --------- | ----- | -------- | --------------- | | `offset` | query | No | Default: `0`. | | `limit` | query | No | Default: `100`. | Apply Optimus Recommendation [#apply-optimus-recommendation] `POST /api/v1/optimus/recommendations/{recommendation_id}/apply` Apply one persisted Optimus recommendation to a managed table policy. **Request body:** `OptimusRecommendationApplyRequest` List Optimus Tables With Recommendations [#list-optimus-tables-with-recommendations] `GET /api/v1/optimus/runs/{run_id}/tables` List tables that have at least one recommendation under this submission. Get Optimus Table Metrics [#get-optimus-table-metrics] `GET /api/v1/optimus/runs/{run_id}/tables/{table_fqn}/metrics` Per-day metric series for a table over the last `days` days, with applied-recommendation markers. | Parameter | In | Required | Description | | --------- | ----- | -------- | --------------------------------------------------------- | | `days` | query | No | Window width in days (inclusive of today). Default: `90`. | *** Object Maintenance [#object-maintenance] Create Location [#create-location] `POST /api/v1/object-maintenance/locations` Register a new object maintenance location. **Request body:** `LocationCreate` List Locations [#list-locations] `GET /api/v1/object-maintenance/locations` List object maintenance locations. Get Location [#get-location] `GET /api/v1/object-maintenance/locations/{location_id}` Get a single object maintenance location. Update Location [#update-location] `PATCH /api/v1/object-maintenance/locations/{location_id}` Partial update of an object maintenance location. **Request body:** `LocationUpdate` Delete Location [#delete-location] `DELETE /api/v1/object-maintenance/locations/{location_id}` Delete an object maintenance location. Dependent prefixes are retained with `location_id = NULL` so history is preserved. Trigger Discovery [#trigger-discovery] `POST /api/v1/object-maintenance/locations/{location_id}/discover` Trigger an object maintenance discovery pass. Runs in the background. List Discovered Prefixes [#list-discovered-prefixes] `GET /api/v1/object-maintenance/locations/{location_id}/discovered-prefixes` List the latest discovery-run results for a location (scoped to the most recent run). List Managed Prefixes [#list-managed-prefixes] `GET /api/v1/object-maintenance/locations/{location_id}/managed-prefixes` List durable managed prefixes for a location. List Prefixes (Cross-Location) [#list-prefixes-cross-location] `GET /api/v1/object-maintenance/prefixes` Flat view of managed prefixes across all locations. | Parameter | In | Required | Description | | ------------------ | ----- | -------- | ---------------------------------------------------- | | `location_id` | query | No | Filter to a single location. | | `management_state` | query | No | Filter: `unmanaged`, `active`, `paused`, or `error`. | | `source_state` | query | No | Filter: `present` or `missing`. | Get Prefix [#get-prefix] `GET /api/v1/object-maintenance/prefixes/{prefix_id}` Get a single managed prefix with parent-location summary and policy details. Enable Maintenance [#enable-maintenance] `POST /api/v1/object-maintenance/locations/{location_id}/prefixes/{prefix_id}/enable` Enable maintenance on a prefix (`unmanaged → active`). Idempotent on already-active prefixes. | Response | Description | | -------- | ----------------------------- | | `200` | Already enabled — no-op. | | `201` | Newly enabled. | | `404` | Prefix or location not found. | | `422` | Invalid lifecycle transition. | Get Prefix Policy [#get-prefix-policy] `GET /api/v1/object-maintenance/prefixes/{prefix_id}/policy` Return the active policy and active version for a prefix. Upsert Prefix Policy [#upsert-prefix-policy] `PUT /api/v1/object-maintenance/prefixes/{prefix_id}/policy` Create or replace the active policy. Appends a new version and deactivates the prior one atomically. Auto-activates the prefix if still unmanaged. **Request body:** `PolicyUpsert` Delete Prefix Policy [#delete-prefix-policy] `DELETE /api/v1/object-maintenance/prefixes/{prefix_id}/policy` Drop the policy for a prefix. Cascades to policy versions. List Prefix Policy Versions [#list-prefix-policy-versions] `GET /api/v1/object-maintenance/prefixes/{prefix_id}/policy/versions` Full version history of the active policy for a prefix, newest-first. Trigger Prefix Action [#trigger-prefix-action] `POST /api/v1/object-maintenance/prefixes/{prefix_id}/actions` Trigger an ad-hoc action against a prefix. **Request body:** `ActionRequest` List Prefix Actions [#list-prefix-actions] `GET /api/v1/object-maintenance/prefixes/{prefix_id}/actions` List actions for a prefix, newest first. List Prefix Runs [#list-prefix-runs] `GET /api/v1/object-maintenance/prefixes/{prefix_id}/runs` List execution rows for a prefix (crunch jobs and maintenance jobs unified). Discovery Schedule Operations [#discovery-schedule-operations] | Endpoint | Description | | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | | `GET /api/v1/object-maintenance/locations/{location_id}/discovery/schedules` | List discovery-attempt history for a location. | | `GET /api/v1/object-maintenance/discovery/schedules/{schedule_id}` | Full detail for a discovery attempt. | | `GET /api/v1/object-maintenance/discovery/schedules/{schedule_id}/state` | Live progress snapshot (console polls at 5s cadence while running). | | `GET /api/v1/object-maintenance/discovery/schedules/{schedule_id}/plan` | Plan summary — bins, total bytes, estimated wall-time. | | `POST /api/v1/object-maintenance/discovery/schedules/{schedule_id}/pause` | Pause a running discovery. | | `POST /api/v1/object-maintenance/discovery/schedules/{schedule_id}/resume` | Resume a paused discovery. | | `POST /api/v1/object-maintenance/discovery/schedules/{schedule_id}/restart` | Restart a discovery from scratch (cancels prior run, clears state). | | `POST /api/v1/object-maintenance/discovery/schedules/{schedule_id}/complete` | Mark a discovery completed (dragon driver terminal callback). | | `POST /api/v1/object-maintenance/discovery/schedules/{schedule_id}/skip-bin/{bin_id}` | Skip a single wedged bin without restarting the full discovery. | | `GET /api/v1/object-maintenance/discovery/schedules/{schedule_id}/effective-config` | Three-layer config resolution for a schedule's parent location. | # Tour of the Granica Console The Granica Console is the central interface for discovering, onboarding, and managing the optimization of your lakehouse data. This tour covers the two primary workspaces: **Table Maintenance**, for tables registered in your data catalogs, and **Object Maintenance**, for raw object store prefixes that live outside any catalog. *** Table Maintenance [#table-maintenance] **Table Maintenance** is where you view all tables synced from your connected catalogs, configure optimization policies, monitor progress, and trigger one-time runs. See all tables, filter, search, and inspect metadata [#see-all-tables-filter-search-and-inspect-metadata] Granica Table Maintenance — table list with filters and status summary Navigate to **Table Maintenance** from the sidebar. The page shows all tables synced from your connected catalogs, with a summary of how many are **managed** (have an active policy) versus **unmanaged**. Only tables 0.1 GB and above are synced from the catalog. Smaller tables are intentionally excluded to focus optimization on high-impact workloads. If you notice missing tables, an Admin can adjust the threshold in [Platform Configuration](/administration/configure-crunch). Use the filters at the top to narrow the list: | Filter | Description | | -------------- | ------------------------------------------------------------------------- | | **Catalog** | Filter to tables from a specific connected catalog | | **Schema** | Narrow to a specific database or schema within the catalog | | **Status** | Show only Managed, Unmanaged, Active, or tables with a specific job state | | **Table Type** | Filter by format — Iceberg, Delta Lake, or Hive | | **Search** | Free-text search by table name | Each row in the table list shows the table name, its full catalog path, table type, current size, estimated DRR, optimization progress (partitions processed out of total), most recent activity timestamp, active policies, and the catalog source. Click any row to open the table detail page. Evaluate a table with "Collect Metadata" [#evaluate-a-table-with-collect-metadata] Before setting a policy on a new table, you can evaluate its optimization potential by collecting metadata. On the table detail page, click **Collect Metadata** to trigger a metadata collection job. Granica analyzes the table's files and partitions and computes the **Estimated DRR** (Data Reduction Ratio) — the projected percentage of storage that Crunch can save for this table. Once metadata collection completes, the Est. DRR column on the table list populates with the result. Use Est. DRR to prioritize which tables to onboard first. Tables with higher estimated DRR will yield the most immediate storage savings. Set policies to schedule compaction and compression [#set-policies-to-schedule-compaction-and-compression] Granica Table Maintenance — Crunch policy configuration panel On the table detail page, the **Policy** panel lets you configure automated optimization schedules. Policies control when Crunch runs, what partitions it processes, and which optimization primitives it applies. **To configure a Crunch policy:** 1. Open the **Crunch** policy section and set Status to **Enabled**. 2. Set the **Schedule** (Daily or Weekly) and **Run Time (UTC)** — when the job should start each day. 3. Set the **Partition Range** — which date partitions to process. For example, Start: 1 day before now and End: 1 day before now processes yesterday's partitions on each run. 4. Configure **Primitives** — the individual optimization operations to apply: * **Compression** — Recompresses files using Granica's adaptive compression engine for maximum data reduction. * **Deduplication** — Removes duplicate rows within a partition based on configured key columns. * **Optimization Type** — Controls compaction behavior. The **Compact** strategy uses Binpack compaction to consolidate small files into target-sized files. Configure the **Target file size** (default 128 MiB) and **Min file size** to skip already-efficient files. 5. Click **Save** to activate the policy. Once saved, Granica schedules the first run according to the configured time and partition range. Policy changes take effect on the next scheduled run. Additional policy types available on the table detail page: * **Vacuum** — Expire old snapshots and delete orphaned files to reclaim storage (Iceberg and Delta Lake). * **Partition Expiration** — Automatically drop partitions older than a configured retention window. Trigger a one-time run (backfill) [#trigger-a-one-time-run-backfill] To process a specific date range outside the normal schedule — for example, to backfill historical partitions or re-crunch a range after a policy change — use the **Actions** tab on the table detail page. Click **New Run**, select the job type (Crunch, Vacuum, or Partition Expiration), set the start and end dates for the partition range, and submit. The run is queued immediately and appears in the activity log. One-time runs are independent of the policy schedule and do not affect future scheduled runs. *** Query Acceleration [#query-acceleration] Query Acceleration is built directly into **Table Maintenance** — there is no separate page. When you connect your query engine's logs (see [Connect Query History](/administration/connect-query-history)), Granica profiles your real query patterns and surfaces workload-aware recommendations alongside each table. * **Est. time saved / mo** — a column on the table list estimating the monthly query time each table's recommendation would save, so you can prioritize the highest-impact tables. * **Query Acceleration recommendation card** — on a table's detail page, a card showing the suggested layout change (such as a sort key or Z-order over the most frequently queried columns) and its projected speedup. * **Applied indicator** — once a table's latest recommendation has been applied, the row shows an **Applied** pill so you can tell at a glance which tables are already optimized for their workload. Query Acceleration recommendations require a Query History connection. Without query history, Crunch still optimizes storage, but the Est. time saved column and recommendation cards remain empty. *** Object Maintenance [#object-maintenance] **Object Maintenance** surfaces the prefixes discovered from your registered [Object Locations](/administration/connect-object-stores) and applies the same optimization capabilities to data that lives outside your catalogs — raw JSON event files, unregistered Parquet dumps, and any object store prefixes not claimed by a connected catalog. Set policies on a prefix [#set-policies-on-a-prefix] Open a discovered prefix from the Object Maintenance page to access its policy configuration. The policy panel works the same way as Table Maintenance: * Enable **Crunch** and configure the schedule, run time, and partition range. * Select which primitives to apply: Compression, Deduplication, and compaction strategy. * Save the policy to start automated background optimization on the prefix. Policies on object prefixes follow the same scheduling model as table policies — Granica runs the configured job at the specified time and processes the configured partition range on each run. Trigger a one-time action [#trigger-a-one-time-action] The **Actions** tab on any object prefix detail page lets you trigger a one-time Crunch run on a specific date range, independent of the configured schedule. Set the start and end dates for the backfill range and submit — the run is queued immediately and tracked in the activity log alongside scheduled runs. # Configure Crunch **Platform Configuration** contains the system-wide knobs that govern how Granica Crunch operates across your entire deployment. Changes take effect immediately for all users. Only users with the Admin role can modify these settings. Navigate to **Settings → Platform Configuration** to access this page. Platform [#platform] General identity and catalog sync settings. | Setting | Default | Description | | -------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Company Name** | — | Your organization's name. Appears in report headers and PDF exports. | | **Min Eligible Table Size (GB)** | — | Tables smaller than this threshold are excluded when Granica syncs tables from a connected catalog. Raising this value focuses Crunch on high-impact tables and reduces noise from small datasets. | | **Quarterly Goal (PB)** | 0 | Target volume of crunched data for the current quarter, used in reporting dashboards. Set to 0 to hide the goal metric from reports. | Savings & Cost [#savings--cost] Parameters used to compute and project the storage savings Crunch delivers. Granica uses these values in the Cloud Savings report and all dollar-denominated savings estimates. | Setting | Default | Range | Description | | ------------------------------ | ------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Storage Cost ($/GB/month)** | 0.021 | 0.001–1 | Your base storage rate before any negotiated discount. The default matches AWS S3 Standard pricing. Adjust for GCP, Azure, or custom rates. | | **Cloud Discount (%)** | 0 | 0–90 | Your negotiated cloud discount (e.g. AWS EDP, GCP CUDs). Applied on top of the base rate. The effective rate shown on the page is `base_rate × (1 − discount%)`. | | **Projection Period (months)** | — | 1–60 | How many months forward the Cloud Savings report projects future savings based on the current compression ratio. | | **Operational Cost ($/PB)** | — | 0–100,000 | Heuristic Crunch operational cost per petabyte of processed data. Used in net savings calculations until direct cloud billing integration (AWS CUR / GCP Billing) is available. | Crunch Intelligence [#crunch-intelligence] Scheduler routing thresholds that determine how Granica distributes and routes crunch jobs. | Setting | Default | Range | Description | | ----------------------------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Hourly Fanout Min Partition Size (GB)** | — | 10–1,000 | When the average daily partition for a table exceeds this size, Granica splits the crunch workload into 24 per-hour jobs instead of a single daily job. This improves parallelism and throughput for large, frequently updated tables. Tables below this threshold use a single daily job. | | **Mutation Interval Threshold (minutes)** | — | 5–120 | The minimum quiet period between data writes required for Granica to route a table to a full crunch job rather than a lighter optimize pass. Lowering this value makes Crunch more aggressive — it will crunch tables even if they have been recently written to. Raising it reserves full crunch runs for tables with longer write-free windows. | Security and Authentication [#security-and-authentication] JWT token lifetime controls. Changes propagate to newly issued tokens within 60 seconds. **Refresh token TTL must always exceed the access token TTL** or users will be unable to silently renew their sessions. | Setting | Default | Range | Description | | --------------------------------------------- | ------------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Console Public URL** | *(request origin)* | — | The public URL at which users reach this Console (e.g. `https://console.example.com`). Pins all absolute URLs the Console emits, including the SAML SP entity ID, ACS URL, SAML metadata endpoint, OIDC post-logout return URL, and proxy redirects. Set this when the `CONSOLE_PUBLIC_URL` environment variable is not configured, or to ensure a stable URL across multiple ingresses or DNS changes. | | **Access Token TTL (seconds)** | 3,600 | 900–28,800 | How long a user's access token stays valid before it must be refreshed. Shorter values limit the exposure window if a token is compromised; longer values reduce the frequency of silent refresh calls. Default is 1 hour (3,600s). | | **Refresh Token TTL — SSO users (seconds)** | 7,200 | 3,600–28,800 | How long an SSO user's refresh token stays valid. When it expires, the user silently re-authenticates through their identity provider. Aligned with typical IdP idle session lengths. Default is 2 hours (7,200s). | | **Refresh Token TTL — local users (seconds)** | 604,800 | 3,600–2,592,000 | How long a local (password) or break-glass user's refresh token stays valid. The default is intentionally longer than SSO because local users must type a password on each re-authentication. Default is 7 days (604,800s). | Email Reports [#email-reports] Controls the theme and recipient list for Granica's scheduled email reports. This section appears only when email reporting is enabled for your deployment. | Setting | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Email Theme** | Visual theme for email reports: **Dark** (matches the Granica Console brand) or **Light**. | | **Daily Report Recipients** | Email addresses that receive the daily Platform Status report. Leave empty to send to all users with the Editor or Admin role. | | **Weekly Report Recipients** | Email addresses that receive the weekly Data Management report. Leave empty to send to all users with any role (Viewer, Editor, or Admin). | Saving changes [#saving-changes] Edit any field and click **Save Changes** in the top-right corner or at the bottom of the page. You can edit multiple settings across sections before saving — all pending changes are applied in a single save. Click **Reset** to discard unsaved edits and revert to the current saved values. Fields marked **Customized** have been changed from their factory default. The page shows who last updated each setting and when, along with the original default value for reference. # Connect Catalogs Catalog Connections is how Granica discovers the tables in your data lakehouse. Once a catalog is connected, Granica continuously syncs table metadata and makes those tables available for compaction, compression, vacuum, partition expiration, and query acceleration policies. Connecting a catalog does not automatically give Crunch access to the actual data. Customers need to work with their data platform and storage team to make sure that the Crunch IAM role (typically named something like `granica-worker-xyz`; find the exact role name in your Terraform output, see [Deploy the admin server](/installation/aws#3-deploy-the-admin-server)) has full read and write access, including listing and head operations, to the S3 bucket(s) associated with the catalog. Granica Catalog Connections — Add Connection dialog Supported catalogs [#supported-catalogs] Granica supports the following catalog types today, with more on the way. Unity Catalog (Databricks) [#unity-catalog-databricks] Connect Granica to a Databricks workspace via a service principal. Granica integrates with Unity Catalog to discover all catalogs, schemas, and tables in the workspace. **Required credentials:** | Field | Description | | ------------- | ---------------------------------------------------------------------------- | | Workspace URL | Your Databricks workspace URL, e.g. `https://workspace.cloud.databricks.com` | | Client ID | Service principal client ID | | Client Secret | Service principal client secret | After providing credentials, click **Test Connection** to discover available catalogs, then select the catalog you want to manage. Hive Metastore (HMS / Iceberg) [#hive-metastore-hms--iceberg] Connect Granica to a Hive Metastore for HMS-managed tables, including Apache Iceberg tables registered in HMS. This covers self-managed Hive deployments as well as managed HMS instances (e.g. Amazon EMR, CDH). HMS endpoint is often private to your network. Make sure Granica can reach it, and work with your network admin if needed. **Required credentials:** | Field | Description | | ------------- | ------------------------------------------------------------------- | | Metastore URI | Thrift URI of the Hive Metastore, e.g. `thrift://10.123.64.35:9083` | | Username | Metastore username (if authentication is enabled) | | Password | Metastore password (if authentication is enabled) | Polaris [#polaris] Granica supports Apache Polaris as a catalog source for Iceberg tables managed via the Polaris REST catalog API. **Required credentials:** | Field | Description | | ------------- | ----------------------------------- | | Catalog URL | Polaris REST catalog endpoint | | Client ID | OAuth2 client ID for Polaris access | | Client Secret | OAuth2 client secret | Coming soon [#coming-soon] Granica is actively expanding catalog support. **AWS Glue** is next on the roadmap, enabling discovery of tables registered in the AWS Glue Data Catalog across S3-backed lakehouses. Additional catalogs will follow. Add a connection [#add-a-connection] 1. Navigate to **Catalog Connections** in the Granica Console sidebar. 2. Click **+ Add Connection**. 3. Select your **Catalog Type** (Unity Catalog, Hive Metastore, or Polaris). 4. Enter a **Connection Name** — a friendly label used to identify this connection in the console. 5. Fill in the catalog-specific credentials (see above). 6. Click **Test Connection** to verify credentials and discover available catalogs. 7. Select the catalog to sync (Unity Catalog only — HMS syncs all available databases). 8. Click **Create Connection**. Granica immediately begins an initial metadata sync. Depending on the size of your catalog, this may take a few minutes. Table sync and eligibility [#table-sync-and-eligibility] Only tables **0.1 GB and above** are synced from the catalog by default. Smaller tables are excluded to focus optimization on high-impact workloads. If tables you expect to see are missing, ask your admin to adjust the threshold in [Platform Configuration](/administration/configure-crunch). Synced tables appear in **Table Maintenance**, where you can onboard them for automated optimization. Manage connections [#manage-connections] From the Catalog Connections page you can: * **Sync** — Trigger a manual metadata sync for a connection to pick up newly added tables. * **Edit** — Update connection credentials or settings. * **Delete** — Remove a connection. This cascades to all synced table metadata. Managed tables with active policies are flagged before deletion. Each connection shows its **sync status** (idle, running, completed, or failed), the **last synced** timestamp, and whether the connection is **Active** or **Inactive**. # Connect Object Stores **Object Locations** is where you register the cloud storage prefixes that Granica should track and optimize. While [Catalog Connections](/administration/connect-catalogs) handle tables registered in Unity Catalog, Hive Metastore, or Polaris, Object Locations covers everything else — raw JSON event files, unregistered Parquet dumps, archive prefixes, and any other data that lives in your object store but is not claimed by a catalog. Once a location is registered, Granica discovers the prefixes within it and makes them available for compression, compaction, vacuum, and orphan file deletion — the same operations it applies to catalog-managed tables. Object Locations is an experimental feature. The initial release supports JSON input (converted to Parquet on output). Parquet-to-Parquet optimization is coming soon. Granica Object Locations page showing registered locations and the Register Location dialog Location table [#location-table] The Object Locations page lists all registered locations with the following columns: | Column | Description | | ---------------------- | ------------------------------------------------------------------------ | | **Name** | Friendly label for the location | | **URI** | The cloud storage prefix being tracked (`s3://`, `gs://`, or `azure://`) | | **Method** | How Granica discovers prefixes within this location | | **Status** | Pending, Discovering, Active, or Archived | | **Surviving Prefixes** | Number of prefixes discovered and accepted after filtering | | **Last Discovery** | Timestamp of the most recent discovery run | Register a location [#register-a-location] Click **+ Register Location** to open the registration dialog. Discovery Method [#discovery-method] The discovery method controls how Granica enumerates the prefixes within your registered location. Use as Registered [#use-as-registered] Treats the URI you provide as a single, exact prefix with no further enumeration. Granica takes it at face value and begins optimizing objects immediately. Use this when you already know the precise prefix you want to manage. * No additional discovery runs are scheduled. * The URI field is required — provide the full `s3://`, `gs://`, or `azure://` path. Inventory Report [#inventory-report] Uses a pre-generated cloud storage inventory manifest (such as an [S3 Inventory](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html) report) to enumerate all objects under a bucket. Granica reads the manifest, filters out any prefixes already claimed by your catalogs or that contain files too young to touch, and produces a clean list of surviving prefixes to optimize. Inventory Report is the recommended method for large buckets. It is far more efficient than scanning the bucket directly — cloud-native inventory exports are pre-computed by the storage provider and avoid the per-object API calls that make direct listing slow and expensive. Inventory manifests registered here are also used by other Granica features, including vacuum and orphan file deletion, so a single inventory location serves multiple optimization workflows. **Additional fields for Inventory Report:** | Field | Description | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Inventory URI** | URI of the inventory manifest destination (e.g. `s3://inventory-bucket/path/`). Granica reads the manifest from this location. The source bucket URI is derived automatically from the manifest — you do not need to enter it separately. | | **Inventory Format** | Format of the manifest files. **Parquet** is supported today; ORC and CSV are coming soon. | | **Discovery Cadence** | How often Granica re-reads the manifest to surface newly appeared prefixes: **Daily** or **Weekly**. You can always trigger a manual discovery run regardless of cadence. | | **Skip Age (days)** | Files newer than this threshold are excluded from discovery to avoid interfering with active writes. Default is 2 days. Valid range: 0–365. | | **Run discovery immediately** | When enabled, Granica queues a discovery run as soon as the location is created. Defaults to on for Inventory Report so you can validate the manifest end-to-end right away. | Listing (coming soon) [#listing-coming-soon] Direct bucket listing via cloud storage APIs. This method will enumerate all objects by scanning the bucket in real time. It is simpler to configure than Inventory Report but significantly slower and more expensive for large buckets. Recommended only for small prefixes or when an inventory export is not available. Name and Description [#name-and-description] Give the location a descriptive **Name** (up to 100 characters) that identifies its purpose, such as `raw-events-prod` or `archive-data-us-east`. An optional **Description** (up to 500 characters) is useful for documenting what data lives under the prefix. Input Format [#input-format] Specifies the file format of objects in the registered location. | Format | Status | | ----------- | ------------------------------------------------------------------- | | **JSON** | Supported — Granica reads JSON files and converts output to Parquet | | **Parquet** | Coming soon | Output is always Parquet regardless of input format. Register Location [#register-location] Click **Register Location** to save. If **Run discovery immediately** is enabled, Granica starts a discovery run in the background. The location status changes from **Pending** to **Discovering** and then to **Active** once the first discovery run completes and surviving prefixes are identified. Manage locations [#manage-locations] Click any row in the location table to open a details panel showing the full configuration, discovered prefixes, and discovery run history. From the **⋯** Actions menu on any row you can: * **Edit** — Update the location name, description, or cadence. * **Run Discovery** — Trigger an immediate discovery run outside the scheduled cadence. * **Archive** — Stop tracking and optimizing this location. Archiving does not delete any data. Archived locations can be viewed but are no longer processed. How discovery filtering works [#how-discovery-filtering-works] When Granica processes an inventory manifest or listing, it applies several filters before accepting a prefix as a surviving location: | Filter | Reason | | ------------------------ | --------------------------------------------------------------------------------------------------------------------- | | **Catalog claimed** | Prefixes already managed by a connected catalog are excluded — they are handled through table-level policies instead. | | **Too young** | Files newer than the Skip Age threshold are excluded to avoid touching data that is still being actively written. | | **Empty** | Prefixes containing no data files are skipped. | | **Non-data files** | Prefixes containing only metadata, manifest, or other non-data files are excluded. | | **Below size threshold** | Very small prefixes below the minimum optimization threshold are skipped. | Only prefixes that pass all filters appear as **surviving prefixes** and are queued for optimization. # Connect Query History Query History Connections tell Granica where to find your engine's query logs. Granica reads those logs daily, analyzes query patterns, and surfaces [Query Acceleration](/crunch/overview) recommendations — such as optimal sort keys, Z-ordering, and file right-sizing — for your most frequently queried tables. Each connection targets a single query-log prefix (S3, GCS, or ABFS) and runs once per UTC day at a configured time. An initial sync runs immediately when a connection is created. Granica Query History Connections — connection list and Add Connection dialog Query History Connections require Admin access. Viewers and Editors cannot create or manage connections. Supported engines [#supported-engines] | Engine | Log format | Notes | | ---------- | ------------------------------ | -------------------------------------------------------------------------------------- | | **Trino** | Trino event listener logs | Point to the root prefix where Trino writes its query event JSON files | | **Spark** | Spark event logs | Point to the Spark History Server log directory (e.g. the `eventlog` prefix on S3/GCS) | | **Athena** | Athena query execution history | Point to the S3 prefix where Athena writes query results and execution metadata | Add a connection [#add-a-connection] 1. Navigate to **Query History Connections** in the Granica Console sidebar. 2. Click **Add Connection**. 3. Fill in the connection form: | Field | Description | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | Optional label shown in the connections table (e.g. `prod-trino-logs`). If left blank, Granica uses the last segment of the log prefix URI. | | **Engine** | The query engine that produced the logs: Trino, Spark, or Athena. | | **Query logs prefix** | S3, GCS, or ABFS URI pointing to the root of the engine's query log output (e.g. `s3://my-bucket/trino-logs/`). | | **Lookback (days)** | How many days of query logs to scan on each daily run. Range: 1–365 days. Default: 7 days. | | **Daily sync time (UTC)** | The UTC time at which the daily sync should run (e.g. `06:00`). The scheduler checks every 15 minutes and fires the sync the first time it ticks past this time. | 4. Click **Create & sync now**. Granica immediately starts an initial sync regardless of the configured daily sync time. Subsequent syncs run daily at the configured UTC time. Monitor connections [#monitor-connections] The connections table shows the status of each configured connection: | Column | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------- | | **Connection** | The connection name (or log prefix if no name was set) and the full URI | | **Engine** | The query engine type (Trino, Spark, or Athena) | | **Lookback** | The number of days of logs scanned per run | | **Daily sync (UTC)** | The scheduled time for the daily sync | | **Last sync** | Status of the most recent run: Never synced, Syncing, Succeeded, or Failed (hover over Failed for the error detail) | | **State** | Active or Paused | While a sync is in progress, the Last sync column shows **Syncing** with a spinner. The table auto-refreshes every 5 seconds when any sync is running. Manage connections [#manage-connections] Click the **⋯** menu on any connection row to access management options: * **Edit** — Update the name, log prefix, engine type, lookback window, or sync time. Changes take effect on the next sweep tick. * **Pause / Resume** — Suspend daily syncs without deleting the connection. Paused connections are dimmed in the table and cannot be manually synced. * **Delete** — Permanently remove the connection and all associated historical recommendations. This cannot be undone. To trigger an immediate sync outside of the scheduled time, click the **sync** (↻) icon on any active connection row. Log prefix requirements [#log-prefix-requirements] Granica needs read access to the configured log prefix in your cloud storage. Ensure the IAM role or service account used by Granica's data plane has `s3:GetObject` / `storage.objects.get` permissions on the log bucket. Trino [#trino] Configure Trino's [event listener](https://trino.io/docs/current/develop/event-listener.html) to write query events to a cloud storage prefix. Point the connection at the root of that prefix. Spark [#spark] Set `spark.eventLog.enabled=true` and `spark.eventLog.dir=s3://your-bucket/spark-logs/` in your Spark configuration. Point the connection at the same `spark.eventLog.dir` prefix. Athena [#athena] Athena writes query execution metadata to the **Query result location** configured in your workgroup settings. Point the connection at that S3 prefix (e.g. `s3://your-bucket/athena-results/`). See also [#see-also] # Manage Users The **User Management** page is where Admins create user accounts, assign roles, and control access to the Granica Console. It is available under **Settings → User Management** and is only accessible to users with the Admin role. Granica User Management page showing user list and Add User dialog User table [#user-table] The user list displays all accounts in your Granica deployment. Each row shows: | Column | Description | | -------------- | ------------------------------------------------------- | | **User** | Full name and email address | | **Username** | Login identifier | | **Role** | Assigned role: Viewer, Editor, or Admin | | **Status** | Active or Inactive | | **Auth Type** | How the user authenticates: Password, OIDC, or SAML 2.0 | | **Last Login** | Timestamp of the most recent successful sign-in | Add a user [#add-a-user] Click **+ Add User** in the top-right corner to open the Add User dialog. There are two account types depending on how the user will authenticate. SSO users (identity provider) [#sso-users-identity-provider] Choose **SSO** when the user will sign in via your organization's identity provider (Okta, Microsoft Entra ID, Google Workspace, etc.). SSO must be configured before creating SSO accounts — see [SSO Integration](/security-and-compliance/sso-integration). 1. Select **SSO** as the authentication type. 2. Choose the protocol: **OIDC** or **SAML 2.0** (the protocol your IdP is configured with). 3. Enter the user's **Full Name**. 4. Enter the user's **Email** address. This must exactly match the email address that the identity provider asserts during login. The username is automatically derived from the email local-part. 5. Select a **Role** (Viewer, Editor, or Admin). 6. Click **Create User**. The user can immediately sign in via your identity provider using that email address. No password is set or required. Bulk add SSO users from a CSV [#bulk-add-sso-users-from-a-csv] When SSO is configured, the Add User dialog shows an **Add One** / **Upload CSV** toggle. Use **Upload CSV** to create many SSO accounts at once instead of adding them one at a time. CSV upload is only available for SSO accounts. It is not offered for password accounts, so the toggle does not appear when the authentication type is set to **Password**. 1. Select **SSO** as the authentication type and choose the protocol (**OIDC** or **SAML 2.0**). The selected protocol applies to every user in the file. 2. Click **Upload CSV**. 3. Select a **Role**. The chosen role is applied to all users in the file. 4. Click the upload area and choose your `.csv` file. 5. Click **Create User** to process the file. **CSV format** — the first row must be a header row containing `full_name` and `email` columns (column order does not matter, and other columns are ignored): ```csv full_name,email Jane Smith,jane@company.com Ravi Patel,ravi@company.com ``` For each row, the username is automatically derived from the email local-part, and the account is created with the role and protocol selected in the dialog. After processing, the dialog reports how many accounts were added and lists any rows that failed with the reason (for example, a duplicate email). Successful rows are still created even when some rows in the same file fail, so you can fix the listed rows and re-upload just those. Password users (local credentials) [#password-users-local-credentials] Choose **Password** when the user will sign in with a username and password managed directly by Granica. This is useful for service accounts, break-glass users, or environments without SSO. 1. Select **Password** as the authentication type. 2. Enter the user's **Full Name**. 3. Enter a **Username** — this is the login identifier and cannot be changed after creation. 4. Enter the user's **Email** address. 5. Enter an initial **Password**. The user can change it after their first login. 6. Select a **Role** (Viewer, Editor, or Admin). 7. Click **Create User**. When SSO is configured, the dialog defaults to SSO authentication. Password authentication remains available as a fallback and is the only option when no SSO protocol has been configured. Assign and change roles [#assign-and-change-roles] The user's role is set when the account is created. To change it later: 1. Find the user in the user table. 2. Click the role badge (e.g., **Viewer**) in the Role column. 3. Select the new role from the dropdown. The change takes effect immediately on the user's next request. See [Role-Based Access Control](/security-and-compliance/role-based-access-control) for a full description of what each role can do. Edit a user [#edit-a-user] Click the **⋯** (Actions) menu on any user row to access management options: | Action | Description | | ------------------------- | --------------------------------------------------------------------------------------------- | | **Edit** | Update the user's full name, email, or role | | **Reset Password** | Set a new password (password accounts only) | | **Deactivate / Activate** | Toggle the user's active status. Deactivated users cannot log in but the account is retained. | | **Delete** | Permanently remove the user account. This action cannot be undone. | Deleting a user is permanent. If you need to temporarily revoke access, use **Deactivate** instead — the account and its history are preserved and can be reactivated later. Protected accounts [#protected-accounts] The built-in `admin` and `granica-admin` accounts are marked as **protected** and cannot be deleted or deactivated through the UI. The `granica-admin` account serves as a break-glass login and is exempt from SSO enforcement. See [SSO Integration](/security-and-compliance/sso-integration#break-glass-access) for details. # Monitoring Platform The Granica Console provides four dedicated monitoring surfaces that give operators, data engineers, and executives a complete picture of optimization activity, platform health, and business value. Each dashboard is updated in near real-time and can be filtered by time window, table type, version, or bucket. Granica Overview dashboard showing KPI cards, Daily Bytes Processed chart, and DRR Over Time chart Overview [#overview] **URL:** `/` (Console home) The Overview dashboard is the primary operational view. It gives a live, at-a-glance summary of how much data Granica has processed and the compression efficiency achieved. All-time KPI cards [#all-time-kpi-cards] The four cards at the top of the page reflect all-time cumulative performance, independent of any date filters: | Metric | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------- | | **Data Processed** | Total volume of data read by Granica across all crunch and optimize jobs | | **After Optimization** | Total volume of data written after optimization — the smaller the gap with Data Processed, the higher the reduction | | **DRR** | Data Reduction Ratio — the percentage of storage saved relative to the original size. Higher is better. | | **Tables Processed** | Total number of distinct tables that have been optimized at least once | Filters [#filters] Below the KPI cards, a filter bar lets you slice the detailed metrics and charts by: * **Time window** — 24h, 7d, 30d, or 90d * **Job type** — All, Crunch, or Optimize * **Version** — Iceberg snapshot or Delta commit version * **Table Type** — Iceberg, Delta Lake, or Hive * **Bucket** — Filter to a specific cloud storage bucket * **Table name** — Free-text search Charts [#charts] | Chart | Description | | ------------------------- | ------------------------------------------------------------------------------- | | **Daily Bytes Processed** | Bar chart showing data volume processed per day within the selected time window | | **DRR (%) Over Time** | Line chart showing how the Data Reduction Ratio has trended day-by-day | Breakdown tables [#breakdown-tables] Below the charts, the Overview page breaks down metrics by four dimensions — switch between tabs to explore: | Tab | What it shows | | ---------------- | ------------------------------------------------------------------------ | | **By Table** | Per-table before/after size, DRR, row count match, and file count change | | **By Bucket** | Aggregated metrics grouped by storage bucket | | **By Partition** | Per-partition breakdown for tables with date or Hive partitioning | | **By Spark App** | Metrics grouped by the Spark application ID that performed the work | Each row shows **Before** and **After** sizes, the **DRR (%)**, a row count **Match** indicator confirming data integrity was preserved, and a **Files** delta showing how many files were consolidated. Attention alerts [#attention-alerts] A yellow alert banner appears at the top of the Overview when optimization runs have failed or require attention. Clicking **View details** jumps directly to the affected jobs in the Activities view. *** Activities [#activities] **URL:** `/activities` The Activities page is the job-level operational log. It shows every crunch and metadata collection job Granica has run, with filtering and search so you can quickly find jobs for a specific table or investigate failures. Tabs [#tabs] | Tab | Description | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Summary** | Aggregate view combining all job types — total jobs, in-progress, completed, failed, and skipped counts, plus average DRR and average job duration. Includes trend charts for DRR, duration, and daily job volume. | | **Crunch Jobs** | Spark-level job detail table. Each row shows the table name, job type (Crunch or Optimize), source (Background or Runtime), before/after row counts, file consolidation, and status. Click a row to open the Spark History Server detail panel for the underlying execution. | | **Metadata Collection** | Table showing metadata collection runs — the lightweight discovery scans that identify which partitions are candidates for crunching. | Filters and search [#filters-and-search] Use the filter bar to narrow results by time window, version, table type, or bucket. Type in the **Search table name** box to filter all tabs to a specific table. Filters persist across tab switches within the Activities page. *** Data Management [#data-management] **URL:** `/reports/data-management` The Data Management report is a weekly executive summary that tracks your organization's optimization goals, quarter-to-date progress, ROI, and the universe of tables eligible for optimization. It can be previewed and sent as a weekly email report. Quarter-to-date progress [#quarter-to-date-progress] If a quarterly goal (in PB) is configured in [Platform Configuration](/administration/configure-crunch), a progress bar shows how much of the goal has been achieved so far this quarter. KPI cards [#kpi-cards] | Metric | Description | | ------------------------- | ------------------------------------------------------------------------------------ | | **Tables Crunched (QTD)** | Number of distinct tables processed since the start of the current quarter | | **Bytes Crunched (QTD)** | Total data volume processed quarter-to-date | | **Optimization Tables** | Number of tables currently available for optimization | | **Forecasted Savings** | Projected annual storage cost savings based on current DRR and storage cost settings | Charts and breakdowns [#charts-and-breakdowns] | Section | Description | | ------------------------------ | ------------------------------------------------------------------------------------------------- | | **Tables & Bytes Crunched** | Time-series chart tracking cumulative quarter-to-date volume | | **ROI Metrics** | Return-on-investment breakdown: gross storage savings, operational cost estimate, and net savings | | **Optimization Opportunities** | Table-level breakdown of eligible tables, their size, and forecasted annual savings if onboarded | *** Cloud Savings [#cloud-savings] **URL:** `/reports/cloud-savings` The Cloud Savings report quantifies the dollar value of storage savings Granica has delivered and projects future savings over your chosen time horizon. Storage cost and discount rates are configured in [Platform Configuration](/administration/configure-crunch#savings--cost). KPI cards [#kpi-cards-1] | Metric | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------ | | **Data Processed** | Total pre-crunch data volume — the baseline that savings are measured against | | **Total Saved** | Total storage volume eliminated across all time | | **DRR** | Cumulative Data Reduction Ratio | | **Current Month** | Dollar savings accrued in the current calendar month | | **Operational Cost** | Heuristic estimate of Granica's processing cost per PB (until direct cloud billing integration is available) | | **Net Savings** | Total savings minus estimated operational cost | Projection period [#projection-period] Select a projection window of **3, 6, 12, or 24 months**. Granica extrapolates savings forward from the current DRR and your configured storage rate, marking projected months visually in the Monthly Savings chart. Charts and breakdowns [#charts-and-breakdowns-1] | Section | Description | | ------------------------------ | ----------------------------------------------------------------------------------------- | | **Monthly Savings** | Bar chart showing actual savings per month with projected months shown in a lighter color | | **Savings by Table** | Per-table breakdown of total storage saved and the DRR for each table | | **How Savings Are Calculated** | Expandable explanation of the savings formula and what the projection assumes | *** Email reports [#email-reports] The **Platform Status** and **Data Management** reports can each be sent as scheduled emails to specific recipients or to all users by role. Configure recipients and the email theme under [Platform Configuration → Email Reports](/administration/configure-crunch#email-reports). | Report | Frequency | Default recipients | | --------------- | --------- | --------------------------------- | | Platform Status | Daily | All Editor and Admin users | | Data Management | Weekly | All users (Viewer, Editor, Admin) | Each report page has a **Preview Email** button to review the email layout before it goes out. # Use Platform Tools The **Platform Tools** section of the sidebar provides direct, authenticated access to four backend services embedded within the Granica Console. All four tools are proxied through the Console's authentication layer — you do not need separate credentials to access them. | Tool | Access | Roles | | ----------------- | ------------------- | ------------- | | **Granica API** | `/granica-api/docs` | Admin only | | **Grafana** | `/grafana/` | Admin, Editor | | **Airflow** | `/airflow/` | Admin only | | **Spark History** | `/sparkhistory/` | Admin, Editor | *** Granica API [#granica-api] **Access level: Admin only** The Granica API tool opens the interactive Swagger/ReDoc documentation for the Granica REST API, served directly from the backend. From this interface, Admins can browse every available endpoint, inspect request and response schemas, and execute API calls live against your Granica deployment. **Exercise extreme caution.** The Granica API gives direct access to every operation the platform can perform — including creating, modifying, and deleting tables, triggering crunch and vacuum jobs, and changing policies. Mutating operations (POST, PUT, PATCH, DELETE) execute immediately against your live data. There is no undo. Use this tool only when you need low-level access that the Console UI does not expose, and prefer read-only GET requests for exploration. **Access control:** * `GET` requests (read) — available to all authenticated users * `POST`, `PUT`, `PATCH`, `DELETE` requests (write) — Editors and Admins only * `/docs`, `/redoc`, `/openapi.json` (API documentation UI) — Admins only API keys with explicit scopes are also enforced here — a key with only `tables:read` cannot trigger a crunch job even through the direct API. See [API Keys](/security-and-compliance/api-token) for scope definitions. For the full API reference, see [Granica APIs V1](/api-reference). *** Grafana [#grafana] **Access level: Admin, Editor** Grafana provides deep operational dashboards for the Granica platform infrastructure — cluster health, job throughput, resource utilization, error rates, and latency metrics collected from Granica's internal telemetry. Grafana is a read-only observability tool. It surfaces metrics from the Granica data plane running in your cloud environment and is the recommended tool for: * Investigating performance anomalies or slow crunch jobs * Monitoring cluster resource utilization (CPU, memory, disk I/O) * Tracking job queue depth and worker availability * Setting up alert rules for infrastructure health Your Granica Console role maps directly to a Grafana org role: Admins get Grafana Admin access, Editors get Grafana Editor access, and Viewers (who cannot access Grafana through the Console) would get Viewer access if granted. No separate Grafana login is required. *** Airflow [#airflow] **Access level: Admin only** Apache Airflow is Granica's workflow scheduler. It orchestrates the DAGs (Directed Acyclic Graphs) that drive background crunch jobs, metadata collection, catalog syncs, and other scheduled platform operations. **Exercise extreme caution.** Airflow gives direct control over the pipelines that process and write your data. Manually triggering, pausing, or clearing DAG runs can cause duplicate processing, missed optimizations, or unintended writes to your datasets. Only interact with Airflow if you have been specifically instructed to do so by Granica support, or if you fully understand the consequences of the operation. Common legitimate use cases for accessing Airflow: * Checking the status of a stuck or delayed DAG run during troubleshooting * Reviewing task logs to diagnose a pipeline failure (read-only) * Coordinating with Granica support on a specific DAG Your Console Admin role maps to Airflow's `Admin` FAB role, which grants full access to all DAGs and operations. Avoid triggering, re-running, or clearing tasks unless directed. *** Spark History [#spark-history] **Access level: Admin, Editor** The Spark History Server (SHS) provides a detailed execution log for every Spark job that Granica has run — including all crunch, optimize, and vacuum jobs. It is the primary tool for diagnosing individual job performance and failures at the Spark level. The Spark History Server is **read-only** — it provides historical execution data and cannot trigger any new operations. It is safe to explore freely. Use the Spark History Server to: * Inspect the execution plan and stage breakdown for a specific crunch job * Identify data skew, spill, or shuffle bottlenecks in a slow job * View detailed task-level timing and I/O metrics * Cross-reference a job with the Activities dashboard using the Spark App ID From the [Activities](/administration/monitoring-platform#activities) dashboard, clicking on a Crunch Jobs row opens the Spark History Server detail panel for that job directly within the Console, so you rarely need to navigate to `/sparkhistory/` independently. *** Accessing platform tools [#accessing-platform-tools] Platform tools are accessible from the **Platform Tools** section at the bottom of the Console sidebar. Each tool can be opened embedded within the Console or in a new browser tab using the external link icon. If a tool is not visible in your sidebar, either your role does not grant access or the tool is not enabled in your deployment. Contact your Granica administrator to request access. # Architecture Granica Crunch is built on a two-plane architecture that separates orchestration from data processing. Understanding this separation is important for evaluating security, data residency, and operational boundaries. Control plane and data plane [#control-plane-and-data-plane] **Data plane** — The data plane runs in your cloud environment and is responsible for all actual data processing. This includes the Spark clusters that read, optimize, and write your Parquet and Iceberg files. Your data is processed in place inside your cloud; it does not need to travel anywhere for Crunch to work. **Control plane** — The control plane hosts the Granica Console, API, Airflow scheduler, and PostgreSQL state store. Depending on your [deployment model](/installation/deployment-models), the control plane runs either in Granica's cloud (Hybrid) or entirely within your cloud (On-Premises). The two planes communicate through a secure tunnel maintained by the Tunnel Agent running in your data plane. Granica uses this tunnel for job scheduling, operational access, and software upgrades — no inbound network access to your environment is required. Deployment models [#deployment-models] Granica supports three deployment models. The right choice depends on your data residency, compliance, and operational requirements. See [Deployment Models](/installation/deployment-models) for a detailed comparison. | Model | Control plane location | Data plane location | Table data leaves your cloud? | | ------------------ | ---------------------- | ------------------- | ------------------------------ | | **Granica Hosted** | Granica's cloud | Granica's cloud | Yes | | **Hybrid** | Granica's cloud | Your cloud | No — only metadata and metrics | | **On-Premises** | Your cloud | Your cloud | No | Hybrid (recommended) [#hybrid-recommended] In the Hybrid model, the control plane (Console, API, Airflow, PostgreSQL) runs in Granica's cloud. The data plane (Spark, Tunnel Agent, Granica Worker) runs inside your cloud environment. This is the most common deployment model. What leaves your cloud in this model: * Spark job progress metrics (e.g. completed task counts, durations) * Job status signals (e.g. RUNNING, FAILED) * Aggregated table metadata used for reporting (e.g. table names, partition counts, DRR) **Your actual table data never leaves your cloud.** On-Premises [#on-premises] In the On-Prem model, both the control plane and the data plane run entirely within your cloud environment. This model is designed for customers with strict data residency or compliance requirements where no traffic of any kind can cross cloud boundaries. Granica Hosted [#granica-hosted] In the Granica Hosted model, Granica manages the entire platform including the data processing infrastructure. Table data and catalog metadata flow in and out of your cloud as part of Crunch operations. This model requires the least customer infrastructure but involves data leaving your cloud environment. Key architectural properties [#key-architectural-properties] Data plane isolation [#data-plane-isolation] The Spark workers that process your data run in your cloud VPC. They read from and write to your own cloud storage (S3, GCS, Azure Blob). At no point do they transmit file contents outside your environment. The Tunnel Agent only carries control signals and metadata — never file data. Single-tenant data plane [#single-tenant-data-plane] Each customer's data plane is a dedicated deployment — a single-tenant EKS cluster in your cloud account or project. Granica does not run multi-tenant compute for data processing. Your data is never co-mingled with another customer's data on shared infrastructure. Components and data flow [#components-and-data-flow] Object data integrity Crunch Data integrity [#crunch-data-integrity] Granica implements multiple levels of data integrity to ensure your data is **always** protected during optimization. Object integrity [#object-integrity] Object data integrity 1. **Pre-Crunch file validation.** Before crunching, Granica reads the source file to verify it is consistent with the native format — for example, confirming a Parquet file is structurally valid before processing begins. 2. **Post-Crunch integrity validation.** Immediately after a Crunch job completes, Granica performs logical data validation by comparing the source and optimized output, and verifying row counts match. Integrity failure handling [#integrity-failure-handling] In the unlikely event of an integrity failure, Crunch stops processing new objects and the Granica team is alerted immediately. Processing resumes only after the failure is investigated and resolved. High availability [#high-availability] Granica provides >99.99% availability, built on cloud-native primitives such as AWS EKS with multi-AZ node groups. High availability All Crunch services run as Kubernetes pods across a cluster of compute instances. A minimum two-node on-demand cluster ensures baseline availability. As workload increases, Granica automatically provisions additional spot instances and scales service pods to match. Pods are distributed across nodes and availability zones using a Broker pod that manages routing to distributed service pods. Elastic scaling [#elastic-scaling] Granica Crunch is a background service and is **not** in the read or write path of your query engines. It operates independently, reading from and writing to cloud storage without affecting query latency. Elastic Scaling Compute resources are fully elastic. Crunch uses autoscaling Kubernetes clusters that scale from zero to as many nodes as needed based on the volume of data queued for optimization, and back to zero during idle periods. Processing throughput reaches 150 MBps per node. Non-disruptive upgrades [#non-disruptive-upgrades] Granica upgrades are transparent to your applications and do not require downtime or application changes. Non-disruptive upgrades Granica uses a rolling upgrade approach across all service pods, containers, and Kubernetes cluster infrastructure. In the Hybrid deployment model, Granica can perform upgrades through the control plane tunnel without requiring direct access to your cloud environment. On-Premises customers can trigger upgrades manually or configure automatic updates. See also [#see-also] # Crunch compatibility Granica Crunch supports a range of platforms, file formats, and cloud providers. This page details what's compatible today and what's coming soon. Supported formats [#supported-formats] | Cloud | Storage | Format | Versions / notes | | -------------- | ------------------------ | ------------------ | -------------------------------------------------- | | **AWS** | Amazon S3 | Delta Lake | 3.2.x, 3.3.x, 4.1.x · Unity | | **AWS** | Amazon S3 | Apache Iceberg | 1.6.1, 1.10.1 · HMS, Polaris, more catalogs coming | | **AWS** | Amazon S3 | Apache Parquet | 1.13.x, 1.15.x · 1.16.x coming soon | | **AWS** | Amazon S3 | Hive | External tables · Managed tables | | **GCP** | Google Cloud Storage | Delta Lake | 3.2.x, 3.3.x, 4.1.x · Unity | | **GCP** | Google Cloud Storage | Apache Iceberg | 1.6.1, 1.10.1 · HMS, Polaris, more catalogs coming | | **GCP** | Google Cloud Storage | Apache Parquet | 1.13.x, 1.15.x · 1.16.x coming soon | | **GCP** | Google Cloud Storage | Hive | External tables · Managed tables | | **Azure** | Blob Storage · ADLS Gen2 | Delta Lake | 3.2.x, 3.3.x, 4.1.x · Unity | | **Azure** | Blob Storage · ADLS Gen2 | Apache Iceberg | 1.6.1, 1.10.1 · HMS, Polaris, more catalogs coming | | **Azure** | Blob Storage · ADLS Gen2 | Apache Parquet | 1.13.x, 1.15.x · 1.16.x coming soon | | **Azure** | Blob Storage · ADLS Gen2 | Hive | External tables · Managed tables | | **All clouds** | S3, GCS, ADLS Gen2 | Object maintenance | Non-ACID, PB-scale object stores | Runtime Crunch compatibility [#runtime-crunch-compatibility] Runtime Crunch currently works with **Apache Spark**. Support for additional engines is planned: * **Apache Spark** — V3.5.x supported, EMR 7.x · Spark V4 and EMR 8.x coming soon * **Apache Flink** — Coming soon * **Trino** — Coming soon Background Crunch compatibility [#background-crunch-compatibility] Background Crunch currently works with **Apache Spark**. Support for additional engines is planned: * **Apache Spark** — V3.5.x supported, EMR 7.x · Spark V4 and EMR 8.x coming soon * **Apache Flink** — Coming soon * **Trino** — Coming soon Crunch maintains full data integrity and format compliance. Crunched files remain valid Parquet files that any standards-compliant reader can process without modification. # How crunching works Once you've deployed Granica into your cloud environment, it's time to get crunching. "Crunching" is our euphemism for data processing in the context of compression optimization, where we "crunch" the data down to its purest information-rich state. Crunch is initiated by setting a policy on a table through the Granica Console or directly through the [Granica API](/api-reference). Once a policy is active, Crunch runs continuously in the background without any further manual steps. Production lakehouse data [#production-lakehouse-data] Production lakehouse data is the columnar data your teams are generating and working with every day. Crunch offers two mechanisms to compress and optimize that data: | | Runtime Crunch | Background Crunch | | ------------------------------------ | ------------------------------------------ | ------------------------------------------------------------------------------------------------- | | Optimizes incoming data (as written) | Yes | No | | Optimizes existing data | No | Yes | | Continuously learns from data | Yes | No | | Compatibility | Apache Spark (Flink and Trino coming soon) | Apache Spark on EMR, Dataproc, HDInsight, and self-managed clusters (Flink and Trino coming soon) | | Availability | Early access | Now | Crunch lexicon [#crunch-lexicon] * **Crunched buckets** — those under active management, processing and monitoring by Crunch * **Crunched objects** — objects evaluated by Crunch for compression optimization * **Vanilla buckets** — those which have not been crunched * **Vanilla objects** — those which have not been crunched * **Ingested** objects — crunched and reduced (background mode only) * **Analyzed** objects — crunched and analyzed to generate optimal recipes (runtime mode only) Background crunch write workflow [#background-crunch-write-workflow] This is the primary mode the Crunch works. It silently optimize tables at rest and carry out maintainece activities on the background without users even noticing. Once your data is crunched you'll see immediate savings in your lakehouse storage costs. Background crunch workflow **1.** An administrator sets a Crunch policy on a table via the Granica Console or API. The Controller retrieves vanilla objects. **2.** The Crunch scheduler determines when to process the table based on the configured SLA and policy. **3.** The Controller sends vanilla objects to a load balanced Compression optimizer. **4.** The Compression optimizer validates policy eligibility, optimizes the compression and encoding, and commits the optimized files using the table format's native commit protocol (Iceberg or Delta Lake snapshot commit, or direct object replacement for Hive) — initiating a reduction in your monthly cloud storage bill. **Crunch is not in the read path** in background mode either. Crunch swaps the original files with smaller, compression-optimized versions. In case of Iceberg and Delta Tables, Crunch commit the smaller and opitmized files into a new snapshot/version. Compatible applications will then begin reading the reduced files normally. Runtime crunch write workflow [#runtime-crunch-write-workflow] In some rare cases, customers may choose to integrate Crunch into their existing ETL engine. In this mode Crunch has two main components: 1. An **ML-powered adaptive compression control system** which analyzes your existing columnar files to create compression optimization recipes. 2. A **runtime optimizer JAR** which integrates into your data platform and is invoked transparently by any applications utilizing an open source Apache Parquet writer, without any code changes. Runtime crunch workflow **1.** An administrator routes vanilla objects to a Compression recipe generator, which analyzes the unique characteristics and structure of the columnar files. **2.** The Compression recipe generator updates the Compression recipe store with new or updated recipes. **3.** Spark-based applications initiate columnar writes using standard commands. The Granica runtime JAR intercepts the write and applies the best available recipe. **4.** The JAR writes data out in standard, lakehouse-native format (typically Parquet). **5.** When needed — for example, when a schema change is detected — the runtime JAR notifies administrator to re-analyze newly created files and update the recipe. **Crunch is not in the read path.** Reading Granica Crunch compressed files is transparent — any application using the open source Parquet reader can read them normally. See also [#see-also] # Granica Crunch overview **Granica Crunch** is a continuous, policy-driven data lakehouse optimization platform that automates compaction, compression, vacuuming, partition lifecycle management, and query acceleration across Hive, Apache Iceberg, Delta Lake tables, and raw files and objects — at a scale that in-house tools and generic schedulers cannot reach. The problem [#the-problem] Data platform teams managing thousands of tables face millions of optimization decisions every day: which files to compact, how aggressively to compress, when to expire snapshots, how to sort data for real query patterns, and how to safely delete PII to meet compliance deadlines. These decisions are either left undone — causing runaway storage costs, scan bloat, and query degradation — or handled by brittle in-house pipelines that break under scale. DAG based schedulers such as Airflow treat all jobs equally, have no awareness of SLAs or resource contention that are unqiue to table mainainence activities, and turn cascading failures into retry spirals and wasteful orphaned files. How Granica Crunch helps [#how-granica-crunch-helps] Crunch combines a central scheduling engine with per-file optimization tracking, intelligent compute sizing, SLA-aware job dispatch, and bin-packing across resource pools. It manages the full optimization lifecycle for hundreds of thousands of tables processing tens of millions of partitions per day. * **20–50% better data reduction** on top of standard ZSTD, achieved by Granica's Rust-based compression engine that adapts to each file's unique structure * **10–20% query speed improvement** on production workloads through data fingerprinting and layout optimization — no manual tuning required. This leads to direct cost savings on compute as well. Key features [#key-features] Intelligent scheduling at scale [#intelligent-scheduling-at-scale] Crunch's scheduler has full visibility into resource pools, job SLAs, and per-partition optimization state. When a job fails, Crunch retries only the minimum necessary work and prevents the retry spirals that plague general-purpose systems. * **File-level optimization tracking** — Crunch knows the state of every file, not just every table * **Intelligent batch decomposition** — large tables are broken into right-sized work units matched to available compute, preventing OOM failures and wasted retries * **Bin-packing and right-sizing** — jobs are packed against the resource pool to maximize utilization; compute is sized per job based on data volume and SLA * **SLA-based scheduling** — jobs run at the cheapest time that still meets the agreed service window Superior compression [#superior-compression] Crunch's compression engine delivers 20–50% better data reduction on top of standard ZSTD. On a TPC-DS `store_sales` table (418 GiB, 301,949 files, 1,823 partitions), Crunch achieved 36.4% data reduction versus 14.2% for Databricks ZSTD — producing 266 GiB of output versus 359 GiB on matched hardware. Query acceleration [#query-acceleration] Crunch profiles actual query and data patterns to select the optimal layout strategy — sort order, Z-order, file sizing, row group sizing — for each table's real access patterns. This delivers query speed improvements without any manual tuning. Multi-format and multi-catalog support [#multi-format-and-multi-catalog-support] Crunch manages all major open table formats natively: * **Delta Lake** — compaction, compression, VACUUM, transaction log cleanup * **Apache Iceberg** — snapshot expiration, orphan file deletion, metadata compaction * **Hive / Parquet** — compaction and compression It integrates with every major data catalog: Hive Metastore, AWS Glue, Unity Catalog, and Polaris. Critically, Crunch detects shared files across table formats — preventing accidental data loss during Hive-to-Iceberg or Hive-to-Delta migrations that format-specific tools cannot detect. Compliance and lifecycle management [#compliance-and-lifecycle-management] Crunch treats compliance as a first-class workflow: * **Partition-level deletion** — delete entire partitions on a defined schedule or on-demand for GDPR "right to be forgotten" requests * **Row-level PII deletion** — row-level deletes using the native table format's delete mechanism for tables where PII is not partition-isolated (GA coming soon) * **Audit trail** — full audit log for compliance reporting Production safety [#production-safety] Crunch was designed with production safety as the primary constraint: * **Read-before-write validation** — validates table state and catalog metadata before issuing any write * **Atomic commits** — all results are committed via the native Delta transaction log or Iceberg commit protocol; no partial writes are visible to readers * **File-level rollback** — if a job is interrupted, exactly the affected files are identified and cleaned up * **Cross-catalog shared-file detection** — before running vacuum or orphan deletion, Crunch checks all connected catalogs for shared file references * **Non-destructive by default** — original files remain in storage until the configured retention window expires, providing a recovery window No vendor lock-in [#no-vendor-lock-in] Crunch works exclusively with open table formats using their native commit protocols. It does not introduce any proprietary file format, metadata extension, or catalog dependency. Tables remain fully readable and writable by any engine — Spark, Trino, Flink, DuckDB, Snowflake, Athena — if Crunch is turned off. See also [#see-also] # Granica Crunch FAQ Compression and data integrity [#compression-and-data-integrity] Is Crunch lossless? [#is-crunch-lossless] Yes. Crunch uses lossless compression optimization. Every byte of your original data is preserved and recoverable. Pre- and post-crunch are compared and validated automatically. What are recipes? Does the recipe generator use AI? [#what-are-recipes-does-the-recipe-generator-use-ai] A **recipe** is a data-specific compression and encoding configuration that Granica derives by analyzing the statistical properties of your files — column cardinality, value distributions, sort order, and co-occurrence patterns. The recipe generator uses ML models trained on your actual data to determine the optimal compression codec, encoding strategy, and sort key for each column in each table. Recipes are stored in the recipe store and applied automatically during Crunch runs. As your data evolves, Granica re-analyzes files and updates recipes — for example, when a schema change is detected. This continuous learning is what allows Crunch to achieve higher DRRs than static, one-size-fits-all compression. How to maintain the Data Reduction Rate (DRR) after data is crunched? [#how-to-maintain-the-data-reduction-rate-drr-after-data-is-crunched] DRR is maintained by keeping your Crunch policy active. Background Crunch runs on a configurable schedule (daily or weekly) and processes new partitions as they arrive, so incoming data is compressed with the same recipes applied to historical data. If your historical data changes after it has been Crunched, for instance, due to late-arriving data or backfilling an ETL job, historical data needs to be recrunched. For late-arriving data patterns, you can simply configure Crunch to process a rolling window of "Today-10 to Today-1", for example, to ensure that all your late arriving data are fully optimized. Crunch is smart to not reprocess files that were not changed in this process, and only focus on changed or newly added files. What happens if Crunch encounters a corrupted file? [#what-happens-if-crunch-encounters-a-corrupted-file] Crunch validates every file before and after processing. If any integrity check fails, Crunch stops processing that bucket and alerts the operations team. No corrupted data is ever written. Query and workload impact [#query-and-workload-impact] Does Crunch affect query performance? [#does-crunch-affect-query-performance] Smaller files generally improve query performance because less data needs to be read from storage. Benchmarks shows that Crunch improves query speed by 10% to 20%. Queries that are IO bounded benefits the most from Crunch. The query acceleration feature further improve query speed by improving file pruning, using techniques such as sorting, z-ordering, stats, and right-sizing files and row groups. How does Crunch affect query stats? [#how-does-crunch-affect-query-stats] Crunch reduces physical file sizes by 15–60% without changing logical row counts or column values. Query engines that rely on file-level statistics (row counts, min/max values, null counts) embedded in Parquet footers will see those statistics preserved — Crunch does not alter them. Smaller file sizes mean less data is scanned from storage per query, which typically reduces scan times and lowers cloud storage egress costs. Is there any impact on data ops when accessing data during crunch time? [#is-there-any-impact-on-data-ops-when-accessing-data-during-crunch-time] No. Crunch is entirely out of the read and write path. While a Crunch job is running on a table, your applications can continue reading and writing normally — Crunch does not hold locks or block queries. For Iceberg and Delta Lake tables, Crunch commits optimized files using the table format's native snapshot commit protocol, so concurrent readers always see a consistent snapshot. For Hive tables, Crunch performs atomic object replacement. In all cases, there is no downtime and no coordination required from your data ops team. How does Crunch handle files and objects that are shared by multiple tables? [#how-does-crunch-handle-files-and-objects-that-are-shared-by-multiple-tables] Crunch tracks objects at the table level using catalog metadata. When a Crunch policy is applied to a table, only the files belonging to that table's current snapshot (as recorded in the table's metadata) are processed. Shared objects — for example, files referenced by multiple Delta Lake or Iceberg table versions simultaneously — are not blindly replaced. Crunch uses the table format's native commit protocol to introduce new optimized files as part of a new snapshot, leaving existing shared references intact. Contact your Granica account team if you have a specific shared-storage topology to evaluate. Compatibility [#compatibility] Does Crunch work with my existing tools? [#does-crunch-work-with-my-existing-tools] Yes. Crunch produces standard, format-compliant files. Any tool that reads Parquet today (Spark, Trino, Presto, Athena, BigQuery, Databricks, etc.) can read Crunched files without modification. Does Crunch support Liquid Clustering? [#does-crunch-support-liquid-clustering] Liquid Clustering is a Delta Lake feature that organizes data using clustering keys rather than partition directories. Crunch's Background mode is compatible with Liquid Clustered tables — it compresses and re-encodes files while preserving the clustering layout and committing changes via Delta's standard commit protocol. Runtime Crunch compatibility with Liquid Clustering, for both Delta tables and Iceberg Tables, is on the roadmap. Contact [sales@granica.ai](mailto:sales@granica.ai) for the latest status. Can Granica operate on Databricks' Spark engine such as Photon? [#can-granica-operate-on-databricks-spark-engine-such-as-photon] Granica's Runtime Crunch JAR integrates at the Apache Spark / Parquet writer layer. Photon is Databricks' proprietary native execution engine and does not expose the same integration points as open-source Spark. Runtime Crunch is therefore not currently supported on Photon. Background Crunch, which operates independently of the query engine by reading and writing objects directly in cloud storage, is compatible with tables written by Databricks and Photon. Contact [sales@granica.ai](mailto:sales@granica.ai) for details on Databricks compatibility. Will Granica support new features developed by Databricks? [#will-granica-support-new-features-developed-by-databricks] Granica tracks the Databricks and Delta Lake ecosystem actively. Features that affect the table format's commit protocol or file layout (such as new clustering strategies, column mapping modes, or deletion vectors) are evaluated for compatibility as they reach general availability. Granica's goal is to remain compatible with the evolving open table format ecosystem. Reach out to [sales@granica.ai](mailto:sales@granica.ai) for roadmap questions about specific Databricks features. What can Granica offer customers migrating between different table formats, such as Hive to Iceberg? [#what-can-granica-offer-customers-migrating-between-different-table-formats-such-as-hive-to-iceberg] Crunch's primary focus is compression optimization, not format conversion. However, Granica supports all three major table formats — Hive, Iceberg, and Delta Lake — so you can apply Crunch policies before, during, and after a migration. If your migration pipeline produces Iceberg or Delta Lake output, Crunch can begin optimizing the new format immediately as partitions are written. For customers actively migrating, contact your Granica account team to plan the policy transition to avoid redundant work on partitions that will be rewritten. Deployment and infrastructure [#deployment-and-infrastructure] What components are deployed? [#what-components-are-deployed] Granica deploys a **data plane** and, depending on your deployment model, a **control plane**: * **Data plane** (always in your cloud): Granica Worker pods (compression optimizers), Spark clusters, and the Tunnel Agent that maintains a secure outbound connection to the control plane. * **Control plane** (in Granica's cloud for Hybrid; in your cloud for On-Premises): Granica Console and API, Airflow scheduler, Log and Metrics stores, and a PostgreSQL state store. All data plane components run as Kubernetes pods on an EKS (AWS) or GKE (GCP) cluster inside your cloud account. See [Architecture](/crunch/architecture) for details. Can we use existing infrastructure such as an EKS cluster? What are the requirements? [#can-we-use-existing-infrastructure-such-as-an-eks-cluster-what-are-the-requirements] Granica provisions and manages its own dedicated EKS (AWS) or GKE (GCP) cluster inside your cloud account. It does not share an existing cluster with your workloads. This ensures Granica can control node configuration, autoscaling, pod scheduling, and upgrade rollouts independently without affecting your existing workloads. If you prefer to use your existing Infra, this can usually be supported. Contact [sales@granica.ai](mailto:sales@granica.ai) for minimum infrastructure requirements for your environment. Does my data leave my environment? [#does-my-data-leave-my-environment] It depends on your [deployment model](/installation/deployment-models). In the **Hybrid** model (most common), the data plane runs in your cloud — your actual table data never leaves your environment. Only control plane signals and aggregated metrics (job status, table names, partition counts) flow to Granica's cloud. In the **On-Premises** model, nothing leaves your cloud. In the **Granica Hosted** model, Granica manages all infrastructure including data processing. How do I integrate Granica into my environment? [#how-do-i-integrate-granica-into-my-environment] Granica typically processes your data in the background, reading from and writing to cloud storage without requiring any application integration. You can control Crunch through its APIs to set table policies, trigger actions, and check job status. For more, see [how Granica Crunch works](/crunch/how-it-works). How much time and effort is required from the customer's data infra/platform team? [#how-much-time-and-effort-is-required-from-the-customers-data-infraplatform-team] Initial deployment typically requires a few days of collaboration between your cloud infrastructure team and Granica's onboarding team — primarily to set up the necessary IAM roles, VPC networking, and cloud account permissions. Once deployed, Granica is self-managed and self-upgrading (in the Hybrid model). Ongoing operational overhead is minimal: administrators configure policies through the Granica Console and monitor progress through the built-in dashboards. No ongoing code changes or pipeline modifications are required for Background Crunch. How do you achieve exabyte-level scale? [#how-do-you-achieve-exabyte-level-scale] Granica automatically scales out additional nodes to dynamically handle arbitrarily large data volumes. Scaling is completely elastic — as load decreases, Granica automatically shuts down unneeded nodes. This minimizes operational costs and maximizes your savings from Crunch. How do you ensure my deployment is a success? [#how-do-you-ensure-my-deployment-is-a-success] Your Granica instance generates usage, system health, and performance telemetry data to enable predictive analysis, alerting, troubleshooting, and overall success. Telemetry data is stored in a cloud storage bucket unique to each customer and deployment, entirely separate from customer data. **No customer data is ever collected or analyzed.** How do I undo a Granica deployment? [#how-do-i-undo-a-granica-deployment] At any time you can uncrunch any data already processed by Granica Crunch to return it to its original form, by simply rewriting the data using the default Parquet writer. However this is not needed because data processed by Crunch follows open standards and are readable by all engines. You can then teardown your deployment to return your environment to its pre-Granica state. Savings and pricing [#savings-and-pricing] How long does it take to see savings? [#how-long-does-it-take-to-see-savings] You can start seeing storage and compute cost reductions within hours of enabling a Crunch policy on your first table or bucket. The exact timeline depends on the volume of data being processed, which is typically limited by the size of the compute pool that is given to Crunch. How is Crunch priced? [#how-is-crunch-priced] Crunch pricing is outcome-based. We make sure the customers always achieve a high ROI by using Crunch. Contact [sales@granica.ai](mailto:sales@granica.ai) for details. What is the estimated cost to crunch per PB? [#what-is-the-estimated-cost-to-crunch-per-pb] Crunch pricing scales with the savings it generates, not the volume processed. Because DRR varies by dataset, per-PB cost estimates depend on your specific data characteristics. Contact [sales@granica.ai](mailto:sales@granica.ai) for a tailored estimate based on your data profile. # Get started For Administrators [#for-administrators] This guide walks a newly onboarded administrator through setting up Granica Crunch for your organization — from first login to enabling your team to start optimizing tables. 1. First login and password change [#1-first-login-and-password-change] When your Granica instance is provisioned, a member of your team is designated as the initial administrator and is given a username and temporary password. On first login, you will be prompted to change your password. As the initial administrator, it is your responsibility to: * Secure your account with a strong password. * Add additional administrators to share platform management responsibilities (see Step 3). 2. Set up SSO integration (recommended) [#2-set-up-sso-integration-recommended] Before onboarding additional users, configure Single Sign-On (SSO) so that your team can authenticate using your organization's identity provider (IdP). SSO is a prerequisite if you plan to invite users or administrators without issuing individual passwords. Granica supports OIDC and SAML 2.0 providers (Okta, Azure AD, Google Workspace, and others). See [SSO Integration](/security-and-compliance/sso-integration) for setup instructions. If you skip SSO, you can still add users with local username/password credentials. SSO is required to allow users to authenticate via your IdP. 3. Manage users and add administrators [#3-manage-users-and-add-administrators] Once SSO is configured (or if you are using local accounts), add the rest of your team. You can assign each user the **Viewer**, **Editor**, or **Admin** role. * To delegate platform management, assign the **Admin** role to additional team members. * For data users who will set optimization policies, assign the **Editor** role. * For read-only access to dashboards and reports, assign the **Viewer** role. See [Manage Users](/administration/manage-users) and [Role-Based Access Control](/security-and-compliance/role-based-access-control) for details. 4. Connect catalogs [#4-connect-catalogs] Connect Granica to your data catalog so that it can discover and sync your tables. This is a prerequisite for users to view tables in Table Maintenance and set optimization policies. Granica supports Unity Catalog (Databricks), Hive Metastore, and Apache Polaris. After connecting a catalog, Granica syncs all eligible tables (0.1 GB and above) and makes them available in Table Maintenance. See [Connect Catalogs](/administration/connect-catalogs) for setup instructions. 5. Connect object stores (optional) [#5-connect-object-stores-optional] If your organization has data that lives outside any catalog — raw Parquet dumps, JSON event files, or unregistered object store prefixes — connect those locations so Granica can manage them in Object Maintenance. This step is a prerequisite for running Crunch on object prefixes that are not covered by a connected catalog. See [Connect Object Stores](/administration/connect-object-stores) for setup instructions. 6. Connect query history (optional) [#6-connect-query-history-optional] Connect your query engine's log output (Trino, Spark, or Athena) to unlock query-aware optimization insights in Table Maintenance: * **Est. time saved/mo** — an estimated total query time reduction per table, projected over 30 days, based on your actual query workload. * **Query Acceleration recommendations** — clustering and Z-ordering suggestions derived from your most frequent query predicate patterns, showing which column combinations are hit most often and the projected speedup if applied. Query Acceleration recommendation showing ZORDER suggestion with predicate column combinations and projected impact Without query history, Crunch still optimizes storage — but these workload-aware insights and the Query Acceleration column on the table list will not be populated. See [Connect Query History](/administration/connect-query-history) for setup instructions. *** For Users [#for-users] Once your administrator has connected catalogs and set up your account, you can start optimizing tables. This guide walks through the typical workflow. 1. Find the tables you want to optimize [#1-find-the-tables-you-want-to-optimize] Navigate to **Table Maintenance** in the sidebar. The table list shows all tables synced from your connected catalogs, along with their size, format, current optimization status, and estimated savings. Use the filters and search to narrow down the list: * Sort by **size** to find your largest tables first — these typically yield the most savings. * Filter by **catalog**, **schema**, or **table type** (Iceberg, Delta Lake, Hive). * Use the search bar to find a table by name. See [Tour of the Granica Console](/index) for a full walkthrough of the table list. 2. Estimate the Data Reduction Rate (DRR) [#2-estimate-the-data-reduction-rate-drr] Before committing to a Crunch policy, evaluate a table's optimization potential by collecting metadata. Open the table detail page and click **Collect Metadata**. Granica analyzes the table's files and partitions and computes the **Estimated DRR** — the projected percentage of storage that Crunch can save for this table. This step typically takes a few minutes depending on table size. Use the Est. DRR to prioritize which tables to onboard first. A higher DRR means more immediate storage savings. 3. Crunch the table [#3-crunch-the-table] If the estimated DRR looks good, you have two options — use one or both depending on your needs: **Set a recurring policy** — for ongoing optimization of newly arrived partitions. Configure a daily or weekly schedule, the partition date range to process, and the optimization primitives (compression, deduplication, compaction). The policy runs automatically on the configured schedule going forward. **Trigger a one-time run** — for crunching existing historical partitions. Go to the **Actions** tab on the table detail page, click **New Run**, set the partition date range, and submit. The job is queued immediately. Most teams do both: a one-time backfill run to optimize historical data, and a recurring policy to keep new partitions optimized as they arrive. See [Tour of the Granica Console](/index) for step-by-step policy configuration details. 4. Monitor progress [#4-monitor-progress] After submitting a run or enabling a policy, check back in the **Activities** section of [Monitoring](/administration/monitoring-platform) to track job status. The activity log shows each run's status (queued, running, succeeded, or failed), the partitions processed, and the bytes saved. Once a run completes, the table's DRR and storage savings appear in the Table Maintenance list and on the Overview dashboard. # Start a Crunch pilot The easiest way to prove out the value of Crunch compression optimization in your environment is to initiate a pilot. A Crunch pilot is quick and easy, typically lasting 2-3 weeks start to finish. Success criteria [#success-criteria] 1. **Storage Reduction:** Quantify the storage reduction capabilities of Granica Crunch on your representative Parquet dataset. 2. **Data Integrity:** Validate that data integrity is maintained throughout the compression lifecycle. 3. **Query Performance:** Validate that query performance improves or, at least, remains unaffected post-compression. By demonstrating the effectiveness of Granica Crunch across a representative sample of your data and your queries, a pilot builds confidence in the ability of Crunch to deliver value across your entire data lakehouse. Deployment options [#deployment-options] The preferred deployment requires a Kubernetes cluster that has access to: * **Data** — S3, GCS * **Catalogs** — Unity Catalog, Glue * **Database** — RDS, Cloud SQL You can pick the data environment that you feel comfortable with. For example, you may start with data in your Dev or QA environment before you transition to production. See the [Crunch FAQ](/faq/crunch) for a full list of supported catalogs and preferred Kubernetes specs. Customers can provision Kubernetes themselves, or delegate provisioning to Granica's Forward Deployment Engineers. For the latter, customers provide a "Maintenance Server" that has permission to provision Kubernetes and Managed Databases. Granica will then assign an IAM role for data access. Deploy Crunch [#deploy-crunch] Once the Kubernetes cluster is ready, deploy the Granica Helm Chart which includes all required components: * APIs for integration * Web Application for table discovery, monitoring, and onboarding The Helm Chart downloads artifacts from Granica Artifactory Store directly. If your company policy requires an internal Artifactory Store, contact our support team to arrange delivery. Get started [#get-started] Our pilot process is simple yet comprehensive. [Contact our product team](https://granica.ai/demo-request) to initiate a pilot today. The process is fast and easy, and you'll soon be on your way to reducing costs and speeding queries in your lakehouse. **Related reading** # AWS This guide will help you set up and destroy a Granica Admin Server with its VPC and subnets. Prerequisites [#prerequisites] * **AWS credentials** sufficient to run Terraform for this module. You do **not** need account administrator access if you follow a minimal FDE policy set. * **Existing VPC (typical):** customer-managed policy from `customer/fde/aws/docs/fde-user-existing-vpc.json` plus optional AWS managed policies for CloudShell / console read. Terraform creates the admin EC2 and `project-n-admin-*` instance profile; **`granica deploy` and EKS** run **on that instance** using its role, not your console user's deploy permissions. * **New VPC:** EC2/VPC permissions for `terraform-aws-modules/vpc` plus the admin instance; see `customer/fde/aws/docs/fde-user-new-vpc.json`. Replace **`ACCOUNT_ID`**. VPC build APIs use `Resource: *` (except **`RunInstances`** / **`TerminateInstances`**, which are scoped + tag conditions for the admin server). * Git installed on your system (or use AWS CloudShell, which includes Git). AWS CloudShell and Terraform disk space [#aws-cloudshell-and-terraform-disk-space] AWS CloudShell gives you a small **home** volume (on the order of **1 GiB**). A normal `terraform init` in `granica-setup/aws` stores the AWS provider (and module/plugin metadata) under `.terraform` in that directory, which often **fills `$HOME`** and breaks installs, shells, or git. **Before the first `terraform init` in this directory**, point Terraform's working data and plugin cache at **`/tmp`**, which typically has more room in CloudShell (treat it as **ephemeral**: new CloudShell sessions need these exports again): ```bash mkdir -p /tmp/granica-setup-aws-tf-data /tmp/terraform-plugin-cache export TF_DATA_DIR=/tmp/granica-setup-aws-tf-data export TF_PLUGIN_CACHE_DIR=/tmp/terraform-plugin-cache ``` Use the **same** `export` lines in any later session before `terraform init`, `terraform apply`, or `terraform destroy` (remote state stays in S3; this only affects local provider/module cache). If you already ran `init` without this and hit "no space left", remove the old local cache under `granica-setup/aws/.terraform` and default plugin dirs under `$HOME/.terraform.d` if present, then set the variables above and run `terraform init` again. Setup [#setup] 1. Install Terraform and clone the Granica Setup repo [#1-install-terraform-and-clone-the-granica-setup-repo] ```bash git clone https://github.com/tfutils/tfenv.git ~/.tfenv mkdir ~/bin ln -s ~/.tfenv/bin/* ~/bin/ export PATH="$HOME/.tfenv/bin:$PATH" tfenv install 1.13.4 tfenv use 1.13.4 terraform --version git clone https://github.com/granica-ai/granica-setup.git cd granica-setup/aws ``` **AWS CloudShell:** run the `mkdir` / `export` block from [AWS CloudShell and Terraform disk space](#aws-cloudshell-and-terraform-disk-space) here, before `terraform init`. 2. Configure deployment [#2-configure-deployment] Create `terraform.tfvars` in this directory. A sample is provided in `terraform.tfvars.sample`: ```hcl aws_region = "your-region" # Region where admin server and Granica will be deployed. E.g. us-east-1. package_url = "https://granica.ai/granica.rpm" server_name = "my-server" # Optional: suffix for admin server name (defaults to "dev") airflow_enabled = true # Optional: enables EFS permissions for Airflow deployment (defaults to false) ``` Create `backend.conf` in this directory so Terraform stores its state in S3. Set `key` to a value unique to this admin server: ```hcl bucket = "your-bucket" # S3 bucket that holds the Terraform state region = "your-state-bucket-region" key = "your-unique-key" # Change this to a unique identifier for your deployment ``` > **Note:** The `key` identifies your deployment and the state stored in the AWS bucket. You can reuse the same key to continue with a previously created deployment. If you reuse a previous key and want to start fresh, make sure the cleanup steps below have been completed first. 2.1 (Optional) Existing VPC [#21-optional-existing-vpc] Set `existing_vpc_id` (and subnets) to deploy into an existing VPC instead of creating a new one: ```hcl existing_vpc_id = "vpc-xxxxxxxxx" existing_private_subnet_ids = ["subnet-aaa", "subnet-bbb"] existing_public_subnet_ids = ["subnet-ccc"] # Required only if public_ip_enabled = true ``` The admin server is placed in the first private (or public, if `public_ip_enabled`) subnet. By default, an S3 Gateway VPC endpoint is *not* created when `existing_vpc_id` is set (to avoid `RouteAlreadyExists`). Set `create_s3_vpc_endpoint = true` to create it anyway. 2.2 (Optional) IAM role/policy naming and permission boundary [#22-optional-iam-rolepolicy-naming-and-permission-boundary] If your AWS account enforces an IAM naming convention, an IAM path, or a permissions boundary, set the variables below. All default to "off" — omit them unless your account requires them: ```hcl role_path = "/OneCloud/" role_name_prefix = "CustomerManagedBasic-" policy_name_prefix = "CustomerManaged_" policy_path = "/" permission_boundary_arn = "arn:aws:iam:::policy/BasicRole_Boundary" permission_boundary_on_admin_role = false ``` 3. Deploy the admin server [#3-deploy-the-admin-server] ```bash terraform init -backend-config=backend.conf terraform apply ``` This creates the admin server (named `granica-admin-server-{server_name}`) along with its VPC/subnets. Granica itself is **not** deployed yet — that runs from the admin server in the next step. 4. Run Granica Setup from the admin server [#4-run-granica-setup-from-the-admin-server] Once `terraform apply` finishes, connect to the admin server and deploy Granica: * **Connect** to the `granica-admin-server-{server_name}` instance via EC2 console **Connect → Session Manager**, or use the `aws ssm start-session` command from `terraform output admin_server_ec2_instance_connect_endpoint_connect_command`. * **Switch to `ec2-user`.** A Session Manager session starts as `ssm-user`, but `config.tfvars` and Granica files are owned by `ec2-user`: ```bash whoami # ssm-user sudo su - ec2-user whoami # ec2-user ``` * **Deploy Granica:** ```bash granica deploy --var-file=config.tfvars ``` Cleanup [#cleanup] Tear down in reverse order of setup. 1. Granica teardown [#1-granica-teardown] * Go to the AWS EC2 console * Connect to the `granica-admin-server-{server_name}` instance * Run: `granica teardown` 2. Admin server destroy [#2-admin-server-destroy] From AWS CloudShell: ```bash # Same TF_DATA_DIR / TF_PLUGIN_CACHE_DIR exports as for init, then: terraform init -backend-config=backend.conf terraform destroy ``` # Azure Coming soon... # Deployment Models Granica supports three deployment models. This page describes each model and what data, if any, crosses your cloud boundaries — helping your InfoSec team understand the design and accelerate approvals. Model 1 — Granica Hosted [#model-1--granica-hosted] In the Granica Hosted model, Granica manages the platform infrastructure on your behalf. Customers' data objects and catalog metadata flow in and out of the customer's cloud as part of normal Crunch operations. **Data that leaves your cloud:** table data and catalog metadata. Granica Hosted deployment model Model 2 — On-Premises [#model-2--on-premises] In the On-Prem model, the Granica platform runs entirely within your cloud environment. No actual table data ever leaves your cloud. 2A — With Tunnel [#2a--with-tunnel] Data that flows out of your cloud boundaries consists only of HTML content representing the Granica Console — a web application used by customers and Granica employees to discover, onboard, and monitor table maintenance activities. This content includes: * Aggregated reports and metrics (e.g. total bytes crunched, average data reduction rates) * Table names and metadata (e.g. timestamps, column names, table types) * Policies applied to tables * Activity and stats (e.g. when a partition was crunched, duration, rows processed) * System and user configurations **Actual table data does not leave your cloud.** On-premises deployment model with tunnel 2B — Without Tunnel [#2b--without-tunnel] No payload crosses your cloud boundaries. **Actual table data does not leave your cloud.** Model 3 — Hybrid [#model-3--hybrid] Data that flows out of your cloud includes Spark job progress (e.g. `numCompletedTasks`), job status (e.g. `RUNNING`), failures (e.g. `OutOfMemoryError`), and performance metrics. **Actual table data does not leave your cloud.** Hybrid deployment model Comparison [#comparison] | | Pros | Cons | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | **Granica Hosted** | Minimal permissions required (data/catalog read/write only); no infrastructure bandwidth needed from customer; Crunch compute cost borne by Granica | Data travels outside the customer environment, which may require InfoSec review | | **On-Premises** | No data leaves the customer cloud — not even logs or metrics | Requires customer infrastructure deployment and ongoing maintenance | | **Hybrid** | No actual table data leaves the customer cloud; only metadata and metrics are shared | — | # GCP Prerequisites [#prerequisites] If you are working in Cloud Shell you must be logged in as Admin. If you are running from your laptop you will need GCloud command line credentials that give administrator access. * Create your own GCP Project (by default you will have admin access for this project). Instructions [#instructions] 1. Enable GCP APIs [#1-enable-gcp-apis] ```bash gcloud services enable storage.googleapis.com gcloud services enable iam.googleapis.com gcloud services enable cloudresourcemanager.googleapis.com gcloud services enable networkmanagement.googleapis.com gcloud services enable container.googleapis.com gcloud services enable logging.googleapis.com gcloud services enable pubsub.googleapis.com gcloud services enable compute.googleapis.com gcloud services enable sqladmin.googleapis.com gcloud services enable servicenetworking.googleapis.com ``` 2. Install Terraform [#2-install-terraform] ```bash git clone https://github.com/tfutils/tfenv.git ~/.tfenv mkdir ~/bin ln -s ~/.tfenv/bin/* ~/bin/ export PATH="$HOME/.tfenv/bin:$PATH" tfenv install 1.13.4 tfenv use 1.13.4 terraform --version git clone https://github.com/granica-ai/granica-setup.git cd granica-setup/gcp ``` 3. Create a GCS bucket for Terraform state [#3-create-a-gcs-bucket-for-terraform-state] ```bash gcloud storage buckets create gs:// --location ``` 4. Configure backend.conf [#4-configure-backendconf] ```hcl bucket = "" prefix = "" ``` 5. Configure terraform.tfvars [#5-configure-terraformtfvars] ```hcl project_id = "your-gcp-project-id" region = "us-central1" zone = "us-central1-a" package_url = "https://granica.ai/granica.rpm" server_name = "CHANGE_ME" ``` 6. Deploy the admin server [#6-deploy-the-admin-server] ```bash terraform init -backend-config=backend.conf terraform apply ``` 7. Log in to the admin server [#7-log-in-to-the-admin-server] ```bash gcloud compute ssh granica-admin-server-{server_name} --project= --zone= --tunnel-through-iap ``` > Use the `gcloud` command printed at the end of `terraform apply` to SSH into the admin server. Once connected: ```bash sudo su - granica # Use the granica user to run granica commands # The Granica RPM takes around 10-15 minutes to install. # Monitor progress: tail -f /var/log/dnf.rpm.log # For more debug output: tail -f /var/log/startup-script.log granica --help granica deploy --var-file config.tfvars # Cluster deployment takes around 10-15 minutes. ``` # API Keys API keys provide machine-to-machine (M2M) access to the Granica REST API without requiring a user session. Use them to integrate Granica into automated pipelines, CI/CD workflows, and infrastructure tooling. API keys are managed under **Settings → API Keys** and are only accessible to users with the Admin role. Granica API Keys page showing the Create Key form with access policy selection How API keys work [#how-api-keys-work] Each API key is a long-lived bearer token. Include it in the `Authorization` header of every API request: ```http Authorization: Bearer ``` Keys are displayed once at creation time and cannot be retrieved afterwards — store them securely in a secrets manager or environment variable immediately after creation. Create an API key [#create-an-api-key] Click **+ Create API Key** to open the creation form inline above the key list. 1. Key Name [#1-key-name] Give the key a descriptive name that identifies its purpose and owner, such as `Production pipeline key` or `CI/CD — data quality checks`. Good names make it easy to audit and revoke keys later. 2. Expiration Date [#2-expiration-date] Optionally set an expiration date between 1 and 365 days from today. Leave the field empty for a key that never expires. Keys automatically become inactive after their expiration date and can no longer be used to authenticate requests. 3. Access Policies [#3-access-policies] Access policies control exactly which API operations the key can perform. Permissions are organized into five policy groups — check individual capabilities within each group or select the entire group at once. If you make no selection, the key is granted **full access** to all API endpoints. **Quick presets** let you apply a standard set of scopes in one click: | Preset | What it includes | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Read Only** | View everything across all groups — tables, crunch jobs, vacuum history, schedules, policies, and config. No write access. | | **Full Maintenance** | Complete table maintenance: crunch, vacuum, partition expiration, and schedule management. Excludes catalog management and platform admin. | | **Vacuum Only** | Snapshot expiration and orphan file cleanup — vacuum read, write, and policy scopes only. | Policy groups and capabilities [#policy-groups-and-capabilities] **Table Discovery & Onboarding** | Capability | Scope key | What it allows | | -------------------------- | --------------------------- | ----------------------------------------------------------------- | | View tables & metadata | `tables:read` | List tables, view schema, partitions, and size metrics | | Create & configure tables | `tables:write` | Onboard, update, and remove tables | | View catalog connections | `catalog_connections:read` | List connections, browse catalogs, schemas, and discovered tables | | Manage catalog connections | `catalog_connections:write` | Create, update, delete connections and trigger syncs | | View onboarding status | `onboarding:read` | Check progress of table onboarding workflows | | Retry failed onboarding | `onboarding:write` | Retry onboarding tasks that have failed | **Crunch — File Optimization** | Capability | Scope key | What it allows | | ------------------------------- | --------------------- | -------------------------------------------------------------------------------------------- | | View crunch jobs & metrics | `crunch:read` | View job status, execution history, compression metrics | | Trigger crunch operations | `crunch:write` | Run on-demand crunch jobs | | View crunch policy & primitives | `crunch_policy:read` | View which primitives are enabled and their configuration | | Configure crunch primitives | `crunch_policy:write` | Enable/disable and configure Compression, Compaction, Sorting, Clustering, and Deduplication | | View crunch schedules | `schedules:read` | View automated crunch schedules | | Manage crunch schedules | `schedules:write` | Create, update, and delete crunch schedules | **Vacuum** | Capability | Scope key | What it allows | | ------------------------- | --------------------- | ------------------------------------------------------------------ | | View vacuum job history | `vacuum:read` | View vacuum job status and cleanup metrics | | Trigger vacuum operations | `vacuum:write` | Run on-demand vacuum to expire snapshots and delete orphaned files | | View vacuum policy | `vacuum_policy:read` | View retention settings and vacuum policy configuration | | Configure vacuum policy | `vacuum_policy:write` | Update vacuum retention policy | **Partition Expiration** | Capability | Scope key | What it allows | | -------------------------------- | ---------------------------- | ----------------------------------------------- | | View partition expiration policy | `partition_exp_policy:read` | View partition retention rules | | Configure partition retention | `partition_exp_policy:write` | Create and update partition expiration policies | **Platform Administration** | Capability | Scope key | What it allows | | ----------------------------- | -------------- | --------------------------------------------- | | View config & health | `config:read` | Read platform configuration and health status | | Update platform configuration | `config:write` | Modify platform-level configuration settings | 4. Create Key [#4-create-key] Click **Create Key**. The full API key value is shown once in the confirmation panel. Copy it immediately — it cannot be retrieved after you close the panel. Store your API key in a secrets manager or environment variable. It is shown only once and cannot be retrieved from the Granica Console after creation. Manage existing keys [#manage-existing-keys] The API Keys page lists all keys with their status, access summary, creator, expiration, and last used timestamp. | Column | Description | | ------------- | -------------------------------------------------------------- | | **Name** | Key label and the user who created it | | **Key** | Key prefix (`abc123...`) — the full value is never shown again | | **Access** | Badge summary of which policy groups the key covers | | **Status** | Active, Expired, or Revoked | | **Created** | Creation date | | **Expires** | Expiration date, or "Never" | | **Last Used** | Timestamp of the most recent authenticated request | Click any row to open the key detail page, which shows the full access policy breakdown and a per-endpoint map of what the key can and cannot call. Revoke a key [#revoke-a-key] Click the **⋯** Actions menu on any active key row and select **Revoke**. Revocation is immediate and permanent — any application using that key loses access instantly. Revoked keys cannot be re-activated; create a new key if access needs to be restored. Using the API key [#using-the-api-key] Pass the key as a Bearer token in the `Authorization` header: ```bash curl -H "Authorization: Bearer " \ https:///api/v1/tables ``` The key is checked against its access policy on every request. If a request targets an endpoint not covered by the key's scopes, it is rejected with `403 Forbidden`. See [Granica APIs V1](/api-reference) for the full API reference. # Role-Based Access Control Granica uses role-based access control (RBAC) to govern what each user can see and do in the Console. Every user is assigned one of three roles: **Viewer**, **Editor**, or **Admin**. Roles are assigned when a user account is created and can be changed at any time by an Admin. Roles overview [#roles-overview] | Capability | Viewer | Editor | Admin | | ----------------------------------------------- | :----: | :----: | :---: | | View dashboard and savings metrics | ✓ | ✓ | ✓ | | View query history | ✓ | ✓ | ✓ | | View table optimization opportunities | ✓ | ✓ | ✓ | | View onboarding status | ✓ | ✓ | ✓ | | Onboard tables for optimization | | ✓ | ✓ | | Run and view evaluations | | ✓ | ✓ | | Create and manage schedules | | ✓ | ✓ | | View platform configuration | | ✓ | ✓ | | Manage users and roles | | | ✓ | | Configure SSO and enforce authentication policy | | | ✓ | | Manage catalog connections | | | ✓ | | Manage platform settings | | | ✓ | | Access all platform capabilities | | | ✓ | Role descriptions [#role-descriptions] Viewer [#viewer] Viewers have read-only access to the Granica Console. They can monitor the platform and review optimization results, but cannot make any changes. **Viewers can:** * View the dashboard, including savings metrics and compression ratios * Browse the table list and inspect per-table details and optimization opportunities * Review query history Viewers cannot onboard tables, create schedules, or modify any settings. Editor [#editor] Editors can configure and run optimizations. This role is appropriate for data engineers and platform engineers who manage day-to-day operations. **Editors can do everything a Viewer can, plus:** * Onboard tables and manage their optimization policies * Run evaluations to assess compression candidates * Create, update, and delete Crunch schedules * View platform configuration Editors cannot manage users, configure SSO, or modify platform-level settings. Admin [#admin] Admins have full access to all platform capabilities. Assign this role to users who are responsible for deploying, configuring, and securing the Granica platform. **Admins can do everything an Editor can, plus:** * Create, edit, deactivate, and delete user accounts * Assign and change user roles * Configure and enforce SSO authentication policy * Manage catalog connections * Modify platform settings (table size thresholds, job configuration, etc.) There should always be at least one Admin user with verified access before enforcing SSO or making changes to authentication policy. See [SSO Integration](/security-and-compliance/sso-integration) for break-glass user configuration. Manage users and roles [#manage-users-and-roles] Admins manage users from **Settings → Users** in the Granica Console. From this page you can: * **Invite a new user** — Enter an email address and select a role. The user receives an invitation email with a link to set their password. * **Change a role** — Click the role badge next to any user and select a new role. The change takes effect on their next request. * **Deactivate a user** — Deactivated users cannot log in but their account and history are retained. * **Delete a user** — Permanently removes the account. See [Manage Users](/administration/manage-users) for step-by-step instructions. API token permissions [#api-token-permissions] API tokens are scoped independently of the user role that created them. When generating a token, you select exactly which resource actions it can perform (for example `tables:read` or `schedule:write`). A token can never exceed the permissions of the creating user's role, but it can be scoped to a subset. See [API Token](/security-and-compliance/api-token) for details on creating and managing tokens. # Security certifications Granica maintains the highest level of data security by incorporating industry leading best practices into our information security program. We are dedicated to obtaining and maintaining industry recognized security and privacy third party certifications, and to working with independent, CBA-registered CPA firms to regularly audit our program and attest to our certifications. **Benefits for our customers:** * Increased visibility and confidence in our information security program and overall operations * Increased ease in onboarding Granica as a vendor **Benefits for Granica:** * Ensures we continue to align with industry best practices to meet the requirements of a strong and comprehensive information security program * Streamlines the process of sharing information on our security program with potential and existing customers Current [#current] * [SOC 2 Type 1 Report](https://www.aicpa.org/interestareas/frc/assuranceadvisoryservices/aicpasoc2report.html): The SOC 2 report focuses on a business's non-financial reporting controls as they relate to security, availability, processing integrity, confidentiality, and privacy of a system, as opposed to SOC 1/SSAE 16 which is focused on the financial reporting controls. The SOC 2 Type 1 (or Type I) report evaluates the effectiveness of the deployed controls *at a point in time*. * [SOC 2 Type 2 Report](https://www.aicpa.org/interestareas/frc/assuranceadvisoryservices/aicpasoc2report.html): The Type 2 (or Type II) report evaluates the effectiveness of deployed controls *over a period of time*. This report provides greater assurance and is more comprehensive than the Type 1 report. Planned (Roadmap) [#planned-roadmap] * [ISO/IEC 27001 Report](https://www.iso.org/isoiec-27001-information-security.html): An international standard that evaluates the effectiveness of an information security management system (ISMS). ISO/IEC 27001 addresses people and processes as well as technology. Requesting a copy of the Granica SOC 2 Reports [#requesting-a-copy-of-the-granica-soc-2-reports] SOC 2 Reports are restricted and cannot be shared publicly. We can only share SOC 2 reports upon request with prospective customers under NDA or with current customers bound by confidentiality agreements. To request our SOC 2 report, contact [security@granica.ai](mailto:security@granica.ai) and provide the following information: * Company Name * Report Requestor name * Report Requestor email * Report Requestor Job Title You will receive an acknowledgement email within one business day. SOC Overview [#soc-overview] The American Institute of CPAs (AICPA) has developed a suite of System and Organizational Controls (SOC) reports. The reports are divided into three categories: * SOC for Service Organizations * SOC for Cybersecurity * SOC for Supply Chain The SOC for Service Organizations category is classified into: * **SOC 1** — Internal Controls over Financial Reporting (ICFR) * **SOC 2** — Trust Services Criteria * **SOC 3** — Trust Service for General Use Report SOC 2 Trust Services Criteria [#soc-2-trust-services-criteria] According to AICPA, the SOC 2 Report covers controls relevant to: * Security * Availability * Processing Integrity * Confidentiality * Privacy The report must contain Security (Common Criteria), with the remaining Trust Services Criteria included if applicable. SOC 2 Type 1 vs Type 2 [#soc-2-type-1-vs-type-2] * **Type 1** evaluates the effectiveness of deployed controls at a *point in time*. * **Type 2** evaluates the effectiveness of deployed controls *over a period of time*, providing greater assurance. HIPAA Compliance [#hipaa-compliance] If a customer's cloud environment is HIPAA compliant and proper controls around data handling, access, and separation are implemented, then using Granica can be part of an overall HIPAA-compliant architecture. This is because Granica does not directly handle protected health information (PHI): * Data stays within the customer's own HIPAA-compliant cloud environment and Granica does not have access to actual data contents * Granica relies on the cloud provider's native encryption for data at rest and in transit * The Granica control plane runs within the customer's VPC and follows their security policies * Data isolation happens automatically if the customer uses separate buckets per tenant # SSO Integration Granica supports Single Sign-On (SSO) via two industry-standard protocols: **OpenID Connect (OIDC)** and **SAML 2.0**. Once configured, users authenticate through your existing identity provider rather than managing separate Granica credentials. Granica SSO configuration page Supported protocols [#supported-protocols] OpenID Connect (OIDC) [#openid-connect-oidc] OIDC is an identity layer built on top of OAuth 2.0. Granica acts as a confidential OIDC client using the Authorization Code flow. Configuration requires a Discovery URL (`.well-known/openid-configuration`), a Client ID, and a Client Secret. Supported OIDC identity providers: | Identity Provider | Setup Guide | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------ | | Okta | [Implement Authorization Code with PKCE](https://developer.okta.com/docs/guides/implement-grant-type/authcodepkce/main/) | | Microsoft Entra ID | [Register an app in Entra](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app) | | Google Workspace | [OpenID Connect on Google](https://developers.google.com/identity/openid-connect/openid-connect) | | Auth0 | [Application Settings](https://auth0.com/docs/get-started/applications/application-settings) | | AWS IAM Identity Center | [Configure OIDC grant](https://docs.aws.amazon.com/singlesignon/latest/userguide/app-config-grant-oidc.html) | | Keycloak | Use the standard OIDC discovery URL from your Keycloak realm. | | Custom / Generic | Any [OpenID Connect Core 1.0](https://openid.net/specs/openid-connect-core-1_0.html) compliant provider. | SAML 2.0 [#saml-20] Granica acts as a SAML Service Provider (SP). Configuration requires importing IdP metadata (via URL or XML file) and registering Granica's SP metadata with your identity provider. Supported SAML 2.0 identity providers: | Identity Provider | Setup Guide | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Okta | Configure via SP metadata URL or XML upload in Okta Admin Console under **Applications → SAML Settings → Edit → Configure SAML**. | | Microsoft Entra ID | Configure via Entra Enterprise Application SAML blade — **Enterprise Applications → Single sign-on → SAML → Basic SAML Configuration**. | | Google Workspace | Configure via **Apps → Web and mobile apps → Add custom SAML app** in Google Admin Console. Note: assertion encryption is not supported for Google Workspace SAML. | | Auth0 | Configure via **Applications → Addons → SAML2 Web App → Settings** by pasting the Granica-generated JSON. | | AWS IAM Identity Center | Configure via **Applications → Add custom SAML 2.0 application** by uploading Granica's SP metadata XML. Note: users must be pre-provisioned in Granica before they can sign in (SCIM auto-provisioning is not supported). | | Custom / Generic | Any SAML 2.0-compliant provider. Use the raw SP values (Entity ID, ACS URL, SLO URL, SP certificate) or download the SP metadata XML. | Configuration steps [#configuration-steps] SSO is configured in the Granica Console under **Settings → SSO**. The wizard walks through five steps: 1. **Vendor** — Pick your identity provider. Granica autofills recommended defaults and shows only the fields your provider needs. 2. **IdP config** — Register Granica as a client or SP in your identity provider using the values provided (redirect URIs, ACS URL, Entity ID, metadata URL, etc.). 3. **Import** — Import your identity provider's metadata back into Granica (discovery URL for OIDC, or IdP metadata URL/XML for SAML). 4. **Security** — Review signing and encryption settings. Granica signs outbound requests by default for SAML; encryption settings can be adjusted here. 5. **Test & enable** — Run a test login to verify the configuration before enabling SSO for all users. Break-glass access [#break-glass-access] Granica supports designated break-glass users who can always log in with local credentials, even if SSO is misconfigured or your identity provider is unavailable. Configure break-glass users in **Settings → SSO** before enforcing SSO to ensure you are never locked out. Enforcement modes [#enforcement-modes] | Mode | Description | | ------------ | ------------------------------------------------------------------------------------------ | | **Disabled** | SSO is not active. Users log in with local credentials. | | **Test** | SSO is configured and can be tested, but local login remains available for all users. | | **Enforced** | All users must authenticate via SSO. Local login is disabled except for break-glass users. |