GCS Parquet Explorer

GCS Parquet Explorer is a single-file Python web application for browsing and querying SAP CDS Parquet data across Google Cloud Storage, Azure Data Lake, and AWS S3. It provides an interactive web UI with DuckDB-powered SQL queries, schema inspection, and Iceberg table support via Polaris REST catalogs.

Key Features

Single-File Design

Everything lives in one .py file: Python backend + embedded HTML/CSS/JS frontend in a raw string (HTML_PAGE = r"""..."""). Because the HTML is in a Python raw string, \u unicode escapes do NOT work in CSS/JS — use literal UTF-8 characters or HTML entities.

Technology Stack

Server vs Local Differences

Configuration

Requirements

duckdb
pyarrow
google-cloud-storage
psutil

Setup

# Install dependencies
pip install duckdb pyarrow google-cloud-storage psutil

# Authenticate with GCS
gcloud auth application-default login

# Run the explorer
python gcs_explorer.py

Host Details

Server Files

systemd Service

[Unit]
Description=GCS Parquet Explorer Web Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/usr/sap
EnvironmentFile=-/usr/sap/gcs_explorer.env
ExecStart=/root/miniconda/bin/python3 -u /usr/sap/gcs_explorer_server.py
Restart=always
RestartSec=5
StandardOutput=append:/var/log/gcs_explorer.log
StandardError=append:/var/log/gcs_explorer.log

[Install]
WantedBy=multi-user.target

Features: auto-restart on crash (5s delay), auto-start on reboot, unbuffered output (-u) for immediate log visibility.

Additional Routes

The server also hosts the SAP Skills Portal static files at the /sap_skills/ route, serving HTML/CSS/JS from /usr/sap/sap_skills/ on the filesystem.

Environment Variables

Why Subprocess Isolation?

DuckDB's C extension holds the Python GIL (Global Interpreter Lock) during network I/O operations such as ATTACH TYPE ICEBERG and iceberg_scan over S3. In the old in-process design, this froze ALL Python threads — including the HTTP server's accept loop. No amount of Python-level locking or watchdogging could prevent this because the GIL itself was blocked in C code.

Solution: Moving DuckDB to a separate subprocess gives it its own GIL that cannot affect the HTTP server. The two processes communicate via multiprocessing.Queue using a command/response pattern.

Process Architecture

Commands

Command Protocol

# Main -> Worker (via cmd_q)
{"id": "uuid", "cmd": "exec_sql", "args": {"query": "SELECT ..."}}

# Worker -> Main (via resp_q)
{"id": "uuid", "status": "ok", "result": {"cols": [...], "data": [[...]]}}

Hang Recovery

When a DuckDB operation hangs (e.g., iceberg_scan over unreachable S3):

1. The send_command() timeout fires
2. DuckDBWorker._kill_and_respawn() sends SIGKILL to the subprocess
3. All pending waiters are woken with "worker restarted" error
4. loaded_tables and polaris_catalogs are cleared
5. A fresh subprocess starts with clean DuckDB
6. The HTTP server never stops responding

Memory Management

The _maybe_evict() function is called before loading any new table. It checks system memory usage via psutil and evicts the oldest tables from the loaded_tables OrderedDict (LRU order) until system memory drops below 85%.

# Memory threshold
MEMORY_THRESHOLD = 0.85  # evict at 85% system memory

DuckDB Initialization

def init_duckdb():
    # Ensure HOME is set - systemd services may not have it
    if not os.environ.get("HOME"):
        os.environ["HOME"] = "/root"
    db_conn = duckdb.connect(":memory:")
    db_conn.execute(f"SET home_directory='{os.environ['HOME']}';")
    db_conn.execute("INSTALL httpfs; LOAD httpfs;")
    db_conn.execute("INSTALL iceberg; LOAD iceberg;")
    db_conn.execute("INSTALL azure; LOAD azure;")
    db_conn.execute("SET azure_transport_option_type='curl';")  # Fixes SSL on SUSE

Overview

The explorer uses the DuckDB Iceberg extension for querying Iceberg tables and the Polaris REST catalog for fast namespace and table listing. Three catalog presets are pre-configured for GCS, Azure, and AWS.

Catalog Presets

Polaris REST API (Fast Listing)

Namespace and table listing uses the Polaris REST API directly instead of DuckDB information_schema queries. This is approximately 250x faster (0.3s vs 70s for Azure).

# Key REST API functions
def _polaris_oauth_token(endpoint, client_id, client_secret):
    """POST to {endpoint}/v1/oauth/tokens with client_credentials grant."""

def _polaris_rest_get(endpoint, path, token):
    """GET {endpoint}{path} with Bearer token."""

def list_polaris_namespaces(alias):
    """GET /v1/{catalog}/namespaces"""

def list_polaris_tables(alias, namespace):
    """GET /v1/{catalog}/namespaces/{ns}/tables"""

AWS Vended Credentials & Query Rewrite

DuckDB's Iceberg extension does NOT send the X-Iceberg-Access-Delegation: vended-credentials header when querying via ATTACH ... TYPE ICEBERG. The workaround implemented in _rewrite_aws_query():

1. Detect SQL query references an AWS Polaris catalog (e.g., aws.namespace.table)
2. Resolve the table's metadata-location via the Polaris REST API with vended-credentials header
3. Fetch vended S3 credentials from the table response config field
4. Create a DuckDB S3 secret with the vended credentials
5. Rewrite the query to use iceberg_scan('s3://...metadata.json')

# User writes:
SELECT * FROM aws.sap_hana_db_saphanadb.mara LIMIT 100

# Server rewrites to:
SELECT * FROM iceberg_scan('s3://ts-s3-multiformat-dest-demo/sap_hana_db_saphanadb/mara/metadata/00092-xxx.metadata.json') LIMIT 100

Credential Management

The server version includes a Manage Credentials UI panel that allows updating OAuth2 client_id and client_secret at runtime via POST /api/polaris/update_credentials. Changes persist in memory until server restart. If a provider is currently connected, it is automatically disconnected so the next connect uses the new credentials.

GET Endpoints

All routes are under the BASE_PATH prefix (/datalake_reader on server).

POST Endpoints

Production Deployment Workflow

# 1. Upload to server
scp /tmp/gcs_explorer_server.py sapidesecc8:/tmp/gcs_explorer_server.py

# 2. Deploy and restart
ssh sapidesecc8 "sudo cp /tmp/gcs_explorer_server.py /usr/sap/gcs_explorer_server.py; sudo systemctl restart gcs-explorer"

# 3. Verify
ssh sapidesecc8 "sudo systemctl status gcs-explorer --no-pager"

Dev Deployment

A separate dev instance runs alongside production on port 8443:

# Dev deployment
scp /tmp/gcs_explorer_server_dev.py sapidesecc8:/tmp/
ssh sapidesecc8 "sudo cp /tmp/gcs_explorer_server_dev.py /usr/sap/gcs_explorer_server_dev.py; sudo systemctl restart gcs-explorer-dev"

Service Management

sudo systemctl start gcs-explorer       # Start
sudo systemctl stop gcs-explorer        # Stop
sudo systemctl restart gcs-explorer     # Restart (after deploy)
sudo systemctl status gcs-explorer      # Check status
sudo systemctl enable gcs-explorer      # Enable on boot (already done)
journalctl -u gcs-explorer -f           # Follow logs

SSL Certificate

Certificate

/usr/sap/gcs_explorer_cert.pem (ZeroSSL, chained: server + intermediate CA)

Private Key

/usr/sap/gcs_explorer_key.pem

Expiration

June 25, 2026

Server Stability & Resilience

Proactive measures (prevent freezes):

Reactive backup measures: