AWS Batch Fargate • Scale-to-Zero EC2 • EMR Sedona • DuckDB 2.0

Serverless Batch Raster, Mesh & Spatial Lakehouse Engines

When geospatial jobs outgrow synchronous request limits—such as multi-gigabyte Cloud-Optimized GeoTIFF (COG) synthesis, multidimensional Zarr v2/v3 weather cubes, LiDAR hydro-conditioning, or billion-row GeoParquet spatial joins—Tech Maven Geospatial orchestrates containerized pipelines on AWS Batch (Fargate & scale-to-zero EC2) and Amazon EMR Serverless.

Inspect 6 Containerized Engines DuckDB 2.0 & Sedona Lakehouse Local-to-AWS Migration Toolkit

Fargate + EC2

Dual AWS Batch compute environments that scale down to 0 vCPUs when queues empty

DuckDB 2.0

Vectorized `SPATIAL_JOIN` on S3 GeoParquet, Apache Iceberg, S3 Tables & PostGIS

Rust Terrio

GDAL-free, memory-safe Rust raster processing & SIMD map algebra container

Atomic S3

COG/Zarr/Parquet assets verified before completion markers & monotonic leases

Isolated Workload Runtimes

Six Purpose-Built Geospatial Container Engines on AWS Batch

Each workload runs in an isolated, digest-pinned container image with dedicated IAM permissions, bounded scratch storage, and explicit CRS/nodata preservation.

1. Scientific Raster & Multidimensional Cube Worker

GDAL, Rasterio, xarray, Zarr & MetPy (`geoproc/`)

Processes meteorological grids (HRRR GRIB2 / NetCDF), multi-spectral satellite imagery, and digital elevation models into Cloud-Optimized GeoTIFFs (COG) and chunked Zarr stores on S3.

Strict Scientific Provenance: Validates coordinate reference systems (CRS), grid resolution compatibility, physical units, valid timestamps, and explicit nodata masks before computation.
Monotonic Lease & Atomic Publishing: Acquires DynamoDB leases that outlive job durations and writes `_SUCCESS` / `latest.json` pointers only after all COG/Zarr objects pass integrity checks.
HRRR & Atmospheric Event Ingestion: EventBridge-triggered S3 ingestion pipelines automatically process new forecast cycles without duplicate runs.

2. Pure-Rust Raster Engine (`terrio_batch/`)

GDAL-Free Rust Terrio Container

Eliminates large C/C++ shared library footprints with a lightweight, memory-safe Rust binary engineered for high-throughput raster algebra and Cloud-Optimized GeoTIFF streaming.

Minimal Cold-Start Image: Distroless/slim container starts in seconds on AWS Batch Fargate with deterministic memory bounds.
Native SIMD Raster Algebra: Executes normalized difference indices (NDVI/NDWI), slope/aspect derivatives, hillshade, and threshold masks directly over S3 byte-range reads.
Zero GDAL Dependency Chain: Immune to PROJ/GDAL version conflicts while preserving GeoTIFF tags, GeoKeys, and overview pyramids.

3. High-Speed Analytical SQL Worker (`duckdb_batch/`)

DuckDB 2.0 GeoParquet, Iceberg & PostGIS Engine

Disposable serverless Fargate workers running DuckDB 2.0 with the `spatial`, `httpfs`, `aws`, `iceberg`, and `postgres` extensions for out-of-core vector and tabular geoprocessing.

Native `SPATIAL_JOIN` Operator: Accelerated R-tree spatial joins across millions of polygons and points stored in S3 GeoParquet without loading data into a persistent DBMS.
Multi-Catalog Federation: Join S3 GeoParquet lakes, AWS S3 Tables (Apache Iceberg), and live Aurora PostGIS tables in a single SQL pipeline.
Verified Extension Security: Extension signature verification is strictly enforced (`allow_unsigned_extensions` is never enabled).

4. Remote Sensing & Hydrological Geomorphometry

Orfeo ToolBox (`otb_batch/`) & WhiteboxTools (`whitebox_batch/`)

Turnkey AWS Batch containers for specialized remote sensing image segmentation, SAR/optical calibration, and LiDAR hydrological terrain modeling.

Orfeo ToolBox (OTB): Radiometric calibration, pan-sharpening, Large-Scale Mean-Shift (LSMS) segmentation, and object-based image classification on S3 imagery.
WhiteboxTools Hydro-Terrain Suite: Depression breaching (`BreachDepressionsLeastCost`), D8/D-Infinity flow accumulation, topographic wetness index (TWI), stream network extraction, and LiDAR LAS/LAZ DEM interpolation.

5. Petabyte Distributed Spatial Lakehouse (`infra/emr-sedona.yaml`)

Amazon EMR Serverless + Apache Sedona

When spatial joins span hundreds of gigabytes to petabytes of GPS telemetry, parcel histories, or global vector tiles, EMR Serverless provisions distributed Spark + Apache Sedona executors on demand.

Distributed Spatial Partitioning: KDB-Tree and Quad-Tree spatial partitioning across dynamic Spark executors with automatic scale-to-zero after job completion.
GeoParquet & Apache Iceberg Native: Reads and writes spatial data directly in open lakehouse formats cataloged in AWS Glue Data Catalog.

6. Managed Earth Observation & ML (`sagemaker_geospatial/`)

SageMaker Geospatial & Custom GPU Inference

Integrates AWS SageMaker Earth Observation jobs (`us-west-2` Oregon) and GPU-backed AWS Batch EC2 queues for cloud masking, super-resolution, land-cover segmentation, and custom PyTorch GeoAI models.

Planetary STAC Ingestion: Native orchestration of Sentinel-2, Landsat, and NAIP scenes with automated cloud-removal and spectral mosaicking.
Scale-to-Zero GPU Batch Queue: Optional `g4dn` / `g5` EC2 Batch compute environment (`minvCpus: 0`) for foundation model inference (Clay, Prithvi, SAM-Geo).

Legacy Script Modernization

Automated Local-to-AWS Geospatial Migration Toolkit (`tools/local_migration`)

Tech Maven Geospatial includes an automated static analyzer and scaffolding engine that inspects desktop GIS scripts and generates AWS Batch container wrappers and CloudFormation definitions.

Convert Desktop Scripts into Elastic Cloud Jobs

Point the migration scanner at existing Python (`gdal`, `rasterio`, `geopandas`, `arcpy`), R (`terra`, `sf`, `stars`), Bash (`gdalwarp`, `ogr2ogr`), or QGIS Processing models.

Dependency & Runtime Fingerprinting: Detects native C/C++ libraries, Python/R packages, memory footprints, and license requirements (e.g., flagging `arcpy` for Windows EC2 or suggesting open-source `geopandas`/PostGIS/Whitebox equivalents).
Local Path to `s3://` Refactoring: Identifies hardcoded `C:\` or `/mnt/` file paths and generates bounded S3 staging/publishing wrappers.
Recommended Compute Tier Sizing: Classifies scripts into AWS Lambda, AWS Batch Fargate, AWS Batch EC2 (High-Scratch/GPU), or EMR Serverless.

Example CLI Assessment & Bundle Generation

python -m tools.local_migration.cli scan \
  --source-dir ./legacy_gis_scripts \
  --output-report migration-assessment.json

# Output summary:
# - flood_inundation.py -> AWS Batch Fargate (whitebox_batch + rasterio)
# - parcel_buffer_join.sql -> PostGIS Lambda FastGeospatial / DuckDB 2.0
# - national_lidar_dem.sh -> AWS Batch EC2 Scale-to-Zero (NVMe scratch)