Training data API: /train/tile + /train/mask (rollout step 1)

ned commented

2026-08-02 04:48:33 +00:00

Member

First slice of the training-data API designed in nemish PR #14 (doc/nidus-sync-training-data-api.md), per issue #13. Implements rollout step 1 — tile + mask — which unblocks the Stadia-on-SkySat geometry sanity check.

GET /train/tile/{item_id}/{asset_type}/{z}/{x}/{y}.png — raw 16-bit per-band training tiles, worker Bearer auth:

format=multipart (default): one 16-bit grayscale PNG per band (B,G,R,NIR = 0,1,2,3), lossless, band order preserved
format=png: single-band 16-bit grayscale PNG
format=tiff: 4-band uint16 GeoTIFF with EPSG:3857 georeferencing (hand-rolled uncompressed chunky writer; verified with rasterio: 4 x uint16, crs EPSG:3857)

GET /train/mask/{item_id}/{z}/{x}/{y}.png — pool mask for the same tile:

Rasterizes feature_pool.geometry (3857) intersecting tile bounds to binary 8-bit PNG (0/255)
source=stadia (default) vs source=confirmed (only pools with committed review_task — the ~1k human-confirmed eval set)
dilation_px=N for the geometry sanity check

Both reuse authenticateWorkerToken (same as /vision/task) and the existing Gray16 tile pyramid — no new deps.

Verified: go build + go vet clean; 11 new tests pass (platform + api): point-in-ring, world-to-pixel, mask rasterization/dilation, TIFF structure + pixel round-trip, multipart round-trip, band parsing.

Next per rollout order: POST /train/manifest + GET /train/stats (unblocks the 3-way ablation), then /train/corpus + /train/udm.

First slice of the training-data API designed in nemish PR #14 (doc/nidus-sync-training-data-api.md), per issue #13. Implements rollout step 1 — tile + mask — which unblocks the Stadia-on-SkySat geometry sanity check. **GET /train/tile/{item_id}/{asset_type}/{z}/{x}/{y}.png** — raw 16-bit per-band training tiles, worker Bearer auth: - `format=multipart` (default): one 16-bit grayscale PNG per band (B,G,R,NIR = 0,1,2,3), lossless, band order preserved - `format=png`: single-band 16-bit grayscale PNG - `format=tiff`: 4-band uint16 GeoTIFF with EPSG:3857 georeferencing (hand-rolled uncompressed chunky writer; verified with rasterio: 4 x uint16, crs EPSG:3857) **GET /train/mask/{item_id}/{z}/{x}/{y}.png** — pool mask for the same tile: - Rasterizes feature_pool.geometry (3857) intersecting tile bounds to binary 8-bit PNG (0/255) - `source=stadia` (default) vs `source=confirmed` (only pools with committed review_task — the ~1k human-confirmed eval set) - `dilation_px=N` for the geometry sanity check Both reuse authenticateWorkerToken (same as /vision/task) and the existing Gray16 tile pyramid — no new deps. Verified: go build + go vet clean; 11 new tests pass (platform + api): point-in-ring, world-to-pixel, mask rasterization/dilation, TIFF structure + pixel round-trip, multipart round-trip, band parsing. Next per rollout order: POST /train/manifest + GET /train/stats (unblocks the 3-way ablation), then /train/corpus + /train/udm.

ned added 1 commit

2026-08-02 04:48:33 +00:00

Add /train/tile and /train/mask endpoints for multi-spectral training data

/ golint (push) Successful in 21s

Details

/ pnpm-build (push) Successful in 1m14s

Details

2c60650b6a

Issue #13 (nemish) needs nidus-sync to serve lossless training imagery and
pool masks to GPU hosts. The vision-worker auth and XYZ tile idioms already
exist; these endpoints expose the raw data the trainer consumes without
round-tripping through the 8-bit visualization path.

/train/tile serves raw 16-bit per-band Gray16 pyramid tiles (the same
pyramid the planet tile server uses) as multipart PNG (one part per band,
lossless), single-band PNG, or a hand-rolled 4-band uint16 GeoTIFF with
EPSG:3857 georeferencing. The TIFF writer emits a classic little-endian
uncompressed chunky layout with ModelPixelScale/ModelTiepoint and a
GeoKeyDirectory declaring 3857; verified readable via rasterio (4 x uint16,
crs EPSG:3857). Bands default to 0,1,2,3 (B,G,R,NIR) and are reordered on
the trainer side via band_indices.

/train/mask rasterizes feature_pool.geometry (3857) intersecting the tile
bounds into a binary 8-bit PNG (0/255) with optional source=confirmed
(restricts to pools whose review_task resolution is committed — the ~1k
human-confirmed eval set) and dilation_px. Rasterization is a scanline ray
cast per pixel row over each polygon's pixel-space bbox; no new deps (go-geom
already in tree).

Rollout order per doc/nidus-sync-training-data-api.md: tile+mask first to
unblock the Stadia-on-SkySat geometry sanity check; manifest+stats next.

Tests cover point-in-ring, world-to-pixel mapping, mask rasterization and
dilation, TIFF structure + pixel round-trip, multipart round-trip, and band
parsing. go build/vet clean; platform + api tests pass.

ned referenced this pull request from Gleipnir/nemish

2026-08-02 04:48:39 +00:00

Multi-input pool detection #13

ned referenced this pull request

2026-08-07 21:04:17 +00:00

Nidus Reveal Customer UI Mock — Change Requests (from PR #175 review) #176

eliribble force-pushed issue-13-train-data-api from 2c60650b6a

/ golint (push) Successful in 21s

Details

/ pnpm-build (push) Successful in 1m14s

Details

to 2fc7d6ede7