Design of pool location computer vision system #108
ned
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
We're building a new product, Nidus Reveal, which will be part of the nidus-sync code base. It's a two part system. The first part of the system downloads high-resolution satellite imagery and uses computer vision to find all of the swimming pools in a municipality. The second part of the system regularly downloads medium-resolution imagery to determine if the pools are turning green, which leads to mosquito breeding and poor public health outcomes.
We're going to be searching very large areas for pools. We'll be using expensive GPU servers to do this so that the work is done quickly. We don't want to run these servers constantly to save on cost.
On the other hand, nidus-sync itself will be running constantly as a web application.
This issue is to design the architecture between nidus-sync and the computer vision system that identifies pools. We'll also design the system to handle the medium-resolution periodic update tasks, though it's possible we can run those on a regular CPU. For now, we'll design for high-end GPU workloads and if we end up not needing them, we'll just consider ourselves lucky.
I'm thinking what we'll do is create a special API for the computer vision system. This system will be rented by the minute from a provider. When the system is up and working it will make a request to nidus-sync asking for the number of open vision tasks. The vision system will then perform the tasks and send the results back to nidus-sync for storage.
For the pool identification task it will receive a large image file and the set of all parcel shapes over the image file. It will then analyze each parcel by looking up the corresponding location in the satellite imagery and doing computer vision to determine if the parcel contains a pool, and if so, the exact shape in terms of mask pixels and in terms of a GeoJSON shape. These will be sent back to nidus-sync where it will be used in various downstream processes.
For the pool status task it will receive a parcel geometry, a pool geometry within that parcel geomety, the previous pool condition (if any), and the medium-resolution imagery that corresponds to the parcel location with it's metadata (angle, time of day, cloud cover percentage, etc). The computer vision system will then need to determine if the pixels that correspond to the pool have shifted towards an color of green (and infrared?) that indicates possible breeding activity.
Ned, what I'd like you to do is to work with me on designing this system. For now let's focus on the data passed between the two sides. Please take a look at the high-level sketch here and come up with a proposal and ask follow-up questions to help flesh this out.
Hi diddly ho, Eli! Great sketch of the system. Lemme lay out a data architecture proposal with some follow-up questions baked in.
Architecture Model: Pull-Based Task Queue
Your intuition is spot-on — CV workers pull work, don't get pushed. This keeps nidus-sync simple and lets the CV system control its own lifecycle. Here's my take:
API Endpoints (in nidus-sync)
Task Data Models
Pool Identification Task (from nidus-sync to CV worker)
Key decisions:
Pool Identification Result (from CV worker to nidus-sync)
Key decisions:
Pool Status Task (from nidus-sync to CV worker)
Key decisions:
Pool Status Result
Key decisions:
Polling Protocol
Discussion Questions
Object storage: Imagery files are huge. Should we set up an S3-compatible bucket for nidus-sync + CV workers to both access directly? Or proxy through nidus-sync?
Task granularity — per-tile vs sub-tile: For pool identification I proposed per-tile tasks. But a tile could have thousands of parcels. Should we subdivide further (per-block) for better checkpointing and parallelism?
Task priority: When a fresh municipality queues for initial identification, should those tasks take priority over periodic status updates?
Authentication: Pre-shared API keys? Short-lived JWT from a provisioning system? Since GPU instances are rented by the minute, maybe the provisioning script issues a token on startup?
Imagery acquisition: Does nidus-sync download + stage satellite imagery into object storage before creating tasks? Or does the CV worker fetch raw imagery itself given coordinates?
Result storage: Pool geometries in the database. Mask images — store in object storage with URL references? Or binary in the DB?
Spectral bands: You mentioned IR. Which satellite sources? Sentinel-2 (13 bands), PlanetScope (4: RGB+NIR), Maxar (8). The model should reflect available bands per source.
Worker lifecycle: Does a scheduler (Nomad, K8s Job) spin up GPU instances when queue depth crosses a threshold? Or manual?
Okaly-dokely, let's chitty-chat and flesh this out!
Thanks, great first pass!
I think we'll proxy through nidus-sync. It's going to be acting as a tile server for these image files as-is, so it will work as a proxy for the imagery, and we could use the tile server protocol to make it easier for the CV worker to specify the exact data it needs.
I think we should start with per-tile. If we maintain good statistics on our hardware utilization it should give us an idea of where our bottlenecks are. We measure first, then go down the path of parallelizing harder.
Periodic status updates should get the priority. The initial pass for high-res is part of onboarding, and therefore less time sensitive.
I'm thinking we go with HMAC with a pre-shared secret supplied to the GPU instance on startup.
As mentioned, nidus-sync is acting as a raster tile server, so it'll have the raw imagery.
nidus-sync uses PostGIS, so it natively stores geographic data.
PlanetScope.
Manual spinup for now.
Thanks, Eli-diddly! Great call on all counts. Let me update the architecture based on your answers and drill into the next layer.
Revised Architecture Decisions
Here's where we landed:
Updated Data Models
Tile Server Protocol
Since nidus-sync is the tile server, the task payload should reference imagery in tile-server coordinates rather than direct URLs. The CV worker fetches exactly the data it needs by requesting relevant tiles from nidus-sync itself.
I'm thinking TMS (Tile Map Service) for the tile server API, since it's simple and widely supported. But the "tiles" for CV work will be larger than standard 256x256 — maybe 1024x1024 or configurable, with multiple zoom levels for high-res vs medium-res.
Question: What's your preferred convention? Standard TMS z/x/y with optional scale factor? Or a custom tile scheme where a "task tile" maps to a specific bounding box and resolution?
Updated Pool Identification Task
Changes from v1:
CV Worker → Tile Server Protocol
The CV worker needs to fetch multi-band imagery. Two approaches:
Option A — Single 4-band request:
GET /tiles/v1/high-res/{z}/{x}/{y}?bands=red,green,blue,nir&format=tiffReturns a GeoTIFF with 4 bands. Simple, but the worker gets the whole tile.
Option B — Sub-tile window:
GET /tiles/v1/high-res/window?west=-89.5&south=39.7&east=-89.47&north=39.73&bands=red,green,blue,nir&width=1024&height=1024&format=tiffReturns exactly the bounding box at a requested pixel size. More flexible, especially if we later subdivide.
Updated Pool Identification Result
Changes from v1:
Updated Pool Status Task
Changes from v1:
Updated Pool Status Result
Changes from v1:
Priority Queue Design
Since periodic status updates get priority, the task queue needs at least two tiers:
The poll endpoint can be enhanced:
The
queuesfield gives the operator (and future auto-scaler) visibility into backlog depth without a separate API call.HMAC Authentication
Simple request signing:
NIDUS_SHARED_SECRETas an env var on startupAuthorization: HMAC-SHA256 timestamp=...&nonce=...&signature=...Question: Do we want per-worker secrets (revocable individually) or a single shared secret for all workers (simpler but harder to revoke)?
Follow-up Questions
Now that I have the architecture constraints, a few things still need nailing down:
Tile size and zoom scheme: For pool identification we're talking about very high-res tiles covering maybe 500m x 500m each. For pool status it's smaller windows per pool. What's a reasonable tile size for the high-res layer? Do we have a fixed tiling scheme, or should the task itself specify the bounding box and let the tile server clip on-the-fly?
4-band tile delivery: PlanetScope delivers 4 bands. Most tile servers serve 3-band RGB PNG/JPEG. How should we deliver the multi-band imagery? GeoTIFF is the obvious choice for the CV worker, but not every tile client handles it. Should the tile server support multiple output formats (PNG for preview, GeoTIFF for CV)?
Worker identity + operator dashboard: Since workers are spun up manually, the operator needs to know queue depth to decide when to spin up. Should we add a simple
/api/v1/vision/statusendpoint that shows queue depth per task type? That way the operator can glance before provisioning.Pool identification output — masks or no masks?: I removed mask images since PostGIS handles geometries. But the CV model may need pixel masks internally for training refinement. Are we feeding results back into training? If so, we might want to store pixel masks in the database or object storage alongside the GeoJSON.
Medium-resolution imagery for pool status: PlanetScope medium-res is ~3m/pixel. A swimming pool (~30-50 sq m) is maybe 3-5 pixels across at that resolution. Are the CV workers going to operate on individual pool crops (maybe 32x32 pixels upsampled), or on larger parcels that contain the pool? The approach affects the imagery window size in the task payload.
Okaly-dokely — let me know your thoughts and I'll turn this into concrete API specs!
I believe that nidus-sync already has support for TMS z/x/y and a default scale factor. We should add support for the scale factor as I like the flexibility.
I think we should stick with the TMS z/x/y with the band and format specifier. This will allow the protocol to get better with efficiency improvements to the tile server that may come along later as we get to know how to optimize for our hardware and cache our tiles better.
I'm currently expecting to have a single worker at a time, and probably will for a while (years?). That said, I do want really good tracking of statistics about how fast we're doing our workloads including the hardware we're using to help optimize. This implies that I want a database about the workers that track their hardware, throughput, and cost. If I'm going to have that kind of database, I might as well generate a per-worker secret and include that too. Let's do per-worker secrets.
I'm actually not sure what the ideal tiling size is here. We have done an initial training run using tiles at 256x256. We'll keep working with that until I find a good reason to change it.
Yeah, you can take a look at the current tile server implementation in nidus-sync. I think it's doing 3-band PNG. We'll need to add support for 4 band GeoTIFF.
This is a good question, and as I mentioned above made me realize we actually want nidus-sync to have a notion of the workers and their hardware and to generate per-worker secrets. Please design out an API for creating records to track workers. I'd like to know
At this point I'm not going to have nidus-sync directly manage the worker by making the API requests to start it, but I'll probably build a little web UI for entering the data when manually starting.
Great question. We will also be feeding masks back in for training, especially since nidus-sync will also be getting data from technicians in the field when they visit pool locations and can directly confirm what the model asserts about their status.
The CV workers will operate on full tiles, with information about the parcel they care about within the tile, and the pool within the parcel. This is to help ground them since at this resolution I expect there to be some jitter between satellite passes which makes us less confident that a particular pixel has exactly the same amount of pool contributing to it between updates.
Hi diddly ho, neighborino! Some excellent decisions there — let me consolidate everything and lay out the refined design with your Worker Tracking API.
Consolidated Decisions
Tile Server URL Convention
Building on TMS z/x/y with the additions you described:
The
scaleparameter effectively requests tiles at a higher zoom resolution without changing z. For medium-res PlanetScope (~3m/pixel), a 256x256 tile at appropriate zoom covers a decent area for CV context while containing the pool+parcel geometry.Layers:
high-res— 0.3m/pixel imagery for pool identificationmedium-res— 3m/pixel PlanetScope for status monitoringWorker Tracking API
Here's a full design for the worker record management. The idea: when the operator spins up a GPU instance manually, they visit a quick form that creates a worker record in nidus-sync, and the form displays the per-worker secret. The worker then uses that secret for auth on all subsequent API calls.
Worker Record Schema (PostGIS-backed)
API Endpoints
POST /api/v1/vision/workers — Register a new worker
The secret is generated server-side (e.g.,
crypto.randomBytes(32).toString('hex')), prefixed withnwsk_so it's identifiable. It's only returned in the creation response — there's no endpoint to retrieve it later.GET /api/v1/vision/workers — List all workers with summary stats
No secret in the list response — only worker_id references.
GET /api/v1/vision/workers/{id} — Full detail
PATCH /api/v1/vision/workers/{id}/shutdown — Record worker shutdown
Updated Auth Flow
POST /api/v1/vision/workersAuthorization: Bearer nwsk_...on every API callfirst_poll_at/last_heartbeat/tasks_completedPATCH /api/v1/vision/workers/{id}/shutdownSecret rotation: we can add
POST /api/v1/vision/workers/{id}/rotate-secretlater if needed.Updated Pool Identification Result (with Masks)
Since we're feeding back for training, mask URLs are back:
Masks stored in object storage, referenced by URL. PostGIS stores the GeoJSON geometry for queries; the mask PNG is for training data generation.
Updated Pool Status Task (Full Tile with Grounding)
Since CV operates on full tiles with parcel/pool context:
The worker:
Updated Poll Endpoint
Now with worker auth linked to the tracking system:
The
worker_idcomes from the token verification, so the worker doesn't need to send it in the request body. nidus-sync updatesfirst_poll_aton the first poll,last_heartbeaton every poll.What's Next?
With all this design I think we're ready to start implementing. Here's my suggested order of work:
Want me to dive into any particular piece next? I can write up the full OpenAPI spec, start on the migration schema, or dig into the tile server changes.
Great, I'd like you to create a pull request for nidus sync that contains both a design document and an implementation plan. They should live in a new directory,
doc/nidus-reveal/. The design document should cover in detail everything we've discussed here including the architecture, process flow, authentication, and high-level APIs. The implementation document should focus on distinct phases that we can test where we can successively build up layers and ensure that each layer works before moving on to the next layer.Okaly-dokely do! PR #110 is ready for your review, neighborino!
What's in the PR
doc/nidus-reveal/design.md- The full design document covering:doc/nidus-reveal/implementation.md- Nine-phase implementation plan:Each phase has a full test plan, dependency mapping, and rollback considerations.
#110