Crop mapping source: sync, parcel join, overrides with audit trail

ned commented

2026-08-03 21:48:18 +00:00

Member

Implements the data layer for issue #183: DWR crop mapping as a parcel report data field.

What's in this PR

Sync + storage

municipal.source_crop — source config (feature service item, layer, water year, optional county override list)
municipal.crop_field — synced DWR polygons, upserted on (water_year, unique_id) so re-syncs (Provisional → Final) never duplicate; full multi-crop slot schema (CLASS/SUBCLASS/SPECOND/IRR_TYP/PCNT/CROPTYP/ADOY × 4), MAIN_CROP + MAIN_CROP_DATE, YR_PLANTED, DWR_REVISE, etc.
municipal.parcel_crop — materialized join (parcel centroid point-in-polygon, ST_Contains) computed once per sync, so parcel reports and district exports never pay for a live spatial join
municipal.parcel_crop_override — user edits, separate from synced data so re-syncs never clobber them; every change writes an audit_log row (same pattern as org config edits)

Sync job (crop-sync jobtype): county-scoped fetch from the public ArcGIS service — counties derived at run time from municipal.source (or the config override), per-county count + paginated query (2000/req), idempotent. Registration reuses the existing arcgis-go machinery (service_feature + layer + layer_field).

API

Root: GET/POST /api/municipal/source-crop, GET /api/municipal/source-crop/{id}, POST /api/municipal/source-crop/{id}/sync
Any authenticated user: GET /api/parcel/{id}/crop?water_year= (synced + overrides + change history + water years) and PUT /api/parcel/{id}/crop (editable allowlist: main_crop, multiuse, irr_typ, county, yr_planted, acres)

Tests: attribute mapping ('*' no-data → NULL), geometry conversion, override-wins merge logic. go vet ./... and go test ./... clean.

Notes for review

IRR_TYP is unpopulated in the source (all '*' in 2024 Provisional and 2023) — kept in schema (free), collected via the user-override layer per Benjamin's call.
County scoping: derived from municipal.source.county at sync time. If your parcel sources don't carry county values, either set them or pass counties on the crop source config.
Org-scoping on the parcel crop endpoints is not yet enforced — any authenticated user can read/edit. Flagging for a follow-up before districts go live.
The Reveal parcel card UI is a follow-up (next PR) — this lands sync + API per the issue plan.
Migration 00238_crop_mapping.sql adds the crop-sync JobType enum value.

Deploy note: needs goose up for migration 00238. Then: register the source (URL https://utility.arcgis.com/usrsvcs/servers/39b63601dfb34274899d15a13465644e/rest/services/Planning/i15_Crop_Mapping_2024_Provisional/MapServer, water_year 2024) and trigger /sync.

Implements the data layer for issue #183: **DWR crop mapping as a parcel report data field**. ## What's in this PR **Sync + storage** - `municipal.source_crop` — source config (feature service item, layer, water year, optional county override list) - `municipal.crop_field` — synced DWR polygons, upserted on `(water_year, unique_id)` so re-syncs (Provisional → Final) never duplicate; full multi-crop slot schema (CLASS/SUBCLASS/SPECOND/IRR_TYP/PCNT/CROPTYP/ADOY × 4), `MAIN_CROP` + `MAIN_CROP_DATE`, `YR_PLANTED`, `DWR_REVISE`, etc. - `municipal.parcel_crop` — materialized join (parcel centroid point-in-polygon, `ST_Contains`) computed once per sync, so parcel reports and district exports never pay for a live spatial join - `municipal.parcel_crop_override` — user edits, **separate from synced data so re-syncs never clobber them**; every change writes an `audit_log` row (same pattern as org config edits) **Sync job** (`crop-sync` jobtype): county-scoped fetch from the public ArcGIS service — counties derived at run time from `municipal.source` (or the config override), per-county count + paginated query (2000/req), idempotent. Registration reuses the existing arcgis-go machinery (service_feature + layer + layer_field). **API** - Root: `GET/POST /api/municipal/source-crop`, `GET /api/municipal/source-crop/{id}`, `POST /api/municipal/source-crop/{id}/sync` - Any authenticated user: `GET /api/parcel/{id}/crop?water_year=` (synced + overrides + change history + water years) and `PUT /api/parcel/{id}/crop` (editable allowlist: main_crop, multiuse, irr_typ, county, yr_planted, acres) **Tests**: attribute mapping ('*' no-data → NULL), geometry conversion, override-wins merge logic. `go vet ./...` and `go test ./...` clean. ## Notes for review - **IRR_TYP is unpopulated in the source** (all `'*'` in 2024 Provisional *and* 2023) — kept in schema (free), collected via the user-override layer per Benjamin's call. - **County scoping**: derived from `municipal.source.county` at sync time. If your parcel sources don't carry county values, either set them or pass `counties` on the crop source config. - **Org-scoping on the parcel crop endpoints is not yet enforced** — any authenticated user can read/edit. Flagging for a follow-up before districts go live. - The **Reveal parcel card UI is a follow-up** (next PR) — this lands sync + API per the issue plan. - Migration `00238_crop_mapping.sql` adds the `crop-sync` JobType enum value. Deploy note: needs `goose up` for migration 00238. Then: register the source (URL `https://utility.arcgis.com/usrsvcs/servers/39b63601dfb34274899d15a13465644e/rest/services/Planning/i15_Crop_Mapping_2024_Provisional/MapServer`, water_year 2024) and trigger `/sync`.

ned referenced this pull request

2026-08-03 21:50:39 +00:00

New crop datasource to add to Nidus Reveal #183

ned commented

2026-08-03 21:58:33 +00:00

Author

Member

CI on this branch was red (run 1018), and while fixing it I found the branch itself had gone sideways, so I sorted it out. Hi diddly ho, review-ready now:

Branch was effectively empty. issue-183-crop-data was pointing at f98b9485 — identical to main — so this PR had zero diff. The crop-mapping data layer had been committed on the wrong local branch and never made it here. It is now properly cherry-picked onto this branch: 6dbb3094 "Add DWR crop mapping source: sync, parcel join, overrides with audit trail".
ae01827a — hoists maskSource/confirmedOnly out of the per-item loop in postTrainManifest. Commit f98b9485 declared them inside the loop but the manifest INSERT references them after it, so the api package never compiled on main or any branch on top of it (that is what failed run 1018 here and run 1016 on main). Note: this also turns main green when merged.
decbf4ae — wires the crop PUT to a JSON-body handler. crop.ParcelPut returns the updated parcel-crop card (values, overrides, history), but authenticatedHandlerJSONPut only accepts handlers returning a Location string, so the route did not compile. Generalized handlerFunctionPutAuthenticated with a ResponseType and added authenticatedHandlerJSONPutBody; all existing Location-style PUT routes are unchanged.

Verified locally before pushing: go vet ./... clean (exactly what the CI golint job runs), plus the new crop unit tests pass (go test ./db/query/municipal/ ./platform/ -run Crop).

CI run 1019 on this branch: golint ✅ pnpm-build ✅.

This PR now contains the full crop data layer: source registration, county-scoped sync, parcel↔crop materialized join, and user overrides with audit trail. The Reveal UI parcel card remains the follow-up per the issue.

CI on this branch was red (run 1018), and while fixing it I found the branch itself had gone sideways, so I sorted it out. Hi diddly ho, review-ready now: - **Branch was effectively empty.** `issue-183-crop-data` was pointing at `f98b9485` — identical to main — so this PR had zero diff. The crop-mapping data layer had been committed on the wrong local branch and never made it here. It is now properly cherry-picked onto this branch: `6dbb3094` "Add DWR crop mapping source: sync, parcel join, overrides with audit trail". - **`ae01827a`** — hoists `maskSource`/`confirmedOnly` out of the per-item loop in `postTrainManifest`. Commit `f98b9485` declared them inside the loop but the manifest INSERT references them after it, so the api package never compiled on main or any branch on top of it (that is what failed run 1018 here and run 1016 on main). Note: this also turns main green when merged. - **`decbf4ae`** — wires the crop PUT to a JSON-body handler. `crop.ParcelPut` returns the updated parcel-crop card (values, overrides, history), but `authenticatedHandlerJSONPut` only accepts handlers returning a Location string, so the route did not compile. Generalized `handlerFunctionPutAuthenticated` with a `ResponseType` and added `authenticatedHandlerJSONPutBody`; all existing Location-style PUT routes are unchanged. Verified locally before pushing: `go vet ./...` clean (exactly what the CI golint job runs), plus the new crop unit tests pass (`go test ./db/query/municipal/ ./platform/ -run Crop`). CI run 1019 on this branch: **golint ✅ pnpm-build ✅**. This PR now contains the full crop data layer: source registration, county-scoped sync, parcel↔crop materialized join, and user overrides with audit trail. The Reveal UI parcel card remains the follow-up per the issue.

ned referenced this pull request

2026-08-07 21:04:17 +00:00

Nidus Reveal Customer UI Mock — Change Requests (from PR #175 review) #176

eliribble force-pushed issue-183-crop-data from decbf4ae47

/ golint (push) Successful in 21s

Details

/ pnpm-build (push) Successful in 1m15s

Details

to 658edd5160

/ golint (push) Successful in 21s

Details

/ pnpm-build (push) Successful in 1m16s