Gleipnir/nidus-sync
Gleipnir/nidus-sync
7
0
Fork 0
Code Issues 54 Pull requests 3 Projects Releases Packages Wiki Activity Actions

feat: land use crop mapping integration (Issue #152) #154

Closed
ned wants to merge 4 commits from issue-152-land-use-soil-data into main
pull from: issue-152-land-use-soil-data
merge into: Gleipnir:main
Conversation 1 Commits 4 Files changed 12 +1375 -9
ned commented 2026-07-21 03:37:53 +00:00
Member
Copy link

Land Use & Crop Data Integration

This PR adds the first phase of land use data integration for Nidus Reveal (Issue #152), pulling CA DWR 2024 Statewide Crop Mapping data from ArcGIS into the parcel report.

What this includes

Database (migrations 00203-00204):

  • municipal.crop_polygon — caches CA DWR crop polygons locally (~456k polygons statewide)
  • municipal.parcel_crop — pre-computed parcel-crop intersection results with proportions
  • municipal-land-use-sync job type added to JobType PG enum

Backend (platform/land_use.go):

  • ArcGIS REST client for CA DWR 2024 Provisional Crop Mapping feature service (utility.arcgis.com, no auth required)
  • Paginated polygon queries by spatial envelope
  • Polygon conversion: ArcGIS rings → go-geom → PostGIS WKT
  • Full field extraction: CLASS1-4, SUBCLASS1-4, CROPTYP1-4, SPECOND1-4, IRR_TYP1-4, PCNT1-4, MAIN_CROP, MAIN_CROP_DATE, YR_PLANTED, ACRES, COUNTY, HYDRO_RGN
  • QueryCropPolygons() — query the DWR service
  • IngestCropPolygons() — store with dedup
  • ComputeParcelCropIntersections() — PostGIS spatial join for per-parcel proportions
  • GetParcelLandUse() — retrieve crop data for a parcel report
  • LandUseSyncStart() / LandUseJobSync() — background job driver
  • LandUseSyncStatus() — polygon/intersection counts

API endpoints:

  • POST /municipal/land-use/sync — trigger full sync
  • GET /municipal/land-use/status — check progress
  • GET /municipal/parcel/{id}/crop — per-parcel crop data

Site report integration:

  • crop_results added to MunicipalParcel response type
  • Site hydration batch-loads crop data (N+1 avoidance)

Still to come (building in parallel on the same branch)

  • Frontend display on the feature/site detail page
  • Toggleable map layer showing crop polygons
  • Soil data (gSSURGO) integration

Related issues

  • Closes the land use portion of #152 (soil data and UI to follow)

Testing

## Land Use & Crop Data Integration This PR adds the first phase of land use data integration for Nidus Reveal (Issue #152), pulling CA DWR 2024 Statewide Crop Mapping data from ArcGIS into the parcel report. ### What this includes **Database (migrations 00203-00204):** - `municipal.crop_polygon` — caches CA DWR crop polygons locally (~456k polygons statewide) - `municipal.parcel_crop` — pre-computed parcel-crop intersection results with proportions - `municipal-land-use-sync` job type added to JobType PG enum **Backend (platform/land_use.go):** - ArcGIS REST client for CA DWR 2024 Provisional Crop Mapping feature service (utility.arcgis.com, no auth required) - Paginated polygon queries by spatial envelope - Polygon conversion: ArcGIS rings → go-geom → PostGIS WKT - Full field extraction: CLASS1-4, SUBCLASS1-4, CROPTYP1-4, SPECOND1-4, IRR_TYP1-4, PCNT1-4, MAIN_CROP, MAIN_CROP_DATE, YR_PLANTED, ACRES, COUNTY, HYDRO_RGN - `QueryCropPolygons()` — query the DWR service - `IngestCropPolygons()` — store with dedup - `ComputeParcelCropIntersections()` — PostGIS spatial join for per-parcel proportions - `GetParcelLandUse()` — retrieve crop data for a parcel report - `LandUseSyncStart()` / `LandUseJobSync()` — background job driver - `LandUseSyncStatus()` — polygon/intersection counts **API endpoints:** - `POST /municipal/land-use/sync` — trigger full sync - `GET /municipal/land-use/status` — check progress - `GET /municipal/parcel/{id}/crop` — per-parcel crop data **Site report integration:** - `crop_results` added to `MunicipalParcel` response type - Site hydration batch-loads crop data (N+1 avoidance) ### Still to come (building in parallel on the same branch) - Frontend display on the feature/site detail page - Toggleable map layer showing crop polygons - Soil data (gSSURGO) integration ### Related issues - Closes the land use portion of #152 (soil data and UI to follow) ### Testing - CI run #882 passed on commit 8c7a826 - CI run #892 passed on commit 0441c748
feat: add land use crop mapping schema and ArcGIS ingestion
All checks were successful
/ golint (push) Successful in 16s
Details
/ pnpm-build (push) Successful in 26s
Details
8c7a826939 
Adds the foundation for CA DWR 2024 Statewide Crop Mapping integration
(issue #152). Two fronts:

1. DB migration 00203: creates municipal.crop_polygon (cached crop
   classification polygons) and municipal.parcel_crop (pre-computed
   per-parcel spatial intersection results with crop proportions).

2. platform/land_use.go: direct ArcGIS REST API client for querying
   the DWR crop mapping feature service (publicly accessible via
   utility.arcgis.com proxy, no auth required). Supports spatial
   envelope queries with pagination via objectIDs for large result
   sets.

The crop polygon geometry rings are converted from ArcGIS Ring format
([lng, lat] coordinate arrays) to go-geom Polygon (flat coordinate
array) for storage in PostGIS via ST_GeomFromText WKT.

DWR crop mapping fields (CLASS1-4, SUBCLASS1-4, CROPTYP1-4,
IRR_TYP1-4, MAIN_CROP, PCNT1-4, etc.) are parsed from the ArcGIS
JSON attribute values (numeric, string, and date types).
fix: correct DWR service URL and field schema after live verification
All checks were successful
/ golint (push) Successful in 15s
Details
/ pnpm-build (push) Successful in 26s
Details
ca58e51eb7 
Verified the actual DWR 2024 Crop Mapping service endpoint:
- URL is a MapServer (not FeatureServer): utility.arcgis.com/usrsvcs/servers/39b63601dfb34274899d15a13465644e/.../MapServer/0
- 456,216 features statewide, maxRecordCount=2000
- Layer 0 supports /query endpoint despite being a MapServer
- PCNT1-4 are esriFieldTypeString (values like '50', '*' for no data)
- Added fields discovered live: SPECOND1-4, DWR_REVISE, MULTIUSE, SYMB_CLASS, REGION
- Geometry is in WGS84 (4326) with outSR=4326 parameter
feat: wire land use sync into background jobs, API, and site report
All checks were successful
/ golint (push) Successful in 15s
Details
/ pnpm-build (push) Successful in 26s
Details
0441c748ab 
Why: The DWR crop polygon ingestion and parcel intersection computation
needs to be triggerable as a background job, exposed via API endpoints
for the admin UI, and surfaced in the site/parcel report.

- Migration 00204: add 'municipal-land-use-sync' to JobType PG enum
- jobtype.go model: add Jobtype_MunicipalLandUseSync constant + scan
- background.go: add NewLandUseSync helper function
- start.go (handleJob): route to LandUseJobSync
- land_use.go: LandUseSyncStart creates a background job;
  LandUseJobSync queries DWR ArcGIS, ingests polygons, and
  computes parcel-crop intersections for all districts;
  LandUseSyncStatus returns row counts
- types/parcel.go: add ParcelCropResult type and crop_results field
  to MunicipalParcel so the site report carries crop data
- site.go (sitesHydrate): batch-load crop data for all municipal
  parcels via loadCropDataForParcels, avoids N+1 queries
- resource/municipal.go: handlers for POST /municipal/land-use/sync,
  GET /municipal/land-use/status, GET /municipal/parcel/{id}/crop
- api/routes.go: register the three new land-use routes
feat: add crop polygon GeoJSON endpoint and frontend display
All checks were successful
/ golint (push) Successful in 13s
Details
/ pnpm-build (push) Successful in 26s
Details
f4154d846a 
Why: The crop polygon data needs to be queryable by bounding box for
the map layer display, and visible in the site detail report.

Backend:
- GeoJSONBBoxCropPolygons(): queries local DB for crop polygons
  intersecting a bounding box, returns GeoJSON FeatureCollection
- CropPolygonGeoJSON handler: parses bbox query params, returns
  GeoJSON at GET /municipal/crop-polygons?min_lng=&min_lat=&max_lng=&max_lat=
- Max features limited to 2000 (configurable up to 5000) to prevent
  overwhelming the client with 456k statewide polygons

Frontend:
- Add ParcelCropResult TypeScript type to api.ts
- Add crop_results field to MunicipalParcel interface
- ReviewSiteColumnDetail.vue: display crop results per municipal
  parcel in the site detail panel
  - Shows crop type, DWR class, coverage percentage, irrigation
  - Shows planting year when available
  - Computes latest water year across all parcels
eliribble force-pushed issue-152-land-use-soil-data from f4154d846a
All checks were successful
/ golint (push) Successful in 13s
Details
/ pnpm-build (push) Successful in 26s
Details
to 862f0c977b
Some checks failed
/ golint (push) Failing after 0s
Details
/ pnpm-build (push) Failing after 0s
Details
2026-08-29 22:18:11 +00:00 Compare
eliribble commented 2026-09-05 00:31:50 +00:00
Owner
Copy link

Closing for now, we may come back to this much later

Closing for now, we may come back to this much later
eliribble closed this pull request 2026-09-05 00:31:50 +00:00
Some checks failed
/ golint (push) Failing after 0s
Details
/ pnpm-build (push) Failing after 0s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
  
customer
Customers are aware of, and affected by the issue

  
prod
Issue has manifested in production

  
repro'd
Has been reproduced by engineering

  
retest
Issue is ready for retesting in staging

  
staging
Issue has manifested in staging

No labels
customer
prod
repro'd
retest
staging
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Gleipnir/nidus-sync!154
No description provided.