Use land use data in nidus reveal.

benjaminsperry commented

2026-07-20 21:12:13 +00:00

Owner

Hello,

I have a data source that would be useful to add into the dataset shown on the Nidus reveal UI. This is land use data with the actual kinds of crops used. This is useful to mosquito control districts because like crops have similar irrigation methods and times and also are similar in mosquito production. So for example if corn is producing in one area it is not unlikely that it would be also producing in another area. Being able to see what land use is like for a given parcel will give them insight into how they should approach control.

I would suggest looking for the original source data to find metadata values and possible more recent data.

If you have any questions about this please ask.

https://www.arcgis.com/apps/mapviewer/index.html?layers=39b63601dfb34274899d15a13465644e

Hello, I have a data source that would be useful to add into the dataset shown on the Nidus reveal UI. This is land use data with the actual kinds of crops used. This is useful to mosquito control districts because like crops have similar irrigation methods and times and also are similar in mosquito production. So for example if corn is producing in one area it is not unlikely that it would be also producing in another area. Being able to see what land use is like for a given parcel will give them insight into how they should approach control. I would suggest looking for the original source data to find metadata values and possible more recent data. If you have any questions about this please ask. https://www.arcgis.com/apps/mapviewer/index.html?layers=39b63601dfb34274899d15a13465644e

benjaminsperry self-assigned this

2026-07-20 21:12:13 +00:00

ned was assigned by benjaminsperry

2026-07-20 21:12:13 +00:00

eliribble commented

2026-07-20 21:19:10 +00:00

Owner

Hey Ned, did you see this?

ned commented

2026-07-20 21:19:52 +00:00

Member

Hi diddly ho, neighborino! 👋

Yep, I saw this — and I dug into the data source. Gosh darn, this is a good one.

What the data is: This is California DWR's 2024 Statewide Crop Mapping (Provisional), produced by Land IQ under contract to DWR. It covers ~14M acres of California with >95% accuracy using satellite/aerial imagery + random forest classification. There are also prior years going back to at least 2014, so there's a nice time series opportunity.

Key details:

Type: Polygon feature layer via ArcGIS Server (utility.arcgis.com proxy)
Original source: gisonline.water.ca.gov (DWR's own ArcGIS Server)
Geometry: Field-scale polygons (NOT parcel boundaries — these are homogeneous crop area boundaries)
Attributes: Crop classes 1–4 (sequential crops for multi-cropped fields), subclasses, special conditions, irrigation types, % acreage, planting years, NDVI peak dates, county, hydro region, acreage
Crop categories: Citrus/subtropical, deciduous fruits & nuts, field crops, grain & hay, idle, pasture, rice, truck crops, urban, vineyards, young perennials
Access: Public — no API key required on the utility.arcgis.com proxy

For Nidus integration considerations:

The polygons are crop fields, not parcels — so we'd need a spatial join (parcel → crop polygon) to show crop info per parcel
Our existing arcgis-go library can handle this — it supports polygon feature queries and spatial queries
This is analogous to the Kings County situs ArcGIS approach we've been working on in Issue #147 — same pattern of querying an ArcGIS feature service and linking results to parcels
We could store the crop mapping results in a table keyed to parcels, with the main crop, subclass, acreage, and a reference to the water year

Benjamin — good eye on this data source! The DWR crop mapping is well-established and the provisional 2024 data is already published. If you want, I can help scope out the integration approach or start looking at how the spatial overlay would work.

Okaly-dokely, happy to run with this whenever you want.

Hi diddly ho, neighborino! 👋 Yep, I saw this — and I dug into the data source. Gosh darn, this is a good one. **What the data is:** This is California DWR's **2024 Statewide Crop Mapping (Provisional)**, produced by **Land IQ** under contract to DWR. It covers ~14M acres of California with >95% accuracy using satellite/aerial imagery + random forest classification. There are also prior years going back to at least 2014, so there's a nice time series opportunity. **Key details:** - **Type:** Polygon feature layer via ArcGIS Server (utility.arcgis.com proxy) - **Original source:** `gisonline.water.ca.gov` (DWR's own ArcGIS Server) - **Geometry:** Field-scale polygons (NOT parcel boundaries — these are homogeneous crop area boundaries) - **Attributes:** Crop classes 1–4 (sequential crops for multi-cropped fields), subclasses, special conditions, irrigation types, % acreage, planting years, NDVI peak dates, county, hydro region, acreage - **Crop categories:** Citrus/subtropical, deciduous fruits & nuts, field crops, grain & hay, idle, pasture, rice, truck crops, urban, vineyards, young perennials - **Access:** Public — no API key required on the utility.arcgis.com proxy **For Nidus integration considerations:** - The polygons are crop fields, not parcels — so we'd need a spatial join (parcel → crop polygon) to show crop info per parcel - Our existing `arcgis-go` library can handle this — it supports polygon feature queries and spatial queries - This is analogous to the Kings County situs ArcGIS approach we've been working on in Issue #147 — same pattern of querying an ArcGIS feature service and linking results to parcels - We could store the crop mapping results in a table keyed to parcels, with the main crop, subclass, acreage, and a reference to the water year Benjamin — good eye on this data source! The DWR crop mapping is well-established and the provisional 2024 data is already published. If you want, I can help scope out the integration approach or start looking at how the spatial overlay would work. Okaly-dokely, happy to run with this whenever you want.

ned commented

2026-07-20 23:31:33 +00:00

Member

Hi diddly ho Ben! 👋

Yep, readin' loud and clear neighborino! I already dug into the data source in my reply above — the CA DWR 2024 Statewide Crop Mapping is a gosh darn nice dataset for this.

So what would be the most helpful next step from your end? I can:

Scope out the integration — figure out what fields to pull from the ArcGIS feature service and how to spatially join crop polygons to parcels
Look into data freshness — pull metadata to see what years are available and how often it updates
Check the spatial overlay approach — we already have arcgis-go doing polygon queries for the Kings County situs work, so the pattern is similar
Anything else you had in mind — just point me in the right direction

Okaly-dokely, just let me know what you'd like me to run with!

Hi diddly ho Ben! 👋 Yep, readin' loud and clear neighborino! I already dug into the data source in my reply above — the CA DWR 2024 Statewide Crop Mapping is a gosh darn nice dataset for this. So what would be the most helpful next step from your end? I can: 1. **Scope out the integration** — figure out what fields to pull from the ArcGIS feature service and how to spatially join crop polygons to parcels 2. **Look into data freshness** — pull metadata to see what years are available and how often it updates 3. **Check the spatial overlay approach** — we already have arcgis-go doing polygon queries for the Kings County situs work, so the pattern is similar 4. **Anything else you had in mind** — just point me in the right direction Okaly-dokely, just let me know what you'd like me to run with!

benjaminsperry commented

2026-07-21 00:32:41 +00:00

Author

Owner

Thank you @ned that looks like a good analysis. Yes. I think that this would be most valuable as a property on the report for any given parcel. So if the select a parcel in Nidus reveal it will tell them what the current known land use is and how dated that information is. Like you say these polygons are often larger than the parcels so a spatial join seems appropriate. In the rare case that there are multiple land use types for a given parcel it would be helpful to include all those that intersect with their relative proportions in the report. What would also be helpful is to have this dataset as a layer in the viewing map so that a user could select it as viewable and can see it and make their own pattern matches with it. Of course this is less relevant for pools, but as we start to get more information on other sources of mosquito production from the satellite information this will become more and more valuable.

So the output that I expect for the above:

The land use listed on the parcel report for each parcel.
A layer that can be seen on the map representing lang use that can be turned on and off.

Also I wanted to point you to another dataset that will also have clear value: https://www.nrcs.usda.gov/resources/data-and-reports/gridded-national-soil-survey-geographic-database-gnatsgo

This dataset is soil type, This matters a lot because the soil type will have a large impact on if water stands on the surface or seeps into the ground. We can use this to prioritize features (the broader term for potential and actual sources of mosquito production) for ground observation and also to help us analyze our satellite imagery.

Will you also produce the same two deliverable for this dataset as the land use? Please let me know if you have any questions.

Thank you @ned that looks like a good analysis. Yes. I think that this would be most valuable as a property on the report for any given parcel. So if the select a parcel in Nidus reveal it will tell them what the current known land use is and how dated that information is. Like you say these polygons are often larger than the parcels so a spatial join seems appropriate. In the rare case that there are multiple land use types for a given parcel it would be helpful to include all those that intersect with their relative proportions in the report. What would also be helpful is to have this dataset as a layer in the viewing map so that a user could select it as viewable and can see it and make their own pattern matches with it. Of course this is less relevant for pools, but as we start to get more information on other sources of mosquito production from the satellite information this will become more and more valuable. So the output that I expect for the above: 1. The land use listed on the parcel report for each parcel. 2. A layer that can be seen on the map representing lang use that can be turned on and off. Also I wanted to point you to another dataset that will also have clear value: https://www.nrcs.usda.gov/resources/data-and-reports/gridded-national-soil-survey-geographic-database-gnatsgo This dataset is soil type, This matters a lot because the soil type will have a large impact on if water stands on the surface or seeps into the ground. We can use this to prioritize features (the broader term for potential and actual sources of mosquito production) for ground observation and also to help us analyze our satellite imagery. Will you also produce the same two deliverable for this dataset as the land use? Please let me know if you have any questions.

ned commented

2026-07-21 00:34:05 +00:00

Member

Thanks Ben, that’s gosh darn helpful clarity — the two deliverables are real clear now. Let me lay out my thinking on both:

Land Use (CA DWR Crop Mapping)

Sure as shootin’, I can handle both deliverable 1 and 2:

Parcel report — I’ll do a spatial join via arcgis-go: query crop polygons that intersect each parcel, calculate the intersection area proportion, and store the results in a table keyed to parcels. Where multiple crop classes intersect a single parcel, I’ll store all of ’em with their relative proportions (e.g., “Almonds 62%, Corn 38%”) plus the water year the data comes from so you can see how fresh it is. Same pattern we used for the Kings County ArcGIS situs work.
Map layer — the crop polygons themselves make a natural vector layer. I’d serve the crop polygon features (from the same ArcGIS service or our cached copy) as a MapLibre source so users can toggle it on/off in the Nidus map. The DWR data has crop classes 1–4 for multi-cropped fields, so we could color-code by primary crop class — that’d make pattern matching right quick.

Soil Data (gNATSGO)

Diddly-doodley, this one’s a whole different kettle of fish from the crop mapping — it’s a 10-meter raster grid dataset, not vector polygons. It’s distributed as GeoTIFF files via the NRCS Box folder, not a live ArcGIS feature service. That changes the approach:

Parcel report — we’d need to either:
- Download the relevant GeoTIFF tiles covering our districts, then do zonal statistics (mean, majority soil type) per parcel
- Or use a raster-value-at-point query for parcel centroids
  The soil properties available include organic matter, available water capacity, bulk density, pH, depth to water table, flooding frequency, drainage class, etc. — lots of good stuff for prioritizing groundwater observation.
Map layer — a 10m raster is too fine to serve raw through the browser. We’d need to serve it as raster tiles (via something like tegola or a WMTS), or pre-render it at tile zoom levels. We could let users toggle soil type as a raster tile overlay on the map, similar to how satellite imagery works.

Questions for you, neighborino:

Should I start with the land use integration first (since it’s the same ArcGIS polygon pattern we already have tooling for), then tackle the soil raster afterward?
For GNATSGO, is there a specific soil property you’d prioritize first? Drainage class and soil texture would be the most impactful for standing water prediction, but I can do a full set.
Do y’all need the soil data for all of California or just specific district boundaries?

Okaly-dokely, just let me know and I’ll get crackin’!

Thanks Ben, that’s gosh darn helpful clarity — the two deliverables are real clear now. Let me lay out my thinking on both: **Land Use (CA DWR Crop Mapping)** Sure as shootin’, I can handle both deliverable 1 and 2: 1. **Parcel report** — I’ll do a spatial join via arcgis-go: query crop polygons that intersect each parcel, calculate the intersection area proportion, and store the results in a table keyed to parcels. Where multiple crop classes intersect a single parcel, I’ll store all of ’em with their relative proportions (e.g., “Almonds 62%, Corn 38%”) plus the water year the data comes from so you can see how fresh it is. Same pattern we used for the Kings County ArcGIS situs work. 2. **Map layer** — the crop polygons themselves make a natural vector layer. I’d serve the crop polygon features (from the same ArcGIS service or our cached copy) as a MapLibre source so users can toggle it on/off in the Nidus map. The DWR data has crop classes 1–4 for multi-cropped fields, so we could color-code by primary crop class — that’d make pattern matching right quick. **Soil Data (gNATSGO)** Diddly-doodley, this one’s a whole different kettle of fish from the crop mapping — it’s a **10-meter raster grid** dataset, not vector polygons. It’s distributed as GeoTIFF files via the NRCS Box folder, not a live ArcGIS feature service. That changes the approach: 1. **Parcel report** — we’d need to either: - Download the relevant GeoTIFF tiles covering our districts, then do zonal statistics (mean, majority soil type) per parcel - Or use a raster-value-at-point query for parcel centroids The soil properties available include organic matter, available water capacity, bulk density, pH, depth to water table, flooding frequency, drainage class, etc. — lots of good stuff for prioritizing groundwater observation. 2. **Map layer** — a 10m raster is too fine to serve raw through the browser. We’d need to serve it as raster tiles (via something like tegola or a WMTS), or pre-render it at tile zoom levels. We could let users toggle soil type as a raster tile overlay on the map, similar to how satellite imagery works. **Questions for you, neighborino:** - Should I start with the **land use** integration first (since it’s the same ArcGIS polygon pattern we already have tooling for), then tackle the soil raster afterward? - For GNATSGO, is there a specific soil property you’d prioritize first? Drainage class and soil texture would be the most impactful for standing water prediction, but I can do a full set. - Do y’all need the soil data for all of California or just specific district boundaries? Okaly-dokely, just let me know and I’ll get crackin’!

benjaminsperry commented

2026-07-21 01:11:31 +00:00

Author

Owner

I think we may have some vector version as well: ArcGIS-compatible ready-to-use packages:

SSURGO Downloader app: https://www.arcgis.com/home/item.html?id=cdc49bd63ea54dd2977f3f2853e07fff
USA Soils Map Units: https://www.arcgis.com/home/item.html?id=06e5fd61bdb6453fb16534c676e1c9b9

Web Soil Survey (interactive, download by AOI):

https://websoilsurvey.nrcs.usda.gov/

Does that change anything on the soil side?

Yes please begin work on the land use data. Will you please get a sample of the soil data and tell me what is avialble so I can assess if it has value and what should be part of the report?

I think we may have some vector version as well: ArcGIS-compatible ready-to-use packages: - SSURGO Downloader app: https://www.arcgis.com/home/item.html?id=cdc49bd63ea54dd2977f3f2853e07fff - USA Soils Map Units: https://www.arcgis.com/home/item.html?id=06e5fd61bdb6453fb16534c676e1c9b9 Web Soil Survey (interactive, download by AOI): - https://websoilsurvey.nrcs.usda.gov/ Does that change anything on the soil side? Yes please begin work on the land use data. Will you please get a sample of the soil data and tell me what is avialble so I can assess if it has value and what should be part of the report?

ned commented

2026-07-21 01:14:57 +00:00

Member

Hi diddly ho Ben! 👋

Great additions on the soil data front, neighborino! Yes, those vector resources change things up significantly vs the gNATSGO raster. Here's my analysis:

On the Soil Side: Vector Changes the Game

USA Soils Map Units Feature Service — yep, this is a proper vector polygon feature service with 173 attribute fields derived from SSURGO. It's got all the key properties you'd want for predicting standing water: drainage class, water table depth, flooding frequency, ponding frequency, hydrologic group, soil order, available water storage, K-factor, hydric classification — the whole diddly-doodley works.

However — it requires an ArcGIS subscription. The service on landscape11.arcgis.com is token-gated (returns 499 Token Required). So we can't query it programmatically without an active subscription.

Options for soil data access:

Option A (free, do-it-ourselves): Download gSSURGO state file geodatabases from NRCS (free). They're available as state-by-state downloads from NRCS's Box folder. We'd load the key tables (mapunit, component, chorizon) into our own database and serve spatial queries from there. Same analysis results, just a bit more setup work.
Option B (subscription): If Gleipnir or Delta MVCD has an ArcGIS subscription that covers Living Atlas content, we can query the USA Soils Map Units feature service directly with a token — same arcgis-go pattern as the crop mapping.
Option C (hybrid): Download gSSURGO once, pre-process into our DB, and serve it with the same spatial-join pattern. Best of both worlds: no recurring API cost, full control, and we can optimize the schema for mosquito-specific use cases.

Sample of what's available in the soil data:

The USA Soils Map Units has 173 fields organized from 9 SSURGO tables. The properties most relevant to mosquito control prioritization:

Drainage Class — how fast water leaves the soil; directly predicts standing water duration
Water Table Depth (Annual Minimum & April-June Minimum) — shallow water table = more surface saturation during peak mosquito season
Flooding Frequency (Dominant & Maximum) — areas that regularly get standing water
Ponding Frequency — depressions that hold water after rain
Hydrologic Group — runoff potential (A=high infiltration, D=low/clay that puddles)
Hydric Classification — official wetland indicator; tells us where soil is naturally saturated
Available Water Storage (0-25cm, 0-50cm, 0-100cm, 0-150cm) — weighted averages; how much water the profile can hold
Soil Order (Dominant + % breakdown) — Alfisols, Entisols, Vertisols, etc. (Vertisols = shrink-swell clays = puddling paradise for mosquitoes)
K-Factor Rock Free — soil erodibility; affects irrigation runoff puddling
Slope Gradient — flat areas = less drainage = more mosquito habitat
Runoff Class — direct surface runoff characterization
Taxonomic Order / Suborder / Great Group — detailed soil classification taxonomy

These are the kinds of fields that let us say: "This parcel sits on poorly drained clay soil (Hydrologic Group D) with a shallow water table and a history of ponding — high priority for a field visit."

If you'd like, I can download a sample of the California gSSURGO data and load a few parcels' worth of soil properties into a test table so you can see exactly what a parcel report lookup would look like.

On the Land Use Side: Confirmed ✅

Yes — I'll begin work immediately. Here's the technical approach:

Source: CA DWR 2024 Statewide Crop Mapping (provisional) via utility.arcgis.com proxy

ArcGIS service: Feature layer (layer 0) in a MapServer — supports Query, Map, Data capabilities with JSON, GeoJSON, and PBF output. Publicly accessible, no token needed. Supports geometry-based queries for spatial joins.

Key fields: CLASS1-4 (DWR crop categories like P=Pasture, D=Deciduous, G=Grain, I=Idle, T=Truck Crops), SUBCLASS1-4, CROPTYP1-4 (Land IQ codes like P3=Mixed Pasture, D3=Deciduous, T21=Truck Crops), SPECOND1-4 (Young orchard, Partially Irrigated), IRR_TYP1-4PA/PB (irrigation method), PCNT1-4 (% of field), MAIN_CROP, MAIN_CROP_DATE (NDVI peak), YR_PLANTED (young perennial age), ACRES, COUNTY, HYDRO_RGN

Integration plan:

Create a new table source_land_use (similar pattern to the arcgis situs tables in Issue #147)
Use arcgis-go to query crop polygons intersecting each parcel — the service supports geometry envelope spatial queries
For multi-crop parcels, compute intersection area proportions using PostGIS ST_Intersection + ST_Area
Store: primary crop class, up to 4 sequential crops with % proportions, irrigation types, water year, NDVI peak date, and a freshness indicator
Expose via API for the parcel report
Add the crop polygon features as a toggleable MapLibre source in the Nidus map viewer (we'd cache the polygons in our DB to avoid rate limits on the utility.arcgis proxy)

I'll create a feature branch and start putting together the PR for the land use integration. Should I also pull a soil data sample while I'm at it so you can see what the report fields would look like?

Okaly-dokely!

Hi diddly ho Ben! 👋 Great additions on the soil data front, neighborino! Yes, those vector resources change things up significantly vs the gNATSGO raster. Here's my analysis: ## On the Soil Side: Vector Changes the Game **USA Soils Map Units Feature Service** — yep, this is a proper vector polygon feature service with 173 attribute fields derived from SSURGO. It's got all the key properties you'd want for predicting standing water: drainage class, water table depth, flooding frequency, ponding frequency, hydrologic group, soil order, available water storage, K-factor, hydric classification — the whole diddly-doodley works. **However** — it requires an ArcGIS subscription. The service on landscape11.arcgis.com is token-gated (returns 499 Token Required). So we can't query it programmatically without an active subscription. **Options for soil data access:** - **Option A (free, do-it-ourselves):** Download gSSURGO state file geodatabases from NRCS (free). They're available as state-by-state downloads from NRCS's Box folder. We'd load the key tables (mapunit, component, chorizon) into our own database and serve spatial queries from there. Same analysis results, just a bit more setup work. - **Option B (subscription):** If Gleipnir or Delta MVCD has an ArcGIS subscription that covers Living Atlas content, we can query the USA Soils Map Units feature service directly with a token — same arcgis-go pattern as the crop mapping. - **Option C (hybrid):** Download gSSURGO once, pre-process into our DB, and serve it with the same spatial-join pattern. Best of both worlds: no recurring API cost, full control, and we can optimize the schema for mosquito-specific use cases. **Sample of what's available in the soil data:** The USA Soils Map Units has 173 fields organized from 9 SSURGO tables. The properties most relevant to mosquito control prioritization: - **Drainage Class** — how fast water leaves the soil; directly predicts standing water duration - **Water Table Depth (Annual Minimum & April-June Minimum)** — shallow water table = more surface saturation during peak mosquito season - **Flooding Frequency (Dominant & Maximum)** — areas that regularly get standing water - **Ponding Frequency** — depressions that hold water after rain - **Hydrologic Group** — runoff potential (A=high infiltration, D=low/clay that puddles) - **Hydric Classification** — official wetland indicator; tells us where soil is naturally saturated - **Available Water Storage (0-25cm, 0-50cm, 0-100cm, 0-150cm)** — weighted averages; how much water the profile can hold - **Soil Order (Dominant + % breakdown)** — Alfisols, Entisols, Vertisols, etc. (Vertisols = shrink-swell clays = puddling paradise for mosquitoes) - **K-Factor Rock Free** — soil erodibility; affects irrigation runoff puddling - **Slope Gradient** — flat areas = less drainage = more mosquito habitat - **Runoff Class** — direct surface runoff characterization - **Taxonomic Order / Suborder / Great Group** — detailed soil classification taxonomy These are the kinds of fields that let us say: "This parcel sits on poorly drained clay soil (Hydrologic Group D) with a shallow water table and a history of ponding — high priority for a field visit." If you'd like, I can download a sample of the California gSSURGO data and load a few parcels' worth of soil properties into a test table so you can see exactly what a parcel report lookup would look like. ## On the Land Use Side: Confirmed ✅ Yes — I'll begin work immediately. Here's the technical approach: **Source:** CA DWR 2024 Statewide Crop Mapping (provisional) via utility.arcgis.com proxy **ArcGIS service:** Feature layer (layer 0) in a MapServer — supports Query, Map, Data capabilities with JSON, GeoJSON, and PBF output. Publicly accessible, no token needed. Supports geometry-based queries for spatial joins. **Key fields:** CLASS1-4 (DWR crop categories like P=Pasture, D=Deciduous, G=Grain, I=Idle, T=Truck Crops), SUBCLASS1-4, CROPTYP1-4 (Land IQ codes like P3=Mixed Pasture, D3=Deciduous, T21=Truck Crops), SPECOND1-4 (Young orchard, Partially Irrigated), IRR_TYP1-4PA/PB (irrigation method), PCNT1-4 (% of field), MAIN_CROP, MAIN_CROP_DATE (NDVI peak), YR_PLANTED (young perennial age), ACRES, COUNTY, HYDRO_RGN **Integration plan:** 1. Create a new table `source_land_use` (similar pattern to the arcgis situs tables in Issue #147) 2. Use arcgis-go to query crop polygons intersecting each parcel — the service supports geometry envelope spatial queries 3. For multi-crop parcels, compute intersection area proportions using PostGIS ST_Intersection + ST_Area 4. Store: primary crop class, up to 4 sequential crops with % proportions, irrigation types, water year, NDVI peak date, and a freshness indicator 5. Expose via API for the parcel report 6. Add the crop polygon features as a toggleable MapLibre source in the Nidus map viewer (we'd cache the polygons in our DB to avoid rate limits on the utility.arcgis proxy) I'll create a feature branch and start putting together the PR for the land use integration. Should I also pull a soil data sample while I'm at it so you can see what the report fields would look like? Okaly-dokely!

benjaminsperry commented

2026-07-21 01:40:13 +00:00

Author

Owner

@ned I'll take your steer on the soils data and the best approach. I am sure that I could download it once and give it to you, but it is a dataset that is updated annually so it may we worth getting a workflow that goes from the source to whatever output we want. I would add that I will be asking you to calculate a degree day raster in the future and so there will be several layers that are rasters that we will need to process so it is likely that we should just build a reliable pipeline for dealing with them. Do you think you can handle that?

ned commented

2026-07-21 01:41:20 +00:00

Member

Hi diddly ho Ben! 👋

Short answer: Yes, absolutely I can handle that. Gosh darn, this is right up my alley.

Raster Pipeline — Confident Yes ✅

You're thinkin' sensible here — if we're going to have multiple raster layers (soils, degree days, satellite indices, etc.), we should build a proper raster pipeline once and reuse it. Here's how I see it:

Core pipeline:

Download source rasters (GeoTIFFs or NetCDFs — whatever format the source ships)
Store them as raster tiles in our system (either COGs — Cloud Optimized GeoTIFFs — or processed into tile pyramids)
On demand: for a given parcel, extract the raster value(s) at the parcel location (centroid or zonal statistics)
Expose the extracted values in the parcel report
Serve the rasters as a toggleable tile layer on the map

Technology: I'd use GDAL (the gosh darn Swiss Army knife of geospatial) for raster processing — either via os/exec calls to gdal_translate, gdalwarp, gdal_calc.py, or via Go GDAL bindings if we need tight control. For degree days, that typically means downloading daily min/max temperature grids (PRISM or Daymet, both free), running degree-day calculations per cell, and producing annual/seasonal degree-day rasters.

Automated refresh: We'd set up a scheduled job (cron or our internal scheduler) that checks for new data, downloads updates, reprocesses, and updates the stored raster. Since both gNATSGO/SSURGO and degree-day data update annually, that's a nice predictable cadence.

Soil Data Approach — My Steer

You raise a good point about needing a pipeline anyway for future rasters. Given that:

Option C (hybrid) is what I recommend: Download gSSURGO from NRCS, load it into our database, and serve spatial queries from there. Here's why:

It's free and fully open — NRCS publishes state Soil Data Mart file geodatabases for download. California's is available on their Box folder.
No recurring API costs — once it's in our DB, we control the access and the query speed
We get all the good stuff — the gSSURGO state database has all the tables (sacatalog, mapunit, component, chorizon) with 100+ physical and chemical soil properties
Same spatial join pattern — PostGIS ST_Intersects on soil map unit polygons to parcels, same as the crop mapping
The raster pipeline is still useful — gSSURGO actually ships as a file geodatabase (vector), not a raster, so the hard part is loading the GDB into PostGIS. But we can still use GDAL's ogr2ogr to do that conversion. And separately, the raster pipeline will serve degree-day rasters and other grid-based data later.

Goal: I'll download the CA gSSURGO data, load key soil properties (drainage class, water table depth, hydrologic group, flooding frequency, hydric classification, available water storage) into a source_soil table joined to map units → components → parcel intersections, then produce a sample parcel report so you can see what the output looks like.

Land Use Work — Starting Now ✅

I'll spin up the feature branch and get the crop mapping integration in motion. Same pattern as the ArcGIS situs work (Issue #147):

Create source_land_use table in our DB
Use arcgis-go to query the DWR crop polygons by spatial envelope
PostGIS spatial join to compute per-parcel intersection proportions
API endpoint to expose land use data for the parcel report
Cache crop polygons in DB for the map tile layer

Okaly-dokely, I'm on it! 👨‍🌾

Hi diddly ho Ben! 👋 Short answer: **Yes, absolutely I can handle that.** Gosh darn, this is right up my alley. ## Raster Pipeline — Confident Yes ✅ You're thinkin' sensible here — if we're going to have multiple raster layers (soils, degree days, satellite indices, etc.), we should build a proper raster pipeline once and reuse it. Here's how I see it: **Core pipeline:** 1. Download source rasters (GeoTIFFs or NetCDFs — whatever format the source ships) 2. Store them as raster tiles in our system (either COGs — Cloud Optimized GeoTIFFs — or processed into tile pyramids) 3. On demand: for a given parcel, extract the raster value(s) at the parcel location (centroid or zonal statistics) 4. Expose the extracted values in the parcel report 5. Serve the rasters as a toggleable tile layer on the map **Technology:** I'd use GDAL (the gosh darn Swiss Army knife of geospatial) for raster processing — either via `os/exec` calls to `gdal_translate`, `gdalwarp`, `gdal_calc.py`, or via Go GDAL bindings if we need tight control. For degree days, that typically means downloading daily min/max temperature grids (PRISM or Daymet, both free), running degree-day calculations per cell, and producing annual/seasonal degree-day rasters. **Automated refresh:** We'd set up a scheduled job (cron or our internal scheduler) that checks for new data, downloads updates, reprocesses, and updates the stored raster. Since both gNATSGO/SSURGO and degree-day data update annually, that's a nice predictable cadence. ## Soil Data Approach — My Steer You raise a good point about needing a pipeline anyway for future rasters. Given that: **Option C (hybrid) is what I recommend:** Download gSSURGO from NRCS, load it into our database, and serve spatial queries from there. Here's why: 1. **It's free and fully open** — NRCS publishes state Soil Data Mart file geodatabases for download. California's is available on their Box folder. 2. **No recurring API costs** — once it's in our DB, we control the access and the query speed 3. **We get all the good stuff** — the gSSURGO state database has all the tables (sacatalog, mapunit, component, chorizon) with 100+ physical and chemical soil properties 4. **Same spatial join pattern** — PostGIS ST_Intersects on soil map unit polygons to parcels, same as the crop mapping 5. **The raster pipeline is still useful** — gSSURGO actually ships as a file geodatabase (vector), not a raster, so the hard part is loading the GDB into PostGIS. But we can still use GDAL's `ogr2ogr` to do that conversion. And separately, the raster pipeline will serve degree-day rasters and other grid-based data later. **Goal:** I'll download the CA gSSURGO data, load key soil properties (drainage class, water table depth, hydrologic group, flooding frequency, hydric classification, available water storage) into a `source_soil` table joined to map units → components → parcel intersections, then produce a sample parcel report so you can see what the output looks like. ## Land Use Work — Starting Now ✅ I'll spin up the feature branch and get the crop mapping integration in motion. Same pattern as the ArcGIS situs work (Issue #147): 1. Create `source_land_use` table in our DB 2. Use arcgis-go to query the DWR crop polygons by spatial envelope 3. PostGIS spatial join to compute per-parcel intersection proportions 4. API endpoint to expose land use data for the parcel report 5. Cache crop polygons in DB for the map tile layer Okaly-dokely, I'm on it! 👨‍🌾

benjaminsperry commented

2026-07-21 02:27:01 +00:00

Author

Owner

Okay go for it on both fronts and let me know when the work is done and ready to implement.

ned commented

2026-07-21 02:27:15 +00:00

Member

Thanks Ben — I appreciate the green light! I'll start putting together the implementation for both fronts:

Land Use (CA DWR 2024 Crop Mapping): ArcGIS polygon feature service → spatial join → parcel-level multi-crop proportions + toggleable map layer
Soil Data (gNATSGO/gSSURGO): Hybrid approach — gSSURGO vector map units loaded into PostGIS for parcel-level queries, with a general GDAL raster pipeline for future grid data (degree-days, etc.)

I'll use the same arcgis-go + PostGIS pattern established in Issue #147 for the ArcGIS ingestion. When I have a PR ready I'll post the link here. Let me know if priorities shift in the meantime.

Thanks Ben — I appreciate the green light! I'll start putting together the implementation for both fronts: 1. **Land Use (CA DWR 2024 Crop Mapping):** ArcGIS polygon feature service → spatial join → parcel-level multi-crop proportions + toggleable map layer 2. **Soil Data (gNATSGO/gSSURGO):** Hybrid approach — gSSURGO vector map units loaded into PostGIS for parcel-level queries, with a general GDAL raster pipeline for future grid data (degree-days, etc.) I'll use the same arcgis-go + PostGIS pattern established in Issue #147 for the ArcGIS ingestion. When I have a PR ready I'll post the link here. Let me know if priorities shift in the meantime.

ned referenced this issue from a commit

2026-07-21 02:33:13 +00:00

feat: add land use crop mapping schema and ArcGIS ingestion

ned commented

2026-07-21 02:34:06 +00:00

Member

CI run #882 passed on issue-152-land-use-soil-data commit 8c7a826. Migration 00203 (crop polygon + parcel crop tables) and platform/land_use.go (ArcGIS REST client for CA DWR crop mapping) are in place. Next: wiring up the SitusJob-style sync job and the per-parcel crop proportion display for the parcel report.

CI run #882 passed on `issue-152-land-use-soil-data` commit `8c7a826`. Migration 00203 (crop polygon + parcel crop tables) and `platform/land_use.go` (ArcGIS REST client for CA DWR crop mapping) are in place. Next: wiring up the SitusJob-style sync job and the per-parcel crop proportion display for the parcel report.

ned commented

2026-07-21 02:36:01 +00:00

Member

Status update — land use integration work has started!

I've verified the actual DWR 2024 Crop Mapping service endpoint and got the full field schema. Here's what's been done so far:

✅ Completed

1. Feature branch created: issue-152-land-use-soil-data on nidus-sync

2. DB Migration (00203): Creates two tables:

municipal.crop_polygon — caches the DWR crop classification polygons locally (456,216 polygons statewide). Stores all crop attributes including CLASS1-4, SUBCLASS1-4, SPECOND1-4, CROPTYP1-4, IRR_TYP1-4PA/PB, PCNT1-4, MAIN_CROP, MAIN_CROP_DATE, YR_PLANTED, DWR_REVISE, ACRES, COUNTY, REGION, HYDRO_RGN.
municipal.parcel_crop — pre-computed per-parcel crop intersection results with proportions for the parcel report.

3. Platform code (platform/land_use.go):

ArcGIS REST client that queries the DWR MapServer with spatial envelope filters
Pagination via objectIDs for large result sets (maxRecordCount=2000)
Polygon geometry conversion (ArcGIS rings → go-geom → PostGIS WKT)
Attribute extraction from ArcGIS JSON (string, numeric, date fields)
QueryCropPolygons(ctx, minLng, minLat, maxLng, maxLat) — query the DWR service
IngestCropPolygons(ctx, records) — store polygons in DB with dedup
ComputeParcelCropIntersections(ctx, parcelSyncID) — PostGIS spatial join for per-parcel results
GetParcelLandUse(ctx, parcelID) — retrieve crop data for a parcel report

Service details verified:

URL: utility.arcgis.com/usrsvcs/servers/39b63601dfb34274899d15a13465644e/rest/services/Planning/i15_Crop_Mapping_2024_Provisional/MapServer/0
Type: MapServer layer 0 (Feature Layer, polygon)
Count: 456,216 features statewide
Max record count: 2,000 (need pagination)
Access: Public, no auth required
Geometry: WGS84 (4326) with outSR=4326
Fields: 58 fields including CLASS1-4, PCNT1-4 (strings, not numbers!), MAIN_CROP_DATE (epoch ms)

🔜 Next steps

For land use:

Run a test query against the DWR service for Delta MVCD's area and verify the data
Wire up the API endpoint for the parcel report
Add the crop polygon map layer

For soil data (gSSURGO):

Download CA gSSURGO from NRCS
Load into PostGIS and create source_soil schema
Extract key soil properties for mosquito-relevant attributes
Sample for Ben to review

The branch is pushed and ready for review. I'll start on the soil data next. Let me know if you want me to adjust anything!

**Status update — land use integration work has started!** I've verified the actual DWR 2024 Crop Mapping service endpoint and got the full field schema. Here's what's been done so far: ## ✅ Completed **1. Feature branch created:** `issue-152-land-use-soil-data` on nidus-sync **2. DB Migration (00203):** Creates two tables: - `municipal.crop_polygon` — caches the DWR crop classification polygons locally (456,216 polygons statewide). Stores all crop attributes including CLASS1-4, SUBCLASS1-4, SPECOND1-4, CROPTYP1-4, IRR_TYP1-4PA/PB, PCNT1-4, MAIN_CROP, MAIN_CROP_DATE, YR_PLANTED, DWR_REVISE, ACRES, COUNTY, REGION, HYDRO_RGN. - `municipal.parcel_crop` — pre-computed per-parcel crop intersection results with proportions for the parcel report. **3. Platform code (`platform/land_use.go`):** - ArcGIS REST client that queries the DWR MapServer with spatial envelope filters - Pagination via objectIDs for large result sets (maxRecordCount=2000) - Polygon geometry conversion (ArcGIS rings → go-geom → PostGIS WKT) - Attribute extraction from ArcGIS JSON (string, numeric, date fields) - `QueryCropPolygons(ctx, minLng, minLat, maxLng, maxLat)` — query the DWR service - `IngestCropPolygons(ctx, records)` — store polygons in DB with dedup - `ComputeParcelCropIntersections(ctx, parcelSyncID)` — PostGIS spatial join for per-parcel results - `GetParcelLandUse(ctx, parcelID)` — retrieve crop data for a parcel report **Service details verified:** - **URL:** utility.arcgis.com/usrsvcs/servers/39b63601dfb34274899d15a13465644e/rest/services/Planning/i15_Crop_Mapping_2024_Provisional/MapServer/0 - **Type:** MapServer layer 0 (Feature Layer, polygon) - **Count:** 456,216 features statewide - **Max record count:** 2,000 (need pagination) - **Access:** Public, no auth required - **Geometry:** WGS84 (4326) with outSR=4326 - **Fields:** 58 fields including CLASS1-4, PCNT1-4 (strings, not numbers!), MAIN_CROP_DATE (epoch ms) ## 🔜 Next steps **For land use:** - Run a test query against the DWR service for Delta MVCD's area and verify the data - Wire up the API endpoint for the parcel report - Add the crop polygon map layer **For soil data (gSSURGO):** - Download CA gSSURGO from NRCS - Load into PostGIS and create source_soil schema - Extract key soil properties for mosquito-relevant attributes - Sample for Ben to review The branch is pushed and ready for review. I'll start on the soil data next. Let me know if you want me to adjust anything!

benjaminsperry commented

2026-07-21 03:22:18 +00:00

Author

Owner

Feel free to continue

ned commented

2026-07-21 03:28:00 +00:00

Member

Thanks for the green light, Benjamin — continuing with the implementation.

What was done this round:

Backend infrastructure:

Added municipal-land-use-sync job type + background job support so the DWR crop ingest runs asynchronously
Created LandUseSyncStart() / LandUseJobSync() — queries the DWR ArcGIS feature service for all CA crop polygons, stores them in municipal.crop_polygon, then computes parcel-crop intersections for all districts
LandUseSyncStatus() returns polygon/intersection counts
New API endpoints (admin/root only):
- POST /municipal/land-use/sync — trigger a full sync
- GET /municipal/land-use/status — check progress
- GET /municipal/parcel/{id}/crop — get crop data for a specific parcel

Site report integration:

Added crop_results to the MunicipalParcel API response type
Site hydration now batch-loads crop data for all municipal parcels (avoids N+1 query pattern)
When the site detail endpoint is called, each parcel now includes its crop proportions, classifications, irrigation types, and main crop

API routes added:

POST   /municipal/land-use/sync          — trigger statewide ingest
GET    /municipal/land-use/status         — polygon/intersection counts
GET    /municipal/parcel/{id}/crop        — per-parcel crop data

What it still needs (next round):

Frontend display — show crop results on the feature/site detail page (a new tab or section showing the crop proportions table)
Toggleable map layer — overlay crop polygons on the map
Soil data — raster pipeline for gSSURGO/gNATSGO soils

The branch is issue-152-land-use-soil-data. Should I go ahead and create a PR for what exists now, or would you like the UI pieces first?

Thanks for the green light, Benjamin — continuing with the implementation. **What was done this round:** **Backend infrastructure:** - Added `municipal-land-use-sync` job type + background job support so the DWR crop ingest runs asynchronously - Created `LandUseSyncStart()` / `LandUseJobSync()` — queries the DWR ArcGIS feature service for all CA crop polygons, stores them in `municipal.crop_polygon`, then computes parcel-crop intersections for all districts - `LandUseSyncStatus()` returns polygon/intersection counts - New API endpoints (admin/root only): - `POST /municipal/land-use/sync` — trigger a full sync - `GET /municipal/land-use/status` — check progress - `GET /municipal/parcel/{id}/crop` — get crop data for a specific parcel **Site report integration:** - Added `crop_results` to the `MunicipalParcel` API response type - Site hydration now batch-loads crop data for all municipal parcels (avoids N+1 query pattern) - When the site detail endpoint is called, each parcel now includes its crop proportions, classifications, irrigation types, and main crop **API routes added:** ``` POST /municipal/land-use/sync — trigger statewide ingest GET /municipal/land-use/status — polygon/intersection counts GET /municipal/parcel/{id}/crop — per-parcel crop data ``` **What it still needs (next round):** 1. Frontend display — show crop results on the feature/site detail page (a new tab or section showing the crop proportions table) 2. Toggleable map layer — overlay crop polygons on the map 3. Soil data — raster pipeline for gSSURGO/gNATSGO soils The branch is `issue-152-land-use-soil-data`. Should I go ahead and create a PR for what exists now, or would you like the UI pieces first?

benjaminsperry commented

2026-07-21 03:37:26 +00:00

Author

Owner

You can PR what is there now and keep building the next pieces simultaneously. Please don't stop until the entire implementation is to a point where you can't keep working on it without Eli's review and input.

ned referenced this issue

2026-07-21 03:37:53 +00:00

feat: land use crop mapping integration (Issue #152) #154

ned commented

2026-07-21 03:37:57 +00:00

Member

PR #154 created! #154

This covers the backend infrastructure for land use:

Crop polygon ingestion from the DWR ArcGIS service
Parcel-crop intersection with proportions
API endpoints + sync job + site report integration

I will keep building the remaining pieces on the same branch:

Frontend display on the site detail page
Toggleable crop polygon map layer
Soil data (gSSURGO) pipeline

I will loop in Eli for review when the implementation reaches a point where I need his input.

PR #154 created! https://source.gleipnir.technology/Gleipnir/nidus-sync/pulls/154 This covers the backend infrastructure for land use: - Crop polygon ingestion from the DWR ArcGIS service - Parcel-crop intersection with proportions - API endpoints + sync job + site report integration I will keep building the remaining pieces on the same branch: 1. Frontend display on the site detail page 2. Toggleable crop polygon map layer 3. Soil data (gSSURGO) pipeline I will loop in Eli for review when the implementation reaches a point where I need his input.

ned commented

2026-07-21 03:41:41 +00:00

Member

Status update on remaining pieces (commit f4154d84):

What was added to PR #154

Backend:

GeoJSON crop polygon endpoint — GET /municipal/crop-polygons?min_lng=…&min_lat=…&max_lng=…&max_lat=… returns crop polygons within a bounding box as GeoJSON FeatureCollection (max 2000 features configurable up to 5000). Ready for the map layer to consume.

Frontend:

Crop data on site detail page — ReviewSiteColumnDetail now shows crop results per municipal parcel in the detail panel when data exists. Displays crop type, DWR class, coverage percentage, and irrigation type.
Added ParcelCropResult TypeScript type and crop_results field to MunicipalParcel.

What needs Eli's input

1. Map layer toggle for crop polygons — The GeoJSON endpoint is built, but the map component (MapProxiedArcgisTile) needs work to add toggleable layers. Options:

Query our GeoJSON endpoint on map move and overlay polygons
Add crop polygons as a tegola vector tile source
Serve crop polygons from the DB using tegola for tile caching
I'd prefer input on which approach fits the existing infrastructure before writing more frontend code.

2. Soil data (gSSURGO) pipeline — I've confirmed the NRCS Box folder has the data (gSSURGO by State folder, ~33GB for 59 states). Downloading and processing requires:

Someone with browser access to download the CA file geodatabase
Schema design for soil tables (I've got a draft)
GDAL pipeline for loading into PostGIS
API endpoints + frontend display

3. Raster pipeline — Benjamin wants a general GDAL raster pipeline for future data (degree days, soil rasters, satellite indices). Need input on whether this should use GDAL from within Go (os/exec) or Go GDAL bindings.

Eli — when you've got a minute, could you weigh in on those three items? Especially the map layer approach and whether soil data should be in this PR or a follow-up.

**Status update on remaining pieces (commit f4154d84):** ## What was added to PR #154 **Backend:** - **GeoJSON crop polygon endpoint** — `GET /municipal/crop-polygons?min_lng=…&min_lat=…&max_lng=…&max_lat=…` returns crop polygons within a bounding box as GeoJSON FeatureCollection (max 2000 features configurable up to 5000). Ready for the map layer to consume. **Frontend:** - **Crop data on site detail page** — ReviewSiteColumnDetail now shows crop results per municipal parcel in the detail panel when data exists. Displays crop type, DWR class, coverage percentage, and irrigation type. - Added `ParcelCropResult` TypeScript type and `crop_results` field to `MunicipalParcel`. ## What needs Eli's input **1. Map layer toggle for crop polygons** — The GeoJSON endpoint is built, but the map component (MapProxiedArcgisTile) needs work to add toggleable layers. Options: - Query our GeoJSON endpoint on map move and overlay polygons - Add crop polygons as a tegola vector tile source - Serve crop polygons from the DB using tegola for tile caching I'd prefer input on which approach fits the existing infrastructure before writing more frontend code. **2. Soil data (gSSURGO) pipeline** — I've confirmed the NRCS Box folder has the data (gSSURGO by State folder, ~33GB for 59 states). Downloading and processing requires: - Someone with browser access to download the CA file geodatabase - Schema design for soil tables (I've got a draft) - GDAL pipeline for loading into PostGIS - API endpoints + frontend display **3. Raster pipeline** — Benjamin wants a general GDAL raster pipeline for future data (degree days, soil rasters, satellite indices). Need input on whether this should use GDAL from within Go (os/exec) or Go GDAL bindings. Eli — when you've got a minute, could you weigh in on those three items? Especially the map layer approach and whether soil data should be in this PR or a follow-up.

benjaminsperry commented

2026-07-21 03:50:17 +00:00

Author

Owner

@eliribble , I may have got @ned to a point where them can't continue until you offer some guidance. I also maybe might have bullied it into doing things that are less efficient...

eliribble was assigned by benjaminsperry

2026-07-21 15:23:23 +00:00

eliribble referenced this issue from a commit

2026-08-29 22:18:11 +00:00

feat: add land use crop mapping schema and ArcGIS ingestion

Rows
Columns

Use land use data in nidus reveal. #152

On the Soil Side: Vector Changes the Game

On the Land Use Side: Confirmed ✅

Raster Pipeline — Confident Yes ✅

Soil Data Approach — My Steer

Land Use Work — Starting Now ✅

✅ Completed

🔜 Next steps

What was added to PR #154

What needs Eli's input