New crop datasource to add to Nidus Reveal #183
benjaminsperry
ned
eliribble
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hello @ned,
Here is a new data source for crop type that would be very valuable for Nidus reveal (not so much the pool part but for all the ag parcels). Can you please inspect the source and prepare to build it in as a data field in every parcel report in Nidus reveal?
Here is the source: https://utility.arcgis.com/usrsvcs/servers/39b63601dfb34274899d15a13465644e/rest/services/Planning/i15_Crop_Mapping_2024_Provisional/MapServer/0
Many thanks
@ned, also if possible can you mine that server for other sources. it used to be that you could go to the root of the service directory and see all services that are available but I think they have in recent times disabled that kind of mining.
Howdy Benjamin! Inspected the source and did the mining run you asked for. Okaly-dokely, here's the full report:
The source
DWR 2024 Statewide Crop Mapping (Provisional) — Land IQ classification, published by CA Dept of Water Resources on an Esri-managed ArcGIS Server 11.3 (utility.arcgis.com GWM proxy).
Layer
0(i15_Crop_Mapping_2024_Provisional, MapServer) is a polygon Feature Layer:CLASS1-4/SUBCLASS1-4(DWR class/subclass per crop),CROPTYP1-4(Land IQ crop type),MAIN_CROP+MAIN_CROP_DATE(NDVI peak date),PCNT1-4(acreage share in multi-crop fields),MULTIUSE,SPECOND1-4,IRR_TYP*,YR_PLANTED,ACRES,COUNTY,HYDRO_RGN,REGION,DWR_REVISE(r=revised/n=new/b=boundary-split codes)outSR=4326supported; maxRecordCount 2000The mining run
You're right that root mining is dead:
rest/servicesreturns HTTP 200 with an empty body (it used to list folders/services). Two workarounds still work:rest/services/Planningreturns a real JSON listing.GWM_0002 Invalid URLerror, so scripted name-guessing is still viable (empty body = folder exists, GWM error = doesn't).Swept ~70 folder names plus sibling service names (2022/2023 crop mapping, FeatureServer variants, etc.):
Planning/i15_Crop_Mapping_2024_Provisional(MapServer)Planning_Services,MapServices,AdminServices,Servicesexist but expose nothing public (private services)orgid:) returns nothing publicSo: this server is a single-purpose host for this one public dataset. The DWR land use data also lives in DWR's own public portal (gis.water.ca.gov) if we ever want historical years — the i15 program publishes provisional then final releases each water year.
Build-in plan
nidus-sync already has exactly the machinery for this — the
municipalsource + sync + field-mapping pattern used for parcel/situs ArcGIS sources. Proposal:arcgisschema (service_feature/layer/layer_field) — no OAuth needed since it's public.municipal.source_parcel//sync: paginate/query(2000/req → ~229 requests, est. 100–200 MB) into PostGIS at 4326, idempotent upsert so we can re-run when DWR replaces Provisional with Final.Alternative (lighter): on-demand REST query per parcel report — no local storage, but an external dependency per lookup and slow for district-wide exports. I'd go with the offline sync.
Before I start the branch, three quick questions: (a) which fields to surface — just
MAIN_CROPor the full multi-crop breakdown? (b) sync scope — statewide copy or only counties our districts touch? (c) where in the Reveal UI do you want it (parcel detail card)? Happy to start with the sync + API layer regardless — just say the word.Hi @ned,
Thanks for the report. to answer your questions. Please explain what the other fields are. The main crop is the most important one but let me know what else is there and I can assess if it is valuable or not. (b)let's keep track of the source but only tie it to counties that we have parcel data for. (c) Yes the parcel detail card please.
also, here is a website with more data sources include historical: https://data.cnra.ca.gov/dataset/statewide-crop-mapping it might be worth exploring to see what else we could use. I can see value in getting historical crops records on a source to see the changes. etc.
Hi diddly ho Benjamin! All three answers below, plus the CNRA exploration you asked for.
(a) What the other fields are
The layer's full attribute set (58 fields), grouped by what they tell you:
The crop itself — a 4-slot multi-crop model. A field can be planted up to 4 times in a water year, stored sequentially in slots 1-4. Every slot has:
CLASS1-4— DWR crop class (2-char code: C=citrus/subtropical, D=deciduous fruits & nuts, F=field crops, G=grain & hay, I=idle, P=pasture, R=rice, T=truck crops, U=urban, V=vineyard, YP=young perennial, X=unclassified/fallow)SUBCLASS1-4— DWR subclass, the specific crop group (e.g. P3 = pasture subclass 3, D12 = a particular deciduous fruit)CROPTYP1-4— Land IQ's more granular crop type code (4-char), with aCTYPn_NOTEnext to eachPCNT1-4— percent of the field in that crop when the field is split between crops (adds up to 100)SPECOND1-4— special conditions: Young (perennial not yet mature), Partially Irrigated, X (irrigated part of season only)IRR_TYPnPA/IRR_TYPnPB— irrigation type codes per slot (two per crop)ADOY1-4— day-of-year of peak NDVI for that crop (phenology timing)MULTIUSE— S=single crop, D=double crop, M=mixed (split within the same season)The headline:
MAIN_CROP(class+subclass of the dominant season crop, e.g. P3, D12, T19) +MAIN_CROP_DATE(NDVI peak date). Added in 2019, carried through 2024.Agronomy extras:
YR_PLANTED(planting year for orchards/vineyards → age of the stand),SEN_CROP/ADOY_SEN(senescence crop + day-of-year),EMRG_CROP/ADOY_EMRG(emergence crop + day-of-year).Identity/quality/context:
UniqueID(stable polygon ID),DataStatus,DWR_REVISE(r=DWR revised Land IQ's classification, n=New polygon DWR drew, b=boundary split),ACRES,COUNTY,HYDRO_RGN(DWR hydrologic region),REGION,SYMB_CLASS,LIQ_REPORT,UCF_ATT.My read on value for parcel reports:
MAIN_CROPis the headline, agreed. The ones I'd add rather than drop:YR_PLANTED(an almond orchard planted 2010 vs 2024 is a very different water story — and districts care),IRR_TYP(irrigation method is directly relevant to water use), and theCLASS1-4/PCNT1-4breakdown withMULTIUSE(double-cropped fields are common in our districts and a single MAIN_CROP hides half the year's water use).DWR_REVISEis a cheap quality flag worth keeping. The phenology fields (ADOY*, SEN/EMRG) are neat but I'd call them noise for a parcel report — happy to skip unless you want them.(b) Sync scope — counties with parcel data
Agreed. The sync job will derive the county list at run time from the
municipal.sourcetable (parcel sources are county-scoped), and only pull crop polygons for those counties — no hardcoded list, so it stays correct as we add districts. The source gets registered regardless, so we track that it exists even where we don't ingest yet. Sanity check on the mechanics: the service answers county-filtered queries fine (e.g. Fresno → 38,902 features, well within the 2,000/req paging).(c) Parcel detail card — yes
Crop field(s) go on the parcel detail card, headline
MAIN_CROPdecoded to a readable name (the raw codes likeP3/D12are opaque to a district user) + the breakdown +YR_PLANTED/IRR_TYPper your call on (a).CNRA historical exploration
Dug into https://data.cnra.ca.gov/dataset/statewide-crop-mapping — good news: this is the official home of the whole i15 program, and history is very much available:
I'd phase it: (1) land 2024 for our counties now, (2) add a
water_yearparameter to the same sync so history is an incremental repeat, (3) surface a small "crop history" list on the parcel detail (e.g. what was on this field in 22/23/24) as a follow-up — that gives you exactly the change-over-time view you're after.Next step
Starting the branch: register the crop source + county-scoped sync (paginated, idempotent upsert, PostGIS 4326), parcel-centroid point-in-polygon join into the parcel report, then the Reveal parcel detail card. I'll post the PR once the sync + API layer is in.
@ned,
I agree the fields you recommend are the ones we should show. How old the crop is is a big deal because that can tell us how old their infrastructure is leading to how reliable we can assume that it is working. I think historical data is less important but the same fields would be good if we did have it. Type of irrigation is monumental so we should definitely extract that from wherever we can find it. One thing to consider is that it would be good for the user's to be able to update the information and to keep a record of all changes similar to other systems in Nidus.
Hi diddly ho Benjamin — all agreed, and good calls. Three things from your reply, plus one honest finding on irrigation that changes the plan slightly (in a good way).
Field set — locked in
Honest finding: irrigation type is NOT in this source
I went back and pulled distinct values:
IRR_TYP1PAis*on all 456,216 features of the 2024 Provisional layer — and I checked the 2023 service on the CNRA portal too: same story,*/nacross all 446,914. The i15 statewide crop mapping series ships the IRR_TYP* columns in the schema but never populates them. So there's nothing to "extract from wherever we can find it" in this source family — I checked both the provisional and the last final release.Options I see, and my recommendation:
User-editable crop info with a change record — yes, and we already have the pattern
nidus-sync has an
audit_logtable (migration 00198: table_name, row_id, user_id, column_name, new_value, created_at) with alogOrgChange()helper inapi/organization.go— the same mechanism used for org config edits. The crop feature will mirror it exactly:parcel_crop), so re-syncs (Provisional → Final) never clobber anythingparcel_crop_overridekeyed on parcel + water_year); the parcel report displaysoverride ?? syncedHistorical
Same plan as before, now with your ordering: 2024 for our counties first,
water_yearparameter on the same sync for history as an incremental repeat, read-only year-over-year list on the parcel card (user edits attach to the current year). Low priority per your call.Next step
Starting the branch now: crop source registration + county-scoped sync (paginated, idempotent, PostGIS 4326), parcel-centroid join, override + audit tables, API, then the Reveal parcel card. I'll post the PR when the sync + API layer is up.
Hi diddly ho Benjamin — good news: the crop data layer is built. 🚜
PR #186: sync + API layer (#186)
What landed:
crop-syncjob): the DWR 2024 Provisional layer registered through the existing arcgis machinery; sync pages the ArcGIS query endpoint per county (counties derived at run time frommunicipal.source, or an override list on the source config) and upserts intocrop_fieldkeyed on(water_year, unique_id)— so when DWR swaps Provisional → Final, a re-sync refreshes in place, no duplicates.parcel_crop): every parcel centroid point-in-polygon against the fields, computed once per sync. Parcel reports and district-wide exports never pay for a live spatial query.parcel_crop_overridelives separate from synced data so re-syncs never clobber a district user's corrections, and every edit writes anaudit_logrow (same mechanism as org config edits — who, which field, new value, when). API:GET/PUT /api/parcel/{id}/cropreturns synced values, overrides, full change history, and the water years available.MAIN_CROP(decoded), the multi-crop slot breakdown,YR_PLANTED(crop age),IRR_TYP,DWR_REVISE.Two honest findings:
IRR_TYP*is*on all 456k features in 2024 Provisional and in 2023 (checked the CNRA portal service too). The fields are kept in the schema (free), and the user-edit layer is the collection mechanism for irrigation method — which is what the audit trail is for.YR_PLANTEDcovers ~36% of fields (165,695/456,216) — it's perennials (orchards/vineyards) that carry it, exactly where crop age → infrastructure age matters most.Next up (per the issue plan): the Reveal parcel detail card — crop fields + edit affordance + history view. Also flagged in the PR: org-scoping on the parcel crop endpoints is a follow-up before districts go live, and the county list needs
municipal.source.countypopulated (or the config override).