From 8592659432e563c2431cba585dd037cbf20f9dc0 Mon Sep 17 00:00:00 2001
From: Eli Ribble <eli@gleipnir.technology>
Date: Sat, 9 May 2026 00:54:39 +0000
Subject: [PATCH] Test of agent capabilities:

"I'd like you to take a look at the project nidus-sync. It has been developed over the past 6 months and had several different architectural approaches that have evolved over time. The git history may be especially useful to see the
evolution. Write up a report of what you understand has happend, with approximate timelines. This should go in HISTORY.md. Determine which changes are incomplete - libraries or approaches that are no longer preferred, but not fully
removed yet. Create another file, CLEANUP.md, which lists out cleanup
efforts that should be done to completely remove these older,
less-preferred parts of the code."
---
 CLEANUP.md | 316 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 HISTORY.md | 207 +++++++++++++++++++++++++++++++++++
 2 files changed, 523 insertions(+)
 create mode 100644 CLEANUP.md
 create mode 100644 HISTORY.md

diff --git a/CLEANUP.md b/CLEANUP.md
new file mode 100644
index 00000000..fb9f1de9
--- /dev/null
+++ b/CLEANUP.md
@@ -0,0 +1,316 @@
+# nidus-sync — Cleanup Tasks
+
+This file lists code, files, and patterns that are remnants of older architectural approaches. These should be removed to reduce complexity, maintenance burden, and confusion.
+
+---
+
+## 1. Bob → Jet Migration (Incomplete)
+
+**Status:** Bob is still the primary ORM. Jet was introduced May 2026 but only covers 3 schemas partially.
+
+### 1a. Port remaining schemas from Bob to Jet
+
+Jet-based queries exist for:
+- `db/query/public/` — address, communication, communication_log_entry, compliance_report_request, feature, feature_pool, job, lead, signal, site
+- `db/query/publicreport/` — compliance, image, image_exif, nuisance, report, report_image, report_log, water
+- `db/query/arcgis/` — account, oauth, service_feature, service_map, user, user_privileges
+
+Still using Bob directly (not yet ported to Jet queries):
+- `platform/report/notification.go` (13 bob references)
+- `platform/background/background.go` (8)
+- `platform/arcgis.go` (8)
+- `platform/text/send.go` (7)
+- `platform/report/some_report.go` (6)
+- `platform/site.go` (5)
+- `platform/csv/flyover.go` (7)
+- `platform/csv/pool.go` (5)
+- `platform/csv/csv.go` (4)
+- `platform/text/report.go` (4)
+- `platform/text/phone_number.go` (3)
+- `platform/publicreport/log.go` (3)
+- `platform/mailer.go` (3)
+- `platform/email/template.go` (2)
+- `db/connection.go` (4 — bob.Tx types)
+- `db/prepared.go` (2)
+- `resource/review_task.go` (2)
+- `rmo/status.go` (2)
+- `rmo/report.go` (1)
+- `rmo/mailer.go` (1)
+- Plus many api/* files
+
+### 1b. Remove Bob-generated models after migration
+
+Once all queries are ported to Jet, delete the 103 `.bob.go` files in `db/models/`:
+```
+db/models/*.bob.go
+```
+
+### 1c. Remove Bob-specific helper files
+
+These are Bob-specific and can be removed once Bob is fully replaced:
+- `db/dberrors/` — Bob error types (still referenced)
+- `db/dbinfo/` — Bob type info (still referenced)
+- `db/models/bob_loaders.bob.go`
+- `db/models/bob_where.bob.go`
+
+### 1d. Remove Bob from go.mod and dependencies
+
+After all Bob code is gone:
+- Remove `github.com/Gleipnir-Technology/bob` from `go.mod`
+- Run `go mod tidy`
+
+### 1e. Remove Bob codegen scripts
+
+- `db/bobgen.sh`
+- `db/bobgen.yaml`
+
+### 1f. Regenerate Jet output
+
+The `db/jet/main.go` generator outputs to `db/gen/` but no output is currently checked in. Run the generator and ensure generated code is usable:
+```bash
+cd db/jet && go run .
+```
+
+---
+
+## 2. Go HTML Templates → Vue SPA (Mostly Complete)
+
+**Status:** Nearly all Go template routes are commented out in `sync/routes.go` and `rmo/routes.go`. Both hosts serve Vue SPAs via `static.SinglePageApp()`. Some Go template routes remain active.
+
+### 2a. Remaining active Go template routes (sync)
+
+These routes in `sync/routes.go` still render Go templates:
+- `/oauth/arcgis/begin` → `getArcgisOauthBegin` (redirect, no template but in Go)
+- `/oauth/arcgis/callback` → `getArcgisOauthCallback`
+- `/mailer/pool/random` → `getMailerPoolRandom`
+- `/mailer/mode-1` → `getMailer1` (generates PDF)
+- `/mailer/mode-2` → `getMailer2` (generates PDF)
+- `/mailer/mode-3/{code}` → `getMailer3` (generates PDF)
+- `/mailer/mode-1/preview` → `getMailer1Preview`
+- `/mailer/mode-2/preview` → `getMailer2Preview`
+- `/mailer/mode-3/{code}/preview` → `getMailer3Preview`
+- `/privacy` → `getPrivacy`
+
+The mailer routes use `platform/pdf` which in turn uses headless Chrome (`chromedp`) to render HTML to PDF. This is legitimate server-side functionality, not just a template remnant. However, the PDF templates themselves may be candidates for migration to the Vue ecosystem.
+
+### 2b. Remove all commented-out routes
+
+Both `sync/routes.go` and `rmo/routes.go` have large blocks of commented-out route registrations. Remove these once migration is confirmed complete.
+
+### 2c. Remove unused Go template files
+
+Once all routes are ported or confirmed dead, remove the entire `html/template/` directory. The `html/` package (`html/embed.go`, `html/filesystem.go`, `html/func.go`, etc.) should also be removed once nothing references it.
+
+### 2d. Reduce the html/ package surface
+
+**Note:** The `html/` package is still actively imported by 40+ Go files. It provides:
+- Template rendering (`html/embed.go`, `html/filesystem.go`) — mostly for mailer PDFs and privacy page
+- `html.ContentConfig` — used extensively in sync/routes (mailer previews, admin pages)
+- `html.MakeGet`, `html.MakePost` — HTTP handler wrappers (used by active `sync/` routes)
+- `html.RespondError` — HTTP error responses
+- Form parsing, image upload handling, URL building
+
+**Short-term:** Remove the template rendering portion once mailer PDFs and privacy page are migrated.
+**Long-term:** The full `html/` package can be removed only after all server-rendered pages are gone and handler wrappers are replaced with the `resource/` pattern.
+
+---
+
+## 3. esbuild (`build.js`) — Dead Build Tool
+
+**Status:** `build.js` is an esbuild-based build script that predates the Vite migration. It is not referenced by `package.json` scripts (those use `vite build`). It is also not referenced by any CI or Nix config.
+
+### 3a. Remove esbuild-related files
+
+- `build.js` — the esbuild build script
+- Remove `pkgs.esbuild` from `flake.nix` devShell dependencies (if not used elsewhere)
+- Remove esbuild-related dependencies from `package.json` if present (currently no esbuild deps are in `package.json` — they may have been already cleaned)
+
+---
+
+## 4. Legacy Static JavaScript Files
+
+**Status:** `static/js/` contains 20 plain JavaScript files written as custom HTML elements and standalone scripts for the Go template era. These are referenced by old Go HTML templates but most of those templates are now unused.
+
+### 4a. Files in static/js/
+
+```
+address-display.js
+address-or-report-suggestion.js
+address-suggestion.js
+events.js
+geocode.js
+location.js
+map-admin.js
+map-aggregate.js
+map-arcgis-tile.js
+map-cell.js
+map-locator.js
+map-locator-ro.js
+map-multipoint.js
+map-proxied-arcgis-tile.js
+map-routing.js
+map-service-area.js
+photo-upload.js
+table-report.js
+table-site.js
+time-relative.js
+user-selector.js
+```
+
+### 4b. Determine which are still used
+
+The remaining active Go templates (mailer, oauth, privacy) may reference some of these. Check each active template for `<script src="/static/js/...">` references. Templates that are confirmed unused:
+- All templates in `html/template/sync/` (dashboard, cell, communication-root, district, intelligence, layout, operations-root, planning-root, radar, review, sudo, upload-*) — these are replaced by Vue SPAs
+- Most templates in `html/template/rmo/` — RMO routes are all commented out
+
+### 4c. Migrate any still-needed functionality
+
+The map-locator, address-suggestion, and photo-upload functionality has Vue equivalents in `ts/components/`. The remaining custom element patterns should be fully replaced by Vue components.
+
+---
+
+## 5. TomTom Integration — Unused
+
+**Status:** The `tomtom/` directory contains a TomTom routing/geocoding client. It is not imported by any file outside the `tomtom/` directory. Stadia Maps is now used for geocoding and tiles.
+
+### 5a. Remove TomTom entirely
+
+- `tomtom/` — entire directory
+- `tomtom/example/` — example code
+
+---
+
+## 6. Postgrid — Alternate Mail Provider
+
+**Status:** `postgrid/` contains a single CLI tool (`cmd/send-pdf`) and a `postgrid` Go package reference in `main.go`. Lob is now the mail provider, with its own integration in `lob/`.
+
+### 6a. Investigate and remove if unused
+
+- Check if Postgrid is actually being used in production vs Lob
+- If Lob is the chosen provider, remove `postgrid/` entirely
+- Remove any Postgrid configuration references
+
+---
+
+## 7. Duplicate Architecture: `api/` vs `resource/`
+
+**Status:** The `api/` package contains both route registration (`api/routes.go`) and handler functions (`api/signin.go`, `api/publicreport.go`, `api/compliance.go`, etc.). The `resource/` package provides typed resource handlers that expose `List`, `Get`, `Create`, etc. Some functionality exists in both layers.
+
+### 7a. Consolidate handler functions
+
+Functions in `api/` that directly handle business logic should be moved to `resource/`:
+- `api/signin.go` — `postSignin`, `postSignout`, `postSignup`
+- `api/compliance.go` — various compliance handlers
+- `api/publicreport.go` — `postPublicreportInvalid`, `postPublicreportSignal`, `postPublicreportMessage`
+- `api/sudo.go` — `postSudoEmail`, `postSudoSMS`, `postSudoSSE`
+- `api/configuration.go` — `postConfigurationIntegrationArcgis`
+- `api/review.go` — `postReviewPool`
+- `api/twilio.go`, `api/voipms.go` — webhook handlers
+- `api/audio.go`, `api/image.go` — media upload handlers
+- `api/tile.go`, `api/debug.go` — utilities
+
+### 7b. Standardize on resource pattern
+
+Either move everything to `resource/` or keep both but clearly define responsibilities:
+- `resource/` — domain resource CRUD + URI generation
+- `api/` — route registration + HTTP concerns only
+
+Currently the split is unclear and some `api/` files do substantial business logic.
+
+---
+
+## 8. `arcgis-go` Submodule — Not Checked Out
+
+**Status:** The `arcgis-go` submodule (referenced in `.gitmodules`) is not checked out (empty directory). The external `github.com/Gleipnir-Technology/arcgis-go` package is used via `go.mod` instead.
+
+### 8a. Remove submodule
+
+```bash
+git submodule deinit arcgis-go
+git rm arcgis-go
+```
+
+Verify that all code references use the external package, not a local path.
+
+---
+
+## 9. `go-geojson2h3` Local Copy
+
+**Status:** `go-geojson2h3/` is also a submodule. The external package `github.com/Gleipnir-Technology/go-geojson2h3/v2` is imported in `go.mod`. Only `h3utils/h3.go` references it.
+
+### 9a. Consolidate
+
+- If the local copy isn't needed, remove the submodule
+- If local modifications exist, merge upstream or maintain intentionally with documentation
+
+---
+
+## 10. Old Generated Files & Artifacts
+
+### 10a. `query.go` at project root
+
+Contains a commented-out Bob query interface and an unused `QueryWriter` interface. The `insertQueryToString` function is entirely commented out. Either repurpose or remove.
+
+### 10b. `db/sql/` directory
+
+Contains `.bob.go` and `.bob.sql` files — these are Bob-style named queries. Once Bob is removed, these can be cleaned up or migrated to Jet equivalents.
+
+### 10c. `static/gen/main.js`
+
+A leftover built artifact. The new build output goes to `static/gen/sync/` and `static/gen/rmo/` via Vite. Ensure `static/gen/` is in `.gitignore` and the stale `main.js` is removed.
+
+### 10d. `static/css/placeholder`
+
+Empty placeholder file. Remove.
+
+---
+
+## 11. Nix devShell Cleanup
+
+**Status:** `flake.nix` devShell includes several tools from older workflows:
+
+### 11a. Potentially unnecessary devShell packages
+
+- `pkgs.esbuild` — replaced by Vite (keep only if `build.js` is retained)
+- `pkgs.dart-sass` — Vue/Vite uses the `sass` npm package; check if Go code invokes dart-sass directly
+- `pkgs.autoprefixer` — may not be needed with Vite's built-in PostCSS
+
+---
+
+## 12. Start Scripts — Consolidate
+
+**Status:** Four start scripts exist:
+
+| Script | Purpose |
+|--------|---------|
+| `start-air.sh` | Development with air (live reload) |
+| `start-flogo.sh` | Unknown (references `flogo`) |
+| `start-nidus-sync.sh` | Production-like direct run |
+| `start-nix-built.sh` | Run Nix-built output |
+
+`start-flogo.sh` may be a remnant. Investigate and remove if unused.
+
+---
+
+## Priority Summary
+
+1. **High impact, low effort:**
+   - Remove `tomtom/` (unused, no imports)
+   - Remove `build.js` (dead, replaced by Vite)
+   - Remove commented-out routes in `sync/routes.go` and `rmo/routes.go`
+   - Remove `query.go` commented-out code
+   - Remove `static/gen/main.js` stale artifact
+   - Remove `static/css/placeholder`
+
+2. **Medium impact, medium effort:**
+   - Remove unused Go HTML templates (confirm which are still active first)
+   - Remove unused `static/js/` files (verify against active templates)
+   - Remove `arcgis-go` submodule
+   - Clean up Nix devShell
+
+3. **High impact, high effort:**
+   - Complete Bob → Jet migration across all schemas
+   - Remove Bob-generated models, helpers, scripts
+   - Remove Bob from go.mod
+   - Consolidate `api/` and `resource/` handler patterns
+   - Remove `html/` package (after all Go templates are gone)
diff --git a/HISTORY.md b/HISTORY.md
new file mode 100644
index 00000000..712c5e39
--- /dev/null
+++ b/HISTORY.md
@@ -0,0 +1,207 @@
+# nidus-sync — Project History
+
+## Overview
+
+nidus-sync is a dual-tenant mosquito abatement platform serving two domains:
+- **RMO** (`report.mosquitoes.online`) — Public-facing mosquito/water/nuisance reporting
+- **Sync** (`sync.nidus.cloud`) — Administrative dashboard for vector control districts
+
+The project was started in November 2025 and has undergone several major architectural shifts across ~1655 commits spanning 6 months.
+
+---
+
+## Timeline
+
+### Phase 1: Foundation (November 2025)
+
+**Nov 3 – Nov 13: Project bootstrap**
+- Initial Go project with Nix build system (`flake.nix`, `default.nix`)
+- Basic `net/http` web serving with `gorilla/mux` routing
+- Go `html/template` server-side rendering
+- Bob ORM integration (`github.com/Gleipnir-Technology/bob`) for PostgreSQL — code-generated models via `bobgen`
+- ArcGIS OAuth integration for user authentication
+- ArcGIS Fieldseeker data synchronization (treatment areas, inspections, breeding sources, etc.)
+- MapBox GL JS integration for heatmap visualization
+- Dashboard with login, basic CRUD mocks
+
+**Nov 13 – Nov 24: Logging & DB restructuring**
+- Migration from standard `log` to `zerolog` for structured, colorized output
+- Database logic moved into a separate `db/` subdirectory
+- Clean shutdown logic, token refresh loops
+
+**Key characteristics:** Monolithic Go server, HTML templates, Bob ORM, MapBox maps, ArcGIS OAuth
+
+---
+
+### Phase 2: Fieldseeker & Schema Evolution (December 2025)
+
+**Dec 2 – Dec 24: Fieldseeker schema v2**
+- Bob codegen updated to latest version
+- Fieldseeker schema captured on OAuth connect and stored locally
+- Dynamic SQL functions replacing hardcoded per-table sync logic
+- Old Fieldseeker tables removed, v2 generated tables used
+- Note/image audio support added
+- MMS file downloads from SMS webhooks
+
+**Key characteristics:** Bob-generated fieldseeker models, prepared SQL functions, SMS/MMS debugging
+
+---
+
+### Phase 3: Architecture Maturation (January 2026)
+
+**Jan 2 – Jan 8: Domain split & template system**
+- WIP pass-through models concept ("Checkpoint on initial idea for passing through models")
+- Massive reorganization: templates split into `rmo/` (public) and `sync/` (admin) subdirectories
+- `html/` package created with embedded template loading
+- Bob submodule removed, `arcgis-go` became external dependency
+- Public report domain support added
+- Version bumped 7 times in rapid iteration (v0.0.4 → v0.0.10)
+
+**Jan 8 – Jan 31: Platform Layer emergence**
+- "Report platform layer" introduced (`a9b0a55f`) — initial abstraction between HTTP handlers and database
+- Address suggestion and map-locator components via custom HTML elements
+- SVG auto-transformation into Go templates
+- Report submission forms wired up (nuisance, water)
+- Email template system
+
+**Key characteristics:** Two-domain architecture (RMO/Sync), `html/` template package, platform layer beginning, custom element web components
+
+---
+
+### Phase 4: Map Migration & Platform Expansion (February 2026)
+
+**Feb 1 – Feb 28: Map provider transition**
+- MapBox → MapLibre GL (open-source fork) via `maplibre-gl`
+- Stadia Maps integration for tile serving and geocoding (Feb 12-14)
+- TomTom routing integration added (Feb 17)
+- Bulk geocoding via Stadia
+- Parcel image generation debugging
+
+**Platform layer expansion:**
+- Emails moved to platform layer
+- Phone/SMS support
+- OAuth integration settings
+- Upload platform functions
+- QR code and image tile moved into platform
+- Admin map components
+
+**Key characteristics:** MapLibre/Stadia replacing MapBox, TomTom added, platform layer expanding, heavy template iteration
+
+---
+
+### Phase 5: VueJS Revolution (March 2026) — 448 commits
+
+**Mar 5 – Mar 12: Pre-Vue cleanup**
+- Stadia Maps client initialization
+- Signal database schema added
+- Review task/mailer schema rework
+- Generated Bob files pruned
+
+**Mar 12: Massive platform layer rework** (`44c4f17f`)
+- User/organization handling restructured in platform layer
+- Signal creation moved inside platform
+
+**Mar 18 – Mar 22: VueJS Migration** (the biggest architectural shift)
+- Mar 18: Auto-generated report IDs
+- Mar 21: **VueJS introduced** — begins with TypeScript bundle, then Vue SFC components, vue-router, Bootstrap/SCSS integration
+- Mar 21: Dashboard, Intelligence, sidebar all moved to Vue
+- Mar 22: **esbuild replaced by Vite** (`47f900ab`) — `vite/` directory with separate configs for `sync` and `rmo` SPAs
+- Mar 22: TypeScript checking clean across entire frontend
+- Mar 23: Public report card component, auth checks off API client
+- Mar 24-31: Communication page ripped into components, impersonation support, users page
+
+**Key characteristics:** VueJS 3 + TypeScript + Vite frontend, Pinia stores, vue-router, SCSS, SPA architecture replacing server-rendered Go templates
+
+---
+
+### Phase 6: Compliance & Communication (April 2026) — 454 commits
+
+**Apr 1 – Apr 9: RMO frontend & resources**
+- Resource layer expanded (user, avatar, district, nuisance, water, compliance resources)
+- RMO frontend checkpoint — Vue ports of public-facing pages
+- TS types migrated into API module
+- Old bundle paths removed, old SPA generation removed
+
+**Apr 10 – Apr 17: Compliance workflow**
+- Compliance report creation, mailer flow
+- Site/pool review tasks
+- Stadia Maps cache, direct tile access
+- OAuth refresh in frontend
+- Image upload components
+
+**Apr 17 – Apr 25: Communication system**
+- Background jobs reworked for shorter transactions
+- Lob (physical mail) integration — direct API client, address creation, letter events
+- QR code generation moved to API
+- Compliance report evidence, mailer views
+- Vue map system generalized (`cad01e68`)
+
+**Apr 25 – Apr 30: Map & communication polish**
+- VueJS reimplementation of address/report suggestion
+- Communication workbench with map, list, detail views
+- Text message log, email/phone display
+- Compliance card detail display
+- SSE event system with status vs resource message distinction
+- Systemd socket activation for downtime-free deploys
+- Sentry error tracking for Vue frontend
+
+**Key characteristics:** Compliance/mailer operational, communication system born, Lob integration, Sentry, generalized Vue map system
+
+---
+
+### Phase 7: Jet Migration & Cleanup (May 2026) — 46 commits so far
+
+**May 1 – May 9: SQL generation transition**
+- **Jet (go-jet/jet) introduced** — type-safe SQL builder replacing Bob's query building
+- Custom Jet generator created with geometry/Box2D type support (`db/jet/main.go`)
+- `publicreport` schema ported to Jet
+- `arcgis` schema ported to Jet (compiles, not fully tested per commit message)
+- New `communication` table added
+- Communication marking workflow (invalid, pending-response, possible-issue, possible-resolved)
+- Linting: `golangci-lint` added to lefthook, per-file linting
+- Cleanup of legacy generated columns (latitude/longitude), string-based queries
+- Centralized error handler for Vue sync app
+
+**Key characteristics:** Bob→Jet transition in progress, communication workflow, code quality improvements
+
+---
+
+## Architectural Patterns (by layer)
+
+### Current architecture stack
+
+```
+┌─────────────────────────────────────────────────┐
+│  Vue 3 SPA (TypeScript)                          │
+│  ts/ — shared components, composables, stores    │
+│  vite/sync/ — admin SPA entry                    │
+│  vite/rmo/ — public SPA entry                    │
+├─────────────────────────────────────────────────┤
+│  Go HTTP Server (gorilla/mux)                    │
+│  api/routes.go — central route registration      │
+│  resource/ — resource handlers (REST patterns)   │
+│  sync/ — remaining Go template routes             │
+│  rmo/ — remaining Go template routes              │
+├─────────────────────────────────────────────────┤
+│  platform/ — business logic layer                │
+│  (address, compliance, communication, district,  │
+│   email, fieldseeker, mailer, publicreport,      │
+│   review, signal, text, user, upload, etc.)      │
+├─────────────────────────────────────────────────┤
+│  db/ — database access                           │
+│  db/models/ — Bob-generated models (103 files)   │
+│  db/query/ — Jet-based query functions           │
+│  db/prepared.go — prepared SQL functions         │
+├─────────────────────────────────────────────────┤
+│  PostgreSQL                                      │
+└─────────────────────────────────────────────────┘
+```
+
+### Pattern: Platform Layer
+Introduced January 2026, the `platform/` package encapsulates business logic between HTTP handlers and the database. It grew from initial report handling to encompass users, organizations, emails, texts, compliance, communications, signals, geocoding, tiles, uploads, and more.
+
+### Pattern: Resource Layer
+Added March–April 2026, `resource/` provides typed REST resource handlers with URI generation (via mux route naming). Resources are instantiated with a `resource.NewRouter()` and expose methods like `List`, `Get`, `Create`, `Update`, `Delete` that return domain types. This replaced ad-hoc handler functions in `api/`.
+
+### Pattern: Dual SPA + API
+Since late March 2026, both domains serve Vue SPAs for most routes, with the Go server acting as an API backend. The `static.SinglePageApp()` handler serves the Vite-built output and falls back to `index.html` for client-side routing. Some Go template routes remain for mailer PDF generation, OAuth flows, and previews.