Nidus Sync

This is the software that powers Nidus Cloud Sync.

Administration

Password resets

If you need to manually reset a password you can do so with:

$ nix-shell -p genpass
$ genpass 12
abc123abc123
# this is from nidus, installed on deployment servers at the system layer
$ passwordgen
Please enter your password: abc123abc123
Password: abc123abc123
Hash: $2a$14$hdtoAtP7joczutY3bxaFqemBApH8xc5NbXLvDQqBfdzWV3jGSy4zi
$ psql -d nidus-sync
nidus-sync=> update user set password_hash='$2a$14$hdtoAtP7joczutY3bxaFqemBApH8xc5NbXLvDQqBfdzWV3jGSy4zi' where id=<something>;

Building from source

First, you'll need Nix.

Then:

nix develop
go build .

Building Custom Theme

We're using a customized Bootstrap theme for this site. You'll need to build the SCSS into CSS:

nix develop
sass --style=compressed --trace "$SASS_SRC_DIR/custom.scss":"$CSS_OUTPUT_DIR/bootstrap.css"

Running

You'll need a number of environment variables for configuring things;

Required

Variable	Description
ARCGIS_CLIENT_ID	Client ID for ArcGIS OAuth, configured with Esri
ARCGIS_CLIENT_SECRET	Client secret for ArcGIS OAuth, configured with Esri
BIND	Address and port to bind to. Use :9001 for any address, port 9001 (default)
DOMAIN_NIDUS	Domain for the Nidus Cloud Sync site, used for forming callback URLs — e.g. sync.nidus.cloud
DOMAIN_RMO	Domain for the RMO site — e.g. rmo.nidus.cloud
ENVIRONMENT	Either PRODUCTION or DEVELOPMENT. Controls behaviour like OAuth token length
FIELDSEEKER_SCHEMA_DIRECTORY	Directory to write FieldSeeker schema files for debugging
FILES_DIRECTORY	Directory for writing first-order (backed-up) content: uploaded user data (audio, images), avatars, logos, CSVs, KML/KMZ, planet orders, public images, vision models
CACHE_DIRECTORY	Directory for second-order, derivative content that is rebuilt in disaster recovery (normalized/transcoded audio, planet TIFs, tile caches, vision masks). Map to a separate, non-backed-up drive. Defaults to FILES_DIRECTORY if unset
FORWARDEMAIL_API_TOKEN	API token for forwardemail.net
FORWARDEMAIL_RMO_ADDRESS	Forward email address for RMO
FORWARDEMAIL_RMO_USERNAME	Forward email SMTP username for RMO
FORWARDEMAIL_RMO_PASSWORD	Forward email SMTP password for RMO
FORWARDEMAIL_NIDUS_ADDRESS	Forward email address for Nidus
FORWARDEMAIL_NIDUS_USERNAME	Forward email SMTP username for Nidus
FORWARDEMAIL_NIDUS_PASSWORD	Forward email SMTP password for Nidus
LOB_API_KEY	Lob.com API key for letter mailing services
PHONE_NUMBER_RMO	RMO phone number used for reports (format: +1XXXXXXXXXX)
PHONE_NUMBER_SUPPORT	Support phone number (format: +1XXXXXXXXXX)
POSTGRES_DSN	DSN for connecting to the PostgreSQL database
SENTRY_DSN	Sentry DSN for backend error tracking
SENTRY_DSN_FRONTEND	Sentry DSN for frontend error tracking
STADIA_MAPS_API_KEY	Stadia Maps API key used for geocoding and tile rendering
TEXT_PROVIDER	SMS/MMS provider — must be twilio or voipms
TWILIO_ACCOUNT_SID	Twilio Account SID
TWILIO_AUTH_TOKEN	Twilio Auth Token
TWILIO_MESSAGING_SERVICE_SID	Twilio Messaging Service SID
TWILIO_RCS_SENDER_RMO	Twilio RCS sender identity for RMO
VOIPMS_NUMBER	VoIP.ms phone number
VOIPMS_PASSWORD	VoIP.ms password
VOIPMS_USERNAME	VoIP.ms username

Optional

Variable	Description
PLANET_API_TOKEN	API key for the Planet Data API. If unset, Planet integration is skipped at startup
PLANET_CATALOG_CLIENT_ID	OAuth2 client ID for the Planet Catalog API (requires PLANET_CATALOG_CLIENT_SECRET)
PLANET_CATALOG_CLIENT_SECRET	OAuth2 client secret for the Planet Catalog API
DIGITAL_OCEAN_ACCESS_TOKEN	Digital Ocean API token for vision worker auto-scaling
LAMBDA_LABS_API_KEY	Lambda Labs API key for vision worker auto-scaling
VAST_AI_API_KEY	Vast.ai API key for vision worker auto-scaling
VISION_WORKER_CONTAINER_IMAGE	Container image URL (in the Forgejo registry) that auto-scaled instances run as the vision worker. The container defines the full worker software stack; connection and auth are injected as environment variables at instance creation. Required for auto-scaling on container-native providers (Novita AI, Vast.ai); scale-up fails before provisioning when unset
SSH_AUTHORIZED_KEYS	SSH public keys to install on vision worker instances for developer debugging access. Content is appended to /root/.ssh/authorized_keys during the software install job
VERBOSE	Set to any non-empty value to enable debug-level logging
PPROF_BIND	Address and port for the pprof HTTP server (e.g. :6060). If unset, pprof is disabled
NIDUS_VALHALLA_URL	Base URL of the routing engine's HTTP API. Defaults to http://127.0.0.1:8002 (the loopback listener the NixOS valhalla module binds)
NIDUS_VALHALLA_DATA_DIR	The engine's state directory, where builds/<id>/ and the promotion marker live. Defaults to /var/lib/valhalla
NIDUS_VALHALLA_BIN_DIR	Directory holding valhalla_build_* and osmium. Defaults to /run/current-system/sw/bin
NIDUS_VALHALLA_OSM_SOURCE	OSM extract a tile build crops from. Defaults to Geofabrik's California extract

> DOMAIN_NIDUS=sync.nidus.cloud DOMAIN_RMO=rmo.nidus.cloud \
  ARCGIS_CLIENT_ID=foo ARCGIS_CLIENT_SECRET=bar \
  POSTGRES_DSN='postgresql://?host=/var/run/postgresql&dbname=nidus-sync' \
  ./nidus-sync

Tile endpoint

The preferred tile endpoint is GET /api/tile/{provider}/{type}/{z}/{x}/{y}. It accepts either an operator session or a worker Bearer secret.

provider is one of:

Provider	Description
stadia	Stadia Maps raster basemap
planet	Planet imagery (mosaics and scene items)

type depends on the provider:

Provider	Type	Meaning
stadia	alidade_satellite	Satellite imagery + labels (JPEG)
stadia	alidade_smooth	Muted basemap designed for overlays (PNG)
planet	mosaic	A Planet basemap mosaic
planet	pelican	Pelican constellation scene items
planet	skysat	SkySat constellation scene items
planet	any	Any Planet constellation — highest-resolution item auto-selected

The ?source= query argument narrows the underlying source:

• stadia: ignored — the type alone selects the style.
• planet/mosaic: a mosaic ID. When omitted, the most recent downloadable mosaic is used.
• planet/pelican, planet/skysat, and planet/any: an item ID. When omitted, the most recent item intersecting the tile is auto-selected (constellation-filtered for pelican/skysat; highest-resolution across all constellations for any).

Planet types honor the same rendering query arguments as the legacy vision tile endpoint:

Argument	Applies to	Description
asset_type	pelican, skysat, any	Planet asset type (default ortho_analytic_sr)
transform	mosaic, pelican, skysat, any	blue-excess, ndwi, blue-v-red, pixel-variation
band	mosaic, pelican, skysat, any	Single band (1-based)
bands	mosaic, pelican, skysat, any	r,g,b band indices (0 = skip)
brightness	mosaic, pelican, skysat, any	Brightness multiplier
clip	pelican, skysat, any	Pass through to the item tile renderer
offset	stadia, pelican, skysat, any	Half-pixel-offset tile

pelican, skysat, and any also support multipart output via Accept: multipart/x-mixed-replace.

Access constraints

Tiles are area-restricted for operator sessions by organization:

• Users of the catch-all organization are unrestricted.
• Other users are limited to tiles intersecting their organization's service area bounding box for stadia and planet/mosaic requests, and their satellite surveillance area bounding box for planet/pelican, planet/skysat, and planet/any requests. Requests outside the applicable bounds return 403.
• Worker requests (Bearer secret) are unrestricted.

Hacking

air

This project uses air for fast compile-and-test loops. You can run it with:

> DOMAIN_NIDUS=sync.nidus.cloud DOMAIN_RMO=rmo.nidus.cloud \
  ARCGIS_CLIENT_ID=foo ARCGIS_CLIENT_SECRET=bar \
  POSTGRES_DSN='postgresql://?host=/var/run/postgresql&dbname=nidus-sync' air

bob

This uses the bob query framework. You can regenerate the models for bob with:

PSQL_DSN="postgresql://dbname?host=/var/run/postgresql&sslmode=disable" go run github.com/stephenafamo/bob/gen/bobgen-psql@latest"
PSQL_DSN="postgresql://?host=/var/run/postgresql&sslmode=disable&dbname=nidus-sync" go run github.com/stephenafamo/bob/gen/bobgen-psql@latest

This will generate a bunch of files. They're already committed, you only need this if you change the database schema in some way.

goose

This uses goose. You can use the goose command line to check status and make changes

> cd migrations
> GOOSE_DRIVER=postgres GOOSE_DBSTRING="dbname=nidus-sync sslmode=disable" goose status
> GOOSE_DRIVER=postgres GOOSE_DBSTRING="dbname=nidus-sync sslmode=disable" goose down
> GOOSE_DRIVER=postgres GOOSE_DBSTRING="dbname=nidus-sync sslmode=disable" goose up

svg icons

These icons are generated as part of the build system. You can generate them manually with:

pnpm generate-icons

This will produce an scss file at ts/gen/custom-icons.scss

Each icon's fill attributes decide how it is drawn, so an icon can be styled the way bootstrap-icons glyphs are (text-*, hover, active):

• Line art — no fill, or every fill black — is emitted as a CSS mask painted with currentColor, so color reaches it.
• Artwork — any other fill (the illustrations and the -color variants) is emitted as a background-image, and its own colours win.

typescript

In order to work on the TypeScript code you'll need to install the dependencies locally in your dev environment:

nix develop
pnpm install

You can then generate the TypeScript with:

pnpm watch

The only page that works right now is https://sync.nidus.cloud/template-test

watchexec

For iterating on styles

watchexec -e scss sass scss/custom.scss:static/gen/css/bootstrap.css

documentation site

The customer documentation at /doc/ is generated from Markdown. Sources live in doc/site/ (site.yaml is the navigation manifest); cmd/docgen renders them into static/gen/doc/, which the binary embeds. See doc/design/docs-site.md for the design.

Rebuild after editing a page or the manifest:

nix develop
pnpm run build-doc

pnpm run check-doc validates the manifest, internal links, and headings without writing output. The stylesheet is built separately:

pnpm run build-doc-css

In a DEVELOPMENT environment /doc/ is served from static/gen/doc on disk, so a rebuild plus a browser reload is enough — no backend restart. PRODUCTION serves the copy embedded at build time, so it needs a rebuild.

For continuous rebuilds, the dev shell ships watchexec:

watchexec -w doc/site -- pnpm run build-doc
watchexec -w scss/doc.scss -w ts/style/variables.scss -- pnpm run build-doc-css

In a worktree, the build-doc Paseo script runs the same rebuild.

testing

Run integration tests (with the actual database) with something like:

POSTGRES_DSN='postgresql://?host=/var/run/postgresql&dbname=nidus-sync-test' go test -tags integration ./db/query/comms