The cloud portion of Nidus
This refactor was born out of the inter-dependency cycles developing between the "background" module and just about every other module which was caused by the background module becoming a dependency of every module that needed to background work and the fact that the background module was also supposedly responsible for the logic for processing those tasks. Instead the "background" module is now very, very shallow and relies entirely on the Postgres NOTIFY logic for triggering jobs. There's a new table, `job` which holds just a type and single row ID. All told, this means that jobs can be added to the queue as part of the API-level or platform-level transaction, ensuring atomicity, and processing coordination is handled by the platform module, which can depend on anything. |
||
|---|---|---|
| api | ||
| arcgis-go@f5ec5c75c1 | ||
| auth | ||
| cmd/passwordgen | ||
| comms | ||
| config | ||
| db | ||
| debug | ||
| go-geojson2h3@c2ff1a96ab | ||
| h3utils | ||
| html | ||
| http | ||
| label-studio | ||
| llm | ||
| minio | ||
| platform | ||
| rmo | ||
| scss | ||
| stadia | ||
| sync | ||
| tomtom | ||
| tools | ||
| .air.toml | ||
| .gitattributes | ||
| .gitignore | ||
| .gitmodules | ||
| .prettierrc | ||
| default.nix | ||
| flake.lock | ||
| flake.nix | ||
| go.mod | ||
| go.sum | ||
| lefthook.yml | ||
| LICENSE | ||
| main.go | ||
| query.go | ||
| README.md | ||
| start-air.sh | ||
| start-flogo.sh | ||
| start-nidus-sync.sh | ||
Nidus Sync
This is the software that powers Nidus Cloud Sync.
Building from source
First, you'll need Nix.
Then:
nix develop
go build .
Building Custom Theme
We're using a customized Bootstrap theme for this site. You'll need to build the SCSS into CSS:
nix develop
sass --style=compressed --trace "$SASS_SRC_DIR/custom.scss":"$CSS_OUTPUT_DIR/bootstrap.css"
Running
You'll need a number of environment variables for configuring things;
- ARCGIS_CLIENT_ID - The client ID for ArcGIS oauth, configured with esri.
- ARCGIS_CLIENT_SECRET - The client secret for ArcGIS oauth, configured with esri.
- BASE_URL - The URL the site is hosted at, used for forming callback URLs. Should be complete like 'https://foo.bar'
- BIND - The address and port to bind to. Use ':9001' for 'any address, port 9001'
- ENVIRONMENT - either 'PRODUCTION' or 'DEVELOPMENT'. It's used to set things like oauth token length.
- MAPBOX_TOKEN - The token to use with mapbox which is important for rendering maps.
- POSTGRES_DSN - The DSN for connecting to the postgres database.
- FIELDSEEKER_SCHEMA_DIRECTORY - The directory to write fieldseeker schema files for debugging.
- USER_FILES_DIRECTORY - The directory for writing uploaded user data files (audio, images).
> BASE_URL=https://sync.nidus.cloud ARCGIS_CLIENT_ID=foo ARCGIS_CLIENT_SECRET=bar POSTGRES_DSN='postgresql://?host=/var/run/postgresql&dbname=nidus-sync' ./nidus-sync
Districts
There's a table containing district information in the database, import.district. It was created with:
psql
CREATE SCHEMA import;
shp2pgsql -s 3857 -c -D -I CA_districts.shp import.district | psql -d nidus-sync
psql
ALTER TABLE import.district ADD COLUMN geom_4326 geometry(MultiPolygon,4326) GENERATED ALWAYS AS (ST_Transform(geom, 4326)) STORED;
Hacking
air
This project uses air for fast compile-and-test loops. You can run it with:
> BASE_URL=https://sync.nidus.cloud ARCGIS_CLIENT_ID=foo ARCGIS_CLIENT_SECRET=bar POSTGRES_DSN='postgresql://?host=/var/run/postgresql&dbname=nidus-sync' air
bob
This uses the bob query framework. You can regenerate the models for bob with:
PSQL_DSN="postgresql://dbname?host=/var/run/postgresql&sslmode=disable" go run github.com/stephenafamo/bob/gen/bobgen-psql@latest"
PSQL_DSN="postgresql://?host=/var/run/postgresql&sslmode=disable&dbname=nidus-sync" go run github.com/stephenafamo/bob/gen/bobgen-psql@latest
This will generate a bunch of files. They're already committed, you only need this if you change the database schema in some way.
goose
This uses goose. You can use the goose command line to check status and make changes
> cd migrations
> GOOSE_DRIVER=postgres GOOSE_DBSTRING="dbname=nidus-sync sslmode=disable" goose status
> GOOSE_DRIVER=postgres GOOSE_DBSTRING="dbname=nidus-sync sslmode=disable" goose down
> GOOSE_DRIVER=postgres GOOSE_DBSTRING="dbname=nidus-sync sslmode=disable" goose up
watchexec
For iterating on styles
watchexec -e *.scss sass scss/custom.scss:html/static/css/bootstrap.css