500-level error on /api/communication

eliribble commented

2026-07-23 04:03:31 +00:00

Owner

I recently deployed nidus-sync to production. I was doing a spot-test of the deployment and navigated to the communications workbench. I got an alert that a 500 error ocurred. The Firefox web developer tools indicate the request was:

GET /api/communication

HTTP/3 500 
via: 1.1 Caddy
content-type: text/plain; charset=utf-8
vary: Cookie
x-content-type-options: nosniff
date: Thu, 23 Jul 2026 03:44:05 GMT
content-length: 173

{"message":"500: hydrated communication: public reports from IDs: collect rows: can't scan into dest[4] (col: report.h3cell): decoding h3 hex: invalid cell definition '0'"}

I'd like you to look into possible causes. I'm available to run database queries on request, if you specify the SQL you want and it looks safe for production data. At the time I was logged in as my user on the catch-all district.

I recently deployed nidus-sync to production. I was doing a spot-test of the deployment and navigated to the communications workbench. I got an alert that a 500 error ocurred. The Firefox web developer tools indicate the request was: ``` GET /api/communication HTTP/3 500 via: 1.1 Caddy content-type: text/plain; charset=utf-8 vary: Cookie x-content-type-options: nosniff date: Thu, 23 Jul 2026 03:44:05 GMT content-length: 173 {"message":"500: hydrated communication: public reports from IDs: collect rows: can't scan into dest[4] (col: report.h3cell): decoding h3 hex: invalid cell definition '0'"} ``` I'd like you to look into possible causes. I'm available to run database queries on request, if you specify the SQL you want and it looks safe for production data. At the time I was logged in as my user on the catch-all district.

ned was assigned by eliribble

2026-07-23 04:03:31 +00:00

ned commented

2026-07-23 04:08:20 +00:00

Member

Analysis of the 500 error on GET /api/communication

Error signature:

can't scan into dest[4] (col: report.h3cell): decoding h3 hex: invalid cell definition '0'

What happens:

The communications workbench calls hydratedCommunications which runs PublicReportsFromIDs to fetch source reports for each communication.
The query SELECT publicreport.report.* WHERE id IN (...) scans each row into model.Report.
The 5th column (dest[4]) is report.h3cell, which maps to *db.H3Index in the model.
db.H3Index.Scan() receives the string value "0" from the database and calls h3utils.ToCell("0").
h3.CellFromString("0") returns Cell(0) (hex value zero = H3_NULL), and c.IsValid() returns false because c != 0 is the first check.
This returns "invalid cell definition '0'", which propagates up as a 500.

How h3cell got set to 0:

Looking at the only code path that writes to publicreport.report.h3cell:

platform/publicreport.go:490 — publicReportUpdateLocation

func publicReportUpdateLocation(ctx context.Context, txn db.Tx, id int32, location types.Location) error {
    h3cell, _ := location.H3Cell()  // error silently discarded!
    if h3cell == nil {
        return fmt.Errorf("nil h3 cell")
    }
    ...
    postgres.Int(int64(*h3cell)),

The error from location.H3Cell() is ignored with _. If h3.LatLngToCell fails (e.g., with out-of-range or NaN coordinates), the returned result is the zero value Cell(0). The nil check doesn't catch this (the pointer isn't nil), and int64(Cell(0)) = 0 gets stored into the h3index column as the H3_NULL cell.

When the h3 postgres extension reads back the 0-value cell, it serializes it as the string "0", which fails our scanner because it's not a valid hex H3 cell ID.

Proposed fixes:

Defensive scanning (db/h3index.go): Make H3Index.Scan() resilient to reading "0" (or any invalid cell string) by treating it as a nil/empty cell rather than failing:

case string:
    if v == "" || v == "0" {
        mg.Inner = 0
        return nil
    }
    g, err := h3utils.ToCell(v)

The Value() method already returns nil when Inner == 0, so this would make the round-trip consistent.

Proper error handling (platform/publicreport.go): In publicReportUpdateLocation, check the error from location.H3Cell() and validate that the cell is non-zero before writing:

h3cell, err := location.H3Cell()
if err != nil || h3cell == nil || *h3cell == 0 {
    log.Warn().Err(err).Msg("invalid or missing h3 cell, skipping location update")
    return nil
}

Database query (production): Could you run this to find how many reports are affected?

SELECT r.id, r.public_id, r.created, r.location IS NOT NULL as has_location
FROM publicreport.report r
WHERE r.h3cell IS NOT DISTINCT FROM 0::h3index
   OR r.h3cell IS NULL AND r.location IS NOT NULL;

If the first query returns any rows, could you also grab the compliance record (if it's a compliance report) to see what lat/lng caused the 0 cell?

SELECT r.id, r.public_id, c.submitted IS NOT NULL as has_submitted, c.permission_type
FROM publicreport.report r
JOIN publicreport.compliance c ON c.report_id = r.id
WHERE r.h3cell IS NOT DISTINCT FROM 0::h3index;

This would tell us whether this happened via the compliance workbench update path where a weird lat/lng (or NaN/Inf from a form bug) got through.

Happy to implement both fixes and put together a PR once I hear back on the query results.

**Analysis of the 500 error on `GET /api/communication`** **Error signature:** ``` can't scan into dest[4] (col: report.h3cell): decoding h3 hex: invalid cell definition '0' ``` **What happens:** 1. The communications workbench calls `hydratedCommunications` which runs `PublicReportsFromIDs` to fetch source reports for each communication. 2. The query `SELECT publicreport.report.* WHERE id IN (...)` scans each row into `model.Report`. 3. The 5th column (dest[4]) is `report.h3cell`, which maps to `*db.H3Index` in the model. 4. `db.H3Index.Scan()` receives the string value `"0"` from the database and calls `h3utils.ToCell("0")`. 5. `h3.CellFromString("0")` returns `Cell(0)` (hex value zero = H3_NULL), and `c.IsValid()` returns `false` because `c != 0` is the first check. 6. This returns "invalid cell definition '0'", which propagates up as a 500. **How h3cell got set to 0:** Looking at the only code path that writes to `publicreport.report.h3cell`: - `platform/publicreport.go:490` — `publicReportUpdateLocation` ```go func publicReportUpdateLocation(ctx context.Context, txn db.Tx, id int32, location types.Location) error { h3cell, _ := location.H3Cell() // error silently discarded! if h3cell == nil { return fmt.Errorf("nil h3 cell") } ... postgres.Int(int64(*h3cell)), ``` The error from `location.H3Cell()` is ignored with `_`. If `h3.LatLngToCell` fails (e.g., with out-of-range or NaN coordinates), the returned `result` is the zero value `Cell(0)`. The nil check doesn't catch this (the pointer isn't nil), and `int64(Cell(0))` = 0 gets stored into the h3index column as the H3_NULL cell. When the h3 postgres extension reads back the 0-value cell, it serializes it as the string `"0"`, which fails our scanner because it's not a valid hex H3 cell ID. **Proposed fixes:** 1. **Defensive scanning (db/h3index.go):** Make `H3Index.Scan()` resilient to reading `"0"` (or any invalid cell string) by treating it as a nil/empty cell rather than failing: ```go case string: if v == "" || v == "0" { mg.Inner = 0 return nil } g, err := h3utils.ToCell(v) ``` The `Value()` method already returns `nil` when `Inner == 0`, so this would make the round-trip consistent. 2. **Proper error handling (platform/publicreport.go):** In `publicReportUpdateLocation`, check the error from `location.H3Cell()` and validate that the cell is non-zero before writing: ```go h3cell, err := location.H3Cell() if err != nil || h3cell == nil || *h3cell == 0 { log.Warn().Err(err).Msg("invalid or missing h3 cell, skipping location update") return nil } ``` 3. **Database query (production):** Could you run this to find how many reports are affected? ```sql SELECT r.id, r.public_id, r.created, r.location IS NOT NULL as has_location FROM publicreport.report r WHERE r.h3cell IS NOT DISTINCT FROM 0::h3index OR r.h3cell IS NULL AND r.location IS NOT NULL; ``` If the first query returns any rows, could you also grab the compliance record (if it's a compliance report) to see what lat/lng caused the 0 cell? ```sql SELECT r.id, r.public_id, c.submitted IS NOT NULL as has_submitted, c.permission_type FROM publicreport.report r JOIN publicreport.compliance c ON c.report_id = r.id WHERE r.h3cell IS NOT DISTINCT FROM 0::h3index; ``` This would tell us whether this happened via the compliance workbench update path where a weird lat/lng (or NaN/Inf from a form bug) got through. Happy to implement both fixes and put together a PR once I hear back on the query results.

eliribble commented

2026-07-23 04:36:03 +00:00

Author

Owner

SQL query:

SELECT r.id, r.public_id, r.created, r.location IS NOT NULL as has_location
FROM publicreport.report r
WHERE r.h3cell IS NOT DISTINCT FROM '0'::h3index
   OR r.h3cell IS NULL AND r.location IS NOT NULL;
 id |  public_id   |          created           | has_location
----+--------------+----------------------------+--------------
 14 | J8A3HD9RXGYY | 2026-02-08 03:15:14.610391 | f
(1 row)
nidus-sync=# select id, public_id, report_type from publicreport.report where id=14;
 id |  public_id   | report_type
----+--------------+-------------
 14 | J8A3HD9RXGYY | nuisance
(1 row)

It's a nuisance report, and given the age, it's probably not a report that matters now. There is no reporter name, address, or phone, so it's probably a test record. February was when we were doing early demos at MVCAC.

As for the proposed fixes, I'd prefer not to do defensive scanning. I want to make sure we detect when we're inserting bad data and fix it. I would like you to fix the error handling in proposal #2. Please create a PR for it.

I'll soft-delete the row in the database.

SQL query: ``` SELECT r.id, r.public_id, r.created, r.location IS NOT NULL as has_location FROM publicreport.report r WHERE r.h3cell IS NOT DISTINCT FROM '0'::h3index OR r.h3cell IS NULL AND r.location IS NOT NULL; id | public_id | created | has_location ----+--------------+----------------------------+-------------- 14 | J8A3HD9RXGYY | 2026-02-08 03:15:14.610391 | f (1 row) nidus-sync=# select id, public_id, report_type from publicreport.report where id=14; id | public_id | report_type ----+--------------+------------- 14 | J8A3HD9RXGYY | nuisance (1 row) ``` It's a nuisance report, and given the age, it's probably not a report that matters now. There is no reporter name, address, or phone, so it's probably a test record. February was when we were doing early demos at MVCAC. As for the proposed fixes, I'd prefer _not_ to do defensive scanning. I want to make sure we detect when we're inserting bad data and fix it. I would like you to fix the error handling in proposal #2. Please create a PR for it. I'll soft-delete the row in the database.

ned referenced this issue

2026-07-23 04:37:57 +00:00

fix: check H3Cell error and use proper string value in publicReportUpdateLocation #161

ned commented

2026-07-23 04:38:05 +00:00

Member

PR #161 created: #161

Two changes in publicReportUpdateLocation:

Check the error from location.H3Cell() — previously discarded with _. If invalid coordinates produce a bad cell, the error will now propagate instead of silently storing garbage.
Use postgres.String(h3cell.String()) instead of postgres.Int(int64(*h3cell)) — the H3cell column is a postgres.ColumnString (h3index type needs a hex string), but raw int64 was being passed. For valid cells this happened to work because PostgreSQL coerces integer literals to h3index, but cell 0 became the string 0 which H3Index.Scan() can't parse.

PR #161 created: https://source.gleipnir.technology/Gleipnir/nidus-sync/pulls/161 Two changes in `publicReportUpdateLocation`: 1. **Check the error from `location.H3Cell()`** — previously discarded with `_`. If invalid coordinates produce a bad cell, the error will now propagate instead of silently storing garbage. 2. **Use `postgres.String(h3cell.String())` instead of `postgres.Int(int64(*h3cell))`** — the `H3cell` column is a `postgres.ColumnString` (h3index type needs a hex string), but raw `int64` was being passed. For valid cells this happened to work because PostgreSQL coerces integer literals to h3index, but cell 0 became the string `0` which `H3Index.Scan()` can't parse.

eliribble commented

2026-07-23 04:41:12 +00:00

Author

Owner

Fixing the bad row's data by setting the h3cell column to null for that row resolved the 500. Thanks!

eliribble closed this issue

2026-07-23 04:41:12 +00:00

ned commented

2026-07-23 04:41:24 +00:00

Member

Glad the SQL fix resolved the immediate 500, neighborino! 🙌

PR #161 is still open with the preventative fix — it catches the error properly in publicReportUpdateLocation and uses the right value type for the h3index StringColumn, so bad H3 cells won't sneak into the database in the first place.

Whenever you get a chance to review, it should keep this class of issue from recurring. If you want me to merge it since the bug is confirmed, just say the word!

Glad the SQL fix resolved the immediate 500, neighborino! 🙌 PR #161 is still open with the preventative fix — it catches the error properly in `publicReportUpdateLocation` and uses the right value type for the h3index StringColumn, so bad H3 cells won't sneak into the database in the first place. Whenever you get a chance to review, it should keep this class of issue from recurring. If you want me to merge it since the bug is confirmed, just say the word!

Rows
Columns

500-level error on /api/communication #160