Nemish

Fine-tune a computer vision model to detect swimming pools in residential areas using overhead aerial or satellite photography.

---

Quick start

New VM?

Often you'll be running this on a new VM, often on Lambda Labs, so Ubuntu 24.04. If so, here's a checklist

• Port forwarding: ssh -A -L 4097:localhost:4096
• Basic packages: sudo apt install fish neovim tmux
• chezmoi for dotfiles: sh -c "$(curl -fsLS https://get.chezmoi.io)"
• Clone dotfiles:

mkdir src
cd src/
git clone https://source.theribbles.org/eliribble/dotfiles.git

• Install the dotfiles: cd dotfiles; ~/bin/chezmoi init .; ~/bin/chezmoi apply
• Enter tmux: tmux
• Install opencode: curl -fsSL https://opencode.ai/install | bash
• Clone this repository: git clone ssh://forgejo@source.gleipnir.technology/Gleipnir/nemish.git
• Run opencode in the repository: cd nemish;~/.opencode/bin/opencode web

Development system

All development happens inside a container. The base images are built from the Nix flake; you then layer project dependencies on top with one of two Containerfiles:

# CPU-only container (CI, linting, smoke tests)
podman build -t nemish:ci -f Containerfile.ci .

# GPU container (CUDA 12.4, production training / inference)
podman build -t nemish:gpu -f Containerfile.gpu .

Once built, run the container with your data mounted:

# Start an interactive shell
podman run --rm -it \
    -v ./data:/data \
    -v ./runs:/data/runs \
    --shm-size=2g \
    nemish:gpu

# Or for GPU access
podman run --rm -it \
    --device nvidia.com/gpu=all \
    -v ./data:/data \
    -v ./runs:/data/runs \
    --shm-size=2g \
    nemish:gpu

Inside the container the nemish command is on PATH. Proceed to Training below.

--shm-size=2g is needed because PyTorch DataLoader workers use /dev/shm for inter-process communication. The default (64 MB) is too small for image tensors.

Nix for static analysis only

The Nix flake is the basis for the container base images and also provides a fast path for linting and type-checking without building a full container:

nix develop
ruff check src/
ruff format src/
mypy --config-file mypy.ini src/

Do not rely on nix develop for day-to-day coding or running the pipeline — use the container instead.

---

Data format

The training pipeline expects this directory layout:

data/
├── images/          # aerial / satellite tiles
│   ├── tile_001.png
│   ├── tile_002.png
│   ├── ...
│   └── tile_999.png
├── masks/           # label masks — one per image, same stem name
│   ├── tile_001.png
│   ├── tile_002.png
│   └── ...
├── train.txt        # (optional) newline-separated stem names for training
└── val.txt          # (optional) newline-separated stem names for validation

Image files

• Format: PNG is preferred. JPEG and GeoTIFF (.tif / .tiff) also work.
• Size: The model resizes everything to image_size × image_size (default 256×256) during loading, so source images can be any resolution. Larger tiles give the model more context at the expense of GPU memory.
• Channels: RGB (3 channels). If your imagery has a near-infrared band, set model.in_channels: 4 in the config.

Mask files

• Format: Single-channel (grayscale) PNG.
• Pixel values: Integer class labels.

  Value	Meaning
  0	Background (not a pool)
  1	Pool

• Same dimensions as the image. The model resizes masks together with images so spatial alignment is preserved through nearest-neighbor interpolation on the mask.
• Filenames must match the image they annotate. If tile_001.png is the image, the mask must be named tile_001.png as well (just in a different folder).

Split files (train.txt / val.txt)

Plain text files listing which samples go into each split, one per line, without the file extension:

# data/train.txt
tile_001
tile_003
tile_007
...

If you omit these files, the pipeline splits the data automatically using data.val_fraction (default 15%).

---

Exporting from Label Studio

Label Studio is the most popular open-source tool for image annotation. This section covers how to get your annotations out of Label Studio and into the format above.

Step 1 — Set up your Label Studio project

1. Create a new project with Labeling Setup → Computer Vision → Semantic Segmentation.
2. Under Labeling Interface, add a Brush with nested Labels tag:

<View>
  <Image name="image" value="$image"/>
  <Brush name="pool" toName="image">
    <Labels name="labels" toName="image">
      <Label value="Swimming Pool" />
    </Labels>
  </Brush>
</View>

3. Import your aerial/satellite tiles through the Label Studio UI.
4. Annotate pools by painting over them with the brush tool.
   • Use a brush size appropriate to your image resolution.
   • Be consistent — label pool water only (or define a convention like "pool water + visible coping" and stick to it across all annotators).

Step 2 — Export from Label Studio

Use the CLI to export via the Label Studio SDK. Credentials come from environment variables, and --project-id can be omitted to list projects interactively.

# Set credentials
export LABEL_STUDIO_URL=https://labelstudio.example.com
export LABEL_STUDIO_API_KEY=your_access_token_here

# Interactive — lists projects, pick one
nemish label-studio export

# Or specify a project directly
nemish label-studio export -p 1

Exports are saved to /data/exports/project-<id>.json by default. After exporting, convert to training format with nemish label-studio convert.

pip install label-studio-sdk

Step 2a — Inspect export statistics

Before converting, verify your export looks right:

nemish label-studio export-stats export.json

Outputs:

• Total tasks in the export
• Completed vs incomplete tasks
• How many tasks show pool areas vs no pools

Step 3 — Convert to training format

Images are fetched from S3; configure access via environment variables:

export S3_BUCKET=pool-tiles
export S3_ENDPOINT=https://garage.example.com
export S3_PREFIX=tiles/
export AWS_ACCESS_KEY_ID=GK...
export AWS_SECRET_ACCESS_KEY=...
export S3_REGION=garage

Then run the conversion:

nemish label-studio convert export.json \
    --val-fraction 0.15 \
    --image-format png \
    --mask-format png \
    --seed 42

The output is written to /data/ inside the container (bind-mount to a host directory if needed).

Step 4 — Verify the conversion

Spot-check samples with the built-in web viewer:

nemish web --data-dir data/ --port 8080

This starts a local web server showing each sample's source image, mask, and a red-tinted overlay side by side. Navigate with the on-screen buttons or the left/right arrow keys.

If you're running on a remote server, create an SSH tunnel:

ssh -L 8080:localhost:8080 your-server

Then open http://localhost:8080 in your browser.

---

Training

Before you start

The pretrained encoder weights are downloaded from Hugging Face Hub. To avoid rate-limiting and enable faster downloads, set a Hugging Face token:

set -Ux HF_TOKEN hf_your_token_here

You can get a free token at https://huggingface.co/settings/tokens. Without one training still works, but downloads may be slower.

CPU smoke test (verify the pipeline before GPU spend)

Before committing to an expensive GPU run, confirm the training pipeline is healthy on your CPU with a small data sample. This catches NaN/Inf loss, broken masks, config errors, and import issues without incurring GPU costs:

# 50 samples, 3 epochs, CPU, no workers — ~2 min
nemish train \
    training.max_train_samples=50 \
    training.num_epochs=3 \
    num_workers=0 \
    device=cpu

What to check:

• No NaN/Inf warnings — if you see [WARN] NaN/Inf loss, something is wrong with your data or model.
• train/loss starts below ~0.7 and trends downward — a loss that starts at absurd values (>100) or stays flat suggests normalization or loss-function issues.
• val/iou_mean is non-zero — IoU above ~0.3 on the first epoch means the model is learning. Zero IoU means the model predicts all-background (bad data, incorrect loss function, or class-imbalance problem).
• Run completes without crashing — catches import errors, missing dependencies, GPU API mismatches, corrupt images/masks.

A full 50-epoch CPU smoke test with 50 samples takes ~12 min and should reach IoU ≥ 0.6. If it doesn't, fix the issue before scaling to GPU.

Production GPU training

Once the smoke test passes, run the full dataset on GPU:

nemish train \
    training.use_amp=true \
    device=auto \
    training.batch_size=256 \
    num_workers=4

Key overrides for GPU runs:

Override	Purpose
training.use_amp=true	Mixed precision — ~2× faster, uses less VRAM
device=auto	Picks CUDA if available, CPU otherwise
training.batch_size=256	Larger batches for GPU throughput (tune to fit VRAM)
num_workers=4	Parallel data loading to keep GPU fed
model.encoder_name=resnet50	Stronger backbone (more VRAM, better accuracy)
training.num_epochs=100	More epochs for convergence

This will:

1. Read images + masks from data/
2. Build a UNet with a ResNet-34 encoder (pretrained on ImageNet)
3. Freeze all BatchNorm layers (keeps pretrained ImageNet statistics, crucial for satellite imagery where small batches produce unstable BN stats)
4. Use CombinedLoss (CrossEntropy + Dice) to handle the extreme foreground/ background class imbalance inherent in pool detection
5. Train with AdamW optimizer and cosine-annealing LR schedule
6. Log loss, IoU, and Dice to TensorBoard
7. Save the best checkpoint (by validation IoU) to runs/<timestamp>/checkpoints/best.pth

Monitoring GPU utilisation

After launching, confirm the GPU is being properly utilised:

# Watch GPU usage (install with: apt install nvtop)
nvtop

# Or with built-in nvidia-smi (refresh every second)
watch -n 1 nvidia-smi

What to look for:

• GPU utilisation — should be >80% during training. Spikes to 100% are normal during forward/backward. Sustained <50% means your data loading or CPU is the bottleneck (increase num_workers, check I/O throughput).
• GPU memory — should be mostly allocated (e.g. 8–12 GB of 10–24 GB total). If utilisation is high but memory is low, increase batch_size. If you see CUDA out of memory errors, reduce batch_size or image_size.
• Temperature — sustained >80°C may trigger thermal throttling. Reduce batch_size or lower num_workers to let the GPU cool between batches. On cloud instances this is usually managed by the provider.
• Power draw — should be near the card's rated TDP during training. Significantly lower power at high utilisation suggests the card is bottlenecked elsewhere (e.g. memory bandwidth on lower-end GPUs).

Past experience with an A10 training run indicates that a good resource utilization (100% utilization, 16/24G VRAM) is at:

nemish train \
    training.use_amp=true \
    device=auto \
    training.batch_size=256 \
    num_workers=4 \

Override config from the command line

nemish train \
    training.num_epochs=100 \
    training.batch_size=4 \
    model.encoder_name=resnet50

Resume from a checkpoint

nemish train \
    training.resume_from=runs/20260619_120000/checkpoints/best.pth

Monitor with TensorBoard

tensorboard --logdir runs/ --port 6006
# Open http://localhost:6006 in a browser

Key metrics to watch:

• val/iou_mean — your primary metric. Above 0.7 is good, above 0.85 is excellent.
• train/loss — should decrease smoothly. Spikes may mean your learning rate is too high.
• val/dice_class_1 — Dice score for the pool class only (ignores background).

Monitor with nvtop

$ apt install nvtop
$ nvtop

Training outputs

Each training run creates a timestamped directory under runs/. Here's what ends up on disk:

runs/
└── 20260621_143052/              # auto-generated experiment name (timestamp)
    ├── tensorboard/              # TensorBoard event files
    │   └── events.out.tfevents...
    └── checkpoints/
        ├── best.pth              # checkpoint with highest val/iou_mean
        ├── epoch_0005.pth        # periodic snapshot (every save_every epochs)
        ├── epoch_0010.pth
        ├── ...
        └── last.pth              # checkpoint from the final epoch

Checkpoint contents — each .pth file is a standard PyTorch checkpoint dictionary:

Key	Contents
epoch	Integer — which epoch this was saved from
model_state_dict	Model weights (loadable with load_state_dict)
optimizer_state_dict	Optimizer state (for resuming training)
metrics	Dict with val/loss, val/iou_mean, etc.

Which checkpoint should I use?

• best.pth — use this for inference / predictions. It's the model with the highest validation IoU across all epochs.
• last.pth — the final model state. Useful if you want to resume training later.
• epoch_NNNN.pth — periodic snapshots. Handy if you notice the model started overfitting and want to pick an earlier epoch.

---

Inference

# Single image
python -m src.inference.predict \
    --checkpoint runs/run_name/checkpoints/best.pth \
    --image data/images/tile_042.png \
    --output predictions/

# All images in a directory
python -m src.inference.predict \
    --checkpoint runs/run_name/checkpoints/best.pth \
    --dir data/images/ \
    --output predictions/

# On GPU
python -m src.inference.predict \
    --checkpoint runs/run_name/checkpoints/best.pth \
    --image tile.png --device cuda

Output masks are written as grayscale PNGs where white (255) = predicted pool.

---

Production daemon

The daemon (nemish daemon) is a long-running worker that polls nidus-sync for inference tasks, runs them on GPU, and submits results back. It uses a pipelined architecture with three phases:

1. Data worker — polls nidus-sync, downloads tile images from S3, enqueues work
2. Inference worker — accumulates batches, runs GPU forward passes, pipelines post-processing
3. Main process — drains results, submits them to nidus-sync in parallel

Environment variables

Variable	Default	Description
NEMISH_DAEMON_BATCH_SIZE	64 (>16 GiB RAM), 32 (>8 GiB), 8 (otherwise)	Images per GPU forward pass
NEMISH_DAEMON_QUEUE_SIZE	10000 (>16 GiB), 5000 (>8 GiB), 1000	Max work-queue entries
NEMISH_TILE_WORKERS	2 × cpu_count (min 4)	Tile download threads
NEMISH_PP_WORKERS	2 × cpu_count (min 4)	Post-processing threads
NEMISH_SUBMIT_WORKERS	2 × cpu_count (min 4)	Result-submission threads
NIDUS_SYNC_URL	(required)	nidus-sync API base URL
NIDUS_SYNC_AUTH	(required)	API authentication token
NIDUS_SYNC_WORKER_ID	(required)	Worker identifier

A10 GPU tuning

The daemon was profiled on an NVIDIA A10 (24 GiB VRAM, Ampere) processing ~213K inference tasks. Key findings:

Batch size	Rate (tasks/s)	GPU utilisation	Notes
64	~89	Sawtooth (100 → 25%)	GPU starved between batches
192	~94	Sawtooth (100 → 25%)	More GPU work, same idle pattern
192 + pipelining + HTTP reuse	~115	Sustained >80%	Post-processing overlapped with GPU

The 28% throughput gain (89 → 115) came from three changes:

1. Bumping NEMISH_DAEMON_BATCH_SIZE to 192 — at 256×256 resolution the A10 has plenty of VRAM headroom. The default batch-size heuristic (64) was undersized for 24 GiB cards.

2. Overlapping GPU compute with post-processing — the inference worker now pipelines the GPU forward pass of batch N+1 concurrently with the mask encoding, polygon extraction, and geometry conversion of batch N. This keeps the GPU busy instead of waiting on CPU-bound post-processing.

3. Thread-local HTTP client reuse — submission threads reuse persistent httpx.Client instances instead of creating a new TCP+TLS connection per result. At 115 results/second this eliminates ~115 connection handshakes per second.

Recommended A10 settings:

NEMISH_DAEMON_BATCH_SIZE=192 nemish daemon

For A10 or similar GPUs (A40, L40S) with ≥24 GiB VRAM, start at batch size 192 and tune upward if utilisation still shows troughs. Cards with 10-16 GiB VRAM should start at 64-128.

VRAM usage at batch 192: ~8.4 GiB of 22.4 GiB available for 256×256 tiles. Larger tile resolutions consume more VRAM — reduce batch size proportionally.

Shutdown behaviour: Ctrl+C triggers a graceful drain — the daemon stops polling, finishes in-flight batches, drains the result queue, and waits for all submissions to complete before exiting (up to 600 s for inference drain). Submissions in flight at shutdown are still delivered.

---

Using the model with Label Studio (pre-annotation)

Once you have a trained model, you can use it to pre-annotate tasks in Label Studio. The model takes a first pass at detecting pools on every image; annotators then review and fix the predictions instead of drawing every pool from scratch. This can dramatically speed up labeling throughput.

How it works

The command nemish label-studio predict:

1. Connects to your Label Studio instance via the SDK.
2. Lists all tasks in the target project.
3. Downloads each image, runs the model, and converts the output mask into Label Studio's RLE brush-label format.
4. Pushes the mask as a prediction on the task. When an annotator opens the task, the prediction appears as a pre-filled brush region — they can accept it, adjust the boundaries, or erase it entirely.

Quick start

# Install the SDK (one-time)
pip install label-studio-sdk

# Set credentials
set -Ux LABEL_STUDIO_URL https://labelstudio.example.com
set -Ux LABEL_STUDIO_API_KEY your_access_token_here

# Push predictions for all tasks in project 1
nemish label-studio predict \
    --checkpoint runs/20260621_143052/checkpoints/best.pth \
    --project-id 1

Common workflows

Dry-run first — simulate without uploading to confirm the model produces sane output on your imagery:

nemish label-studio predict \
    --checkpoint runs/run_name/checkpoints/best.pth \
    --project-id 1 --dry-run

Target specific tasks — only pre-annotate tasks 42, 43, and 44:

nemish label-studio predict \
    --checkpoint runs/run_name/checkpoints/best.pth \
    --project-id 1 --task-ids 42 43 44

Skip already-predicted tasks — safe to re-run after adding new images to the project; tasks that already have predictions are left alone:

nemish label-studio predict \
    --checkpoint runs/run_name/checkpoints/best.pth \
    --project-id 1 --skip-existing

Use GPU for faster inference on projects with many images:

nemish label-studio predict \
    --checkpoint runs/run_name/checkpoints/best.pth \
    --project-id 1 --device cuda

Tune batch processing for higher throughput on large projects:

# Increase GPU batch size (default 8 — try 16 or 24 on GPUs with ≥16 GB VRAM)
nemish label-studio predict \
    --checkpoint runs/run_name/checkpoints/best.pth \
    --project-id 1 --device cuda \
    --inference-batch-size 16

# Speed up task listing with page size (default 100)

The pipeline processes images in batches for maximum GPU utilisation:

1. Downloads a batch of images from S3 in parallel (up to 8 concurrent requests)
2. Runs GPU inference on the entire batch at once
3. Uploads predictions to Label Studio in parallel (up to 8 concurrent requests)

This keeps the GPU busy and minimises idle time waiting for I/O. Progress is reported every 10 seconds with tasks/second, success/error counts, and ETA.

Task list caching

The task list is cached locally to avoid re-downloading it on every run (default TTL: 60 minutes). The cache lives at .cache/tasks_<project_id>.json.

Important: If predictions are created (by any process) while a cached task list is still valid, a subsequent run will use the stale cache and may skip the anti-duplicate check — potentially creating duplicate predictions on tasks that were already annotated since the cache was written. To force a fresh task list:

# Delete the cache file before running
rm .cache/tasks_1.json

Image downloads from S3 are never cached — only the task metadata list is.

Use a different label name if your labeling config doesn't use "Swimming Pool":

nemish label-studio predict \
    --checkpoint runs/run_name/checkpoints/best.pth \
    --project-id 1 --label-name "Pool" \
    --from-name "labels" --to-name "image"

What annotators see

When a task has a prediction, Label Studio displays the pre-filled mask alongside the image. Annotators can:

• Press Ctrl+Enter to accept the prediction as-is (it becomes the annotation).
• Use the brush/eraser tools to refine the mask.
• Delete the prediction and draw from scratch if the model was wrong.

Iterative improvement (active learning)

This workflow works particularly well in a loop:

1. Label a small initial set of images in Label Studio (50–100 tiles).
2. Export the annotations and train a first model (following the sections above).
3. Pre-annotate the remaining unlabeled tiles with the model.
4. Review and correct the predictions — much faster than labeling from scratch.
5. Retrain with the expanded dataset (original labels + corrected predictions).
6. Repeat until the model is good enough that you only need to spot-check.

Each iteration should improve the model, which in turn produces better pre-annotations, which makes each labeling pass faster.

Labeling config requirements

Your Label Studio project must use a Brush with nested Labels tag (semantic segmentation with a brush tool). The default tag names match the setup described in Exporting from Label Studio:

<View>
  <Image name="image" value="$image"/>
  <Brush name="pool" toName="image">
    <Labels name="labels" toName="image">
      <Label value="Swimming Pool" />
    </Labels>
  </Brush>
</View>

If your config uses different tag names, pass --from-name, --to-name, and --label-name to match.

---

Multi-model training

The project supports multiple model architectures registered through a ModelSpec abstraction. Two families are included:

Model Type	Architecture	Task	Output
segmentation	SMP UNet / DeepLabV3+ / FPN / MANet	Semantic segmentation	Per-pixel class mask
detection	Faster R-CNN (ResNet-50 FPN)	Object detection	Bounding boxes + scores

Training different model types

# Train a segmentation model (default)
nemish train

# Train a detection model (bboxes derived from masks automatically)
nemish train \
    model.model_type=detection \
    training.batch_size=4 \
    training.learning_rate=0.0005

Both models use the same data/ directory layout (images + masks). The detection model derives bounding boxes from masks via connected-component labeling — no separate annotation format needed.

Analysing specific Label Studio tasks

Download images from specific LS tasks, run inference with multiple models, and save predictions for visual comparison:

nemish label-studio check-tasks \
    -p 8 \
    --task-ids 122497 122498 122499 \
    --checkpoints runs/seg/best.pth runs/det/best.pth \
    --labels "Segmentation" "Detection" \
    --include-ground-truth \
    --device cuda

The output directory layout under /data/check-tasks/:

/data/check-tasks/
├── images/               # source images downloaded from S3
│   ├── task_122497.png
│   └── ...
├── masks/                # ground truth masks (if --include-ground-truth)
│   ├── task_122497.png
│   └── ...
├── Segmentation/         # predictions from model A
│   ├── task_122497_mask.png
│   └── ...
├── Detection/            # predictions from model B
│   ├── task_122497_mask.png
│   └── ...
└── models.json           # metadata about the models used

Interactive comparison viewer

nemish web --mode check-tasks
# Open http://localhost:8080 in a browser
# Remote server: ssh -L 8080:localhost:8080 your-server

The comparison viewer shows the source image, ground truth mask, and one colour-coded prediction panel per model. Use the left/right arrow keys or the on-screen buttons to navigate.

Static comparison report

For batch metrics across an entire data directory:

nemish compare-models \
    --checkpoints runs/seg/best.pth runs/det/best.pth \
    --labels "UNet" "Faster-RCNN" \
    --data-dir data/ \
    --output comparisons/run1 \
    --max-samples 50 \
    --device cuda

Generates per-image visualizations in comparisons/run1/visualizations/ and a metrics summary JSON at comparisons/run1/metrics.json.

---

Archiving and reusing models

Once you have a well-performing model, you can package it into a portable archive for transfer between inference systems or long-term storage.

Archive a run

# Package a training run (best + last checkpoints, tensorboard, summary)
nemish archive runs/stadia_seg_v1

# Custom output path
nemish archive runs/stadia_seg_v1 -o models/nemish-detector-v2.tar.gz

The archive contains only the essential outputs — no input data or intermediate epoch snapshots:

stadia_seg_v1/
├── checkpoints/
│   ├── best.pth          # best validation checkpoint
│   └── last.pth          # final epoch (resume-capable)
├── tensorboard/          # metric history
│   └── events.out.tfevents...
└── summary.json          # metrics, model config, parameter count

Inspect an archive

nemish archive --list models/stadia_seg_v1.tar.gz

Prints model type, best metrics, parameter count, and file listing.

Transfer and reuse

Copy the archive to another system and extract:

scp models/stadia_seg_v1.tar.gz gpu-server:/home/user/models/
ssh gpu-server
tar xzf models/stadia_seg_v1.tar.gz

The extracted directory works directly with inference and Label Studio scripts:

# Single-image inference
python -m src.inference.predict \
    --checkpoint stadia_seg_v1/checkpoints/best.pth \
    --image tile.png --device cuda

# Label Studio pre-annotation
nemish label-studio predict \
    --checkpoint stadia_seg_v1/checkpoints/best.pth \
    --project-id 1 --device cuda

# Resume training from the last checkpoint
nemish train \
    training.resume_from=stadia_seg_v1/checkpoints/last.pth

The checkpoint format is self-contained — it embeds all model type and architecture metadata, so no config file is needed to reconstruct the model for inference.

---

Project structure

src/
├── cli.py                     # CLI entry point (nemish command)
├── configs/
│   └── default.yaml           # Bundled training config
├── train.py                   # Training entry point
├── archive.py                 # Package training runs into portable archives
├── compare.py                 # Multi-model comparison
├── label_studio.py            # Label Studio integration
├── s3.py                      # S3-compatible storage utilities
├── image_utils.py             # RLE decoding, polygon rasterization
├── workers.py                 # Progress bar and parallel worker tasks
├── data/
│   └── dataset.py             # PoolDataset + DataLoader builder + bbox utilities
├── models/
│   ├── spec.py                # ModelSpec base class (ABC)
│   ├── registry.py            # Model registry: map names → spec instances
│   ├── segmentation.py        # SegmentationSpec (SMP UNet/DeepLabV3+/FPN/MANet)
│   ├── detection.py           # DetectionSpec (torchvision Faster R-CNN)
│   └── factory.py             # count_parameters() utility
├── training/
│   ├── trainer.py             # Model-type-agnostic training loop
│   ├── metrics.py             # IoU, Dice, pixel accuracy, box IoU/F1
│   └── losses.py              # DiceLoss, CombinedLoss (CE + Dice)
├── inference/
│   └── predict.py             # CLI for running a trained model on new images
├── utils/
│   └── config.py              # Typed configuration dataclass
└── web/
    ├── server.py              # HTTP server for data / model visualisation
    ├── helpers.py             # Overlay generation, stem collection
    └── templates/             # HTML templates

tests/
└── test_metrics.py             # Unit tests for metrics and losses

---

## Common issues

**Training produces NaN/Inf loss**

This usually means one of:
- **Missing or incorrect normalization**: The dataset must apply ImageNet
  normalization (`mean=[0.485,0.456,0.406]`, `std=[0.229,0.224,0.225]`).
  Pretrained encoders expect this — raw [0,1] pixel values cause extreme
  activations.  Verify that `src/data/dataset.py` applies normalization
  in `PoolDataset.__getitem__()`.
- **Loss/optimizer mismatch**: The model uses `CombinedLoss` (CrossEntropy +
  Dice) with `AdamW`.  If you've modified the loss function, make sure it's
  compatible with the model's 2-class output and the optimizer type.
- **Broken masks**: All masks should be 0 (background) or 255 (pool) pixels,
  then divided to 0/1 during loading.  Non-binary masks produce unstable
  gradients.  Run ``nemish label-studio convert`` to regenerate.

Run a CPU smoke test (`training.max_train_samples=50 num_workers=0`) first —
NaN on CPU is easier to debug and doesn't cost GPU time.

**Validation IoU is zero or near-zero**
→ The model is predicting all-background.  Common causes: (1) masks have
intermediate values (not binary 0/1), (2) `CombinedLoss` CE weight is too
high relative to Dice, (3) extreme foreground/background imbalance combined
with small `batch_size`.  Try reducing CE weight: in `src/training/losses.py`,
change `CombinedLoss(ce_weight=0.3, dice_weight=0.7)`.

**Training loss decreases but validation IoU flatlines**
→ The model is overfitting on the training set.  This is expected with very
small `max_train_samples`.  Remove it to use the full dataset.  Also try:
add `model.architecture=deeplabv3+` for a stronger decoder, increase
`data.image_size` for more spatial context, or add dropout to the decoder.