nixos-systems

These are the system definitions for various systems being run by Gleipnir Technology.

Current Method

Convert to NixOS with nixos-anywhere

First log in to the host using regular credentials. Set up an ssh key for root access.

SSH key for root access

Assuming you're using a Debian base for these instructions.

$ su
# apt install sudo
# echo 'ssh-ed25519 AAA....JGTm3 me@somewhere' > ~/.ssh/authorized_keys
# chmod 600 ~/.ssh/authorized_keys

Now log out and see if you can SSH into the host as root. nixos-anywhere will be using SSH as root quite a bit.

Copy a baseline config

Start by copying an existing host config to a new subdirectory. For me right now I'm working on an AMD Legacy Quadrcore, so I"ll copy one of those:

cp host/nocix/123456 host/nocix/654321

You'll then need to update flake.nix to have an entry for the new host configuration.

Configure root disk

Get the disk layout using /sbin/fdisk -l. You're looking to figure out which disk is the boot disk and which isn't. Use ls -l /dev/disk/by-id/ to figure out the ID, which should be stable across reboots. Then update the host/nocix/amd-legacy-quadcore-123456/disk-config.nix file for the matching provider to ensure that the boot disk gets written.

Configure network

Then check the network configuration via ip route and ip addr or /etc/network/interfaces. Update the network configuration at host/nocix/amd-legacy-quadcore-123456/network.nix to match.

Generate the hardware configuration

Generate the hardware configuration

# apt install curl
# curl --proto '=https' --tlsv1.2 -L https://nixos.org/nix/install | sh -s -- --daemon
# nix-shell -p nixos-install-tools
# nixos-generate-config

This will generate files at /etc/nixos. You specifically want the /etc/nixos/hardware-configuration.nix. Copy it to the new host config on the management system.

At this point you have a fully-defined system in ./host/nocix/654321. In order to run nixos-anywhere you'll need to copy its configuration to the area that nixos-anywhere reads from.

cp ./host/nocix/654321/* nixos-anywhere/nocix/

This should copy over the configuration.nix, disk-config.nix, hardware-configuration.nix, and networking.nix files over which nixos-anywhere will read from. Before you can continue, comment out the auto-generated fileSystems entries from hardware-configuration.nix, since they'll conflict with the disko-based config in disk-config.nix:

/*
  fileSystems."/" =
    { device = "/dev/disk/by-uuid/6760cb51-d755-4499-8c72-89237ac7e2df";
      fsType = "ext4";
    };

  fileSystems."/boot/efi" =
    { device = "/dev/disk/by-uuid/F49C-CDD1";
      fsType = "vfat";
      options = [ "fmask=0077" "dmask=0077" ];
    };

  swapDevices =
    [ { device = "/dev/disk/by-uuid/ec2e393e-16ca-4196-9b2f-66695e5fcd1e"; }
    ];
*/

Now you can then actually deploy the NixOS anywhere with:

$ cd nixos-anywhere
$ nix run github:nix-community/nixos-anywhere -- --flake ./#nocix --target-host root@1.2.3.4

This will take a while, maybe 10 minutes, and disconnect and reboot the server. From there you can ping it and wait for it to come back, usually around 10 minutes again. I think. I don't watch it.

Deploy full system

At this point you've got a server that has NixOS on it, but it's just a bare system missing most of its purpose. That's because we used nixos-anywhere and a special flake instead of our regular flakes.

You can start with copyng files from a working host

$ cp -R host/nocix/amd-legacy-quadcore-123 host/nocix/amd-legacy-quadcore-456

Then copy over the configuration files created during the nixos-anywhere steps above

$ cp nixos-anywhere/nocix/hardware-configuranion.nix nixos-anywhere/nocix/network.nix host/nocix/amd-legacy-quadcore-456

Before this will work you need to commit the files created in the above steps - nix flakes ignore files on disk that aren't in source code management if you're operating in a git repo context.

This may take a while, maybe 20 minutes, but after you'll have a fully-functioning NixOS system with the correct SSH keys.

Adding a new host to the secrets

The host should create its own new ssh host key that lives at /etc/ssh/ssh_host_ed25519_key.pub. We're going to convert that into a sops key and add it to our key material.

On the server you just added run:

$ nix-shell -p ssh-to-age --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'
age1lzzlx60f9ra4evdkn4l9px735mz7uxml5467ptzc4hg3t86gn9mq3ddsxy

This will produce an age key. Copy that into .sops.yaml. You'll want to add it to the list of keys at the top, then add it to any sections that the host should be able to read. Once that's done you'll need to add the key into each secret file's encryption with the following on a dev machine:

$ nix-shell -p sops --run "sops updatekeys secrets/example.yaml"

See the official sops-nix docs for details.

Switch-fix

I've added a tool, switch-fix to the deployment. From the docs:

"you set-rollback before you do your nixos-rebuild switch."
"you do cancel-rollback if it works"
"you wait if it doesn't, and the system should come back."

Successful avenues of exploration

nixos-anywhere

See failed section for stuff before this

Eventually got around to building a s-2vcpu-4gb system:

$ nix run github:nix-community/nixos-anywhere -- --flake ./nixos-anywhere/flake.nix#digitalocean --target-host root@128.199.4.31

This worked and I was then able to ssh in as root. Interestingly, it has no /etc/nixos/* files (but the directory does exist). Resulting disk images:

# fdisk -l
Disk /dev/vda: 80 GiB, 85899345920 bytes, 167772160 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E0C9241B-89EA-4E2C-A0CD-04A3FDEFEDA2

Device       Start       End   Sectors  Size Type
/dev/vda1     2048      4095      2048    1M BIOS boot
/dev/vda2     4096   1028095   1024000  500M EFI System
/dev/vda3  1028096 167770111 166742016 79.5G Linux filesystem


Disk /dev/vdb: 482 KiB, 493568 bytes, 964 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pool-root: 79.51 GiB, 85370863616 bytes, 166739968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Next we'll try something at half that size

s-1vcpu-2gb works fine. We won't bother going smaller at this point.

Failed Avenues of Exploration

These are things that I wanted to have work, but just didn't in the way I had hoped.

Custom System Image

The idea here is to use NixOS to create a custom system image, upload it, and then start servers based on that image. It actually works really well, except that Digital Ocean categorically does not support IPv6 networking on custom images. That's bollocks.

Hat tip to the negation for some ideas here.

Build a custom image for Digital Ocean:

> nix-build digitalocean/custom-image.nix
...
/nix/store/rm84j1a5bskhg2z8gz633m4apjyg848c-digital-ocean-image
> ls -lh result/
nixos-image-digital-ocean-25.05pre-git-x86_64-linux.qcow2.gz  nix-support/

In order to "upload" the image to Digital Ocean you'll need to make the image available via URL. We can use Gleipnir static storage as an example:

rsync result/nixos-image-digital-ocean-25.05pre-git-x86_64-linux.qcow2.gz static.gleipnir.technology:/tmp

Make sure the image is accessible via a public URL. Then upload either through the web interface, or using doctl

> doctl compute image create "Gleipnir NixOS 25.05" -v --image-description "NixOS 25.05 with ssh keys for eliribble baked in" --image-distribution nixos-25.05 --image-url "https://static.gleipnir.technology/nixos-image-digital-ocean-25.05pre-git-x86_64-linux.qcow2.gz" --region sfo3 --tag-names nixos
ID           Name                    Type      Distribution    Slug    Public    Min Disk    Created
192948683    Gleipnir NixOS 25.05    custom    Unknown OS              false     0           2025-07-10T20:22:43Z1G

Then start a droplet using that image:

> doctl compute droplet create "test2.nidus.cloud" --enable-ipv6 --image 192948683 --project-id ce2159e8-02f5-4169-8943-f34ccf812d23 --region sfo3 --size s-1vcpu-1gb --ssh-keys 48777034 --tag-name nixos --wait
Error: POST https://api.digitalocean.com/v2/droplets: 422 (request "116c778d-8e72-4099-a7c6-c3ad37557c4c") image is not compatible with ipv6

Oh. Well that sucks. Digital Ocean can't do IPv6 on custom images.

NixOS Infect via cloud-init

The idea here is to spawn a stock Debian system from Digital Ocean's supported image, then immediately infect it on boot to become a real NixOS system, but with proper IPv6 networking

I tried creating a cloud-init function based on NixOS-infect. You can see the content in digitalocean/infect-nixos.yaml. I added it to the startup command via doctl compute droplet create ... --user-data-file digitalocean/infect-nixos.yaml. This may have a way of working, but I don't get a log and it doesn't get infected, so something fundamental isn't working. I abandoned it.

Nixos-anywhere Investigation

First we start up a really small s-1vcpu-1gb. Then we try to install nixos via nixos-anywhere. Notice the --no-disko-deps which is recommended for very low RAM systems:

$ nix run github:nix-community/nixos-anywhere -- --no-disko-deps --flake ./nixos-anywhere#digitalocean --target-host root@64.23.242.187

After an hour it was railed on the CPU at 100% and had been for an hour with no network data going anywhere. I gave up. Must be too small. Tried again with a larger system, s-2vcpu-4gb:

See successful investigations for what happened after that.

Garage

In order to set up garage initially you'll need to get the RPC secret out of /var/run/secrets/garage-env on a garage node, or from secrets/garage-env. You'll then run

sudo -u garage garage --rpc-secret <secret> status

To see garage. You'll start with output like:

==== HEALTHY NODES ====
ID                Hostname                   Address               Tags  Zone  Capacity          DataAvail  Version
eb133fbcbabc2f18  nocix-amd-legacy-octacore  107.150.59.202:10110              NO ROLE ASSIGNED             cargo:2.2.0

You'll need to configure garage based on "Creating a cluster layout" from https://garagehq.deuxfleurs.fr/documentation/quick-start/:

Creating a cluster layout for a Garage deployment means informing Garage of the disk space available on each node of the cluster using the -c flag, as well as the name of the zone (e.g. datacenter) each machine is located in using the -z flag.

For our test deployment, we are have only one node with zone named dc1 and a capacity of 1G, though the capacity is ignored for a single node deployment and can be changed later when adding new nodes.

sudo -u garage garage layout assign -z dc1 -c 1G <node_id>

where <node_id> corresponds to the identifier of the node shown by garage status (first column). You can enter simply a prefix of that identifier. For instance here you could write just garage layout assign -z dc1 -c 1G 563e.

The layout then has to be applied to the cluster, using:

sudo -u garage garage layout apply --version 1

for our use that would be:

$ sudo -u garage garage --rpc-secret <secret> layout assign -z dc1 -c 10G eb133fbcbabc2f18
Role changes are staged but not yet committed.
Use `garage layout show` to view staged role changes,
and `garage layout apply` to enact staged changes.
$ sudo -u garage garage --rpc-secret <secret> layout apply --version 1
==== COMPUTATION OF A NEW PARTITION ASSIGNATION ====

Partitions are replicated 1 times on at least 1 distinct zones.

Optimal partition size:                     39.1 MB
Usable capacity / total cluster capacity:   10.0 GB / 10.0 GB (100.0 %)
Effective capacity (replication factor 1):  10.0 GB

dc1                 Tags  Partitions        Capacity  Usable capacity
  eb133fbcbabc2f18  []    256 (256 new)     10.0 GB   10.0 GB (100.0%)
  TOTAL                   256 (256 unique)  10.0 GB   10.0 GB (100.0%)


New cluster layout with updated role assignment has been applied in cluster.
Data will now be moved around between nodes accordingly.
$ sudo -u garage garage --rpc-secret <secret> bucket create nix-cache
==== BUCKET INFORMATION ====
Bucket:          84271248d9358db636b86a466c54b1b0dbf46509ebb7d464b3da8fd95142c94a
Created:         2026-05-22 22:26:04.432 +00:00

Size:            0 B (0 B)
Objects:         0

Website access:  false

Global alias:    nix-cache

==== KEYS FOR THIS BUCKET ====
Permissions  Access key    Local aliases
$ sudo -u garage garage --rpc-secret <secret> key create nix-cache-key
==== ACCESS KEY INFORMATION ====
Key ID:              <keyid>
Key name:            nix-cache-key
Secret key:          <keysecret>
Created:             2026-05-22 22:26:56.939 +00:00
Validity:            valid
Expiration:          never

Can create buckets:  false

==== BUCKETS FOR THIS KEY ====
Permissions  ID  Global aliases  Local aliases
$ sudo -u garage garage --rpc-secret <secret> bucket allow --read --write --owner nix-cache --key nix-cache-key
==== BUCKET INFORMATION ====
Bucket:          84271248d9358db636b86a466c54b1b0dbf46509ebb7d464b3da8fd95142c94a
Created:         2026-05-22 22:26:04.432 +00:00

Size:            0 B (0 B)
Objects:         0

Website access:  false

Global alias:    nix-cache

==== KEYS FOR THIS BUCKET ====
Permissions  Access key                                 Local aliases
RWO          GKa3e0921fe0d3955fa8fee429  nix-cache-key

Then you add the key ID and secret to the correct secrets file for attic.

Connection settings

For various programs that connect to S3 via the usual connection settings, here's what you'll use for garage:

• Bucket Name - whatever you set
• Region Name - 'garage'
• S3 Endpoint - https://s3.garage.gleipnir.technology
• Access Key ID - created when you create a key
• Secret Access Key - created when you create a key

Nix Cache

We're doing nix caching via attic. In order to push new builds from the Forgejo Action runners to the nix cache and perform deployments there is some manual work beyond just running nix.

Initial Setup

The very first time you set up a corp-like server you'll need to:

$ cat /etc/systemd/system/atticd.service | grep atticd
...
/nix/store/ymln4v0gfczxbi4a6g2cmwdfh8n12sdw-attic-0-unstable-2025-09-24/bin/atticd
/run/atticd/server.toml
...
$ /nix/store/ymln4v0gfczxbi4a6g2cmwdfh8n12sdw-attic-0-unstable-2025-09-24/bin/atticadm -f /run/atticd/server.toml make-token --validity 10y --push '*' --pull '*' --create-cache '*' --sub 'admin'
eyJhbGciOiJIUzI1NiI...snip...kaFiHM7UMGU
$ attic login gleipnir https://nix-cache.gleipnir.technology <secret from above>
✍️ Configuring server "gleipnir"
$ attic cache create staging
✨ Created cache "staging" on "gleipnir"
$ attic cache create prod
✨ Created cache "prod" on "gleipnir"

Testing runners

You'll need to generate tokens through the attic daemon. This can be done on the server running atticd (corp) with

$ /nix/store/ymln4v0gfczxbi4a6g2cmwdfh8n12sdw-attic-0-unstable-2025-09-24/bin/atticadm -f /run/atticd/server.toml make-token --validity 10y --push '*' --pull '*' --sub 'eliribble'

You need to SSH in to the action runner and login to attic with attic login gleipnir https://nix-cache.gleipnir.technology <token>

At that point you can do a sample build-and-push cycle with

$ nix build
$ attic push staging result

This allows you to mimic what the runners will do

Updating runners

Runners get their attic config from an attic config file written as part of the forgejo-action-runner. The key is shared between them and created with:

$ /nix/store/ymln4v0gfczxbi4a6g2cmwdfh8n12sdw-attic-0-unstable-2025-09-24/bin/atticadm -f /run/atticd/server.toml make-token --validity 10y --push '*' --pull '*' --sub 'nocix-amd-legacy-octacore-292465'

This is then added to the secrets/attic-config.yaml file. The runners should be able to push newly built derivations to the cache as part of their workflow definitions with that config.