Table of Contents
WIP !
top of head notes
think of way to NOT map host ports on docker (22:22) rather map to an arbitrary port (22:3083) and still get the build to recognize the remote builder
custom seccomp filter for cross-compilation builds
why would i use nix and docker ?
this is probably the question on everyones mind right now and while many of the nix users i have conversed with started using nix purely to get away from docker i am here to bring peace between the communities and show how they can be used in tandem to solve nix multi-platform builds
vain pov
the problem arose when i was working on building a nixos iso that would contain a pre configured configuration.nix file so I can install nixos on my old intel macbook pro and turn it into a homelab of sorts and have a reproducible iso that i can use to reproduce my config on other intel macs (i have a lot). obviously downloading and flashing the nixos iso onto a flash drive from https://nixos.org and then putting my configuration.nix file onto the machine would be the easier solution here but out of curiosity and the need for reproducibility i decided to try to build the iso myself using a flake.
here is the current example flake.nix file i am trying to build (i will go into these files ltr):
{
description = "nix docker rb";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
outputs = { self, nixpkgs }:
let
supportedSystems = [
"x86_64-linux"
"aarch64-darwin"
];
# https://zerforschen.plus/posts/why-i-do-not-use-flake-utils/
forAllSystems = nixpkgs.lib.genAttrs supportedSystems;
in
{
nixosConfigurations.iso = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
./configuration.nix
"${nixpkgs}/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix"
];
};
packages = forAllSystems (system: {
iso = self.nixosConfigurations.iso.config.system.build.isoImage;
});
};
}
configuration.nix:
{ pkgs, ... }:
{
nix.extraOptions = "experimental-features = nix-command flakes"
programs.fish.enable = true;
users.defaultUserShell = pkgs.fish;
# packages that will be available globally on the system
environment.systemPackages = with pkgs; [
vim
git
tailscale
curl
];
}
why is this a problem ?
in short darwin platforms are the least supported by nix. we can see this from the nixpkgs reference manual so trying to build most packages on my m2 macbook air result in an error since the package doesnt have support for darwin systems and/or arm/silicon architecture.
is this even a problem ?
yes and no. multiplatform builds with nix are "solved" usually with 5 options. I'll dive into each one and why I decided against going that route.
- A remote builder with the correct architecture and OS to support your build (this can be a physical/virtual machine with nix installed on it)
physical/virutal machines come with their problems/limitations. physical machines obviously require you to have another machine that matches the OS and platform needed for the build and then requires you to setup port forwarding or tailscale on the machine so it can be accessed from anywhere. virtual machines on macos come with limited options as well you can decide to pay for a parallels license or use virtual box/UTM to keep things free but all of these options require tedious setup and having to interact with a ui which is not my preferred method of building things since I would like for builds to be automated and easily reproducible.
- CI builder i.e github actions runner building your flake
this option is definetly something you want to implement no matter what but unfortunatlely it doesn't make sense to use for local development due to it being so time consuming. I don't want to push to github for every test change just to verify if the change will actually build/run on top of an inevitably messy commit.
- Cross compilation with dockerTools or hydra
this option is kind of split into three. while cross compilation exists for nix its documentation is incomplete and trying to figure out which dependencies are missing for every cross compilation build is VERY time consuming and results in a messy flake
cross compilation with dockerTools is also in theory a solution. I haven't seen it successfully done myself and for my use case it didn't make sense to attempt since dockerTools requires the linux OS to build (which is part of the problem I am trying to solve) also images built with dockerTools tend to be larger in image size as well
theory dockerTools cross compilation (requires a linux builder):
{
outputs = { self, nixpkgs }: {
# Regular build
# packages.${localsystem}.${name} = drv { import pkgs = nixpkgs.legacypackages.${localsystem}; }
packages.aarch64-darwin.containerImage = import ./docker.nix { pkgs = nixpkgs.legacyPackages.aarch64-darwin; };
# Cross build
# packages.${localSystem}."${name}-${crossSystem}" = drv { import nixpkgs { localSystem = localSystem; crossSystem = crossSystem;}
packages.aarch64-darwin.containerImage-x86_64-linux = import ./docker.nix { pkgs = import nixpkgs { localSystem = "aarch64-darwin"; crossSystem = "x86_64-linux"; }; };
};
}
from my research ive also heard mentions of using hydra for cross compilation but i've never seen it working as intended
- enable NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM
you can enable the NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM environment variable on your host system but this is more of a hammer and nail approach and won't fix most multiplatform builds.
- nix-darwin linux-builder
nix-darwin also mentions in their docs that it contains a linux builder for nix builds on macos that require a linux OS. this was my initial approach but this also means installing nix-darwin onto your host system and you can't define the platform for the builder from what i've seen which doesnt solve the main multiplatform issue.
da solution waow
we will use docker as a remote builder so I can specify the platform and operating system I would like to use for the build. in my use case I need an x86_64 linux builder but i will also show how you can potentially build arm packages as well.
dockerfile big waow
FROM nixos/nix:2.31.1-amd64
RUN set -eux; \
{ \
echo "filter-syscalls = false"; \
echo "max-jobs = auto"; \
echo "cores = 0"; \
} >> /etc/nix/nix.conf
RUN nix-env -f '<nixpkgs>' -iA \
gnused \
openssh \
&& nix-store --gc
RUN set -eux; \
mkdir -p /etc/ssh /var/empty /run /var/log /root/.ssh && \
touch /var/log/lastlog && \
echo "sshd:x:498:65534::/var/empty:/run/current-system/sw/bin/nologin" >> /etc/passwd; \
cp /root/.nix-profile/etc/ssh/sshd_config /etc/ssh/sshd_config && \
sed -i 's/root:!:/root:*:/' /etc/shadow; \
sed -i '/^PermitRootLogin/d' /etc/ssh/sshd_config && echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config; \
ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N "" -t rsa && \
ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N "" -t ed25519 && \
echo "source /root/.nix-profile/etc/profile.d/nix.sh" >> /etc/bashrc && \
echo "source /etc/bashrc" >> /etc/profile
COPY entrypoint.sh /
RUN chmod 755 /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
entrypoint script small waow
#!/usr/bin/env bash
set -e
echo "$1" > /root/.ssh/authorized_keys
sshd=$(readlink -f $(which sshd))
exec $sshd -D -e
dockerfile deep dive
lets go into what im actually telling you to run here before you think im using your computer for crypto mining.
we are using the official nix amd64 image version 2.31.1
RUN set -eux; \
{ \
echo "filter-syscalls = false"; \
echo "max-jobs = auto"; \
echo "cores = 0"; \
} >> /etc/nix/nix.conf
in the first block we are appending the following to our nix configuration:
filter-syscalls = false
the nix daemon runs as root since it needs run priveleged operations i.e starting a builder under a different uid so it natively enables seccomp filtering (SECure COMPuting). seccomp is a linux kernel feature that adds the seccomp() system call which allows a process (in our case the nix-daemon) to restrict the syscalls it can make by passing and comes with two modes: strict mode which limits a process to only 4 syscalls and filter mode or seccomp-bpf. seccomp filtering provides a means for a process to specify a specific filter for incoming syscalls you can think of it as a "syscall firewall". the nix-daemon creates and installs a seccomp-bpf filter.
cBPF and eBPF
classic BPF or cBPF is a small bytecode with two 32-bit registers to perform basic filtering on packets and syscalls. we use cBPF bytecode to populate the code field in the following sock_filter struct. the kernel then automatically translates this cBPF bytecode to eBPF which is the modern version of cBPF. more on cBPF and eBPF here
filter mode
Filter mode is an improvement to the original strict mode where allowed syscalls are defined by a pointer to a Berkeley Packey Filter (BPF). the nix daemon provides a c struct sock_fprog which is just a pointer to an array of sock_filter instructions:
struct sock_fprog {
unsigned short len; /* Number of BPF instructions */
struct sock_filter *filter; /* Pointer to array of
BPF instructions */
};
a filter program must contain one or more of the following BPF instructions:
struct sock_filter { /* Filter block */
__u16 code; /* Actual filter code (where the cBPF bytecode lives) */
__u8 jt; /* Jump true */
__u8 jf; /* Jump false */
__u32 k; /* Generic multiuse field */
};
when a process makes a system call the kernel pauses and creates a seccomp_data struct which contains information about our syscall attempt sort of like an "id-card":
struct seccomp_data {
int nr; /* System call number */
__u32 arch; /* cpu architecture making the call (where our problem most likely lies) */
__u64 instruction_pointer; /* where in the program the syscall was made */
__u64 args[6]; /* args passed to syscall */
};
in our case the nix-daemon makes the seccomp() syscall to load the BPF program (the array of sock_filter structs defined in sock_fprog) into the kernel. after the kernel translates the cBPF bytecode to eBPF it executes our BPF program passing in the seccomp_data struct as its input. if an invalid value is detected the syscall fails returning EINVAL or "Invalid Argument". since the nix build is being executed from our host machine the syscalls have to be translated through the emulation layer before reaching our amd container and since the translation of syscalls between arm and amd is not perfect this is more than likely where most most of the nix cross compilation errors lie.
Heres an example error when filter-syscalls is set to true to further support this:
$ nix build .#packages.x86_64-linux.docker --builders 'ssh://nix-docker x86_64-linux'
error: build of '/nix/store/yaf6fy6wdjfkd2qka9ifgjsasy9zj2q1-build-spago-style.drv' on 'ssh://nix-docker' failed: while setting up the build environment: unable to load seccomp BPF program: Invalid argument
error: builder for '/nix/store/yaf6fy6wdjfkd2qka9ifgjsasy9zj2q1-build-spago-style.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/jcaq8l839jcv1w5vb7mli4rkyy9pkc38-lions-client.drv' failed to build
error: 1 dependencies of derivation '/nix/store/0xw07gaii4r1syn28fkn8ays101z2r45-lions-all-client-assets.drv' failed to build
error: 1 dependencies of derivation '/nix/store/jhf2ka90dpbjfq6ajyv7mkn1l7jmj92y-lions-website.drv' failed to build
error: 1 dependencies of derivation '/nix/store/dpg0qb0valcxppm2l5f5g41kfn66isbk-server-config.json.drv' failed to build
error: 1 dependencies of derivation '/nix/store/niqlffyvy725j844dwxivys13pb84kn0-docker-image-server.tar.gz.drv' failed to build
max-jobs = auto,cores = 0
these configurations are being set in the name of faster builds and enabling parallelism.
max-jobs is the max number of jobs nix will try to build locally in parallel. we use the special value auto to tell nix to use all of the cpus on our system for parallelism.
cores defines the number of cpu cores to use for each build job. it also contains a special value 0 which allows nix to use all available cpu cores in the system.
in the next portion of our dockerfile:
RUN nix-env -f '<nixpkgs>' -iA \
gnused \
openssh \
&& nix-store --gc
we are simply installing sed for parsing text and openssh so we can communicate with the container via ssh.
nix-store --gc is used to clean up any unnecessary packages living in the nix store.
RUN set -eux; \
mkdir -p /etc/ssh /var/empty /run /var/log /root/.ssh && \
touch /var/log/lastlog && \
echo "sshd:x:498:65534::/var/empty:/run/current-system/sw/bin/nologin" >> /etc/passwd; \
cp /root/.nix-profile/etc/ssh/sshd_config /etc/ssh/sshd_config && \
sed -i 's/root:!:/root:*:/' /etc/shadow; \
sed -i '/^PermitRootLogin/d' /etc/ssh/sshd_config && echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config; \
ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N "" -t rsa && \
ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N "" -t ed25519 && \
echo "source /root/.nix-profile/etc/profile.d/nix.sh" >> /etc/bashrc && \
echo "source /etc/bashrc" >> /etc/profile
we will focus on the important parts of this RUN block.
echo "sshd:x:498:65534::/var/empty:/run/current-system/sw/bin/nologin" >> /etc/passwd;
since we are installing openssh to our nix profile we need to create the necessary sshd unpriveleged user or else docker will throw an