Sandbox isolation

Isolation here is deliberately layered. No single mechanism is trusted to hold on its own; each exists to contain the failure of the one above it.

The layers

The microVM boundary

When KVM_ENABLED=true, launcher/src/main.rs (a small Rust binary) boots a libkrun microVM and execs into it:

krun_create_ctx()
krun_set_vm_config(ctx, vcpus, ram_mib)   // LAUNCHER_VCPUS, LAUNCHER_RAM_MIB
krun_set_root(ctx, root_path)
krun_add_virtiofs(ctx, tag, path)          // packages and workspace
krun_set_port_map(ctx, port_map)           // egress gateway only
krun_set_rlimits(ctx, rlimits)
krun_set_exec(ctx, exec_path, argv, envp)
krun_start_enter(ctx)

The launcher installs a seccomp filter of its own before entering the guest.

NsJail then runs inside that guest. Untrusted code faces the guest kernel, so a kernel-level exploit compromises a disposable VM whose entire purpose was to run that one job.

No privileged mode and no Linux capabilities are required: only access to /dev/kvm. On Kubernetes a device plugin can provide that without a hostPath (workerSandbox.kvmDevicePlugin).

NsJail configuration

api/config/sandbox.cfg holds the static policy: namespace flags, the mount table, UID/GID mapping, default cgroup limits. Per-execution values (timeouts, memory, the seccomp policy, environment) are passed as CLI arguments by api/src/nsjail.ts.

keep_caps: false        # no capabilities retained
clone_newnet: true      # empty network namespace
clone_newcgroup: true
mount_proc: false       # procfs mounted explicitly, not implicitly

cgroup_mem_max: 536870912
cgroup_mem_swap_max: 0  # no swap: memory limits cannot be evaded
cgroup_pids_max: 64

chroot is used rather than pivot_root, for compatibility with the RuntimeDefault seccomp profile: pivot_root would otherwise force Unconfined at the container level.

The per-job /mnt/data bind is appended as a config overlay rather than passed via NsJail's -B flag, because the CLI form cannot carry noexec, nosuid, and nodev: flags that matter on the one writable mount in the sandbox.

The seccomp policy

Built in api/src/nsjail.ts as a Kafel policy string, in two tiers.

KILL: the process dies with SIGSYS

socket(domain) { domain == AF_VSOCK }
ioctl(fd, request) { (request & 0xFF00) == KVM_IOCTL_MAGIC }

ERRNO: the call fails, the process survives

Some things must fail gracefully, because language runtimes probe for them at startup and would die on SIGSYS.

ERRNO(38) { clone3 }                       # ENOSYS: runtimes fall back to clone
ERRNO(1)  { io_uring_setup, io_uring_enter, io_uring_register,
            sched_setaffinity, vmsplice, ... }

Nested namespace creation:

clone(flags) { (flags & CLONE_NAMESPACE_FLAGS) != 0 }

Signalling the NsJail monitor, PID 1 of the sandbox's PID namespace:

kill(pid)               { pid == 0 || pid == 1 || pid == 0xFFFFFFFF }
tkill(tid)              { tid == 1 }
tgkill(tgid, tid)       { tgid == 1 || tid == 1 }
rt_sigqueueinfo(pid)    { pid == 0 || pid == 1 || pid == 0xFFFFFFFF }
pidfd_open(pid)         { pid == 1 }
pidfd_send_signal

With clone_newpid a tenant cannot reach other tenants, but killing their own namespace's PID 1 still races NsJail's cleanup ordering. pid == 0 (process group) and pid == -1 (everything signalable) are blocked too, since their fan-out would catch the monitor.

EPERM matches the kernel's standard "you may not signal init" behaviour, so runtimes probing with kill(pid, 0) get a familiar error instead of dying.

pidfd_send_signal targets a pidfd rather than a numeric pid, so it cannot be filtered by destination, and a pidfd to PID 1 is obtainable via openat("/proc/1") since Linux 5.4. It is refused outright; no supported runtime uses it.

Socket families:

socket(domain) { domain == AF_INET || domain == AF_INET6 || domain == AF_NETLINK
              || domain == AF_KEY  || domain == AF_RXRPC || domain == AF_ALG }

AF_NETLINK and AF_KEY are kernel-control interfaces. AF_ALG is the kernel crypto API and AF_RXRPC the RxRPC family: both named in past kernel vulnerabilities.

The policy ends USE sandbox DEFAULT ALLOW: an explicit deny-list over a permissive default, since the allowed set for arbitrary language runtimes is impractical to enumerate.

Filesystem layout

/mnt/data/          Working directory: bind mount, writable, noexec/nosuid/nodev
/tmp/               tmpfs (20 MB), writable
/usr/               Host /usr, read-only bind
/bin  -> /usr/bin   merged-usr symlinks
/lib  -> /usr/lib
/lib64 -> /usr/lib64
/proc/              procfs, read-only (Bun requires it)
/pkgs/              Language runtimes, read-only bind
/dev/null, /dev/urandom, /dev/zero

/mnt/data is the only writable location that survives the process, and it is mounted noexec.

procfs masking

Several /proc entries are masked with bind mounts over /dev/null because they leak host information:

• /proc/cgroups, /proc/1/cgroup (cgroup topology and the total cgroup count across the host
• /proc/self/maps) ASLR layout (further mitigated by seccomp blocking ptrace and bpf)

With clone_newnet: true, /proc/net reflects only the sandbox's empty network namespace, so it needs no masking.

Per-job UID isolation

SANDBOX_PER_JOB_UIDS=true
SANDBOX_JOB_UID_BASE=200000
SANDBOX_JOB_GID_BASE=200000

api/src/workspace-isolation.ts leases a distinct UID/GID per concurrent job from a pool. Two jobs on the same runner are then isolated by ordinary filesystem permissions as well as by namespaces: a second, independent mechanism for the same property.

Workspaces are created at mode 0711: not listable, but traversable, which NsJail's bind-mount setup requires on the source path and its ancestors. Staged files are 0600; read-only inputs 0444.

Without per-job UIDs, everything runs as nobody (65534).

Resource limits

cgroup_mem_swap_max: 0 matters more than it looks, with swap available, a memory limit can be evaded by pushing pages out.

SANDBOX_LIMIT_OVERRIDES accepts a JSON object of per-runtime overrides, for runtimes with genuinely different needs.

The setup gate

api/src/nsjail-setup-gate.ts serialises sandbox setup behind a watchdog. Concurrent setup was found to race in ways that could wedge a runner. The watchdog fires when setup exceeds its budget, and codeapi_sandbox_nsjail_setup_gate_watchdog_fires_total should stay at zero in a healthy deployment.

Host-level hardening

Two further layers, both off by default because they need node-level installation:

• seccomp profile: seccomp/ ships a profile applied at the container level, on top of NsJail's inner Kafel policy. Enable with workerSandbox.seccomp.enabled=true; it must be pre-installed on every node at /var/lib/kubelet/seccomp/profiles/nsjail.json.
• AppArmor: apparmor/sandbox-nsjail constrains the sandbox runner process itself.

Without the custom profile, KVM-mode pods use RuntimeDefault and non-KVM pods are forced to Unconfined by NsJail's pivot_root requirement: part of why the chroot choice above exists.

What is deliberately not claimed

• Not a defence against denial of service by resource exhaustion beyond the configured limits. Set them appropriately.
• Side channels (timing, cache) are not addressed.
• NsJail-only mode does not defend against kernel vulnerabilities.
• The safety of dynamic dependencies rests on wheels containing no install-time code, and on the index you point at.