Skip to content

Koios: private per-job /tmp on the local NVMe

Since 10 August 2026 every job on the Koios compute nodes gets its own /tmp, on the node's NVMe disk, that no other job can see and that is removed when the job ends. The rollout started on 30 July.

How it is set up

The change in slurm.conf on slurmserver2.koios was one line (PrologFlags=contain was already set):

NamespaceType=namespace/linux

The plugin is configured in /etc/slurm/namespace.yaml (last edited on 10 August 2026):

defaults:
  base_path: "none"
node_confs:
  - nodes: ["n1", "n2", ..., "n27"]   # shortened: the file lists n1 to n27 one per line
    options:
      auto_base_path: false
      base_path: "/mnt/.nvme"
      dirs: "/tmp"
      shared: true
  • base_path: "/mnt/.nvme": on each node, the 1.7 TB NVMe volume vg0-tmp is mounted at /mnt/.nvme. Each job gets a directory /mnt/.nvme/<jobid>.
  • dirs: "/tmp": that directory is mounted as /tmp inside the job's own mount namespace, so the job sees only its own files.
  • shared: true: mounts made on the node after the job starts (for example by autofs) still propagate into the job's namespace.
  • defaults: base_path: "none": nodes not listed (the login and small nodes) keep a normal /tmp.

When the job ends, Slurm removes /mnt/.nvme/<jobid> and everything in it. On a node, findmnt | grep /mnt/.nvme lists the directories of the running jobs.

Phoebe

Phoebe does not use a namespace plugin (NamespaceType is unset). /tmp on its nodes is one NVMe partition shared by all jobs, and the only cleanup is the standard systemd-tmpfiles rule that deletes files not used for 10 days. Users are told to use /tmp/$USER/$SLURM_JOB_ID and remove it themselves; see storage and software.

Koios is the test bed for this setup. Once it has proven itself there, Phoebe will get the same configuration.

On Koios the plugin was first enabled on one canary node, n10, on 30 July. Then shared: true was added so that CVMFS mounts made by autofs reach running jobs. After that the remaining nodes were added one at a time, from n1 on 31 July to n27 on 10 August. Phoebe can follow the same order, and its nodes already have a separate NVMe volume vg0-tmp mounted at /tmp (1.7 TB on the CPU nodes, 3.2 TB on the GPU nodes).