Skip to content

Phoebe

Phoebe is the current CEICO HPC cluster and the main system for new work. Log in through the front-end node phoebe.fzu.cz.

  • 1408 CPU cores
  • 16 NVIDIA A100 GPUs
  • 2 TB RAM per GPU node
  • 218 TB shared storage
  • 100 Gb/s InfiniBand

System overview

Phoebe consists of 20 compute nodes, each with 64 CPU cores (2× AMD EPYC 75431), 512 GB of RAM and 1.7 TB of fast local NVMe2 disk. Two additional "fat" GPU nodes each carry 8 NVIDIA A1003 cards, 2 TB of RAM and 3.4 TB of local NVMe storage. Three small nodes with 8 cores and 64 GB of RAM each serve light interactive work. They are virtual machines, also useful for software that runs better on a small machine than on a large SMP node, such as some legacy codes.

Software, user and project data are stored on 218 TB of hybrid storage built from both solid state and rotational drives. All components are connected by a low-latency 100 Gbit InfiniBand fabric. All nodes run Rocky Linux 8. See storage for where to keep your data.

pcs hostnames resource ncores fcpu (base) fcpu (max) RAM local storage notes
20 n[1-20] CPU compute nodes 64 2.8 GHz 3.7 GHz 512 GB 1.7 TB -
2 gpu[1-2] GPU-accelerated fat nodes 64 2.8 GHz 3.7 GHz 2 TB 3.4 TB 8× NVIDIA A100
3 s[1-3] small nodes for light interactive work 8 - - 64 GB - small_int partition
1 phoebe.fzu.cz login front-end node (virtual machine) 24 vCPU 3.5 GHz 4.0 GHz 384 GB - AMD EPYC 73F3 host

More detail: Phoebe infrastructure. If Phoebe helped your research, please acknowledge it in your publications.

Slurm partitions

Jobs go to the cpu partition unless you ask for another one with --partition. The *_int partitions are meant for interactive work (see interactive session).

Partition Nodes Per node Time limit Use
cpu (default) n[4-20] 64 cores (128 threads), 512 GB 18 days 8 h batch CPU jobs
cpu_int n[1-20] 64 cores (128 threads), 512 GB 20 days 10 h interactive CPU work
rocky10 gpu[1-2] 64 cores, 8× A100 80 GB, 2 TB 14 days 4 h batch and interactive GPU work (Rocky Linux 10)
small_int s[1-3] 8 cores, 64 GB 7 days 7 h light interactive work
preempt n[1-20] 64 cores (128 threads), 512 GB 5 days jobs that may be preempted

sinfo also lists the partitions debug and project001. They are reserved; don't submit jobs to them. The former GPU partitions gpu, gpu1, gpu2 and gpu_int have no nodes.

Limits change from time to time; sinfo on the login node shows the current values.

Pictures from the datacenter

Status LEDs of the compute nodes glowing in the dark server room

Status LEDs in the dark server room

Disk shelves with the storage servers below

Disk shelves and storage servers

Back of the rack with power, Ethernet and InfiniBand cabling

Power, Ethernet and InfiniBand cabling

Front view of the compute nodes

Compute nodes, front view

About the name

Line drawing of a server

In Greek mythology, Phoebe (ˈfiːbi), sister of Κοῖος (Koios), was one of the first generation of Titans, the sons and daughters of Uranus and Gaia.4 Koios is also the name of our previous cluster.