Phoebe¶
Phoebe is the current CEICO HPC cluster and the main system
for new work. Log in through the front-end node phoebe.fzu.cz.
- 1408 CPU cores
- 16 NVIDIA A100 GPUs
- 2 TB RAM per GPU node
- 218 TB shared storage
- 100 Gb/s InfiniBand
System overview¶
Phoebe consists of 20 compute nodes, each with 64 CPU cores (2× AMD EPYC 75431), 512 GB of RAM and 1.7 TB of fast local NVMe2 disk. Two additional "fat" GPU nodes each carry 8 NVIDIA A1003 cards, 2 TB of RAM and 3.4 TB of local NVMe storage. Three small nodes with 8 cores and 64 GB of RAM each serve light interactive work. They are virtual machines, also useful for software that runs better on a small machine than on a large SMP node, such as some legacy codes.
Software, user and project data are stored on 218 TB of hybrid storage built from both solid state and rotational drives. All components are connected by a low-latency 100 Gbit InfiniBand fabric. All nodes run Rocky Linux 8. See storage for where to keep your data.
| pcs | hostnames | resource | ncores | fcpu (base) | fcpu (max) | RAM | local storage | notes |
|---|---|---|---|---|---|---|---|---|
| 20 | n[1-20] |
CPU compute nodes | 64 | 2.8 GHz | 3.7 GHz | 512 GB | 1.7 TB | - |
| 2 | gpu[1-2] |
GPU-accelerated fat nodes | 64 | 2.8 GHz | 3.7 GHz | 2 TB | 3.4 TB | 8× NVIDIA A100 |
| 3 | s[1-3] |
small nodes for light interactive work | 8 | - | - | 64 GB | - | small_int partition |
| 1 | phoebe.fzu.cz |
login front-end node (virtual machine) | 24 vCPU | 3.5 GHz | 4.0 GHz | 384 GB | - | AMD EPYC 73F3 host |
More detail: Phoebe infrastructure. If Phoebe helped your research, please acknowledge it in your publications.
Slurm partitions¶
Jobs go to the cpu partition unless you ask for another one with --partition. The
*_int partitions are meant for interactive work (see
interactive session).
| Partition | Nodes | Per node | Time limit | Use |
|---|---|---|---|---|
cpu (default) |
n[4-20] |
64 cores (128 threads), 512 GB | 18 days 8 h | batch CPU jobs |
cpu_int |
n[1-20] |
64 cores (128 threads), 512 GB | 20 days 10 h | interactive CPU work |
gpu |
gpu[1-2] |
64 cores, 8× A100 80 GB, 2 TB | 18 days 8 h | batch GPU jobs |
gpu1, gpu2 |
gpu1 or gpu2 |
as gpu |
14 days 4 h | pin a job to one GPU node |
gpu_int |
gpu[1-2] |
as gpu |
20 days 10 h | interactive GPU work |
small_int |
s[1-3] |
8 cores, 64 GB | 7 days 7 h | light interactive work |
preempt |
n[1-20] |
64 cores (128 threads), 512 GB | 5 days | jobs that may be preempted |
sinfo also lists the partitions debug and project001. They are reserved; don't submit jobs
to them.
Limits change from time to time; sinfo on the login node shows the current values.
Pictures from the datacenter¶
Status LEDs in the dark server room
Disk shelves and storage servers
Power, Ethernet and InfiniBand cabling
Compute nodes, front view
About the name¶
In Greek mythology, Phoebe (ˈfiːbi), sister of Κοῖος (Koios), was one of the first generation of Titans, the sons and daughters of Uranus and Gaia.4 Koios is also the name of our previous cluster.