Performance

The GPFS shows optimal performance with sequential read and write patterns (typically large files). Avoid random and high frequency access patterns (typically small files). Avoid the creation of large numbers of small files (>1000) in a single directory. Being a network filesystem there is always a small latency for every I/O operation involved during which your calculation remains idle. Since the GPFS is a shared ressource, the performance for all other users may vary and strongly depend on the filesystem load created by a single user either globally or within a single node.

If you cannot avoid highly frequent I/O operations, it is almost always much more efficient to use the local node filesystem or the RAM disk (see below).

In order to help you decide for the optimal storage for your use case we ran a couple of benchmarks.



Benchmark commands using fio 
fio --rw=read --name=/hpc/gpfs2/u/$USER/test --size=50G
fio --rw=read --name=/hpc/gpfs2/u/$USER/test --size=50G --bs=4M
fio --rw=read --name=/home/ltmp/test --size=50G
fio --rw=read --name=/home/ltmp/test --size=50G --bs=4M
fio --rw=read --name=/dev/shm/test --size=50G
fio --rw=read --name=/dev/shm/test --size=50G --bs=4M

fio --rw=write --name=/hpc/gpfs2/u/$USER/test --size=50G
fio --rw=write --name=/hpc/gpfs2/u/$USER/test --size=50G --bs=4M
fio --rw=write --name=/home/ltmp/test --size=50G
fio --rw=write --name=/home/ltmp/test --size=50G --bs=4M
fio --rw=write --name=/dev/shm/test --size=50G
fio --rw=write --name=/dev/shm/test --size=50G --bs=4M

fio --rw=randread --name=/hpc/gpfs2/u/$USER/test --size=1G
fio --rw=randread --name=/hpc/gpfs2/u/$USER/test --size=1G --bs=4M
fio --rw=randread --name=/home/ltmp/test --size=5G
fio --rw=randread --name=/home/ltmp/test --size=5G --bs=4M
fio --rw=randread --name=/dev/shm/test --size=50G
fio --rw=randread --name=/dev/shm/test --size=50G --bs=4M

fio --rw=randwrite --name=/hpc/gpfs2/u/$USER/test --size=1G
fio --rw=randwrite --name=/hpc/gpfs2/u/$USER/test --size=1G --bs=4M
fio --rw=randwrite --name=/home/ltmp/test --size=5G
fio --rw=randwrite --name=/home/ltmp/test --size=5G --bs=4M
fio --rw=randwrite --name=/dev/shm/test --size=50G
fio --rw=randwrite --name=/dev/shm/test --size=50G --bs=4M

Programming pitfalls

Unwanted "benchmarking" of the network filesystem (GPFS) should be avoided! Especially self-written scripts often bypass buffering or do unnecessary I/O like opening/closing files or flushing buffers in hot loops. The following show bad examples and their solutions in Python and C++:

(Python) BAD: Opening a file in a hot loop, Runtime: 9m50s 
seed = 42

for i in range(1,20_000_000):
    sample = ...
    with open(f"data_{seed}.dat", "a") as file:
        file.write(f"{sample}\n")
(Python) GOOD: Opening a file outside of the hot loop, Runtime: 7s 
seed = 42

with open(f"data_{seed}.dat", "a") as file: 
	for i in range(1,20_000_000):
    	sample = ...
        file.write(f"{sample}\n")
(C++) BAD: line flush after every write, Runtime: ~100s 
#include <fstream>
#include <string>

int main() {
    int seed = 42;

    std::ofstream file("data_" + std::to_string(seed) + ".dat", std::ios::app);

    for (long long i = 1; i < 20000000LL; ++i) {
        double sample = /* ... */;
        file << sample << std::endl; // std::endl writes an newline AND flushes to disk immediately
    }

    return 0; // file is automatically flushed and closed
}
(C++) GOOD: line flush after steam buffer is full (automatically managed), Runtime: ~4s 
#include <fstream>
#include <string>

int main() {
    int seed = 42;

    std::ofstream file("data_" + std::to_string(seed) + ".dat", std::ios::app);

    for (long long i = 1; i < 20000000LL; ++i) {
        double sample = /* ... */;
        file << sample << '\n'; // just write the newline but don't flush to disk
    }

    return 0; // file is automatically flushed and closed
}




part of icons are from social media marketing icons Social media marketing icons created by Freepik - Flaticon