MEMORY INTENSIVE
SERVICE (DURHAM)

System names: COSMA7, COSMA8

The workflow prototype underlying the design of the Memory Intensive Service (COSMA 7 and 8) is large-scale cosmological simulation of the formation and evolution of galaxies in the universe. Such simulations are extremely challenging due to the complex interplay between diverse physical processes operating on length-scales ranging from 10⁷ km to 10²² km, and timescales ranging from hours to tens of billions of years. To enable this calculational complexity, COSMA 8 has 1TB of memory on each node – by mid-2023 the total memory footprint will be an impressive 528 TB — ensuring that each node holds enough data to keep its 128 AMD cores busy. Even so, the nodes need to communicate efficiently, and a non-blocking InfiniBand network is essential. 

However, it’s not enough simply to carry out lots of calculations – we also need to save the current state of the simulation by copying every detail from the node memory into the storage system. While previous checkpoints took over 12 hours, COSMA 8’s large volume of flash-based storage makes it possible to perform a complete 528TB checkpoint in under 10 minutes! This time-saving by itself makes large-scale calculations roughly 10% more efficient.

Finally, we need somewhere to keep the output data – COSMA has 14PB of spinning disk storage, and an additional 26PB of tape storage for those data not actively in use.

The careful design matching the needs of simulation workflows, combined with expert support and a flexible allocation model, makes it possible to reserve up to 80% of the COSMA system for a single calculation making it the capability system of choice for cosmology. Due to its greater efficiency and flexibility, it has been used in preference to European systems with four times as many cores.