Improve profile reading and storing
At the moment, profiles are read by collective MPI IO operations and stored in each process. This can be improved by parallelizing the reading of all profile files and storing the data in a shared memory section on each computing node. The steps to take are the following:
- create a split MPI communicator for each node
- allocate one shared memory segment per profile on each node
- make sure that each profile file is read by only one process in the split MPI communicator
- store profile content to respective shared memory segment
- make sure that the FBScomponents library reads correctly from the shared memory segments
This method should no longer require collective MPI IO operations to read the profile files and reduce the required memory for profile storing.
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information