Throughput computing series: System Concurrency and Parallelism
By glennf on Jan 16, 2008
Let us consider the following five open source packages:
The following experiments will time the installation of these packages in both a serial, parallel, and concurrent fashion.
Parallel buildsAfter the "configure" phase is complete, these packages are all compiled using
gmake. This is where parallelism within each job can be used to speed the install process. By using the
"gmake -j"option, the level of parallelism can specified for each of the packages. This can dramatically improve the overall compile time as seen below.
- Jobs were ran in a serial fashion but with parallelism within the job itself.
- 79% reduction in compile time at 32 threads/job.
Concurrency and ParallelismThe build process for the various packages are not each able to be parallelized perfectly. In fact, the best overall gain of any of the packages is 6x out of 32. This is where concurrency comes into play. If we start all the compiles at the same time and use parallelism as well, this further reduces the overall build time.
- All 5 jobs were run concurrently with 1 and 32 (threads/job).
- 88% overall reduction in compile time from serial to parallel with concurrency.
- 42% reduction in compile time over parallel jobs ran serially.