High Performance Computing

Jobs that require more than one processor need to be compiled in a special way. The way they are submitted to the queuing system is also a bit more complicated, involving use of a 'parallel environment', specified via the -pe option to qsub. Streamline have provided wrapper scripts (ompisub, mvapich2sub) to make the submission of MPI parallel jobs easier in most cases. Those which use multiple processors within one compute node may use OpenMP or MPI. Those using more than one node must use MPI.

To run an MPI code, use of the supplied wrapper scripts is recommended. e.g. for OpenMPI:

> ompisub 2x8 ./mpitest
Generating SGE job file for a 16 cpu mpich job with SMP=8 from executable
/work/work1/egzcb/mpitest.
QSUB mpirun -np 16 /work/work1/egzcb/mpitest
Done.
Submitting SGE job as follows:
qsub -pe openmpi 2 /work/work1/egzcb/./mpitest.sh
Sending standard output to file: /work/work1/egzcb/mpitest.sh.o28416
Sending standard error to file: /work/work1/egzcb/mpitest.sh.e28416
Use the qstat command to query the job queue. e.g
qstat
job-ID prior name user state submit/start at queue
slots ja-task-ID
---------------------------------------------------------------------------
--------------------------------------
28409 0.60500 mpitest.sh egzcb qw 04/22/2009 15:45:53
parallel-test.q@compE198.grid. 2
Job submission complete.

This job will run a total of 16 processes, on 2 compute nodes, as is suitable for towerE with 8 processing cores per node. On towers A-D, the number of processing cores is only 2, so to get 16 processes, the command would be

> ompisub 8x2 ./mpitest

Note that the ompisub script invokes the qsub command with appropriate options, including specifying the appropriate parallel environment (pe). The script also creates the mpitest.sh script file for submission. Creation of these scripts without using the wrapper script, and direct submission via qsub is an alternative approach, but is only recommended for experienced users.

If using mvapich, mvapich2sub can be used as the wrapper script in the same way as ompisub in the example above.

See the Programming section for more information on compiling and running parallel programs.

To run an OpenMP code, a job script needs to be created, and submitted it to the smp parallel environment. Using the hello example above here is a script called run.sh

#! /bin/bash
#$ -V -cwd -pe smp 1
export OMP_NUM_THREADS=4 
echo Running on ; hostname
./hello

To submit the script:

> qsub run.sh
Your job 208 ("run.sh") has been submitted

The output file run.sh.o208 after the job has completed:

Running on
comp07
hello world from thread = 0
number of threads = 4
hello world from thread = 1
hello world from thread = 3
hello world from thread = 2

There are two points to note. Firstly the second line of the run.sh script contains -pe smp 1. This ensures that the script is submitted to the smp parallel environment. Secondly the environment variable OMP NUM THREADS should be set to the number of required threads. It is recommended the value does not exceed the number of processing cores per node. If the OMP NUM THREADS variable is not set then the default value depends on which compiler was used. This is 1 for PGI compilers, but equals the number of cores on the node for Intel and Gnu compilers.