12. Customizing Batch Jobs for LSF

This chapter describes how to customize your batch jobs to take advantage of LSF and LSF Batch features.

Environment Variables

When LSF Batch runs a batch job it sets several environment variables. Batch jobs can use the values of these variables to control how they execute. The environment variables set by LSF Batch are:

LSB_CHKPNT_DIR
This variable is set each time a checkpointed job is submitted. The value of the variable is chkpntdir/jobId, a subdirectory of the checkpoint directory that is specified when the job is submitted. The subdirectory is identified by the job ID of the submitted job.

LSB_JOBID
The LSF Batch job ID number.

LSB_JOBFILENAME
The full path name of the batch job file. This is a /bin/sh script on UNIX systems or a .BAT command script on Windows NT systems that invokes the batch job.

LSB_HOSTS
The list of hosts selected by LSF Batch to run the batch job. If the job is run on a single processor, the value of LSB_HOSTS is the name of the execution host. For parallel jobs, the names of all execution hosts are listed separated by spaces. The batch job file is run on the first host in the list.

LSB_QUEUE
The name of the batch queue from which the job was dispatched.

LSB_JOBNAME
The name of the batch job as specified by the -J job_name argument to bsub. If this argument was not given, the job name is the actual batch command as specified on the bsub command line.

LSB_RESTART
If this batch job was submitted with the -r option to bsub, has run previously, and has been restarted because of a host failure, LSB_RESTART is set to the value Y. If this is not a restarted job LSB_RESTART is not set.

LSB_EXIT_PRE_ABORT
The value of this parameter can be used by a queue or job-level pre-execution command so that the command can exit with this value, if it wants the job be aborted instead of being requeued or executed.

LSB_EXIT_REQUEUE
This variable is a list of exit values defined in the queue's REQUEUE_EXIT_VALUE parameter. If this variable is defined, a job will be requeued if it exits with one of these values. This variable is not set if the queue does not have REQUEUE_EXIT_VALUE defined.

LSB_JOB_STARTER
This variable is defined if a job starter command is defined for the queue. See `Queue-Level Job Starters' on page 129 of the LSF Batch Administrator's Guide for more information.

LSB_INTERACTIVE
This variable is set to `Y' if the job is an interactive job. An interactive job is submitted using the -I option to bsub. This variable is not defined if the job is not interactive. See `Interactive Batch Job Support' on page 145.

LS_JOBPID
The process ID of the job. This is always a shell script process that runs the actual job.

LS_SUBCWD
This is the directory on the submission host where the job was submitted. By default LSF Batch assumes a uniform user name and user ID space exists among all the hosts in the cluster, that is, a job submitted by a given user will run under the same user's account on the execution host. For situations where non-uniform user id/user name space exists, account mapping must be used to determine the account used to run a job. See `User Controlled Account Mapping' on page 86.

LSB_JOBINDEX
This variable is set only if the job is an element of a job array. The value of the variable is the index of the job into the job array.

Parallel Jobs

Each parallel programming package has different requirements for specifying and communicating with all the hosts used by a parallel job. LSF is not tailored to work with a specific parallel programming package. Instead, LSF provides a generic interface so that any parallel package can be supported by writing shell scripts or wrapper programs. Example shell scripts are provided for running PVM, P4, MPI, and POE programs as parallel batch jobs.

Getting the Host List

The hosts allocated for the parallel job are passed to the batch job in the LSB_HOSTS environment variable. Some applications can take this list of hosts directly as a command line parameter. For other applications you may need to process the host list. The following example shows a /bin/sh script that processes all the hosts in the host list, including identifying the host where the job script is executing.

#!/bin/sh
# Process the list of host names in LSB_HOSTS

for host in $LSB_HOSTS ; do
handle_host $host
done

LSF comes with a few scripts for running parallel jobs under LSF Batch, such as pvmjob, poejob, mpijob, p4job, etc. These scripts are installed in the LSF_BINDIR as defined in lsf.conf file. You can modify these scripts to support more parallel packages.

Starting Parallel Tasks With lstools

For simple parallel jobs you can use the lstools commands to start parts of the job on other hosts. Because the lstools commands handle signals transparently, LSF Batch can suspend and resume all components of your job without additional programming.

The simplest parallel job runs an identical copy of the executable on every host. The lsgrun command takes a list of host names and runs the specified task on each host. The lsgrun -p option specifies that the task should be run in parallel on each host. The example below submits a job that uses lsgrun to run myjob on all the selected batch hosts in parallel:

% bsub -n 10 'lsgrun -p -m "$LSB_HOSTS" myjob'
Job <3856> is submitted to default queue <normal>.

For more complicated jobs, you can write a shell script that runs lsrun in the background to start each component.

Using LSF Make to Run Parallel Batch Jobs

For parallel jobs that have a variety of different components to run, you can use LSF Make. Create a makefile that lists all the components of your batch job and then submit the LSF Make command to LSF Batch. The following example shows a bsub command and Makefile for a simple parallel job.

% bsub -n 4 lsmake -f Parjob.makefile
Job <3858> is submitted to default queue <normal>.

% cat Parjob.makefile
# Makefile to run example parallel job using lsbatch and LSF Make

all: part1 part2 part3 part4

part1 part2 part3: myjob data.$@

part4: myjob2 data.part1 data.part2 data.part3

The batch job has four components. The first three components run the myjob command on the data.part1, data.part2 and data.part3 files. The fourth component runs the myjob2 command on all three data files. There are no dependencies between the components, so LSF Make runs them in parallel.

Submitting PVM Jobs to LSF Batch

PVM is a parallel programming system distributed by Oak Ridge National Laboratories. PVM programs are controlled by a file, the PVM hosts file, that contains host names and other information. The pvmjob shell script supplied with LSF can be used to run PVM programs as parallel LSF Batch jobs. The pvmjob script reads the LSF Batch environment variables, sets up the PVM hosts file and then runs the PVM job. If your PVM job needs special options in the hosts file, you can modify the pvmjob script.

For example, if the command line to run your PVM job is:

% myjob data1 -o out1

the following command submits this job to LSF Batch to run on 10 hosts:

% bsub -n 10 pvmjob myjob data1 -o out1

Other parallel programming packages can be supported in the same way. The p4job shell script runs jobs that use the P4 parallel programming library. Other packages can be handled by creating similar scripts.

Submitting MPI Jobs to LSF Batch

The Message Passing Interface (MPI) is a portable library that supports parallel programming. LSF supports MPICH, a joint implementation of MPI by Argonne National Laboratory and Mississippi State University. This version supports both TCP/IP and IBM's Message Passing Library (MPL) communication protocols.

LSF provides an mpijob shell script that you can use to submit MPI jobs to LSF Batch. The mpijob script writes the hosts allocated to the job by the LSF Batch system to a file and supplies the file as an option to MPICH's mpirun command. The syntax of the mpijob command is:

mpijob option mpirun program arguments

Here, option is one of the following:

-tcp Write the LSF Batch hosts to a PROCGROUP file, supply the -p4pg procgroup_file option to the mpirun command, and use the TCP/IP protocol. This is the default.

-mpl Write the LSF Batch hosts to a MACHINE file, supply the -machinefile machine_file option to the mpirun command, and use the MPL on an SP-2 system.

The following examples show how to use mpijob to submit MPI jobs to LSF Batch.

To submit a job requesting four hosts and using the default TCP/IP protocol, use:

% bsub -n 4 mpijob mpirun myjob

Note

Before you can submit a job to a particular pool of IBM SP-2 nodes, an LSF administrator must install the SP-2 ELIM. The SP-2 ELIM provides the pool number and lock status of each node.

To submit the same job to run on four nodes in pool 1 on an IBM SP-2 system using MPL, use:

% bsub -n 4 -R "pool == 1" mpijob -mpl mpirun myjob

To submit the same job to run on four nodes in pool 1 that are not locked (dedicated to using the High Performance Switch) on an SP-2 system using MPL, use:

% bsub -n 4 -q mpiq -R "pool == 1 && lock == 0" mpijob -mpl mpirun myjob

Note

Before you can submit a job using the IBM SP-2 High Performance Switch in dedicated mode, an LSF administrator must set up a queue for automatic requeue on job failure. The job queue will automatically requeue a job that failed because an SP-2 node was locked after LSF Batch selected the node but before the job was dispatched.

Submitting POE Jobs to LSF Batch

The Parallel Operating Environment (POE) is an execution environment provided by IBM on SP-2 systems to hide the differences between serial and parallel execution.

LSF provides a poejob shell script that you can use to submit POE jobs to LSF Batch. The poejob script translates the hosts allocated to the job by the LSF Batch system into an appropriate POE host list and sets up environment variables necessary to run the job.

The poejob script does not set the MP_EUILIB and MP_EUIDEVICE environment variables, so you have to do this.

% setenv MP_EUILIB us

By default, MP_EUIDEVICE is css0. Or:

% setenv MP_EUILIB ip
% setenv MP_EUIDEVICE en0

The following are examples of how to submit POE jobs.

To submit a job requesting four SP-2 nodes configured for the poeq queue, use:

% bsub -n 4 -q poeq poejob myjob

By using LSF resource requirements, you can select appropriate nodes for your job.

To submit the same job requesting four SP-2 nodes from pool 2 configured for the poeq queue, use:

% bsub -n 4 -R "pool == 2" -q poeq poejob myjob

To submit the same job requesting four SP-2 nodes from pool 2 with at least 20 megabytes of swap space, use:

% bsub -n 4 -R "(pool == 2) && (swap > 20)" -q poeq poejob myjob

To submit the same job requesting four SP-2 nodes from pool 2 that are not locked (dedicated to using the High Performance Switch), use:

% bsub -n 4 -R "(pool == 2) && (lock == 0)" -q poeq poejob myjob

Using a Job Starter for Parallel Jobs

The above examples use a script to run parallel jobs under LSF Batch. Alternatively, your LSF administrator could configure the script into your queue as a job starter. With a job starter configured at the queue, you can submit the above parallel jobs without having to type the script name. See `Queue-Level Job Starters' on page 129 of the LSF Batch Administrator's Guide for more information about job starters.

To see if your queue already has a job starter defined, run bqueues -l command.