This section describes the system resources monitored by LSF JobScheduler, and the use of resource specifications. Specific topics covered in this section are:
A computer network may be thought of as a collection of resources used to execute programs. Different applications often require different resources. For example, a number crunching program may take a lot of CPU power, but a large spreadsheet may need a lot of memory to run well. Some applications may run only on machines of a specific type, and not on others.
In LSF JobScheduler, resources are handled by naming them and tracking information relevant to them. LSF JobScheduler does its scheduling according to applications' resource requirements and the resources available on individual hosts. Resource names are case sensitive, and can be up to 29 characters in length (excluding some characters reserved as operators in resource requirement strings).
LSF JobScheduler classifies resources in different ways.
general resources
These are resources that are available on all hosts, e.g. all the load indices, number of processors on a host, total swap space, host status.
special resources
These are resources that are only associated with some hosts, e.g. FileServer, solaris, compServer.
dynamic resources
These are resources that change their values dynamically, e.g. all the load indices, host status.
static resources
These are resources that do not change their values dynamically, e.g. all resources except load indices and host status are static resources.
numerical resources
These are resources that take numerical values, e.g. all the load indices, number of processors on a host, host CPU factor.
string resources
These are resources that take string values, e.g. host type, host model, host status.
boolean resources
These are resources that denote the availability of specific features, e.g. hspice, FileServer, SYSV, aix.
configured resources
These are resources defined by user sites, e.g. all resources defined in the LSF configuration files.
built-in resources
These are resources that are always defined by LSF, e.g. load indices, number of CPUs, total swap space.
host-based resources
These are resources that are not shared among hosts, but are tied to individual hosts, e.g. CPU, memory, swap space. An application must run on a particular host to access such resources.
shared resources
These are resources that are not associated with individual hosts in the same way, but are "owned" by the entire cluster, or a subset of hosts within the cluster, e.g. floating licenses, shared file systems. An application can access such a resource from any host which is configured to share it, but doing so affects its value as seen by other hosts.
You can list the resources available in your cluster using the lsinfo command.
Load indices measure the availability of dynamic, non-shared resources on hosts in the LSF cluster. Load indices built into the LIM are updated at fixed time intervals. External load indices are updated when new values are received from the external load collection program, ELIM, configured by the LSF administrator. Load indices are numeric in value.
Table 1 summarizes the load indices collected by the LIM.
Load indices can be viewed using the lsload command:
% lsload
HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem
hosta ok 0.0 0.0 0.1 1% 0.0 1 224 43M 67M 3M
hostb ok 0.0 0.0 0.0 3% 0.0 3 0 38M 40M 7M
hostc busy *6.2 6.9 9.5 85% 1.1 30 0 5M 400M 385M
hostd busy 0.1 0.1 0.3 7% *17 6 0 9M 23M 28M
hoste unavail
The r15s, r1m and r15m load indices are the 15-second, 1-minute and 15-minute average CPU run queue lengths. This is the average number of processes ready to use the CPU during the given interval.
On multiprocessor systems more than one process can execute at a time. LSF scales the run queue value on multiprocessor systems to make the CPU load of uniprocessors and multiprocessors comparable. The scaled value is called the effective run queue length. The -E option of lsload shows the effective run queue length.
LSF also adjusts the CPU run queue based on the relative speeds of the processors (the CPU factor). The normalized run queue length is adjusted for both number of processors and CPU speed. The host with the lowest normalized run queue length will run a CPU intensive job the fastest. The -N option shows the normalized CPU run queue lengths.
Static resources represent host information that does not change over time, such as the maximum RAM available to user processes, or the number of processors in a machine. Most static resources are determined by the LIM at startup time. Table 2 lists the static resources reported by LSF.
The type and model static resources are strings specifying the host type and model.
The CPU factor is the speed of the host's CPU relative to other hosts in the cluster. If one processor is twice the speed of another, its CPU factor should be twice as large. The CPU factors are defined by the LSF administrator. For multiprocessor hosts the CPU factor is the speed of a single processor; LSF automatically scales the host CPU load to account for additional processors.
The server static resource is boolean; its value is 1 if the host is configured to run jobs and 0 if the host is a client.
Static resources can be used to select appropriate hosts for particular jobs based on binary architecture, relative CPU speed, and system configuration.
To view the static resources are available in your cluster, run the lshosts command.
A shared resource is a resource that is not tied to a specific host, but is associated with the entire cluster, or a specific subset of hosts within the cluster. Examples of shared resources include:
An application may use a shared resource by running on any host from which that resource is accessible. For example, in a cluster in which each host has a local disk but can also access a disk on a file server, the disk on the file server is a shared resource, and the local disk is a host-based resource. There will be one value for the entire cluster which measures the utilization of the shared resource, but each host-based resource is measured separately.
A shared resource may be configured to be dynamic or static. In the above example, the total space on the shared disk may be static while the amount of space currently free is dynamic. A site may also configure the shared resource to report numeric, string or boolean values.
All shared resources must be configured by the LSF administrator. Run the lshosts -s command to view the static shared resources currently configured in your cluster. Run the lsload -s command to view the dynamic shared resources configured in your cluster.
A resource requirement string describes the resources a job needs. LSF JobScheduler uses resource requirements to select hosts for remote execution and job execution.
A resource requirement string is divided into three sections:
The syntax of a resource requirement expression is:
select[selectstring] order[orderstring] rusage[usagestring]
The square brackets are an essential part of the resource requirement expression.
The section names are select, order, and rusage. The syntax for each of selectstring, orderstring and usagestring is defined below.
If no section name is given, then the entire string is treated as a selection string. The select keyword may be omitted if the selection string is the first string in the resource requirement.
The selection string specifies the characteristics a host must have to match the resource requirement. It is a logical expression built from a set of resource names. The lsinfo command lists all the resource names and their descriptions. The resource names swap, idle, login, and cpu are accepted as aliases for swp, it, ls, and r1m respectively.
The selection string can combine resource names with logical and arithmetic operators. Non-zero arithmetic values are treated as logical TRUE, and zero as logical FALSE. Boolean resources (for example, server to denote LSF server hosts) have a value of one if they are defined for a host, and zero otherwise.
Table 3 below shows the operators that can be used in selection strings. The operators are listed in order of decreasing precedence.
The selection string is evaluated for each host. If the result is non-zero, then that host is selected. For example:
select[(swp > 50 && type == MIPS) || (swp > 35 && type == ALPHA)] select[((2*r15s + 3*r1m + r15m) / 6 < 1.0) && !fs && (cpuf > 4.0)]
For the string resources type and model, the special value any selects any value and local selects the same value as that of the local host. For example, type==local selects hosts of the same type as the host submitting the job. If a job can run on any type of host, include type==any in the resource requirements. If no type is specified, the default is type==local unless a model or boolean resource is specified, in which case it is type==any.
The order string allows the selected hosts to be sorted according to the values of resources. The syntax of the order string is:
[-]res[:[-]res]...
Each res must be a dynamic load index; that is, one of the indices r15s, r1m, r15m, ut, pg, io, ls, it, tmp, swp, mem, or an external load index defined by the LSF administrator. For example, swp:r1m:tmp:r15s is a valid order string.
The values of r15s, r1m, and r15m used for sorting are the normalized load indices returned by lsload -N (see `Load Indices' on page 47).
The order string is used for host sorting and selection. The ordering begins with the rightmost index in the order string and proceeds from right to left. The hosts are sorted into order based on each load index, and if more hosts are available than were requested, the JobScheduler drops the least desirable hosts according to that index. The remaining hosts are then sorted by the next index.
After the hosts are sorted by the leftmost index in the order string, the final phase of sorting orders the hosts according to their status, with hosts that are currently not available for load sharing (that is, not in the ok state) listed at the end.
Because the hosts are resorted for each load index, only the host status and the leftmost index in the order string actually affect the order in which hosts are listed. The other indices are only used to drop undesirable hosts from the list.
When sorting is done on each index, the direction in which the hosts are sorted (increasing or decreasing values) is determined by the default order returned by lsinfo for that index. This direction is chosen such that after sorting, the hosts are ordered from best to worst on that index.
When an index name is preceded by a minus sign `-', the sorting order is reversed so that hosts are ordered from worst to best on that index.
The default sorting order is r1m:pg.
This string defines the expected resource usage of the task. It is used to specify job resource reservations.
The syntax of the resource usage string is:
res=value[:res=value]...[:res=value][:duration=value][:decay=value]
The res parameter can be any load index. The value parameter is the initial reserved amount. If res or value is not given, the default is not to reserve that resource.
The duration parameter is the time period within which the specified resources should be reserved. It is specified in minutes by default. If the value is followed by the letter 'h', it is specified in hours. For example, 'duration=30' and 'duration=2h' specify a duration of 30 minutes and two hours respectively. If duration is not specified, the default is to reserve the total amount for the lifetime of the job.
The decay parameter indicates how the reserved amount should decrease over the duration. A value of 1, 'decay=1', indicates that system should linearly decrease the amount reserved over the duration. The default decay value is 0, which causes the total amount to be reserved for the entire duration. Values other than 0 or 1 are unsupported. If duration is not specified, decay is ignored.
rusage[mem=50:duration=100:decay=1
The above example indicates that 50MB memory should be reserved for the job. As the job runs, the amount reserved will decrease at approximately 0.5 megabytes per minute until the 100 minutes is up.
LSF resource requirement syntax is straightforward to use, even though it may look complicated. The following examples illustrate how to specify a resource requirement when submitting a job to LSF JobScheduler. These examples show the command line interface to better illustrate the concepts involved (although it is easier to use the GUI for job creation).
This example creates a job that requires more than 50MB of swap space and requires an aix type of host to run.
% bsub -R "swp > 50 && type == aix" myJob
The section name "select" can be omitted because the select string is the first string in the resource requirement. We did not specify "order" or "rusage" in this example, which tells LSF JobScheduler to order hosts by default ordering ("r1m:pg"), and not to reserve resources.
This example specifies a shared resource as a resource requirement.
% bsub -R "avail_scratch > 200 && swap > 50" myJob
Assume that avail_scratch is a shared resource for scratch space in a shared file system and all hosts in the cluster have access to the shared scratch space. The job will only be scheduled if the value of the avail_scratch resource is more than 200 MB and will go to a host with at least 50MB of available swap space.
It is possible for a system to be configured so that only some hosts within the LSF cluster have access to the scratch space. In order to exclude hosts which cannot access a shared resource, the "defined(resource_name)" function must be specified in the resource requirement string.
% bsub -R "defined(avail_scratch) && avail_scratch > 100 && swap > 100" myJob
This command will exclude any hosts which cannot access the scratch resource. The LSF administrator configures which hosts do and do not have access to a particular shared resource.
Some applications require resources other than the default set. LSF can store resource requirements for specific applications so that users do not have to specify resource requirements each time a job is submitted.
The resource requirements of applications are stored in the remote task list file. A task is a command or a user-created executable program; the terms application or job are also used to refer to tasks. The remote task list file contains the resource requirements of each task.
There are three sets of task list files: the system-wide
default file lsf.task, the cluster default file lsf.task.cluster,
and the user file $HOME/.lsftask. The system and cluster default
files are set by LSF administrator and they apply to all users. The user file
specifies the tasks to be added to or removed from the system lists for your
jobs. Resource requirements specified in your user file override those in the
system lists.
When you submit a job to LSF JobScheduler, the system automatically picks up the job's default resource requirement string from the remote task list files, unless you explicitly override the default by specifying the resource requirement string on the command line or job submission GUI.
The lsrtasks command inspects and modifies the remote task list. Invoking lsrtasks commands with no arguments displays the resource requirements of tasks in the remote list, separated from the task name by `/'.
% lsrtasks
cc/cpu compress/-:cpu:mem compressdir/cpu:mem
f77/cpu verilog/cpu && cadence synopsys/swp >150 && cpu
dsim/type == any hspice/cpu && cadence nas/swp > 200 && cpu
cfd3d/type == SG1 && cpu epi/hpux11 sparc regression/cpu
cc/type == local compress/cpu
You can specify resource requirements when tasks are added to the user's remote task list. If the task to be added is already in the list, its resource requirements are replaced.
% lsrtasks + myjob/swap>=100 && cpu
This command adds myjob, along with its resource requirement, to the remote tasks list.