[Contents] [Index] [Top] [Bottom] [Prev] [Next]

B. Frame Arrays

Extending job array support, the frame array enables a subset of jobs in an array to be specified as a manageable unit. It can be associated with its own attributes such as resource requirements and job processing parameters. A frame array can be scheduled, modified, and controlled just like a single job.

In this appendix, we assume that you are already familiar with job arrays in LSF.

Overview

The frame array provides a structure that allows all the frames that make up a scene to be submitted using a single command. The frame array also provides the ability to control and manipulate individual frames, frame sub-sequences, and the whole scene (frame array). It groups frames (elements in a frame array) into user specified manageable units (chunks) that are submitted to the LSF Batch system as individual jobs. All jobs derived from a frame array share the same job ID and submission parameters.

Frames in a frame array are referenced using an array index, which supports negative indices. The dimension and structure of a frame array are defined when the array is submitted. Frames are scheduled to run independently of each other using the various policies that apply to the submitting user.

Frame arrays are especially useful for the Computer Graphics Industry (CGI) where the process of rendering a sequence of individual frames to build a scene is an everyday occurrence. The frames that make up a scene represent independent entities. They can be processed separately, therefore using the frame array is a natural approach to improving rendering throughput.

The default size of a frame array is 1000 jobs, the maximum size is 2046 jobs. The MAX_JOB_ARRAY_SIZE parameter specified in the lsb.params file sets the size of a frame array. The frame array and the individual frames are modified and controlled using the following new commands:

gbsubsubmits a frame array to the LSF Batch system

gbjobsdisplays the frame status for frame array jobs

gbstopsuspends a frame array or individual frames

gbresumeresumes a suspended frame array or individual frames

gbkillsends a signal to a frame array or individual frames

Distribution

The files that allow LSF to use frame arrays are available from Platform. Installation instructions are included.

Frame Array Concepts

frame
The basic element that makes up a frame array, hence the name. As the name implies, a frame is one picture of a series on a length of film. A number of frames make up a scene.

job
In the context of frame arrays, a job is the smallest object of control within a frame array. Jobs are defined by the chunk parameter.

chunk
The number of frames that are to be grouped together to form a job.

frame array
The structure used to define the dimension (number of frames) and structure (chunks) of the frames that make up the scene. The frame array is the structure that is submitted to the LSF Batch system.

Submitting Frame Arrays

A frame array is created at the time of submission. The gbsub command inherits all the options of the bsub command and extends the -J option to specify the elements of the array. Each element of the array corresponds to a single frame and is identified by an index which must be a unique integer. The index values in an array do not have to be consecutive, and a combination of individual index values and index ranges are used to define a frame array.

Syntax

% gbsub -J "frameArrayName[indexList, . . .]" command

Note

The frame array specification must be enclosed in double quotes.

The square brackets, [ ], around indexList must be entered exactly as shown.

Note: The frame array syntax breaks the convention of using square brackets to indicate optional items

frameArrayNameSpecifies a user defined string used to name the frame array. Any combination of the following characters make up a valid frameArrayName:

a-z | A-Z | 0-9 | . | - | _

indexListSpecifies the dimension, structure, and indices of the frame array in the following format:

indexList = start [- end [x step [: chunk]]]

startA unique integer specifying the start of a range of frame array indices. If a start value is specified without an end value, start specifies an individual job of the frame array.

endA unique integer specifying the end of a range of frame array indices. The start-end construct specifies a range of frames.

stepAn integer specifying the value to increment the index values for the respective range of frames. If omitted, the default value is 1.

chunkA single integer specifying the number of frames to group (chunk) into a job. If omitted, default value is 1.

A job is the smallest object of control within the LSF Batch system, when chunk size is larger than 1 the individual frames that make up the job are controlled as a single job. When the number of frames cannot be evenly divided by chunk size, the last job (chunk) will contain fewer frames.

The following formulas help to explain the relationship between frames, steps, and chunks.

Note

In these examples, if the number of frames or jobs turns out to be a fraction, it should always be rounded up to the next integer. For example, 9/3 would be 3, but 10/3 would be 4.

(end - start) + 1 Number of Frames = ----------------- step

Number of Frames Number of Jobs = ---------------- chunk

Examples

[1] specifies 1 frame with the index of 1 submitted as 1 job.
[1, 2, 3, 4, 5] specifies 5 frames with indices 1 - 5 submitted as 5 jobs .
[1-5] specifies 5 frames with indices 1 - 5 submitted as 5 jobs. Step and chunk values are default (i.e., 1). Index values are determined by starting at 1 and adding 1, not incrementing past the end value.
[1-10x2] specifies 5 frames with indices 1, 3, 5, 7, and 9 submitted as 5 jobs. Step value is 2, chunk value is default (i.e., 1). Index values are determined by starting at 1 and adding 2, not incrementing past the end value.
[1-10x2:3] specifies a frame array of 5 frames with indices 1, 3, 5, 7, and 9 submitted as 2 jobs. Step value is 2 and chunk value is 3. Index values are determined by starting at 1 and adding 2. Jobs are determined by putting frames in groups of 3. When chunk size does not evenly divide frame size the remainder frames (i.e., < chunk) make up the last job.
[-99-0, 1-100] specifies a frame array with indices -99 to 0 and 1 to 100 submitted as 199 jobs. Step and chunk values are default (i.e., 1).
[-99-0x10:5, 1-100x5:10] specifies the following frame array:
- Step value 10, chunk size 5, with indices -99, -89, -79, -69, -59, -49, -39, -29, -19, and -9 submitted as 2 jobs:
  - -99, -89, -79, -69, -59
  - -49, -39, -29, -19, -9
- Step value 5, chunk size 10, with indices 1, 6, 11, 16, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, and 96 submitted as 2 jobs:
  - 1, 6, 11, 16, 21, 26, 31, 36, 41, 46
  - 51, 56, 61, 66, 71, 76, 81, 86, 91, 96

Tracking Frame Arrays

The gbjobs command is used to track frame arrays. It inherits all the options of bjobs command with the addition of the -Jn "frameArrayName" and -Js "frameArrayName" options. The -Jn option displays frame jobs (chunks) arranged by the frame index and job state. The -Js option displays the status of all jobs in a frame array.

Examples

A frame array is submitted:

% gbsub -J "myFrame[-100--50x10:3,-40--20x10:3,-10,0,100-10x-10]" command

The command gbjobs -Jn "myFrame" produces output that looks like:

JOBGID USER FRAMES STAT 101 user1 myFrame[-100--50x10:3] RUN 101 user1 myFrame[-40--20x10:3] PEND 101 user1 mtFrame[-10] PEND 101 user1 myFrame[0] PEND 101 user1 myFrame[100-10x-10] PSUSP

The command gbjobs -Js "myFrame" produces output that looks like:

JOBGID FRAMES PEND RUN DONE EXIT SUSP OTH

101 myFrame[-100--20x10:3,-10,0,100-10x-10] 3 2 10

Controlling Frame Arrays

The gbstop, gbresume, and gbkill commands are used to control frame arrays. These commands inherit all the options of the bstop, bresume and bkill commands respectively, except for -R, -d, and -a, which are not supported. For frame arrays the -J "frameArrayName[indexList]" option operates at the job level, which is defined by the chunk size at submission time. These commands echo the frames on which they will be operating.

Syntax

gbstop [options] -J "frameArrayName[indexList, . . .]" command

gbresume [options] -J "frameArrayName[indexList, . . .]" command

gbkill [options] -J "frameArrayName[indexList, . . .]" command

-J "frameArrayName[indexList]"The syntax for frameArrayName[indexList] is the same used with the gbsub command. The asterisk, *, wildcard is supported for frameArrayName. Care must be used with the wildcard, as it will cause the command to operate all arrays whose names match the pattern. Chunk size is ignored for these commands. If only frameArrayName is specified (i.e., without an indexList) the command will operate on all jobs whose name matches frameArrayName.

If a mistake is made when issuing a control (i.e., indexing a frame that does not exist) the following error will be reported:

lsb_openjobinfo: No matching job found

Examples

A frame array is submitted using the following specification:

% gbsub -J "myFrameArray[1-10x2:3]" command

[1-10x2:3] specifies a frame array of 5 frames with indices 1, 3, 5, 7, and 9 submitted as 2 jobs. Step value is 2 and chunk value is 3. Index values are determined by starting at 1 and adding 2. Jobs are determined by putting frames in groups of 3. When chunk size does not evenly divide frame size the remainder frames (i.e., < chunk) make up the last job.

gbkill -J "myFrameArray[10]" command
specifies the frame at index 10 is to be killed. If myFrameArray[10] does not exist, the following error message is issued in response: lsb_openjobinfo: No matching job found.

gbstop -J "myFrameArray[1]" command
specifies the job at index 1 is to be stopped. Since that frame belongs to a job (chunk) whose size is larger than 1, all frames in the job (1 and 3) are stopped.

gbresume -J "myFrameArray[3]" command
specifies the frame at index 3 is to be started. Since that frame belongs to a job (chunk) whose size is larger than 1, all frames in the job (1 and 3) are started.

[Contents] [Index] [Top] [Bottom] [Prev] [Next]

doc@platform.com