A. Installation on AFS

Introduction

Installing LSF in AFS involves running the lsfsetup program from the standard LSF distribution, and then installing the additional LSF AFS distribution.

LSF manages user permission for NFS, AFS, and DFS accesses, so users can use LSF no matter what type of filesystem their files are stored on. The choice of installation directory for LSF does not affect user access to load sharing.

Pre-Installation

Before installing LSF, you need to choose the primary LSF administrator, and decide where to store the LSF configuration and executable files. Complete instructions for LSF installation are in `Default Installation' on page 13 or `Custom Installation' on page 23. You may use either installation type before installing the LSF AFS distribution.

Choosing the LSF Administrator

The root account cannot be used as the primary LSF administrator if the LSF configuration files are to be stored in AFS, because in this case, the primary LSF administrator must be defined in AFS.

LSF Installation Directory

LSF Batch needs read/write access to the working directories under LSB_SHAREDIR, which contain the LSF Batch log files. Before running the lsfsetup program, you have to decide where to store your LSB_SHAREDIR. It can be stored locally, on an NFS-mounted filesystem, or on an AFS filesystem.

LSB_SHAREDIR on an AFS Filesystem

LSB_SHAREDIR should be defined in AFS only if the potential master hosts are trusted. The LSF system elects the master host in the order that the hosts appear in the lsf.cluster.cluster file. The first host is the default master. If the first host is down, the second host takes over as master, and so on. `Installing the LSF AFS Distribution' gives additional information on configuring LSF to use LSB_SHAREDIR on AFS.

LSB_SHAREDIR on a Local Filesystem

If you install the LSF Batch working directory in a local filesystem on one host, only that host can act as the LSF master. You must list this host first in the lsf.cluster.cluster file. If this host goes down, LSF Batch becomes unavailable. Batch processing resumes when the master host becomes available again. Interactive load sharing is still available while the host is down.

LSB_SHAREDIR on an NFS Filesystem

With this setup, the master daemon can be run on the hosts that have this directory mounted.

Additional Notes

You must not define LSF_RES_ACCTDIR and LSF_LOGDIR to be in AFS as the files in this directory are always written as the root user.

The other configuration directories are accessed read-only by the LSF daemons and thus can be defined in AFS if the ACL for these directories contain system:any_user rl.

Installing LSF

Follow the instructions in `Default Installation' on page 13 or `Custom Installation' on page 23 to install the standard LSF distribution. If some of your directories are defined in AFS, you must klog as the primary LSF administrator before running lsfsetup.

After Installing LSF

After running the lsfsetup program, add the following line to the lsf.conf file (stored in LSF_TOP/etc by default):

LSF_AFS_CELLNAME=cell_name

At this point, you can create @sys symbolic links so that LSF_BINDIR, LSF_LIBDIR, and LSF_SERVERDIR access the corresponding architecture directory.

Installing the LSF AFS Distribution

This LSF AFS distribution is named depending on the LSF version and host type, for example lsf3.2_solaris_afs.tar.Z.

1. Get the additional LSF AFS tar distribution file from tape or downloaded from Platform Computing's WWW or FTP sites. Uncompress the compressed tar distribution file.
2. Copy the following executables from the LSF AFS distribution directory to the LSF_SERVERDIR directory:

   res sbatchd

These executables are the same as the ones in the main distribution except that they are linked with AFS libraries.

gettok puttok

gettok gets the AFS token(s) from the kernel and prints out the tokens in ASCII format to standard output. puttok reads the AFS token(s) in the format generated by gettok, from standard input and sets the token(s) into the kernel.

esub eexec

These are shell scripts invoked by LSF to support token forwarding from the submission host to the execution host, and for supporting token renewal. Sites can modify these scripts to further customize token processing (for example, using site-specific encryption software).

3. If LSB_SHAREDIR is defined in AFS, the master batch daemon, mbatchd, must be configured so that it has the AFS token to write to LSB_SHAREDIR. The mbatchd.sc wrapper script in the LSF AFS distribution directory provides two methods for renewing the AFS token for the master batch daemon--plaintext password, and rtok daemon.

Plaintext Password

This method involves storing the primary LSF administrator's plaintext AFS password in a local file on the hosts that potentially can become the master.

• Rename mbatchd in LSF_SERVERDIR to mbatchd.real.
• Edit mbatchd.sc shell script in the AFS distribution directory:
  • Set LSF_ADMIN to the name of LSF primary administrator.
  • Set RTYPE=1
  • Set PWFILE to the absolute path of the file containing the plaintext password. This file should be accessible only by root.
• Copy mbatchd.sc to LSF_SERVERDIR as mbatchd.

rtok Daemon

This method involves setting up the token renewal daemon on the AFS server.

• Rename mbatchd in LSF_SERVERDIR to mbatchd.real.
• Set up token renewal kit as described in `AFS Token Renewal Kit' on page 103.
• Edit the mbatchd.sc shell script in the AFS distribution directory:
  • Set LSF_ADMIN to the name of LSF primary administrator.
  • Set RTYPE=2
  • Set RTOKD_HOST to the AFS server host name running rtokd.
  • Set RTOK_PORT to rtokd's port number.
  • If PGP is used, follow the procedure described in `AFS Token Encryption' (below) to set up PGP for root on the potential master hosts, and on the AFS server host.
• Copy mbatchd.sc to LSF_SERVERDIR as mbatchd.

4. If privileged port authentication is used and LSF_BINDIR is defined in AFS, you will need to change the ownership of the setuid executables in LSF_BINDIR to root. First, find all the binaries in LSF_BINDIR that are installed with the setuid bit on:

% ls -l | grep rws 

Then klog to a user ID with AFS administrator privileges, and run:

% chown root setuid_binaries 

Alternatively, you can simply chown all the LSF binaries under LSF_BINDIR to root if the directory contains only LSF binaries:

 
% chown root *
 

AFS Token Encryption

By default, the AFS esub and eexec scripts do not use encryption when transferring the AFS tokens between the submission and execution hosts. A site can modify these scripts to add site-specific encryption. The esub and eexec scripts in the LSF AFS distribution give an example of how to use PGP for encryption.

To configure LSF to use PGP:

1. Install the PGP package on all LSF hosts.
2. Define LSF_EEXEC_USER=root in /etc/lsf.sudoers on all LSF server hosts.
3. Create the root account's PGP directory on all the LSF server hosts. This root_pgp_dir (for example, /local/etc/.pgp) should be a local directory and only be accessible by root.
4. On an LSF server host, create root's public key and private keys by running (using Bourne shell syntax):

   % PGPPATH=root_pgp_dir
% export PGPPATH
% pgp -kg lsf_pgp_name

   where lsf_pgp_name can be any name you choose; for example,

   % PGPPATH=/local/etc/.pgp
% export PGPPATH
% pgp -kg lsf

5. Store the plain text pass phrase into $PGPPATH/.p. Make sure this file is owned by root and accessible only by root.

   Extract the public key to a share file so users from any LSF host can access the file on encryption:

   % pgp -kx lsf_pgp_name lsf_public_key_file

   For example:

   % pgp -kx lsf /usr/local/lsf/conf/lsfpublic.pgp

6. Securely replicate the PGPPATH contents onto all the other LSF server hosts.
7. Edit the eexec script, and
   • Re-define ROOT_PGPPATH to your root_pgp_dir directory if necessary.
   • Define PGP_EXEC_PATH to point to the directory containing the pgp executable.
   • Remove the line: PGPPATH=""
8. Edit the esub script, and re-define LSF_PGP_NAME to your lsf_pgp_name definition.
9. Each LSF user who wants to use PGP encryption must do the following setup:
   • Set up PGP by running pgp -kg. Make sure the PGPPATH environment variable is set.
   • Add lsf_pgp_name's public key to his/her public key ring by running:

     % pgp -ka user_name lsf_public_key_file

     For example, the user john would run,

     % pgp -ka john /usr/local/lsf/conf/lsfpublic.pgp

AFS Token Renewal Kit

The AFS token renewal kit in the LSF AFS distribution assumes that root is trusted on all the LSF server hosts. Since the token renewal kit uses the esub/eexec mechanism, a site can write its own token renewal system if a higher level of security is required.

The token renewal kit consists of a client program, rtok, and a server program, rtokd. rtok is used to request the rtokd daemon running on the AFS server host to renew the token for a user. rtok/rtokd also uses the eexec, esub, gettok, and puttok executables from the AFS distribution.

Setup on the AFS server host

If the AFS server host is of a host type that is not in your LSF cluster then you will need to get the LSF AFS distribution for that platform.

The assumption here is that the AFS server host is not part of the LSF cluster, and that none of the LSF binaries (in particular LSF_SERVERDIR) are accessible.

Define a directory to store the rtokd, gettok, puttok, eexec, and esub executables from the AFS distribution. If PGP is used, be sure to make the necessary modification to the esub script as described in `AFS Token Encryption' on page 101.

Set up a client_hostsfile file containing the list of machines that are authorized to renew tokens. This file is in the same format as /etc/hosts.

As root, start the rtokd server:

% absolute_path/rtokd -l /tmp -p portno client_hostsfile & 

where

-l /tmp indicates that the messages are logged to /tmp/rtokd.log.hostname. If -l is not given, the messages are printed to standard error. If -l syslog is given, the messages are logged to syslog.

-p portno indicates the port number to which the daemon will listen for client requests.

client_hostfile is the path name of the file containing the addresses/names of the hosts, which are authorized to renew tokens.

By default, rtokd reads /usr/afs/db/kaserver.DB0 for the user's DES keys. If kaserver.DB0 is in another location, the full path can be specified using the -f option.

Add rtokd to the system startup file (for example, rc.local).

Setup on LSF Server Hosts

1. Copy the rtokd, gettok, esub, puttok, eexec, and rtok executables into the LSF_SERVERDIR directory as defined in your lsf.conf file.
2. Set the owner of rtok to root, and its permissions to 0700.
3. Define LSF_EEXEC_USER=root in the /etc/lsf.sudoers file.
4. If PGP is used, follow the instructions in `AFS Token Encryption' on page 101 on how to modify the esub and eexec scripts to support encryption.
5. Edit the eexec script to set RTOK_PORT to the rtokd's port number, and set RTOKD_HOST to the name of the AFS server host running rtokd.