Hp XC System 3.x Software Manuel d'utilisateur

HP XC System SoftwareUser's GuideVersion 3.2.1HP Part Number: A-XCUSR-321Published: October 2007

10

sometime in the future, depending on resource availability andbatch system scheduling policies.Batch job submissions typically provide instructions on

LSF-HPC allocates the appropriate whole node for exclusiveuse by the serial job in the same manner as it does for paralleljobs, hence the name “pseudo

The HP XC system has several features that make it optimal for running parallel applications,particularly (but not exclusively) MPI applications. You

Figure 10-1 How LSF-HPC and SLURM Launch and Manage a JobN 1 6N16User124666677775job_starter.sh$ srun -nl myscriptLogin node$ bsub-n4 -ext ”SLURM[node

4. LSF-HPC prepares the user environment for the job on the LSF execution host node anddispatches the job with the job_starter.sh script. This user en

10.10.1 Examining System Core StatusThe bhosts command displays LSF-HPC resource usage information. This command is usefulto examine the status of the

• The maxmem column displays minimum maxmem over all available computer nodes in thelsf partition.• The maxtmp column (not shown) displays minimum max

10.11 Getting Information About JobsThere are several ways you can get information about a specific job after it has been submittedto LSF-HPC integrat

Example 10-3 Job Allocation Information for a Running Job$ bjobs -l 24Job <24>, User <lsfadmin>, Project <default>,

Example 10-5 Using the bjobs Command (Short Output)$ bjobs 24JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME24 msmith

List of Tables1-1 Determining the Node Platform...241-

Table 10-2 Output Provided by the bhist CommandDescriptionFieldThe job ID that LSF-HPC assigned to the job.JOBIDThe user who submitted the job.USERThe

$ bjobs -l 99 | grep slurmdate and time stamp: slurm_id=123;ncpus=8;slurm_alloc=n[13-16];The SLURM JOBID is 123 for the LSF JOBID 99.You can also find

$ bjobs -l 124 | grep slurmdate and time stamp: slurm_id=150;ncpus=8;slurm_alloc=n[1-4];LSF allocated nodes n[1-4] for this job. The SLURM JOBID is 15

Example 10-10 Launching an Interactive MPI Job on All Cores in the AllocationThis example assumes 2 cores per node.$ mpirun -srun --jobid=150 -n8 hell

10.14 LSF-HPC Equivalents of SLURM srun OptionsTable 10-3 describes the srun options and lists their LSF-HPC equivalents.Table 10-3 LSF-HPC Equivalent

Table 10-3 LSF-HPC Equivalents of SLURM srun Options (continued)LSF-HPC EquivalentDescriptionsrun OptionYou can use when launching parallel tasks.Spec

Table 10-3 LSF-HPC Equivalents of SLURM srun Options (continued)LSF-HPC EquivalentDescriptionsrun OptionUse as an argument to srun when launchingparal

11 Advanced TopicsThis chapter covers topics intended for the advanced user. This chapter addresses the followingtopics:• “Enabling Remote Execution w

$ echo $DISPLAY:0Next, get the name of the local machine serving your display monitor:$ hostnamemymachineThen, use the host name of your local machine

$ sinfoPARTITION AVAIL TIMELIMIT NODES STATE NODELISTlsf up infinite 2 idle n[46,48]According to the information returned about this

12

One way is to prefix the actual compilation line in the rule with an srun command. So, insteadof executing cc foo.c -o foo.o it would execute srun cc

then \ echo "Making $$i ..."; \ (cd $$i; make); \ echo ""; \ fi; \ done

$(MAKE) $(MAKE_J) struct_matrix_vector/libHYPRE_mv.astruct_linear_solvers/libHYPRE_ls.a utilities/libHYPRE_utilities.a $(PREFIX) $(MAK

11.5 I/O Performance ConsiderationsBefore building and running your parallel application, I/O performance issues on the HP XCcluster must be considere

respectively. These subsections are not full solutions for integrating MPICH with the HP XCSystem Software.Figure 11-1 MPICH Wrapper Script#!/bin/cshs

A ExamplesThis appendix provides examples that illustrate how to build and run applications on the HPXC system. The examples in this section show you

Examine the partition information:$ sinfoPARTITION AVAIL TIMELIMIT NODES STATE NODELISTlsf up infinite 6 idle n[5-10]Examine the loca

date and time stamp: Submitted from host <lsfhost.localdomain>, CWD <$HOME>, 2 Processors Requested;date and time stam

example steps through a series of commands that illustrate what occurs when you launch aninteractive shell.Examine the LSF execution host information:

Summary of time in seconds spent in various states by date and time PEND PSUSP RUN USUSP SSUSP UNKWN TOTAL 11 0 124 0 0 0

List of Examples5-1 Submitting a Job from the Standard Input...545-2

srun hostnamesrun uname -aRun the job:$ bsub -I -n4 myjobscript.shJob <1006> is submitted to default queue <normal>.<<Waiting for di

Show the SLURM job ID:$ env | grep SLURMSLURM_JOBID=74SLURM_NPROCS=8Run some commands from the pseudo-terminal:$ srun hostnamen13n13n14n14n15n15n16n16

View the node state:$ sinfoPARTITION AVAIL TIMELIMIT NODES STATE NODELISTlsf up infinite 4 idle n[13-16]A.7 Submitting an HP-MPI Jo

EXTERNAL MESSAGES:MSG_ID FROM POST_TIME MESSAGE ATTACHMENT0 - - - - 1 lsfadmin

134

GlossaryAadministrationbranchThe half (branch) of the administration network that contains all of the general-purposeadministration ports to the nodes

operating system and its loader. Together, these provide a standard environment for bootingan operating system and running preboot applications.enclos

image server A node specifically designated to hold images that will be distributed to one or more clientsystems. In a standard HP XC installation, th

LVS Linux Virtual Server. Provides a centralized login capability for system users. LVS handlesincoming login requests and directs them to a node with

onboardadministratorSee OA.PparallelapplicationAn application that uses a distributed programming model and can run on multiple processors.An HP XC MP

14

an HP XC system, the use of SMP technology increases the number of CPUs (amount ofcomputational power) available per unit of space.ssh Secure Shell. A

IndexAACML library, 49application development, 43building parallel applications, 49building serial applications, 46communication between nodes, 123com

CP3000, 24MKL library, 49system interconnect, 26CP3000BL, 24CP4000, 24ACML library, 49compilers, 44, 48designing libraries for, 50MKL library, 49softw

job accounting, 94job allocation informationobtaining, 107job manager, 96job scheduler, 96JOBID translation, 110Llaunching jobssrun, 91libraries, 31bu

building, 49compiling and linking, 49debugging, 67debugging with TotalView, 68developing, 43environment for developing, 28examples of, 125partitionrep

setting preferences, 69setting up, 68tuning applications, 85UUnified Parallel C (see UPC)UPC, 43user environment, 37utilization metrics, 73VVampir, 87

About This DocumentThis document provides information about using the features and functions of the HP XC SystemSoftware. It describes how the HP XC u

audit(5) A manpage. The manpage name is audit, and it is located inSection 5.CommandA command name or qualified command phrase.Computer outputText dis

Provides an overview of the HP XC system administrativeenvironment, cluster administration tasks, node maintenancetasks, LSF® administration tasks, an

Supplementary Software Products This section provides links to third-party and open sourcesoftware products that are integrated into the HP XC System

• http://www.llnl.gov/linux/pdsh/Home page for the parallel distributed shell (pdsh), which executes commands across HPXC client nodes in parallel.• h

© Copyright 2003, 2005, 2006, 2007 Hewlett-Packard Development Company, L.P.Confidential computer software. Valid license from HP required for possess

MPI Web Sites• http://www.mpi-forum.orgContains the official MPI standards documents, errata, and archives of the MPI Forum. TheMPI Forum is an open g

Manpages for third-party software components might be provided as a part of the deliverablesfor that component.Using discover(8) as an example, you ca

22

1 Overview of the User EnvironmentThe HP XC system is a collection of computer nodes, networks, storage, and software, built intoa cluster, that work

$ head /proc/cpuinfoTable 1-1 presents the representative output for each of the platforms. This output may differaccording to changes in models and s

distributes login requests from users. A node with the login role is referredto as a login node in this manual.compute role The compute role is assign

the HP XC. So, for example, if the HP XC system interconnect is based on a Quadrics® QsNetII® switch, then the SFS will serve files over ports on that

Additional information on supported system interconnects is provided in the HP XC HardwarePreparation Guide.1.1.8 Network Address Translation (NAT)The

Modulefiles can be loaded into the your environment automatically when you log in to thesystem, or any time you need to alter the environment. The HP

1.4.2 Serial ApplicationsYou can build and run serial applications under the HP XC development environment. A serialapplication is a command or applic

Table of ContentsAbout This Document...15Intended

1.5.3 Standard LSFStandard LSF is also available on the HP XC system. The information for using standard LSF isdocumented in the LSF manuals from Plat

— however, it manages the global MPI exchange so that all processes can communicate witheach other.See the HP-MPI documentation for more information.1

32

2 Using the SystemThis chapter describes the tasks and commands that the general user must know to use thesystem. It addresses the following topics:•

overview about some basic ways of running and managing jobs. Full information and detailsabout the HP XC job launch environment are provided in “Using

For more information about using this command and a sample of its output, see “ExaminingSystem Core Status” (page 105)• The LSF lshosts command displa

2.3.1 Determining the LSF Cluster Name and the LSF Execution HostThe lsid command returns the LSF cluster name, the LSF-HPC version, and the name of t

3 Configuring Your Environment with ModulefilesThe HP XC system supports the use of Modules software to make it easier to configure andmodify the your

access the mpi** scripts and libraries. You can specify the compiler it uses through a variety ofmechanisms long after the modulefile is loaded.The pr

Table 3-1 Supplied Modulefiles (continued)Sets the HP XC User Environment to Use:ModulefileIntel C/C++ Version 8.1 compilers.icc/8.1/defaultIntel C/C+

2.3.1 Determining the LSF Cluster Name and the LSF Execution Host...362.4 Getting System Help and Information..

Each module supplies its own online help. See “Viewing Modulefile-Specific Help” for informationon how to view it.3.3 Modulefiles Automatically Loaded

For example, if you wanted to automatically load the TotalView modulefile when you log in,edit your shell startup script to include the following inst

In this example, a user attempted to load the ifort/8.0 modulefile. After the user issued thecommand to load the modulefile, an error message occurred

4 Developing ApplicationsThis chapter discusses topics associated with developing applications in the HP XC environment.Before reading this chapter, y

HP UPC is a parallel extension of the C programming language, which runs on both commontypes of multiprocessor systems: those with a common global add

4.3 Examining Nodes and Partitions Before Running JobsBefore launching an application, you can determine the availability and status of the system&apo

4.6.1 Serial Application Build EnvironmentYou can build and run serial applications in the HP XC programming environment. A serialapplication is a com

4.7.1.1 ModulefilesThe basics of your working environment are set up automatically by your system administratorduring the installation of HP XC. Howev

To compile programs that use SHMEM, it is necessary to include the shmem.h file and to usethe SHMEM and Elan libraries. For example:$ gcc -o shping sh

4.7.1.12 MKL LibraryMKL is a math library that references pthreads, and in enabled environments, can use multiplethreads. MKL can be linked in a singl

5.2 Submitting a Serial Job Using LSF-HPC...535.2.1 Submitting a

To compile and link a C application using the mpicc command:$ mpicc -o mycode hello.c To compile and link a Fortran application using the mpif90 comma

names. However, HP recommends an alternative method. The dynamic linker, during its attemptto load libraries, will suffix candidate directories with t

NOTE: There is no shortcut as there is for the dynamic loader.52 Developing Applications

5 Submitting JobsThis chapter describes how to submit jobs on the HP XC system; it addresses the following topics:• “Overview of Job Submission” (page

The srun command is only necessary to launch the job on the allocated node if the HP XC JOBSTARTER script is not configured to run a job on the comput

The following is the C source code for this program; the file name is hw_hostname.c. #include <unistd.h> #include <stdio.h> int main() {

bsub -n num-procs [bsub-options] srun [srun-options] jobname [job-options]The bsub command submits the job to LSF-HPC.The -n num-procs parameter, whi

The srun command, used by the mpirun command to launch the MPI tasks in parallel in thelsf partition, determines the number of tasks to launch from th

With LSF-HPC integrated with SLURM, you can use the LSF-SLURM External Scheduler tospecify SLURM options that specify the minimum number of nodes requ

Example 5-9 Using the External Scheduler to Submit a Job to Run on Specific Nodes$ bsub -n4 -ext "SLURM[nodelist=n6,n8]" -I srun hostnameJob

9.3.3 Using the srun Command with LSF-HPC...929.4 Monitoring Jobs with the

Example 5-13 Using the External Scheduler to Constrain Launching to Nodes with a Given Feature$ bsub -n 10 -ext "SLURM[constraint=dualcore]"

Example 5-15 Submitting a Batch Script with the LSF-SLURM External Scheduler Option$ bsub -n4 -ext "SLURM[nodes=4]" -I ./myscript.shJob <

Example 5-18 Environment Variables Available in a Batch Job Script$ cat ./envscript.sh#!/bin/sh name=`hostname` echo "hostname = $name" echo

The ping_pong_ring application is submitted twice in a Makefile named mymake; the firsttime as run1 and the second as run2:$ cat mymake PPR_ARGS=1000

1This line attempts to submita program that does notexist.The following command line makes the program and executes it:$ bsub -o %J.out -n2 -ext "

5.6 Submitting a Job from a Host Other Than an HP XC HostTo submit a job from a host other than an HP XC host to the HP XC system, use the LSF -Roptio

66

6 Debugging ApplicationsThis chapter describes how to debug serial and parallel applications in the HP XC developmentenvironment. In general, effectiv

6.2.1 Debugging with TotalViewTotalView is a full-featured, debugger based on GUI and specifically designed to fill therequirements of parallel applic

6.2.1.3 Using TotalView with SLURMUse the following commands to allocate the nodes you need before you debug an applicationwith SLURM, as shown here:$

A.2 Launching a Serial Interactive Shell Through LSF-HPC...125A.3 Running LSF-HPC Jobs with

6.2.1.6 Debugging an ApplicationThis section describes how to use TotalView to debug an application.1. Compile the application to be debugged. For exa

6.2.1.7 Debugging Running ApplicationsAs an alternative to the method described in “Debugging an Application”, it is also possible to"attach"

72

7 Monitoring Node ActivityThis chapter describes the optional utilities that provide performance information about the setof nodes associated with you

Figure 7-1 The xcxclus Utility DisplayThe icons show most node utilization statistics as a percentage of the total resource utilization.For example, F

1The node designator is on the upper left of the icon.2The left portion of the icon represents the Ethernet connection or connections.In this illustra

Enables you to display core utilization in terms of user or system statistics,or both.CPU InfoEnables you to specify the data in terms of bandwidth or

Figure 7-4 Plotting the Data from the xcxclus UtilityThe xcxclus utility uses the GNUplot open source plotting program.See xcxclus(1) for more informa

Use the -node option to specify another node in your allocation, for example, enter thefollowing to display the performance metrics for node n6:$ xcxp

Displays the color map for the data on interrupts.EthernetDisplays the color map for the data on interrupts.InterruptsDisplays the color map for the d

8

Figure 7-7 Plotting Node Data from the xcxperf UtilityThe xcxperf utility plots the performance data and displays as much data that can be shownon one

NOTE: The --nodelist=nodelist option is particularly useful fordetermining problematic nodes.If you use this option and the --nnodes=n option, the --n

network_stress network_bidirectional network_unidirectionalBy default, the ovp command reports if the nodes passed or failed the given test.

Details of this verification have been recorded in:HOME_DIRECTORY/ovp_n16_mmddyy.logThe following example tests the memory of nodes n11, n12, n13, n14

84

8 Tuning ApplicationsThis chapter discusses how to tune applications in the HP XC environment.8.1 Using the Intel Trace Collector and Intel Trace Anal

Example 8-1 The vtjacobic Example ProgramFor the purposes of this example, the examples directory under /opt/IntelTrace/ITC iscopied to the user'

8.2 The Intel Trace Collector and Analyzer with HP-MPI on HP XCNOTE: The Intel Trace Collector (OTA) was formerly known as VampirTrace. The Intel Trac

Running a ProgramEnsure that the -static-libcxa flag is used when you use mpirun.mpich to launch a C orFortran program.The following is a C example ca

[0] Intel Trace Collector INFO: Writing tracefile vtjacobif.stf in /nis.home/user_name/xc_PDE_work/ITC_examples_xc6000mpirun exits with status: 0Runni

List of Figures4-1 Library Directory Structure...

90

9 Using SLURMHP XC uses the Simple Linux Utility for Resource Management (SLURM) for system resourcemanagement and job scheduling.This chapter address

The srun command handles both serial and parallel jobs.The srun command has a significant number of options to control the execution of yourapplicatio

Example 9-2 Displaying Queued Jobs by Their JobIDs$ squeue --jobs 12345,12346 JOBID PARTITION NAME USER ST TIME_USED NODES NODELIST(REASON)

Example 9-8 Reporting Reasons for Downed, Drained, and Draining Nodes$ sinfo -R REASON NODELIST Memory errors

10 Using LSF-HPCThe Load Sharing Facility (LSF-HPC) from Platform Computing Corporation is a batch systemresource manager used on the HP XC system.On

LSF-HPC is installed and configured on all nodes of the HP XC system by default. Nodes withoutthe compute role are closed with '0' job slots

SLURM_JOBID This environment variable is created so that subsequent srun commandsmake use of the SLURM allocation created by LSF-HPC for the job. This

Example 10-2 Examples of Launching LSF-HPC Jobs Without the srun CommandThe following bsub command line invokes the bash shell to run the hostname com

• LSF-HPC integrated with SLURM only runs daemons on one node within the HP XC system.This node hosts an HP XC LSF Alias, which is an IP address and c