Commit 395bcc0f authored by Laura Ketzer's avatar Laura Ketzer
Browse files

Delete AIP_Cluster_instructions.txt

parent 15a7a219
Using jobqueue
The GNU parallel command is capable to run multiple jobs on different machines, so it is siutable for job farming.
A nice example is provided by D.Streich on how to accelerate your serial analysis on the cluster:
#!/bin/bash
## Define the queue in which job will be submitted
#PBS -q medium
## Define number of requested nodes, cpu per node and maximum time of the job
#PBS -l nodes=5:ppn=8,walltime=48:00:00
## send me a mail on beginning, end and abortion of the job
#PBS -m bea
## put all sreen messages (normal and error) in the same log file
#PBS -j oe
## Do not restart job when aborted
#PBS -r n
## Write the name of used nodes to a file
nodefile=my.nodefile
headnode=`head -1 $PBS_NODEFILE`
cat $PBS_NODEFILE | uniq | awk '{print $1 ":8"}' > $nodefile
nhosts=`wc -l $nodefile | awk '{print $1}'`
nproc=$((nhosts*8))
echo "headnode=$headnode"
echo "nhosts=$nhosts"
echo "nproc=$nproc"
## here the actual program starts
galaxies='ngc5023 ngc4244 ic5052'
pops='RGB AGB lowHeB upHeB MS'
## parallel starts multiple versions of the given command. It acts like a parallel version of
#for gal in galaxies:
# for pop in pops:
# sh runfit.sh gal pop
/usr/local/bin/parallel -jobs 3 --joblog $PBS_O_WORKDIR/parallel.log --sshloginfile $PBS_NODEFILE sh $PBS_O_WORKDIR/runfit.sh ::: $galaxies ::: $pops
## NOTE: -jobs N sets the number of jobs per node (I used 3, because my runfit script already uses up to three cores)
Assuming you have jobfile like this:
myserial.exe 0000 > myscript_0000.log
myserial.exe 0001 > myscript_0001.log
myserial.exe 0002 > myscript_0002.log
myserial.exe 0003 > myscript_0003.log
myserial.exe 0004 > myscript_0004.log
myserial.exe 0005 > myscript_0005.log
myserial.exe 0006 > myscript_0006.log
myserial.exe 0007 > myscript_0007.log
myserial.exe 0008 > myscript_0008.log
myserial.exe 0009 > myscript_0009.log
myserial.exe 0010 > myscript_0010.log
etc.....
The your pbs jobscript will be:
$ cat serial2parallel.pbs
#PBS -q short
## Define number of requested nodes, cpu per node and maximum time of the job
#PBS -l nodes=2:ppn=8,walltime=00:35:00
## send me a mail on beginning, end and abortion of the job
##PBS -m e
## put all sreen messages (normal and error) in the same log file
##PBS -j oe
## Do not restart job when aborted
##PBS -r n
## Write the name of used nodes to a file
cd $PBS_O_WORKDIR
nodefile="$PBS_O_WORKDIR/mynodes.txt"
cat $PBS_NODEFILE |sort|uniq > $nodefile
## here the actual program starts
cat jobfile | /usr/local/bin/parallel -P 16 --sshloginfile $nodefile --workdir $PBS_O_WORKDIR --joblog $PBS_O_WORKDIR/parallel.log
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment