Commit 1a6510f3 authored by Laura Ketzer's avatar Laura Ketzer
Browse files

.gitignore fix

parent 15a7a219
Using jobqueue
The GNU parallel command is capable to run multiple jobs on different machines, so it is siutable for job farming.
A nice example is provided by D.Streich on how to accelerate your serial analysis on the cluster:
#!/bin/bash
## Define the queue in which job will be submitted
#PBS -q medium
## Define number of requested nodes, cpu per node and maximum time of the job
#PBS -l nodes=5:ppn=8,walltime=48:00:00
## send me a mail on beginning, end and abortion of the job
#PBS -m bea
## put all sreen messages (normal and error) in the same log file
#PBS -j oe
## Do not restart job when aborted
#PBS -r n
## Write the name of used nodes to a file
nodefile=my.nodefile
headnode=`head -1 $PBS_NODEFILE`
cat $PBS_NODEFILE | uniq | awk '{print $1 ":8"}' > $nodefile
nhosts=`wc -l $nodefile | awk '{print $1}'`
nproc=$((nhosts*8))
echo "headnode=$headnode"
echo "nhosts=$nhosts"
echo "nproc=$nproc"
## here the actual program starts
galaxies='ngc5023 ngc4244 ic5052'
pops='RGB AGB lowHeB upHeB MS'
## parallel starts multiple versions of the given command. It acts like a parallel version of
#for gal in galaxies:
# for pop in pops:
# sh runfit.sh gal pop
/usr/local/bin/parallel -jobs 3 --joblog $PBS_O_WORKDIR/parallel.log --sshloginfile $PBS_NODEFILE sh $PBS_O_WORKDIR/runfit.sh ::: $galaxies ::: $pops
## NOTE: -jobs N sets the number of jobs per node (I used 3, because my runfit script already uses up to three cores)
Assuming you have jobfile like this:
myserial.exe 0000 > myscript_0000.log
myserial.exe 0001 > myscript_0001.log
myserial.exe 0002 > myscript_0002.log
myserial.exe 0003 > myscript_0003.log
myserial.exe 0004 > myscript_0004.log
myserial.exe 0005 > myscript_0005.log
myserial.exe 0006 > myscript_0006.log
myserial.exe 0007 > myscript_0007.log
myserial.exe 0008 > myscript_0008.log
myserial.exe 0009 > myscript_0009.log
myserial.exe 0010 > myscript_0010.log
etc.....
The your pbs jobscript will be:
$ cat serial2parallel.pbs
#PBS -q short
## Define number of requested nodes, cpu per node and maximum time of the job
#PBS -l nodes=2:ppn=8,walltime=00:35:00
## send me a mail on beginning, end and abortion of the job
##PBS -m e
## put all sreen messages (normal and error) in the same log file
##PBS -j oe
## Do not restart job when aborted
##PBS -r n
## Write the name of used nodes to a file
cd $PBS_O_WORKDIR
nodefile="$PBS_O_WORKDIR/mynodes.txt"
cat $PBS_NODEFILE |sort|uniq > $nodefile
## here the actual program starts
cat jobfile | /usr/local/bin/parallel -P 16 --sshloginfile $nodefile --workdir $PBS_O_WORKDIR --joblog $PBS_O_WORKDIR/parallel.log
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment