Date: Wed, 6 Sep 2000 12:59:52 -0700
From: FERRON@fusion.gat.com
To: Turnbull@fusion.gat.com, ferron@fusion.gat.com, Lao@fusion.gat.com,
Garofalo@fusion.gat.com, chum@fusion.gat.com, jayakuma@fusion.gat.com,
snyder@fusion.gat.com, osborne@fusion.gat.com, brennan@fusion.gat.com,
Freemanj@fusion.gat.com
Subject: Latest GATO version available
Folks,
The most recent version of GATO is available. Below is the latest version of
my notes describing how to get it.
John
Information on running GATO
===========================
version 6/29/2000 by John Ferron /u/ferron/gato/howto.txt
Changes: 6/29/2000 Add sections 6 through 8
7/06/2000 Add section 9
8/16/2000 The most recent released executable is in
/d/gato/Alpha_executables_v2a. Text below modified to
specify this directory.
8/25/2000 Add information on running gato on katze. See item 10.
New script available which should be used on katze.
Large results files are compressed with gzip.
9/06/2000 The most recent released executable is in
/d/gato/Alpha_executables_v3. Text below modified to
specify this directory. Note that to prevent GATO
from executing the inverse iterations, set nitmax=0
in this new version (previously it was nitmax=-1).
The script script/setup_restart has been modified for this
change.
Here is a summary of how to use my set of scripts for running gato.
Note that if you want to run on katze, you should pay special attention
to item 10 below.
See item 11 for a description of how to build the gato executables if that is
necessary.
This information should be relevant to any GATO installation (NERSC, General
Atomics etc.).
1. Set up the assumed directory structure:
o Make a directory called, for example, gato.
o Within this directory make directories called run, input, results.
o Optionally make other directories: script, inputmaster. (These
directories are necessary only if you want to make your own versions
of scripts and master input files, which is rather probable.)
o Make links to installation directories
(here I assume that your gato directory is in your home directory and
that the gato files are in /d/gato):
cd ~/gato
ln -s /d/gato/Alpha_executables_v3 ./sources2000
ln -s /d/gato/script ./script
ln -s /d/gato/inputmaster ./inputmaster
The script and inputmaster links are only necessary if you want
to use the default set of scripts and input files. Otherwise, you
will have made your own script and inputmaster directories as
mentioned above.
The paths for the links shown here are for running at General
Atomics. You would need to adjust the paths appropriately for use on
different machines (such as NERSC).
Note that the path to the directory with the latest set of executables
will change occasionally. In case there are jobs in progress using older
executables, those older directories will be retained.
o The purpose for these directories:
gato/run = location where the script for a particular convergence
study will be created. This directory will contain
files which can be deleted when the gato run is
complete.
gato/input = location where the gato input files and the
equilibrium files will be set up prior to the gato run
being started. This directory will contain
files which can be deleted when the gato run is
complete.
gato/results = the directory where the results from the run will
be placed. Each gato run will have its own
subdirectory.
gato/inputmaster = contains the template files for gato input
files for convergence studies at various values
of toroidal mode
number. These files are the basis to create the input
files for a
particular run. You might want to copy my example
files and modify
them for your own purposes.
gato/script = directory containing the template files for scripts
to run a convergence study.
Also contains the scripts for creating
files for a convergence study.
o You need an environment variable TMPDIR that points to the area
which will be used to hold temporary files during a gato run. Gato
can require a large amount of space, depending on the mesh density
(see below). Make certain that the area indicated by this environment
variable has enough space. All temporary files are deleted at the end
of a gato run. (Except: on katze TMPDIR is ignored, see item 10.)
o The executable "/d/gato/Alpha_executables_v3/replace" must be in your
path shell variable. At General Atomics this is the same as Joe Freeman's
routine that should already be in everyone's path.
2. Copy your equilibrium input file to gato/input.
My scripts assume that the name of the equilibrium file is toq_case
efit_case depending on whether the equilibrium is from toq or EFIT.
Here, "case" is the case label which I like to keep to 4 characters
or less so that the full job name appears in the NERSC job queue information
displays.
3. Create the files required for a convergence study.
o The basic script is ~/gato/script/create_converge. Two other useful
scripts which call create_converge are
~/gato/script/create_converge_conformal
~/gato/script/create_converge_d3d
Read the documentation
at the beginning of these files for details on using them.
o Set your default directory to the gato area.
cd ~/gato
o Execute the script for creating the convergence study files. Here
the example is for a case called aa10 with toroidal mode number 1:
script/create_converge_conformal aa10 1
o After the script is run the scripts to run gato will be in
~/gato/run. For example ~/gato/run/aa10n11 is the script to run
case aa10 for toroidal mode number 1 at the first group of mesh sizes.
~/gato/run/aa10n12 runs the second group of mesh sizes. The mesh density
runs are grouped together for historical reasons (best fit to NERSC
queues). You may want to alter this grouping. As the group number
increases, the required disk space and CPU time increase.
4. Run as many of the convergence study scripts as desired. For example,
to run it interactively:
run/aan11
At General Atomics, I find this command convenient to start the run:
nice +19 run/aan11 >& log/log_date &
where "log" is a directory to place the log for this run. This command
starts the job in the background (so that you can log out) at the lowest
possible priority (useful for running jobs on badger, ulam etc so that
only the extra CPU time is utilized).
Or, submit the script to a NERSC batch queue (and wait...):
cqsub run/aan11
5. After the convergence study runs are complete:
o Look in the directory results/Caa10n1 for the output files.
o you can collect the growth rate values:
script/collect_summary aa10n1
o A simple IDL routine to make a plot of the convergence results is:
script/plot_summary.pro
6. Choosing output files to be saved:
o GATO produces a large number of files while it executes. Some of
these files can be very large. Many files are used as scratch storage
and are not worth saving after the run. However, there are some files
that should be saved to record the run's results.
o If GATO fails for
some reason, it is desirable to make certain that a few key files
are retained. This is particularly important at NERSC because there the
content of the temporary directory is removed when the batch job completes.
o So, the basic script that runs a single GATO job at a single mesh density,
script/runGATO_f90.script_newg, creates most files in the temporary
directory indicated by the TMPDIR environment variable (except on katze
where the temporary directory is hardwired). The files
that might be useful to diagnose a job failure are created in the
job directory in the gato/results area. Links to these files are
created in the temporary directory.
o After a GATO job completes successfully, the
script/runGATO_f90.script_newg script copies the essential output files
to the gato/results area. Other files are simply deleted. You may need to
modify the script/runGATO_f90.script_newg script to save other files that
you require. The necessary copy commands are in the script, but
are commented out by default. Large results files are compressed with gzip
to save space.
7. Plotting GATO results:
o GATO produces a 2 large plot files as one of its primary outputs. There
are parameters that can be included in the GATO input file to control
the plots that are produced. Even so, you might want to print just a
selection of the plots in the output file. Here are examples of
commands to do this.
- The version of gato using the pgplot package (e.g. Alpha version at
General Atomics) produces a PostScript file. This command prints
selected pages from this file:
psselect -p1,4,6,10 outpt4.ps | lp -dhplj21_duplex
or, for gzipped files:
gunzip -c outpt4.ps.gz | psselect -p1,4,6,10 | lp -dhplj21_duplex
- At NERSC, a cgm file is produced. This command prints
selected pages from this file:
ictrans -d ps.mono -e '1 4 6 10 plot' outpt.cgm4 | lp -dhplj21_duplex
8. Restarting a GATO job in the middle:
o The most time-intensive portion of the GATO run is the execution of
the eigenvalue solver code seigh. This code executes typically 5-10
Cholesky decompositions, each of which takes a significant amount of time.
If the GATO run fails for some reason in the middle of this set of
decompositions it is possible to start the job again where it was at
the end of the last decomposition that was fully completed. GATO writes
a file after each decomposition containing the necessary information.
This can save significant time for large mesh densities.
o There is a script which will do the work necessary to prepare for
restarting a GATO run. This script saves some of the files from the
previous run in a subdirectory named with the time and date just in case
they are needed later. Then, the GATO input file is modified to set
the flag indicating that the job should be restarted. See the script
comments for details. Here is an example of restarting the jobs in
group number 1 for case aa10 at toroidal mode number 3:
script/setup_restart aa10 3 1
o When a job is restarted, since the first two portions of a GATO run
(smaph, swnwh) take relatively little time, they are always rerun
completely.
o After running script/setup_restart,
simply run your job again as described above in item 4.
9. Here is a sample command to start a script running in the background
at a low priority with all output sent to a log file.
nice +19 do_gato >& log/log20000706 &
Here, for example, the script do_gato contains commands to run portions of a
convergence study. E.g.:
#! /bin/csh -f
run/aa37n21
run/aa37n22
run/aa37n23
run/aa37n24
10.The machine katze.gat.com is available specifically for running GATO.
This machine is equipped specially so that large mesh density jobs
can be run, so often there will be jobs running there requiring large
disk space and large amounts of CPU time. So, users will need to
cooperate to make certain that this resource is used without problems.
There are special disks on katze to hold the GATO temporary files.
To use these disks properly, the cooperative user will use the
most recent script for
running gato (listed above in item 4, script/runGATO_f90.script_newg).
This script will distribute temporary files evenly among the available
scratch areas. Each user interested in running GATO on katze should request
that directories be made in the katze scratch areas. These directories
should only contain files when there is a running GATO job. Don't store
other files there.
Jobs with relatively small mesh density (300x600 for isym=1,
180x360 for isym=0) should be fairly easily runable without conflicting
with other users. For larger jobs we need to work out a scheduling
system.
One suggestion is to utilize CPU time available on other machines to
do the calculation up to the point where the scratch space is needed.
In the GATO input file, set nitmax = 0 so that the inverse iterations
will not be run. Then run your job on any available CPU. Then restart
your job on katze to run the inverse iterations which require
large scratch space. Follow the guide in item 8 for information on
restarting a job easily.
The execution script for GATO will automatically clean up the temporary
storage areas. However, if the script is terminated abnormally, this
will not happen. So, please check your scratch directory areas
(in /scratch0,/scratch1,/scratch2,/scratch3) to make certain that no
old files are left there. We need to maximize the available
scratch storage space.
katze has a license for only 2 users. It is best to use the
"penetrate katze" command to log in.
11.Building GATO:
The source for GATO is in /d/gato/sourcemaster_v3. There is a make file
in that directory which makes building the executables easy. Read
the comments in /d/gato/sourcemaster_v3/Makefile for more information.
12.Disk and CPU time requirements:
See /u/ferron/gato/notes.txt for some information on disk and CPU time test
results.
This page http://lithos.gat.com/THEORY/gato/Gato_summary.html was updated Wednesday, 9 August 2006 at 03:53:58 PM by Joe Freeman.