Information on Running GATO



Date:    Wed, 6 Sep 2000 12:59:52 -0700
From:    FERRON@fusion.gat.com
To:      Turnbull@fusion.gat.com, ferron@fusion.gat.com, Lao@fusion.gat.com,
         Garofalo@fusion.gat.com, chum@fusion.gat.com, jayakuma@fusion.gat.com,
         snyder@fusion.gat.com, osborne@fusion.gat.com, brennan@fusion.gat.com,
         Freemanj@fusion.gat.com
Subject: Latest GATO version available

Folks,
The most recent version of GATO is available.  Below is the latest version of
my notes describing how to get it.
John

Information on running GATO
===========================

version  6/29/2000 by John Ferron /u/ferron/gato/howto.txt
Changes: 6/29/2000 Add sections 6 through 8
         7/06/2000 Add section 9
         8/16/2000 The most recent released executable is in
                   /d/gato/Alpha_executables_v2a.  Text below modified to
                   specify this directory.
         8/25/2000 Add information on running gato on katze. See item 10.
                   New script available which should be used on katze.
                   Large results files are compressed with gzip.
         9/06/2000 The most recent released executable is in
                   /d/gato/Alpha_executables_v3.  Text below modified to
                   specify this directory. Note that to prevent GATO
                   from executing the inverse iterations, set nitmax=0
                   in this new version (previously it was nitmax=-1).
                   The script script/setup_restart has been modified for this
                   change. 

Here is a summary of how to use my set of scripts for running gato.
Note that if you want to run on katze, you should pay special attention 
to item 10 below.

See item 11 for a description of how to build the gato executables if that is
necessary. 

This information should be relevant to any GATO installation (NERSC, General
Atomics etc.).


1. Set up the assumed directory structure:
   o Make a directory called, for example, gato.  
   o Within this directory make directories called run, input, results.
   o Optionally make other directories: script, inputmaster. (These
     directories are necessary only if you want to make your own versions
     of scripts and master input files, which is rather probable.)
   o Make links to installation directories
     (here I assume that your gato directory is in your home directory and
     that the gato files are in /d/gato):

       cd ~/gato
       ln -s /d/gato/Alpha_executables_v3 ./sources2000
       ln -s /d/gato/script                ./script
       ln -s /d/gato/inputmaster           ./inputmaster

     The script and inputmaster links are only necessary if you want
     to use the default set of scripts and input files. Otherwise, you
     will have made your own script and inputmaster directories as
     mentioned above.

     The paths for the links shown here are for running at General
     Atomics. You would need to adjust the paths appropriately for use on
     different machines  (such as NERSC).

     Note that the path to the directory with the latest set of executables
     will change occasionally.  In case there are jobs in progress using older
     executables, those older directories will be retained.

   o The purpose for these directories:
     gato/run = location where the script for a particular convergence
                study will be  created.  This directory will contain
                files which can be deleted when the gato run is
                complete.
     gato/input = location where the  gato input files and the
                equilibrium files will be set up prior to the gato run
                being started. This directory will contain
                files which can be deleted when the gato run is
                complete.
     gato/results = the directory where the results from the run will
                be placed.  Each gato run will have its own
                subdirectory.
     gato/inputmaster = contains the template files for gato input
                files for convergence studies at various values    
                of toroidal mode
                number.  These files are the basis to create the input
                files for a
                particular run.  You might want to copy my example 
                files and modify
                them for your own purposes.
     gato/script = directory containing the template files for scripts
                to run a convergence study.  
                Also contains the scripts for creating
                files for a convergence study. 
   o You need an environment variable TMPDIR that points to the area
     which will be used to hold temporary files during a gato run.  Gato
     can require a large amount of space, depending on the mesh density 
     (see below).  Make certain that the area indicated by this environment 
     variable has enough space.  All temporary files are deleted at the end 
     of a gato run.  (Except: on katze TMPDIR is ignored, see item 10.)
   o The executable "/d/gato/Alpha_executables_v3/replace" must be in your
     path shell variable.  At General Atomics this is the same as Joe Freeman's
     routine that should already be in everyone's path.

2. Copy your equilibrium input file to gato/input.
   My scripts assume that the name of the equilibrium file is toq_case
   efit_case depending on whether the equilibrium is from toq or EFIT.
   Here, "case" is the case label which I like to keep to  4 characters
   or less so that the full job name appears in the NERSC job queue information
   displays.

3. Create the files required for a convergence study.
   o The basic script is ~/gato/script/create_converge.  Two other useful
   scripts which call create_converge are 

     ~/gato/script/create_converge_conformal
     ~/gato/script/create_converge_d3d

   Read the documentation
   at the beginning of these files for details on using them.
   o Set your default directory to the gato area.
      cd  ~/gato
   o Execute the script for creating the convergence study files.  Here
     the example is for a case called aa10 with toroidal mode number 1:
     script/create_converge_conformal aa10 1
   o After the script is run the scripts to run gato will be in
   ~/gato/run.  For example ~/gato/run/aa10n11 is the script to run
   case aa10 for toroidal mode number 1 at the first group of mesh sizes.
   ~/gato/run/aa10n12 runs the second group of mesh sizes.  The mesh density
   runs are grouped together for historical reasons (best fit to NERSC
    queues).  You may want to alter this grouping.  As the group number 
   increases, the required disk space and CPU time increase.

4. Run as many of the convergence study scripts as desired.  For example,
   to run it interactively:

     run/aan11

   At General Atomics, I find this command convenient to start the run:

     nice +19 run/aan11 >& log/log_date &

   where "log" is a directory to place the log for this run.  This command
   starts the job in the background (so that you can log out) at the lowest
   possible priority (useful for running jobs on badger, ulam etc so that
   only the extra CPU time is utilized).

   Or, submit the script to a NERSC batch queue (and wait...):

     cqsub run/aan11

5. After the convergence study runs are complete:
   o Look in the directory  results/Caa10n1 for the output files.
   o you can collect the growth rate values:
   script/collect_summary aa10n1
   o A simple IDL routine to make a plot of the convergence results is:
   script/plot_summary.pro

6. Choosing output files to be saved:
   o GATO produces a large number of files while it executes.  Some of
   these files can be very large.  Many files are used as scratch storage
   and are not worth saving after the run.  However, there are some files
   that should be saved to record the run's results.  
   o If GATO fails for
   some reason, it is desirable to make certain that a few key files
   are retained.  This is particularly important at NERSC because there the
   content of the temporary directory is removed when the batch job completes.
   o So, the basic script that runs a single GATO job at a single mesh density,
   script/runGATO_f90.script_newg, creates most files in the temporary
   directory indicated by the TMPDIR environment variable (except on katze
   where the temporary directory is hardwired).  The files
   that might be useful to diagnose a job failure are created in the
   job directory in the gato/results area.  Links to these files are
   created in the temporary directory.
   o After a GATO job completes successfully, the 
   script/runGATO_f90.script_newg script copies the essential output files
   to the gato/results area.  Other files are simply deleted.  You may need to
   modify the script/runGATO_f90.script_newg script to save other files that
   you require.  The necessary copy commands are in the script, but
   are commented out by default. Large results files are compressed with gzip
   to save space.

7. Plotting GATO results:
   o GATO produces a 2 large plot files as one of its primary outputs.  There
   are parameters that can be included in the GATO input file to control
   the plots that are produced.  Even so, you might want to print just a
   selection of the plots in the output file.  Here are examples of
   commands to do this.
      - The version of gato using the pgplot package (e.g. Alpha version at 
   General Atomics) produces a PostScript file.  This command prints
   selected pages from this file:
   psselect -p1,4,6,10  outpt4.ps | lp -dhplj21_duplex
   or, for gzipped files:
   gunzip -c outpt4.ps.gz | psselect -p1,4,6,10 | lp -dhplj21_duplex
      - At NERSC, a cgm file is produced.  This command prints
   selected pages from this file:
   ictrans -d ps.mono -e '1 4 6 10 plot' outpt.cgm4 | lp -dhplj21_duplex

8. Restarting a GATO job in the middle:
   o The most time-intensive portion of the GATO run is the execution of
   the eigenvalue solver code seigh. This code executes typically 5-10
   Cholesky decompositions, each of which takes a significant amount of time.
   If the GATO run fails for some reason in the middle of this set of
   decompositions it is possible to start the job again where it was at
   the  end of the last decomposition that was fully completed.  GATO writes
   a file after each decomposition containing the necessary information.
   This can save significant time for large mesh densities.
    o There is a script which will do the work necessary to prepare for
   restarting a GATO run. This script saves some of the files from the
   previous run in a subdirectory named with the time and date just in case
   they are needed later.  Then, the GATO input file is modified to set 
   the flag indicating that the job should be restarted. See the script
   comments for details.  Here is an example of restarting the jobs in
   group number 1 for case aa10 at toroidal mode number 3:
   script/setup_restart aa10 3 1
   o When a job is restarted, since the first two portions of a GATO run
    (smaph, swnwh) take relatively little time, they are always rerun
   completely.
   o After running script/setup_restart, 
   simply run your job again as described above in item 4.

9. Here is a sample command to start a script running in the background
   at a low priority with all output sent to a log file.

   nice +19 do_gato >& log/log20000706 &

   Here, for example, the script do_gato contains commands to run portions of a
   convergence study.  E.g.:

#! /bin/csh -f
run/aa37n21
run/aa37n22
run/aa37n23
run/aa37n24


10.The machine katze.gat.com is available specifically for running GATO.
   This machine is equipped specially so that large mesh density jobs
   can be run, so often there will be jobs running there requiring large 
   disk space and large amounts of CPU time.  So, users will need to
   cooperate to make certain that this resource is used without problems.

   There are special disks on katze to hold the GATO temporary files.
   To use these disks properly, the cooperative user will use the 
   most recent script for
   running gato (listed above in item 4, script/runGATO_f90.script_newg).
   This script will distribute temporary files evenly among the available
   scratch areas.  Each user interested in running GATO on katze should request
   that directories be made in the katze scratch areas. These directories
   should only contain files when there is a running GATO job.  Don't store
   other files there.

   Jobs with relatively small mesh density (300x600 for isym=1,
   180x360 for isym=0) should be fairly easily runable without conflicting
   with other users.  For larger jobs we need to work out a scheduling
   system.

   One suggestion is to utilize CPU time available on other machines to
   do the calculation up to the point where the scratch space is needed.
   In the GATO input file, set nitmax = 0 so that the inverse iterations
   will not be run.  Then run your job on any available CPU.  Then restart
   your job on katze to run the inverse iterations which require 
   large scratch space. Follow the guide in item 8 for information on
   restarting a job easily.

   The execution script for GATO will automatically clean up the temporary
   storage areas.  However, if the script is terminated abnormally, this
   will not happen.  So, please check your scratch directory areas
   (in /scratch0,/scratch1,/scratch2,/scratch3) to make certain that no
   old files are left there.  We need to maximize the available 
   scratch storage space.

   katze has a license for only 2 users.  It is best to use the
   "penetrate katze" command to log in.

11.Building GATO:
   The source for GATO is in /d/gato/sourcemaster_v3.  There is a make file
   in that directory which makes building the executables easy.  Read
   the comments in /d/gato/sourcemaster_v3/Makefile for more information.

12.Disk and CPU time requirements:
   See /u/ferron/gato/notes.txt for some information on disk and CPU time test
   results.


This page http://lithos.gat.com/THEORY/gato/Gato_summary.html was updated Wednesday, 9 August 2006 at 03:53:58 PM by Joe Freeman.

Back to the top