Pete Politzer Committee Report on DIII-D Data Handling and Analysis
Task Force members: K. Burrell, T. Casper, J. DeBoo, M. Fenstermacher, T. Luce, B. McHarg, Q. Peng, P. Politzer (chmn.), with a lot of help from P. Henline and R. Groebner.
1. Introduction
This DIII-D Data Analysis Task Force was set up to "… assess the adequacy of existing tools and procedures for data handling and analysis, in order to develop an integrated plan for the future to increase our scientific output …" We were asked to (1) develop an understanding of how DIII-D presently does data analysis, (2) to find out how other similar groups do it, (3) to recommend near term changes that will have an immediate impact on DIII-D's scientific output, and (4) to recommend organizational or structural changes which would increase our output for the long term. This report presents our recommendations for items 3 and 4.
The use of computers is an integral part of every aspect of our work. These have become so pervasive that there is nothing we do, from tokamak control to document preparation which does not involve computer systems and data handling. Furthermore, everything is linked and it is increasingly difficult to draw clear boundaries between different sorts of uses for computers.
However, there are three broad categories into which computer and programming activities can be divided. These are (a) tokamak operations, real-time control, data acquisition, etc., (b) the infrastructure - computers, networks, storage, etc., as well as associated software, and (c) data handling, analysis, modeling, scientific programming, etc. The emphasis of this review has been primarily on the third aspect (this is what the physics staff sees most), but because of the close coupling and the impact that any computer activity has on overall productivity, inevitably we have gotten into issues in the other areas.
Our recommendations fall into two categories. Of the many specific tasks suggested we have identified those which have sufficient importance for the DIII-D program and which would improve our productivity the most, and which therefore should be done within the next year. A few longer term tasks that need to be started within the next year are also noted. We also make recommendations for organizational changes that we feel are necessary in order to ensure that DIIID addresses the important tasks and continues to focus on those projects which have the highest priority and impact. These include recommendations for increased staffing and budget allocation for hardware. A particularly important observation is that the level of computer support in the DIII-D program has fallen well below the level needed for optimum productivity. The area of data analysis has suffered particularly in these cuts, and the connection between the physics staff and the computer division has become tenuous. The organizational changes we recommend are intended to address this problem directly.
2. Organization
A common opinion within the DIII-D program (among both the data analysts and the computer staff) is that there is no shared view of how things should be done. Every individual or small working group has developed its own analysis path, including storage of data, analysis methods and tools, databases, etc. Little attention is paid to compatibility and sharing of data, to sharing tools, or to making results widely available. The result, as many people have said, is "every man for himself."
A positive consequence of this is that the tools and data handling methods developed are very well suited to the immediate needs of the group that uses them. With a premium placed on getting "new" results, this system is very efficient for the short term. The negative consequences are that an inordinate amount of staff time is devoted to code development and maintenance. Further, tools tend to be highly idiosyncratic and can only be used by (or with a great deal of help from) the developer. This further subtracts from the time available to actually do the analyses. A third problem is that data gathered and processed by one group is inaccessible to others, leading to a significant level of duplication of effort.
A contributing factor to this state of things is the approach that has been taken by the computer division. In part as a consequence of major reductions in staff and in budget, the DIII-D computer division has increasingly focused its attention on its tokamak operations role. Even though the computer staff recognizes and tries to understand the requirements in all of these areas, the ordering of priorities has been operations first, infrastructure second, and data analysis last.* Even in the general area of operations, support for diagnostics and rf systems has been weak.
As a result, in addition to the development of data handling and analysis tools by individuals for their own use, groups within DIII-D have acquired their own workstations, and have hired their own programmers to do the necessary work. While this greatly benefits those who have the resources, others tend to be locked out. Groups zealously guard access to their workstations, and are reluctant to allow their programmers to work for others. Similarly, there is a tendency for assets (people and systems) brought to DIII-D by a collaborating institution to be reserved for the use of that group.
Recommendations:
A. Additional resources
An additional 5 professional computer staff members are needed. The emphasis should be primarily on support for applications programming.
An increment of $200K in the budget for software, principally for acquisition and installation of new database software.
An increment of $400K in the hardware budget. This would cover improving and expanding data storage, and additional computing capability to address the urgent tasks in the coming year. Note that the budgeted capital in FY96 for computers was $0.
B. Organization
Move the computer resources to the Program Support Division (reporting to K. Shoolbred). This includes all of the personnel in the present Computer group, plus others as appropriate (to be decided on a case-by-case basis). We believe that, because of the pervasive need for computer services of all kinds, this is necessary in order to maintain open communication with all of the user groups, to equitably evaluate priorities across the whole DIII-D program, and to apply resources to high priority tasks as needed.† If there is any reason that this cannot be done, the management of support for users and for applications programming should nonetheless be separate from Operations.
An advisory committee should be established. The purpose of this committee is to provide for interaction between the DIII-D operations, diagnostic, and physics groups, and the computer group. A major function would be to review and establish priorities for relatively long term tasks.
Definition of computer support tasks undertaken by collaborators should be folded into the overall priority assessment process, and judged and assigned along with all other DIII-D needs.
3. High priority tasks
These are things which should either be finished before operations start next spring, which should be completed within the next year, or which will take longer than a year but should be started on this year. Note that these are not in any particular order, either by topic or within topics.
3.a. Access to data
This is the immediate issue which prompted the establishment of the Data Analysis Task Force. The problem is that, at present, restoring shot files from tape to disk for analysis can take more than 24 hours if the system is busy, and that the lifetime for keeping a shot on disk is short.
The immediate crunch has been alleviated by the installation of additional disk capacity, doubling the number of shots that can be kept on disk. Also, part of the work has been done to set up a system for restoring only portions of shots (usually, only a small portion of the shot data is wanted). Several parallel approaches to the problem of data access are recommended. These address the partitioning of stored data, the storage media, keeping certain data always on disk, and caching.
Recommendations:
Immediately add disk space (done July 96).
Refine and implement plan for requesting and restoring portions of shot files. This has been started, and is expected to be completed by November.
Establish a central archive of EFIT output files (e0 files, g0 files). This would include e0 files for all shots, and g0 files as analyses are done (including all g0 files generated in the control room would be excessive - the intent is to keep only the slices that someone is interested in). Some way of incorporating additional identifying information, such as the snap file used and who did the analysis, is needed.
Decide on whether to pursue a robot (e.g., optical disk juke box) storage system (terabyte capacity). The information has to be gathered and specifications developed before the end of the year. The decision whether to proceed will depend, in part, on the extent to which the partitioning system improves access. This decision should be made by December.
Develop a system for keeping a modest amount of frequently used data permanently on disk. Integrate this with the LLNL cache of data needed for EFIT. A definite plan for what to keep, where, and how should be prepared by next spring.
3.b. Databases
Improvement of databases can provide major leverage on long-term productivity. However, the present system is antiquated, difficult to use, and no longer adequately serves the needs of the physics group. Furthermore, it uses S1032 software which is only available on the VAXs. As a result there has been a proliferation of private databases of varying size and complexity. In turn, this has led to duplication of both analyses and experiments, and to the creation of large private archives. Furthermore, having a very limited entry of slices into the database can bias any statistical conclusions that might be drawn. Another problem is that the concept of "blessed" data has evolved, and it is no longer feasible to have every datum entered into the database examined by hand.
We have identified three different requirements for databases: DIII-D needs a broad, but shallow database containing basic information from essentially all DIII-D discharges. This will facilitate surveys and searches of past experiments to find what has already been done, to determine trends, to select shots for detailed analysis, and to plan new experiments. It should include a small number of slices (5-10 perhaps) from each shot to provide some information about time history. Data entry would be automatic.
At the other extreme we need an archive for highly analyzed, publication-quality data. This will serve for publication, for outside distribution, and for testing models. All of the supporting data files (e.g., EFIT & ONETWO output) would be included.
In between are the many private, special purpose databases. We recognize the need for such archives, but suggest establishing a directory of such databases, and accumulating a set of tools for viewing and extracting information. This will allow the DIII-D physics community to know what data has been analyzed, how, and by whom (something presently lacking). This network of databases should work toward a common format for data entry and examination.
Finally, we have to detach ourselves from reliance on the VAX platform, and select a new software system which works under UNIX.
Implementation of a new database system will require work on the development of tools and procedures for both entry and examination of data, as well as the work needed to install the system itself.
Recommendations:
Set up a small expert task force to prepare an integrated plan for new databases. Many detailed questions of requirements, formats, tools, and basic software have to be clearly formulated and answered. We would expect to be ready to proceed with the purchase of a new software system and with the commitment of manpower by the end of November.
3.c. Code development and support (general)
The area of code development has not received much in the way of professional support within the DIII-D program. Adding people to the DIII-D staff with specific expertise in this area will greatly enhance our productivity.
Recommendations:
If we do nothing else, it is important to add professional staff to help with code support, maintenance, implementation of better algorithms, benchmarking, porting, etc. (This is discussed under Organization.)
The absence of documentation (including information about the existence of codes) has impeded utilization of many codes, functions, utilities, and other tools. A simple, descriptive documentation of both major codes and useful tools people write for their own use should be implemented, using web pages. For IDL routines, there is an IDL utility that will extract comments automatically, and create an html file. We should, using whatever carrots and sticks are available, encourage a minimum standard documentation.
We also need a common library of basic tools and routines for such things as getting particular diagnostic data, mapping and other simple calculations from EFIT output, etc.
We note the need of the divertor and SOL group for a consistent user interface, and for integration of their many diagnostics. While the direction and specification must come from the physicists, professional programming help should be made available.
A small expert task force should be established to consider how to deal with large quantities of 2D (and 3-D) data. The questions include storage, retrieval, and viewing of processed data, how to incorporate such data into the databases, etc. A formal proposal on how to proceed should be prepared by the end of December.
3.d. Code development and support (specific codes & analyses)
The following is a list of additions and modifications to existing codes, or development of new codes which should be undertaken this year. Note there are code development tasks presently in progress that are not listed here (plasma control, CERFIT upgrades, Thomson control programming, …). The listing here of high priority new items for this year does not imply that present tasks are less important (see footnote on page 2).
EFIT: - make available (web pages) complete, up-to-date documentation - an easy to use interface - make 65x65 meshes the default (after 4D is benchmarked and stable)
4D: - make available (web pages) complete, up-to-date documentation - complete the replacement of the functionality of ENERGY - set up and carry out a program to benchmark 4D (a physics task) - collect, document, and maintain a 4D function library - set up a procedure to monitor and eliminate runaway processes
CER: - finish port of CERQUICK to HPs (find a compiler expert) - set up and carry out a program to benchmark CERQUICK (a physics task)
ONETWO: - finish time-dependent modifications (we needed to list something already done) - automate the use of standard (IDL, netcdf?) file formats for input & output
Stability codes: - port GATO to the HPs, and provide a dedicated workstation for running it - develop a script/interface to allow wider use of GATO
Divertor codes: - Again, we note the need of the divertor and SOL group for a portfolio of mapping routines, data viewing routines, new analysis routines, etc., all with common interfaces. While it is the primary job of the physicists to do this, professional assistance with programming, numerical methods, and other aspects of software engineering should be available now. This will avoid the problems that have arisen with (for example) 4D, where the professionals are asked to "clean up" a code system after a lot of work has been done.
- Another task which should be undertaken by the divertor group (with help) is the development of an analysis code (or an analysis mode for UEDGE), to replace the cumbersome iterative use of complex modeling to fit data.
Spectral analysis codes: - The physics group should decide (in a timely fashion) whether to undertake the development of a new spectral analysis code (with 2-D capability, capability of mapping to flux surfaces, …). This would eliminate the need to port NEWSPEC and MODE1 to the HPs.
Other observations: - Also raised was the need to automate the kinetic EFIT process, for which the next step is to eliminate the need for ONETWO by separating the beam pressure calculation. - We take note of the suggestions made regarding the benefits of Matlab vs. IDL. However, this is really a user-driven choice, and for most people the choice has already been made.
3.e. Data acquisition and control room analyses
Recommendations:
Implement use of the second data highway to speed acquisition. This is being done.
Have an automatic EFIT in place before operations next year. Replace and retire MFIT.
Improve the display and control of the fast delay trigger and other timing. This needs a graphic display. This should be done in consultation with the diagnosticians and physicists most concerned about fluctuation analyses (magnetics, density) to optimize the display.
Survey the data acquisition system for possible choke points (e.g., the CER microVAX). Develop a proposal to relieve any problems. Also, publicize the capability to re-order the acquisition sequence.
3.f. User support, networks, & other
Recommendations:
Set up a system for regular, standardized backups of Macs and PCs.
Upgrade the building 13 network. Some work is being done.
Improve familiarity with IDL. Provide more introductory books and manuals. Maybe a brief familiarization session.
Set up a small expert task force to recommend whether and how to implement some level of a distributed computing environment. This can make the network invisible to the users, and can make much better use of available CPU cycles.
.