Exploring Coupled Atmosphere-Ocean Models Using Vis5D William Hibbard1, John Anderson1, Ian Foster2, Brian Paul1, Robert Jacob1, Chad Schafer2, Mary Tyree1 1Space Science and Engineering Center University of Wisconsin-Madison Madison, WI 53706 2Mathematics and Computer Science Division Argonne National Laboratory Argonne, IL 60439 contact: William Hibbard SSEC, 1225 W. Dayton St., Madison, WI 53706 608-263-4427 fax: 608-263-6738 email: whibbard@macc.wisc.edu revised: 20 December 1995 Abstract: A distributed client / server system can be used to visualize very large simulation data sets. An example of a very large simulation data set is a 100 year simulation of the Earth's coupled atmosphere-ocean system. This model run was produced by an Argonne National Laboratory / University of Wisconsin collaborative project which is studying atmosphere / ocean coupling dynamics in order to understand the intrinsic low frequency variability of the climate system. This understanding is crucial for the prediction and detection of human impacts on the Earth's climate. In order to visually explore this simulation, an IBM SP-2 is used as a data server and a pair of SGI Onyx's driving a CAVE is used as a graphics client. The SP-2 server divides the data set into sections that will fit in the memory of the graphics client. The data set is divided along the time axis. One data set section covers the entire 100 year span of the simulation at reduced time resolution, while other data set sections cover short sub-intervals at full time resolution. The visualization user interface allows users to switch between low and high time resolution. 1. Introduction Beginning in the late 1980's, high performance workstations have enabled rapid improvements in scientists' capability to visualize their data [DeFanti, Brown, McCormack, 1989]. Prior to that period, scientists could display images and graphics on computer screens, and could animate sequences of images or graphics by pre-computing them and storing them in large memories. The revolution in visualization was created by the capability of workstations to compute images and graphics (including 3-D graphics) at the rate required for animation. This capability enabled the creation of interactive software systems that provided users with immediate control over displays, for example to rotate a scene in 3-D. General purpose systems like AVS [Upson, Faulhaber, Kamins, Laidlaw, Schlegel, Vroom, Gurwitz and van Dam, 1989], Iris Explorer and Data Explorer [Lucas, Abrams, Collins, Epstein, Gresh and McAuliffe, 1992] help scientists to build visualization applications suited to their data, while specialized systems like Vis5D [Hibbard and Santek, 1990] provide user interfaces and performance levels tuned to the specific needs of focused user communities. Interactive systems help scientists to cope with the problem posed by the size of their data sets. That is, scientific data sets have become much too large and complex to be encoded in a single image or even a single animation sequence. Interactive systems enable scientists to browse through large data sets, searching for problems and diagnosing their causes. However, workstation memory capacities limit the size of data sets that can be visualized. Current workstations typically have primary memory capacities in the tens of megabytes and secondary storage (disk) capacities in the gigabytes. Furthermore, tertiary storage devices on workstations (tapes) are too slow to support interactive visualization. In order to visualize very large data sets, on the order of tens to hundreds of gigabytes, access to supercomputer storage devices via high speed networks is required. The interactive graphics capabilities of workstation must be combined with the storage capacities of supercomputers, connected by high speed network. In this paper we report on experiments adapting Vis5D to work in a distributed environment in order to visualize very large data sets. These experiments have been performed using the Blanca Gigabit Network Testbed, and using the Information Wide Area Year (I-WAY) network and Global Information Infrastructure (GII) Testbed deployed at the IEEE Supercomputing '95 Conference. The Vis5D system creates the illusion of a virtual Earth environment behind the workstation screen. In this paper we report on experiments adapting Vis5D to create an immersive virtual reality using the CAVE [Cruz-Neira, Sandin and DeFanti, 1993] display system. The CAVE gives the user the illusion of being located in a 3-D graphical space by surrounding the user with graphics projection screens, by providing binocular stereo, and by adjusting 3-D perspective according to the user's head location. We also report on experiments using the CAVE user interface for controlling visualization in a distributed computing environment. 2. Interactive Visualization of Atmosphere and Ocean Simulations The atmosphere and oceans fill three-dimensional volumes with fluids that move over time and whose states are described by sets of variables such as temperature, pressure, velocity (wind or current), and chemical concentrations (principally water in the atmosphere and salt in the oceans). The output data of numerical simulations of the atmosphere and oceans typically consist of four-dimensional fields for each of these variables, sampled over a three-dimensional spatial grid and a sequence of times. Such data sets can be regarded as five-dimensional rectangles of data, where the fifth data dimension enumerates the 4-D fields corresponding to different state variables. The Vis5D system was developed to provide scientists with interactive 3-D visualization of the five-dimensional rectangles of data produced by atmosphere and ocean simulations. The diagram in Figure 1 shows how Vis5D organizes the 5-D data rectangle into a 2-D array of 3-D spatial grids. The diagram also shows how such data are transformed into images by a two stage process. The first stage derives various 3-D geometric representations of the 3-D grids, such as iso-surfaces, sets of iso-lines on horizontal and vertical 2-D slices, pseudo-colored images on 2-D slices, and volume renderings (i.e., transparent fogs). The second stage renders these 3-D geometries onto a 2-D workstation screen in a way that creates the visual illusion of three dimensions. Vis5D also includes flow rendering techniques for vector fields associated with wind or current motion. These include 2-D arrays of vectors on horizontal and vertical planes, streamlines on 2-D planes, and 3-D particle motion trajectories. Figure 2 is a screen shot of Vis5D. The 3-D images rendered from model data are displayed in the window on the right, and the arrays of icons on the left give users interactive control over those 3-D images. Users can select arbitrary combinations of model fields and rendering techniques, and can control the details of those rendering techniques (e.g., values for iso-surfaces, locations of 2-D plane slices, color maps for pseudo-colored images, and locations of starting points for computing motion trajectories). Users can also rotate their view of the 3-D scene and control animation to visualize model dynamics. Figure 2 was generated from a simulation by the Limited Area Mesoscale Prediction System (LAMPS). It shows a U. S. topographic map, iso-lines of potential temperature on a vertical plane running East to West, a pseudo-color image of wind speed on a vertical plane running North to South, a yellow iso-surface of water vapor (at 5 g/kg), cyan wind trajectory ribbons, and a white volume rendering of cloud water. By animating such 3-D images scientists can quickly search through very large data sets to find problems with their simulations. Then by comparing different model fields and by stepping forward and backward through time, scientists can analyze the basic causes of those problems. In Figure 3, Vis5D is used to compare clouds predicted by the CIMSS Regional Assimilation System (CRAS) with clouds observed by the Geostationary Operational Meteorological Satellite (GOES). The gray-scale satellite images are texture mapped onto U. S. topography and the predicted cloud fields are volume rendered as a yellow fog. Such comparisons between simulation and observation are an important area for future development of Vis5D. 3. Visualization in a Distributed Computing Environment The fact that Vis5D runs on simple workstations (it has recently even been ported to run on Pentium processors under Linux) makes it accessible to most scientists and has contributed to its popularity. However, workstation storage capacities limit the sizes of data sets that can be visualized. Thus we have experimented with ways to exploit distributed computing resources. In particular, we have split Vis5D into a graphics client running on a workstation and a data server running on a supercomputer. Initially this was done as part of the Blanca Gigabit Network Testbed [Hibbard, Santek and Tripoli, 1991; Paul, Battaiola and Hibbard, 1993]. The diagram in Figure 4 shows that the server stores the model output grids and transforms them into 3-D geometric primitives (i.e., the first stage of the transformation pipeline seen in Figure 1) while the client renders these geometric primitives into 3-D images (i.e., the second stage of transformation pipeline). In this design, network transactions are relatively small and user interactions requiring quick responses (e.g., animation and changing the levels of iso-surfaces) must go through the network loop to the server. The inherent parallelism of this design effectively utilized the large resources of gigabit networks and dedicated supercomputers. For example, this design achieved a sustained network bandwidth of 95 Mbps using a dedicated Cray YMP server. However, long delays (e.g., seconds to tens of seconds) are common with shared access to networks and supercomputers, and with the design shown in Figure 4 most user interactions were subject to network delays. In fact, the only user interactions not subject to network delays were 3-D rotations. One way to diminish the impact of network delays is to make the size of network transactions roughly equal to the length of expected delays. Thus our approach with the I-WAY and the Supercomputing 95 GII was to divide large simulation data sets into a number of subsets whose transmission takes between seconds and tens of seconds. The I-WAY network consisted primarily of OC-3 links running at 155 Mbps (roughly 20 MB/s). The corresponding transaction size was between tens and hundreds of megabytes and our model output data set was divided into subsets of this size. The division was done along the time axis, so that each subset was a five dimensional rectangle with a smaller number of time steps than the original data set. Figure 5 illustrates this division along the time axis. There was one subset that covered the entire time span of the original data set at a reduced time resolution, and many subsets that covered subintervals at the original time resolution. The low time resolution subset was always available in the workstation client, to give users an overview of their entire data sets. The user could access the high time resolution subsets one at a time to concentrate on the detailed dynamics of short time intervals. Our approaches with the Blanca Gigabit Testbed and with the I-WAY differ in where we place the network connection in the Vis5D data flow. For our work with the Blanca network we transmitted 3-D geometries over the network, whereas for our work with the I-WAY network we transmitted gridded data. This choice is partly dictated by our expectation of computational resource levels in the server (i.e., shared versus dedicated) since the transformation of gridded data into 3-D geometries is computationally intensive. However, this choice is primarily driven by our desire to make most user interactions independent of the delays associated with network transactions. By transmitting subsets of the data in its rawest (i.e., gridded) form, most interactions only involve data residing locally in the client. We note that it would also be possible to transmit rendered 3-D images. However, this would force even 3-D rotation interactions to go through the server. It would also place the rendering task in the supercomputer server which is considerably less efficient than using 3-D graphics hardware in the workstation client. 4. Visualization with a Virtual Reality Interface The hardware environment of the Supercomputing 95 GII emphasized both distributed computing and virtual reality. Since the Vis5D system is based on a virtual world interface it can be adapted to immersive virtual reality quite easily. In fact, Vis5D was previously adapted to run in the virtual reality CAVE [Cruz-Neira, Sandin and DeFanti, 1993] for the VROOM at Siggraph 94 [Hibbard, Paul, Battaiola, Tripoli, Pokrandt and Cohen, 1994]. The CAVE system gives the user the illusion of being located in a 3-D graphical space by surrounding the user with graphics projection screens, by providing binocular stereo, and by adjusting 3-D perspective according to the user's head location. When Vis5D runs in the CAVE, users have the illusion of being located in the Earth's atmosphere or oceans and watching dynamics evolve around them. The Vis5D demo in the Siggraph 94 CAVE showed that day's weather prediction over Florida (the conference was located in Florida). Figure 6 shows a typical Vis5D scene of Florida weather during the Siggraph 94 conference. While the 3-D scenes generated by Vis5D map quite naturally to the CAVE's virtual reality interface, Vis5D's user interaction icons do not. For the Siggraph 94 demo, a few of the icon functions were mapped to the CAVE's 3 button wand (i.e., the 3-D analog of a mouse), but most of the Vis5D icon-based interactions were not available during the demo. The Vis5D user interface does include some direct manipulation, such moving rotating the 3-D scene, dragging plane slices through the 3-D volume, and locating the starting point for motion trajectories. These direct manipulation interactions did map naturally to the CAVE, although with some modification. For example, 3-D scene rotation is unnecessary since users can simple rotate their heads. However, users are able to fly around in the CAVE's virtual 3-D space in the direction they pointed the wand. 5. Coupled Atmosphere-Ocean Simulations The coupled atmosphere/ocean climate model system was developed by a collaborative effort involving scientists at the University of Wisconsin-Madison and Argonne National laboratory for the purpose of studying climate variability on century time scales. The goal of the initial system development was to build a coupled model system which could run on parallel machines such as the IBM SP-2 at speeds in excess of 10,000 * real-time. At this throughput rate the model system will produce a 1,000 year climate system integration is just over 1 month and in the process 12,000 monthly model history model history files totaling about 100 GB will be created. The first version of the model was completed in Fall 1995 and is currently running in excess of 3,500 * real-time on the Argonne SP-2. The data set which was visualized in the SC 95 demonstration consisted of a 100 year test integration which was the first long production run of the coupled model. The initial model system consisted of three functional elements, the first of which was an atmosphere model which is based on the National Center for Atmospheric Research (NCAR) Community Climate Model (CCM2) which was been adapted to the parallel MPI environment by Argonne and Oak Ridge National Laboratories [Drake, Foster, Michalakes, Toonen and Worley, 1995]. This atmospheric model interfaced to the University of Wisconsin parallel ocean model using a parallel coupler module which computes fluxes of heat, moisture, and momentum between the model systems. Each of these modules was responsible for saving history and restart files during the run. 6. Visualizing Large Coupled Atmosphere - Ocean Simulations The principle challenge for the Vis5D demo at Siggraph 94 was providing real time weather predictions, which required the use of networking once per day to transfer model output data sets from Wisconsin to Florida. The principle challenge for the Vis5D demo at Supercomputing 95 was providing access to model output data sets too large to fit on the workstation, which required use of networking continuously during the demos. The large size of the history file data base precludes the simple transfer of the data set to a user workstation due to limitations in both local storage and communications bandwidth. The use of the client-server model to implement the visualization process was an effort to provide interactive access to the data with reasonable response times. This built on the idea that in most cases the user does not need full access to all of the data simultaneously. Rather we provided the ability to scan the model history files with reduced time resolution and then select limited duration sections to examine with the full time resolution. In order to implement the data server we used the parallel I/O and data archive systems on the Argonne SP-2 to stage model data to the node local disks on an SP-2 partition consisting of several nodes using an MPI programming model. The client ran on a pair of SGI Onyxs with Reality Engine graphics accelerators located at the site of Supercomputing 95 in San Diego. The client and server communicated using sockets over the I-WAY network. Figure 7 is a diagram of the top-level hardware organization. Each of 12 SP-2 PE's stored roughly 1/12 of the total data set. During our experiments at Supercomputing 95 the data set was divided into high time resolution subsets of roughly 25 MB each. The latency between the request for a high time resolution subset until it appeared in the display was typically 25 to 30 seconds. Figure 8 shows a typical Vis5D scene of the coupled atmosphere - ocean model. It includes a pseudo-colored depiction of sea surface temperatures and upper air wind vectors, over global topography and bathymetry. For the Supercomputing 95 demo we added a small panel of icons embedded in the 3-D virtual space. These include animation controls, an indicator of the location of the current scene along the overall 100 year time line, and icons for switching between the low time resolution subset and one of the high time resolution subsets. The choice of a particular high time resolution subset is governed by the current location along the time axis. The panel also includes an icon for toggling topography and bathymetry on and off, and icons for switching between different modes. Unlike the normal situation in Vis5D, where users can select arbitrary combinations of model fields and rendering techniques, the Vis5D CAVE demos include a small list of different modes, each with a fixed combination of fields and rendering techniques. As with the Siggraph 94 Vis5D CAVE demo, direct manipulation via the wand is available for flying through the 3-D scene and for moving plane slices through the volume. Because of the long time steps (monthly) of the coupled atmosphere - ocean model data set, motion trajectories are not possible for this demo. 7. Conclusion The Vis5D system enables scientists to use simple workstations to interactively visualize their moderately-sized numerical simulations of the Earth's atmosphere and oceans. The system can be adapted to use distributed computing and virtual reality techniques to provide interactive visualization of coupled atmosphere - ocean simulations too large for simple workstations. The Blanca Gigabit Network Testbed, the CAVE virtual reality system, and most recently the I-WAY network and associated GII Testbed at the IEEE Supercomputing 95 Conference have provided the resources required to experiment with ways to adapt Vis5D to distributed computing and virtual reality. These experiments point to the need for novel user interfaces for adapting workstation-based virtual world systems to immersive virtual reality, and to the need to make user interactions as independent as possible of delays associated with networks and remote supercomputers. In particular, the experiments with the GII Testbed demonstrated a successful client-server architecture for visualizing large simulation data sets. We plan to develop this client-server architecture as a standard part of the Vis5D system. Acknowledgements This work was supported by the Department of Energy CHAMMP Program and by the National Science Foundation and the Advanced Research Projects Agency under Cooperative Agreement NCR-8919038 with the Corporation for National Research Initiatives. References Cruz-Neira, C., D. J. Sandin, and T. A. DeFanti, 1993; Surround-screen projection-based virtual reality: the design and implementation of the CAVE. Proc. Siggraph 93. 135-142. DeFanti, T. A., M. D. Brown, and B. H. McCormick, 1989; Visualization: expanding scientific and engineering research opportunities. IEEE Computer 22(8), 12-25. Drake, J., I. Foster, J. Michalakes, B. Toonen, and P. Worley, 1995; Design and implementation of a parallel climate model. Parallel Computing, in press. Hibbard, W., 1986; Computer generated imagery for 4-D meteorological data. Bull. Amer. Met. Soc., 67, 1362-1369. Hibbard, W., 1986; 4-D display of meteorological data. Proceedings, 1986 Workshop on Interactive 3D Graphics. Chapel Hill, Siggraph, 23-36. Hibbard, W., and D. Santek, 1989; Visualizing large data sets in the earth sciences. IEEE Computer 22(8), 53-57. Hibbard, W., and D. Santek, 1990; The VIS-5D system for easy interactive visualization. Proc. Visualization '90, IEEE. 28-35. Hibbard, W., D. Santek, and G. Tripoli, 1991; Interactive atmospheric data access via high speed networks. Computer Networks and ISDN Systems, 22, 103-109. Hibbard, W., and B. Paul, 1992; Distributed visualization at the Space Science and Engineering Center. Notes for Siggraph course 7, Distributed Scientific Visualization on High-Performance Networks, 4.1-4.22. Hibbard, W., 1992; Systems issues for the 120 Terabyte GOES archive. Proc., Managing Terabyte Databases in the 90s and beyond. NOAA and the MITRE Corp. 24-25. Hibbard, W. L., B. E. Paul, A. L. Battaiola, D. A. Santek, M-F. Voidrot-Martinez and C. R. Dyer, 1994; Interactive Visualization of Computations in the Earth and Space Sciences. IEEE Computer 27(7), 65-72. Hibbard, W., B. Paul, A. Battaiola, G. Tripoli, P. Pokrandt, and S. Cohen, 1994; The Siggraph '94 daily weather forecast. Siggraph '94 Visual Proceedings, 263. Lucas, B., G. D. Abrams, N. S. Collins, D. A. Epstein, D. L. Gresh, and K. P. McAuliffe, 1992; An architecture for a scientific visualization system. Proc. Visualization '92, IEEE, 107-114. McCormick, B., T. DeFanti, and M. Brown, 1987; Visualization in Scientific Computing. Computer Graphics, 21(6). Paul, B., A. Battailoa and W. Hibbard, 1993; Progress with VIS-5D / distributed VIS-5D. Preprints, Conf. Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology. Anaheim, Amer. Meteor. Soc., 162-164. Upson, C., T. Faulhaber, Jr., D. Kamins, D. Laidlaw, D. Schlegel, J. Vroom, R. Gurwitz, and A. van Dam, 1989; The application visualization system: A computational environment for scientific visualization. Computer Graphics and Applications, 9(4), 30-42. Captions for color figures: Figure 2. Iso-lines of potential temperature and a pseudo-color image of wind speed on vertical planes, a yellow iso- surface of water vapor at 5 g/kg, cyan wind trajectory ribbons, and a white volume rendering of cloud water, over a U. S. topographic map. Figure 3. Comparison of a yellow volume rendering of predicted clouds with a satellite image texture mapped onto U. S. topography. Figure 6. Low altitude wind vectors showing colliding sea breezes with a white volume rendering of thunderstorm clouds, over Florida, as seen in the CAVE. Figure 8. Pseudo-colored sea surface temperatures on a horizontal plane and upper atmosphere wind vectors over global topography/bathymetry, as seen in the CAVE. Captions for B&W diagram figures: Figure 1. Vis5D transforms simulations of the Earth's atmosphere and oceans into an interactive graphical environment. Figure 4. Distributed Vis5D organization. Figure 5. A large data set is partitioned into subsets along the time axis. Figure 7. Top-level hardware organization. Author Biographies: William L. Hibbard is a Scientist at the Space Science and Engineering Center of the University of Wisconsin-Madison. He was the Principal Investigator of the NASA grant that supported the development of the Vis5D and VisAD systems, and was an investigator of the Blanca Gigabit Network Testbed. Hibbard has been a member of the Program Committees for the IEEE Visualization Conferences for 1990 through 1996. He received a B.A. degree in Mathematics in 1970, an M.S. degree in Computer Sciences in 1974, and a Ph.D. degree in Computer Sciences in 1995, all from the University of Wisconsin - Madison. See http://www.ssec.wisc.edu/~billh/vis.html for more information. John Anderson is a Professor in the Department of Atmospheric and Oceanic Sciences and Associate Director of the Space Science and Engineering Center at the University of Wisconsin - Madison. He is the originator and PI of the Wisconsin Model Engine Project and a member of the DOE CHAMMP science team. He is a member of the NASA Earth Observing System Laser Atmospheric Wind Sounder (LAWS) science team. He has been strongly involved in the development of radar signal and data processing algorithms having been the system engineer for the MIT - Lincoln Laboratory MTD-2 radar which led to the purchase of the ASR-9 radar series by the Federal Aviation Administration. Anderson received an S.B. in Physics from the Massachusetts Inst. of Technology in 1976, an S.M. in Meteorology from the Massachusetts Inst. of Technology in 1982, and a Ph.D. in Atmos. Sci. from Colorado State University in 1984. He was elected to Sigma Xi in 1976, received the Colorado State University President's Fellowship in 1983, and received the AMS Max A. Eaton Award in 1985. Ian Foster received his BSc (Hons I) in Computer Science from the University of Canterbury, New Zealand, and his PhD in Computer Science from Imperial College, London. In 1989, the Strand programming system which he co-designed was awarded the British Computer Society's award for technical innovation. He is currently a Scientist in the Mathematics and Computer Science Division at Argonne National Laboratory, and Associate Professor of Computer Science at the University of Chicago. Dr. Foster has published three books and numerous papers on various aspects of parallel and distributed computing. His current research focuses on the techniques required to integrate high- performance computing into large-scale internetworked environments. He recently served as Software Architect for the I-WAY distributed computing and networking experiment. Brian Paul is a Researcher at the Space Science and Engineering Center at the University of Wisconsin - Madison. He is author of the Mesa 3-D graphics library, and co-author of the Vis5D and VisAD systems. His interests include computer graphics and visualization. He received his BS in Computer Science at the University of Wisconsin - Oshkosh in 1990 and his MS in computer science from the University of Wisconsin - Madison in 1994. Brian is a member of the Association for Computing Machinery. He may be contacted at brianp@ssec.wisc.edu. Chad Schafer is currently a research assistant in the Mathematics and Computer Science Division at Argonne National Laboratory. His work at Argonne has involved the development a coupled ocean/atmosphere climate model. In 1995 he received a B.S. from Western Michigan University, and in the fall of 1996 he will begin work towards a Ph.D. in statistics at the University of Illinois at Urbana-Champaign. Robert Jacob is a Research Assistant at the Space Science and Engineering Center at the University of Wisconsin-Madison. He is also a dissertator in the Department of Atmospheric and Oceanic Sciences at UW-Madison. His interests are in long term climate change, ocean-atmosphere dynamics and numerical modeling. He received B.S. degrees in Physics and in Mathematics from the University of Texas at Austin in 1990. He can be reached at rob@ssec.wisc.edu. Mary K. Tyree is a Research Associate at the Space Science and Engineering Center of the University of Wisconsin-Madison. She has also worked as a Physicist/Research Scientist at the Lawrence Livermore National Laboratory and as a Programmer/Analyst at Scripps Institution of Oceanography. She received a B.S. degree in Meteorology in 1985 and an M.S. degree in Meteorology in 1988, both from the University of Wisconsin - Madison.