Interactive Computer Graphics for Understanding Science William Hibbard Space Science and Engineering Center University of Wisconsin - Madison The two most familiar uses of computer graphics are movies and video games. Movies generally need graphics that are indistinguishable from physical reality whereas video games need graphics that can be generated quickly enough to be perceived as smooth motion. These two needs are incompatible, at least at the speeds of current computers. While the realistic computer graphics of movies have much greater emotional impact than the fast graphics of video games, the latter define a new interactive medium because their speed allows them to be computed in response to user actions. That is, they define two-way communications between users and computations. Users see graphical depictions of the numbers produced by computations, and control those computations by moving a mouse or joystick. In video games, the subject matter of computations is generally characters chasing and shooting at each other. However, the subject matter may be anything, and there is a less familiar use of computer graphics for interacting with scientific computations. About 1988 computers became fast enough to generate 3-D graphics at interactive speeds. This triggered a variety of new software systems for interacting with scientific computations, including general purpose systems like IBM's Data Explorer and Stellar Computer's Application Visualization System, and special purpose systems like NASA's FAST for aerodynamic simulations and our own Vis5D for weather and other environmental simulations. Vis5D grew out of the long term interest in weather graphics at the Space Science and Engineering Center (SSEC). In the early 1980's we began experiments with 3-D graphics generated from all sorts of weather data, including observations from surface stations, balloons, aircraft and ships, images from satellites, radars and lidars, and the output of numerical weather models. These experiments led to two conclusions: 1. 3-D weather graphics are only useful when they can be interactively rotated. 3-D graphics are inherently ambiguous because many points in the 3-D atmosphere are projected onto the same point on a 2-D display screen or on users' 2-D retinas. The most effective way to resolve this ambiguity is to allow users to interactively rotate their viewpoint, to change the mapping from 3-D to 2-D. 2. It is easier to make 3-D graphics from simulations than from observations, because of the errors and coverage gaps of observations, and because of the greater variety of data collection platforms and data organizations for observations. Based on these conclusions, Vis5D was written in 1988 to generate interactive 3-D graphics from weather simulations. Users do not interact directly with the computations of weather models, since they run too slowly to be visually interesting. Rather, users explore the large volumes of data left by weather models after they finish, by interacting with the computations that convert that output data into 3-D images. While the initial goal of interactive graphics was providing the rotation necessary to resolve 3-D ambiguity, we exploited the interactive capability in many other ways to enable scientists to explore their data. Vis5D's users can interactively choose from a variety of ways of depicting temperature, pressure, humidity and other atmospheric state variables. These may be quantitative and focused, such as planar colored slices through data volumes, or qualitative and holistic, such as transparent colored "fogs" showing entire data volumes at once. Users can also interactively choose to: view atmospheric state variables in various combinations to search for cause and effect relations between them; move forward and backward through time in order to trace physical events; and make and display computations such as the ratio of cloud water to total water. The development of Vis5D was supported by NASA and EPA, and we believe that publicly supported software should be free to the public. Thus we began giving Vis5D away in 1989, and it is available to anyone at http://www.ssec.wisc.edu/~billh/vis5d.html. The first computers capable of interactive 3-D graphics were quite expensive and discouraged widespread use of Vis5D. However, the price of interactive 3-D graphics has declined to the point where Vis5D is widely used at universities and research labs around the world, and even by some people on their home computers. One of the benefits of giving Vis5D away is that energetic users add new capabilities and give the changes back to us, to be shared with other users. For example, two students at Rijks University in the Netherlands modified Vis5D so that it could run on a wider variety of computers, and a staff member at Beijing University is currently modifying Vis5D to run on Windows NT and adding a user interface option based on Chinese characters. Recognizing the value of such user contributions, we restructured Vis5D in 1995 to make it easier for others to add their own modifications. There are now dozens of derived versions branching out from Vis5D. One of these is called Cave5D and provides Vis5D graphics in immersive virtual reality. This involves specialized graphical displays requiring users to wear electronic goggles, giving users the illusion of being immersed inside of weather graphics. The development of Cave5D has been a collaboration between SSEC, Old Dominion University and the University of Illinois, as part of the NSF supported National Computational Science Alliance (NCSA). We used Cave5D at the 1994 Siggraph Conference in Orlando to show attendees each day's Florida weather. Figure 1 is a Cave5D image showing how colliding sea breezes create thunderstorms over central Florida. Other virtual reality versions are being developed at the National Center for Atmospheric Research and at the NASA Goddard Space Flight Center. The most exciting Vis5D collaborations are with labs and agencies with public scientific responsibilities. For example, we are collaborating with the EPA to develop a new version of Vis5D useful for interactive exploration of linked simulations of multiple environmental systems such as weather, air pollution, oceans, ground water and lakes. The graphical depictions of each simulation are overlaid in the same virtual earth display so that scientists can see how their simulated environmental systems interact with each other. This version is also useful for visualizing multiple simulations of weather known as ensemble forecasts. Given the limited accuracy of weather forecasts, one new approach is to run perhaps 50 slightly different computer forecasts of the next few days weather and use them to estimate probabilities of weather events. These forecasts are called ensemble members. If 45 of 50 ensemble members say it will rain, then rain is likely, but if 25 of 50 say it will rain, then it is anyone's guess. The new version of Vis5D provides a spread sheet style of capability to visualize ensemble members side-by-side on the display screen. When users rotate the display of one ensemble member, or make other interactive changes to the display, these changes are reflected in the displays of all ensemble members. The U. S. Environmental Modeling Center and the European Center for Medium-range Weather Forecasting run ensemble forecasts and we are actively collaborating with them to use the new Vis5D. Vis5D has proven the value of interactive 3-D displays for the scientists who develop weather models. However, there are many more meteorologists who use the output of weather models to provide forecasts to the public. We are collaborating with the National Weather Service's (NWS) Forecast Systems Lab (FSL) to determine whether Vis5D graphics are useful to forecasters in NWS field offices. Model developers require a very wide degree of freedom in their user interface for choosing graphics, in order to understand arbitrary mechanisms and interactions within their models, whereas forecasters require user interfaces that provide quick access to a standardized set of graphics. Thus, FSL is designing and evaluating a new Vis5D user interface for forecasters. Simulations are easy to visualize because they represent an idealized mathematical world. Observations are full of errors, have uneven coverage in space and time, and are often indirect. Software for visualizing observations must have much greater flexibility, which implies a more complex user interface and fewer assumptions that can be exploited for computational efficiency. Thus visualizing observations has been a real challenge for software designers. Vis5D does allow scientists to compare simulations with satellite images, and a capability is in the works to compare simulations with observations by balloons, aircraft and other in situ instruments. However, it would be a mistake to make Vis5D too general. It has been successful just because of its focus on the problem of visualizing environmental simulations. Thus in 1991 we started work on another system, VisAD, to generate graphical depictions of the full variety of environmental observations and simulations and to give users a greater variety of ways of interacting with computations involving those data. A major part of observational science is developing appropriate computational algorithms for analyzing observations. Examples include: detecting errors, filling in gaps in coverage, converting indirect measurements into more useful quantities like temperature and pressure, and reconstructing the 3-D structure of the atmosphere from 2-D images. Thus, in addition to enabling scientists to interact with computations, VisAD enables them to interactively experiment with the algorithms behind those computations. The name VisAD stands for "Visualization for Algorithm Development" to stress the significance of this type of interaction between user and computer. Interactive experiments with science algorithms can be useful in may way. For example, the Diffuse X-ray Spectrometer (DXS) flew on the Space Shuttle in 1993 and returned several million events, each potentially an X-ray emanating from interstellar gas. However, most of the events were spurious, so the DXS scientists used VisAD to experiment with algorithms for discriminating good events from bad. VisAD was also used for experiments with algorithms for discriminating clouds in weather satellite images, and visual experiments with the way that multi-spectral satellite instruments observe vertical columns of atmosphere. Experiments with algorithms are very effective for students, and VisAD has been used to teach numerical modeling concepts via an interactive 2-D shallow fluid model, and to teach about chaos and atmospheric turbulence. Our collaboration with scientists was very useful in giving us feedback about the design of VisAD. This collaboration indicated the importance of computations that combine data from multiple sources and of enabling multiple users to work together in developing algorithms. In other words, visualization systems need to exploit Internet connections between data sources and between scientists in new ways. In 1995 the Java programming language was introduced as a way to more fully exploit Internet connections between computers. Java allows computers to send each other programs to run (Java applets in web pages are a simple example of this), and Java makes it possible to write programs that "run on the network" rather than on any particular computer. So we rewrote VisAD in Java. Now we and others are writing VisAD applications that enable many scientists to share the same graphics and user interfaces as if they were sitting in front of the same computer. This is called collaborative graphics. Whereas interactive graphics define communications between one user and computations, collaborative graphics define communications between multiple users and shared computations. Collaborative graphics will be critical for scientists at different institutions and in different disciplines to work together to build a coherent model of the earth as a whole. We worked with a group of scientists who design weather satellites to write a VisAD application, named GoesCollaboration, for visual experiments with the way that multi-spectral satellite instruments observe vertical columns of atmosphere. Each pixel in a satellite image corresponds to an entire vertical column of atmosphere - this is analogous to the ambiguity in 3-D graphics. Vertical distributions of temperature, water vapor and ozone can be estimated by collecting satellite observations in many different spectral bands. The GoesCollaboration application displays graphs of temperature, water vapor and ozone in a vertical column, along with other graphics that show how satellites will observe this vertical column and what errors to expect in the process of estimating vertical structure from satellite observations. Users can redraw the graphs of the vertical column, triggering recomputation of satellite responses and expected estimation errors. Multiple users at different computers share the displays and interactions of this application. When any user redraws the vertical graphs, they all see the same computational responses. The GoesCollaboration application is a practical illustration of the ways that computers can define new communication media: the results of computations are communicated to scientists by graphical depictions of data; scientists communicate back to computations by redrawing those graphical depictions; and scientists communicate with each other via the computational consequences of those redrawn graphics. We believe that this flexibility will spawn a wide variety of collaborative and interactive visualization applications for anyone using numerical data, including physical scientists, social scientists, business people, and educators. We are making VisAD freely available to the public at http://www.ssec.wisc.edu/~billh/visad.html and look forward to many useful collaborations to develop applications based on it. Interactive and collaborative computer graphics are still in their infancy. Current computer speeds limit the quality of images that can be produced fast enough to be perceived as smooth motion, and limit the complexity of virtual worlds that can be simulated by computations. However, progress with computer and communication speeds will make shared interactive graphical worlds the medium of the 21st century, just as movies and television have been the media of the 20th century. It is very exciting to participate in the development of a new medium, even when the purpose is science rather than entertainment. I encourage students with interests in computer science and art to become involved. Author Biosketch: Dr. Hibbard is a Scientist at the University of Wisconsin Space Science and Engineering Center. His SSEC Visualization Project focuses on making advanced visualization techniques useful in the daily work of physical scientists. He can be reached via the World Wide Web at http://www.ssec.wisc.edu/~billh/vis.html.