Rapid advances in computer technology have propelled visualization to new prominence as a basic tool for scientific research. Maps, of course, were key visualization tools for both the earth and the social sciences well before the advent of "scientific" visualization. Cartographic methods (many developed more than a century ago), have been cited as a basis for most representation methods now used in scientific visualization (see Collins 1993). The interaction between cartography and scientific visualization is, however, not a one-way street. Technology developed for scientific visualization is making practical the dynamic/interactive maps cartographers have advocated (and produced in prototype form) for decades (e.g., Thrower 1959; Cornwell 1966; Tobler 1970; Moellering 1973; Moellering 1984; MacEachren 1987; Slocum 1988; Beard and Buttenfield 1991).
The integration of technology for scientific visualization and established methods for representing geo-referenced data is facilitating fundamental changes in how science interacts with these data and in the nature of knowledge that is created through manipulating and linking graphic representations of all forms (not only maps, but images, diagrams, graphs, etc.). This interface between cartography and scientific visualization and between technology for mapping and ways in which mapping can facilitate geographic thinking has been labeled Geographic Visualization (geovisualization) (MacEachren and Monmonier 1992). geovisualization has been characterized as a kind of geo-information use – with emphasis on individuals using interactive visual tools in a search for unknowns (MacEachren 1994).
The potential for productive integration of scientific visualization and geographic representation methods (i.e., for the fruitful application of geovisualization) is particularly high in the context of research on environmental change. Environmental change, by definition, has both spatial and temporal components. Most environmental data are geo-referenced and maps are often used by both scientists and policy analysts to understand the spatial distribution of individual environmental variables as well as the spatial associations among environmental (and other) variables. Map series are often used to analyze change in these distributions and associations.
As the understanding of environmental change has become an important input to policy decisions, increased attention has been given to issues of environmental data reliability. Like the data they are associated with, reliability estimates (and other forms of metadata1) exhibit spatial and temporal variation. Tools are needed that facilitate visualization of the spatial and temporal components of environmental data together with the spatial and temporal components of associated reliability estimates. This paper presents a conceptual overview of interface design issues associated with such tools followed by a description of a prototype geovisualization system designed to facilitate analysis of the spatial and temporal aspects of environmental data and its reliability.
In relation to geographic information systems (GIS), Frank (1993) suggests that "the user interface is the system." His point is that the user interface is often the only part of a system that the user has direct contact with and that its design is thus a crucial factor in the success of a GIS. This perspective is, perhaps, even more applicable to geovisualization (at least when emphasis is on interactive tools that facilitate data exploration). A change in the display is often considered to be a secondary consequence of a user action in a GIS. For geovisualization, however, it is these changes to the visual display that are the primary purpose of system use. The result of a geovisualization operation is a change in the visual interface to data. A successful geovisualization will facilitate creative thinking by allowing a user to change the display quickly in predictable ways – thus using the system (and the power of vision) to notice patterns and associations (MacEachren and Ganter 1990).
Like map design, interface design is best approached at multiple levels within which related decisions are made. An interface can be considered to be a system for processing information, translating back and forth between "languages" understood by the user and the computer platform. A useful starting point for identifying appropriate levels of analysis, therefore, is suggested by Marr (1982). He proposed that any information processing system can profitably be analyzed at three linked, but independent, levels: the level of computational theory (where we describe what a process must do and why, along with a logical strategy for carrying out the process), the level of representation and algorithms (where we deal with how theory might be applied to the specification of particular operations), and the device/hardware implementation level (where we deal with how particular representations and algorithms might be implemented with the available materials).
Foley et al., (1990) have offered a related multi-component approach that was specifically developed to address issues of interface design (in particular, interfaces for computer graphics systems). They proposed two major components of design, related to meaning (or content) and to form. Within meaning, two elements are defined, "conceptual design" and "functional design. Conceptual design is characterized as defining objects together with properties of, relations between, and operations on these objects. Functional design specifies the information needed for operations, the errors that can occur, how the errors are handled, and the potential results of operations. In relation to Marr’s (1982) tri-level model, both of these elements seem to correspond with the level of representation and algorithm.2 The form elements defined by Foley et al., (1990) are "sequencing design" and "binding design." These elements deal, respectively, with the ordering of inputs and outputs and how the inputs and outputs of meaning are actually formed from hardware primitives. Form, then, seems to be largely defined as a hardware implementation issue.
Lindholm and Sarjakoski (1994) offer an extension of the above ideas to geovisualization. They credit Foley et al., (1990) as the source for their initial ideas, but the perspective presented also has similarities to Marr’s (1982) levels of analysis for information systems in general. Like Marr, Lindholm and Sarjakoski specify three levels of analysis which they label conceptual, functional, and appearance. The conceptual level corresponds largely with Marr’s level of computational theory and deals with the use and operation of the system in a general way. The critical issues to be dealt with here are what the system is for and what outcome is desired from system use. Lindholm and Sarjakoski also give considerable attention (at the conceptual level) to who the users are and what kind of data is to be visualized (issues not discussed by Marr (1982)). Their discussion of kinds of data to visualize is related to the conceptual design level definition of objects and their properties proposed by Foley et al., (1990), thus to Marr’s (1982) representation and algorithm level.
Lindholm and Sarjakoski’s functional level is where particular operations that the user can perform are specified and the meaning and interaction style for these operations are defined. This level seems to merge elements of functional and sequencing design as described by Foley et al. (1990). It thus deals with issues that, according to Marr (1982), are at both the representation/algorithm and the hardware implementation levels.
Lindholm and Sarjakoski’s third level, appearance, deals with "the output language (how to present the application and data to the user)." (Lindholm and Sarjakoski 1994; p. 172) Emphasis is on appearance of the display, including both the controls and any graphic elements that will appear. Issues dealt with here correspond closely to those described by Foley, et al., (1990) under the heading of "binding design," and to Marr's (1982) hardware implementation level.
Here we present a synthesis of the above perspectives. This synthesis follows Marr’s (1982) lead in putting more emphasis on the goals of using a system than on the details of implementing the system on a particular hardware-software configuration. We draw heavily on Lindholm and Sarjakoski’s efforts to adapt general interface design theory to geovisualization, but attempt to retain the distinction between representation and implementation delineated by Marr. Our approach distinguishes among three levels of analysis: conceptual, operation, and implementation.
Conceptual level issues are those associated with a system as a connection to information, rather than with specifics of the interface between a user and the system. Following Lindholm and Sarjakoski (1994), we define conceptual issues for geovisualization in terms of a set of questions to address: what need is met by the system, how is this goal reached, what should be the result of working with the system, and who is the system designed for? The latter seems particularly important for geovisualization.
In relation to these conceptual issues, a critical distinction is that between experts and novices and the associated differences in data exploration strategies that these two groups employ (McGuinness 1994). Different user groups are also likely to differ in the relative importance they place upon the simplicity, flexibility, and power of geovisualization tools. The past several decades of cartographic design research have focused on design of single "optimal" maps for a hypothetical "average map user." Geovisualization emphasizes the use of multiple maps (along with images, text, graphs, etc.) and is targeted (at least at present) to specialists.
Conceptual level goals can be met (or at least addressed) by subdividing them into a series of steps, each of which involves performing a particular operation or function on available information. The operation level of interface design involves the delineation of the appropriate operations to match conceptual level goals (e.g., for a goal of comparing two distributions, an overlay function may be considered useful). Decisions at this level are (or at least should be) independent of the particular hardware/software environment that might be used to implement the desired operations and, thus, should be made prior to consideration of the implementation level. In practice, however, tools available often put limits on the functions that may be specified. The distinction between the operation and implementation levels, then, is probably not as distinct as presented here.
Whether or not the operation and implementation level are distinct stages of interface design in practice, there are fundamental distinctions that are clearly made at an operation level. The most important is, perhaps, that between the "data manipulation" and the "abstract worlds" perspective on what the operations apply to (Gould 1993). With a data manipulation perspective, operations are treated as functions that can be performed on data (e.g., matrix addition, polygon overlay, etc.). The abstract worlds perspective, on the other hand, treats operations as applying to the world (e.g., compare education with income, determine a flood plane boundary, etc.).3
Beyond this distinction between operations directed to data versus to phenomena, it is useful to categorize operations in relation to the aspects of the information to which they are directed. The key distinction here is among operations directed to spatial, temporal, and attribute aspects of the data (or of the world). One of the most comprehensive attempts to delineate visualization operations is that reported by Keller and Keller (1992) and credited to Wehrend. More than 50 specific operations are described. Most of the operations specified are directly (or at least indirectly) applicable to geovisualization.
The implementation level includes consideration of anything that the user will have to see and decipher in order to interact with the system. This level also includes consideration of issues that underpin display generation, such as method of data storage and retrieval, choice of hardware/software platform, optimization of program routines, etc. These are issues that relate to the creation of the system as a practical reality based on the theoretical construct formed at the conceptual and operation levels.
Although there are many issues to be considered at the implementation level, the most important of them for a geovisualization user and the ones that cartographers may have the most to contribute to are those that deal with the appearance of the system. Among these considerations are the kinds of controls that a user is given to initiate various operations, the appearance of maps (color schemes, legends, etc.) and other representations that result from applying the operations, and the general appearance of the overall display (i.e., where to place controls and windows on the screen, how to maintain consistency in appearance and controls, etc.).
Among the most important choices at the implementation level is that of
interaction style. Typically, more than one interaction style will be incorporated
in a particular application. The critical decision for an interface designer
is which interaction style to match with each operation. Schneiderman (1987)
identifies five categories of interaction: menu selection, form fill-in, command
language, natural language and direct manipulation – each is briefly described
below.
• Command line interfaces (Figure 1) involve keyboard input of
commands and command parameters, either interactively or to be stored for
later use. This is a very powerful and flexible sort of interface but there
are several drawbacks. It requires that the user learn the language and it
generally can provide only limited protection against errors caused by mistyping
or incorrect command parameters.
• Natural language interfaces (Figure 2) also involve typed commands,
but now in the language that the user speaks and hears every day (such as
English). This makes the interface more comfortable for most users. Natural
language interfaces are likely to become more common as voice recognition
and speech synthesis hardware and software become more affordable. In spite
of the promise of such an interface style in general, however, natural language
is often strained when trying to discuss spatial concepts (which is why there
are maps in the first place) thus making it less than perfect for geovisualization. A
further issue, if the system will be used internationally, is that spatial
concepts differ among languages (Mark and Egenhofer 1995).
• Form fill-in presents the user with a set of fields to be filled
in (generally by typing responses from the keyboard) (Figure 3). Form fill-in
is effective for entering information into a geographic data base and for
retrieving information about specific locations. This interface style is not
well suited to exploratory analysis, except perhaps in limited situations
such as providing numerical bounds for data categories.
• A menu selection interface, (as the name implies), allows the user to interact with the system by selecting from menus or lists of possible actions (Figure 4). This approach reduces errors and helps remind the user what his/her options are at any point. Menu selection, however, reduces flexibility by limiting options to a small number of predefined choices. An additional problem is that long menus can be confusing for a user to peruse and the presence of multiple menus and submenus is a hindrance to an experienced user.
• With direct manipulation interfaces (as originally conceived)
the user, through a mouse or other pointing device, manipulates display elements
(e.g., buttons or icons) representing system operations. Direct manipulation
has progressed to a point where it is now possible to identify three sub-categories.
The first includes those controls that perform an operation or initiate a
change in the display when selected (by "clicking" on them). Such controls
are conceptually equivalent to menu selection, the difference is simply in
the details of how choices are accessed (and often in the use of verbal labels
on menu items versus graphic icons for selection "buttons"). 
A second sub-category of direct manipulation includes control devices that
allow a user to select parameters for an operation (often from a set of choices
too large to display as separate buttons or list in a menu). The color wheels
typical of graphic design packages and slider bars typical of scientific visualization
software are representative of this category (Figure 5). These controls often
allow selection along a continuous scale of real numbers (rather than limiting
choices to the discrete alternatives of menus and buttons). The third sub-category
of direct manipulation includes tools that allow a user to manipulate the
data display directly (e.g., Monmonier’s (1989) "geographic brushing"
or Buttenfield and Weber’s (1994) "proactive" graphics in which
symbols in an animated map act as "hot" links to text, static graphics,
or other animated views).
As noted above, operation level decisions are often difficult to separate from the implementation level. Whether a particular operation is practical or not is often a function of the hardware/software platform used for implementation. In practice, therefore, the operation and implementation level are typically addressed simultaneously. In general, any specific tasks conceived of at the operation level can be presented through any of the above interface styles. With today's visualization toolkits, we are not restricted to one style for all functions. An important job for the GVIS interface designer, therefore, is to choose the style for each operation that is easiest for the user to understand and that facilitates use with a minimum of error.
The remainder of the paper describes a prototype geovisualization directed to analysis of environmental change data. The particular goal for the system described is to facilitate an understanding of spatial variation in the reliability of environmental data (and in abstractions of those data). This understanding of reliability variation must, of course, be integrated with information gathered from exploration of the spatial data sets incorporated into the system. Following from the approach to interface design outlined above, discussion of our prototype geovisualization is organized into conceptual, operation, and implementation level issues. It should be noted, however, that much of the initial system was designed before we had worked out the details of the structured approach presented above. It was, in fact, the process of creating a prototype that provided the perspective from which our approach to geovisualization interface design evolved.
The prototype (named R-VIS, for Reliability VISualization) was initiated specifically as a tool for analyzing dissolved inorganic nitrogen in the Chesapeake Bay. The phenomenon of dissolved inorganic nitrogen, however, is representative of a range of environmental variables which are distributed continuously through space and time. As a result, the data quality issues, data processing techniques, and visualization methods incorporated in the prototype are relevant to a broader set of problems.
Each of the four conceptual level questions identified are addressed below as they relate to this primary application planned for the prototype.
• What need is (or should be) met by R-VIS? The general need that R-VIS is intended to address is the representation of spatially varied environmental data and reliability metadata that change in magnitude and distribution over time. Specifically, the goal is to examine spatial and temporal aspects of dissolved inorganic nitrogen in the Chesapeake Bay along with the reliability (uncertainty, validity, quality) of the original data and of derivations of those data used for analysis.
• How are the goals to be reached? The combined goals of examining spatial and temporal variation in data and reliability metadata can be reached (at least in part) by giving users a set of tools that allow data and metadata to be examined independently, simultaneously, and as an integrated concept.
• What should be the result of working with the application? Working with R-VIS should result in more robust hypotheses about the spatio-temporal variation in DIN. It should also foster more informed decision making in which analysts understand that the situation could be worse (or better) than the best guess perspective (as shown by a single map display) might indicate.
• Who are the anticipated users? There are two intended user groups. First are environmental scientists. They are expected to explore data in a search for spatial, temporal, and spatio-temporal patterns and relationships. For these users, reliability is important in judging whether patterns noticed are real. The second intended user group are policy analysts. For this group, reliability of information is critical to decision-making about whether or not to change current policies related to sources of DIN.
The conceptual level goals delineated above provide a general framework for system design. Matching this framework to characteristics of the available data leads to the second design stage, specification of operation level tasks that the system must perform. Before describing specific operation level tasks included in R-VIS, therefore, a brief account of the data sets made available by the Environmental Protection Agency (EPA) and the reliability estimates derived from them is needed.4
The dissolved inorganic nitrogen (DIN) data provided by the EPA for use in this project are typical real-world data, replete with sources of potential uncertainty. This uncertainty occurs in spite of a careful, well-conceived plan for data collection. The EPA-sponsored data collection effort was designed to measure changes in nitrogen levels from 1984 through the year 2000. The data provided for prototyping span a six year period. Data were (usually) collected at fixed sample points, once every two weeks in summer (April through November) and once per month in winter (December through March). Spatially, data consist of samples from only 49 points arranged somewhat irregularly across the Bay. Water samples were taken at two or four depths for each point. Particularly troublesome issues with these data are the Bay’s irregular shape, the uneven distribution of sample points, and the complete absence of samples in tributaries extending out from the core of the Bay. Estimates of DIN for large portions of the Bay, therefore, are based on extrapolation beyond the envelope defined by data collection locations, an inherently less reliable process than interpolation within this envelope.
With any multi-year study, various political and economic factors influence data quality over time. In this case, three different laboratories, using three different methods for measuring DIN were involved in the early years of the project. In addition, techniques used at each laboratory changed over the project’s initial three years. Standardization of techniques was not achieved until 1987. As the technology for measurement evolved, the threshold for DIN detection dropped (Chesapeake Bay Program 1992). What appears to be a decline in nitrogen levels, then, may simply be a function of more sensitive analytical techniques. Even if detection thresholds had remained constant, uncertainty exists for any below-detection measurements. Should they be treated as zero, as equal to the threshold, or as some intermediate value? Initial data analysis by the EPA contractors showed limited variation in a volumetric interpolation of a three-dimensional grid. As a result, the below-threshold values were set to 1/2 the detection limit (Chesapeake Bay Program 1992). The CBP report does not, however, consider temporal implications of this choice (i.e., if DIN drops over time, from current values that are typically above 1.0 mg/l, to a level near the target of 0.15 mg/l, then the value assigned to measurements below the detection limit becomes more relevant).
Data represent a spatially continuous surface (or volume). This surface/volume can be modeled by interpolating from the 49 samples (taken at two or four depths) to create estimates for locations on a surface at any level in the Bay or for locations in three-dimensions within the Bay. In the R-VIS prototype described here, we have initially limited analysis to a single layer (the surface), through one year of data although it is a goal to expand the coverage both depthwise and temporally.5
For the current prototype, raw data have not been aggregated and measurements of DIN were provided to a precision of +/- 0.01 mg/l (with typical measurements over 1.0 mg/l). Precision of attribute information, therefore, was not an issue. Although accuracy of attribute measurements at sample sites has varied somewhat over time due to changes in analytical procedures, this variation is also minimal compared to variation in accuracy of DIN estimates for locations intermediate to sample locations. Thus, we focus our attention on spatial and temporal variation in reliability of these estimated values.
Reliability of intermediate value estimates will be a function of the interpolation method used, the density of sample points, where those points are located, and the variability of the surface sampled (plus the accuracy of initial measurements) (MacEachren and Davidson 1987). For any given interpolation method, cross-validation (also known as jackknifing) provides a way to assess reliability of interpolated values (Clark 1979). This technique involves comparison of known values to estimated values at those same locations. We selected four cruises (data collection trips) for one year (at quarterly intervals) and calculated a jackknife estimate for each of the 49 sample locations on each cruise. Jackknifing provides a reliability gauge for interpolated values, a way to assess likely changes in reliability over time, and a tool for comparing interpolation techniques. The ‘errors’ it measures, however, do not in fact exist. They are estimates of what would happen if individual data values were missing from the interpolation process (which happened occasionally).
To obtain a true spatial estimate of reliability, a method of determining confidence limits around estimated values is needed. Kriging, in addition to providing the "best linear unbiased estimate" of values for each grid point, results in a confidence matrix for the values calculated. In effect, it takes into account the combined influence of sample point location, sample point density, and data surface variability. We calculated kriged surfaces and their associated variance matrices for cruises in our sample year.
Based on what R-VIS is for, the kinds of data and reliability estimates available, and who the users might be, several specific operations were required.
• Representation of data for specific times. Whether the user is a scientist or policy analyst, he/she will require access to representations depicting estimates of DIN across the Bay at particular times. Operations that allow a set of these time slices (what Monmonier (1990) labeled "chess" maps) to be viewed in sequence as well as the ability to compare time slices (e.g., May DIN levels in successive years) are needed. Since there might be seasonal or annual cycles, we also wanted the user to be able to choose any data surface by date and aggregate information for the same point in the cycle across years.
• Representation of reliability (metadata) for specific times. As noted above, we have estimated two kinds of reliability for this data set. The first is determined by jackknifing and provides a mean reliability estimate for the entire map. Jackknifing also results in an estimate of the reliability with which missing data can be approximated at each sample point. In contrast to jackknifing, kriging generates a reliability surface in the form of confidence limits (associated with the model fit) for each grid cell calculated. These confidence limits are a measure of likelihood that a value is within some interval of the interpolated value. An additional goal is to assess and represent temporal aspects of reliability associated with factors of data collection (such as the number of days it took to collect each sample). Although we limited ourselves to only two types of reliability estimation in the construction of R-VIS, many more estimates could clearly be usefully derived and made available for exploration.6 The tools that we have implemented allow for the display of both point and area measures of reliability and could thus be used for the representation of many other data reliability measures.
• Comparison of data and metadata for specific times. Three methods of comparison are desirable. One is to examine data and metadata in sequence (perhaps repeatedly). A second is paired comparison (through side-by-side maps). A third alternative involves generating composite maps of data and metadata superimposed or merged at the same display location. Methods provided should facilitate the analysis of data and metadata individually along with the analysis of any systematic relationships between the two.
• Representing change in data and metadata over time. Change can be represented through comparison of time slices (see above). In addition to comparing two (or a small number of) time slices (or aggregates), it should be possible for the user to examine the process of change across time via animated sequences of maps. Also, in contrast to emphasizing time slices or aggregates, maps that depict change directly can be produced in which the difference between two time periods (rather than either individual time period) is depicted.
• Representing differences in data and metadata with depth. DIN concentrations (and reliability of DIN estimates) are likely to vary with depth in the Bay. In addition to allowing users to access representations that depict information at the surface, therefore, the system should allow an analyst to explore both data and metadata at different depths.
• Representing subsets of data and metadata. To examine both data and metadata, operations that highlight particular data values or ranges are needed. For the data, one important value is the target DIN level of 0.15 mg/l. Highlighting locations that either meet or exceed this target would obviously be a useful analytical tool. For reliability, a threshold might be set, below which data are not considered reliable enough to use with confidence. A method for de-emphasizing (or entirely omitting) these locations from a data representation is desirable.
• Representing the impact of interpolation algorithms and their parameters. R-VIS is directed to assessing the reliability of estimates intermediate to (and beyond) locations at which measurements are taken. This reliability will be, in part, a function of the interpolation method applied for generating a model of a surface (or volume) based on a limited set of irregularly distributed samples. In addition to using R-VIS for exploring the distribution of data and metadata, therefore, it should also be possible to compare results derived from different interpolation methods.
• Enhancing spatial patterns in data and metadata. In geographic visualization, there are often multiple variables being explored and relationships between variables are being sought. In R-VIS, these variables are the DIN data and the reliability of that data. Tools are needed that enhance the ability of R-VIS users to find patterns in the variables individually as well as relationships between the data and metadata. These tools need to be more sophisticated than merely side-by-side display of the two data sets. Although side-by-side display allows the easy comparison of distributions, tools are needed that both focus in on specific ranges of the data/metadata and filter out local extremes in the data/metadata to clarify overall patterns.
The operation level deals with matching conceptual level goals to specific characteristics of available information. Similarly, the implementation level links the desired operation level tasks with specific features of available software/hardware tools. For R-VIS, the software development environment selected is the Interactive Data Language (IDL).7 One reason for choosing IDL is that it is available for a range of hardware platforms from Cray supercomputers to desktop Macintoshes. For the project reported here, development was carried out on workstations in a UNIX environment.
IDL allows the designer to build an interface by combining and modifying tools called ‘widgets’ (IMSL, 1992). A widget can be a simple or complex graphical object (ranging from a push-button to a display window containing a map). Interaction with a given widget produces what is referred to as an "event" from that widget. When the user generates an event (by pushing a button, moving a slider, etc.), the software is then able to respond to the event by performing some function.
Six different types of widget are available in IDL. Two are non-manipulable and deal with display of information on the screen:
Base widgets: allow the designer to specify a base (including its position on the screen, color, etc.) upon which other widgets will appear.
Drawing widgets: allow the user to display graphics. They must be rectangular but can be of any size.
The remaining four kinds of widget provide the controls through which users interact with the information displays. These widgets can be matched with several of the interface styles discussed above:
List widgets: consist of either pull-down or pop-up menus that allow choices of actions from a list.
Text widgets: a version of form fill-in that provides for the entry or display of text (or numerical information).
Button widgets: a form of direct manipulation in which some action will be taken if a user clicks on them. Buttons are visible and labeled button-shaped objects appearing in the margins of the display. Although not a built-in feature of IDL it is possible to simulate transparent "virtual" buttons that result in "hot" locations within the information display itself.
Slider widgets: an alternative form of direct manipulation in which a user moves a pointer back and forth on a sliding scale that is linked to a program function.
We made a decision early in the project that our default display would
contain two map windows with a centrally located control panel (Figure 6 - click
on figure for larger version)
.
This choice was due, in part, to the shape of the Chesapeake Bay in relation
to a typical display screen – thus making map pairs possible with no decrease
in scale (over a single map). In addition, the high priority placed on representation
of reliability suggested that side-by-side views of data and reliability might
be frequently consulted and would be the preferable starting point for analysis.
Our user interface went through many revisions as modules were added to the system and the interface continues to be revised. The procedure followed in building the interface has been one of matching the available widgets to desired operations and then testing these widgets to insure that they generate the correct events. To simplify description of the specific functions implemented in R-VIS, we organize our discussion on the basis of operational level tasks that have been implemented. It should be noted, however, that actual design of R-VIS was not as systematic as this presentation implies.
Since our default display includes representations of data and reliability
in adjacent windows, tools provided for accessing data and reliability at particular
times are linked (it would make little sense to view data for one time and reliability
for another). The primary tool for selecting specific times is a slider bar
(Figure 7)
.
The slider bar allows the user to "page" forward or backward through
the surfaces in chronological order (by clicking on an arrow) or to select any
particular time slice desired (by sliding the control forward or backward until
the appropriate label appears above the slider). Although the prototype displays
these labels as index numbers, an operational tool would display actual dates.
Linking the slider to a time line would also facilitate its use.
In addition to implementing a tool for dynamically selecting time slices for display, several decisions had to be made about the representation of individual time slices and about additional controls applied to these representations. One consideration that affected all others was map scale. There were many constraints here, including screen size, desired resolution, and the size of array needed to define the map. Since IDL is primarily a raster system, map scale and resolution of display must be considered in conjunction. There is a trade-off between a desire by analysts for spatial precision in the maps and the resolution at which the computer system can support interactive analysis. In addition, small cells would imply higher interpolation accuracy than the sparse 49 point sample network can achieve. We opted for four square kilometer cells, resulting in a 90-by-172 array for each map. On our display device, this translated into twenty-five screen pixels per cell. These cells can be seen individually on the display, yet are not so distinct as to produce a disjointed or blocky look. The resolution is only one-half that used by the EPA contractors in previous analysis of the same data. The lower resolution is a better match to the sample network (of forty-nine sites to cover Chesapeake Bay). It helps to suggest the uncertainty inherent in interpolation from such a sparse sample.
In relation to symbolization methods, color schemes were given particular attention as we implemented R-VIS. IDL allows colors to be defined in relation to several color models including RGB and HLS. Because we wanted to discuss various display schemes in terms we, and most cartographers, were familiar with, we adopted the hue/lightness/saturation system. Brewer’s (1994) color syntactics for univariate and bivariate mapping played a key role in designing color schemes to match operation level goals.
Following MacEachren's (1992) proposals about uncertainty representation we experimented with color saturation as a way to depict uncertainty (reliability). We initially depicted reliability in a range from saturated to unsaturated red (representing reliable to unreliable data). This was matched in the data depiction with a range in lightness (light red for low values and dark for high). These schemes did not provide enough contrast for the data sets we tried, with the saturation range being particularly ineffective. As an alternative, identical value/saturation scales using different hues for data and reliability metadata proved to be more successful at highlighting extremes in both. For data, the scheme runs from a dark relatively unsaturated red for high values through a lighter more saturated red in the middle range to a light desaturated red for low values (see left side map in Figure 6). The reliability depiction uses a similar range of blues (see right-hand map in Figure 6). When applied to side-by-side map pairs, this dual-hue depiction makes interpretation easy for the user because the schemes do not require switching "schemata" when looking from one map to another.8 Once the user has seen that dark=more and light=less on the data map, he will not have to switch this interpretation when looking at the metadata surface.
Intuition suggests that high DIN values, being those that exceed tolerances by the largest margin, should be highlighted, and thus represented by the most noticeable color. Given the light-gray background that we chose, we decided to follow the convention of dark=more, selecting a menacing dark red for high DIN values. Lower values are shown in lighter colors. We chose a light gray background because the extremes of a black or white background were harsh and overwhelming. Light rather than dark gray was selected, in part, due to McGranaghan’s (1989) evidence that using lighter colors for backgrounds lowers the chance that the map reader will become confused about which end of the color value sequence represents high data values.
In addition to representing data and reliability surfaces, a method was also needed to display the results of jackknifing at sample points. Although the sample points are actually the most certain in any exact interpolation, depicting the potential error that would occur if samples were missing gives an overview of the relative magnitude and distribution of uncertainty. Since there are samples missing from some cruises, the procedure also allows the analyst to gauge how significant these missing data will be. Displays generated illustrate the relative dominance of specific sample sites in areas of sparse data. When examined over time, this point-based depiction of potential uncertainty illustrates that the potential for error varies in space-time, not just in space.
The representation technique devised to depict jackknifing uses a dynamically
controlled graduated point symbol located at each sample station. Initially,
the analyst sees narrow vertical "bars" scaled by height proportional
to the potential error at the sample location 
(Figure
8a). These bars are really triangles with a narrow initial base that is dynamically
controlled by a slider widget. With this control, the user can hide symbols
entirely or turn up the "visual magnitude" (Figure 8b). As the base
expands there are three clues to the size of the potential error. One is the
height (which is always present), the second is triangle area, and the third
is the "pointedness" of the triangle. An isosceles triangle was chosen
as the symbol because its height and area are linearly proportional (so both
visual cues represent data in the same way) and the sharp points of high-error
locations should make them stand out. The interactive nature of the variable/magnitude
graduated symbols seems particularly suited to metadata representation. The
analyst can bring the metadata to the fore by increasing symbol size, then de-emphasize
the metadata by reducing triangle width back to a point at which height remains
the sole cue, or by turning the triangles off entirely.
Information about data reliability is intended to provide the analyst with a way to gauge the confidence that should be attached to any features or patterns noticed in map representations. To be useful, this information must be presented in conjunction with the data to which it applies. MacEachren (1992) suggests that at least three possible display strategies are available: paired comparison (i.e., side-by-side maps), composite maps in which data and metadata are combined, and alternation (or toggling) between data and metadata representations (if the tool for display is dynamic). Each has several variations, some of which we consider below.
Map pairs were selected as the default presentation form. Map pairs make it easy for an analyst to examine data and reliability independently. We can generally assume that most analysts will want to consider the data independently before taking data reliability into account, and that they will want to assess reliability before looking for relationships between data and reliability.
After viewing the initial side-by-side display, the analyst can select among various options for overlay and merging of data and reliability maps. We implement the overlay technique in a manner similar to that presented by DiBiase, et al., (1994) for display of multivariate information. One variable (the data) is depicted with area shading and the second (the metadata) is depicted with line or point symbols. For depicting confidence values derived by kriging, our overlay uses weighted isolines – where the isolines are thicker for higher uncertainty (DiBiase, et al. 1993). The second kind of overlay uses scaleable triangles representing the potential errors at sample points when/if those sample points are missing (see above for a description of these dynamically manipulable point symbols). Both of these are tools that can be turned on and off within R-VIS.
Overlay methods emphasize the data while allowing analysts to check the reliability in map areas that seem to be particularly good or bad in terms of meeting the EPA dissolved inorganic nitrogen targets. With overlay, users are able to look past either data or metadata to interpret the other.
A merged display style, in contrast to overlay, uses bivariate maps to put emphasis on relationships between data and reliability. With a merged display, data and reliability estimates are visually integrated in such a way that it becomes difficult to pay attention to either independently. On these maps, a unique symbol (color) is used for each data reliability category. As a result, a bivariate map would make it possible to discover a data-metadata relationship, such as one in which areas that appear to be meeting standards also have high probabilities of being unreliable.
For bivariate maps in R-VIS, users can select among several color schemes
that focus attention on different aspects of the data and metadata. Selection
is achieved through a pop-up menu. One available scheme suggests uncertainty
about data by matching uncertainty with a range in color saturation and data
with a range in color lightness (value) (Figure 9 - click on figure to see larger
version)
.9
With this bivariate map, data in areas having clearly identifiable colors can
be relied upon and data in areas with grayish, "uncertain" colors
can not.
A second color scheme is designed to emphasize either positive or negative correlation between data and uncertainty estimates. Palettes that achieve this goal share the feature of merging sequential value ranges of two hues, selected so that their combination produces a logical sequence along the diagonal of the 3-x-3 legend (e.g., grays of increasing darkness). Based upon Brewer’s (1994) recommendations we used red and blue as the two hues in this scheme. The user can invert the order of the color sequence matched to the data in order to search for negative correlation between data and metadata.
For applications in which it makes sense to treat uncertainty as a binary choice (i.e., low enough to ignore or high enough to worry about), a scheme is provided that emphasizes the extremes of these reliability categories as qualitatively different. The three classes are represented by a hue difference (e.g., high certainty is shown in blue, low certainty in red, and intermediate certainty in gray).
Perhaps
the most effective representation scheme developed thus far is one that does
not rely exclusively on color. This scheme evolved from discussions about how
the graphic variable of "clarity" might be implemented in R-VIS (see
MacEachren 1995 for a discussion of clarity). In essence, the goal was to make
reliable data clearly visible and unreliable data hard to see. In practice,
implementation of this idea resulted in a bivariate map in which a lightness/saturation
range (described above) is used to depict data and symbol size is used to depict
reliability. Larger symbols (completely filled grid cells of the raster map)
represent the most reliable data, symbols of intermediate size (filling 9/25ths
of the cell) represent data of intermediate reliability, and very small symbols
(filling 1/25th of the cell) represent unreliable data (Figure 10). An added
factor in favor of this representation form is that it works in black and white
as well as color.
The final method for comparing data and reliability implemented in R-VIS thus far is the use of dynamic alternation. Alternation is achieved in two ways, interactively and through animation. Interactive alternation simply allows the user to toggle back and forth between a data and a reliability display registered to the same screen location. Animation performs this alternation automatically, multiple times. It is with multiple viewing that relationships between data and reliability distributions (if there are any) begin to become apparent.
At this stage of R-VIS development, two methods for examining change over time have been implemented (and both can be applied to either data or metadata). The first, is simply to use the time slice slider to select times in succession. Clicking on and holding the arrow of the slider results in continuous stepping through the available times. The alternative to this frame by frame presentation is to animate a temporal sequence. This module runs the sequence at a rate sufficient to give an impression of continuous change. With animation, emphasis is placed on the process of change rather than the individual states in time.
There have been specific events during the span of the CBP data collection and analysis that lead to uncertainty in the data. These include seasonal changes in frequency of sampling, changes in length of specific cruises, and "instantaneous" changes in analytical method. Each can be signified by visually noticeable changes in representations. Abrupt switches in analytical methods are best represented with a flashing signal or some other sort of flag (e.g., a sonic alarm). These relatively significant, but short duration, events would go unnoticed if we merely adjusted a static variable (e.g., saturation of hues) that was being used to depict DIN concentrations.
An issue with animated presentation of the DIN data for Chesapeake Bay is the systematic change in temporal resolution during each year (with sampling twice as often in summer as winter). If all available data are presented in the animation, potential confusion can result about the rate of change across the year (with changes seeming quicker in winter where samples are twice as far apart in time). One solution is to adjust the tempo of the animation so that the time slices with greater temporal spacing have a greater duration on the screen. The resulting animation will differ in smoothness. This difference will reflect changes in temporal resolution, but might be misinterpreted as differences in rate of change. Another way to deal with the issue of varying temporal resolution is to present only a subset of the data (one value per month) so that the time space between frames is equal. The drawback here, of course, is loss of what might be significant detail. A third alternative is to interpolate between times during the winter in order to generate a consistent two week temporal spacing. In this case, the animation will be smooth, rates of change can be correctly interpreted, but reliability of the depiction will vary between those times slices that were measured and those that were estimated.
We chose a variation of the second alternative in which we averaged the summer biweekly samples to produce one monthly surface. To distinguish between monthly samples (having zero variance) and monthly means of biweekly samples we have experimented with resolution in our coastline generalization (matching fine detail with zero variance and a coarse grid generalization with aggregated data). This resolution-based technique is also suited to cueing analysts to changing temporal precision (due to varied cruise length) during each year.
In addition to directly representing temporal change in reliability, time series animations of reliability maps can be used to examine space-time patterns. By "playing" pairs of side-by-side data/metadata maps, the analyst can begin to explore whether or not the seasonal cycle identified in DIN concentration is mirrored by a similar cycle in uncertainty, and if either or both change their spatial distribution appreciably over time (as a result of a new point pollution source).
A dynamic environment such as IDL provides the potential to apply the exploratory data analysis technique of focusing to the analysis. Focusing refers to dynamic interaction with data in which an analyst can highlight a specific subsection of a larger data set for more detailed analysis (Buja, McDonald et al. 1991).
In
R-VIS, slider bars are provided that allow the analyst to set thresholds for
the lowest DIN concentration to be represented and for the lowest uncertainty
(i.e., highest reliability) (Figure 11 - click on figure for larger
version). The analyst could, for example, set the data display to depict
only those locations for which the DIN concentration was greater than or equal
to twice the year 2000 target concentration and set the uncertainty tolerance
so that only those locations that were less certain than +/- one half of the
target goal were highlighted. The most important feature of the focusing tool
is that it is dynamic. As the slider is moved the map changes, so that finding
reliability patterns, data patterns, or their relationships is not dependent
upon some preconceived threshold values (such as those just offered as examples).
R-VIS is an ongoing project. At present, two operation level tasks have yet to be matched with implementation level tools. In addition, several modifications and extensions of tools already implemented are being considered. In this section, we discuss future plans for R-VIS and methods being considered for assessing R-VIS usefulness for the target users. We end with some brief comments on the potential impacts of geovisualization on cartography and on the role of geovisualization in science and decision-making.
R-VIS is undergoing constant modifications and may never be a finished product. IDL has capabilities that we still do not appreciate and new ideas are suggested regularly. There are several sets of tools that still need to be added to meet the goals determined at the operation level.
Multivariate pattern identification seems to be an ideal application for bivariate maps. R-VIS already has the ability to generate bivariate maps. However, two ideas that apply to bivariate pattern identification have not yet been implemented.
Carstensen (1986) pointed out that axis scaling for bivariate maps significantly influences the ability of map users to relate the two variables being shown. Although Carstensen was working in the context of static maps (in which a single choice must be made about how axes should be scaled), dynamic alteration of data classification and symbolization (color to data assignments) is suggested by his ideas.
A second idea is presented by Eyton (1984) with his alternative bivariate classification scheme. Eyton’s idea is particularly appealing for geovisualization because, like Carstensen’s axis scaling, it seems ideal for interactive exploration of geographic distributions. He suggests a bivariate scheme which has only four categories created by breaking each of the variables into two classes at their mean. For the bivariate categories in which the two variables are similar (e.g. both variables are below the mean), white and black are used. For the mismatched categories (i. e., one high and one low class), two complementary colors are suggested. The intriguing idea is that the middle 50% of the values from each data range are collected into a fifth class which is symbolized by a medium value gray (Eyton 1984). This has the effect of focusing the analysts’ attention on the extreme data values rather than the "average" ones. In an interactive environment, of course, the proportion of data values contained in the fifth class could be dynamically manipulated.
R-VIS users will also be searching for patterns within data and reliability metadata individually. Univariate pattern identification can be facilitated through application of "filtering." Filtering involves smoothing out local extremes in a data surface. Visually, filtering has the effect of reducing the contrast in an image. From a data analysis viewpoint, the utility of such a tool is that it shifts the emphasis of a map from local, usually short-term features to longer-term trends that occur over larger areas. These larger trends are more likely to be important to analysts, especially within the context of exploring environmental data that is gathered over many years, as is the case with the DIN data used in the prototype. Filtering could also be applied in a temporal sense by averaging "adjacent" cruises (as was done for the summer months, although not with this goal in mind) or by averaging data from similar dates over several years. Success in spatio-temporal filtering has been demonstrated by Dorling and Openshaw (1992) in an application built to explore data about childhood leukemia. They report that filtering helped them to explore the processes that influenced the incidence of the disease and to identify disease "hot spots".
Thus far there has been no attempt within R-VIS to incorporate the data recorded at depths other than the surface. This decision was made early in the conceptual stage to make design of the prototype practical within constraints imposed by available software and hardware. A couple of possibilities exist for the inclusion of the third dimension into R-VIS. The most obvious is to follow the lead of the original Chesapeake Bay Project (1992) and slice the Bay into horizontal sheets. The CBP interpolated vertically at each of the data collection sites and then used horizontal interpolation to fill in values for each of the depth slices. This method could be made more robust by applying a three-dimensional interpolation algorithm.
One example of the application of three-dimensional interpolation is given by Mitasova et al. (1993). They were working with the same Chesapeake Bay data and they used dynamic isosurfaces to represent depth. The isosurfaces were also flexible in that any variable (depth, data reliability, etc.) could be mapped to the vertical dimension. The version of IDL that was used to create our prototype had the ability to drape surfaces over terrain representations but did not have the ability to perform true three-dimensional operations such as three-dimensional interpolation.10
We are currently exploring alternative ways to include the depth dimension in the next version of R-VIS so that the user can explore patterns not only at the surface, but throughout the volume of the Chesapeake Bay.
Although there are many sources of uncertainty, we have highlighted that due to interpolation from a sparse set of sample points. Thus far, we have relied upon confidence matrices generated by kriging as an estimate of reliability. An alternative measure of the reliability of interpolated values is their robustness through several different interpolation procedures. In the future, R-VIS users will be able to compare the surfaces calculated by several different algorithms. One way of accomplishing this comparison is by toggling or flickering back and forth between several interpolated surfaces. This would allow the analyst to check for "map stability." Areas that did not change noticeably between different methods of calculating the intermediate values would be deemed highly reliable whereas areas whose values were very dependent on the interpolation method used would be significantly less reliable. This tool for examining pattern "robustness" would allow the user to reap the benefits of dynamic displays and eliminates some of the concerns raised in the past about single-map (or "best-map") solutions (Monmonier 1991).
IDL was not originally conceived of as a language for geographic visualization. However, it was meant for people who were trying to manipulate numerical data sets (hence the name Interactive Data Language). Therefore, there are many graphing functions readily available within the language. Future versions of R-VIS will take advantage of these capabilities by letting the user ‘link’ to graphs as well as other non-map graphic elements during the data exploration process.
To date, there has been no assessment of R-VIS by users. This raises the question of appropriate methods to assess a geographic visualization system. geovisualization is intended for use by experts in a private interactive way to solve problems or generate hypotheses. Obviously, this seems to rule out the common cartographic testing method where large numbers of relatively novice subjects are gathered and asked to complete simple tasks to evaluate the effectiveness of a map. Instead, testing methods involving fewer subjects interacting more intensively with the application seem to be required. Experiments in which "experts" use the geovisualization to solve realistic problems or to generate hypotheses about the data set being examined are one possibility. For R-VIS these experts could be scientists or policy analysts who are familiar with the Chesapeake Bay Project or visualization specialists experienced in environmental science applications.
When working with expert users, various alternatives to task performance measures should be considered. One is the use of focus groups. Focus group interviews can reveal a range and depth of information that is not possible by having subjects perform a set of tasks. These ideas may even branch into general principles applicable to geographic visualization in general rather than emphasizing requirements of a single application. Much of this depth and breadth of insight can be attributed to the interaction of the members of the focus group under the direction of a trained moderator. This interaction is something that individuals completing tasks independently are not able to benefit from. Focus groups have been used in cartography to increase insight into the effectiveness of dynamic maps (Monmonier and Gluck 1994). Although Monmonier and Gluck felt that the focus groups they used to obtain reactions to dynamic maps were successful, they list some limitations of focus groups:
Despite these limitations, Monmonier and Gluck recommend focus groups as a way to obtain preliminary feedback on dynamic maps. This method has one additional drawback when being applied to visualization assessment. Either all of the subjects would have to have a chance to use the system extensively before the focus group meeting or they would only be exposed to a walk-through of the system’s capabilities, an alternative that does not accurately depict the environment that visualization takes place in.
Another method that can be combined with the expert problem-solving/hypothesis-generating approach outlined above, and serve as a complement to focus groups, is protocol analysis. Protocol analysis offers a method to examine the knowledge structures, problem solving strategies and tactics of subjects as they complete tasks (McGuinness 1994). Protocol analysis is carried out by having the subjects "think aloud" as they solve problems. The problems being attempted are usually realistic problems that would actually be faced in a real-world application of the system. The thoughts are recorded and later coded into categories that reveal the kind of thinking that is taking place Kulhavy, et al. 1992). This method allows exploration of the mental processes and strategies being applied during visualization without compromising the private exploratory nature of it.
Another method of accessing the strategies employed by visualization users is that demonstrated by McGuinness and Ross (1995). They allowed subjects to use a GIS with several different display options in order to explore problem scenarios. After the subjects were finished interacting with the system, McGuinness and Ross had them answer a set of questions about their strategies for exploring the databases. The subjects were given a choice of strategies: random search, systematic search, focused/selective search, combination of these, and none of these. Finally the subjects were asked to justify their approach. This "cognitive interview" technique seems to be a useful approach, perhaps especially when combined with protocol analysis. The combination would provide researchers with both a measure of the strategies that were actually employed as well as the strategy that was being attempted.
We are nearing the end of the first decade since the landmark report on Visualization in Scientific Computing (McCormick et al. 1987). Mapping methods developed in the eighteenth, nineteenth, and early twentieth centuries (together with methods of graphing and scientific illustration) are recognized by computer scientists as the source of all visual representation techniques now part of "visualization in scientific computing" (VISC). Computers have facilitated the use of "mapping" as a method for exploring data in many branches of science, not just those dealing with geo-referenced data (Hall 1992). Dramatic changes are under way in the nature of maps and related geographic representations and in concomitant uses of these tools as facilitators of geographic thinking.
One of the most sweeping changes in the last decade (or longer) is the advent of the World Wide Web (WWW). Users are now much more attuned to the idea of interactivity. Now that the average scientific computer user has experienced hypertext links and WWW search engines, they expect to be able to manipulate the program they are using so that they can see what they want. Another facet of this change is that computer users are more sophisticated in terms of the interface styles that they understand. However, both of these observations need to be weighed against the still sizable minority who are not comfortable with computers and thus need simple, friendly interfaces. For those with access, the WWW allows people to explore graphic as well as textual information from sites around the world simply, and relatively quickly. As we are making our final revisions to this paper, the Deasy GeoGraphics Laboratory at Penn State is in the final stages of developing a WWW-accessible interactive atlas/GIS of Pennsylvania. The application, undertaken for the State of Pennsylvania, is written in Java™ and will allow users not only to access maps and data through the WWW, but also to perform basic overlay operations in real-time. This project might be considered the first instance of real-time geographic visualization via the Web.
The Internet is but one part of the emerging pattern of increased computer literacy and availability for many people. This link between technology and its uses has the potential to foster links between streams of cartographic research that have been on divergent paths since the mid-1970s (i.e., cognitive and analytical/computer cartography) (Taylor 1994).
R-VIS is an attempt to use knowledge gained from both of these paths in order to create a system that can be used by people who are not geographers but who are studying geographic distributions. This is a new era for cartographers, who are comfortable designing maps but have less experience designing user interfaces and computer information systems. Much can be learned from those who have more experience: computer programmers, human-factors engineers and GIS designers, but cartographers also need to contribute our expertise with the display of geo-referenced data.11 If we are active in this process of designing tools for analysts to use in decision-support we will have an impact on the decisions being made and the basis for those decisions. Environmental study is a perfect example of a field where geographic visualization (and GIS) can be powerful tools because of the spatial and temporal nature of the data. If we do nothing, then people without our training will build the tools that we are most qualified to construct and decisions that could have benefited from our expertise will be made without our input.
Brewer, Cynthia A. 1994. Color Use Guidelines for Mapping and Visualization. In MacEachren, A. M. and D. R. F. Taylor eds. 1994. Visualization in Modern Cartography 123-148. Oxford, United Kingdom: Elsevier.
Brown, A. and C. P. J. M. van Elzakker. 1993. The Use of Colour in the Cartographic Representation of Information Quality Generated by a GIS. Proceedings, 16th Conference of the International Cartographic Association 707-720. Cologne, Germany, May 3-9.
Buja, A., J. A. McDonald, J. Michalak, and W. Stuetzle. 1991. Interactive Data Visualization Using Focusing and Linking. Proceedings, Visualization ‘91, IEEE Conference on Visualization 156-163. San Diego, CA.
Beard, M. K. and B. P. Buttenfield. 1991. NCGIA Research Initiative 7: Visualization of Spatial Data Quality. NCGIA.
Buttenfield, B. P. and M. K. Beard. 1994. Graphical and Geographical Components of Data Quality. In Hearnshaw, H. and D. Unwin eds. 1994 Visualization in Geographic Information Systems 150-157. London, United Kingdom: John Wiley & Sons.
Buttenfield, B. P. and C. R. Weber. 1994. Proactive Graphics for Exploratory Visualization of Biogeographical Data. Cartographic Perspectives 19: 8-18.
Carstensen, Laurence W. 1986. Bivariate Choropleth Mapping: The Effects of Axis Scaling. The American Cartographer 13(1): 27-42.
Chesapeake Bay Program. 1992. Trends in Nitrogen in the Chesapeake Bay (1984-1990), Produced under contract to the U. S. EPA. Contract No. 68-WO-0043, CBP/TRS: (68/92).
Clark, I. 1979. Practical Geostatistics. London: Applied Science Publishers.
Collins, B. M. 1993. Data Visualization – Has it All Been Done Before? In Watson, R. A. E. ed. 1993. Animation and Scientific Visualization: Tools and Applications 3-28. New York: Academic Press.
Cornwell, B. and A. H. Robinson. 1966. Possibilities for Computer Animated Films in Cartography. Cartographic Journal 3: 79-82.
Cressie, N. 1991. Statistics for Spatial Data. New York: Wiley.
DiBiase, D., T. Paradis, and J. L. Sloan II. 1993. Weighted Isolines: An Alternative Method for Depicting Statistical Surfaces. Professional Geographer 46(2): 218-228.
DiBiase, D. C. Reeves, A. M. MacEachren, M. von Wyss, J. B. Krygier, J. L. Sloan, and M. C. Detweiler. 1994. Multivariate Display of Geographic Data: Applications in Earth System Science. In MacEachren, A. M. and D. R. F. Taylor eds. 1994. Visualization in Modern Cartography 287-312. Oxford, United Kingdom: Elsevier.
Dorling, D. and S. Openshaw. 1992. Using Computer Animation to Visualize Space-Time Patterns. Environment and Planning B: Planning and Design 19: 639-650.
Eyton, J. R. 1984. Complementary-Color Two-Variable Maps. Annals of the AAG 74(3): 477-490.
Foley, J. D., A. van Dam, S. K. Feiner, and J. F. Hughes. 1990. Computer Graphics: Principles and Practice. Reading: Addison-Wesley.
Frank, A. U. 1993. The Use of Geographical Information System: The User Interface Is the System. In Medyckyj-Scott, D. and H. Hearnshaw eds. 1993. Human Factors in Geographical Information Systems 3-14. London, United Kingdom: Belhaven Press.
Gould, M. D. 1993. Two Views of the User Interface. In Medyckyj-Scott, D. and H. Hearnshaw eds. 1993. Human Factors in Geographical Information Systems 101-110. London, United Kingdom: Belhaven Press.
Hall, S. S. 1992. Mapping the Next Millennium: The Discovery of New Geographies. New York: Random House.
Heuvelink, G. B. M. 1993. Error Propagation in Quantitative Spatial Modelling: Applications in geographical information systems. Utrecht, Faculteit Ruimtelijke Wetenschappen Universiteit Utrecht.
Hootsmans, R. 1996. Fuzzy Sets and Series Analysis for Visual Decision Support in Spatial Data Exploration. Utrecht, Faculteit Ruimtelijke Wetenschappen Universiteit Utrecht.
IMSL, Inc. 1992. IMSL/IDL User's Guide. Sugar Land, TX: IMSL, Inc.
Keller, P. and M. Keller. 1992. Visual Cues: Practical Data Visualization. Los Alamitos, CA: IEEE Computer Society Press and Piscataway, NJ: IEEE Press.
Kulhavy, R. W., D. R. Pridemore, and W. A. Stock. 1992. Cartographic Experience and Thinking Aloud about Thematic Maps. Cartographica 29(1): 1-9.
Lindholm, M. & T. Sarjakoski. 1994. Designing a Visualization Interface. In MacEachren, A. M. and D. R. F. Taylor eds. 1994. Visualization in Modern Cartography 167-184. Oxford, United Kingdom: Elsevier.
MacEachren, A. M. 1987. The Evolution of Computer Mapping and Its Implications for Geography. Journal of Geography 86(3): 100-108.
MacEachren, Alan M. 1992. Visualizing Uncertain Information. Cartographic Perspectives 13: 10-19.
MacEachren, Alan M. 1994. In MacEachren, A. M. and D. R. F. Taylor eds. 1994. Visualization in Modern Cartography 1-12. Oxford, United Kingdom: Elsevier.
MacEachren, Alan M. 1995. How Maps Work: Issues in Representation and Design. Guilford Press.
MacEachren, A. M. and J. V. Davidson. 1987. Sampling and Isometric Mapping of Continuous Geographic Surfaces. The American Cartographer 14(4): 299-320.
MacEachren, A. M. and J. H. Ganter. 1990. A Pattern Identification Approach to Cartographic Visualization. Cartographica 27(2): 64-81.
MacEachren, A. M. and M. Monmonier. 1992. Geographic Visualization: Introduction. Cartography and Geographic Information Systems 19(4): 197-200.
Mark, D. M. and M. J. Egenhofer. 1995. Topology of Prototypical Spatial Relations Between Lines and Regions in English and Spanish. Auto Carto 12, ACSM/ASPRS Annual Convention & Exposition Technical Papers 4: 245-254. Charlotte, North Carolina, February 27 - March 2, 1995.
Marr, David. 1982. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. San Francisco, CA: W. H. Freeman.
Marr, David. 1985. Vision: The Philosophy and the Approach. In Aitkenhead, A. M., and J. M. Slack eds. 1985. Issues in Cognitive Modeling 103-126. London, United Kingdom: Erlbaum.
McCormick, B. H., T. A. DeFanti, et al. ed. 1987. Visualization in Scientific Computing report to the National Science Foundation by the Panel on Graphics, Image Processing and Workstations.
McGranaghan, Matthew. 1989. Ordering Choropleth Map Symbols: The Effect of Background. American Cartographer 16(4): 279-285.
McGuinness, Carol. 1994. Expert/Novice Use of Visualization Tools. In MacEachren, A. M. and D. R. F. Taylor eds. 1994. Visualization in Modern Cartography 185-200. Oxford, United Kingdom: Elsevier.
McGuinness, C. and V. Ross. 1995. User Interactions with Alternative Map Displays in a GIS Environment. Proceedings, 17th Conference of the International Cartographic Association 2004-2014. Barcelona, Spain, September 3-9, 1995.
Mitasova, H., D. P. Gerdes, W. Brown, I. Kosinovsky, J. Hofierka. 1993. Multidimensional Interpolation, Analysis, and Visualization for Environmental Modelling. Proceedings, GIS/LIS 1993 Annual Conference & Exposition 2:550-556. Minneapolis, MN.
Moellering, H. 1973. The Automatic Mapping of Traffic Crashes. Surveying and Mapping 33(4): 467-476.
Moellering, H. 1984. Real Maps, Virtual Maps and Interactive Cartography. In Gaile Cort, G. L. ed. 1984. Spatial Statistics and Models 109-132. Boston: D. Reidel Pub. Co.
Monmonier, M. 1989. Geographic Brushing: Enhancing Exploratory Analysis of the Scatterplot Matrix. Geographical Analysis 21(1): 81-84.
Monmonier, M. 1990. Strategies for the Visualization of Geographic Time-Series Data. Cartographica 27(1): 30-45.
Monmonier, M. 1991. Ethics and Map Design: Six Strategies for Confronting the Traditional One-Map Solution. Cartographic Perspectives (10): 3-8.
Monmonier, M. and M. Gluck. 1994. Focus Groups for Design Improvement in Dynamic Cartography. Cartography and Geographic Information Systems 21(1): 37-47.
Schneiderman, B. 1987. Designing the User Interface. Reading: Addison-Wesley.
Schweizer, D. M. and M. F. Goodchild. 1992. Data Quality and Choropleth Maps: An Experiment With the Use of Color. Proceedings, GIS/LIS ‘92 2:686-699. San Jose, CA, November 10-12, ACSM & ASPRS.
Slocum, T. A. 1988. Developing an Information System for Choropleth Maps. Proceedings: Third International Symposium on Spatial Data Handling, Sydney, Australia, 293-305.
Taylor, D. R. F. 1994. Perspectives on Visualization and Modern Cartography. In MacEachren, A. M. and D. R. F. Taylor eds. 1994. Visualization in Modern Cartography 333-342. Oxford, United Kingdom: Elsevier.
Thrower, Norman. 1959. Animated Cartography. The Professional Geographer 11(6): 9-12.
Tobler, Waldo. 1970. A Computer Movie Simulating Urban Growth in the Detroit Region. Economic Geography 46(2): 234-240.