Current computing systems do not support human work effectively. They restrict human-computer interaction to one mode at a time and are designed with an assumption that use will be by individuals (rather than groups), directing (rather than interacting with) the system. To support the ways in which humans work and interact, a new paradigm for computing is required that is multimodal, rather than unimodal, collaborative, rather than personal, and dialogue-enabled, rather than unidirectional.

This research will develop principles for, implement, and assess natural, multimodal, multiuser dialogue-enabled interfaces to geographic information systems (GIS) that make use of large-screen displays and virtual environment technology. Progress toward multimodal human-computer interfaces will require both an understanding of individual modalities and the fusion of information at various levels. Specific human-computer interaction (HCI) goals for this research are to achieve natural, human-centered interfaces to complex information systems and to facilitate a two-way dialogue between human and computer. The project is concerned, specifically, with the use of computer vision and speech processing as a means of interpreting and integrating information from three modalities, spoken words, free hand gestures and gaze. It is also concerned with how to enable a human-computer dialogue with an interactive, multi-layered map in the context of a GIS and with map-mediated dialogue between human collaborators. The work will be bootstrapped through use of an existing test-bed that integrates gesture and speech for simple map queries (iMap, developed by one of the team, Sharma).

Estimates suggest that as much as 80% of all digital data contain some form of georeferencing (coordinates, addresses, zip codes). As a result, GIS have become an important tool for managing and analyzing data across a range of science, policy analysis, resource management, business planning, and educational applications. In spite of the potential applications for GIS and the growing availability of geospatial data, GIS are underutilized because current systems are hard to use. As a result, there is a substantial gap between current GIS applications and the promise of GIS (a) to facilitate public access to the large volume of geospatial information being compiled and support public input to important decisions affecting their communities and environments, and (b) to support time-critical decision-making by expert analysts.

The proposed research will advance the science of natural, multimodal HCI, while addressing, directly, the challenges of developing new knowledge through which advances in IT can enable universal participation in geospatial information access and applications. The focus on a natural, multimodal, multiuser, dialogue-enabled interface will support more effective decision-making about risks to public safety and health (and responses to those risks) and will support input to important public decisions by a wider range of participants. Related to natural interfaces generally, and GIS specifically; the research will investigate how characteristics of tasks, information, and users influence the nature of human-computer and human-human computer-mediated dialogue. In doing so, the research will consider how these factors influence the strategies required to achieve an effective easy-to-use interface that supports information access and productive work by individuals and groups. To develop the natural interface envisioned requires careful attention to the human user at all stages of design, development, and deployment, thus a human-centered systems approach. Specifically, we will address the goals delineated above through a conceptual framework for human-centered multimodal collaborative interfaces to GIS that emphasizes the development of shared semantic frameworks through which dialogue can take place.