Humans use information to address their desire and need to learn, communicate, explain, and make decisions. Information is stored in computing systems as “bits” that are either zero or one and collections of bits such as the eight-bit bytes used to encode up to 256 distinct letters, digits, and characters. A computing system’s bits and bytes are converted into spoken or written words, symbols, or images, for humans to understand. Humans define and develop both data and information and their use by computing systems so that they contribute to achieving human goals.
The HCI&IM R&D vision is to make possible the following benefits:
• Information that is available everywhere, at any time, and to everyone regardless of their abilities
• Broadened human use of this information through the ability to: o Access information in different contexts – examples include: ƒ Having the same information presented differently for audiences as varied as school children and researchers ƒ Providing the same content to different devices o Interact with this information using a variety of devices o Meet needs that can vary from access to manipulation, analysis, and control
• Comprehensive management of this vast information environment
Why Do Humans Use Computing Systems?
• To write and talk to each other
• To exchange artifacts of personal interest such as photos, music, and videos
• To exchange artifacts used in the workplace such as text files, drawings, and visualizations
What Do Humans Use Computing Systems For?
In little over 50 years, computing systems have become part of virtually every human activity. The following are key areas of interest to the Federal government in which computing systems are used:
• National defense and national security
• Homeland security • Infrastructure operation and protection
• The workplace
• Education and training
• Financial, personnel, and clientele records management
• Health care
•Manufacturing
• Research and development (R&D) in science, engineering, and technology
R&D communities have developed computing system capabilities to meet their needs, such as:
• Scientific calculations that implement mathematical formulas and increasingly produce huge computer-based data sets
• Computer-based sensors that collect huge data sets
• Software programs that search for patterns in such data sets
• Computer-generated visualizations (images or videos created by controlled mapping of interpreted data bits) for human pattern recognition and for studying multiple data sets
• Computer-based information for use by other researchers
The broader population (which includes R&D communities) has found it beneficial to adapt these systems for new and different capabilities, such as:
• Moving more information on line by creating digital content and by converting material from non-digital form to digital form
• Using general-purpose computing systems to manage the telecommunications infrastructure or control industrial processes, for example
• Using special-purpose computing systems and computing systems that have special physical features, such as the small purpose-built computers in today’s automobiles that communicate with computers in satellites to provide location information; the car’s computers present this information in different forms to the car and to humans
• Using global-scale computer-based networks to access computing systems that contain the data and information and provide the services they wish to use; the Internet and the Web are examples
• Interacting with computing systems and each other either simultaneously or by connecting to networks and computing systems at different times; an example is school children around the world communicating with one other, contributing to on-going scientific data collection, and “seeing” natural phenomena such as California’s Monterey Bay during a scientific dive through advances in computing and human interaction technologies and ease of use, human use of the on-line world will continue to grow.
How Do Humans and Computing Systems Use Data and Information?
• To be able to cycle from bits to various humanly comprehensible representations of data and information, and in some cases to cycle back to bits
• To enable humans to interact with the content in its various forms in order to obtain additional information (from the bits) and to manipulate representations to better understand content
How Do Humans Use Data and Information?
• Understand and learn about the world from direct observation
• Understand and learn about the world from artifacts (for example, off-line artifacts such as books and sculptures, and data and information – which include digital representations of books and sculptures – in on-line artifacts)
• Create new information
• Make decisions
• Control processes (such as the operation of a nuclear power plant)
• Communicate with other people
• Communicate with computing systems
• Share what they have learned and created with others • Explain, inform, and teach
How Do Computing Systems Use Data and Information?
• Perform numerical calculations
• Use mathematics to model and simulate real-world phenomena
• Compare data sets and find statistical patterns
• Use mathematical logic to reason deductively about information
• Make decisions (based on rules established in computer processes set up by humans)
• Connect with other computers. Reasons for doing so include to: o Collect and find data and information (results of a google search of the Web are an example) and download or send what is found o Collect data from computer-based sensors placed in the real world o Send instructions to computer-based actuators placed in the real world
However, computers are currently less able than humans to:
• Create information
• Create new ways to communicate effectively with humans
• Validate that their output is true (for example, that it accurately represents the real world)
How Can Humans and Computing Systems Best Work Together?
- Air Traffic Management
- Human Genome Science
HCI&IM Research Needs
• Information Creation, Organization, Access, and Use
• Managing Information as an Asset
• Human-Computer Interaction and Interaction Devices
• Evaluation Methods and Metrics
Information Creation, Organization, Access, and Use
To meet these HCI&IM agency needs, R&D is needed in the following areas:
• Human perceptual, cognitive, and neural processes for obtaining and using information
• Human perceptual, cognitive, and neural processes for obtaining and using information
• The processes that computing systems use to analyze data and information, such as the uses of machine-created information
• The processes humans use (for example, observing, reading, writing, taking notes, and interacting) to obtain meaning from information
• Scientific theories of information content that allow its presentation in multiple forms and formats without changing its value
• Usability, that is, ways to provide information that is easy to create, organize, access, and use
• System use by multiple communities or user groups in multiple domains, in order to establish general methods for building capabilities that transfer across communities and domains
• Scientific theories underlying the use of models to generate data
• Developing and evaluating scientific, economic, and social domain models underlying information use
• The integration of cognitive principles of use in the specification, design, and implementation of interactive presentations, such as visualizations, so that they are appropriate for the information content
• The integration of cognitive principles of use in the specification, design, and implementation of interactive presentations, such as visualizations, so that they are appropriate for the information content
• Interoperability of data, information, and associated software
• Methods for designing, building, and maintaining systems for a usable, extendable IM environment
• Methods for assessing the effectiveness of IM systems
Managing Information as an Asset
HCI&IM R&D per se has its own rich set of topics for R&D in managing very large, distributed, heterogeneous multi-modal collections of data and information. These topics include:
• Digitizing legacy information and creating on-line descriptions of off-line material
• Digitizing legacy information and creating on-line descriptions of off-line material
• Cataloging, searching, finding, discovering, viewing, processing, and disseminating data and information
• Metadata and new ways to index and find information
• Multi-modal and multi-mode access
• Interoperability of data, information, and the software that accesses and uses them
• Guaranteeing 24/7 accessibility and dissemination • Provenance, access and version control, accuracy and integrity
• Technologies for guaranteeing security, privacy, and confidentiality
• Technologies for guaranteeing security, privacy, and confidentiality
• Long-term archival storage and preservation
• Designing, building, and maintaining information management systems
• Scalability of the collections and their management
Human-Computer Interaction and Interaction Devices
Humans use computing systems to augment their own capabilities. To do so, they interact with the data and information in those computing systems, in several ways:
• They seek content from information sources.
• They seek content from information sources.
o The content can be presented in different modalities and different modes – visually (text, images), audibly (spoken and non-speech sounds such as music), or haptically (touch and pressure), etc.
• They interact with what they find.
o The means they currently use include writing or drawing, speaking or singing, pointing or touching, and moving hands or eyes.
o They use those means to further query the information, have it presented in different ways, or give directions (for example, stop a simulation, change some numbers, and restart the simulation using those new numbers).
• They control computing-enabled devices.
HCI&IM R&D needs include:
• Basic understanding of the internal human perceptual, cognitive, and neural processes of obtaining and using information
• Basic understanding of the internal human perceptual, cognitive, and neural processes of obtaining and using information
• Basic understanding and best employment of media, modalities, and modes to maximize human ability to seek, access, and use information
• The science of usability, that is, of delivering information in a usable manner, which includes providing optimal interaction capabilities for all possible human use
• Basic understanding of how humans share information and the methods they use
• Basic understanding of how groups of people work in a shared information space environment
• Basic understanding of how teams (groups organized to work together) work in a shared information space environment and how they evolved to become a team
• Basic understanding of how teams (groups organized to work together) work in a shared information space environment and how they evolved to become a team
• Basic understanding of how humans use information in individual and group or team problem solving, planning, decision-making, and explaining
• Understanding how computing systems use data and information to maximize synergistic human-machine capability
• Models of humans, computing systems, and the synergies between them to aid in interactive system design
To that end, they are conducting R&D in interactive technologies including:
• Decision support
• Decision support
• Designing interfaces for specific tasks and for multi-tasking • Integrating user intentions into system or interface design
• Intelligent assistive devices and technologies ranging from handheld devices to robotic assistants
• Interactions in multimodal and multimode environments
• Modeling presentation, use, and sharing of data and information
• Multimedia technologies • Pervasive and immersive environments
• Security
• Spoken and written languages, including translation and speech to text
• Understanding of collaboration and development of collaboration technologies
• Universal accessibility
• Usability studies
Evaluation Methods and Metrics
In abstract worlds outside the physical realm, human uses determine if information is valid, and in this abstract world of information, evaluation challenges abound, as seen in the following research needs:
• Theory of evaluation for IM
• Theory of evaluation for IM
• Theory-based evaluation methods for HCI&IM
• Theory of evaluation for interactive information systems
• Theory of information validation
• Metrics of total system performance
o Under basic and a variety of other conditions
o Error bounding criteria for system acceptability and usability
• Predictive methods for human performance while using computing systems
• Models of total system function
Methods need to be developed to validate:
Methods need to be developed to validate:
• Models of natural systems:
o Physical systems such as weather
o Biological systems such as DNA and cells
o Cognitive systems and performance
o Groups and teams
• Mathematical models of analytical methods
• Models of properties of information content such as uncertainty and error propagation during inference
• Models for evaluating design before implementation, especially where the human is a critical link in system use
HCI&IM CG Government Workshop (Held October 22, 2001)
DARPA
She then discussed DARPA’s HCI&IM investments. DARPA is focused on developing applications that are relevant to the military. DARPA/ITO’s HCI&IM efforts are chiefly in HCI, particularly programs in spoken languages (acronyms are defined in Appendix 2):
• Communicator: dialog management to enable people to converse with a computer to create, access, and manage information and solve problems
• Communicator: dialog management to enable people to converse with a computer to create, access, and manage information and solve problems
• SPINE: speech in noisy environments
• TIDES: translation, topic detection, extraction, summarization
• EARS: speech to text, including summarization and metadata extraction
• DAML: DARPA Agent Markup Language that facilitates the Semantic Web
• Augmented cognition: use fundamental understanding of cognition and guidelines for user interfaces to enhance human abilities in diverse, stressful, operational environments
Suggested new areas of HCI R&D include:
• Implicit interactions, perceptive user interfaces, and intelligent user interfaces that have autonomy to aid users in multimodal environments
• Implicit interactions, perceptive user interfaces, and intelligent user interfaces that have autonomy to aid users in multimodal environments
• Interactions with distributed, embedded computers or with teams of robots
DARPA/ITO IM programs include:
• Bio-surveillance, focusing on receiving and correlating data from heterogeneous data sources and detecting patterns in them
DARPA/ITO IM programs include:
• Bio-surveillance, focusing on receiving and correlating data from heterogeneous data sources and detecting patterns in them
• Rapid knowledge formation to encode a broad range of knowledge into reusable databases, then automatically reason across domains
Suggested areas for new IM R&D are:
• Distributed, replicable storage to support mobile uses
Suggested areas for new IM R&D are:
• Distributed, replicable storage to support mobile uses
• Information management for large-scale distributed systems
• Context aware information retrieval
• Locally optimized information retrieval for bandwidth-constrained situations
• Abstractions of data for storage
• Secure distributed storage with automatic replication
EPA
Current EPA activities include:
• Multimedia (air, water, soil) Integrated Modeling System (MIMS)
• Multimedia (air, water, soil) Integrated Modeling System (MIMS)
• Software infrastructure and tools for constructing, composing, executing, and evaluating multimedia models
23
• Enablers that let environmental decision makers, scientists, and modelers work at higher conceptual levels
Planned EPA HCI&IM activities include:
• General model of environmental data and associated programming libraries
23
• Enablers that let environmental decision makers, scientists, and modelers work at higher conceptual levels
Planned EPA HCI&IM activities include:
• General model of environmental data and associated programming libraries
• Interdisciplinary data analysis and visualization tools
Fine identified some of EPA’s HCI&IM needs:
• Integrated data analysis approaches to exploring relationships among diverse types of data, including data mining and pattern recognition technologies
Fine identified some of EPA’s HCI&IM needs:
• Integrated data analysis approaches to exploring relationships among diverse types of data, including data mining and pattern recognition technologies
• Integrated search and retrieval across multiple digital libraries
• Efficient management and distribution of large data sets
• Approaches for efficiently and appropriately resolving spatial and temporal scale differences
• Paradigms and tools for automatically building consistent systems of models and data or for identifying semantic inconsistencies in such systems
• Approaches for joint cognitive decision making for issues that involve physical, chemical, biological, social, economic, and regulatory influences and effects
• Long-term archival of large data sets
DOE/SC
the DOE/SC response to this challenge includes:
• A coordinated program of technology R&D and pilot projects to pioneer new approaches to flexible, secure, coordinated resource sharing through pilot collaboratories, middleware, and network research
• Scientific Discovery through Advanced Computing (SciDAC), DOE/SC’s new integrated program to:
o Create a new generation of scientific simulation codes o Create the mathematical and computer systems software needed for these simulation codes to effectively and efficiently use terascale (trillions of operations per second) computing systems o Create a collaboratory software environment to enable geographically separated scientists to work together effectively as a team and access both facilities and data remotely, sometimes synchronously and sometimes not
DOE/SC’s fusion energy, particle physics, and Earth system grids face problems including:
• Moving large amounts of data
• A coordinated program of technology R&D and pilot projects to pioneer new approaches to flexible, secure, coordinated resource sharing through pilot collaboratories, middleware, and network research
• Scientific Discovery through Advanced Computing (SciDAC), DOE/SC’s new integrated program to:
o Create a new generation of scientific simulation codes o Create the mathematical and computer systems software needed for these simulation codes to effectively and efficiently use terascale (trillions of operations per second) computing systems o Create a collaboratory software environment to enable geographically separated scientists to work together effectively as a team and access both facilities and data remotely, sometimes synchronously and sometimes not
DOE/SC’s fusion energy, particle physics, and Earth system grids face problems including:
• Moving large amounts of data
• Supporting a distributed research science base
• Discovering data and information
• Providing access to experimental devices
The vision for DOE/SC’s science grid is to provide:
• An environment for scientific applications
The vision for DOE/SC’s science grid is to provide:
• An environment for scientific applications
• Middleware to put together grid services to let applications talk at various levels such as at the network level
• Access to computing, visualization, and storage resources
• Public key infrastructure and certificate authority with policies that allow a community to interact with other communities both inside DOE and in other agencies and internationally
• Reliable and secure group communications
NOAA
NOAA’s HCI&IM gaps include:
• Managing very large environmental databases and information sets (including event recognition and sharing vocabulary and thesaurus)
• Managing very large environmental databases and information sets (including event recognition and sharing vocabulary and thesaurus)
• Adaptive environments
• Language integration — being able to use language interfaces and thesaurus
NIH
NIH’s HCI&IM research needs are similar to those of EPA, DOE/SC, and NOAA:
• Support collaboration by distributed research communities
• Support collaboration by distributed research communities
• Develop and share large data sets and information
• Develop data and information collection, manipulation, analysis, distribution, retrieval, visualization, maintenance, and preservation methods and tools
NSF
Strong described several grants that illustrate NSF/ITR’s HCI&IM investments:
• Multilingual access to large spoken archives
• Multilingual access to large spoken archives
• Molecular ocean data assimilation
• An ensemble approach to data assimilation in the Earth sciences
• Cognitive and social design of robotic assistants
• Adaptable voice translation for minority languages
• Research on the perceptual aspects of locomotion interfaces
• Digital clay for shape input and display
• Capturing, coordinating, and remembering human experience
• Computational tools for modeling, visualizing, and analyzing historic and archaeological sites
Strong then noted some features of NSF’s HCI&IM investments:
• ITR is a generic program that is highly competitive.
Strong then noted some features of NSF’s HCI&IM investments:
• ITR is a generic program that is highly competitive.
• Peer-reviewed projects tend toward being incremental rather than revolutionary.
• A new BioInfo program will help address computer science/bioinformatics needs.
• Informatics and data management in massive databases are still gaps since science research data sets are generally not large.
There was a discussion of NSF’s possible investment in R&D in webs of sensors and associated data management for advanced warnings and for modeling the prevalence of diseases in animals, for example. Such technologies are of interest to public health organizations, and they could be applied in other areas such as weather sensors for NOAA use and chemical sensors.
Strong interacts with the intelligence community and discussed some of their needs:
• There is substantial technology for knowledge discovery in databases, but it doesn’t scale sufficiently.
There was a discussion of NSF’s possible investment in R&D in webs of sensors and associated data management for advanced warnings and for modeling the prevalence of diseases in animals, for example. Such technologies are of interest to public health organizations, and they could be applied in other areas such as weather sensors for NOAA use and chemical sensors.
Strong interacts with the intelligence community and discussed some of their needs:
• There is substantial technology for knowledge discovery in databases, but it doesn’t scale sufficiently.
• There is a need for knowledge representation technologies not just for heterogeneous data but multimodal (visual, audio, haptic) data, and a need to relate information to gain understanding.
• There is a need for knowledge sharing across traditional boundaries such as sharing data across research disciplines.
NASA
The NASA HCI&IM research focuses on technologies that enhance cognitive and collaborative performance of expert teams. NASA mission teams include users at all levels of operations:
• Mission control
• Mission control
• In-space scientific experimentation
• Scientific use of collected data
• Systems design, testing, evaluation, and maintenance
• Training for all participants in mission planning and rehearsal
NIST
Herman identified the following gaps in HCI&IM R&D at NIST:
• Greater support in IT accessibility
• Greater support in IT accessibility
• Evaluation and standards for intelligent information systems
• Immersive, interactive environments
• Universal access
GSA
The goal is a citizen-connected Government that provides universal access to both science and service communities. One way is to eliminate stove-piping. Toward that end, Turnbull identified common elements of current and future plans:
• Open, standards-based Federal enterprise architecture
• Open, standards-based Federal enterprise architecture
• Knowledge sharing among communities of practice such as scientific user communities
• Adoption of inclusive, universally accessible IT infrastructures
• Validation of IT usability and accessibility performance
AHRQ
AHRQ works to improve decisions in three areas – clinical, health care, and public policy. The agency funds scientific research and synthesizes and develops evidence-based medical information to inform the health care choices of health care practitioners and patients. AHRQ research advances the use of information technology for coordinating patient care and for conducting quality and outcomes research. AHRQ goals are to understand patient safety incidents and reduce medical errors and injuries through the use of information technologies, such as the Internet, that enhance information access, provider-patient communication, and shared decision-making. AHRQ research programs focus on IT in two areas:
• Improving patient safety
• Improving patient safety
• Improving the quality of patient care by evaluating clinical applications that use health data standards and information technology
OTHER GOVERNMENT PROGRAMS
NSF’s Digital Libraries and Digital Government Programs and their Joint Workshop with the Library of Congress on Preservation
On October 28, 2002, Lawrence E. Brandt, Digital Government (DG) Program Manager at NSF, and Stephen M. Griffin, Program Manager for the Digital Libraries (DL) Program at NSF gave a three-part briefing that summarized their individual programs and then focused on a workshop they co-sponsored with the Library of Congress.
• The DG Program supports multi-disciplinary cross-sector (academia, industry, and Government) collaborative information technology research to improve the availability and use of on-line government information.
• DL research is grounded in computer science and engineering research, informed by domain research across disciplines, applicable to a broad set of scientific and nonscientific problem domains and characterized by novel collaborative efforts focused on the creation, collection, organization, use, and preservation of large volumes of digital information in a rapidly changing, globally-linked knowledge environment.
• The DG and DL programs and the Library of Congress jointly sponsored a Workshop on Research Challenges in Digital Archiving that was held at the Library of Congress on April 12-13, 2002. The workshop’s goal was to better understand issues in the long-term preservation of emerging vast on-line resources. The issues are of immediate concern due to the ephemeral nature of on-line material and the speed at which the underlying technology changes, leading to increased loss over time. While preserving bits is necessary and challenging, logical preservation is also needed. Challenges particular to digital archiving include the long-term perspective; interrupted management; changing user communities, requirements, and expectations; and systems and technologies evolving around the data
• The DG Program supports multi-disciplinary cross-sector (academia, industry, and Government) collaborative information technology research to improve the availability and use of on-line government information.
• DL research is grounded in computer science and engineering research, informed by domain research across disciplines, applicable to a broad set of scientific and nonscientific problem domains and characterized by novel collaborative efforts focused on the creation, collection, organization, use, and preservation of large volumes of digital information in a rapidly changing, globally-linked knowledge environment.
• The DG and DL programs and the Library of Congress jointly sponsored a Workshop on Research Challenges in Digital Archiving that was held at the Library of Congress on April 12-13, 2002. The workshop’s goal was to better understand issues in the long-term preservation of emerging vast on-line resources. The issues are of immediate concern due to the ephemeral nature of on-line material and the speed at which the underlying technology changes, leading to increased loss over time. While preserving bits is necessary and challenging, logical preservation is also needed. Challenges particular to digital archiving include the long-term perspective; interrupted management; changing user communities, requirements, and expectations; and systems and technologies evolving around the data
Air Force Research Laboratory Human Effectiveness Directorate
Maris M. Vikmanis, Chief of the Crew System Interface Division, began by describing his Division’s mission, which is to conceive, develop, integrate, and transition information display and performance aiding technology to assure the preeminence of U.S. air and space forces. The Division’s branches address aspects of the mission:
• Visual Display Systems Branch – helmet displays, flat panel displays, night vision systems, sensor cueing, vision science, optics/transparencies
• Visual Display Systems Branch – helmet displays, flat panel displays, night vision systems, sensor cueing, vision science, optics/transparencies
• Aural Displays and Bioacoustics Branch – voice communications, acoustic signatures, vibration, 3-D audio, noise suppression
• Human Interface Technology Branch – multi-sensor displays, hands-free control, crew station fit, operator functional state, aiding and adaptation
• Information Analysis and Exploitation Branch – speech processing, voice recognition, battlespace awareness, cognitive display
• Crew System Development Branch – requirements, analysis, design integration, test and evaluation, cockpit concepts, modeling and simulation tools, real-time mission simulation
DOE/OSTI and CENDI
Web sites such as science.gov let users search both these databases and other Web sites that contain technical reports, journal articles, and other published material. Challenges OSTI is addressing include:
• Knowing if a search has overlooked a key resource. A solution is to take advantage of today’s network speed to search every possible source.
• Knowing if a search has overlooked a key resource. A solution is to take advantage of today’s network speed to search every possible source.
• Developing an architecture that collects search results to selectively allow user downloads, which is especially useful for large distributed data sets
CENDI has held workshops on:
• Business Continuity and Disaster Recovery
• Business Continuity and Disaster Recovery
• Managing and Preserving Electronic Resources: The OAIS Reference Model
• PKI and Digital Signatures: From E-Commerce to E-Information Management
• Evaluating Our Web Presence: Challenges, Metrics, Results
• Handles as Persistent Identifiers
CENDI projects and reports include:
• Science.gov • Copyright FAQs
CENDI projects and reports include:
• Science.gov • Copyright FAQs
• License Agreements for Electronic Products and Services: FAQ
• Evaluating Our Web Presence: Challenges, Metrics, Results
