White
Paper for NRO Director’s Innovation Initiative
Mindtel,
LLC
September
26, 2001 [second version]
Methods for Perceptually Enhanced Human Interaction
with Information: the Perceptualization Environment
A.1 Overview
In response to the NRO Director’s Innovation Initiative, MindTel will propose the ‘perceptualization environment’ as a multi-modal, multi-sensory, high performance communications tool for handling the escalating quantities and types of information the Office will confront in the near and distant future. Mindtel’s primary goal is to develop an interface medium for drastically increasing the accessibility and, perhaps more importantly, the intelligibility of variant intelligence data. Intelligibility, or the graspable, available meaning within information, is determined by the manner(s) in which it is presented to a user. The representation and comprehensibility of information are the conceptual and technological challenges we seek to address with the perceptualization environment.
Mindtel has developed powerful and generic hardware/software solutions to handle core functional elements of this communications environment. Yet, the greater import of our work will be in the development and implementation of conceptual models with functional prototypes. The latter will be initial demonstrations of how the many streams of critical, time sensitive information may be innovatively woven together and presented to the user with a high degree of content dynamicity and user control. Also, the final presentation of information to the user will manifest perceptual diversity; diversity in using the tool to integrate and render information in perceptual styles which depend on mission needs and user biases.
The question before us in the conceptualization and implementation of the environment is the following: how can a single user simultaneously access greater and greater quantities of information while having that information be presented in a way that profoundly enhances the rate and richness of its understanding? Our goal is to make key people smarter by fitting them with the most powerful and forward thinking technology by which their interaction with mission-critical information may be enhanced. In this document, we will sketch the idea of the perceptualization environment for human-information interaction. The NRO user will be gathering information from wide ranging sources (e.g., sensors, GPS, data bases, etc.). However, our purpose is the development of a generic interface tool which may be deployed in any number of diverse contexts of need. Therefore, the specific content of a particular information realm is to be separated conceptually from the perceptualization chamber itself. In the document, we will focus on both the perceptual and cognitive aspects of the user as they concern
- the
perceptual form in which the information is being represented
- the cognitive state of the human during perception of represented information
Our discussion will begin with the conceptual tools we employ in thinking about interface solutions.
Key terms: neurocosmology, PSI, PHX, PHI, perceptualization of information; state specific sciences; perceptual state space modulation; neuromodulation;
A.2 Conceptual Schema
The human interaction with information, at the level of its structure, is a system made up of multiple components. For conceptual and operational purposes we created a schema for representing these various elements and their dynamical relationships to one another and to the system as a whole; we call it neurocosmology. Neurocosmology is a simple schema for representing key features of the human-information system interactive looping we call the cybernetic loop. Beginning with perception and ending with expression, the schema is made up of the following elements: mind, body, sensor, computation, and rendering. The linguistic shorthand for these, along with their corresponding color sequence in the individual shapes of the image below, are PSI (orange), PHX (gray), SNS (blue), KHI (yellow), and SKY (pink). Within each of the colored shapes is an arbitrary symbol representing each of the elements in the loop.
To summarize the scenario: 1) the user’s mind (PSI/orange) has a thought to query some information; 2) that thought translates into some brain (PHX/gray) state which causes some eye and hand movements to 3) manipulate a mouse and keyboard (SNS/blue) which then 4) triggers a sequence of computations (KHI/yellow) resulting finally in some information appearing to the user (SKY/pink). For the purposes of this document we will limit our discussion to the diagram elements of PSI, PHX, and SKY. These are the mind, the brain, and the actual information the user sees, hears, and/or feels[1] For now, SKY will be limited to those representations of information which are visual.
A.2.1 Mind—PSI
Little needs to be said at this point to define the mind. Obviously, the mind is where information has its end, its meaning. As related to some particular knowledge, problem, or task area, we have information in our mind which ever seeks more related information to refine cognition and action in that particular area. Also occurring in this process is the mind determining which information is actually valuable in a particular context as it interacts with content. Our assumption is that the measure of a powerful and efficient interface system is ultimately in how it helps or hinders the mind as it grasps and processes critical information. More sophisticated and powerful systems for gathering and processing data are being developed constantly. The ability of these systems to generate information is staggering. However, what seems to have been left out of this picture is the development of technologies for making the job of the analyst, the mind in the cybernetic loop, easier, faster, and more powerful. For analysts and decision makers to better serve the purposes of the intelligence community, the perception and cognition of vast information resources must be the starting point for interface development. Mindtel’s contribution has been and continues to be in building information/communications systems around the mind and its physiological vectors into the world, the brain and the body.
A.2.2 Body—PHX
PHX refers to those aspects of the interaction with information based on the body itself. We make the distinction between perceptual PHX, or pPHX, and expressional PHX, or ePHX. The latter, which incidentally is some bodily extension like a hand as it connects to SNS/blue, we will not address at this time, but it is very important as it facilitates decision commands and related input from the user.[2] pPHX covers visual, aural, and tactile modalities of perception. Together, these offer a large array of possibilities for getting information into the mind from out of a user sensitive information system. In the consideration of perceptual PHX, we want to become better informed as to the optimal manner by which each sense system collects information. That is, there are ways of representing information to the eyes, ears, and skin surfaces which take advantage of the unique physiological mechanisms of those senses for gathering information and presenting it to the mind via the brain. Physioinformatics is the term we use to designate this conceptual and development orientation which seeks to exploit the capacities of the senses and the brain for accelerating and enhancing the human interaction with information. Once optimal sensory parameters have been determined, then we figure out which are the best kinds of renderings, or representations, of the information. Again, SKY is the element of our schema that refers to what the user actually sees, hears, and feels from the information being generated.
A.2.3 Rendering—SKY
Inside the percpetualization environment, the user is in the middle of a rich multisensory matrix. Many diverse kinds of visual displays are situated within the environment. In addition, auditory displays will also surround the user. Tactile rendering devices will be attached to an array of body surfaces. Visual SKY is the first rendering problem we plan to take on in the initial prototyping of the perceptualization environment. The eyes are the primary sensory gate for most information at present. Our assumption has been that the natural bias towards the visual, as opposed to the auditory for example, is only a hindrance as the amount of information in all fields increases exponentially. Therefore, eventually, we aim to include sound and tactile renderings in the perceptualization environment.
A.3 Rendered Information
The grant for which we plan to submit is the short term (nine month) grant for the NRO’s DII. Therefore, Mindtel’s plan for an initial prototype will center on 3 streams of information coming into the perceptualization environment from geographically separate spaces. Presently, Mindtel has succeeded in moving towards key features of pPHX. Our work has focused on the three information streams of sensors, human input, and databases.
A.3.1 Sensors and Human Input
The perceptualization environment is designed to serve the intensive information needs of communications/information specialists in any domain open to the benefit of such a technology. To give a context to what we are proposing for sensors and human input we could use an example from a refugee camp scenario which includes medical complications. We have prototyped a system which allows us to
a. collect data from diverse sensors on the ground
· physiological: heart, blood pressure, blood glucose, etc.
· chemical: air quality, water quality
· whether: temperature, wind direction
b. collect data from human input devices
· PDA’s: Palm Pilots equipped with software that allows a user to walk through a refugee camp and enter data into a form covering elements like number of water containers, pounds of soy protein, lamp fuel, pints of blood, bandages, and so on.
The images below are screen shots of the system we are developing for the sensor and human input into the perceptualization environment.
In this case, SKY is a topographic map of Hawaii with some emerging geometries. These shapes may represent features on the ground as indicated in the categories to the left.
As more information comes in through sensors and human input via the PDA into the form fields of those doing assessment on the ground, the interface’s SKY is added to and perceptually enriched in real time. Here we can see place names and geometries emerging on the heels of the data streams.
This image shows more fully the evolution of SKY in real time as information continues to come into the system. The images seen here are not complicated. However, the innovation we bring to this element of the interface is in the abstraction of some information as geometries. These geometry data are then fused with the topographic map to represent many parameters from ground data localized by means of GPS coordinates. The latter are provided as part of our ground based sensor/PDA technology. The next and very important piece of this perceptualization picture is in the representation of information queried from electronic databases.
A.3.2. Text into Geometries—Bibliomics
For a number of years Mindtel has envisioned a technology for accessing and representing data contained within electronic data storage systems in new ways. The majority of information an intelligence[3] user would need to get at is stored in databases. These databases are often located in geographically diverse locations. Moreover, data is often stored in different formats with different securities. The goal for the perceptualization environment is to have data from these many and diverse sources available and accessible to particular users in particular contexts of need.
Bibliomics is a representational technology Mindtel conceived to handle the vast and growing sources of data needed by a whole host of users from those working the areas of intelligence, medicine, disaster relief, stock trading, and many others. The idea of Bibliomics is to take textual/numerical data and begin to find ways of abstracting critical elements of it for presentation to a user who either has scarce time for its raw form, or else who needs to ‘see’ features of that data normally invisible when it exists as text or numbers. The latter covers that information in data we do not even know could be there. New representations will often disclose what was not know to exist before it could be represented.
This means that textual SKY will be
substantially transformed because human visual information processing is
hindered as text can be a non-optimal representation when confronting the
factors of content volume and time limits (e.g., intelligence
emergencies). Therefore, signifiers
other than text ought to be explored and experimented with. Mindtel is doing this work by looking to
visual forms of representation which are open to multi-modal features for
conveying meaning (e.g., geometric shapes, colors, movements, shape changes,
etc.). Below are some examples of how
we might transform textual data into new kinds of signifiers for rapid and
meaning-rich visualizations. These are
only arbitrary forms which if useful would then be assigned multi-modal
significance. For example what if
visual SKY for Nasdaq output could be condensed and more meaningfully
abstracted in some way like this? 
Or what if movements of terrorist operatives and their diverse resources could be condensed from multiple data stores and more quickly and meaningfully visualized?
For the sake of example, think in the following manner about this image. The small flat image on the left is a geographical topography akin to the one of Hawaii in C.3.1 above. Whereas the geographical images representing Hawaii ground information were more basic, the image on the right side above is a much richer geometric complexification of the diverse intelligence data initiated on the left. In the above example the following elements could be represented: the intelligence user has input a query requesting everything from the geographic location of suspected or known terrorist personnel clusters, to amount of money and locations of their bank accounts, to their spending patterns of the past 2 years including what/where they have purchased and in what amounts. Furthermore, queries could look for the location of crop dusters both owned and for sale, or else one could query certain chemicals, where they are stored, who is buying them, in what amounts. The queried information could incorporate a large number of variables. The image on the right would be a typical Bibliomic data fusion and dynamic geometrification of the many database streams collected in the search.
Or what if a disaster relief team was concerned about water supplies in a region?
This set of images could be an abstract and dynamic visualization of the needed data.
These are merely examples of the kinds of thinking we are proposing for how to address the problem of altering the representations of information. The essential point is to open up new ways of thinking and talking about the perceptualization and expression of the vast and diverse data which we and our sponsors will desire from the perceptualization environment
A.4 Cognitive State Enhancements
‘State-specific sciences’ refers to the idea that states of
brain/PHX affect states of the mind/PSI (i.e., perception and cognition). Therefore, it is reasonable to assume that
alterations in brain state will bring out some correlative alterations in
perceptual cognitive state; changes in perception disclose heretofore invisible
information.
Given that an ASC is an overall qualitative and quantitative shift in the complex functioning of consciousness, such that there are new ‘logics’ and perceptions (which would constitute a paradigm shift), it is quite reasonable to hypothesize that communication may take a different pattern. For two observers, both of whom, we assume, are fluent in communicating with each other in a given SoC, communication about some new observations may seem adequate to them, or may be improved of deteriorated in specific ways. To an outside observer, an observer in a different SoC, the communication between these two observers may seem ‘deteriorated.’ (Tart, 1972)
State specific sciences are those scientific inquiries which take into account ‘states of consciousness’ (SoC) when interacting with particular data. For the well grounded suspicion is that we do not know all that is to be accessed in a collection of data. Thus, altering our perceptual cognitive states at calculated points during the design and execution of the interaction process may open us to entire realms of information which would have remained totally invisible.
A.4.1 Perceptual State Space Modulation
The idea of perceptual state space modulation is our attempt to conceptualize what it would look like to introduce state specific sciences into the discussion of human-information interaction. Not only do new representations of information promise to show us new and powerful contents of our information which we could not have otherwise seen, but so might changes in brain state give us new perceptions of that information. Ultimately, it is the thoughts which one has about information which leads to decisions and thus processes set in motion to benefit most from that information.
Any logic consists of a basic set of assumptions and a set of rules for manipulating information, based on those assumption. Change the assumptions, or change the rules, and there may be entirely different outcomes from the same data … By changing the paradigm, altering the SoC, the nature of theory building may change radically. Thus a person in SoC2 might come to very different conclusions about the nature of the same events that he observed in SoC1. An investigator in SoC may comment on the comprehensibility of the second person’s ideas from the point of view (paradigm) of SoC1, but can say nothing about their inherent validity. A scientist who could enter either SoC 1 or 2, however, could pronounce on the comprehensibility of the other’s theory, and the adherence of that theory to the rules and logic of SoC 2. Thus scientists trained in the same SoC may check on the logical validity of each other’s theorizing. We have then the possibility of a state specific logic underlying theorizing in various SoC’s…If such sciences could be created, we would have a group of highly skilled, dedicated, and trained practitioners able to achieve certain SoC’s and able to agree with one another that they have attained a common state. While in the SoC, they might then investigate other areas of interest, whether these be totally internal phenomena of that given state, the interaction of that state with external, physical reality, or people in other SoC’s. (Tart, 1972)
A.4.2 Modalities
How might one think about modulating states of PHX so as to modulate states of PSI? We have identified several. The most obvious one is pharmacological. Beyond chemical induction of a change in state of the brain, another means of inducing alternative SoC is with bioenergetic modulations. Microwaves and other kinds of energy modulations have been shown to have diverse influences on neurophysiology. Perhaps it would be possible to also include certain frequency modulations into the perceptualization environment. Acoustic modulations may also be an avenue for perceptual state space modulation. Manipulated sound can have effects on the central nervous system which may enhance perception and cognition in important ways. For example, the Monroe Institute has long conducted acoustic-neurologic research. With modulation of specific sounds into both ears at the same they are suggested a re-entraining of the two hemispheres of the brain. Hemispheric synchronization is the phenomena whereby communication between the hemispheres is optimized/altered along some axis resulting in perceptual and cognitive alterations. Mindtel’s research in this area thus far has centered on acoustic state modulation. This aspect of our thinking about the perceptualization environment is likely to show value once substantial progress has been made in the development of the perceptualization technologies: SKY renderers, and data fusion technologies.
A.5 Summary
In this paper, our purpose has been to climb out of the box
to think in new ways about the perceptual basis of human interaction with vast
and diverse data sources . What we are proposing is far outside the normative
realm of how to construct interfaces and represent information. Thinking in these ways will require
potential users to understand the challenge which confronts them in this time
of intelligence gathering. Without
pushing the boundaries of what can count as an acceptable means for accessing
information and making decisions based upon it, unrealized possibilities to
help go unseen. We are aiming towards a
new kind of perceptually based literacy to aid any process of high quantity,
high performance interaction with information.
Arriving at this place in our thinking has been the result of conceptual cross linking based on a desire
for both a flexibility of functionality and massively enhanced perceptual
dimensionality. The perceptualization
environment is a representational knowledge matrix which addresses the problem
of the insufficient language structures we currently use in digital communications. The new language we are working on will no
doubt lead to a kind of perceptual cross pollination in the different realms
where this technology is eventually applied.
Our desire is to be funded in sufficient measure to construct a robust
prototype which collects data from sensors, human input, and databases and
turns it into “sentences” in our new perceptual languages.
