Tokyo -- The Sony Corp. has developed an integrated portable elec- tronic "book reader" that weds a 3-inch CD-ROM drive and 256 x 200 pixel LCD display. The DD-1 Data DiscMan adheres to the electronic book specifications outlined by the Electronic Book Community industry association, founded five months ago. Using rotating 3-inch CD-ROM media, the system delivers 200 Mbytes of data on CD platters that cost under $2 each to replicate in volume. If you think of one book as a megabyte of text, this could store about 200 books, so filling the platters is really a matter of cost, not technology. However, video sequences and graphics eat up space much faster. The book could only hold about half an hour of encoded video, for instance.
Current applications focus on relatively long-lived reference material. The initial DD-1 system delivers five dictionaries to the user, using retrieval software contained on a slip-in IC decoder. These are English and Japanese language dictionaries. A small typewriter keyboard lets a user select particular entries or portions of the text. Eighteen CD electronic book titles were released at prices that are comparable to hardcover books. They included reference books and guides to medicine, travel and entertainment. However, with only ten lines of readable text on display at any one time, real reading, as in reading novels, works of nonfiction, comedies, etc., would not be very successful, disrupted by the need to redisplay or scroll the page too frequently.
The $379 unit has a three-hour rechargeable battery lifetime. It is equipped with a video output jack allowing the electronic book to be displayed on an outboard TV screen or video display. The DD-1 can also be used to enjoy 3-inch CD audio platters in stereo.
Not too many years ago this would have been taken as science fiction. Now, we can see the inadequacies immediately. Not enough memory of an interactive kind to permit annotation and real writing with these materials as sources. No provision for imagery on the portable unit itself. Missing color and 3-D graphics. There are so many functionalities we take for granted or find essential for highly interactive media.
And yet the pieces are building up dramatically and quickly. There are a host of technologies that are coming available. DVI, CDI, three dimensional sketch, draw, and animation systems. Digitizing systesm for sounds, sights, and printed materials. The hardware is progressing so quickly that the implications for its use are lagging far behind. This is particularly true of the possible space of new inventions that coudl affect how we use graphs and charts, icons, animations and video. In particular, althought it is widely acknowledged that human perception and thinking is dominated by visual components, no clear theoretical framework for visual thinking has yet been developed. In particular, no agreed on approach to developing visual abstractions for visual thinking has found general scietific acceptance. It is clear that the host of new technologies will dramatically affect how we manipulate and use visual imagery, and that there is a great need for improved understanding in this area. Some developing ideas will be outlined below, but first, we will examine the technology of multimedia.
Multimedia Technology
DVI
Palenque Videodisc
CDI
Voyager Series
Videodiscs of American History of the 20th Century.
At the 1990 National Educational Computing Conference (NECC) multimedia applications to education conitnued to evoke amazement and interest. James E. Dezell, IBM's Vice President of Educational Systems, led off with a first day keynote address, "Restructuring Education Through Multi-Media Technology." He demonstrated a hypermedia prototype called "Ulysses" in which they had designed an enrichment of a poem by Tennyson. Viewers were treated to an interactive spectacle of different actors reading in full motion video, literary critics arguing, Ted Kennedy quoting at the Democratic convention, Joseph Cambell explaining, and much more. Click click click.
A little later the same day another large computer company responded. Bernard R. Gifford, Vice President, Education, of Apple had a very similar title, "From Change to Transformation: The Role of Technogy and Education." Again a dazzling description from many different perspectives, this time of the change from an industrial to an information society, fully informed with eye-catching graphics and animation. It ended, of course, in a hypermedia paean to hypermedia.
It is clear that large organizations with much money at stake are casting their lot with the prospect that hypermedia technology will affect the way we learn and work, and that we may be able to learn and work, anytime adn anywhere, together.
What are all these hypermedia technologies then: CD-ROM, CD-I, DVI, OCR, etc., and how do they work? What is the right level to deal with these issues in a book on Instruction?
HyperGraphics
Graphs and charts can be liberated from simple static displays into moving invitations to explore relations. for instance, here are two sets of numbers. Can you find some relationships among them:
Table 9.1
Data that may have a clean relationship.
____________________________________________
s q
____________________________________________
36 88
67.25 224.7
93 365.3
141.75 687
483.8 4332.1
____________________________________________
Using a simple statistical package, one can immediately look at the relationship in a graph.
Figure 9.1 A scattergram of the relationship of the data in Table 9.1
This seems to be a very nice fit. A relationship of .992 is rare in the social sciences. Yet, there seems to be a decided curve to the points as they are plotted. The numbers in column 2 are increasing faster than those in column 1. Perhaps a non-linear increase like this could be mimcked by squaring the numbers in column 1. When they are plotted:
Figure 9.2 A scattergram of the relationship of the data in Table 9.1, with
column 1 squared.
Intriguingly, this has reversed the state of affairs. The correlation remains exactly the same, but now the square of column 1 is increasing faster than column 2. Perhaps an exponent exactly halfway between 1 and 2 would do the trick.
Figure 9.2 A scattergram of the relationship of the data in Table 9.1, with
column 1 raised to the power of 1.5
In fact it does the trick exactly. The relationship we have empirically uncovered can be stated in a simple equation as q = .407 s 3/2 Since 1.5 is the same as 3/2, we can square both sides and convert the equation into:
q2 = .166 s3
In fact, these are data taken from a modern encyclopedia relating the time it takes a planet to complete one revolution around the sun with its distance from the sun. The Equation we have empirically derived is the same as Johannes Kepler's third law of planetary motion, formulated in 1618, from very similar data (Qin and Simon, 1990). You may recognize the middle entry (93 million miles from the sun, with an orbit of 365.3 days) as Earth.
It is wonderful that with so little effort, we can establish scientific relationships that first took decades to formulate.
However, we can go far beyond these simple graphic systems to produce graphs and charts of startling power and dynamism. It is not enough to convert these numeric relationships into static graphs. We can make these graphs interactive and dynamic to reveal the full power of their interplay.
Imagine trying to understand the way a simple lens works. Some of us are nearsighted and others are farsighted. What is the difference in the glasses needed by each type of visual error? An interactive image of the lens, the optical components of an eye, and the dynamic path of light (rays or waves)
Visual Abstractions and Thinking
Visual Knowledge Representation in MultiMedia Cyberspaces.
How should knowledge be structured to take full advantage of advanced computer environments for communication and training? Basic theories about visual communication need to be drastically improved if the rapid progress in computer technologies is to be fully leveraged in future instruction. The prospect of computer CyberSpaces in which people are both shielded from their surroundings and immersed in a symbolic virtual reality raises fundamental questions about the nature of optimal communication techniques to be used in such exotic environments. The base communication technologies that will be packaged in CyberSpace environments -- hypertext and expert systems -- have primarily focused on symbolic and text representation systems to manipulate knowledge. The coming ability to create entire personal worlds of experience in which people can share their knowledge and decisions adds new value to visual representation systems that can capture and communicate both concrete reality and complex abstract ideas. The experiments to be carried out will begin to define a theory of visual thinking that can be realized in CyberSpace systems for command, control, and training. Since a complete theory of visual thinking is still many years off, the research will proceed in a bootstrapping approach that will use existing technologies to validate competing sets of basic theoretical principles derived from past research in expert systems and hypertext.
Expert Systems
The growth of Artificial Intelligence (AI) in the last decade has been steady and broad. Two main technologies of AI have been particularly important for Army Training: expert systems and knowledge engineering. Expert systems have been used widely as help and explanation systems, decision aids, planners, advisors and managers, and in many small implementations throughout industry and government. Expert systems are important components of Intelligent Training Systems. In order to produce expert systems efficiently and quickly, knowledge engineering must be understood and enhanced. The bottleneck to accelerated expert system deployment is in knowledge engineering. Yet,a vast storehouse of important knowledge is being lost is being lost as experts retire. Their skills and valuable judgments need to be captured in ways that computers and the next generations can use.
Research Issue: The key psychological contribution to the development of expert systems is an understanding of knowledge structures (Anderson, 1983; Clancey, 1988; Kurland & Tenney, 1988): how concepts grow and interlink as they are being learned. This understanding has informed the process of knowledge engineering, adding a broad array of techniques and technologies to the armament of extracting and codifying knowledge from experts and novices. Although these techniques are far advanced over the state of the art a decade ago, increasingly faster and cheaper machines with augmented memory are creating ever greater demands for simpler and more effective knowledge structuring technologies.
Hypertext
Research on hypertext focuses on how to use point-and-click technology to communicate ideas easily and quickly. Hypertext is usually defined as non
linear text, but it is more accurately viewed as a node and link database ( Shneiderman, 1989). When the links are constructed to reflect human associations among ideas, hypertext becomes a dynamic, interactive knowledge base that complements expert system technologies. In fact, it has been called "expertext" (Rada, 1990). Also, the nodes of the hypertext can include graphics, animations, video, and sound in its structures to form hypermedia. It is the enormous convenience that hypertext has added to mixing pictures and text that motivates this research on visual thinking. The need for an overarching theory that connects pictures (icons, graphics, animations, three dimensional designs, videos) with text is made even more compelling with the development of technologies for virtual realities that extend hypertext into CyberSpace.
MultiMedia CyberSpaces: Computational power and portability have made available binocular goggles, power datagloves, and telecommunicating sensors that extend an individual's reach and both bring the soldier into contact with others (at a distance) through telepresence, and isolate the same person from the external world with an artificial covering or exoskeleton. How people perform in these artificial realities, how they best communicate with others to cooperate and compete in these computational work environments, opens a brave new world of cognitive psychological research. In general, these artificial realities not only encapsulate people; but they can also be seen as natural ways of exploiting hypertext and expert systems technology to improve command, control, and training systems by making them more accessible and useful to a broader range of users. When the users may speak different languages, visual communication becomes even more useful.
Multilingual Users: A picture is worth a thousand words, when a person understands those words. How much more is it worth to speakers of different languages? Visual communication through
hypertext and CyberSpace techniques offers an additional channel of
communication among multilingual users. If visual abstractions,
(icons, images, graphs, animations, and videos) can be realized in a
principled, theory-driven way, then such communication could
connect multilingual users directly, thought to visual thought.
What follows is my best attempt to begin to bring some order to this bubbling and frothy area of multimedia. In order to define a knowledge representation scheme that is
appropriate to visual thinking and communication a theoretical
framework must first be shaped from existing components extracted
from research on AI and hypertext. A key ingredient is taken from
contemporary approaches to knowledge representation --- schema,
frame, and script theories (Psotka, Massey, and Mutter, 1988). A
second component is derived from protoype and exemplar theories of
pattern recognition (Psotka, 1983). The third leg of the beginning of a
theory of visual thinking derives from on-line hypertext dictionaries
that structure words and knowledge not alphabetically, but in accord
with simple but meaningful relations that bring together all the
related meanings at a glance (Miller et al., 1988). These components
of a theory of visual thinking and communication suggest a series of
hypotheses that are empirically and experimentally testable. The
hypotheses are ordered in a sequence that lays out an appropriate
program of research below:
Hypothesis1: Meronymic relations among concepts will be most stable
and informative for creating visual hierarchies. Meronymic relations
connect parts of things to their wholes. For example, a tread is part of
a tracked vehicle. A wheel is part of a tracked vehicle. Creating a
database of images that can be reusable and recombinable into
different combinations (e.g. using the same wheel for a tank and a
personnel carrier and perhaps even a HUMMV!) is at the moment an
art and an unconstrained tsk. Discovering the principles that help to
define how to structure these images will be the first goal of this
project.
Hypothesis 2. Synonymic relations will shape the form of the best
exemplar or prototype image. Synonyms are those concepts with
strongly overlapping meanings. For example, convertible, coupe, jeep,
limousine, sedan, jalopy, wreck, and hummer are all kinds of cars.
Choosing the best example of a "car" will be strongly constrained by
this synonym set, and the choice of a best example of a visual image
for each of these synonyms.
Hypothesis 3. Metanymic relations will add increased variability to
the visual image. Metanymic relations connect the same kinds of
things together. For example, the taxonomy formed by the sequence of
ordered concepts: house, dwelling, building, structure, material, and
thing; goes from more specific to more abstract in an ordered sequence
of metanymic relations. As the image becomes more abstract, certain
components will "blur" while others will remain recognizable. How
far up the hierarchy one can go and still maintain a useful and viable
visual image for communication purposes will be an important
principle to clarify in this research. The principles uncovered in
creating meronymic images will help suggest techniques and principles
for metanymic images.
Hypothesis 4. Antonymic relations will define "contrast sets" for
training purposes. Antonyms are frequently used bipolar
classifications of objects, such as friend -foe; black - white; and
sometimes opposite ends of a graded continuum, such as long - short.
Visual images selected for one pole will need to take into account the
other. Images of "friendly" tanks will need to be selected to
emphasize differences in features and components from "enemy"
tanks.
Hypothesis 5. Images of "features" will be affected by the same
principles (hypotheses 1 to 4) as images of concepts. The features
that are the meronymic "parts" of images will be constructed using the
same principles as the "whole" concepts. For example, images of
wheels or turrets themselves have components, synonyms, antonyms,
and higher-order metanymic relations that constrain the choice of
good exemplar images.
Hypothesis 6. The context of an abstract image determines its shape.
The existing database of component, synonymous, and subordinate
imagery must show flexibility and variability in its assembly into
higher-order, more abstract units, dependent on the context of its use.
It is not enough to have rigid, monolithic knowledge representations
fully pre-coded. Some mechanism for dynamic assembly of these data
into context-sensitive schemas must be found. A spinning wheel or a
wheel of fortune is not the same as a jeep's wheel, and so the context
of spinning, gambling, or warfighting must block the other
representations.
Hypothesis 7. Abstract images come to resemble icons. The
relationship between icons and real images has been a problematic one
that needs to be resolved within this theoretical framework. "Icons"
are visual symbols for entities or actions. Icons can be used to
powerfully represent information. The iconography of tactical maps
has developed over many centuries. Hypertext makes it possible to
fundamentally alter the power of icons. Buttoning an icon can give
additional or complete information about the unit it represents.
The opportunity to link part of an image to an elaboration of that part
is potentially very useful. For instance, a town on a map may be
linked to a video of someone driving through that town. Selecting
appropriate icons may no longer be a designer's art but a functional
consequence of well-selected abstract imagery.
Hypothesis 8. Abstract images will be preferred to drawings and text.
The flexibility of manipulating abstract images, of composing images
into new patterns, creating icons, and attaching animations and video
to the icons will provide a powerful alternative to sketching by hand.
Hypothesis 9. Abstract images will be useful for generic training.
Oftentimes, training is given on a specific device, when there are many
variants of the device in use. Training is then fragile and brittle, and
will not transfer easily to the new (or more often, old) device
configuration. Training with abstract images of devices will help
transfer skills across families of devices.
Hypothesis 10. Abstract images will be useful for creating analogies
and "mental models". The flow of electricity is often visualized as the
flow of water. The solar system is often seen as a model for atomic
structures. These family resemblances based on perceptual properties
will be derivable from the construction of abstract images.
Hypothesis 11. Graphs and charts for numeric data will be composable
from abstract images. Instead of examining a curve that relates the
strength of two variables in lines in an abstract cartesian plane,
dynagraphs with abstract imagery can create an animated
demonstration of the relationship that can be seen, heard, and felt.
Hypothesis 12: CyberSpace techniques add kinaesthesis, sound, depth,
and realism to visual abstract imagery.
Yet, when asked, hypothetically, whether they would include a diagram if giving such information, most say yes. Furthermore, when asked how they would like to be given such information, people say they would like a diagram. .D3 "Wright\ \'89"
One dimension of grouptext is communication between people. The telephone which was developed in 1876 is an early example of an electrical aid to communication. The use of telephone connections between computers supports electronic mail. The combination of .D1 "telephone" links with .D1 "television" links allows people to hear and see one another at the same time. However, the combination of technologies to support electronic communication does not always lead to the beneficial effects that might be predicted.
The first commercial application of the telephone plus television was called the "PicturePhone" . When AT&T introduced the PicturePhone at the 1964 World's Fair the product was expected to sell very well. Julius Molnar, executive vice-president of Bell Laboratories wrote in the Bell Laboratories Record in 1969:
Rarely does an individual or an organization have an opportunity to create something of broad utility that will enrich the daily lives of everybody. Alexander Graham Bell with his invention of the telephone in 1876, and the various people who subsequently developed it for general use, perceived such an opportunity and exploited it for the great benefit of society. Today there stands before us an opportunity of equal magnitude
Regular users of PicturePhone over the network between Bell Laboratories and AT&T's headquarters agreed that conversations over PicturePhone conveyed important information over and above that carried by voice alone. The enthusiasm for PicturePhone from its creators at AT&T was not, however, shared by other users. These new users felt self-conscious about being on television and did not feel whatever value was gained by the extra information outweighed the equipment or social costs. A special issue of the in 1969 devoted to telecommunications described