Spatial Schemas in Depictions

Barbara Tversky
Stanford University




Department of Psychology, Bldg. 420
Stanford University
Stanford, California 94305-2130
Voice: (650) 725-2440
Fax: (650) 725-5699
email: bt@psych.stanford.edu

In press in M. Gattis, Editor, Spatial schemas and abstract thought. 
Cambridge: MIT Press.  The author is grateful to Merideth Gattis and 
five anonymous reviewers for helpful comments on an earlier draft, 
and to Rochel Gelman, Jeff Zacks, and Julie Bauer Morrison for 
discussions on the general ideas.  Some of the research was supported 
by grants from Interval Research Corporation, by NSF-IST Grant 
8403273 to Stanford University, and by grants from the Human 
Development Institute and the Cognitive Psychology Institute of the 
Hebrew University.


  Abstract:
Depictions, such as maps, that portray visible things are ancient 
whereas graphics, such as charts and diagrams, that portray things 
that are inherently not visible, are relatively modern inventions. 
An analysis of historical and developmental graphic inventions 
suggests that they convey meaning by using elements and space 
naturally. Elements are based on likenesses, "figures of depiction" 
and analogs to physical devices.  Spatial relations are used 
metaphorically to convey other relations, based on proximity, at 
nominal, ordinal, and interval levels.  Graphics serve a variety of 
functions, among them, attracting attention, supporting memory, 
providing models, and facilitating inference and discovery.


Spatial Schemas in Depictions
Barbara Tversky
Stanford University

1. Introduction:  A Brief History of Graphics.

We take it for granted that human beings have crafted tools to 
improve their physical well-being from the beginning of humanity. 
More recently, it has become apparent that other primates do the same 
(e. g., Goodall, 1986; Whiten, Goodall, Mc Grew, Nishida, Reynolds, 
Sugiyama, Tutin, Wrangham, and Boesch, 1999).  These tools augment 
our physical capacity, they allow us to obtain food and protection 
more effectively than our bodies alone could do.  Less noticed is the 
fact that we craft tools to augment our mental capacity as well (e. 
g., Donald, 1991; Norman, 1993).  The fabrication of cognitive tools 
by other primates has not been shown.  One of the oldest of these 
tools is graphics, from maps in the sand to neon on the Ginza.  The 
key to these is using space and elements in it to convey meaning.

Long before there was written language, there were depictions, of 
myriad varieties.  Only a few of the multitude of cave paintings, 
petroglyphs, bone incisions, graffiti, clay impressions, stone 
carvings, and wood markings that people created and used remain from 
ancient cultures. Some of these depictions probably had religious 
significance, but many were used to communicate, to keep track of 
events in time, to note ownership and transactions of ownership, to 
map places, and to record songs and sayings (e. g., Coulmas, 1989; De 
Francis, 1989; Gelb, 1963; Mallery, 1893/1972; Schmandt-Besserat, 
1992).  As such, they served as permanent records of history, 
commemorations of cultural past.  Because pictures represent meaning 
more directly than alphabetic written languages, we can guess at 
their meanings today.  In rare cases, we have the benefit of 
contemporaneous translations.  While collecting petroglyphs and other 
examples of pictographic communications from native Americans in the 
last century, Mallery was able to speak with many still using them 
(1893/1972).  Several of these left by a Native American who lived 
near Long Lake in Maine in the mid-nineteenth century appear in 
Figure 1.  They are leave-taking notices posted on bark to inform 
visitors where he had gone.
-------------------------
  Figure 1 about here
------------------------
In many places in the world, the use of pictures to communicate 
developed into complete written languages.  All such languages 
invented ways to represent concepts that are difficult to depict, 
such as proper names, abstract entities, causality, quantification, 
negations, and the like as (e.  g., Coulmas, 1989; DeFrancis, 1989; 
Gelb, 1963).  As pictures evolved into written languages, two things 
occurred: they became schematized and they lost their transparency. 
Figure 2 shows how Sumerian characters that began as fairly 
transparent sketches of the concepts they conveyed became schematized 
in writing and lost any resemblance to those concepts.  Some 
originally pictoric written languages transformed to use written 
marks to represent sound rather than meaning directly.  This had two 
simultaneous advantages: it reduced the numbers of pictographs that 
had to be learned and it solved the problem of representing concepts 
difficult to depict.  The spread of the alphabet and, much later, the 
invention of the printing press decreased reliance on pictures for 
communication.  With the increasing ease of reproducing written 
language and increasing literacy, depictions became more decorative 
than communicative.
-------------------------
  Figure 1 about here
------------------------

Now, pictures, depictions, and visualizations as communications are 
on the rise again.  As with the proliferation of written language, 
this is partly due to technologies for creating, reproducing, and 
transmitting pictures.  And as with the proliferation of written 
language, some of the spread of depictions is due to intellectual 
insights.  For this, the basic insight is using depictions to 
represent abstract meaning by means of visual and spatial metaphors 
and figures of depiction. Using space also capitalizes on people's 
extensive experience learning about space and making spatial 
inferences.
-------------------------
  Figure 1 about here
------------------------

Although space in depictions has long been used to convey concrete 
ideas, the use of space to convey abstract ideas is more recent. 
Early uses of space in depictions for the most part portrayed things 
that were inherently visualizable, such as objects, buildings, or 
environments, in pictographs, architectural plans, or maps.  Maps 
appear in many forms, many taking liberty with spatial metrics, such 
as the T-O maps popular in medieval times, so-called because they 
were O-shaped, with an internal T.  An example appears in Figure 3. 
In these maps, East, the Orient, the rising sun, Asia, was at the 
top, above the crossbar of the T, which was formed by the Dan and 
Nile Rivers.  Below the crossbar to the left was Europe and to the 
right, Africa, separated by the Mediterranean forming the vertical 
bar of T.  Maps are necessarily schematic; at the least, they omit 
information, presumably retaining the information relevant to their 
expected use.  Maps may even mix perspectives, as in landmark maps, 
appearing as early as medieval times, where countries, rivers, and 
roads are laid out as if from above, with landmarks portrayed 
frontally superimposed.  Figure 4 shows one of these (for more 
examples of maps ancient and modern, see Brown, 1979; Harley and 
Woodward, 1987, 1992, 1994; Southworth and Southworth, 1982). In 
contrast to these ancient depictions, many contemporary depictions 
are visualizations of things that are not inherently visualizable, 
such as temporal, quantitative, causal, or social relations.  These 
depictions depend on analogy rather than schematic miniaturization or 
enlargement.


-------------------------
  Figure 3 about here
------------------------
-------------------------
  Figure 4 about here
------------------------

Graphs are perhaps the most prevalent example of depictions of 
abstract concepts, and were invented as recently as the late 
eighteenth century (e. g., Beniger and Robyn, 1978; Carswell and 
Wickens, 1988; Tufte, 1983), although they probably had their roots 
in mathematical notation, especially Cartesian coordinate systems. 
Two Europeans, Playfair in England and Lambert in Switzerland, are 
credited with being the first to promulgate their use, for the most 
part to portray economic and political data.

Notably, those early graphs, X-Y plots with time as one of the 
variables, are still the most common type of graph in scientific 
journals (Cleveland, 1984).  This is despite the continuing clever 
inventions of visualizations from Florence Nightingale's polar plots 
of mortality across hospitals (Cohen, 1984) to box plots and 
stem-and-leaf diagrams (Tukey, 1977) to starfields, cone trees, and 
hyperbolic trees (e. g., Card, Mackinlay, and Shneiderman, 1999) to 
the latest presentations at the InfoVis meetings or in Newsweek. 
Some of these visualizations have been widely adopted.  Bar graphs 
and pie charts are common for representing quantitative data, with 
flow charts, trees, and networks widely used for qualitative data. 
Icons appear in airports, train stations, and highways all over the 
world, and menus of icons on information highways over the world. 
Many are used to portray concepts and information that are difficult 
to visualize, such as large data bases.  Creating an effective visual 
webrowser continues to challenge the ingenuity of many.

2. How Graphics Convey Meaning

Graphics consist of elements that are systematically arranged in 
space. The choices of elements and spatial arrangements are usually 
not accidental or arbitrary.  Many graphic conventions have been 
invented and reinvented by adults and children across cultures and 
time.  Many have analogs in language and in gesture.  Many are rooted 
in natural cognitive correspondences, "figures of depictions," and 
spatial metaphors, and have parallels in Gestalt principles of 
perceptual organization.  In this paper, I present an analysis of 
graphic inventions based on how their elements and spatial 
arrangements are used to convey meaning.  The evidence I will bring 
to bear is eclectic and unconventional, drawing from examinations of 
historical graphic inventions, children's graphic inventions, and 
language, as well as more conventional psychological research. 
Later, I examine how two contemporary graphic inventions, animation 
and 3-D, communicate, survey the various functions graphics serve, 
and draw conclusions for their design.

With some exceptions, graphic elements are generally used to 
represent elements in the world and graphic space is used to 
represent the relations between elements.  One notable exception is 
mathematics, where elements such as + and - indicate relations.  The 
dichotomy, into elements and relations, maps loosely onto the "what" 
vs. "where" distinction in vision and in spatial cognition, that is, 
information related to objects on the one hand and to spatial 
relations among objects on the other.

The fact that graphic displays are external augments many of their 
functions.  Externalizing a representation reduces demand on memory 
and facilitates information processing.  Spatially organized 
information can be accessed, integrated, and operated on quickly and 
easily, especially when the spatial organization reflects conceptual 
organization. Moreover, external representations are public.  Several 
people can simultaneously inspect the same graphic display, and refer 
to it by pointing and other devices in ways apparent to all, 
facilitating group communication (e. g., Engle, 1998).

2.1. Elements: Figures of Depiction.

Sometimes icons as elements are effective in representing meaning 
directly,  for example, highway signs of a picnic table or a water 
tap on the route  to the location of actual ones.  For concepts not 
easily depicted, icons present challenges.  They can, however, 
represent concepts indirectly, using a number of "figures of 
depiction," analogous to figures of speech (Tversky, 1995).  One 
common type of figure of depiction is metonymy, where an associated 
object represents the concept.  In computer interfaces, a picture of 
a folder represents a file of words and a picture of a trash can 
represents a place for unwanted folders. Analogous examples in 
language include using "the crown" to represent the king and "the 
White House" to represent the president.  Synecdoche, where a part is 
used to represent a whole, or a whole for a part, is another common 
figure of depiction.  Returning to highway signs, an icon of a place 
setting near a freeway exit indicates a nearby restaurant and an icon 
of a gaspump a nearby gas station. Analogous examples in language 
include "give a hand" for help and "head count" for number of people. 
Figures of depiction frequent early pictographic writing (Coulmas, 
1989; Gelb, 1963;).  For example, early Sumerian writing used a foot 
to indicate "to go" and an ox's head to indicate an ox.  Figure 5 
presents early ideographs for several concepts from a number of 
written languages, illustrating metonymy and synecdoche.  Children's 
spontaneous writing and depictions also illustrate these principles 
(e. g., Hughes, 1986; Levin and Tolchinsky-Landsman, 1989).  Like the 
inventors of pictographic languages, children find it easier to 
depict objects, especially concrete ones, than operations.  For 
abstract objects and operations, children use metonymy and 
synecdoche.  For example, children draw hands or legs to indicate 
addition or subtraction.  Interestingly, the latter was also used in 
hieroglyphics (Hughes, 1986).  As seen in the examples of public 
icons in Figures 6 and 7, depicting objects or things is more direct 
than depicting relations among objects.  The public symbols developed 
by a different local, national, and international organizations for 
restaurants and coffee shops are highly similar whereas those 
developed for purchasing tickets or for baggage claim are less so.
-------------------------
  Figure 5 about here
------------------------
-------------------------
  Figure 6 about here
------------------------
-------------------------
  Figure 7 about here
------------------------

The meanings of these depictions are somewhat transparent.  Often, 
they can be guessed, sometimes with help of context.  Even when 
guessing is unsuccessful, the range of meanings possible is highly 
limited.  Because of their transparency, these icons are easily 
associated to their meanings, and thus easily remembered (for similar 
arguments in the context of ASL and gesture, see Macken, Perry and 
Haas, 1993).  The meanings of computer icons cannot always be 
conveyed by a single word. The alternatives, verbal commands, are not 
always transparent and frequently need to be learned.  Does "delete" 
or "remove" eliminate a file?  Do we "exit" or "quit" a program? 
Depictions have other advantages over words.  Meaning is extracted 
from pictures faster than from words (Smith and McGee, 1980).  Icons 
can be "read" by people who do not read the local language.

A new use of depictions has appeared in email.  Seemingly inspired by 
smiley faces, and probably because it is inherently more casual than 
other written communication, computer vernacular has added signs for 
the emotional expression normally conveyed in face-to-face 
communication by intonation and gesture.  These signs combine symbols 
found on keyboards to denote facial expressions, usually turned 90 
degrees, such as :) or ;).

2.2. Spatial Arrays of Elements

Graphs, charts, and diagrams convey qualitative and quantitative 
information using natural correspondences and spatial metaphors, some 
applied to the spatial elements, and others applied to the arrays of 
spatial elements.  For spatial arrays, the most basic  metaphor is 
proximity: proximity in space is used to indicate proximity on some 
other property, such as time or value.  Spatial arrays convey 
conceptual information metaphorically at different levels of 
precision, corresponding to the four traditional scale types, 
nominal, ordinal, interval, and ratio (Stevens, 1946).  These are 
ordered inclusively by the degree of information preserved in the 
mapping.  Spontaneously produced graphic displays reflect these scale 
types.  Children, for example, represent nominal relations in graphic 
displays at an earlier age than ordinal relations, and ordinal 
relations at an earlier age than interval relations (Tversky, 
Kugelmass, and Winter, 1991).  For elements, graphic displays use a 
variety of visual analogs to physical devices to convey information 
of varying quantitative precision.

2.21. Nominal scales are essentially clustering by category. Here, 
elements are divided into classes, sharing a single property or set 
of properties.  Graphic devices indicating nominal relations often 
use the simplest form of proximity, grouping.  This form, like many 
of the uses of spatial arrangement to convey meaning, capitalizes on 
Gestalt principles of perceptual grouping.  In perception, things 
that are near by in space tend to be grouped and separated from 
things that are distant. To use this for conveying abstract meanings 
simply requires placing things that are related in close proximity 
and placing things that are not related farther away in space. One 
use of this device that we take for granted are the spaces between 
words in writing.  Although it is easy to overlook spacing as a 
graphic device, early writing did not put spaces between words.  The 
Roman alphabet for the most part maps sounds to written characters, 
but beyond the alphabet, many conventions of writing are based on 
spatial correspondences, not sound correspondences.  Indentation 
and/or spacing before a new paragraph is another example of using 
separation in space to indicate separation of ideas.  Some of the 
spatial and visual features used to convey nominal and ordinal 
relations appear in Table 1.

Another spatial device for delineating a category is a list, where 
all the items that need to be purchased or tasks that need to be done 
are written in a single column or row.  Items are separated by empty 
space, and the items begin at the same point in each row, indicating 
equivalence.  For lists, there is often only a single category; 
organization into a column indicates that the items are not randomly 
selected, but rather, share a property.  Multiple lists are also 
common, for example, the list of chores of each housemate or a 
shopping list divided by departments or stores.  A table is an 
elaboration of a list, using the same spatial device to organize both 
rows and columns (Stenning and Oberlander, 1995).  A table 
cross-classifies items by several categories simultaneously.  Items 
within each column and within each row are related, but on different 
features.  In many cases, tables combine categorical and quantitative 
information.  Examples include a list of countries with their GNP's 
for each of the last ten years, or a list of schools, with their 
average achievement scores on a variety of tests. Tables 
cross-classify.  Items within each column and within each row are 
related, but on different features.  Train schedules are yet another 
example, where the first column is typically the stations, the places 
where the train stops, and subsequent columns are the times for each 
train.  For train schedules, a blank space where there would 
ordinarily be a time indicates a non-event, that is, this train 
doesn't stop at that station.  Using spatially-arrayed rows and 
columns, tables group and juxtapose simultaneously.

Empty space is not the only spatial device used to indicate grouping. 
Often special signs, usually visual ones rather than strictly spatial 
ones, are added.  These seem to fall into two classes, those based on 
enclosure (cf. Gestalt Principle of Grouping) and those based on 
similarity (cf. Gestalt Principle of Similarity).  Signs used for 
enclosure resemble physical structures that enclose actual things, 
such as bowls and fences.  On paper, these include parentheses, 
circles, boxes, and frames.  Tables often add boxes to emphasize the 
structures of rows and columns or to enclose related items and 
separate different ones.  Newspapers use frames to distinguish one 
classified ad from another.  Parentheses and brackets in writing are 
in essence degenerate enclosures  The curved or bent lines, segments 
of circles or rectangles, face each other to include the related 
words and to separate them from the rest of the sentence.

Complete circles have been useful in visualizing syllogisms and in 
promoting inference as in Euler or Venn diagrams or in contemporary 
adaptations of them (e. g., Shin, 1991; Stenning and Oberlander, 
1995). Circles enclose items belonging to the same set.  Circles with 
no physical contact indicate sets with no common items, and 
physically overlapping circles indicate sets with at least some 
common items.  The classic Euler circles have been used as a 
reasoning heuristic as they are not sufficient for actual deduction. 
There are ongoing attempts to develop completely diagrammatic 
reasoning systems.  One attempt has been to enrich Euler diagrams 
with spatial signs based on similarity, such as filling in similar 
regions with similar and dissimilar regions with different marks, 
color, shading, cross-hatching, and other patterns (e. g., Shin, 
1991).  Others have been developing heterogeneous reasoning systems 
based on a a combination of graphics and language, utilizing each for 
the contents and contexts in which inferences are easiest (Barwise 
and Etchemendy, 1995).

Other visual features of diagrams can be used to convey categorical 
relations.  Maps use colors as well as lines to indicate political 
boundaries and geographic features.  For geographic features, many of 
the correspondences are natural ones.  For example, deserts are 
colored beige whereas forests are colored green, and lakes and seas 
are colored blue.  Colors can also be used to convey quantitative 
information, for example, deeper (darker) blues indicating deeper 
water.  Color categories and gradations can be used metaphorically as 
well.  Examples include the use of colors to indicate degree of 
excitation in PET or fMRI images of the thinking brain.  In addition 
to color, other visual features, such as size, shape, and font, can 
be used to signify groups based on similarity.

2.22. Ordinal relations can vary from a partial order, where one or 
more elements have precedence over others, to a complete order, where 
all elements are ordered with respect to some property or properties. 
There are two separable issues in mapping order onto space.  One is 
the devices used to indicate order, and the other is the direction of 
order. The direction of indicating order will be discussed after 
interval relations, as the same principles apply.

Now to ways of indicating order, first using space.  Many of the 
devices used to indicate groupings can be adapted to indicate order. 
Writing is ordered, so one of the simplest spatial devices to 
indicate rank on some property is to write items according to the 
order on the property, for example, writing countries in order of 
GNP, or people in order of age. Empty space is used to convey order, 
as in indentation in outlines, where successively subordinate items 
are successively indented relative to superordinate items.

Visual devices are also useful for indicating order. Primary among 
them are lines.  Lines form the skeletons of trees and graphs, both 
of which are commonly used to display ordered concepts, to indicate 
asymmetry on a variety of relations, including kind of, part of, 
reports to, and derived from.  Examples include hierarchical 
displays, as in linguistic trees, evolutionary trees, and 
organizational charts.  Other visual and spatial devices used to 
display order rest on the metaphor of salience. More salient features 
have more of the relevant property.  Such features include size, 
color, highlighting, and superposition.  Some visuo/spatial devices 
rely on what can be called natural cognitive correspondences.  For 
example, high temperatures or greater activity are associated with 
"warm" colors and low temperatures and lower activity with "cold" 
colors, in scientific charts.  This association most likely derives 
from the colors of things varying in temperature, such as fire and 
ice.

Arrows are a special kind of line, with one end marked, inducing an 
asymmetry.  Although they have many uses, a primary one is to 
indicate direction, an asymmetric relation. Like bars and lines, 
arrows seem to be based on physical analogs. One obvious analog is 
the physical object arrow, invented by many different cultures for 
hunting.  It is not the hunting or piercing aspects of physical 
arrows that have been adopted in diagrams, but rather the 
directionality.  Hunting arrows are asymmetric, as a consequence of 
which they fly more easily in one direction than the other.  Another 
more abstract analog is the idea of convergence captured by the > 
("V") of a diagram arrow.  Like a funnel or river straits, it directs 
anything captured by the wide part to the point, and straight 
outwards from there.  Arrows are frequently used to signal direction 
in space.  In diagrams, arrows are also commonly used to indicate 
direction in time.  Production charts and computer flow diagrams, for 
examples, use arrows to denote the sequence of processes. Terms for 
time, such as "before" and "after," and indeed thinking about time, 
frequently derive from terms for and thinking about space (e. g., 
Boroditsky, 1999; Clark, 1973; Gentner, this volume).

2.23. Interval and ratio relations apply more constraints of the 
spatial proximity metaphor than ordinal relations.  In graphic 
displays of interval information, the distances between elements are 
meaningful; that is, greater space corresponds to more on the 
relevant dimension.  This is not the case for ordinal mappings. 
Ratio displays of information, are even more restrictive.  For them, 
zero and the ratios of the distances are meaningful.

The most common graphic displays of interval and ratio information 
are X-Y plots, where distance in the display corresponds to distance 
on the relevant property or properties.  Musical notation is a 
specialized interval scale that makes use of a limited visual 
alphabet corresponding to modes of execution of notes as well as a 
spatial scale corresponding to pitch.  Finally, for displaying ratio 
information, pie charts can be useful, where the area of the pie 
corresponds to the proportion on the relevant variable.

2.24. Expressing mixed cases: Bars and lines. Most graphs display two 
variables simultaneously, average rainfall over months, school 
achievement scores by district, chocolate ice cream sales by stores, 
and so on.  Bars charts are a common way to convey quantities of 
discrete sets.  They are useful for displaying quantities for several 
variables at once where the height or length of the bar corresponds 
to the quantity on the relevant variable.  Isotypes were invented by 
Otto and Marie Neurath in the 30's as part of a larger movement to 
increase communication across languages and cultures.  That movement 
included efforts to develop picture languages and Esperanto. 
Isotypes combine icons and bar charts to render quantities on 
different variables more readily interpretable (Neurath, 1936). For 
example, in order to display the yearly productivity by sector for a 
number of countries, a unit of output for each sector is represented 
by an isotype, or icon that is readily interpretable, a shaft of 
wheat for units grain, an ingot for units of steel, an oil well for 
units of petroleum.  The number of icons per sector is proportional 
to output in that sector.  Because of their physical resemblance to a 
product of each sector, icons facilitate comparison across countries 
or years for the sectors.

When two variables are displayed simultaneously, one may be 
categorical and the other quantitative.  Going from categorical 
grouping to conceptual ordering requires a conceptual leap.  Grouping 
entails seeing that items in Group A share one or more properties and 
items in Group B share one or  more different properties.  Group A 
and Group B do not have to be related in any way; they are the 
proverbial "apples and oranges." Different school districts and 
different ice cream stores can be regarded as separate and different 
entities with different properties. Ordering entails seeing that 
Group A and Group B differ on a property (or properties) that itself 
differs systematically.  Specifically, the systematicity underlying 
an order requires a property that is similar enough across groups 
that it can be arranged as including more or less of the property. 
Put differently, categorical relations depend on within group 
property similarity and between group property difference. Ordinal 
relations depend on graded property similarity between groups. Note 
that the division into categorical and quantitative relations depends 
in part on perspective.  Let us return to the school district 
example.  One way of looking at separate school districts is as 
separate entities.  Another way of looking at them is to order them 
by parental income, thus turning a categorical variable in to a 
(sensible) quantitative one.

Categorical relations separate and ordinal ones connect.  Visual 
devices that indicate boundaries or containers are natural analogs 
for categories just as visual devices that indicate connections are 
natural analogs for orders.  Visual boundaries and containers include 
the brackets and circles and frames discussed in the section on 
categorical relations.  Like those, visual devices that indicate 
connections resemble physical structures that link things, like 
outstretched arms, paths, chains, and ropes.  Like paths or 
outstretched arms, lines link one concept to another, bringing 
noncontiguous things into contiguity, making distal items proximal. 
Lines, sometimes whole and continuous  ( _____ ), sometimes partial 
or broken ( ....... ) are used to link related items.

That categorical or discrete concepts are naturally mapped to 
entities that contain and ordinal concepts are naturally mapped to 
entities that connect seems to underlie the use and interpretation of 
bars and lines in graphs (Zacks and Tversky, in press).  Bars are 
container-like and lines connect.  Zacks and Tversky (in press) 
distinguished two related uses. The use that is statistical lore, and 
sometimes explicit advice, is the bar-line data use: display discrete 
data with bars and continuous data with lines.  But we have already 
seen that the same data can be viewed in different ways.  The more 
subtle use is the bar-line message use: interpret/produce bars as 
discrete comparisons and lines as trends.

Evidence not only for data use but also for message use of bars and 
lines appeared in graphic productions and interpretations.  One set 
of participants wrote interpretations for graphs of height of 
males/females, a discrete domain, or 10 year olds/12 year olds, a 
continuous domain.  Examples appear in Figure 8. Discrete 
interpretations used terms like "higher,"  "less," and "fewer," 
whereas trend interpretations used terms like "rising" and 
"decreasing," and sometimes "function," "relationship," and "trend." 
More line graphs were interpreted as trends than bar graphs, 
consistent with the bar/line message use.  More trend interpretations 
were produced for the continuous domain than for the discrete domain, 
consistent with the bar/line data use.  Another set of participants 
produced graphs for discrete or trend descriptions of height in 
males/females or 10 year olds/12 year olds.
-------------------------
  Figure 1 about here
------------------------

As for interpretations, productions showed the effects of both data 
and message uses of bars and lines.  More line graphs were produced 
for trend descriptions that used terms like "increasing" than for 
discrete descriptions that used terms like "greater", supporting the 
bar/line message use.  More trend descriptions were produced for the 
continuous (age) domain than the discrete (sex) domain, supporting 
the bar/line data use.  Overall, the effects of the graph form, bar 
or line, were stronger than the effects of the conceptual domain, 
discrete or continuous.

These correspondences between choice of graphic representation and 
choice of interpretation appeared despite the fact that few people in 
the informant population are aware of the bar/line data use, and even 
fewer of the bar/line message use.  The choice of visual devices for 
discrete, categorical concepts and for ordinal or continuous ones 
appears to be naturally derived from physical devices that contain or 
connect.

2.25. Directionality.
In spite of the uncountable number of possibilities for indicating 
order in graphic displays, the actual choices are remarkably limited. 
In principle, elements could be ordered in any number of orientations 
in a display.  Nevertheless, graphic displays tend to order elements 
either vertically or horizontally or both.  Similarly, languages are 
written either horizontally or vertically, in rows or in columns. 
There are reasons grounded in perception for the preference for 
vertical and horizontal orientations. The perceptual world has two 
dominant axes:  a vertical axis defined by gravity  and by all the 
things on earth correlated with gravity; and a horizontal axis 
defined by the horizon and by all the things on earth parallel to it. 
Vision is especially acute along the vertical and horizontal axes 
(Howard, 1982).  Memory is poorer for the orientation of oblique 
lines, and slightly oblique lines are perceived and remembered as 
more vertical or horizontal than they were (Howard, 1982; Tversky, 
1981; for exceptions that depend on graphic interpretation, see 
Schiano and Tversky, 1992).

Vertical asymmetry.  Of all the possible orientations, then, graphic 
displays ordinarily only use the vertical and horizontal.  What's 
more, they use these orientations differently. Vertical arrays take 
precedence over horizontal ones. Just as for the choice of 
dimensions, the precedence of the vertical is also rooted in 
perception (Clark, 1973; Cooper and Ross, 1975; Lakoff and Johnson, 
1980; Franklin and Tversky, 1990).  Gravity is correlated with 
vertical, and people are oriented vertically.  The vertical axis of 
the world has a natural asymmetry, the ground and the sky, whereas 
the horizontal axis of the world does not.   The dominance of the 
vertical over the horizontal is reflected in the dominance of columns 
over rows.  Similarly, bar charts typically contain vertical columns 
(the exceptions seem to be aimed at conforming to accompanying text).

Natural correspondences with vertical. There is another plausible 
reason for the dominance of the vertical over the horizontal.  Not 
only does the vertical take precedence over the horizontal, but there 
is a natural direction of correspondence for the vertical, though not 
for the horizontal.  In language, concepts like more and better and 
stronger are associated with upward direction, and concepts like less 
and worse and weaker with downward direction (Clark, 1973; Cooper and 
Ross, 1975; Lakoff and Johnson, 1980).  People and plants, indeed 
most life forms, grow upwards as they mature, becoming bigger, 
stronger, and (arguably) better.  Healthy and happy people stand 
tall; sick or sad ones droop or lie down.  More of any quantity makes 
a higher pile.  The associations of up with quantity, mood, health, 
power, status, and more derive from physical correspondences in the 
world.  It is no accident that in most bar charts and X-Y plots, 
increases go from down to up.  The association of all good things 
with up is widely reflected in language as well. Inflation and 
unemployment are exceptions, but principled ones, as the numbers used 
to express inflation and unemployment go up.  Language reflects these 
natural correspondences.  We speak of someone "at the top of the 
heap," of doing the "highest good," of "feeling up," of being "on top 
of things," of having "high status" or "high ideals," of doing a 
"top-notch job," of reaching "peak performance," of going "above and 
beyond the line of duty."  In gesture, we show success or approval 
with thumbs up, or give someone a congratulatory high five.  The 
correspondence of pitch with the vertical seems to rest on another 
natural relation.  We produce higher notes at higher places in the 
throat, and lower notes at lower places.  It just so happens that 
higher notes correspond to higher frequency waves, but that may 
simply be a happy coincidence.

Horizontal neutral. In contrast, the horizontal axis is standardly 
used for neutral dimensions, most notably, time.  Similarly, with few 
exceptions (curiously, economics), neutral or independent variables 
are plotted along the horizontal axis, and the variables of interest, 
the dependent variables, along the vertical axis.  This arrangement 
of variables facilitates inferences of rates of change from line 
slopes (Gattis and Holyoak, 1996).  Although graphic conventions 
stipulate that increases plotted horizontally proceed from left to 
right, directionality along the horizontal axis does not seem to rest 
in natural correspondences. The world is asymmetric along the 
vertical axis, but not along the horizontal axis.  Right-left 
reflections of pictures are hardly noticed but top-bottom reflections 
are (e. g., Yin, 1969). Languages are just as likely to be written 
left to write as right to left (and in some cases, both), but they 
always begin at the top.

Use of space cross-culturally and developmentally. Children and 
adults from cultures where language is written left to right as well 
as from cultures where language is written right to left mapped 
increases on a variety of quantitative variables from down to up, but 
almost never mapped increases from up to down.  However, people from 
both writing cultures mapped increases in quantity and preference 
from both left to right and right to left equally often.  The 
relative frequency of using each direction to represent quantitative 
variables did not depend on the direction of written language 
(Tversky, Kugelmass, and Winter, 1991).  Despite the fact that most 
people are right-handed and that terms like dexterity derived from 
"right" in many languages have positive connotations and terms like 
sinister derived from "left" have negative connotations, the 
horizontal axis in graphic displays seems to be neutral.  Consistent 
with that, we refer to one side of an issue as "on the one hand," and 
the other side as "on the other hand," which has prompted some 
politicians to ask for one-handed advisors.  And in politics, both 
the right and the left claim the moral high ground.

Children's and adults' mappings of temporal relations showed a 
different pattern from their mappings of quantitative and preference 
relations (Tversky, Kugelmass, and Winter, 1991).  For time, they not 
only preferred to use the horizontal axis, they also used the 
direction of writing to determine the direction of temporal 
increases, so that people who wrote from left to right tended to map 
temporal concepts from left to right and people who wrote from right 
to left tended to map temporal concepts from right to left.  This 
pattern of findings fits with the claim that neutral concepts such as 
time tend to be mapped onto the horizontal axis.  The fact that the 
direction of mapping time corresponded to the direction of writing 
but the direction of mapping preference and quantitative variables 
did not may  be because temporal sequences seem to be incorporated 
into writing more than quantitative concepts, for example, in 
schedules, calendars, invitations, and announcements of meetings.

Directionality in charts and diagrams. Compatible with the previous 
arguments and evidence, ordinal charts and networks tend to be 
vertically organized.  A survey of the standard scientific charts in 
all the textbooks in biology, geology, and linguistics at the 
Stanford Undergraduate Library revealed vertical organization in all 
but two of 48 charts (Tversky, 1995).  Furthermore, within each type 
of chart, there was agreement as to what appeared at the top.  In 17 
out of the 18 evolutionary charts, Homo sapiens, that is, the present 
age, was at the top.  In 15 out of the 16 geological charts, the 
present era was at the top, and in 13 out of the 14 linguistic trees, 
the proto-language was at the top.  In these charts, in contrast to 
X-Y graphs, time runs vertically, but time does not seem to account 
for the direction, partly because time is not ordered consistently 
across the charts.  Rather, at the top of each chart is an ideal.  In 
the case of evolution, it is humankind, regarded by some as the 
pinnacle of evolution, a view some biologists discourage (e. g., 
Gould, 1977).  In the case of geology, the top is the richness and 
accessibility of the present era.  In the case of language trees, the 
top is the proto-language, the most ancient theoretical case, the 
origin from which others diverged.  In organizational charts, say of 
the government or large corporations, power and control are at the 
top.  For diagramming sentences or the human body, the whole is at 
the top, and parts and sub-parts occupy lower levels.  In charts such as these, the vertical relations are meaningful, denoting an asymmetry 
on the mapped relation, but the horizontal relations are often 
arbitrary.

3. Modern Graphics:  3D and Animation.

Advances in computers have made more graphic devices available to 
professionals and amateurs alike, most notably, 3D and animation. 
Although both can be useful, they should come with questions:  First, 
how well are these devices perceived and conceived?  And second, what 
sorts of meanings do they naturally convey?

	3.1 3-D

Different forms of 3-D have been used by artists all over the world. 
Post-Renaissance Western art has favored 1-point perspective, though 
modern and contemporary Western art seems to have abandoned that. 
Eastern art, as well as children's spontaneous art, have used other 
devices to convey depth, such as occlusion, size, and height in the 
picture plane (e. g., Braine, Schauble, Kugelmass, and Winter, 1993; 
Willats, 1997).

For 3D, the first thought is that it is a natural for conveying 
information, concrete or abstract, that is inherently 
three-dimensional. Some encouragement for that view comes from work 
on drawings of objects. People naturally interpret object-like 
drawings, those, for example, with closed contours and hints of 
symmetry, as two-dimensional representations of three-dimensional 
objects (Hochberg, 1964; McBeath, Schiano, and Tversky, 1997). 
Perceiving and comprehending the 3-D details, however, can be 
problematic from flattened images.  Such perceptual difficulties 
abound, especially for odd views or scenes with multiple objects. 
The retina itself is essentially flat, and beyond small distances, 
perception in depth depends on a set of cues such as occlusion, 
foreshortening, and relative size rather than on stereoscopic vision. 
Those clues are only clues, however; they are fallible .  Their 
limitations yield error in depth perception and in object recognition 
alike (e. g., Loomis, DaSilva, Fujita, and Fukusima, 1992).  3D 
interfaces suffer from exactly the same limitations.  Even within the 
bounds of stereoscopy, only one view of an object or scene is present 
at once.  This means that mental representations of the three 
dimensions must be constructed from separate views.  It also means 
that some objects may be occluded by others, as frequently happens in 
3-D depth arrays of bars.  Frontal views of 3-D bars often become 
reversible figures, hard to stabilize in order to inspect.
The difficulties of conceptualizing in three dimensions from a flat 
display make it no accident that maps of the three-dimensional world 
are schematized to two dimensions.  Nor is it an accident that 
architects and engineers prefer to design two-dimensional planes of 
their projects before proceeding to three-dimensional representations 
(e. g., Arnheim, 1977; Suwa and Tversky, 1996).  Not just production, 
but also interpretation of 3D graphics is difficult for novices (e. 
g., Cooper, Schacter, Ballesteros, and Moore, 1992; Gobert, 1999). 
Similarly, constructing three-dimensional conceptual representations 
from two-dimensional graphic representations is also difficult (e. 
g., Shah and Carpenter, 1995).  When asked to provide the information 
conveyed in a 2-D line graph portraying three variables, respondents 
described the relations between the in terms of the variables plotted 
on the X and Y axes rather than in terms of the indirectly 
represented Z axis.

Although 3D is popular, it is hard to perceive and hard to conceive, 
even when used for objects and scenes that are naturally three 
dimensional.  But the proliferation of graphics software has allowed 
creation of 3D graphics even for things that are not naturally 3D, 
most notably 3-D bar graphs.  This practice has been decried by many 
(e. g., Tufte, 1983; Wainer, 1980).  The rising cost of oil, for 
instance, has been portrayed by proportionately larger oil barrels. 
Only the relative heights of the barrels are meaningful, but viewers 
cannot help but respond to the areas or implied volumes.  Despite the 
difficulties of 3-D, users express preferences for 3D bars or lines 
in certain situations, for example, for showing data for others 
rather than self, or for remembering data as opposed to examining it 
(Levy, Zacks, Tversky, and Schiano, 1996).  Some of these intuitions 
are not off the mark.  Estimates in perception from 3D bars are 
indeed worse than from old-fashioned 2D bars,  but estimates in 
memory are no worse from 3D than from 2D bars (Zacks, Levy, Tversky, 
and Schiano, 1998).

	3.2. Animation.

Animation, graphics that change, can be eye-catching, as web 
advertisers have discovered.  For conveying real-time on-line 
processes, simple animations can be informative.  Take for example, 
the bar that appears when accessing a website.  It fills at the rate 
that information is downloaded, allowing the user to estimate how 
much time it will take to acquire the information.  Because animation 
entails changes in graphics over time, it seems a natural for 
conveying actions and processes that are inherently dynamic or that 
take place over time.  This should hold for both actions that are 
concrete, such as the motions of people and machines and for 
processes that are more abstract, such as weather systems or the 
temporal/causal relations of algorithms.  However, as for 3D, this 
view does not take into account human perceptions and conceptions of 
dynamic events.  Animations vary in complexity.  One simple animation 
is the path of a point or object.  Another is the sequential 
highlighting of parts of a display.  Movements of two- or 
three-dimensional bodies, especially an irregular ones, and movements 
along oblique or irregular paths are more complex.  Even more complex 
animations include the movements of parts in relation to one another, 
such as a system of gears or pulleys or the changes in position of 
the limbs during walking.  Still more complex are the movements of 
two bodies or systems with respect to one another.  For an animation 
to be effective, it first must be correctly perceived.

Many dynamic events, especially complex ones, take place too quickly 
for human observers to perceive.  For example, generations of artists 
painted the legs of galloping horses incorrectly until Muybridge's 
(1957) photographs revealed the correct configuration.  This is 
complex motion, parts moving in complex patterns relative to one 
another.  Yet there is evidence that even the simple path of a 
pendulum is often misperceived.  Observers who select the correct 
trajectory from an animation may nevertheless fail to draw the 
correct trajectory (Kaiser, Profitt, Whelan, and Hecht, 1992). 
What's more paths or trajectories of moving objects are often 
distorted, perceived as closer to horizontal or vertical than they 
actually are (e. g., Pani, Jeffres, Shippey, and Schwartz, 1996; 
Shiffrar and Shepard, 1991).  Even events that are slow enough to be 
accurately perceived may be conceived of as discrete steps rather 
than as continuous movement (e. g., Zacks and Tversky, 1999).  In 
verifying assertions about the behavior of pulley systems, people 
mentally animate the systems one pulley at a time in sequence rather 
than imagining the entire system in action (Hegarty, 1992).  Put 
differently, people conceive of the continuous action of a pulley 
system as a sequence of discrete steps.

In addition to the difficulties of perceiving them accurately. 
animations have a substantive disadvantage relative to static 
diagrams. Like speech, animations are transient, ephemeral, and not 
readily reinspected.  A series of static diagrams, freeze frames, can 
simultaneously portray the sequential stages of an action, like the 
galloping of a horse's legs or the steps in operating a coffee maker. 
Similarly, an annotated diagram, such as a flow chart, can show the 
successive steps of a complex action all at once.  Compared to 
animations, static diagrams can more easily be inspected and 
reinspected, related and interrelated to insure understanding.

Misperceptions of dynamic events and conceptions of them as sequences 
of steps rather than continuos activity may account for many of the 
failures to find benefits of animation in conveying information, concrete or abstract (e.  g., Kaiser, Proffitt, Whelan, and Hecht 1992; 
Hegarty, Narayanan, Cate, and Holmquist, 1999; Lowe, 1999; Pane, 
Corbett, and John, 1996; Rieber, Boyce, and Assad, 1990; Stasko and 
Lawrence, 1998). There are, however, some cases where animation has 
been more successful than static diagrams, and these cases are 
instructive.  One notable case is in teaching speed = distance x time 
problems, where motion was abstracted to a moving point (Baek and 
Lane, 1988).  Two points starting in the same place moving at 
different speeds and arriving at different stopping points were 
simple enough to perceive and interpret.  In this case, the animation 
is not only simple, it also abstracts the information essential to 
solving the problems, the differences in rates of movement.  Since 
the rates are constant and the path a straight line, the events are 
perceived as continuous, not likely to be conceptually segmented. 
Another successful application of animation also used simple 
animations, in this case, sequentially highlighting parts of a 
diagram to guide viewers' successive foci of attention.  This 
entailed a discrete rather than continuous use of animation. 
Diagrams can be visually complex, but unlike for reading text, there 
is no established order of scanning them.  Sequential highlighting 
can provide a sensible order of integrating parts of a diagram.  It 
borrows a device common in language, using linear order to reveal 
underlying structure.  For example, cities scattered spatially on a 
map can be organized by historical era or by size.  Sequential 
highlighting indeed facilitated organization of a diagram by the 
highlighted features, determining the character of the mental 
representation.  However, it had no benefits to memory relative to 
static diagrams (Betrancourt and Tversky, 1999). Note that each of 
these animations is perceptually simple, the smooth horizontal 
movement of a single point or the sequential highlighting of 
successive parts.  Note also that each animation conveys a 
cognitively simple idea, the relative speeds of two simultaneously 
moving objects or the order of spatially scattered elements.

These considerations and findings suggest that graphic animations may 
succeed with the right combination of information processing ease and 
cognitive naturalness.  Animations can facilitate only when they are 
simple and slow enough to be perceived.  Animations are natural for 
conveying processes or events that take place in time or for guiding 
focus of attention over time.  Useful animations should reflect 
people's conceptions of the processes and events they portray; they 
should be continuous if conceived of as continuous, discrete if 
conceived of as discrete.  Finally, they should abstract and portray 
the essence of the changes over time.

4. Other Approaches.

The approach taken here has been to search for ways that space and 
the things in it have been used to represent both spatial and 
nonspatial elements and relations, to search for natural 
correspondences between space and thought.  Others have analyzed 
graphics from different perspectives, among them, linguists, computer 
scientists, psychologists, statisticians, designers, and 
philosophers.  I review these analyses briefly for the interested 
reader.  Their insights sometimes complement and sometimes parallel 
the observations and findings put forth here. Some of these 
researchers have analyzed the graphics themselves, others have 
examined how people comprehend, use, or construct them, and still 
others have made recommendations for design.  Inevitably, some have 
focused on more than one of these goals, as the goals interact. 
Analyses of actual graphics gives clues to how they are used and 
produced by people and how they are used gives clues to how they 
should be designed.

In an influential treatise, Bertin (1981) put forth a comprehensive 
semiotic analysis of the functions of graphics and the processes used 
to interpret them that established the field and defined the issues. 
According to Bertin, the functions of graphs are to record, 
communicate, andprocess information, and the goal of a good graphic 
is simplification to those ends.  This work was picked up first by 
statisticians and designers, most notably Tufte and Wainer, and later 
by psychologists.  Tufte (1983, 1990, 1997) has exhorted graphic 
designers to refrain from "chart junk," extraneous marks that convey 
no additional information, adopting by contrast a minimalist view 
that does not take into account conventions or need for redundancy. 
Wainer (1984, 1992) has gathered a set of useful prescriptions and 
insightful examples for graph construction, drawing on work in 
semiotics, design, and information processing.

Within psychology, the interests of researchers have varied. Ittelson 
(1996) has called attention to  differences in information processing 
of "markings," deliberate, two-dimensional inscriptions on surfaces 
of objects and other visual stimuli that may not have communicative 
intent.  Thus, interpreting a graphic depends on understanding that 
it can represent something other than itself, a concept that children 
gradually come to appreciate DeLoache, Miller, and Rosengren (1997). 
After an extensive survey of graphics used primarily in education, 
Winn (1987) analyzed how information  is conveyed in charts, 
diagrams, and graphs.  Larkin and Simon (1987) compared sentential 
and diagrammatic external representations, especially in teaching, 
highlighting the  advantages of diagrammatic ones  for tasks where 
spatial proximity conveys useful information. Continuing this line of 
thinking, Stenning and Oberlander (1995) presented a formal analysis 
of  the advantages and disadvantages of  diagrammatic and sentential 
representations in drawing inferences.  They argue that diagrams 
allow expression of some abstractions, much like natural language, 
but are not as expressive as sentential logics.

Several projects have been directly concerned with how people 
perceive, misperceive, interpret, remember, and produce graphics, 
raising implications for design.  Though a statistician, Cleveland 
(1984; 1985) has examined the psychophysical advantages and 
disadvantages of using different graphic devices for conveying 
quantity, position, angle, length, slope, and more, for efficiency in 
extracting different kinds of information from displays.  He and his 
collaborators have produced convincing cases where conventional data 
displays can be easily misconstrued by human users.  Kosslyn (1989; 
1994), using principles adopted from visual information processing 
and Goodman's (1978) analysis of symbol systems, has developed a set 
of prescriptives for graph design, based on an analysis of the 
syntax, semantics, and pragmatics underlying graphs.  Pinker (1990) 
provided a framework for understanding information extraction from 
graphics that separates processes involved in constructing a visual 
description of the physical aspects of the graph from those involved 
in constructing a graph schema of the mapping of the physical aspects 
to mathematical scales.  Based on Pinker's framework, Carpenter and 
Shah (1998) proposed a model of graph comprehension in which pattern 
recognition, translation of visual features to conceptual relations, 
and determining referents of quantified concepts, are integrated and 
iterated.  Carswell and Wickens (Carswell, 1992; Carswell & Wickens, 
1988; 1990) have demonstrated effects of perceptual analysis of 
integrality on graph comprehension, and others have shown systematic 
biases in interpretation or memory dependent on the form of graphic 
displays (Gattis & Holyoak, 1996; Levy, Zacks, Tversky, & Schiano, 
1996; Schiano & Tversky, 1992; Shah & Carpenter, 1995; Spence & 
Lewandowsky, 1991; Tversky & Schiano, 1989).

A number of observers of society have discussed the role of external 
devices, especially visualizations, in human cognition and behavior. 
Donald (1991) has speculated on the effects of the creation of mental 
artifacts on cultural change.  Norman (1993) has critiqued modern 
inventions that, as he says, are supposed to make us smart, but don't 
always succeed.  Kirsh (1995) has analyzed situations, such as 
preparing meals, playing tetris, and counting money, in which people 
array artifacts spatially and manipulate those arrays to facilitate 
memory and inference.

Visualizations have become increasingly important in organizing large 
data bases and enabling efficient search through them.  Navigation 
metaphors for these tasks abound, both in language and in graphics. 
Researchers in human computer interaction have also been active in 
the invention of graphics and visualizations, and in the development 
of prescriptions for design (e.g., Card, Mackinlay, and Shneiderman, 
1999).  Their "mantra" for creating effective visualizations of large 
sets of data: Overview first, zoom and filter, then details-on-demand 
(p. 625).  Though there is some support for this mantra, research in 
cognition on basic level concepts (e. g., Rosch, 1978) and on 
reasoning (e. g., Cheng and Holyoak, 1985) suggests that an effective 
entry into a complex system might be a thorough understanding of a 
concrete example. Once an exemplary example has been mastered, 
abstraction to generalities and inspection of details are anchored 
and supported.

The burgeoning interest in both descriptive and prescriptive aspects 
of comprehension and production of charts, graphs, and diagrams has 
been addressed in an increasing number of interdisciplinary 
conferences. Papers from the first of these, along with other seminal 
papers, were collected in a volume edited by Glasgow, Narayanan, and 
Chandrasekaran (1995) that called the field diagrammatic reasoning. 
Comics are now regarded as serious literature.  They use a variety of 
pictoric and spatial devices, some readily interpretable, some 
conventional, to convey motion, emotion, and other ideas (McCloud, 
1994).  A recent novel, self-described as a "Novel of Business," 
found it useful to include graphics as part of its' narrative (Bing, 
1998).

5. Some Functions of Graphic Displays.

Despite their variability of form and content, a number of cognitive 
principles underlie graphic displays.  These are evident in the many 
functions they serve as well as in the way information is conveyed in 
them.  Some of their many overlapping functions are reviewed below. 
As with functions, goals, and constraints on other aspects of human 
behavior, so the functions, goals, and constraints of graphic 
displays sometimes work at odds with each other, and sometimes work 
in concert.

	5.1. Attract attention and interest.

One prevalent function of graphic displays is to attract attention 
and interest.  A related function is aesthetic; graphics may be 
pleasing or shocking or repulsive or calming or funny.

5.2. Record information.

An ancient function of graphics is to provide records.  For example, 
tallies developed to keep track of property, beginning with a simple 
one-mark one-piece of property relation, developing into numerals as 
tallies became cumbersome for large sums and calculations (e. g., 
Schmandt-Besserat, 1992).  Various native American tribes kept track 
of their tribal numbers with icons for members and recorded their 
history year by year with depictions of a major event of the year 
(Mallery, 1972).

5. 3. Memory.

A related function of graphic displays is to facilitate memory.  This 
surely was and is one of the functions of writing, whether 
pictographic or alphabetic.  A modern example is the use of menus, 
especially icon menus, in computer user interfaces. Providing a menu 
turns what would otherwise be a recall task into a recognition task. 
Instead of having to call to mind all the possible commands in a 
program or files on a drive, a user has only to select the command or 
file that is needed from a list.  There is yet another way that 
graphs promote memory.  Menus and icons are typically displayed in 
standard places in an array. As anyone who has returned to a previous 
home after a long lapse of time knows, places are excellent cues to 
memory.  Ancient lore, the Method of Loci, and modern research 
support the intuition that space is not only an excellent cue but 
also an excellent organizer of memory (e. g., Bower, 1970; Franklin, 
Tversky, and Coon, 1992; Small, 1997; Taylor and Tversky, 1997; 
Yates, 1969).

5.4  Communication.

In addition to facilitating memory, graphic displays also facilitate 
communication.  As for memory, this has also been an important 
function of writing, to allow communication out of earshot (or 
eyeshot).  Graphic displays allow private, mental conceptualizations 
to be made public, where they can be shared, examined, and revised.

5.5. Provide models of actual and theoretical worlds.

Maps, architectural drawings, molecules, circuit diagrams, 
organizational charts, and flow diagrams are just some of the myriad 
examples of diagrams serving as models of worlds and the things in 
them.  In general, these are models, and not simply shrunken or 
expanded worlds. Effective diagrams omit many features of the modeled 
world and distort others, and even add features that are not in the 
modeled world.  Maps, for example, are not drawn strictly to scale. 
Roadmaps exaggerate the sizes of highways and streets so that they 
can be seen and used.  Maps introduce symbolic elements, for 
railroads, ocean depth, towns, and more, that require a key and/or 
convention to interpret.  The essence of creating an effective 
external representation is to abstract those features that are 
essential and to eliminate those that are not, that only serve as 
clutter.  For dynamic systems, successful diagrams must illuminate 
the causal chain of events over and above the parts of the systems 
and their interconnections (e.  g., Kieras, 1992; Kieras and Bovair, 
1984; Mayer and Gallini, 1990).  Of course, this is not as 
straightforward as it sounds, partly because it is difficult to 
anticipate all the uses an external representation will have, partly 
because successful communication rests on redundancy.  Well-designed 
diagrams facilitate learning and operation of a system but 
poorly-designed diagrams do not (e. g., Glenberg and Langston, 1992; 
Kieras and Bovair, 1984; Mayer and Gallini, 1990; Scaife and Rogers, 
1996). Current trends in computer graphics go against the maxim of 
abstracting the essentials.  The aim of at least some areas of 
computer graphics seems to be creating as much detail and realism as 
possible.  At the other extreme are designers (e. g., Tufte, 1983) 
advocating graphic design that is so minimal that it may not give 
sufficient clues to interpretation.

5.6. Convey meaning, facilitate discovery and inference.

Effective graphics make it easy for users to extract information and 
draw inferences from them.  Maps, for example, facilitate determining 
routes and estimating distances.  A map of cholera cases in London 
during an epidemic made it easier to find the contaminated water pump 
(Wainer, 1992).  Plotting change rather than absolute levels of a 
measure can lead to very different inferences (Cleveland, 1985). 
Indeed, the advice in How to Lie with Statistics (Huff, 1954) has 
been used for good or bad over and over.  The format of physics 
diagrams (Narayanan, Suwa, & Motoda, 1994),  architectural sketches 
(Suwa & Tversky, 1996), and graphs (e. g., Gattis, this volume; 
Gattis and Holyoak, 1996; Shah and Carpenter, 1995) bias users 
towards some kinds of inferences more readily than others.

6. Basis for Metaphors and Cognitive Correspondences.

A major purpose of graphic displays is to represent visually concepts 
and relations that are not inherently visual.  Graphic displays use 
representations of elements and the spatial relations among them to 
do so.  One type of element, common in early writing systems and 
recent computer interfaces, is an icon, which conveys meaning 
naturally through resemblance to something in the world or figures of 
depiction.  Another type of element is a schematic pictorial device, 
such as brackets or frames or lines, which convey meaning naturally 
through resemblance to physical objects that contain or link. 
Spatial relations convey meaning naturally by using spatial distance 
to represent distance on some abstract relation.  Ancient graphic 
creations as well as recent inventions by children and professionals often spontaneously adopt these metaphoric and analogic uses of space a 
nd the things in it.  People have rich experience observing and 
interacting with the physical world,  and consequently extensive 
knowledge about the appearance and behavior of things in it.  It is 
natural for this concrete experience and knowledge to serve as a 
basis for pictorial, verbal, and gestural expression.

This review suggests a perhaps deceptively simple maxim:  Use spatial 
elements and relations naturally. Naturalness is found in natural 
correspondences, "figures of depiction," physical analogs, and 
spatial metaphors, derived from extensive human experience with the 
concrete world.  It is revealed in language and in gesture as well as 
in a long history of graphic inventions.
References

Arnheim, R. (1977). Dynamics of architectural form.  Berkeley: 
University of California Press.
 
Baek Y.K. & Lane, B.H. (1988). Color, graphics, and animation in a 
computer-assisted learning tutorial lesson. Journal of Computer-based 
Instruction, 15 (4), 131-135.
 
Barwise, J. and Etchemendy, J. (1995). Heterogenous logic.  In B. 
Chandrasekeran, J. Glasgow, and N. H. Narayanan (Editors), 
Diagrammatic reasoning:  Cognitive and computational perspectives. 
Pp. 211-234. Cambridge, MA: MIT Press.
 
Beniger, J. R. & Robyn, D. L. (1978). Quantitative graphics in 
statistics. The American Statistician, 32, 1-11.
 
Bertin, J. (1981). Graphics and graphic-information-processing. N. 
Y.: Walter de Gruyter.
 
Betrancourt, M. and Tversky, B. (1999). Simple animations for 
organizing diagrams.  Manuscript submitted for publication.

  Bing, S. (1998). Lloyd:  What happened.  N. Y.:  Crown.
 
Boroditsky, L. (1999). Metaphoric structuring: Understanding time 
through spatial metaphors.  Manuscript submitted for publication.
 
Bower, G. H. (1970). Analysis of a mnemonic device.  American 
Scientist, 58, 496-510.
 
Braine, L. G., Schauble, L. Kugelmass, S. and Winter, A. (1993). 
Representation of depth by children:  Spatial strategies and lateral 
biases.  Developmental Psychology, 29, 466-479.
 
Brown, L. (1979). The story of maps.  NY:  Dover.
 
Card, S. K., Mackinlay, J. D., and Shneiderman, B. (1999).  Readings 
in information visualization:  Using vision to think.  San Francisco: 
Morgan Stanley.
 
Carpenter, P. A. and Shah, P. (1998). A model of the perceptual and 
conceptual processes in graph comprehension.  Journal of Experimental 
Psychology:  Applied, 4, 75-100.
 
Carswell, C. M. (1992). Reading graphs:  Interaction of processing 
requirements and stimulus structure.  In B. Burns (Ed.), Percepts, 
concepts, and categories. Pp. 605-645.  Amsterdam:  Elsevier.
 
Carswell, C. M. and Wickens, C. D. (1988). Comparative graphics: 
history and applications of perceptual integrality theory and the 
proximity compatibility hypothesis.  Technical Report, Institute of 
Aviation, University of Illinois at Urbana-Champaign.
 
Carswell, C. M. & Wickens, C. D. (1990). The perceptual interaction 
of graphic attributes:  Configurality, stimulus homogeneity, and 
object integration.  Perception and Psychophysics, 47, 157-168.
 
Cheng, P. and Holyoak, K. J. (1985). Pragmatic reasoning schemas. 
Cognitive Psychology, 17, 391-416.
 
Clark, H. H. (1973). Space, time, semantics, and the child.  In T. E. 
Moore (Ed.), Cognitive development and the acquisition of language. 
Pp. 27-63.  New York:  Academic Press.
 
Cleveland, W. S. (1984). Graphs in scientific publications. The 
American Statistician, 38, 261-269.
 
Cleveland, W. S. (1985). The elements of graphing data. Monterey, CA: 
Wadsworth.
 
Cohen, I. B. (1984). Florence Nightingale.  Scientific American, 
March, 128-137.
 
Cooper, L. A., Schacter, D. L., Ballesteros, S., and Moore, C. 
(1992). Priming and recognition of transformed three-dimensional 
objects: Effects of size and reflection.  Journal of Experimental 
Psychology: Learning, Memory, and Cognition, 18, 43-57.
 
Cooper, W. E. & Ross, J. R. (1975). World order.  In R. E. Grossman, 
L. J. San, & T. J. Vance, (Eds.), Papers from the Parasession on 
Functionalism.  Chicago: Chicago Linguistic Society.

Coulmas, F. (1989).  The writing systems of the world. Oxford:  Basil 
Blackwell.
 
DeFrancis, J. (1989). Visible speech:  The diverse oneness of writing 
systems.  Honolulu:  University of Hawaii Press.
 
DeLoache, J. S., Miller, K. F., and Rosengren, K. S. (1997). The 
credible shrinking room:  Very young children's performance with 
symbolic and nonsymbolic relations.  Psychological Science, 8, 308-313
 
Donald, M. (1991). Origins of the modern mind.  Cambridge:  Harvard 
University Press.
 
Engle, R.A. (1998).  Not channels but composite signals: Speech, 
gesture, diagrams and object demonstrations are integrated in 
multimodal explanations.  In M.A. Gernsbacher & S.J. Derry (Eds.) 
Proceedings of the Twentieth Annual Conference of the Cognitive 
Science Society.  Mahwah, NJ: Erlbaum.
 
Franklin, N. and Tversky, B. (1990). Searching imagined environments. 
Journal of Experimental Psychology:  General, 119, 63-76.
 
Franklin, N., Tversky, B., and Coon, V. (1992). Switching points of 
view in spatial mental models acquired from text. Memory and 
Cognition, 20, 507-518.
 
Gattis, M. and Holyoak, K. J. (1996).  Mapping conceptual to spatial 
relations in visual reasoning.  Journal of Experimental Psychology: 
Learning, Memory, and Cognition, 22, 1-9.
 
Gentner, D. (in press). Metaphoric mappings from space to time.  In 
M. Gattis (Editor), Spatial schemas.
 
Gelb, I. (1963).  A study of writing. Second edition. Chicago: 
University of Chicago Press.
 
Glasgow, J., Narayanan, N. H., and Chandrasekaran, B. (1995). 
Diagrammatic Reasoning: Cognitive and Computational Perspectives. 
Cambridge:  MIT Press.
 
Glenberg, A. M. and Langston, W. E. (1992).  Comprehension of 
illustrated text:  Pictures help to build mental models.  Journal of 
Memory and Language, 31, 129-151.
  Gobert, J. D. (1999).  Expertise in the comprehension of 
architectural plans.  In J. Gero and B. Tversky (Editors), Visual and 
spatial reasoning in design.  Pp. 185-205.  Sydney, Australia:  Key 
Centre of Design Computing and Cognition.
 
Goodall, J. (1986). The chimpanzees of Gombe:  Patterns of behavior. 
Cambridge, MA:  Harvard University Press.
 
Goodman, Nelson. (1978). Languages of art:  An approach to a theory 
of symbols. New York:  Bobbs-Merrill, 1968.
 
Gould, S. J. (1977). Ever since Darwin: Reflections in natural 
history. NY:  Norton.
 
Harley, J. B. and Woodward, D. (Eds.) (1987).  The history of 
cartography.  Vol. 1: Cartography in prehistoric, ancient and 
medieval Europe and the Mediterranean.  (1992) Vol. 2, Book 1: 
Cartography in the traditional Islamic and South Asian Societies. 
(1994) Vol. 2, Book 2: Cartography in the traditional East and 
Southeast Asian societies. Chicago: University of Chicago Press.
 
Hegarty, M. (1992). Mental animation:  Inferring motion from static 
displays of mechanical systems.  Journal of Experimental Psychology: 
Learning, Memory, and Cognition, 18, 1084-1102.
 
Hegarty, M., Narayanan, N.H., Cate, C., and Holmquist, S. (1999). 
Individual differences in understanding machines from diagrams, text, 
and hypermedia presentations.  Paper presented at the meeting of the 
Society for Applied Research in Memory and Cognition.  June 9-11. 
Boulder, CO.

Hochberg, J. E. (1964). Perception.  Englewood Cliffs, NJ: Prentice-Hall.

Howard, I. P. (1982). Human visual orientation. New York: Wiley.

Huff, D. (1954).  How to lie with statistics.  New York:  Norton.

Hughes, M. (1986). Children and number:  Difficulties in learning 
mathematics.  Oxford:  Blackwell.

Ittelson, W. H. (1996). Visual perception of markings.  Psychonomic 
Bulletin & Review, 3, 171-187.

Kaiser, M.K., Proffitt, D.R., Whelan, S.M. & Hecht, H. (1992). 
Influence of animation on dynamical judgements. Journal of 
experimental Psychology: Human Perception and performance, 18(3), 
669-690

Kieras, D (1992). Diagrammatic display for engineered systems : 
effects on human performance in interacting with malfunctioning 
systems. International Journal on Man-Machine Studies, 36, 861-895.

Kieras, D. E. and Bovair, S. (1984). The role of a mental model in 
learning to operate a device. Cognitive Science, 11, 255-273.

Kirsh, D. (1995). The intelligent use of space.  Artificial 
Intelligence, 73, 31-68.

Kosslyn, S. M. (1989) Understanding charts and graphs. Applied 
Cognitive Psychology, 3, 185-223;
Kosslyn, S. M. (1994). Elements of graph design.  New York:  Freeman.

Lakoff, G. & Johnson, M. (1980). Metaphors we live by. Chicago: 
University of Chicago Press.

Larkin, J. H. and Simon, H. A. (1987). Why a diagram is (sometimes) 
worth ten thousand words.  Cognitive Science, 11, 65-99.

Levin, I. & Tolchinsky Landsmann, L. (1989). Becoming literate: 
Referential and phonetic strategies in early reading and writing. 
International Journal of Behavioural Development, 12, 369-384

Levy, E., Zacks, J., Tversky, B. and Schiano, D. (1996). Gratuitous 
graphics:  Putting preferences in perspective. Human factors in 
computing systems:  Conference proceedings (pp. 42-49).  NY:  ACM.

Loomis, J. M., Da Silva, J. A., Fujita, N. and Fukusima, S. S. 
(1992). Visual space perception and visually directed action. Journal 
of Experimental Psychology: Human Perception and Performance, 18, 
906-922.
 
Lowe, R. (1999).  Extracting information from an animation during 
complex visual processing.  European Journal of the Psychology of 
Education, 14, 225-244.

Macken, E., Perry, J. and Haas, C. (1993).  Richly grounded symbols 
in ASL.  Sign Language Studies, 81, 375-394.
 
Mallery, G. (1893/1972). Picture writing of the American Indians. 
(Originally published by Government Printing Office).  NY:  Dover.
 
Mayer, R. E. & Gallini, J. K. (1990). When is an illustration worth 
ten thousand words?  Journal of Educational Psychology, 82, 715-726.
 
McBeath, M. K., Schiano, D. J., and Tversky, B. (1997). 
Three-dimensional bilateral symmetry bias in judgments of figural 
identity and orientation.  Psychological Science, 8, 217-223.
 
McCloud, S. (1994). Understanding comics.  N.Y.: HarperCollins.
 
Modley, R. (1976). Handbook of pictorial symbols.  N. Y.: Dover.
 
Muybridge, E. (1957). Animals in motion.  Edited by L. S. Brown.  NY:  Dover.
 
Narayanan, N. H., Suwa, M., and Motoda, H. (1994). A study of 
diagrammatic reasoning from verbal and gestural data.  Proceedings of 
the 16th Annual Conference of the Cognitive Science Society.
 
Neurath, O. (1936). International Picture language:  The first rules 
of isotype. London:  Kegan Paul, Trench, Trubner & Co., Ltd.
 
Norman, D. A. (1993). Things that make us smart.  Reading, MA: Addison-Wesley.

Pane, J.F., Corbett, A.T., & John, B.E. (1996). Assessing dynamics in 
computer-based instructions in Proceedings of CHI. Pp. 197-204.  ACM 
Press.

Pani, J. R., Jeffres, J. A., Shippey, G. T., & Schwartz, K. T., 
(1996). Imagining projective transformations:  Aligned orientations 
in spatial organization.  Cognitive Psychology, 31, 125-167.
 
Pinker, S. (1990). A theory of graph comprehension.   In R. Freedle 
(Ed.), Artificial intelligence and the future of testing. Pp. 73-126. 
Hillsdale, N. J.:  Erlbaum.
 
Rieber, L. P., Boyce, M. J., and Assad, C. (1990).  The effects of 
computer animation on adult learning and retrieval tasks.  Journal of 
Computer-Based Instruction, 17, 46-52.
 
Rosch, E. (1978).  Principles of categorization.  In E. Rosch and B. 
B. Lloyd (Editors), Cognition and categorization.  Pp. 27-48. 
Hillsdale, NJ:  Erlbaum.
 
Scaife, M. and Rogers, Y. (1996).  External cognition:  How do 
graphical representations work?  International Journal of 
Human-Computer Studies, 45, 185-213.
 
Schiano, D. and Tversky, B.  (1992).  Structure and strategy in 
viewing simple graphs.  Memory and Cognition, 20, 12-20.
 
Schmandt-Besserat, D. (1992). Before writing, Volume 1: From counting 
to cuneiform.  Austin:  University of Texas Press.
 
Shiffrar, M. M. & Shepard, R. N. (1991). Comparison of cube rotations 
around axes inclined relative to the environment or to the cube. 
Journal of Experimental Psychology:  Human Perception and 
Performance, 17, 44-54
 
Shah, P. and Carpenter, P. A. (1995). Conceptual limitations in 
comprehending line graphs.  Journal of Experimental Psychology: 
General, 124, 43-61.
 
Shin, S. J. (1991). An information-theoretic analysis of valid 
reasoning with Venn diagrams.  In J. Barwise (Editor), Situation 
theory and its applications, Part 2.  NY:  Cambridge University 
Press.
 
Small, J. P. (1997). Wax tablets of the mind.  New York:  Routledge, Paul.
 
Smith, M. C. and McGee, L. E. (1980). Tracing the time course of 
picture-word processing.  Journal of Experimental Psychology: 
General, 109, 373-392.
 
Spence, I. & Lewandowsky, S. (1991). Displaying proportions and 
percentages.  Applied Cognitive Psychology, 5, 61-77.
 
Stasko, J. and Lawrence, A. (1998). Empirically assessing algorithm 
animations as learning aids.  In J. Stasko, J. Domingue, M. H. Brown, 
and B. A. Price (Editors).  Software visualization.  Pp. 419-438. 
Cambridge, MA:  MIT Press.
 
Stenning, K. and Oberlander, J. (1995). A cognitive theory of 
graphical and linguistic reasoning:  Logic and implementation. 
Cognitive Science, 19, 97-140.
 
Stevens, S. S. (1946). On the theory of scales of measurement. 
Science, 103, 677-680.
 
Southworth, M. and Southworth, S. (1982).  Maps:  A visual survey and 
design guide. Boston:  Little Brown and Company.
 
Suwa, M. and Tversky, B. (1996). What architects see in their 
sketches: Implications for design tools.  Human factors in computing 
systems: Conference companion  (pp. 191-192).  NY:  ACM.
 
Taylor, H. A. and Tversky, B. (1997). Indexing events in memory: 
Evidence for index preferences.  Memory, 5, 509-542.
 
Tufte, E. R. (1983). The visual display of quantitative information. 
Cheshire, CT: Graphics Press.
 
Tufte, E. R. (1990). Envisioning information.  Cheshire, CT: Graphics Press.
 
Tufte, E. R. (1997). Visual explanations. Cheshire, CT:  Graphics
Press.
 
Tukey, J. W. (1977). Exploratory data analysis.  Reading, MA: Addison-Wesley.
 
Tversky, B. (1981). Distortions in memory for maps. Cognitive 
Psychology, 13,  407-433.
 
Tversky, B. (1995).  Cognitive origins of graphic conventions. In F. 
T. Marchese (Editor).  Understanding images. Pp. 29-53. New York: 
Springer-Verlag.
 
Tversky, B., Kugelmass, S. and Winter, A.  (1991). Cross-cultural and 
developmental trends in graphic productions.  Cognitive Psychology, 
23, 515-557.
 
Tversky, B. and Schiano, D. (1989). Perceptual and conceptual factors 
in distortions in memory for maps and graphs.  Journal of 
Experimental Psychology:  General, 118, 387-398.

Wainer, H. (1980). Making newspaper graphs fit to print.  In P. A. 
Kolers, M. E. Wrolstad and H. Bouma (Editors), Processing of visible 
language. Vol. 2. Pp. 125-142.  NY:  Plenum.
 
Wainer, H. (1984). How to display data badly. The American 
Statistician, 38, 137-147.
 
Wainer, H. (1992). Understanding graphs and tables.  Educational 
Researcher, 21, 14-23.

Whiten, A., Goodall, J., Mc Grew, W. C., Nishida, T., Reynolds, V. 
Sugiyama, Y., Tutin, C. E. G., Wrangham, R. W., Boesch, C. (1999). 
Culture in chimpanzees.  Nature, 399, 682-685.

Willats, J. (1997). Art and representation: New principles in the 
analysis of pictures. Princeton:  Princeton University Press.
 
Winn, W. D. (1987). Charts, graphs and diagrams in educational 
materials.  In D. M. Willows and H. A. Haughton (Eds.). The 
Psychology of illustration. N. Y.:  Springer-Verlag.
 
Yates, F. A. (1969). The art of memory. New York:  Penguin.
 
Yin, R. K. (1969).  Looking at upside-down faces. Journal of 
Experimental Psychology, 81, 141-45.
 
Zacks, J., Levy, E., Tversky, B., and Schiano, D. J. (1998). Reading 
bar graphs:  Effects of depth cues and graphical context. Journal of 
Experimental Psychology:  Applied, 4, 119-138.

Zacks, J. and Tversky, B. (In press).  Bars and lines:  A study of 
graphic communication.  Memory and Cognition.
 
Zacks, J. and Tversky, B. (1999).  Perceiving, remembering and 
communication structure in events.  Manuscript submitted for 
publication.

Figures

Figure 1.  Birch bark leave notices left by a Native American living 
near Long Lake, Maine in the mid-1800's.  Top panel:  I am going 
across the lake to hunt deer.  Middle panel:  I am going towards the 
lake and will turn off at the point where there is a pointer before 
reaching the lake. Lowest panel:  I am going hunting and will be gone 
all winter. From Mallery (1972), p. 331.

Figure 2. Pictorial origins of ten cuneiform signs.  Note the 
rotation the forms underwent to accommodate production.  From Gelb 
(1963), P. 70.

Figure 3. A T-O map, popular in the Middle Ages.  Asia (East) is at 
the top, Europe (north) to the right, Africa (south) to the right. 
 From Brown (1977), P. 118.

Figure 4. A map that mixes perspective to show both the street plan 
and the frontal views of landmarks.  From Southworth and Southworth 
(1982).
P. 74.

Figure 5. Pictorial signs in the Sumerian, Egyptian, Hittite, and 
Chinese languages.  From Gelb (1963), p. 98.

Figure 6.  Various icons developed by civil authorities for 
restaurant and coffee shop.  From Modley (1976), p. 67.

Figure 7. Various icons developed by civil authorities for ticket 
purchase and baggage claim.  From Modley (1976), p. 70.

Figure 8.  Bars and lines used by Zacks and Tversky (1999) for 
eliciting descriptions of relations.  Despite content, informants 
tended to describe relations portrayed as bars in terms of discrete 
comparisons and those portrayed as lines in terms of trends.  From 
Zacks and Tversky (1999), P. ?.


Table 1
Visual and Spatial Devices Used to Convey
Categorical and Nominal Relations

Categorical/Nominal Relations
spacesbetweenwords
spacing by rows and columns
( ), [ ], boxes, circles, Venn diagrams
visual devices: color, shading, cross-hatching
Ordinal
Simple subordination
indentation
size
superposition
highlighting
punctuation
Complex subordination
order
trees (hierarchies)
-----
----->


		Spatial Schemas in Depictions
		7