Graphics and Cognition

How Do We Perceive Charts?

Susan Vanderplas

2023-10-12

Outline

  1. A quick cognitive psychology primer

  2. Applications to Charts

  3. Models of graph cognition

Cognitive Psychology Primer

Time Stimulus Perception Cognition

Attention

  • Focused vs. Divided attention

    • Divided = multiple input streams must be processed (e.g. driving a car)

    • Focused = single input stream

  • Attention shifts with gaze systematically throughout a visual stimulus

  • Information which is extraneous to the task may not be stored or remembered

Written text. An annotation layer on top shows how the eye fixates sequentially on each word/syllable (approximately) and then jumps to the next word in the sequence.

Introspection

As you look at the chart I’m about to show you, what do you focus on first? second? third?

Mentally record how you read in the chart and what you’re looking at over time.

Most of the time, we are concerned with attentive perception. Some factors are processed preattentively (without conscious attention), but for everything else, introspection is a valid observational method.

Introspection

Memory

Long Term Memory

Long Term Memory Information Retrieval

  • unlimited storage capacity (theoretically)
  • maintained for long periods
  • two types:
    • declarative/explicit - you can consciously evoke this
    • non-declarative/implicit - procedural memories, such as ‘how to ride a bike’

Memory

Short Term Memory

  • Not a term used any more. Replaced by “Working Memory”

Memory

Working Memory

Image from https://www.simplypsychology.org/working-memory.html

  • Proposed by Baddeley and Hitch (1974)
  • Limited capacity
  • Components are thought to be relatively independent of each other

Application to Charts

Perception

  • Sensory input
  • Recognition of type of chart
  • Takes place very quickly - preattentively

perception

Gestalt Grouping

  • Gestalt heuristics used to group points into clusters/trends

Figure/Ground Segregation Closure Proximity Good Continuation Law of Simplicity Similarity CommonRegion

gestaltgrouping

Integration with Domain Knowledge

  • Begin to assign meaning to the relationship between the data and the chart labels/titles

  • Retrieve knowledge from long term memory and make that available in working memory

integrationwith domainknowledge

Integration with Domain Knowledge

Domain knowledge includes:

  • How graphs are usually structured (e.g. y-axis increases from bottom to top)

  • Conventions for use of filled vs. empty space

  • Knowledge of relationships, events, etc. that might be impactful

Figure from Padilla et al. Cognitive Research: Principles and Implications (2018) 3:29

Storytelling

  • Assign meaning to relationships

  • Fit visual statistics

  • Look for things that do and don’t fit a rough working hypothesis about the data

visualstatistics storytelling outlierassessment

Inference

  • Numerical estimation from the chart with uncertainty

  • Visual search, followed by estimation, calculation, and inference.

y(x) y(x) x inference estimation visual search calculation

Prediction and Application

  • Moving beyond the data shown to interpret and apply meaning

  • Draw conclusions

  • Make predictions about the future

prediction application

Decision Making with Graphics

What is the average number of Adelie and Chinstrap penguins measured?

What steps do you go through to calculate this average?

General Process

From Padilla et al (2018), an adaptation of Pinker (1990)

Adding in Working Memory

From Padilla et al (2018)

Fast and Slow Decisions

From Padilla et al (2018)

Fast Decisions

Visual heuristics replace calculations, requiring very little working memory

Slow Decisions

Working memory required for each estimation and calculation step

Cross-domain Findings

Major takeaways from different experiments across domains and application areas:

  1. Visualizations direct viewers bottom-up attention, which can both help and hinder decision making

  2. The visual encoding technique gives rise to visuospatial biases

  3. Visualizations that have greater cognitive fit produce faster and more effective decisions

  4. Knowledge-driven processes can interact with the effects of the encoding technique.

Directing bottom-up attention

From Padilla et al. (2018)

Directing bottom-up attention

Attention focused on the pictogram instead of the base rate, leading to decisions that are suboptimal.

From Padilla et al. (2018)

Directing bottom-up attention

If a path intersects with a point of interest, resulting decisions tend to be biased.

From Padilla et al. (2018)

Visual-spatial Biases

From Padilla et al. (2018), rearranged to fit on slides

Biases:

  • Anchoring

  • Anecdotal evidence

  • Containment - what is in the container is different from what is outside the container. Ex: Binning continuous data

  • Deterministic construal - what is shown is deterministic instead of probabilistic

  • Quality bias - high quality image = high quality science

Cognitive Fit

If there is a mismatch between the visualization type and the decision making component, working memory must be used to compensate

From Padilla et al. (2018), reproduced from Hegarty et al. (2012)

Knowledge-Driven Processing

  • Instructions/training (short term knowledge)

  • Individual differences (e.g. math skills, background knowledge, interests)

  • Knowledge can be used to overcome e.g. familiarity bias (preference for familiar but ineffective visualizations)

  • Subject-matter expertise

From Padilla et al. (2018), a reproduction of stimuli used by Galesic et al (2009).