10  Data Visualization

10.3.1.1 Color

Our eyes are optimized for perceiving the yellow/green region of the color spectrum. Why? Well, our sun produces yellow light, and plants tend to be green. It’s pretty important to be able to distinguish different shades of green (evolutionarily speaking) because it impacts your ability to feed yourself. There aren’t that many purple or blue predators, so there is less selection pressure to improve perception of that part of the visual spectrum.

Not everyone perceives color in the same way. Some individuals are colorblind or color deficient. We have 3 cones used for color detection, as well as cells called rods which detect light intensity (brightness/darkness). In about 5% of the population (10% of XY individuals, <1% of XX individuals), one or more of the cones may be missing or malformed, leading to color blindness - a reduced ability to perceive different shades. The rods, however, function normally in almost all of the population, which means that light/dark contrasts are extremely safe, while contrasts based on the hue of the color are problematic in some instances.

You can take a test designed to screen for colorblindness here

Your monitor may affect how you score on these tests - I am colorblind, but on some monitors, I can pass the test, and on some, I perform worse than normal. A different test is available here.

In reality, I know that I have issues with perceiving some shades of red, green, and brown. I have particular trouble with very dark or very light colors, especially when they are close to grey or brown.

It is possible to simulate the effect of color blindness and color deficiency on an image.

In addition to colorblindness, there are other factors than the actual color value which are important in how we experience color, such as context.

Our brains are extremely dependent on context and make excellent use of the large amounts of experience we have with the real world. As a result, we implicitly “remove” the effect of things like shadows as we make sense of the input to the visual system. This can result in odd things, like the checkerboard and shadow shown above - because we’re correcting for the shadow, B looks lighter than A even though when the context is removed they are clearly the same shade.

Implications and Guidelines

• Do not use rainbow color gradient schemes - because of the unequal perception of different wavelengths, these schemes are misleading - the color distance does not match the perceptual distance.
• Avoid any scheme that uses green-yellow-red signaling if you have a target audience that may include colorblind people.
• To “colorblind-proof” a graphic, you can use a couple of strategies:
  • double encoding - where you use color, use another aesthetic (line type, shape) as well to help your colorblind readers out
  • If you can print your chart out in black and white and still read it, it will be safe for colorblind users. This is the only foolproof way to do it!
  • If you are using a color gradient, use a monochromatic color scheme where possible. This is perceived as light -> dark by colorblind people, so it will be correctly perceived no matter what color you use.
  • If you have a bidirectional scale (e.g. showing positive and negative values), the safest scheme to use is purple - white - orange. In any color scale that is multi-hue, it is important to transition through white, instead of from one color to another directly.
• Be conscious of what certain colors “mean”
  • Leveraging common associations can make it easier to read a color scale and remember what it stands for (e.g. blue for cold, orange/red for hot is a natural scale, red = Republican and blue = Democrat in the US, white -> blue gradients for showing rainfall totals)
  • Some colors can can provoke emotional responses that may not be desirable.²
  • It is also important to be conscious of the social baggage that certain color schemes may have - the pink/blue color scheme often used to denote gender can be unnecessarily polarizing, and it may be easier to use a colder color (blue or purple) for men and a warmer color (yellow, orange, lighter green) for women³.
• There are packages such as RColorBrewer and dichromat that have color palettes which are aesthetically pleasing, and, in many cases, colorblind friendly (dichromat is better for that than RColorBrewer). You can also take a look at other ways to find nice color palettes.

10.3.1.2 Short Term Memory

We have a limited amount of memory that we can instantaneously utilize. This mental space, called short-term memory, holds information for active use, but only for a limited amount of time.

Without rehearsing the information (repeating it over and over to yourself), the try it out task may have been challenging. Short term memory has a capacity of between 3 and 9 “bits” of information.

In charts and graphs, short term memory is important because we need to be able to associate information from e.g. a key, legend, or caption with information plotted on the graph. As a result, if you try to plot more than ~6 categories of information, your reader will have to shift between the legend and the graph repeatedly, increasing the amount of cognitive labor required to digest the information in the chart.

Where possible, try to keep your legends to 6 or 7 characteristics.

Implications and Guidelines

• Limit the number of categories in your legends to minimize the short term memory demands on your reader.

  • When using continuous color schemes, you may want to use a log scale to better show differences in value across orders of magnitude.

• Use colors and symbols which have implicit meaning to minimize the need to refer to the legend.

• Add annotations on the plot, where possible, to reduce the need to re-read captions.

10.3.1.3 Grouping and Sense-making

Imposing order on visual chaos.

The ambiguous figures shown above demonstrate that our brains are actively imposing order upon the visual stimuli we encounter. There are some heuristics for how this order is applied which impact our perception of statistical graphs.

The catchphrase of Gestalt psychology is

The whole is greater than the sum of the parts

That is, what we perceive and the meaning we derive from the visual scene is more than the individual components of that visual scene.

You can read about the gestalt rules here, but they are also demonstrated in the figure above.

In graphics, we can leverage the gestalt principles of grouping to create order and meaning. If we color points by another variable, we are creating groups of similar points which assist with the perception of groups instead of individual observations. If we add a trend line, we create the perception that the points are moving “with” the line (in most cases), or occasionally, that the line is dividing up two groups of points. Depending on what features of the data you wish to emphasize, you might choose different aesthetics mappings, facet variables, and factor orders.

fbiwide <- read.csv("https://github.com/srvanderplas/Stat151/raw/main/data/fbiwide.csv")
library(dplyr)

fbiwide %>% 
  filter(Year %in% c(1980, 2010)) %>%
  filter(State %in% c("Alabama", "Alaska", "Arizona", "Arkansas")) %>% 
  ggplot(aes(x = State, y = Murder/Population*100000, fill = factor(Year))) + 
  geom_col(position = "dodge") +
  coord_flip() + 
  ylab("Murders per 100,000 residents")

import pandas as pd
fbiwide = r.fbiwide
fbiwide = fbiwide.assign(YearFactor = pd.Categorical(fbiwide.Year))
fbiwide = fbiwide.assign(Murder100k = fbiwide.Murder/fbiwide.Population * 100000)

yr1980_2010 = fbiwide[fbiwide.Year.isin([1980,2010])]
subdata = yr1980_2010[yr1980_2010.State.isin(["Alabama", "Alaska", "Arizona", "Arkansas"])]

(
ggplot(aes(x = "State", y = "Murder100k", fill = "YearFactor"), data = subdata) + 
  geom_col(stat='identity', position = "dodge") +
  coord_flip() +
  ylab("Murders per 100,000 residents")
)

<ggplot: (8755707330292)>

fbiwide %>% 
  filter(Year %in% c(1980, 2010)) %>%
  ggplot(aes(x = Year, y = Murder/Population*100000, group = State)) + 
  geom_line() + 
  ylab("Murders per 100,000 residents")

(
ggplot(aes(x = "Year", y = "Murder100k", group = "State"), data = yr1980_2010) + 
  geom_line() + 
  ylab("Murders per 100,000 residents")
)

<ggplot: (8755707272217)>

fbiwide %>% 
  filter(Year %in% c(1980, 2010)) %>%
  ggplot(aes(x = factor(Year), y = Murder/Population*100000)) + 
  geom_boxplot() + 
  ylab("Murders per 100,000 residents")

(
ggplot(aes(x = "YearFactor", y = "Murder100k"), data = yr1980_2010) + 
  geom_boxplot() + 
  ylab("Murders per 100,000 residents")
)

<ggplot: (8755709423144)>

The aesthetic mappings and choices you make when creating plots have a huge impact on the conclusions that you (and others) can easily make when examining those plots.⁴