Cultivating Insights: Harnessing the Power of Data Visualization in Agriculture

Susan Vanderplas

2024-01-09

Why Data Visualization?

Why Data Visualization?

• Data is easier to understand visually

• Build trust in models and results

• Identify patterns, trends, outliers

I’m here to talk to you today about data visualization, and I think the first question we should answer is why.

I’m a statistician who studies data visualization, which means graphs are my favorite tool to use to understand data.

From my perspective, data is easier to understand visually, and if I can see why the model is suggesting I do X, Y, or Z, then I’m more likely to trust it than if I don’t see any pictures or how the data supports that conclusion.

Statistically, we use charts to identify patterns, trends, and outliers. What I’m going to talk through today is how we take that basic theme and apply it to on-farm data.

I’m not an agronomist or an economist, and I can barely keep houseplants alive, so you’ll have to bear with me. I’m using some example data and I’ve fit some very basic models that are probably not correct, but are useful for demonstrating how to use different types of charts.

In statistics, we have a saying that “all models are wrong, but some are useful”, and that’s what I’m focusing on today.

So I hope you’ll overlook any major mistakes I’ve made in modeling and instead focus on the pictures, because that’s what I actually know and am here to talk about.

I’m going to talk about the use of data visualizations through the cycle of a single year’s on-farm experiment. We’ll start with exploratory data analysis - looking at data collected throughout the growing season, and then talk about the use of graphics to understand how certain variables affect yield (or other variables of interest). Finally, we’ll talk about using graphics to make decisions about the next year.

Exploratory Data Analysis

Exploratory Data Analysis

• Identify issues with data quality/import (“gut check”)

• Look at raw data

• Develop initial hypotheses and questions

Exploratory data analysis is data analysis that takes place before you fit models.

It’s mostly done with data visualization, but can also include summary statistics and assessment of outliers (but those can be done with data visualization as well, so that’s how I’m going to do it).

Exploratory Data Analysis

During Design:

We start during the design process - before you implement an experiment it’s good to check that the experiment is actually set up in a way that meets your expectations. If it doesn’t, you can always ask your crop consultant or research contact to explain why something is the way it is - you might find an error, and you might learn something new.

In addition, you know your fields better than we do - our goal is to write software that does these things automatically, but there are always exceptions and special cases that the software doesn’t cover. Here, I’d be interested to know why one half of the field is much more densly managed than the other, but that might be something obvious to you. Otherwise, it might be good to look for e.g. changes in the seed rate that are too sharp (as I undrestand equipment can sometimes have trouble handling that) or areas that may be tricky and need a bit more focus when actually executing the experimental design during planting.

Exploratory Data Analysis

After Application:

Once the seed is planted, it’s useful to go and look at the difference between the planned experiment and the as-applied experiment. Deviations from the plan reduce the ability to determine how yield is affected by seed rate, and so it’s in everyone’s interest to minimize these issues.

Exploratory Data Analysis

After Application:

It’s nice to have the spatial view, but it can be more helpful from a design perspective to understand how the planned rate compares to the deviation. Here, I’ve created a binned plot – this is like a scatterplot, but the rectangle is colored in based on how many points would be in that bin, so that we can see the different counts more effectively. I’ve added the actual counts to the side of each bin as well, to help you see what I’m seeing.

What concerns me as a statistician isn’t the deviations that are within about 10 each way, it’s the ~64 sub plots that have high seed rates and have a -20 deviation. That could reduce our ability to see an effect in the higher seed rates, because we don’t have as many data points in that area.

What we see here is that most seed rates are relatively accurate (right at 0) but that at lower planned seed rates we tend to over-seed and at higher planned seed rates, we tend to under-seed. This suggests an equipment issue. In addition, notice that there are

Exploratory Data Analysis

After Harvest:

After harvest, we start by just looking at yield rates over time. I’m sure you’ve seen similar plots before, but they probably have multiple colors corresponding to different levels. Perceptually, it’s better to use a single-hue scale for each variable, and green is the color humans can see the most shades of, so I wanted to use it for something important.

We could flip back and forth between this and the design plot, or try to overlay them, but that gets pretty confusing to look at and isn’t all that effective perceptually, either.

What we’ll do instead is move on to the model portion of our discussion - we’ll let the statistical models do the hard work of determining what variables are important and then we’ll look at what the model says visually to see if the results make sense.

Understand Results

Understand Results

• Visuals help us understand what a model is doing

• Useful for identifying areas where models may be less accurate

• Look at raw data vs. model results

When we’re using graphics to understand results, we need to keep in mind that our goals have changed - instead of looking at raw data, we may be looking at residuals (differences between the model’s predictions and reality) or partial effects (what a specific variable is predicted to be contributing to the outcome (which is usually yield).

We want to be looking for areas where the model doesn’t seem to be fitting properly or areas that don’t make total sense - think of this as playing “devil’s advocate” and questioning the model’s conclusions to make a stronger model.

Understand Results

Parametric Coefficients
effect	Estimate	Std. Error	t value	Pr(>|t|)
(Intercept)	39.5926	0.2215	178.7559	0.0000

Approximate Significance of Smooth Terms
effect	edf	Ref.df	F	p-value
s(s_rate)	0.9772	9.0000	0.1442	0.2558
s(elev)	6.0431	9.0000	3.2842	0.0000
s(slope)	4.7708	9.0000	2.8008	0.0000
s(clay)	2.4558	9.0000	1.8240	0.0001
s(silt)	1.4403	9.0000	0.7279	0.0061
te(x_coord,y_coord)	20.3908	24.0000	14.1260	0.0000

First off, I’d like to point out the idfference between what the model print-out shows and what the graphics show. If you’re like me, tables like this don’t make a terrible amount of sense. I can translate them for you - essentially, in this model, we have smooth curves that model seed rate, elevation, slope, clay and silt percentage, and spatial correlation. The numbers in this table aren’t that useful, because we’re using smoothers - they mostly tell you just whether or not the variable is statistically significant - whether it seems to account for more variability than what would be expected by random chance.

In this case, seed rate doesn’t seem to be significant, though all of the variables I included that you have no control over seem to be significant. So that isn’t great - it means that at least according to the parameters of this experiment, seed rate doesn’t matter as much as other factors outside of your control. But mostly, a table of numbers doesn’t give you much to interrogate.

Understand Model Results

Instead, with this type of model, it’s often better to look at so-called ‘partial effect plots’ - basically, plots of the effect of a single variable, with all other variables removed.

In each of these plots, you can see the number of observations at each point below the plot, in what’s called a “rug chart” - a little tic for each observation. The black line shows the smoothed effect, and the grey area around it shows the variability in that effect - with more points, there’s lower variability. Note that the scales on the y axis are different as well - you can’t compare the variability around one line to that around the next.

So what do we see from these plots? We see that seed rate might account for -1 to 1 unit of yield variability, and because we can draw a straight line through that grey band at 0, we know that seed rate isn’t significant.

We can see that as elevation increases, the predicted effect due to elevation is a positive change in yield. Similarly, as slope increases, we see a decrease in predicted yield, with everything else held constant. The percentages of silt and clay are likely tied together, which explains the u-shaped relationship between clay and yield and the inverse relationship between silt and yield. These are relatively small, overall, though.

To get a full picture of what’s contributing to this model, though, we have to look at spatial effects too.

Understand Model Results

Here, we see the observations in the field as dots underneath the partial effect plot that is a purple to orange gradient, where purple means high and orange means low. This gradient type is safe for all types of colorblindness, which is why I use it instead of a more common rainbow-type scheme or even a red-yellow-green traffic light scheme.

What we see here is a set of contours outlining a line of approximately equal performance, that produce something like a height map showing areas with higher yield and lower yield as predicted by spatial location (e.g. x and y coordinate of the center of the region).

While this doesn’t mean that much to me, as I’m not familiar with the field in question, I’m pretty sure that a similar plot of a field you are familiar with is something that may be a useful “gut check” - you know your own fields, and likely have an idea of which areas were performing well or poorly - perhaps there was some wind damage along the edge of the field, but the center survived? Or maybe the center of this field is higher than the edges? We could examine these hypotheses with new plots, but my point is that as you look at a similar plot for your own field, you should be coming up with questions and hypotheses and checking those against your expectations as well as the model’s predictions.

Graphics aren’t the final stop here - they’re a way for you to examine the model and understand where it is performing well or poorly.

Make Decisions

Make Decisions

For this part of the talk, I’m switching example fields to one with an experiment designed to manipulate the nitrogen rate. As the main intervention in the last field wasn’t actually significant, it wasn’t all that useful to talk about how to use the model results to make decisions.

Of course, Taro, David, Brittani, and John have all talked this morning about how they’re creating reports and models to help you make decisions. In this part of the talk, I’m simply going to highlight how graphics can help you make sense of both the automatic prescriptions their systems spit out, and how you can look at the charts to understand the why of those predictions.

Visualizations are really important for helping us feel comfortable with the results from a statistical model - I don’t trust models unless I can see why they’re making predictions and what those predictions mean in practice.

Make Decisions

Parametric Coefficients
effect	Estimate	Std. Error	t value	Pr(>|t|)
(Intercept)	246.0769	1.8919	130.0671	0.0000
munameDrummer silty clay loam	21.3936	13.0254	1.6424	0.1011
munameElpaso silty clay loam	6.8863	6.3001	1.0930	0.2749
munameHoughton muck	−1.8043	12.5800	−0.1434	0.8860
munameLisbon silt loam	−16.0409	9.4682	−1.6942	0.0909
munameOctagon silt loam	−2.2365	2.9266	−0.7642	0.4451

Approximate Significance of Smooth Terms
effect	edf	Ref.df	F	p-value
s(n_rate)	2.0872	9.0000	7.9926	0.0000
s(elev)	8.0000	9.0000	3.6272	0.0000
s(slope)	1.9356	9.0000	1.9140	0.0000
s(elev,slope)	16.5476	27.0000	1.7851	0.0000
te(x_coord,y_coord)	22.3023	24.0000	5.7360	0.0000

So just as a starter, in this model for this field, I’m modeling each type of soil as a separate intercept - that is, each soil type is expected to have a different base yield rate, but to respond the same based on nitrogen rate, elevation, slope, etc. (this is obviously ridiculous, I would assume, but again, all models are wrong, some are useful, so let’s work with this simplification because it keeps the table at a manageable size for this presentation. It’s already pretty big.)

Make Decisions

So our first task after checking the model summary is to look at the profiles of each partial effect. Here, we see that n_rate has a serious effect (up to a point) in that yield increases from 100 to 200, and then levels off and/or starts to decrease. That’s (from what I know) expected - after the plant is no longer deficient in nitrogen, there isn’t much benefit to adding more. We also see that the elevation effect here is pretty huge - higher yield at lower elevation, lower yield at higher elevation. We also see that the slope has a pretty significant effect as well.

This isn’t the whole story, though, because I included an interaction effect between elevation and slope.

Make Decisions

Here, we can look at the interaction (ignoring the single-variable effects we included and looked at before). There are two things to note here: the effect size is smaller (about 20-30, instead of limits of -50 to 50 or -200 to 200 in the single-variable plots). The second thing to note is that the plot looks pretty lumpy, and that’s largely because the points are concentrated in a few different clusters. That is, while there may be differences here, they could well be driven by the fact that slope and elevation might change at the same time as e.g. soil type or other variables that are important.

The takeaway here is to be a bit suspicious and to look for other causes. I’m not digging further into this because of time, but it is absolutely ok to question why a variable might be significant and think about other causes for that effect.

Make Decisions

Speaking of other causes, let’s look at the spatial effects. Here, it seems very much like the model predicts much lower yield at the edge of the plot than it does in the top/center region. This could well be due to contouring or other effects.

Make Decisions

In fact, if we plot the partial effects and then also plot the elevation, we can see that this is largely just another way to say higher elevations seem to perform better. Sometimes, there are multiple effects that are correlated and included in the model through separate variables.

Make Decisions

So, with all of this, what are we to do? Let’s go back to the fact that we’ve modeled yield as a function of soil type, location, elevation, slope, and nitrogen rate (which we manipulated with an experiment).

Let’s return to the raw data for a moment. We can fit simple models to just each type of soil (ignoring spatial effects, elevation, etc. for the moment) and see if we can see a difference in the yield. Our statistical model is of course much more precise and we probably can trust it, but it always makes me feel better to see the effect in the raw data as well.

Here, I’ve drawn simple models for the 3 soil types that have enough data points to be able to confidently do so. In Databrook silt loam, we see that there is a slight effect and yield tops out at around a nitrogen rate of 200. In El Paso silty clay loam, we see that the rate is largely not important until we get past 225 or so, at which point yield seems to increase a bit. And in Octagon silt loam, there is a steady increase in yield over the tested nitrogen rates.

If I were deciding what to do next year, I’d probably look at optimizing nitrogen around 225 or so. Of course, this isn’t taking into account the tradeoff between cost of inputs and profit – so you’d need to talk to David about that tradeoff, and the software should be helping with that tradeoff. But as a statistician, I feel more confident in the model’s predictions given that I’ve seen the same basic thing in the raw data… I just had to pull it out and visualize it.

Make Decisions

Here’s the smooth our model fit to the data, showing the partial effect for nitrogen rate on yield. It, too, tops out at about 225, indicating that if nitrogen were free, we could optimize yield that way (since none of the other modeled factors are things within our control).

Why Data Visualization?

• Data is easier to understand visually

• Build trust in models and results

• Identify patterns, trends, outliers

To recap - data visualization is an important tool for assessing models and data. Our goal is to show you plots that allow you to become familiar with the models we’ve built, identifying patterns, but also understanding where the model might be weak (e.g. where there aren’t enough data points) and why some results might be less useful because of differences between as-applied and planned application rates.

If you’re interested in learning more, or want to help us test the visualization part of the interactive platform, please let David know, and we’ll be in touch when we have a prototype available.

Questions?