In an effort to move the public toward vegan eating, Mercy For Animals and other groups often share online videos of farmed animal cruelty.
One way they do this is by running online ads that lead people to a landing page with video of farmed animal cruelty like this one. But is that an effective strategy? Does watching video of farmed animal cruelty actually change people’s diets and attitudes?
To find out, we contracted with the independent research firm Edge Research to conduct an experiment: We ran ads like this one on Facebook targeting women ages 13–25. Half of the people who clicked on the ad—we’ll call them the experimental group—were randomly sent to our regular landing page with video of farmed animal cruelty.
The other half who clicked—we’ll call them the control group—were randomly sent to a different landing page that looked similar but focused on an unrelated social issue, fighting tropical diseases.
By comparing the experimental and control groups over the next few months, we could see whether the farmed animal cruelty video impacted diets and attitudes. (If we saw differences between the two groups, it would likely be due to the video, as the people in each group were similar in almost all respects except for which video they viewed.)
Two to four months later, participants from both groups were encouraged to fill out a survey about their diets and attitudes. Around 800 people from each group did so. Edge Research then “weighted” the data so the respondents from each group were identical in gender, geography, and age.
Finally, Edge compared the groups’ answers to see whether the farmed animal cruelty video had an impact. Here’s what Edge and MFA found.
Special thanks to Peter Hurford, David Chudzicki, Allison Smith, Jeff Kaufman, and several anonymous donors for their financial support for the study, and to Peter Hurford for work on the study design.
Consumption of Animal Products
We asked participants how many servings of beef, pork, chicken, fish, eggs, and dairy they had consumed over the previous two days. We didn’t find a statistically significant difference in reported diet between the two groups.
The experimental group actually reported eating slightly more animal products, but the difference was not significant. That means the difference may well have been due to chance.
Is it possible showing people cruelty to farmed animals increases meat consumption? It’s a frightening proposition that should be studied. However, since other studies have found that showing footage of farmed animal cruelty decreases meat consumption, and since the difference in our study was not close to statistically significant, we think it’s unlikely.
Because of the extremely low power of our study, we don’t actually know whether the two groups’ diets were the same or slightly different. Our study was powered to detect a 10% difference between the groups, and since the differences between the groups were much smaller than that, we can’t be confident about whether the differences between the groups were due to chance or were the true group means. So unfortunately, our pool of participants wasn’t large enough to answer our key question. Based on our study design, it appears we would have needed tens to hundreds of thousands of participants to properly answer this question.
The bottom line is this: We don’t know whether showing people farmed animal cruelty video in this context will cause them to slightly increase, slightly decrease, or sustain their consumption of animal products a few months later.
Consumption of Animal Products by Age Group
We also looked at the experimental and control groups by age subgroups to see whether age mattered at all. We compared the groups in two ways: (1) with a Bonferroni correction to account for the fact that we were conducting multiple group comparisons, and (2) without the Bonferroni correction.
We compared the groups using conservative and non-conservative approaches to try to create a range that possibly contained the truth. Without the Bonferroni correction, we found several statistically significant differences between the experimental and control groups. However, with the Bonferroni correction, we found no statistically significant differences between any age groups.
Participants 13–16 years of age in the experimental group appeared to be eating slightly fewer animal products than those of the same age in the control group, but none of those differences was statistically significant and may well have been due to chance.
Participants 17–20 and 21–25 in the experimental group appeared to be eating slightly to dramatically more animal products than those of the same age in the control group. However, none of these differences was statistically significant. Had we not applied the Bonferroni correction, nearly half of these differences would have been statistically significant at the 85% or 95% level. However, the size of the differences is so large we suspect they’re most likely due to chance. It’s extremely unlikely, for example, that watching a farmed animal cruelty video would cause the average 23 year old to eat 50% more pork, or the average 19 year old to eat 30% more chicken as suggested by the data here.
Nevertheless, the possibility that viewing farmed animal cruelty might increase meat consumption is frightening and deserves further research. The possibility that 13–16-year-old women are more receptive to farmed animal cruelty video than 17–25-year-old women is also worth investigating.
Vegetarians and Meat Eaters
We compared one more thing: the number of vegetarians and meat eaters in the experimental and control groups. We took all of the people who said they had eaten at least one serving of pork, beef, chicken, and fish in the preceding two days and classified them as meat eaters. Then we took all the people who said they had eaten zero servings of meat in the preceding two days and classified them as vegetarians.
We found that 6.7% of people in the control group were vegetarian and 8.5% of people in the experimental group were vegetarian. Although it suffered from the same low power issue discussed above, this difference was just shy of statistically significant at the 95% level.
We think that this result is particularly interesting because it overcomes some of the issues with self-reported consumption. For example, most of us can be certain as to whether we ate meat in the past two days. However, if you did eat meat, it’s very tricky to remember how much you ate. So classifying people as meat eaters or vegetarians helps reduce the mistakes that we make when remembering our consumption.
Self-Reported Dietary Change
We next asked participants whether they changed their diets in the preceding four months. (Four months stretched back to before they viewed one of the videos.)
There was one statistically significant difference at the 95% level: People in the experimental group were more likely to identify as vegetarian. (When we say “statistically significant at the 95% level,” we mean that there was a 5% chance we had found the effect size that we found when there was truly no difference between the experimental and control groups.)
This was interesting because when we looked at what people reported actually eating, there was no statistically significant difference in rates of actual vegetarianism/veganism.
We don’t know why viewing farmed animal cruelty made people more likely to identify as vegetarian. However, since self-identity influences behavior, this is likely a positive thing. Those who view themselves as vegetarian may be more likely to actually go vegetarian, stick with vegetarian eating, and so on.
Intended Future Meat Consumption
Next, we asked participants how much meat they intended to eat four months into the future.
Several statistically significant differences were found. At the 95% level, members of the control group were more likely to report that they didn’t intend to change the amount of meat they ate in the next four months. At the 85% level, members of the experimental group were more likely to report that they intended to eat less meat; they were also more likely to report that they intended to stop eating meat entirely. If the “less meat” and “no meat” categories were combined into one general “reducer” category, we would find a statistically significant difference at the 95% level, with the experimental group more likely to intend to eat less meat four months into the future.
Finally, we asked three questions about attitudes. Answers to these questions have correlated with rates of vegetarian eating and meat reduction in other studies. (In other words, people who hold these attitudes are more likely to cut out or cut back on meat consumption.)
We found only one statistically significant difference, at the 85% level. Those in the experimental group were more likely to strongly agree that cows, pigs, and chickens are intelligent, emotional individuals with unique personalities.
Those in the experimental group were also slightly more likely to strongly agree with the other two attitude questions, but the differences were not statistically significant and therefore may well be due to chance.
MFA’s Takeaways: What We Did and Didn’t Learn
Our hope for the study was to get information useful in deciding whether we should increase, decrease, eliminate, or maintain the amount of funding we allocate to online ads that drive people to video of farmed animal cruelty. Unfortunately, we don’t feel the study shed useful light on that question.
Discouragingly, we found no statistically significant differences in reported meat, dairy, and egg consumption. But because we powered the study to detect only a 10% difference, we can’t be confident there was no difference or a modest positive or negative difference. Since even a tiny reduction (for example 0.1%) in meat consumption could make an online advertising program worthwhile, there’s no useful information we can take away from participants’ reported food choices.
Encouragingly, the study did find that viewing farmed animal cruelty video online makes people more likely to identify as vegetarian, more likely to hold attitudes correlated with meat reduction, and more likely to intend to cut out or cut back on meat consumption, at least in the short term. While those are all very good findings, and while changes to identity, attitude, and intention generally correlate with behavioral change, they are not the same as behavioral change. Animals are only spared once we have successfully changed behavior, and in this study we weren’t able to accurately measure whether (or how) the video changed behavior.The study's results did find that there was close to a statistically significant difference in the rates of vegetarianism between the experimental and control groups (p=.07), but the low power means this result is not necessarily reliable.
Lastly, we learned a few more things through further research that made it difficult to draw practical conclusions from the study:
- Large-scale studies such as this one have found that for online advertising campaigns, the majority of impact comes from changing the behaviors of those who view the ad but never click on it. In our study, we looked only at those who actually clicked on the ad. Therefore, our study wasn’t a study of the overall impact of online pro-vegetarian ads; rather it was a study on the impact of viewing online farmed animal cruelty video after clicking on an ad.
- Large-scale studies of online advertising have also found that sample sizes of more than 1 million people are typically needed before statistically significant behavioral changes can be detected. We spoke to numerous data collection companies as well as Facebook and accessing that much data for a study like this is not possible.
- Numerous studies have found that self-reports on dietary choices are extremely unreliable. On top of that, we’ve also found that diet-change interventions (like the one we did with the experimental group) can change how people self-report their food choices. This combination of unreliability and discrepancy suggests the self-reports of servings of animal products eaten may not be valid.
Based on all of the above, we don’t feel the study provides any concrete practical guidance. Therefore, the study won’t cause us to reallocate funding for our online advertising in a positive or negative direction.
The greatest value of this study has been bringing to light valuable lessons and information we hope will be useful to future MFA research and to other organizations that plan to conduct similar research on vegetarian advocacy interventions.
In light of this study, we hope to carry out further research in an effort to determine the following:
- Whether viewing farmed animal cruelty video and similar material increases meat consumption
- Whether teenage women (ages 13–16) are more responsive than slightly older young women (ages 17–25) to farmed animal cruelty video and similar material
- Dietary and attitudinal changes within the small subset of ad clickers who pledge to go vegetarian and sign up for MFA’s vegetarian eating email series on the MFA landing page.