Regarding Dog Food Advisor’s Post on Sample Error
Dog Food Advisory recently posted on their website about Clean Label Project and sample error. The following is our response.
Hi DogFoodAdvisor,
Clean Label Project is a national non-profit with the mission to bring truth and transparency to consumer product labeling. We do this by using data and science to reveal true product quality and purity and empower consumer to make their own choices. Through the resulting shifting economics, we aim to serve as the catalyst to change the definition of food and consumer product safety in America.
Given the humanization of pet food over the past several years coupled with the scandals and recalls, we were especially interested in what was truly behind all these “Feed them like Family”, “Natural”, “Human-grade”, etc. claims. For us, it was about seeing past the comfort and security that pet food brands marketing departments sell, and get right to the data and science about the true ingredient quality. This impartiality only comes through testing. When we began this pet food study, we assumed that these companies were regularly screening for environmental toxins alongside more conventional foodborne contaminants (like salmonella). Our results clearly indicate otherwise. We believe consumers have a right to know what’s in the products they buy.
When we initially started this project, we utilized Nielson reports to pull the products that made up 90% of the overall retail sales of pet foods. From there, we visited specialty pet food stores and spoke with consumers and team members to ask what people were buying. Our approach to the sampling was simple and why Clean Label Project is unique is clear- we simulate the consumer shopping experience. We went to grocery stores, pet food stores, and online retailers and purchased products just like any consumer would. We tested over 1000 of America’s best-selling dry and wet dog foods, cat foods, and treats for over 130 environmental and industrial contaminants and toxins like heavy metals, pesticide residues, antibiotic residues, plasticizers, melamine, acrylamide, and mycotoxins. We amassed over 130,000 data points, benchmarked them, and put the findings on our website in the form of a 5-star rating system. The products that we personally purchased and tested are literally the exact same products that are in pantries across America. For us, we don’t trust label claims, we trust analytical chemistry because gas chromatography and Inductively coupled plasma mass spectrometry doesn’t lie.
We agree that sampling error is a risk in any scientific study, and you have correctly pointed out that random sampling of sufficient size is essential to minimizing the risk of sampling error leading to a false positive or false negative result (so-called “Type I” and “Type II” errors). However, there are two important distinctions here that we would like to stress as to why we chose the sampling strategy that we did.
First, at the product level – while you make a valid point that sampling repeatedly from different batches for an individual product would result in a more accurate representation of the “true” contaminant level of a single product this, in our opinion, misses important quality and supplier assurance implications of our results. If these companies, as many of them have repeatedly assured us, are conducting rigorous supplier assurance and quality assurance programs, single “fluke” high values should be exceedingly rare. While sample variability does occur, a robust quality program should severely limit the variability (and the levels of these contaminants). This is particularly important given that there is no maximum tolerance level for these contaminants in the pet food space. As a side note, the argument that “the product is only loaded with known carcinogens occasionally” doesn’t reassure us very much. Consider the analogy of food borne pathogens like salmonella, e.coli, or listeria – certainly there is variability as to how much of these bacteria are present in raw ingredients, but rigorous quality assurance programs or kill steps bring the variability in finished products to almost zero. This is why the presence of these bacteria cause headlines when they occur in finished foods – we have systems in place that should prevent them. This is not the case for the contaminants we measure in the pet food space—but it should be. All brands, regardless of their score, should be vigilant regarding these contaminants and should take proactive steps to improve the status of the industry.
Second, at the brand level: It is important to note that decisions about a brand (for example, the brand report cards released September 18, 2017) are not based off single product ratings, instead they are based on the average performance of multiple products within a brand. When data is aggregated across a group, the “true” value being estimated is that of the brand, not the product. As such, when we give the “thumbs up” or “thumbs down” to a brand, this is based on the weight of evidence from multiple products. This means that our brand ratings, and the conclusions we draw about the performance of brands, are arguably the least likely to be impacted by sampling error.
For us at Clean Label Project, we refer to ourselves as the environmental and industrial contaminant and toxin people. That’s who we are, the mission we hold, and the conversation we are looking to have with consumers. To us, so often we hear about food safety issues happening at burrito restaurants and cruise ships (e.coli, listeria, salmonella, etc.), but what people don’t talk about is the long term adverse health effects associated with chronic exposure to industrial and environmental toxins and contaminants with links to diseases like cancer- for ALL living things. To us, first, it’s do no harm, start with high quality – not harmful ingredients – and then dive into how to formulate the most nutritious foods. This should not be a novel concept.
To us at Clean Label Project, there is no such thing as healthy poison.
