Ontario must be a strange place. Aquarians are 23% more likely to experience chest pain than all other signs of the Zodiac. Their hospitals admit 13% more people with the sign of Pisces for heart failure. Librans suffer from 37% more fractures of the pelvis than other patients. Or so we learn from a publication (1) described in Ingredients by George Zaidan. No, our Zodiac signs don’t condemn us to greater incidents of specific medical ailments. Mining for significant relationships in large databases may lead us astray. Are the relationships data miners find real or due to Random Chance?
Do the revelations about Zodiac signs and disease discredit studies about ultra-processed food? No. They do call into question the whole process of mining of large databases. No rational hypothesis exists that Geminis exhibit higher rates of alcohol dependence syndrome than Aquarians. Do diets high in sugar, fat, and salt increase chances of chronic diseases? Mining this dataset based on a stated hypothesis for a specific disease makes sense. The studies group together all packaged products with five or more ingredients. They call this category ultra-processed foods (UPFs).
Did the investigators state a specific hypothesis before starting the study? Or did they approach the dataset with the idea to run every analysis they could think of? Did they only report the ones that turned up statistically significant? Nothing we know to this point clarifies the situation. If they started with a hypothesis and tested it, that is a valid course of study. If they only sought significant, publishable results, they committed p-hacking, a questionable scientific practice. Who cares? Their work becomes equal to the Zodiac studies. Such research demonizes otherwise healthy foods without valid scientific evidence. How do we know which UPFs are unhealthy? Which UPFs may even promote good health? For example a sample hypothesis could read:
Consumption of large amounts of ultra-processed foods increases levels of heart attacks?
Zaidan and I do not see that the researchers followed this path.
What is so bad about p-hacking if it helps us eat healthier? P-hacking places the emphasis on publication and publicity not on accuracy and usefulness. How do we know if the conclusions are legitimate? Important decisions about what to eat hang in the balance. Web stories gain credibility based on clicks rather than the quality of their data. The reputations of scientists rise based on citations not merit. We become pawns in social experiments based on hype not based on science. Zaidan suspects the conclusions because the researchers didn’t pre-register their hypotheses. Data miners hit paydirt, but can they distinguish gold from fool’s gold? If it’s significant, it’s published, and it’s accepted as truth. If no relationships are significant, we never hear about them.
Ultra-processed foods represent a wide range of foods up to 61% of the American diet. Yes, they include cakes, cookies, ice cream, sodas, and candy. They also include whole-grain cereals, plant-based meat substitutes, hard liquor, gluten-free bread, and lactose-free milk. Are some of these foods healthier than others in the category or are all UPFs equally unhealthy? Is the five-ingredient rule valid or simplistic? Based on the evidence he reviewed, Zaidan rejects the conclusion that UPFs are dangerous.
Didn’t the NIH study prove that ultra-processed foods produce weight gain? Data enlighten. They do not prove! Zaidan commends the NIH study as the first randomized control study on the topic. Kevin Hall, the leader of the study, excelled in designing it. He preregistered his hypotheses and experimental design. His data analysis showed no p-hacking. These types of experiments challenge researchers, however, in their degree of difficulty and incredible expenses. Zaidan welcomes studies like the NIH one. Then he criticizes the conclusions more harshly than I did. He states that confining the subjects to a hospital for a month fails to approximate real life. The subjects experienced many invasive medical procedures that defy reality. The study covered too small a sample size over too short a period of time. Zaidan praises Hall for a good start. He presents no alternatives for a better study. So where does that leave us?
What can we learn from Brazil? Brazil is initiating a massive experiment on its population. Dr. Carlos Monteiro used NOVA to design a radical new set of dietary guidelines. These guidelines call for avoiding all UPFs and fast foods. They encourage home meal preparation and eating regular meals with family and friends. Officials promise better health with mass adaptation of these guidelines. Not everyone agrees. Food engineers in Brazil dispute the benefits of NOVA and the guidelines. What could possibly go wrong by adopting these guidelines? Brazilians could refuse to follow the guidelines. Most Americans pay scant attention to our dietary guidelines. Excitement about the guidelines could stimulate mass changes in the Brazilian diet. After the initial euphoria, a gradual reversion back to current dietary habits could occur.
Let’s assume that Brazil adopts the guidelines and sticks with them? First, expect a shock to the availability of food in the market. Supply chains will have difficulty keeping enough fresh foods on the shelves. Prices might skyrocket. Second, benefits in health incomes need a long time to appear. Immune systems do not improve overnight. Susceptibility to chronic disease will not vanish soon enough. Weight loss takes a long time to manifest itself. Impatient journalists may wonder why health isn’t dramatically improving. Consumers may become discouraged. Manufacturers of supplements and potions begin to hype their wares. Miracle diets start proliferating in the media. Forbidden foods tempt the weak. Could this effort collapse? Who knows? Its success depends on the will of the Brazilian people.
How can data miners provide us with a clearer picture? The mass category of UPFs makes up of over 60% of the American diet. Are all UPFs unhealthy? Are some unhealthy and others not dangerous to our health? Why won’t they break UPFs into at least four subcategories? Then perform the same tests on each of these categories AND report all the results. Based on other studies, junk foods could pose the greatest risk. Distilled spirits might show dangers as well. Convenience foods may or may not be unhealthy. Functional foods may even decrease chances of heart disease and other health hazards. Data miners could then drill down within each group to determine which products are more dangerous than others.
My subgroupings are not the only way to go. Other scientists could subdivide these products into different groups. For specific foods identified by the miners, biochemists would propose potential mechanisms. They could then test their theories and refine them. We would then have a clearer idea of which foods pose a danger and which foods do not.
Where is the mechanism? Why do we need one? As I began my research career, I designed observational experiments. My graduate student, Tsung-She Cheng, and I studied changes in fresh tomatoes during storage. Tomatoes lose quality when stored under refrigeration. We developed an objective for our study with fresh tomatoes (2):
to characterize the color development and gaseous evolution rates of tomato fruit during ripening after different lengths of chilling exposure.
I learned that grant proposals require proposing and testing a mechanism. Observing and reporting results are not good enough. Mechanisms provide biochemical explanations for changes observed in a series of experiments. I teamed up with Dr. Al Purvis. We developed a physiological model to explain chilling effects in susceptible plant tissue (3). Then his graduate student, Jim Gegogeine, tested our proposed mechanism on green bell peppers (4).
My readers may not care why tomatoes and bell peppers don’t hold up well in refrigerated storage. Understanding this mechanism, however, gives the research credibility. It also points to better management practices. The results don’t count unless the mechanism makes sense. Readers do care what foods contribute to the development of chronic diseases like diabetes. Do UPFs cause chronic diseases and lead to premature death? Are these studies valid? Or are they an example of Random Chance produced by data mining and p-hacking?
The beginnings of a mechanism emerge. A study measured telomere length in elderly Spanish citizens (5). Telomere length decreases as we age. The group who ate the most UPFs exhibited shorter telomeres than those who ate the least. This study collected real data on real people. It used a mass database, but I saw no evidence of data mining. The study measured a meaningful consequence, premature shortening of telomere length. It reports telomere length as a function of a real activity, consuming more UPFs. It does not identify which of the 50 types UPFs cause the shortening of telomere lengths. The authors mention that many of these foods are high in sugar, salt, oil, or food additives. Many UPFs are not high in sugar, salt, or oil. Almost all UPFs contain food additives. The study does not identify the types of ingredients responsible for shorter telomeres.
At this point they have the beginning of a mechanism. Now, investigators must identify a few suspect ingredients in UPFs that they suspect are unhealthy. Next, they should propose biochemical steps linking those ingredients to shorter telomere length. Then, comes a test of their proposed mechanism in an animal model or cell line. Otherwise we may be dealing with Random Chance and the Scorpio curse!
Next week: Tomatoland: From Harvest of Shame to Harvest of Hope
(1) Austin, P.C., Mamdani, M.M., Juurlink, D.N. and Hux, J.E. 2006. Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological signs and health. Journal of Clinical Epidemiology 59:964-969.
2) Cheng, T.-S. and Shewfelt, R.L. 1988. Effect of chilling exposure of tomatoes during subsequent ripening. Journal of Food Science 53:1160-1162.
(3) Shewfelt, R.L. and Purvis, A.C. 1995. Toward a comprehensive model for lipid-peroxidation in plant-tissue disorders. HortScience 30:213-218.
(4) Purvis, A.C., Shewfelt, R.L. and Gegogeine, J.W. 1995. Superoxide production by mitochondria isolated from green bell pepper fruit. Physiologia Plantarum 94:743-749.
(5) Alonso-Pedrero, L., Ojeda-Rodriquez, A., Martinez-Gonzalez, M.A., Zalba, G., Bes-Rastollo, M., and Marti, A. 2020. Ultra-processed food consumption and the risk of short telomeres in an elderly population of the Seguimiento Universidad de Navarra (SUN) project. American Journal of Clinical Nutrition 111:1250-1266.