Getting to Know Your Information
By Carolina Nunes, MS, Bastyr University Dietetic Intern (2019)
Countless websites and blogs are dedicated to diet and health-related topics, and information about the gluten-free diet, celiac disease, and gluten sensitivity seems to be everywhere. Not all sources, however, are committed to using high standards for their reporting and information. Additionally, applicability of research is sometimes over-stated or misconstrued, or opinions may be veiled as fact-based science. Following are some guidelines to help you sort through and evaluate it all.
- Websites and blogs
Check the “about us/about me” section to find out whether a site has any affiliation that might act as motivation for bias, and be wary of those that are trying to sell something, since this could be incentive for making exaggerated claims. Websites affiliated with universities, government agencies, reputable healthcare institutions, non-profit organizations, or larger publishing agencies such as The New York Times and Cochrane.org tend to have stricter editorial processes and are less likely to disseminate false or misleading information. Independent bloggers can of course also have high standards, but on the other hand, virtually anyone can start a blog. Read about a blogger’s background to find out whether the author has academic or other relevant qualifications, and take a look at the comments section for additional clues on quality of content.
Blog posts and opinion pieces about health-related issues are often based on scientific studies and may offer links to the studies they are based on. Don’t hesitate to follow links provided or track down the studies on your own to build your own opinion about a subject.
- Scientific Studies
There are numerous types of scientific studies, and how a study should be interpreted and understood will vary accordingly.
Two common types of studies which attempt to understand mechanisms related to specific metabolic phenomena are in-vitro studies, which take place in a test tube, and animal studies. In these, researchers have greater control over the conditions of an experiment and are therefore more able to isolate and evaluate causes and effects. But the conclusions from these studies are not generalizable for humans because of the inherent complexity of the human organism.
Findings based on microorganisms in a test tube, or on other species often serve as a starting point and basis for continued research on a given topic.
Studies performed with human subjects vary widely in terms of how many individuals are included. The smallest studies are case reports, or case studies, where a researcher or health care professional reports his or her experience with a single patient. These reports may provide insights about some conditions, but their conclusions cannot be generalized to a larger population. For example, a treatment might be effective for one person because of their genetic makeup, or because of external factors that are not applicable to the population at large.
In general, the more people in a study group the more broadly applicable the findings. A study that observes a group of people constitutes a “cohort analysis” or “observational study.” In these studies, researchers observe a sample of the population, measure specific markers over a relatively long period of time, and then look for relationships between the markers they have observed & measured, and/or for relationships between the markers and characteristics of the individuals studied. An example of observational research would be a study which sets out to assess whether cigarette smoking is related to causing lung cancer. In such a study, the researchers would include individuals who smoked and individuals who didn’t smoke, and would track them over time to see if they develop lung cancer. These types of studies are useful in many cases, but it is important to interpret their results appropriately.
Most observational studies can identify and highlight what is known as “correlation” in the data. In the cigarette and lung cancer example, the study might be able to note that cigarette smoking is associated (or correlated) with lung cancer, but would not be able to conclude that smoking causes lung cancer. That study wouldn’t be able to tell whether smoking causes cancer, or whether a predisposition to cancer somehow makes people more likely to smoke. When reading results of these types of study where the researchers or reporters claim that certain results show that something causes something else, it is critical to understand how their conclusion was reached.
A more specified type of study that is used to establish a cause and effect relationship between two factors is called a “randomized controlled trial”, or RCT for short. An RCT is often used to understand, for example, if a new drug has the intended effect, or if a certain diet is better for health. In these studies, subjects are randomly assigned to one group or the other, and other factors that could influence results are controlled for. For example, a person would be assigned to the group that will eat the “new” diet, or to the group that will continue their usual diet, but both groups would have similar amounts, for example, of exercise, since this factor could also affect their health. Because relevant factors are controlled for, results from RCT’s allow researchers to more clearly focus on the true effect which any specific intervention (e.g. diet, or drug) may have.
But even these studies have their limitations. In general, the larger the sample, the stronger the confidence in the results. For example, suppose you come across an RCT in which 10 patients with celiac disease were assigned to a new drug, and 8 of these were able to eat gluten without any adverse effect while on the drug, and another RCT where 300 patients were assigned to the drug and 60 of these could consume gluten with no adverse effects. It may seem that the smaller RCT demonstrates that the drug is effective, since 80% of participants had a positive result. Meanwhile results from the larger RCT would not support effectiveness, since only 20% of participants experienced a positive result. In such a situation, it is more likely that the larger study is the one reaching the better conclusion, if properly adjusted for relevant factors.
Two more types of studies to consider are systematic reviews and meta-analyses. These are considered more powerful because they can analyze results across a much larger sample that would be impractical to work with in observational studies or RCTs. In these studies, researchers analyze results from multiple previous studies on a subject. In the case of systematic reviews, results are analyzed and summarized, while in the case of a meta-analysis, results are mathematically combined. Continuing our earlier drug example, a systematic review or meta-analysis would take the 2 RCTs performed along with many other RCTs and observational studies available, analyze the results and then reach a stronger conclusion about whether the drug is effective or not.
In addition to the inherent strength and generalizability of a study itself, scientific research is also subject to bias coming from external factors. One potential source of bias is a conflict of interest. Suppose that a researcher is hired by a drug manufacturer to conduct an experiment to investigate whether or not a new drug is effective. The expectation is that research be conducted in an objective, scientific manner. However, if a researcher has financial ties with the manufacturer of the product being investigated, it is possible that he or she may potentially be inclined to give less, or altered, consideration to evidence that does not support the efficacy of the drug.
Studies may also be directly funded by companies that have strong interest in the results. For example, a wheat company which is working towards developing a lower gluten wheat product could be interested in funding a study looking at whether “low gluten” wheat is better tolerated, and might decide not to publish a study when the results contradict their interests. It is important to keep in mind that the presence of a conflict of interest does not necessarily diminish the validity of a study. But conflicts of interest such as those described above do raise a flag that a study needs to be considered more carefully. When evaluating a study, take this information into consideration. Depending on the journal where an article is published, “conflict of interest” and “funding/support” information may be listed in separate sections.
You now have a general idea about how to determine whether or not a piece of “news” or advice is reputable, depending on the website or blog where it appears, and also depending on the scientific studies on which it is based (if any). The good news is that it’s unlikely for a reputable source to disseminate unfounded information, so once you have established that an information outlet is credible you can relax the criteria, albeit you should never blindly trust its information. In general, if something seems too good to be true it deserves being double checked.
-Harvard T.H. Chan School of Public Health. Research Study Types. 2019. Accessed April 24, 2019. https://www.hsph.harvard.edu/nutritionsource/research-study-types/