Understanding the Design of Different Scientific Studies

Twenty-five years ago, the chances were slim that a study in a scientific journal would make the evening news. But no longer. Today, the public regularly reads or hears about a new study linking a specific food or food ingredient with a potential benefit or health threat.

Clearly, this information is compelling news. But, there is another reality associated with the reporting of new studies: increased public confusion about what the findings really mean. Behind this confusion is the public’s unfamiliarity with the scientific process that can make the evolutionary nature of research appear confusing and at times, contradictory. An even more significant factor is how those who interpret the study’s findings -- the scientists, the journals, the media, the government and the public interest groups -- relate the information to the public.

Since the studies reported by the media are usually published in peer-reviewed scientific or medical journals and not intended for the lay public, the Harvard School of Public Health and the International Food Information Council (IFIC) worked with leading nutrition researchers, food scientists, journal editors and print and broadcast reporters to develop specific guidelines about communicating new study findings. These guidelines stress the need for reporters to understand the different types of research studies so they can accurately relate the findings to the general public. At the same time, the guidelines make clear that other factors may influence the results of a study and their applicability to the general public, such as the number of subjects studied, the specific population, and the length of time that the subjects were monitored.

Because being able to identify and explain these factors is necessary for the accurate reporting of all scientific studies, IFIC published a white paper for journalists that describes how to understand and interpret food and health-related scientific studies. The following summarizes IFIC’s descriptions of the types of research and how they are used.

Types of Scientific Research

When it comes to conducting scientific studies, there are basically two categories of research: observational and experimental. In addition, researchers use the meta-analysis to reconcile differences among studies and aggregate relevant findings. The following describes these different types of research in greater detail.

Observational Research

Although observational research may be used in the laboratory, it is primarily conducted in a natural setting to study the relationship between a specific factor and some aspect of health or illness. For this reason, observational research may suggest an association but cannot be used to determine cause and effect. An example of observational research would be a study focusing on the body weight of healthy women aged 50 or older and its relationship to blood pressure in that group.

Compared to experimental studies, observational research designs are very simple. Fundamentally, the research team simply observes specific variables and linkages and uses statistical methods to determine if there is an association. But, there are many problems with this approach. The results of the research may confuse the true effect of a variable and the possible effects of other factors in the environment. That is why observational research is often followed by experimental studies that can validate cause and effect.

The most frequent type of observational research is the epidemiologic study, considered the basic science of public health. Usually focused on studying large groups --sometimes tens or hundreds of thousands of people -- epidemiology seeks to identify possible factors that increase the risk or probability of a disease. Through one type of epidemiological research -- the analytical study -- scientists observe certain behaviors, such as food choices, and track whether certain outcomes, such as the development of disease, occur. The other category of epidemiology -- the descriptive study -- collects information to characterize and summarize the health event or problem. For example, a descriptive study may examine the deaths associated with tractor accidents by such factors as time of day, the growing season, and the age of the individual.

Besides these different categories of epidemiology, scientists use a number of methods to conduct epidemiological studies, all of which have a number of significant limitations. One type of study design, the cross-sectional study, is basically the same as a survey. In this type of study, the epidemiologist defines the population to be studied and then collects information from members of the group about their disease and exposure status. Since the data represent a point in time, it's like taking a "snapshot" of the population. Cross-sectional studies are good for examining the relationship between a variable and a disease, but not for determining cause and effect, which requires data over time.

In a cohort study, scientists select the study population according to their exposure, regardless of whether the group has the disease or health outcome being studied. The researchers then determine the outcomes of interest and compare the results on the basis of the individuals' exposures. Cohort studies are often referred to as prospective studies because they follow the study population forward in time, from suspected cause to effect. An example would be dividing a group of people on the basis of their smoking status and following them for 20 years to see if they develop lung cancer.

Another option used by epidemiologists is the case-control study, where the research team works backward, from the effect to the suspected cause. For this reason, case-control studies are often referred to as retrospective studies. Participants are selected on the basis of the presence or absence of the disease or outcome in question, so there is one group of people (case-subjects) with the health problem and one without (controls). These groups are then compared to determine the presence of specific exposures or risk factors. An example of a case-control study would be to select a group of people with lung cancer and a group without and then compare the two groups for their history of exposure to smoking.

Regardless of the method used, the important thing to understand about the results of epidemiologic studies is that they are observations of associations, nothing more. By conducting observational studies, scientists can add valuable information to the existing literature on a particular topic, which helps them design future research studies, including clinical trials.

Experimental Research

Basic research generates data by investigating biochemical substances or biological processes. It is often conducted to confirm observations or to discover how a particular process works. For example, an experiment might be conducted to examine how vitamin E helps prevent oxidation of LDL (low-density lipoprotein) cholesterol, a process believed to play a role in the development of heart disease.

Basic research is often conducted in vitro -- such as in test tubes -- or with animals. Research with animals is an important tool in determining how humans may react when exposed to particular substances. However, it is important to note that, due to differences in physiology and the fact that animals are routinely exposed to levels of compounds far higher than those that human populations typically encounter, one cannot assume that results from animal studies can be generalized to humans.

When conducting experimental research, study subjects, whether human or animal, are selected according to relevant characteristics and are then randomly assigned to either an experimental group -- the group that will receive the treatment or intervention -- or a control group, which does not get the treatment. Random assignment ensures that factors, known as variables, that may affect the outcome of the study are distributed equally among the groups and therefore cannot lead to differences in the effect of a treatment. As a result, any differences in results between the groups can be attributed to the treatment. However, controlled experimental research can be fraught with errors, which is why it is important to know how the survey was designed and conducted.

The most significant type of experimental research is the clinical trial that uses human subjects to evaluate the effectiveness and safety of a nutrient or medical treatment by monitoring its effect on large groups of people. Researchers affiliated with a hospital or university medical program, independent researchers, or private industry generally conduct the clinical trials that may be small, with a limited number of participants, or may be large intervention trials that seek to discover the outcome of treatments on entire populations. The more participants in the study, the greater the likelihood that the results can be replicated in the general population.

Besides the size of the study, the other critical factor is how the research is designed. Here, the scientific community places the most value on the double-blind, placebo-controlled study that uses random assignment of subjects to experimental and control groups. Considered the "gold standard" of clinical research studies, the double-blind, placebo-controlled study provides dependable findings that are free of bias introduced by either the subject or the researcher. In this type of study, neither the subject nor the researcher conducting the study knows whether the test substance or a placebo has been administered. For the results to be valid and to ensure that the subject cannot violate the "blinding," the placebo and the test substance must be virtually identical (i.e., look, smell, and taste similar).

When conducting clinical trials, researchers go through a series of steps, called phases, each of which is designed to answer a separate research question. The National Institutes of Health defines the four phases of clinical trial research as follows:

Phase I: Researchers test a new treatment or intervention in a small group of people for the first time to evaluate its safety, determine a safe dosage range, and identify side effects.

Phase II: The treatment is given to a larger group of people to see if it is effective and to further evaluate its safety.

Phase III: The treatment is given to large groups of people to confirm its effectiveness, monitor side effects, compare it to commonly used treatments, and collect information that will allow the drug or treatment to be used safely.

Phase IV: Studies are done after the treatment has been marketed to gather information about its effect in various populations and any side effects associated with long-term use.
Obviously, the findings of a Phase III study are more significant for the American public than a Phase I or Phase II trial, where more study needs to be conducted. Thus, while preliminary findings may make for interesting reading, it is important for the public to recognize that the study only represents an initial step in the research process and needs to be confirmed through additional studies.

The Meta-Analysis

Conducted to reconcile differences among studies or to aggregate relevant findings across studies, the meta-analysis is a statistical method of combining results from separate studies to derive overall conclusions about a question or hypothesis. This method is most appropriate when examining studies that look at the same question and use similar methods to measure relevant variables. For example, using one type of meta-analysis, scientists examined the relationship between weight reduction and blood lipid levels. Although individual studies showed inconsistent results, pooling of data from 70 similar studies showed significant decreases in the levels of total cholesterol and other blood lipids due to weight loss.

The technique of meta-analysis is not without limitations, however. Data from flawed studies may be included, or the analysis may include data from studies that use different methods to measure variables, resulting in a comparison of apples to oranges. That is why IFIC and health educators recommend careful review of the methodology to ensure that the objective of the analysis is clearly stated and that the researchers explain the limitations of their findings so that the results can be put into context.

Understanding the Methodology

Besides knowing the limitations of the different types of studies, it is also important to understand how the study was conducted. That is why health educators stress the need to pay attention to the study’s methodology and specifically:

• The setting of the study (in a clinic, laboratory, population, etc.)
  
• How variables were controlled (how did they adjust for specific subject qualities or outside influences that could affect the results?)
  
• The sample size
  
• The number of study groups
  
• The treatment or variables being observed (e.g., a vitamin supplement or specific diets)
  
• The length of the study
  
• How the data were collected
  
• How and by what statistical procedures the data were analyzed

When evaluating the findings of a study, one of the most significant factors is study design and especially, randomness in selection of the study’s participants. If the subjects were selected randomly, the study results are more predictive of the population. However, in cases where the only participants in the study are those that volunteered because they belong to an interest group or a media organization, there is more potential for bias in conducting the study.

Another important factor is sample size. While a small sample size does not mean the study is flawed, there are clearly limitations to the conclusions that can be drawn. That is why federal agencies, like the Food and Drug Administration, rely on multiple studies with large numbers of subjects before approving a new treatment or intervention.

Of equal significance when interpreting a study’s findings is understanding the statistical significance. When conducting both observational and experimental research, scientists use statistical measures to convey the existence and strength of relationships. But while presenting the findings in an organized fashion, statistics do not provide information about cause and effect. Moreover, a statistically significant finding does not guarantee that the research is without biases or confounding factors that could make the statistical value irrelevant. Statistical significance is only part of the picture; to get the whole picture, one must consider the context of the study and what other research on the subject reveals.

Communicating Risk

Of all the challenges when interpreting the findings of a research study is communicating about potential risk. Here, educators stress the importance of understanding the differences between relative and absolute risks and using these distinctions when explaining the findings of a given study.

Absolute risk is defined as the chance of a person developing a specific disease over a specified time-period. For example, a woman's lifetime absolute risk of breast cancer is one in nine. That is to say, one woman in every nine will develop breast cancer at some point in their lives.

In contrast, relative risk puts the chance in comparative terms by describing the outcome rate for people exposed to the factor in question compared with the outcome rate for those not exposed to the factor. Although relative risks are the most commonly used measure of morbidity or mortality in the medical literature today, in most cases the absolute risk is a far more relevant statistic for the public.

For example, suppose that a study shows that a man who brushes his teeth only once a day is 50 percent more likely to have all his teeth fall out in the next 10 years than others who brush their teeth twice per day. This is the relative risk. Yet, the absolute risk that all of the man's teeth will fall out may be only 1 percent. In this case, the relative risk makes the problem seem more important than it really is. Therefore, it is important to consider both relative risk and absolute risk when discussing study results.

In Summary

In summary, those who interpret the findings of scientific studies for the public are the gatekeepers of today’s food and health information. By determining what consumers hear, read and believe about the safety and benefits of specific foods, nutrients and medical treatments, they have the responsibility to explain the results of new scientific studies and to put this information into proper context for the lay public. This requires providing adequate information on the study’s original purpose, research design and methods of data collection and analysis. It also necessitates an understanding of the limitations of specific types of research and recognizing when a study is preliminary or the findings differ from previous research.

But most importantly, properly interpreting a new study’s findings requires putting the potential risk into perspective. It also requires clarification about cause and effect, which can only be demonstrated through rigorous experimental studies, not epidemiological research.

How emerging science is communicated can have a powerful effect on the American public. That is why it is so important for all communicators -- scientists, journal editors, and the media -- to provide the context that will enable the average person to weigh the information appropriately. As Timothy Johnson, MD, the medical editor of ABC Good Morning America, stated in a preface to the Harvard guidelines: “I think what the public wants is for us to be honest with each study as it comes along and try to put it into perspective, but keep reminding people that it’s the totality of the evidence as it unfolds that warrants their attention.”