Internal
and External Validity
RES731
Sean
Delevan
31 July
2015
Professor
XXXXXX XXXXX
Validity
refers to the degree in which accurate assumptions can be made merely by the
results of a research study. Internal validity is described as “validity of the
inference that the independent and dependent variables are causally related”
(Christensen et al, 2011, pg.2/25). Comparatively, external validity is
explained to be the determination regarding whether the “causal relationship
holds over people, settings, treatment variable, measurement variables, and
time” (Christensen et al, 2011, pg.3/25). Although each are present in research
and are necessary to ensure reliability and validity are achieved, there are
annotatable threats associated with both internal and external validity that
have to be considered during the research process. The purpose of this paper is
to first review the notable threats to internal and external validity and then
present ways in which to reduce the threats mentioned. The format of this paper
will be in an APA appendices style so it is clear and easy to understand how
the information pertains to each section.
First
it is important to understand that internal validity, as it relates to
research, is explained to be “how well and experiment is done and if it avoids
confounding variables” (Campbell, 1986). There are seven clearly stated threats
associated with internal validity that must be considered during research.
Although the threats are present, there are ways in which researchers are able
to circumvent these threats to ensure overall validity and reliability in
research. The threats associated with internal validity are as follows:
History, Maturation, Instrumentation, Testing, Regression, Attrition, and
Selection.
The
maximization of internal validity can generally be accomplished in four general
ways according to research presented by Fraenkel and Wallen (1993). The first
method of internal validity maximization is through the standardization of
conditions wherein the research is occurring. This will aid in the lessened
occurrence of threats to history and instrumentation. Additionally, researchers
that are able to get more information about research study participants will
see more success than those that don’t when it comes to minimizing threats
associated with attrition and selection. Just as the obtaining of information
regarding participants is critical to the success of the research and the
lessened propensity of threats associated with such, the same is so for
understanding procedural details.
Researchers
must be prudent when it comes to genuinely understanding the details associated
with the research procedures to ensure the minimization of threats associated
with history and instrumentation. Finally, Fraenkel and Wallen (1993) submit
that research will be exponentially more successful and experience far less
influence from threats when the most appropriate research design is selected.
If an inadequate research design is used, this will ultimately threaten the
overall internal validity of the research making it unreliable.
Further
ways in which researchers are able to reduce the risks associated with history
would be the utilization of control groups that are an identical reflection of
those that are involved in the experimental groups. This will aid in the
assurance that the experiences are concurrent between the groups. Furthermore,
the shorter the time frame of the research/experiment, the better it will be if
the goal is to minimize threats associated with history since history threats
are typically going to occur over an extended period of time. Ways that researchers
can minimize the threats associated with maturation are going to be similar in
nature to the aforementioned suggestions for reducing threats associated with
history; implementation of a control group similar to the experimental group
and a shortened duration of experimentation.
Schlueter,
E., & Scheepers, P. (2010) explain that the most advantageous ways in which
to reduce the threats associated with experimental testing is to use a research
design that does not require the use of a pretest. “If baseline or pretreatment
data is required, the use of data collection techniques that participants are
unaware of will minimize the effects of testing” (Schlueter et al., 2010).
Threats to testing comparable to instrumentation can also be lessened if
researchers make it a point to standardize testing instruments so they can be
specifically controlled and measurement procedures can be the same across the
board. This will help to alleviate misinterpretations and inadequate training
and observations.
Finally,
internal validity threats such as regression and selection can be avoided by
the utilization of large experimental groups that genuinely represent the
larger population that the study is hoping to reflect upon. Doing this, and
using follow-up procedures for those that leave the study early will help to
present other pertinent data rather than leaving a gap. For information to be
considered valid and genuine, the selection process should be represented as
random selection so that group differences can be minimized. This can also be
achieved through quasi-experimental research designs.
External
validity, equally as important as internal validity in research is explained to
be “validity of generalized inferences in research based on experiments”
(Calder, Phillips, & Tybout, 1982). In layman’s terms, external validity is
a reflection of the extent to which results can be generalized and applied to
others. For external validity to be present in research, researchers must have
a hypothesis suggesting that the variables being utilized in the research are
comparable to the larger population the research is attempting to explain.
Christensen
et al. (2011) explains that the most reliable ways in which to reduce the
threats associated with population validity are for “researchers to generalize
from the sample to the accessible population from which the sample was drawn.
This can be easily accomplished if the researcher randomly selects the sample
from the accessible population” (Christensen et al., 2011, pg.17/25). This type
of random selection is similar in nature to the aforementioned suggesting
regarding internal validity threats associated with attrition and selection so
it will not be a drastic differentiation between the two.
Threats
associated with ecological validity are noted to be present when the treatment
effects are not independent of the experimental setting. There are a number of
ways that this threat can be avoided, the most important of which would be to
ensure that explicit descriptions are provided regarding the experimental
treatment and the purpose of the experiment. “If the researcher fails to
adequately describe how he or she conducted a study, it is difficult to
determine whether the results are applicable to other settings” (Kavaliers
& Choleris, 2001). Secondly, the application of multiple treatment
interfaces will help to determine if the treatments in the study work on
individual bases, thereby reducing the threats associated with ecological
validity.
Other
threats associated with external validity in research include temporal validity
which is explained to be “the extent to which the results of an experiment or
other type of research study can be generalized across time” (Christensen et
al., 2011, pg.18/25). One of the easiest ways to circumvent issues associated
with this type of threat to research is for researchers to be prudent in
identifying and selecting more predictable time patterns. This is achieved when
researchers are able to differentiate, and determine applicability as it relates
to seasonal variations and cyclical variations in research. Cyclical variations
are likely to be used the most because they provide a rhythm that is easy to
identify regardless of the length of the study.
Finally,
treatment variation validity and outcome validity threats can be lessened, if
not altogether avoided by ensuring the generalizability of the results across
different variables that are both independent and dependent on one another. “Outcome
validity refers to the extent to which the same effect is measured by all
related outcome measures” (Christensen et al., 2011, pg.19/25). So long as
these methods are used by researchers to lessen the threat to internal and
external validity, the reliability of the research and research outcomes will
be considerably more ‘valid’.
Appendix A:
Threats
to Internal Validity
Threat 1: History – Any
past event that can produce the outcome during a study prior to post testing of
the dependent variable. “history threat exists in this one-group design if an
event occurs (other than the treatment) that can affect the dependent variable”
(Christensen et al. 2011). This includes differential
history which occurs when one group experiences the changes of the history
event but the other group in the study does not.
Threat 2: Maturation-
Changes in internal conditions such as “age, learning, hunger, boredom, and
hunger that are not related to specific external events” (Christensen et al.
2011). Maturation changes are associated with the individual and are realized
as biological and/or psychological processes that unwittingly change the
outcome of the research.
Threat 3: Instrumentation-
“Changes that occur over time, during the course of the study, in the
measurement of the dependent variable” (Christensen et al., 2011, pg.12/25).
Research that requires human observance is most likely to suffer from this type
of threat.
Threat 4: Testing- Taking
similar tests more than once can change the overall outcome as participants become
more familiar with the process. Repetitive tests can desensitize participants
and alter the genuine outcome, thus resulting in unreliable results.
Threat 5: Regression-
Tendency of extreme scores to regress toward a more average score during
retesting. When initial tests generate poor results, retaking the same test
would expect participants to score higher.
Threat 6: Attrition- This
occurs when participants drop out of a study. “Some individuals do not complete
a research study for a variety of reasons, such as failure to show up at the
scheduled time and place or not participating in all phases of the study”
(Christensen et al., 2011, pg.14/25). This is most common in psychological and
behavioral studies.
Threat 7: Selection- This
threat is present when a differential selection procedure is used for placing
participants in certain study groups. Groups in a study might possess different
characteristics such as age, ability, or gender, which might affect the
results.
Appendix B
Threats
to External Validity
Threat 1: Population Validity- A
testing sample may not actually represent that larger population similar to
research participants. The accessible population may be substantially lower
than the target population which will skew testing results.
Threat 2: Ecological Validity- “The
generalizability of results of a study across different settings or from one
set of environmental conditions to another” (Christensen et al., 2011,
pg18/25). Lab experiments don’t generally produce a generalized assumptions for
non-lab settings.
Threat 3: Temporal Validity-
Testing results’ ability to be generalized across an extended period of time.
Simply stated, this is a threat if the findings of the study are not held as
true over an extended period of time.
Threat 4: Treatment Variation Validity- When
the generalizability of study results vary due to different testing
administrators. One administrator might be more apt to help participants than
another which then changes the reliability and validity of the study
altogether.
Threat 5: Outcome Validity-
“Refers to the generalizability of results across different but related
dependent variables” (Christensen et al., 2011, pg.19/25). This measures the
same effect that an independent variable has on various dependent variables.
References
Campbell, D. T. (1986). Relabeling internal and
external validity for applied social scientists. New Directions for
Program Evaluation, 1986(31), 67-77.
Calder, B. J., Phillips, L. W., & Tybout,
A. M. (1982). The concept of external validity. Journal of Consumer
Research, 240-244.
Christensen, R., Burke, J., & Turner, L.
(2011). Research methods, design, and
analysis, Eleventh Edition. Pearson Education.
Fraenkel, J. R., Wallen, N. E., & Hyun, H.
H. (1993). How to design and
evaluate research in education (Vol.
7). New York: McGraw-Hill.
Kavaliers, M., & Choleris, E. (2001).
Antipredator responses and defensive behavior: ecological and ethological
approaches for the neurosciences.Neuroscience & Biobehavioral Reviews, 25(7),
577-586
Schlueter,
E., & Scheepers, P. (2010). The relationship between outgroup size and
anti-outgroup attitudes: A theoretical synthesis and empirical test of group
threat-and intergroup contact theory. Social Science Research, 39(2),
285-295.
