Internal and External Validity...Industrial Organizational Psychology Paper submitted by Sean Delevan

Internal and External Validity

RES731

Sean Delevan

31 July 2015

Professor XXXXXX XXXXX

Validity refers to the degree in which accurate assumptions can be made merely by the results of a research study. Internal validity is described as “validity of the inference that the independent and dependent variables are causally related” (Christensen et al, 2011, pg.2/25). Comparatively, external validity is explained to be the determination regarding whether the “causal relationship holds over people, settings, treatment variable, measurement variables, and time” (Christensen et al, 2011, pg.3/25). Although each are present in research and are necessary to ensure reliability and validity are achieved, there are annotatable threats associated with both internal and external validity that have to be considered during the research process. The purpose of this paper is to first review the notable threats to internal and external validity and then present ways in which to reduce the threats mentioned. The format of this paper will be in an APA appendices style so it is clear and easy to understand how the information pertains to each section.

First it is important to understand that internal validity, as it relates to research, is explained to be “how well and experiment is done and if it avoids confounding variables” (Campbell, 1986). There are seven clearly stated threats associated with internal validity that must be considered during research. Although the threats are present, there are ways in which researchers are able to circumvent these threats to ensure overall validity and reliability in research. The threats associated with internal validity are as follows: History, Maturation, Instrumentation, Testing, Regression, Attrition, and Selection.

The maximization of internal validity can generally be accomplished in four general ways according to research presented by Fraenkel and Wallen (1993). The first method of internal validity maximization is through the standardization of conditions wherein the research is occurring. This will aid in the lessened occurrence of threats to history and instrumentation. Additionally, researchers that are able to get more information about research study participants will see more success than those that don’t when it comes to minimizing threats associated with attrition and selection. Just as the obtaining of information regarding participants is critical to the success of the research and the lessened propensity of threats associated with such, the same is so for understanding procedural details.

Researchers must be prudent when it comes to genuinely understanding the details associated with the research procedures to ensure the minimization of threats associated with history and instrumentation. Finally, Fraenkel and Wallen (1993) submit that research will be exponentially more successful and experience far less influence from threats when the most appropriate research design is selected. If an inadequate research design is used, this will ultimately threaten the overall internal validity of the research making it unreliable.

Further ways in which researchers are able to reduce the risks associated with history would be the utilization of control groups that are an identical reflection of those that are involved in the experimental groups. This will aid in the assurance that the experiences are concurrent between the groups. Furthermore, the shorter the time frame of the research/experiment, the better it will be if the goal is to minimize threats associated with history since history threats are typically going to occur over an extended period of time. Ways that researchers can minimize the threats associated with maturation are going to be similar in nature to the aforementioned suggestions for reducing threats associated with history; implementation of a control group similar to the experimental group and a shortened duration of experimentation.

Schlueter, E., & Scheepers, P. (2010) explain that the most advantageous ways in which to reduce the threats associated with experimental testing is to use a research design that does not require the use of a pretest. “If baseline or pretreatment data is required, the use of data collection techniques that participants are unaware of will minimize the effects of testing” (Schlueter et al., 2010). Threats to testing comparable to instrumentation can also be lessened if researchers make it a point to standardize testing instruments so they can be specifically controlled and measurement procedures can be the same across the board. This will help to alleviate misinterpretations and inadequate training and observations.

Finally, internal validity threats such as regression and selection can be avoided by the utilization of large experimental groups that genuinely represent the larger population that the study is hoping to reflect upon. Doing this, and using follow-up procedures for those that leave the study early will help to present other pertinent data rather than leaving a gap. For information to be considered valid and genuine, the selection process should be represented as random selection so that group differences can be minimized. This can also be achieved through quasi-experimental research designs.

External validity, equally as important as internal validity in research is explained to be “validity of generalized inferences in research based on experiments” (Calder, Phillips, & Tybout, 1982). In layman’s terms, external validity is a reflection of the extent to which results can be generalized and applied to others. For external validity to be present in research, researchers must have a hypothesis suggesting that the variables being utilized in the research are comparable to the larger population the research is attempting to explain.

Christensen et al. (2011) explains that the most reliable ways in which to reduce the threats associated with population validity are for “researchers to generalize from the sample to the accessible population from which the sample was drawn. This can be easily accomplished if the researcher randomly selects the sample from the accessible population” (Christensen et al., 2011, pg.17/25). This type of random selection is similar in nature to the aforementioned suggesting regarding internal validity threats associated with attrition and selection so it will not be a drastic differentiation between the two.

Threats associated with ecological validity are noted to be present when the treatment effects are not independent of the experimental setting. There are a number of ways that this threat can be avoided, the most important of which would be to ensure that explicit descriptions are provided regarding the experimental treatment and the purpose of the experiment. “If the researcher fails to adequately describe how he or she conducted a study, it is difficult to determine whether the results are applicable to other settings” (Kavaliers & Choleris, 2001). Secondly, the application of multiple treatment interfaces will help to determine if the treatments in the study work on individual bases, thereby reducing the threats associated with ecological validity.

Other threats associated with external validity in research include temporal validity which is explained to be “the extent to which the results of an experiment or other type of research study can be generalized across time” (Christensen et al., 2011, pg.18/25). One of the easiest ways to circumvent issues associated with this type of threat to research is for researchers to be prudent in identifying and selecting more predictable time patterns. This is achieved when researchers are able to differentiate, and determine applicability as it relates to seasonal variations and cyclical variations in research. Cyclical variations are likely to be used the most because they provide a rhythm that is easy to identify regardless of the length of the study.

Finally, treatment variation validity and outcome validity threats can be lessened, if not altogether avoided by ensuring the generalizability of the results across different variables that are both independent and dependent on one another. “Outcome validity refers to the extent to which the same effect is measured by all related outcome measures” (Christensen et al., 2011, pg.19/25). So long as these methods are used by researchers to lessen the threat to internal and external validity, the reliability of the research and research outcomes will be considerably more ‘valid’.

Appendix A:

Threats to Internal Validity

Threat 1: History – Any past event that can produce the outcome during a study prior to post testing of the dependent variable. “history threat exists in this one-group design if an event occurs (other than the treatment) that can affect the dependent variable” (Christensen et al. 2011). This includes differential history which occurs when one group experiences the changes of the history event but the other group in the study does not.

Threat 2: Maturation- Changes in internal conditions such as “age, learning, hunger, boredom, and hunger that are not related to specific external events” (Christensen et al. 2011). Maturation changes are associated with the individual and are realized as biological and/or psychological processes that unwittingly change the outcome of the research.

Threat 3: Instrumentation- “Changes that occur over time, during the course of the study, in the measurement of the dependent variable” (Christensen et al., 2011, pg.12/25). Research that requires human observance is most likely to suffer from this type of threat.

Threat 4: Testing- Taking similar tests more than once can change the overall outcome as participants become more familiar with the process. Repetitive tests can desensitize participants and alter the genuine outcome, thus resulting in unreliable results.

Threat 5: Regression- Tendency of extreme scores to regress toward a more average score during retesting. When initial tests generate poor results, retaking the same test would expect participants to score higher.

Threat 6: Attrition- This occurs when participants drop out of a study. “Some individuals do not complete a research study for a variety of reasons, such as failure to show up at the scheduled time and place or not participating in all phases of the study” (Christensen et al., 2011, pg.14/25). This is most common in psychological and behavioral studies.

Threat 7: Selection- This threat is present when a differential selection procedure is used for placing participants in certain study groups. Groups in a study might possess different characteristics such as age, ability, or gender, which might affect the results.

Appendix B

Threats to External Validity

Threat 1: Population Validity- A testing sample may not actually represent that larger population similar to research participants. The accessible population may be substantially lower than the target population which will skew testing results.

Threat 2: Ecological Validity- “The generalizability of results of a study across different settings or from one set of environmental conditions to another” (Christensen et al., 2011, pg18/25). Lab experiments don’t generally produce a generalized assumptions for non-lab settings.

Threat 3: Temporal Validity- Testing results’ ability to be generalized across an extended period of time. Simply stated, this is a threat if the findings of the study are not held as true over an extended period of time.

Threat 4: Treatment Variation Validity- When the generalizability of study results vary due to different testing administrators. One administrator might be more apt to help participants than another which then changes the reliability and validity of the study altogether.

Threat 5: Outcome Validity- “Refers to the generalizability of results across different but related dependent variables” (Christensen et al., 2011, pg.19/25). This measures the same effect that an independent variable has on various dependent variables.

References

Campbell, D. T. (1986). Relabeling internal and external validity for applied social scientists. New Directions for Program Evaluation, 1986(31), 67-77.

Calder, B. J., Phillips, L. W., & Tybout, A. M. (1982). The concept of external validity. Journal of Consumer Research, 240-244.

Christensen, R., Burke, J., & Turner, L. (2011). Research methods, design, and analysis, Eleventh Edition. Pearson Education.

Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (1993). How to design and evaluate research in education (Vol. 7). New York: McGraw-Hill.

Kavaliers, M., & Choleris, E. (2001). Antipredator responses and defensive behavior: ecological and ethological approaches for the neurosciences.Neuroscience & Biobehavioral Reviews, 25(7), 577-586

Schlueter, E., & Scheepers, P. (2010). The relationship between outgroup size and anti-outgroup attitudes: A theoretical synthesis and empirical test of group threat-and intergroup contact theory. Social Science Research, 39(2), 285-295.