Research Methods Tutorials: Introductions to Quantitative and Qualitative Methods

Watch this 3:43 video from Matthew B. Courtney, Ed.D. on the difference between reliability and validity.

In the video, Dr. Courtney introduces the concepts of reliability and validity. Below we'll re-iterate some of the concepts in the video and share additional resources for further exploration of this topic. 

You can think about these concepts by asking yourself these questions:

Reliability: Would your test yield the same result every time?

Validity: Does your test measure what you're intending it to?

To fully answer these questions and prove the reliability and validity of your instrument further testing is required. 

Some ways of measuring reliability include testing and re-testing the same group of participants at different times to see how close the answers are, measure the internal consistency of responses, introduce alternate forms of an existing item and comparing the responses. Reliability is usually expressed as a correlation coefficient or r value between two different sets of data. Generally, an r value greater than 0.70 is preferred. 

Validity can be assessed a bit more simply. Here are two simple methods:

1. One way of doing this is sharing your survey instrument with a few individuals that have no formal training in your field or discipline. This is called Face Validity. Do these individuals feel that you're content matches what you're attempting to measure. It is essentially a set of fresh eyes. 
2. Another method is called Content Validity and involves sharing your instrument with experts in the field or some aspect of your study. They can use their expertise and experiences to judge the level at which your survey is measuring your intended variable. 

There are some factors found in the literature that may skew or distort your data when using a Likert Scale. These are difficult to account for in asynchronous surveys where the investigator in unable to observe or interact with the respondent. You'll see a few of them described below: 

Central Tendency Bias: A likelihood of some respondents to avoid answering on the far ends of the scale.

Acquiescence Bias: The tendency of some respondents to agree with the test item or statements as they are presented.

Environmental: The environment of the respondent at the time they are completing the survey is typically outside of the investigator's control. Adverse temperature or conditions may impact how individuals respond.

Fatigue: Responses may be influenced by the length of the survey instrument or the number of surveys that they are being asked to complete. Both of which can impact how focused respondents are while completing the survey.