Unit 08: How to Evaluate Scientific Reliability and Validity

Unit 8: How to Evaluate Scientific Reliability and Validity

Unit 8: Assignment #1 (due before 11:59 pm Central on Wednesday October 6):

To learn what reliability and validity mean in research:
1. Read an excerpt from Shuttleworth’s (no date) article, “Validity and Reliability” (web link).
2. Read a student from Bangor University’s (2011) blog post, “Reliability vs. Validity” (web link).
To learn what external versus internal validity means in research, read Bradley’s (no date), “Reliability, Validity, and Bias” (original PDF).
1. Although you should be sure you understand both terms, reliability and validity, and their difference, this assignment will focus on reliability. (Another assignment will focus on validity.)
2. You’ve probably noticed that reliability is, in many ways, the same as replicability and reproducibility: Reliable findings can be replicated and reproduced.
3. Therefore, the issues you learned about replicability and reproducibility in Unit 4 (particularly, Unit 4: Assignments #5 and #6) are issues of reliability.
To dig even deeper into the issues of reliability, replicability, and reproducibility, read Munafó et al.’s (2017) article, “A Manifesto for Reproducible Science” (Text-Only Word version) (web link).
1. NOTE: Munafó et al.’s (2017) article is complex (it’s not Huffington Post-style writing).
  - You might need to read it a few times to understand it. Please be persistent.
2. Munafó et al. (2017) suggest 10 procedures for making science, including psychological science, more reproducible (replicable and reliable).
  - Their 10 suggested procedures are listed in Table 1 on the third page, and each is explained in the article.
3. Choose three of Munafó et al.’s (2017) 10 suggested procedures.
  - Choose the three suggested procedures that you like the most (or that you understand the best).
4. Write one five-paragraph Examples-Style essay to support the thesis “Scientists Can Improve Research Reliability.”
  - For this essay, the three procedures you have chosen from Munafó et al.’s (2017) article will be your three ‘examples’ in your Examples-Style essay.
If you need to review how to write a Five-Paragraph Essay, including how to write an Examples-Style essay, watch again Professor Gernsbacher’s lecture video, “The Five-Paragraph Model” (a transcript of the video is available in Word).
1. Check your essay to make sure your Introduction Paragraph has a hook and a Thesis Statement.
2. Check your Thesis Statement to make sure that it is one sentence that incorporates your three examples/procedures.
3. Check your essay to make sure it has three Supporting Paragraphs, each introducing one of your three examples/procedures.
4. Check your essay to make sure it has a Conclusion Paragraph.
5. Check your Conclusion Paragraph to make sure it has one sentence that restates your Thesis Statement (summarizing your three examples/procedures).
6. Check ALL five of your paragraphs to make sure each paragraph has a Topic Sentence, three Supporting Sentences, and a Conclusion Sentence.
Because you’ll be drawing a lot of ideas from Munafó et al.’s (2017) article, review Unit 6: Assignment #4 to make sure you don’t plagiarize.
1. Be sure to include in-text citations to Munafó et al.’s (2017) article whenever needed, which is — of course — after every sentence in which you paraphrase any idea from their article.
2. Try to not make Munafó and his co-authors the grammatical subjects of your sentences. Instead, make the ideas (e.g., the three examples/procedures you’ve chosen) the grammatical subjects of your sentences.
3. Remember, from Unit 6, how to synthesize psychological science, not Mad Lib it. Be sure to review the Synthesizing Research cheat-sheet (Word).
4. Include a full citation to Munafó et al.’s (2017) article, in APA style, at the end of your essay.
5. Save your essay as a PDF and name the file YourLastname_ReliabilityEssay.pdf.
Go to the Unit 8: Assignment #1 Discussion Board and attach your essay, saved as a PDF.

Unit 8: Assignment #2 (due before 11:59 pm Central on Thursday October 7):

Reliable and valid studies have reliable and valid operational definitions. To learn what operational definitions are, read the APA Dictionary of Psychology’s (no date) definition of “Operational Definition,” and read this (no date) definition from Quality Advisor℠.
To understand why consistent, precise, logical, and justifiable operational definitions are crucial for reliability and validity, read Dewey’s (2007) chapter, “Operational Definitions” (web link).
To appreciate the difficulty of generating reliable, precise, and valid operational definitions, read this excerpt (full chapter) from Lammers and Badia ’s (2013) textbook, Fundamentals of Behavioral Research.
To demonstrate the difficulty of generating reliable, precise, and valid operational definitions, complete this fillable PDF, which contains a list of 10 fortunes, such as one would see in a Magic 8 Ball game. Each fortune has at least one term bolded.
1. First, download the unfilled PDF and save it on your own computer.
  - Rename the unfilled PDF to be YourLastName_PSY-311_OperationalDefinitions _Fillable.pdf. (In other words, add your LastName to the beginning of the filename.)
  - On your computer, open a PDF writer app, such as Preview, Adobe Reader, or the like.
  - Be sure to open your PDF writer app before you open the unfilled PDF from your computer.
  - From within your PDF writer app, open the unfilled PDF, which you have already saved onto your computer and re-named.
2. Second, using your PDF writer app, fill in the PDF with your operational definition of each of the bolded terms.
  - For example, if the fortune is “You will inherit some money or a small piece of land,” fill in the PDF with your operational definition of “some money” and “a small piece of land.”
    - What is your operational definition of “some money“? Is it $50? $100? $1000? $100,000? One million dollars? Five million dollars?
    - What is your operational definition of “a small piece of land“? Is it the size of a typical front yard? The size of football field? An acre? Several acres? A hundred acres?
  - As Lammers and Badia  write in their (2013) textbook, “there are no correct answers to these questions. However, the researcher must decide what is going to be meant by each of the variables in a particular study.” And that is what you will be doing in this assignment!
3. Third, teach a friend (or family member) what operational definitions are.
  - Then, ask this friend/family member to tell you their operational definition of each of the bolded terms.
  - Fill in your friend/family member’s operational definitions on your PDF.
  - Although you will first need to explain to your friend/family member what an operational definition is, do not tell your friend/family member your operational definitions of the bolded terms before your friend/family member tells you their operational definitions
4. Fourth, teach a second friend (or family member) what operational definitions are.
  - Then, ask this second friend/family member to tell you their operational definition of each of the bolded terms.
  - Fill in your second friend/family member’s operational definitions on your PDF.
  - Again, you will need to explain to your friend/family member what an operational definition is, BUT do not tell your second friend/family member your operational definitions of the bolded terms (or your first friend/family member’s operational definitions) before your second friend/family member tells you their operational definitions.
Go to the Unit 8: Assignment #2 Discussion Board and make a new post of at least 200 words in which you do the following:
1. Identify which of the bolded terms you and your friends/family members showed the most and the least reliability (consistency) on when each of you (independently) defined those terms operationally.
  - Speculate why those terms led to the most versus least reliability (consistency).
2. Identify which of the bolded terms you and your friends/family members operationally defined in the most versus least valid (credible and believable) ways.
  - Speculate why those terms led to the most versus least validity (credibility).
3. Attach your filled-in PDF.

Unit 8: Assignment #3 (due before 11:59 pm Central on Friday October 8):

Although the construct of validity is a general research methods term, when the construct of validity is applied to tests, such as IQ tests or personality inventories, we use the specific term “test validity.”
To learn what the specific term “test validity” means and to be introduced to the major ways that test validity can be assessed, read a section from Heffner’s (no date) “Chapter 7.3 Test Validity” (web link).
Now, think back to one of the exams you took last semester in one of your courses. In fact, think about the most memorable exam you took (either because it was memorably positive or memorably negative). Refer to this exam as “Text X.”
Go to the Unit 8: Assignment #3 Discussion Board and make a new post of at least 200 words in which you discuss Test X’s
1. construct validity (did the exam measure what it was supposed to measure?);
2. concurrent validity (was the exam like other comparable exams, for example, the exams your friends took in a similar course?);
3. content validity (were the items on the test representative of the content you were expected to learn?); and
4. predictive validity (how well did the exam predict your ability to use the information in the future?).

Unit 8: Assignment #4 (due before 11:59 pm Central on Saturday October 9):

Because we’re a bit more than halfway through the course, it’s a good time to revisit things you’ve learned so far in the course (which is why traditional lecture-based courses usually have midterm exams around the middle of the term).
Listed below are Assignments you previously completed (or were supposed to complete). Choose one Assignment from the list that you enjoyed doing before and re-do that Assignment, according to the Assignment’s original instructions AND the additional requirements specified below.
1. Unit 1: Assignment #3 (choose three situations that are different from the three situations you chose before and are different from the three situations that other students chose before)
2. Unit 1: Assignment #5 (teach the Cognitive Biases to three people who are different from the three people you taught before)
3. Unit 2: Assignment #5 (make a different teaching document than you made before, aimed at a different audience than you aimed at before and using a different medium than you used before)
4. Unit 3: Assignment #4 (write your two essays on two quotes that are different from the two quotes you wrote essays on before; however, as before, one of your two essays must be based on a “Reasons/Arguments Prompt” quote and the other must be based on an “Examples Prompt” quote)
5. Unit 3: Assignment #5 (write your two essays in response to two tasks that are different from the two tasks you completed before)
6. Unit 4: Assignment #2 (find and analyze three different news stories than you found before)
7. Unit 4: Assignment #3 (teach correlation is not causation to three people who are different from the three people you taught before)
8. Unit 5: Assignment #1 (choose three topics that are different from the topics you chose; therefore, you’ll need to find three articles that are different from the articles you found before)
9. Unit 5: Assignment #2 (find and analyze ten popular press articles that are different from the ten popular press articles you found before)
10. Unit 7: Assignment #2 (make a different teaching document than you made before, aimed at a different audience than you aimed at before and using a different medium than you used before)
11. Unit 7: Assignment #3 (choose a different set of three of your 9 articles than you analyzed before)
Go to the Unit 8: Assignment #4 Discussion Board and make a new post containing everything that was previously required for the Assignment you’ve now re-done.
1. Also, explain in just a sentence or two why you chose this Assignment to do again.
2. Be sure to state somewhere in your post the Unit number and Assignment number (e.g., Unit 1: Assignment #3) of the Assignment you chose to do again.

Unit 8: Assignment #5 (due before 11:59 pm Central on Monday October 11):

To prepare for administering online surveys (Unit 10), you need to complete online ethics training (research ethics will be covered in Unit 9). The training is provided by the Collaborative Institutional Training Initiative (CITI Program). Researchers typically refer to it simple as “CITI Training.” After you complete the training, you will receive a completion certificate; this will be valid for 3 years. To complete the training:
1. Follow these instructions—provided by WSU’s Institutional Review Board—to complete the CITI Human Subjects training.
2. When you complete the trainings, save your competition certificate to your computer as a PDF. It should look like this sample certificate (but with your name and details).
  1. Download the sample certificate.
  2. Compare your certificate to the sample certificate. Double check and make sure that your certificate is for the course titled “Students conducting no more than minimal risk research.”
3. On Blackboard, go to Unit 8, Assignment #5 and attach your CITI completion certificate (PDF).
  1. The Unit 8, Assignment #5 link is an Assignment link, rather than a Discussion Board link, so it will look a bit different that what you’re used to.
  2. Click “Browse My Computer” to upload the PDF file from your computer.
  3. Make sure that you do not upload the 2-page completion report, which CITI also gives you. The completion report is not the same as the certificate. You should only upload the 1-page certificate (and not the 2-page completion report).

Unit 8: Assignment #6 (due before 11:59 pm Central on Tuesday October 12):

At this point in the course we will construct new Chat Groups. Please follow closely the instructions below.
If your last name is LAST alphabetically in your “old” Chat Group (see “Groups and Contacts” on Blackboard), these are your responsibilities for forming a new Chat Group:
1. You’re responsible for recruiting new persons to your new Chat Group. You can contact students via their shockers.wichita.edu email. Make sure to check for the updated list on Blackboard under “Groups and Contacts.” That list indicates which students are actively participating in the course. You’ll want to make sure that you are recruiting new persons who are actively participating in the course.
2. You may recruit any student to your new Chat Group except
  - you may NOT recruit a student who was a member of your “old” Chat Group (because the goal during the second half of the course is to mix up the Chat Groups a bit); and
  - you may NOT recruit a student whose last name is also last alphabetically in their “old” Chat Group (because those students whose last names are last alphabetically in their old Chat Groups will also be responsible for recruiting students to their new Chat Group).
3. If your “old” Chat Group had three members (including you), then your new Chat Group should also have three members (including you); if your “old” Chat Group had only two members, then your new Chat Group will have only two members (including you).
4. When recruiting new members to your new Chat Group:
  - It doesn’t matter if the students you are recruiting were previously in a two-member or three-member “old” Chat Group.
  - It only matters how many members you (the person whose last name comes LAST alphabetically and therefore the person doing the recruiting) had in your “old” Chat Group.
  - Similarly, it doesn’t matter if the students you are recruiting previously worked together in their “old” Chat Group; it only matters that none of the students you are recruiting are students YOU worked with before in your “old” Chat Group.
5. Appoint one member of your new Chat Group to be the person who will set up the group text chat room for all subsequent chats.
  - Remind this person that the instructions for setting up a new chat room are below in III.
  - You must NOT be the Chat Group member to set up the chat room for each chat; rather, you must appoint someone else — a member of your new Chat Group — to be the person who will set up the group text chat rooms for your chats.
6. Arrange with your new Chat Group a time when all of you can meet online for one hour to hold your Group Chat for the next Unit (Unit 9: Assignment #6).
7. Write down
  - the names of the members of your new Chat Group (including you);
  - who (of the new member/s of your new Chat Group) you appointed to set up the chat room for each chat; and
  - the name of your new Chat Group (which will be the name of your “old” Chat Group)
  - the date and time that your new Chat Group has agreed to meet to hold your Group Chat for the next Unit (Unit 9: Assignment #6).
If you are appointed to set up the new chat room for your new Chat Group, these are your responsibilities:
1. Read the instructions in the Course How To for “How to Set Up a Group Text Chat Room.”
2. Read the instructions in the Course How To for “How to Save and Attach a Group Text Chat Transcript.”
3. Arrange with your new Chat Group a time when all of you can meet online for one hour to hold your Group Chat for the next Unit (Unit 9: Assignment #6)
4. If the due date for this assignment is approaching, and you haven’t been contacted about being in a new Chat Group, reach out to those students who are responsible for recruiting new Chat Groups.
If you are in a NEW THREE-student Chat Group AND you were NOT appointed to set up the new chat room for your new Chat Group, write down:
1. the names of the members of your new Chat Group (including you);
2. the name of your new Chat Group (which will be the name of the “old” Chat Group of the student who recruited you to the new Chat Group);
3. the name of the student who recruited you to the new Chat Group; and
4. the date and time that your new Chat Group has agreed to meet to hold your Group Chat for the next Unit (Unit 9: Assignment #6)
5. If the due date for this assignment is approaching, and you haven’t been contacted about being in a new Chat Group, reach out to those students who are responsible for recruiting new Chat Groups.
EVERYONE: Continuing our theme of re-visiting at the middle of the term what you’ve learned so far in the course:
1. Ensure that your Course Journal is up-to-date and contains entries for Units 1 through 8, as well as your Goals Setting Entry.
2. Save your Course Journal, along with your Goals Setting Entry as a PDF, titled LastName_CourseJournal_1-8.pdf.
3. If your Course Journal is hand-written (or hand-drawn), you will need to scan all the pages into one PDF.
4. Go to the Unit 8: Assignment #6 Discussion Board and make a new post to which you attach your Course Journal PDF. In addition, in the post itself, do one of the following:
  - If your last name came LAST alphabetically in your “old” Chat Group, post the names of the members of your new Chat Group (including you); the name of your new Chat Group; who you appointed to set up the chats; and the date and time that your new Chat Group has agreed to meet to hold your Group Chat for the next Unit (Unit 9: Assignment #6).
  - If you are appointed to set up the chat rooms for your new Chat Group, post the names of the members of your new Chat Group (including you); the name of your new Chat Group; the name of the student who recruited you to the new Chat Group; and the date and time that your new Chat Group has agreed to meet to hold your Group Chat for the next Unit (Unit 9: Assignment #6).
  - If you are in a THREE-student NEW Chat Group AND you were NOT appointed to set up the new chat room, post the names of the members of your new Chat Group (including you); the name of your new Chat Group; the name of the student who recruited you to the new Chat Group; and the date and time that your new Chat Group has agreed to meet to hold your Group Chat for the next Unit (Unit 9: Assignment #6).

Congratulations, you have finished Unit 8! Onward to Unit 9!

Open-Access Active-Learning Research Methods Course by Morton Ann Gernsbacher, PhD is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. The materials have been modified to add various ADA-compliant accessibility features, in some cases including alternative text-only versions.
Permissions beyond the scope of this license may be available at http://www.gernsbacherlab.org.