Unit 11: How to Design and Administer Online Surveys

Unit 11: Assignment #1 (due before 11:59 pm Central on Wednesday October 27):

In this Unit, you will be constructing and administering online surveys. To learn why it’s important that you learn how to design and administer online surveys:
1. Review from Unit 1: Assignment #6 Brooklyn College’s 10 Things Employers Want You to Learn in College (scanned PDF version); (Text-Only PDF version).
2. . Note that on p. 12, under the heading “Gathering Information,” the skill of “Construct Surveys” is listed.
3. Read Tague’s (2004a) brief introduction, “When to Use a Survey,” which addresses the use of surveys outside of psychological science (i.e., beyond the purposes of basic scientific research).
4. Read cognitive psychologist J. P. de Ruiter’s (2017) tweet (PDF version); (Text-Only Word Version) about the need to be knowledgeable when constructing surveys.
At WSU, administering surveys as a class assignment requires a special type of IRB (Institutional Review Board) ethical approval. It’s called “Clearance – Class Projects.”
1. You do not need to do anything yourself to obtain this clearance. This is something that Joel has done on your behalf.
2. You should have completed the CITI training course (Unit 8, Assignment #5). If you did so, then you are cleared to administer surveys as part of the class assignment. But note that you are only cleared to administer surveys from the list of topics below (c).
  1. If you did not submit the appropriate CITI training complete certificate as part of Unit 8, Assignment #5, then you are not permitted to collect data from anyone for the class project. Contact Joel immediately.
3. You can view a copy of the IRB clearance/approval on Blackboard. Look for “IRB Approval” in the menu on the left-hand side.
  1. You are only authorized to collect data if your name is listed on the IRB clearance/approval document. If your name is not listed, contact Joel immediately.
4. As part of the clearance/approval, you are not permitted to refer to your project as “research” (e.g., in the online survey you will create). Instead, you should always refer to it as a “class project.”
To consider topics for your surveys:
1. Read closely through the PSY 311: Online Survey Topics handout.
2. Be sure to read the two important notes on the right-hand side of the first page (one about assessing your access to research participants and the other about not soliciting students in this class or other classes who you don’t already know).
Select two survey topics (from the PSY 311: Online Survey Topics handout). Be sure to select two topics
1. that you’re interested in investigating, and
2. for which you have access to the appropriate survey participants.
Read and synthesize previously published psychological science on each of the two topics you selected.
1. For each of the two survey topics you’ve selected, search Google Scholar for relevant scientific articles (using the procedure you learned in Unit 5).
  - Remember you can also find related articles by examining the “Cited By” tool and by examining an article’s reference list.
2. For each of the two survey topics you’ve selected, find three scientific articles (via Google Scholar).
  - The three articles can be at any level in the Hierarchy of Scientific Evidence, and
  - all three articles can be at the same level in the Hierarchy of Scientific Evidence.
3. For each of the two survey topics you’ve selected:
  - Read and analyze the three scientific articles you’ve found (using the procedure you learned in Unit 5).
  - Write one paragraph synthesizing the three articles you’ve found (using the procedure you learned in Unit 6).
  - Remember to synthesize the findings and not Mad Lib them; to write about behavior and phenomena, not researchers and their studies; to place each article’s in-text citation at the end of the sentence (in parentheses); and to synthesize conflicting results (e.g., using “However, ….”).
  - Lastly, include the three scientific articles’ full APA-style citations after each of your two synthesis paragraphs (using the procedure you learned in Unit 5).
Go to the Unit 11: Assignment #1 Discussion Board and make a new post that includes the following six parts:
1. In at least 50 words,
  - explain why you think constructing and administering surveys is a skill employers want you to learn in college, and
  - state at least two uses of online surveys for purposes other than basic scientific research. (These uses can come from Tague’s article or from surveys you’ve seen yourself.)
2. In at least 50 words,
  - explain whether your designing and administering a survey as a class assignment requires IRB (Institutional Review Board) approval.
  - Do you agree with this policy? Why or why not?
3. Post the statements:
  - I understand that the two surveys I will be constructing and administering in this course are solely for the purpose of a class assignment.
  - I understand that I am not allowed to solicit as research participants people I do not know.
  - I understand that I am not allowed to ask instructors (including the instructor of this class) if I can solicit research participants from their class.
  - I understand that my research participants can be other members of this class, but I must already know the students, and I cannot ‘cold-call’ other class members.
4. State the two topics you’ve selected for your two surveys.
5. Post the two paragraphs you’ve written that synthesize previous research on your two survey topics.
6. Include the APA-style full citations at the end of both paragraphs.

Unit 11: Assignment #2 (due before 11:59 pm Central on Thursday October 28):

To get a basic introduction to writing survey items, read Science Buddies’ (no date) article, “Designing a Survey.” This article provides a basic introduction; if you already have some experience writing survey items, you may skim (though not skip) this article.
To become more informed about writing survey items:
1. Read Beretta’s (2014) article, “Top Ten Common Problems in Designing Effective Survey Questions (PDF Version); (Text-Only Word version). Make sure you understand all ten problems Beretta describes – and you know how to avoid all ten problems.
2. Read Pew Research Center’s (no date) article, “Questionnaire Design.” Make sure you understand
  - open- versus closed-ended questions (the latter are what Science Buddies refers to as ‘structured questions’);
  - the importance of asking questions that are clear and well-specified;
  - what double-barreled questions and double-negatives are (and how to avoid them);
  - what acquiescence bias and social desirability bias are (and how to avoid them);
  - factors to consider in ordering your questions; and
  - the importance of placing demographic items last.
3. To appreciate the power of question wording, look at Britain Elects’ (2017) pair of tweets (PDF version); (Text-Only Word version).
4. To appreciate the importance of including the least number of survey items possibles, look at cognitive psychologist, J. P. de Ruiter’s (2018) tweet (PDF version); (Text-Only Word version).
5. Read Harvard University’s Program on Survey Research (2007) “Tip Sheet on Question Wording.” Make sure you understand
  - how to avoid technical jargon, vague or imprecise terms;
  - how to avoid double-barreled questions (again!);
  - how to avoid leading, emotional, or evocative language; and
  - how to use ordinal scales, reference frames, and unique answer choices that cover all response options.
6. Read Peters’ (no date) article, “How to Design a Survey” (PDF Version); (Text-Only Word version). Make sure you understand the differences between
  - categorical (also known as nominal) and
  - ordinal survey items (and survey responses).
Make a teaching document that captures ALL the information you learned in steps a. and b. above.
1. Your audience for your teaching document needs to be either other college students or people in industry (not psychological scientists).
2. Your medium for your teaching document needs to be either a PPT or an Infographic.
3. Save your teaching document as a PDF, named YourLastname_SurveyDesign.pdf.
Test yourself on Professor Rennison’s (no date), “Examples of Bad Questions & How to Fix Them (PDF version); (Text-Only Word version).
Go to the Unit 11: Assignment #2 Discussion Board and make a new post in which you
1. attach your teaching document;
2. tell us the intended audience of your teaching document and why you chose that intended audience; and
3. tell us how well you did identifying the examples of bad questions and fixing them (on Professor Rennison’s quiz).

Unit 11: Assignment #3 (due before 11:59 pm Central on Friday October 29):

Now it’s time to apply everything you’ve learned about writing survey items and write the items for your two surveys.
1. For each of your two surveys, you must write no fewer than 5 and no more than 10 items.
2. For each of your two surveys, you must include at least one and no more than two open-ended items (which count toward your total 5 to 10 total items per survey).
3. For each of your two surveys, you must include
  - at least one categorical/nominal survey item and
  - at least one ordinal survey item.
4. For each of your two surveys, if you include demographic items, you must place them at the end (unless there’s a good reason to place them earlier), and your demographic items count toward your total 5 to 10 total items per survey.
After you’ve written a draft of your items for each of your two surveys:
1. Check your items against the teaching document you made in Unit 11: Assignment #2.
2. Be sure to follow the beneficial procedures you told others to follow.
3. Be sure you don’t commit any errors you told others not to commit.
Save your two sets of items in one PDF, making it clear in your PDF which survey topic each set of items was written to assess. Name your PDF YourLastname_SurveyItems.pdf.
Go to the Unit 11: Assignment #3 Discussion Board and attach your PDF that includes your two sets of items.

Unit 11: Assignment #4 (due before 11:59 pm Central on Sunday October 31):

Explore and then select one of the following (free) online survey platforms:
1. SurveyMonkey;
2. Google Forms;
3. LimeSurvey; or
4. SurveyGizmo.
Using the online survey platform you’ve selected, create each of your two surveys. For the title of each of your surveys, use your research topic (e.g., Multi-Tasking while Studying; Student Debt; Texting while Driving; Food Speed versus Food Quality).
1. Make sure you include a consent form at the start of each survey.
  1. Copy the text from this Consent Form template.
  2. Paste the text into your survey platform and replace/remove the red font.
  3. You should split the Consent Form into two “questions” (all on the same “page” or “section”). The first will be a multiple-choice question (yes/no). Here, you should paste the bulk of the consent form, ending with the question “I have read the above and agree to participate in this class project.” The second question should also be a multiple-choice question (yes/no); it should just contain the question “I am age 18 or over.”
  4. You should make those two questions “required” questions, meaning that participants must complete them to proceed to the main survey questions.
  5. You need to make sure that people can only continue to the main part of the survey if they answer “yes” to both questions. If they answer “no” to one or both questions at the end of the consent form, then they should not be able to start the survey. Most survey platforms have settings to allow this.
  6. Here is an example that was created using Google Forms.
  7. Make sure that as you prepare the consent form and survey:
    1. You do not state that your class project is approved by the WSU Institutional Review Board (IRB). The appropriate way to think about this is that you have been given clearance to conduct a class project. You have NOT been given approval to conduct research. There is a technical difference between a class project and research.
    2. You do not use the word “research” in any of your materials. Simply state this is a class project for your Research Methods course.
Read Tague’s (2004b) steps for pilot-testing a survey, highlighted in yellow in the brief article, “How to Administer a Survey.”
Collect pilot data from three participants on each of your two surveys.
1. Your three pilot participants for each of your two surveys can be the same three people.
2. Or your pilot participants can be two different sets of three people or a mixture of same and different.
3. But you must collect pilot data from three pilot participants on each of your two surveys.
Go to the Unit 11: Assignment #4 Discussion Board and make a new post in which you
1. provide a link to each of your two surveys (using the title of each of your surveys for the words that are linked);
2. make sure the links you provide for each of your surveys do not require someone else to log in to that app or program (remember you can test your links before you click “Submit Response” by right-clicking and selecting “Open in New Tab” or something similar); and
3. write at least 200 words about what you learned from collecting pilot data about each of your two surveys, including what changes you need to make to each survey before you administer the survey to non-pilot participants
4. IMPORTANT: After completing Unit 11: Assignment #4, begin collecting your actual (not pilot) data!

Unit 11: Assignment #5 (due before 11:59 pm Central on Monday November 1):

Read Boynton and Greenhalgh’s (2004) article, “Hands-on Guide to Questionnaire Research: Selecting, Designing, and Developing your Questionnaire.”
Pay specific attention to the sections titled “Could you use an existing instrument?” and “Is the questionnaire valid and reliable?”
Find (using Google Scholar or any other search mechanism) one existing instrument (survey) that you might want to use someday in another psychological science research project.
1. The survey must be available via Open Access (not paywalled and not licensed).
2. The survey must have been used in previously published psychological science research.
3. The survey must have some data previously reported about its validity and reliability.
4. The survey does not need to be relevant to the two surveys you constructed in this Unit (but if the existing survey is relevant, that’s all the better).
Go to the Unit 11: Assignment #5 Discussion Board and make a new post in which you
1. identify and explain, in at least 200 words, the existing survey, using as a rough model the identification and explanation of the Subjective Happiness Scale (note, however, this explanation is unfortunately fewer than 200 words);
  - your explanation should also include information about the survey’s validity and reliability; and
  - your explanation should have appropriate in-text citations (at the ends of sentences in parentheses, although such in-text citations are unfortunately not illustrated in the model).
2. list at least three “key references” (in full APA-style citation) for the instrument (the “key references” can include scientific articles that have used the instrument); and
3. attach (in PDF) a copy of the instrument (for example, this PDF of the Subjective Happiness Scale).

Unit 11: Assignment #6 (due before 11:59 pm Central on Tuesday November 2):

Meet online with your Chat Group for a one-hour text-based Chat.
Prior to your Chat Group meeting, all members of your Chat Group MUST have completed Assignments #1, #2, #3, and #4 in this Unit.
During the one-hour text-based Chat, each Chat Group member will develop and write two Research Proposals, one for each of their two surveys, using this Research Proposal Outline.
- NOTE: The Research Proposal Outline is purposely not a fillable PDF.
- It is not a fillable PDF because you will need to copy the contents from the Research Proposal Outline and paste that content into a Word (or other) doc, which is where you will fill in the content you supply.
For your first survey and with the help of your Chat Group:
1. Fill in your Research Proposal’s Title with the title of your survey. Bold and center your Research Proposal’s Title.
2. Write the introduction to your Research Proposal by doing the following:
  - Write a Hook Paragraph that is an engaging introduction to your survey’s topic. (Note that this is a Hook Paragraph rather than simply a Hook Sentence.)
  - Add in the Literature Review that you wrote in Unit 11: Assignment #1.
  - Complete the Overview Paragraph by filling the blanks that are in the Research Proposal Outline.
  - Your Hook Paragraph, your Literature Review, and your Overview Paragraph form the Introduction to your Research Proposal. However, none of these paragraphs will have sub-headings. Neither does your Introduction have a sub-heading.
3. Write the Method section of your Research Proposal by doing the following:
  - Complete your Participants paragraph by filling the blanks that are in the Research Proposal Outline.
  - Complete your Materials paragraph by filling the blanks that are in the Research Proposal Outline.
  - Complete your Procedure paragraph by filling the blanks that are in the Research Proposal Outline.
  - Copy (without having to fill in any blanks) the Analyses paragraph of the Research Proposal Outline.
  - Make sure you understand why you will not be using inferential statistics to analyze your survey data (because inferential statistics are inappropriate with the size samples you are collecting and because inferential statistics are unwarranted without a priori hypotheses); if you don’t understand these limitations, discuss these issues with your Chat Group.
  - Your Participants paragraph, your Materials paragraph, your Procedures paragraph, and your Analyses paragraph form the Method section of your Research Proposal. All of these paragraphs DO have sub-headings, and the Method section has a centered and bolded sub-heading.
4. Complete your References section by filling in the APA-style references you wrote in Unit 11: Assignment #1. (Your References section also has a sub-heading that is centered and bolded.)
5. Save your first survey’s Research Proposal as a PDF that is named LastName_ResearchProposal_One.pdf and that
  - is double-spaced throughout;
  - has one-inch margins on the top, bottom, left and right;
  - has page numbers on the bottom; and
  - has a title page that looks like this.
6. Repeat steps 1 through 5 above for your second survey, except this time save your second Research Proposal as a PDF that is named LastName_ResearchProposal_Two.pdf.
At the end of your one-hour Chat:
1. Nominate one member of your Chat Group (who participated in the Chat) to make a post on the Unit 11: Assignment #6 Discussion Board that summarizes your Group Chat in at least 200 words.
  - Attach the Research Proposal for your first survey to this Discussion Board Post.
  - Because the Discussion Board will allow only one file to be attached to each post, make a reply post to your own post. Use your reply post to attach the Research Proposal for your second survey.
2. Nominate another member of your Chat Group (who also participated in the Chat) to save the Chat transcript as a PDF, as described in the Course How To (under the topic, “How to Save and Attach a Group Text Chat Transcript”), and attach the Chat transcript to a Unit 11: Assignment #6 Discussion Board post.
  - Because the Discussion Board allows only one file to be attached to each post, make a reply post to your own post. Use your reply post to attach the Research Proposal for your first survey.
  - Make a second reply post and use that second reply post to attach your Research Proposal for your second survey.
3. Nominate another member of your Chat Group (who also participated in the Chat) to make another post on the Unit 11: Assignment #6 Discussion Board that states the name of your Chat Group, the names of the Chat Group members who participated the Chat, the date of your Chat, and the start and stop time of your Chat
  - Attach the Research Proposal for your first survey to this Discussion Board Post.
  - Because the Discussion Board will allow only one file to be attached to each post, make a reply post to your own post. Use your reply post to attach the Research Proposal for your second survey.
4. If only two students participated in the Chat, then one of those two students needs to do two of the above three tasks.
5. Before ending the Group Chat, arrange the date and time for the Group Chat you will need to hold during the next Unit (Unit 12: Assignment #6).
All members of the Chat Group:
1. must proceed to collecting data using their two surveys. Remember:
  - at least 10 participants per survey, and the 10 participants can be the same participants for both surveys or some combination of same and different;
  - however, none of the 10 participants on either survey can be the same as your pilot-participants for that survey; AND
  - none of the 10 participants on either survey can be members of your current Chat Group, but they can be members of your previous Chat Group if those students are not also in members of your current Chat Group.
2. must record a typical Unit entry in their own Course Journal for Unit 11.

Congratulations, you have finished Unit 11! Onward to Unit 12!

Open-Access Active-Learning Research Methods Course by Morton Ann Gernsbacher, PhD is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. The materials have been modified to add various ADA-compliant accessibility features, in some cases including alternative text-only versions.
Permissions beyond the scope of this license may be available at http://www.gernsbacherlab.org.