evaluating evidence

ISS 305:003 – Evaluating Evidence Assignment #3

Due Dates and Grades • Assignment #3 is due at 4pm on 12/1/22 (a Thursday) to the D2L paper folder. • This assignment is worth 50 points (10% of your final grade). • Papers turned in after 4pm on 12/1/22 will receive a 10 point deduction for each day they are late (e.g., a paper turned

in 2 days late will be marked down 20 points). I include Saturday and Sunday when I count the number of days the paper is late.

Overview The purposes/goals/objectives of this paper is to describe, interpret, report on, and assess the replication crisis. Specifically, you will decide what field and research design is more susceptible to the replication crisis and the problems associated with it, and perform an interview with a friend or family member where you apply the major concepts and theories relevant to ISS 305 that you have learned so far. This assignment covers ISS Undergraduate Learning Goals #1, #2, and #5. Guidelines for the paper To earn the highest possible score, be sure to do the following:

• Address #s 1-5. • Write the paper in a numbered format, corresponding to each of the numbered questions for you to answer. • Throughout the paper, use details to support your main argument and repeatedly link to material covered

throughout the course. There are multiple places links can, and should, be made. • For full points, be thoughtful, organized, have good sentence and paragraph structure, and be free of punctuation

and spelling errors. • You do not need a separate introduction or conclusion section of the paper. Just provide responses to the

numbered prompts. • Be sure to work independently and present only your own work. Plagiarism (using someone else’s words or

ideas without giving them credit) will result in 0 points for the paper and course. o Besides when directly quoting your friend or family member, do not take direct quotes and

insert them in as answers. You need to develop original work and content for this course. It is expected that you gather information and display it in a manner that isn't just copying and pasting it into your paper. Cite the work (see FAQ #5), but do not copy-and-paste the original content.

o Turnitin will be used. See the syllabus for more details. Papers should be:

• 2-4 pages in length (maximum 5 pages) • Double-spaced, 12 point Times New Roman font with 1-inch margins on all sides • Do not include identifying information in your paper (e.g., name or student number). • Upload your paper to the D2L paper folder.

Papers do not need to be:

• In APA format, MLA, Chicago style, or whatever style you have been taught. If you want to include citations and a reference section, great! In fact, you are encouraged to do so! However, the formatting choice of the citations and the reference section are up to you.

A FAQ is provided at the end of this document (after the rubric).

1. Do you think the problems discussed in the “Real vs. Illusory Effects” lecture series (especially in Part 7) about the replication crisis, file drawer problem, multiple comparisons, and p-hacking are particularly likely in a certain area/field of research? Why? Or, why not? What about a certain research design (survey, experiment, etc.)? Why? Or, why not? Apply course material to help support your answer.

2. Have a friend or family member read this article - https://www.vox.com/future-perfect/21504366/science- replication-crisis-peer-review-statistics. Then interview them. You can ask any questions you like, but get a sense of if they sound like a Candide, skeptic, or Skeptic after reading the article. Then explain why you believe they fit into one of the three categories. Apply course material to help support your answer.

3. Give your friend or family member one piece of advice from the course that you think is the best advice for approaching new evidence. Specifically, what is something from the course that you would tell them to use when evaluating evidence in the future and why? Apply course material to help support your answer.

4. What did your friend or family member think about the advice you gave them in #3? Did the advice change their mind for how they will approach new evidence in the future? If yes, how so and why? If no, why not?

5. Imagine that you and a group of people are presented with some new evidence trying to convince you of something. (Could be anything – taking a new medication, buying a certain kind of car, which dog breed is the best, etc.) Based on your interview and discussion with your friend or family member, how will you approach talking about evaluating new evidence with this group of people? Specifically, what is your strategy for how to discuss “becoming a better research consumer” with them? Apply course material to help support your answer.

Paper Rubric Criteria Failing Below Average Average Very Good /

Strong Exceptional / Superior

#1 10 pts possible

0 – 3 points – No Answer provided, or answer provided has nothing to do with the questions asked (0 points); area/field and design mentioned but no supporting details are provided (1 points); area/field and design mentioned are presented, but very minimal supporting details are provided (2 – 3 points)

4 – 5 points – Area/field and design are mentioned, but the supporting details for why (or, why not) are minimal.

6 – 7 points – Area/field and design are presented and supporting details are provided, but lack in development. Specifically, only surface level details for why (or, why not) are provided.

8 – 9 points – Area/field and design are presented and supporting details are provided, appropriate, and are past surface level details; but still lacking in full development for why (or, why not).

10 points – Area/field and design are presented and supporting details for why (or, why not) are fully developed and appropriate.

#2 5 pts possible

0 – 1 points – No answer provided, or answer provided has nothing to do with the question.

2 point – Some description of why the friend or family member fit into one of the three categories is included, but no clear supporting details.

3 points – Description of why the friend or family member fit into one of the three categories is included, but only surface level details are provided.

4 points – Description of why the friend or family member fit into one of the three categories is included and supporting details are provided, appropriate, and are past surface level details; but still lacking in full development.

5 points – Clear description of why the friend or family member fit into one of the three categories is included. The supporting details are fully developed and appropriate.

#3 5 pts possible

0 – 1 points – No answer provided, or answer provided has nothing to do with the question.

2 point – Some description of the “best advice” is included, but no clear supporting details.

3 points – Description of the “best advice” is included, but only surface level details are provided.

4 points – Description of the “best advice” is included and supporting details are provided, appropriate, and are past surface level details; but still lacking in full development.

5 points – Clear description of the “best advice” is included. The supporting details are fully developed and appropriate.

#4 5 pts possible

0 – 1 points – No answer provided, or answer provided has nothing to do with the question.

2 points – Some description of what the friend or family member thought about the advice is included, but no clear supporting details.

3 points – Description of what the friend or family member thought about the advice is included, but only surface level details are provided.

4 points – Description of what the friend or family member thought about the advice is included and supporting details are provided, appropriate, and are past surface level details; but still lacking in full development.

5 points – Clear description of what the friend or family member thought about the advice is included. The supporting details are fully developed and appropriate.

#5 5 pts possible

0 – 1 points – No answer provided, or answer provided has nothing to do with the question.

2 points – Some description of how to approach “becoming a better research consumer” with other people is included, but no clear supporting details.

3 points – Description of how to approach “becoming a better research consumer” with other people is included, but only surface level details are provided.

4 points – Description of how to approach “becoming a better research consumer” with other people is included and supporting details are provided, appropriate, and are past surface level details; but still lacking in full development.

5 points – Clear description of how to approach “becoming a better research consumer” with other people is included. The supporting details are fully developed and appropriate.

Incorporation of information from lecture and the readings 15 pts possible

0 – 3 points – No attempt to incorporate information from the course material (0 points). One to two minimal attempts are made (e.g., just providing a concept’s name) to provide linking material (1 point). Three to four minimal attempts are made (2 – 3 points).

4 – 9 points – Attempts are made to incorporate linking materials, but are: 1) Not used correctly and lacking in detail and depth (4 – 5 points). 2) Mostly used correctly, but lacking in detail and depth (6 – 7 points). 3) Used correctly, but lacking in detail and depth (8 – 9 points).

10 – 11 points – Links are made with the course material and are used correctly, but supporting details did not sufficiently make the appropriate linked point.

12 – 13 points – Links are made with the course material and are used correctly, but some supporting details are not always relevant and do not always hit the mark of fully developed incorporation.

14 – 15 points – Links are clearly made with the course material, used correctly, relevant, and: Supporting details are provided (14 points). Supporting details are fully developed and appropriate (15 points).

Formatting / organization, grammar / spelling, overall quality 5 pts possible

0 – 1 points – Formatting guidelines not followed; and many problems with sentence structure, grammatical errors, or spelling errors.

2 points – Either: Formatting guidelines not followed. Or, many problems with sentence structure, grammatical errors, or spelling errors.

3 points – Formatting guidelines somewhat followed; and there are a few problems with sentence structure, grammatical errors, or spelling errors.

4 points – Formatting guidelines followed, but there are a few problems with sentence structure, grammatical errors, or spelling errors.

5 points – Formatting guidelines followed. There may be one or two mechanical errors that are not serious.

Frequently Asked Questions (FAQ)

1) What does “numbered format” mean? a. Numbered format means to put in the numbers and respond to those prompts. For example, this FAQ is in

a numbered format.

2) Does the paper actually have to be 2-4 pages in length? a. Not necessarily. Not including a page length freaks students out. I’m making a guess on the page length

based on past assignments in this course.

3) The paper is worth 50 points. How are the points broken down (i.e., what should I focus on)? a. See the rubric that is provided and pay special attention to the parts that are worth the most. Incorporation

of course information is worth 15 points, or 30% of the assignment!

4) Can we use quotes? a. Besides when directly quoting your friend or family member, do not take direct quotes and

insert them in as answers. In general, you should shy away from direct quotes in your writing. You should be gathering information and organizing it in a way that can be easily digested by a reader. Therefore, use your own words with citations (see FAQ #5). People tend to overuse quotes and it makes documents hard to navigate, and get a good sense if you actually understand the course material. Work on your writing by summarizing the work into your own words.

5) How do you want us to cite the readings or the lectures? a. This course is not designed to teach you how to use APA formatting, MLA, Chicago style, etc. Therefore,

most of the formatting is up to you, but please note the “Papers should be:” section found on page 1 for more information on what is required for formatting. You can easily say things like, “As pointed out in lecture”, “As referenced in X reading”, “As Dr. Weaver so amazingly explained”, etc.

6) How many links / applications to the material is enough?

a. This is tough to answer. You could link to five things and go really in-depth and be good. Or, you could just say five vocabulary type words from the course and do nothing else, which would not be good. You need to show application of the material into the assignment and that you understand the material you are using. For example – Was enough information provided to make the linked point? Was it correctly used?

7) What kind of links / applications to the material are you looking for? a. A good place to start is to think about the major concepts or theories we have covered. Like – the

replication crises and the problems associated with it, fallacies, heuristics, etc. Then you could incorporate examples or findings that were covered to drive home the point you are making.

b. Note that some links are not as impactful as others. For example, pointing out that heuristics are “rules of thumb” is not an impactful link, but explaining how or why or when that might influence decisions, perspectives, etc. is impactful if correctly used.