Questions to Ponder as you evaluate these criterion
You are NOT expected to answer the following questions as part of your review, and an individual paper does not need to satisfy all of these questions. The questions are simply provided to give you points to ponder as you read the paper and decide on your rating for the paper you have been asked to review. The questions may help when you provide feedback to the editors and authors regarding the reasoning behind your numerical ranking for each item.
Originality
Interest/Value to the PER Community and Content Novelty
- Is the paper sufficiently novel and interesting to warrant publication at this point in time?
- Does it add to the canon of knowledge?
- Is the research question an important one?
- In order to determine its originality and appropriateness for the PERC Proceedings, it might be helpful to think of the research in terms of what percentile it is in. Is it in the top 25% of papers in this field?
- Are the conclusions novel or are they too similar to work already published?
- Who will be interested in reading the paper, and why?
- What are the main claims of the paper and how significant are they?
- How does the paper stand out from others in its field?
- Are the claims novel? If not, which published papers compromise novelty?
Research Design
Strength of Research
The research problem
- What specific question (or questions) does this paper address?
- Does the paper make a strong argument for its place in the body of existing knowledge?
- Does the paper present the necessary theoretical and experimental background, including citations of current literature? Do you have suggestions for additional references?
Methodology
Type of study/research design
- What type of study is it (qualitative/quantitative, experimental/quasi-experimental, ethnographic, case study, theoretical, etc.)?
- Is there a justification for using this type of research design?
- Is the research design appropriate/suitable for answering the question(s) posed?
- Was the methodology effectively implemented?
- Are alternative approaches mentioned, along with reasons they do not apply?
- What was the role of the researcher (passive observer, instructor, unfamiliar interviewer, etc.)?
Subjects
- Is the context, population, etc. adequately described?
- How were subjects selected?
- Could the way in which subjects were selected have impacted the study?
Data collection
- Are the study's procedures, protocols, and instruments described and justified? Are data, statistical results, populations, instrumentation, interview/observation protocols, and other procedures adequately reported? If the methods are new, are they explained in sufficient detail?
- Was the sampling appropriate? Are there multiple sources/types of data? Are there enough data to answer the research questions?
- Does the author accurately and clearly explain the type of data collected and how the data was collected?
- Is there sufficient detail to facilitate replication of the study? Is there sufficient information present for you to replicate the research?
Results
This is where the author(s) should explain in words what he/she/they discovered in the research. It should be clearly laid out and in a logical sequence. Interpretation of results should not be included in this section. You will need to consider if the appropriate analysis has been conducted. Are the statistics correct? If you are not comfortable with statistics, please advise the editor when you submit your report but continue with your review.
- Is the data analysis done properly and adequately explained? A reader with the original data should be able to replicate the analysis.
- For a quantitative study, was sufficient statistical information presented? Were the correct statistical tests applied? Do the data and statistics presented in the paper follow the appropriate reporting conventions?
- For a qualitative study, was the analysis process thoroughly described? Are there adequate (but not excessive) quotations clearly illustrating the researcher's points? How representative are the data that were selected?
- Are validity and reliability issues addressed? Have threats to validity and estimates of reliability been reported?
- Have triangulation and other attempts to establish consistency in the data been adequately reported and addressed? Do all the different types of data fit together into a consistent set of results? Were there any discrepant findings? If so, how were they explained?
Conclusions
- Are the research questions answered?
- Do the data and methods substantiate the conclusions and interpretations? Does the paper make a convincing analysis of the results? Are the conclusions supported by unambiguous interpretations of the data? Are there other experiments or work that would strengthen the paper further?
- Are the results appropriately discussed in the context of previous literature? Are comparisons made to previous studies? Have the authors indicated how the results relate to expectations and to earlier research? Does the conclusion explain how the research has moved the body of scientific knowledge forward?
- Does the paper support or contradict previous theories? Is there a connection to a theoretical framework (either supporting or refuting)?
- Did the author(s) consider alternate explanations when applicable?
- Could the author(s) synthesize additional, valid conclusions from the data?
- Are limitations of the study's findings or applicability discussed? Are the conclusions appropriate given the limitations?
- Is there a need to include implications for instruction?
Structure
Organization of Ideas
- What is the purpose of the paper? Who is the audience? Does the paper meet its goals?
- Is the paper clearly laid out? Should specific sections be changed, added or dropped? Should specific sections be reorganized?
- Does the paper identify the procedures followed? Are these ordered in a meaningful way?
- Are the headings helpful?
- Have appropriate keywords been indicated? Are there others that apply? (Keyword choice is critical for efficient searches of the literature. Poor keywords can keep an excellent paper from ever being found.)
Grammar and Formatting
Grammar
- Is the paper clearly written? If not, how could it be made more clear or accessible to undergraduates and researchers in other areas?
- Is the paper easy to read?
- Is the English clear and concise?
- Are there grammatical or spelling errors?
- Is the overall quality of writing (vocabulary, grammar, and style) satisfactory?
Formatting
- Does the paper follow PERC Proceedings guidelines for paper submission?
- Are all the key elements (where relevant) present: abstract, introduction, methodology, results, conclusions?
- When considering the whole paper, do the figures and tables inform the reader, are they an important part of the story?
- Are the tables and figures understandable and readable? Should any be reformatted?
- Are data tables and figures complete or is there information missing?
- Are tables and figures used appropriately? Too few or too many can hinder a reader. Should any be added or deleted?
- Do the figures describe the data accurately?
- Are the figures consistent, e.g. bars in charts are the same width, the scales on the axis are logical?
The following should also be considered and issues included in the feedback provided to both the Editor and authors, but NOT used as grounds for rejection.
Title
- Does the title clearly describe the paper? Is it sufficiently descriptive? (Keep in mind that this is all that most people will read when browsing a paper database or a journal's table of contents.) Does the title contain enough information to indicate the uniqueness of the study?
- Are there keywords in the title that will facilitate electronic searches? Should the title be changed? If so, in what way?
Abstract
The abstract is not a "teaser" that presents just enough information to arouse curiosity. All important conclusions should be included.
- Is it detailed enough to communicate the main point of the study and the findings?
- Is it clear and succinct?
- Is it missing critical information?
- Should the abstract be changed? If so, in what way?
- Does it reflect the content of the paper?
Introduction/Background
Normally, the introduction should summarize relevant research to provide context, and explain what other authors' findings, if any, are being challenged or extended. It should describe the experiment, any hypotheses and the general experimental design or method.
- Is sufficient detail presented to provide a rationale for the study? The literature review should lead the reader directly to the questions addressed by the research.
- Is too much well known material included? Is information presented that is already available to researchers in earlier papers?
- Is the justification for the study based on vague generalizations or specific references to previous findings?
- Is the underlying theoretical framework explained?
- Are the researcher's background and possible biases fully described?
- Are the questions addressed by the study clearly presented?
- Does the paper describe what the author hoped to achieve accurately and clearly state the problem being investigated?
References
- Is the bibliography an adequate survey of the literature?
- Are the references accurate?
- Does each reference really make the point mentioned in the paper?
- Does the researcher rely too heavily on one type of literature (e.g., quantitative or qualitative) or their own previous work?
- If the paper builds upon previous research, does it reference that work appropriately? Are there any important works that have been omitted?
- Is it current?
- Are the citations in the approved format?
