Research Methods and Design – Measurement

Part 1: Measurement

1. H1: Students in a gamified classroom will show a statistically significant increase on end of instruction exam scores when compared to students in a traditional classroom.

Variables:

Class model (gamification) – Nominal (control = 0 and experimental = 1)

end of instruction test scores – Interval

H2: Teachers using a gamified class model will be able to cover more units of curriculum during the school year than teachers who use a more traditional classroom model.

Variables:

Class model (gamification) – Nominal (control = 0 and experimental = 1)

Units of curriculum – Ordinal

2. The instrument that will be used to measure the end of instruction test scores will be the Algebra 2 exam that is administered by the Oklahoma State Department of Education (OSDE).   The tests are kept in a secure location, but blueprint information is available at http://www.ok.gov/sde/ and includes items such as:

• Number Sense and Algebraic Operations
• Relations and Functions
• Data Analysis, Probability, & Statistics

3. It was very difficult to find the specific measures used to determine reliability for the Oklahoma End of Instruction (EOI) Algebra 2 exam. The assessment is created by Measured Progress and given to the OSDE. The OSDE administers this test to all students in the state of Oklahoma in the spring of each year. Validity is addressed on the Measured Progress website by Michael Nering who is the senior psychometrician for Measured Progress. He uses math for an example by stating “a math test is valid, if and only if, the test questions assess a student’s math ability” (Nering, 2015). He also discusses two specific forms of validity: predictive and concurrent. Predictive validity can be used if a future behavior related to the assessment can be predicted. Concurrent validity is used when a newer assessment is highly correlated with a known assessment that has been considered valid (Nering, 2015).

4. Reliability for the Algebra 2 EOI was also difficult to find. Nering addressed reliability in the same article. Will the results be “repeatable” (Nering, 2015) is central to reliability. He also explains that the reliability can be determined by a coefficient (Nering, 2015), which would indicate the extent to which the scores were clustered together. It is also important to understand that reliability cannot be exactly determined but must be estimated (Nering, 2015).

5. Something that could influence results on the EOI is the ability and/or the experience of the teacher. The culture inside of a classroom could have an impact, one teacher may be very distant to her students and one may not be. A student could have started the year in the wrong class and the situation was not corrected appropriately. A student could be on an individualized educational plan and is being required to take a test without modification. A student may be experiencing a significant life event during testing.

References

Nering, M. (2015). Introduction to reliability. Retrieved from

http://www.measuredprogress.org/statistical-analysis-introduction-to-reliablity

Part 2: Data Collection Methods

1. For the end of instruction test scores, the data that the testing company releases to the schools will be used. The data from the students who participated in the study will be used. Test scores from the three control courses will be compared to test scores from the three experimental courses. To determine the curriculum that is covered, the teachers will be asked what units of study they were able to teach before the end of instruction exam was administered.

2. All students (including those on an individualized educational plan) who are enrolled in an Algebra 2 course must take the Algebra 2 end of instruction exam (EOI) in April. The EOI is administered by the state of Oklahoma and every high school in the state must give the exam. The test will be locally administered by on site staff who will ensure that testing is done according to protocols established by the state of Oklahoma. Students will have two hours to take the Algebra 2 EOI on a computer. If the two hour block ends before the student has completed the test, the student will be given additional time. The test is done completely online and all information will go directly to the testing company to be collected and scored. Results will be released to the schools as soon as possible by the testing company and disseminated to the teachers.

3. Potential issues include some student not taking the EOI for various reasons due to absence from school on testing days. The courses will be taught by different teachers, it is possible that some teachers will teach different topics differently and this could impact student learning. A student could have a significant life event that impacts their testing or could have impacted their attendance during the year and this would have an effect on overall performance. The testing website could crash and cause students to re-take portions of the test or the entire test.

4. t-test between the test scores from the control classes with the test scores from the experimental classes