HOTs multiple choice and essay questions: A validated ınstrument to measure higher-order thinking skills of prospective teachers

Ika Maryani; Zuhdan Kun  Prasetyo; Insih Wilujeng; Siwi Purwanti; Meita   Fitrianawati

doi:10.36681/

Authors

Ika Maryani Universitas Ahmad Dahlan
Zuhdan Kun Prasetyo
Insih Wilujeng Universitas Negeri Yogyakarta
Siwi Purwanti Universitas Ahmad Dahlan
Meita Fitrianawati

DOI:

https://doi.org/10.36681/

Keywords:

Higher-order thinking skill, multiple choice, essay questions, instrument

Abstract

Higher-order thinking skills (HOTs) are very crucial thinking skills needed by teachers to train students to develop 21st-century learning. This study aimed to develop Multiple Choice and Essay Questions to measure the HOTs of the prospective teachers of the elementary school education department. This study used a 4-D model by Thiagarajan which involved experts at natural science, evaluation studies, and primary school pedagogy in the content validation. We also involved 156 prospective teachers as the test subjects. The assessment of instrument quality by experts showed that the question quality was very good. This research succeeded in developing 10 multiple choice questions and 5 essays. The validity test by Rasch Model showed that there were 7 multiple choice questions classified as fit, and 3 questions were classified as a misfit, while the 2 essay questions are invalid and the other (3 questions) as valid. The reliability test with KR-20 on multiple-choice questions and Cronbach's alpha for the essay questions resulted reliable questions. The discrimination index showed discarded, sufficient, good, and very good. The item difficulty index showed that 3 questions are moderate (num 7, 1, 5) and 7 questions are difficult (num 4, 10, 6, 3, 2, 8, 9). The distractor efficiency showed that 59.2% of distractors worked, and 40.8% did not work. This instrument can be used to analyze prospective teachers’ HOTs. This data can be used as the reference for developing competency improvement programs for prospective teachers, for example through the HOTs-oriented learning models.

Downloads

Download data is not yet available.

References

Abdulghani, H., Ahmad, F., Aldrees, A., Khalil, M., & Ponnamperuma, G. (2014). The relationship between non-functioning distractors and item difficulty of multiple choice questions: A descriptive analysis. Journal of Health Specialties, 2(4), 148. https://doi.org/10.4103/1658600x.142784

Ahmad, S., Prahmana, R. C. I., Kenedi, A. K., Helsa, Y., Arianil, Y., & Zainil, M. (2018). The instruments of higher order thinking skills. Journal of Physics: Conference Series, 943(1). https://doi.org/10.1088/1742-6596/943/1/012053

Ahmad, Syafri, Kenedi, A. K., & Masniladevi, M. (2018). Instrumen Hots Matematika Bagi Mahasiswa Pgsd. JURNAL PAJAR (Pendidikan Dan Pengajaran), 2(6), 905. https://doi.org/10.33578/pjr.v2i6.6530

Arifin, Z. (2017). Mengembangkan instrumen pengukur critical thinking skills siswa pada pembelajaran matematika abad 21. Jurnal THEOREMS (The Original Research of Mathematics), 1(2). Azwar, S. (2015). Tes Prestasi: fungsi dan pengembangan pengukuran prestasi belajar. Pustaka Pelajar.

Bajpai, R., & Bajpai, S. (2014). Goodness of Measurement: Reliability and Validity. International Journal of Medical Science and Public Health, 3(2), 112. https://doi.org/10.5455/ijmsph.2013.191120133

Beck, C. T., Keddy, B. A., & Cohen, M. Z. (1994). Reliability and Validity Issues in Phenomenological Research. Western Journal of Nursing Research, 16(3), 254–267. https://doi.org/10.1177/019394599401600303

Bhakti, C. P., & Maryani, I. (2017). Peran LPTK dalam Pengembangan Kompetensi Pedagogik Calon Guru. Jurnal Pendidikan (Teori Dan Praktik), 1(2), 98. https://doi.org/10.26740/jp.v1n2.p98-106

Burud, I., Nagandla, K., & Agarwal, P. (2019). Impact of distractors in item analysis of multiple choice questions. International Journal of Research in Medical Sciences, 7(4), 1136-1139. DOI:10.18203/2320-60. https://doi.org/10.18203/2320-6012.ijrms20191313

Chang, J. O., Levy, S. S., Seay, S. W., & Goble, D. J. (2014). An Alternative to the Balance Error Scoring System. Clinical Journal of Sport Medicine, 24(3), 256–262. https://doi.org/10.1097/JSM.0000000000000016

Chang, S. C., & Hwang, G. J. (2018). Impacts of an augmented reality-based flipped learning guiding approach on students’ scientific project performance and perceptions. Computers and Education, 125, 226–239. https://doi.org/10.1016/j.compedu.2018.06.007

Chauhan, P., Chauhan, G. R., Chauhan, B. R., Vaza, J. V., & Rathod, S. P. (2015). Relationship Between Difficulty Index and Distracter Effectiveness in Single Best-Answer Stem Type Multiple Choice Questions. International Journal of Anatomy and Research, 3(4), 1607–1610. https://doi.org/10.16965/ijar.2015.299

Chebii, R., Wachanga, S., & Kiboss, J. (2012). Effects of Science Process Skills Mastery Learning Approach on Students’ Acquisition of Selected Chemistry Practical Skills in School. Creative Education, 03(08), 1291–1296. https://doi.org/10.4236/ce.2012.38188

Creswell, J. W. (2012). Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative (4th ed.). Pearson Education Inc.

David, D., Kartowagiran, B., & Harjo, S. P. (2016). Evaluasi Dan Strategi Pengembangan SMA Indonesisch Nerderlandsche School (INS) Kayutanam. Jurnal Penelitian Dan Evaluasi Pendidikan, 20(1), 27. https://doi.org/10.21831/pep.v20i1.7518

Eleje, L. I., & Esomonu, N. P. M. (2018). Test of Achievement in Quantitative Economics for Secondary Schools: Construction and Validation Using Item Response Theory. Asian Journal of Education and Training, 4(1), 18–28. https://doi.org/10.20448/journal.522.2018.41.18.28

Fauzi, A., & Sa’diyah, W. (2019). Students’ metacognitive skills from the viewpoint of answering biological questions: Is it already good? Jurnal Pendidikan IPA Indonesia, 8(3), 317–327. https://doi.org/10.15294/jpii.v8i3.19457

Ghosh, S., Bowles, M., Ranmuthugala, D., & Brooks, B. (2016). Authentic assessment in seafarer education: using literature review to investigate its validity and reliability through rubrics. WMU Journal of Maritime Affairs, 15(2), 317–336. https://doi.org/10.1007/s13437-015-0094-0

Golafshani, N. (2003). Understanding and Validity in Qualitative Research. The Qualitative Report, 8(4), 597–607. https://doi.org/10.17763/haer.62.3.8323320856251826

Hartman, H., & Johnson, P. (2018). The effectiveness of multimedia for teaching drug mechanisms of action to undergraduate health students. Computers and Education, 125, 202–211. https://doi.org/10.1016/j.compedu.2018.06.014

Haviz, M., Maris, I. M., Adripen, Lufri, David, & Fudholi, A. (2020). Assessing pre-service teachers’ perception on 21st century skills in Indonesia. Journal of Turkish Science Education, 17(3), 351–363. https://doi.org/10.36681/tused.2020.32

He, W., Holton, A., Farkas, G., & Warschauer, M. (2016). The effects of flipped instruction on out-ofclass study time, exam performance, and student perceptions. Learning and Instruction, 45, 61–71. https://doi.org/10.1016/j.learninstruc.2016.07.001

Heale, R., & Twycross, A. (2015). Validity and reliability in quantitative studies. In Evidence-Based Nursing (Vol. 18, Issue 3, pp. 66–67). BMJ Publishing Group. https://doi.org/10.1136/eb-2015102129

Herrmann-Abell, F, C., & DeBoer, G. E. (2011). Using distractor-driven standards-based multiplechoice assessments and Rasch modeling to investigate hierarchies of chemistry misconceptions and detect structural problems with individual items. Chemistry Education Research and Practice, 12(2), 184–192.

Hingorjo, M. R., & Jaleel, F. (2012). Analysis of One-Best MCQs : the Difficulty Index , Discrimination Index and Distractor Efficiency. JPMA-Journal of the Pakistan Medical Association, 62(2), 142–147.

Istiyono, E., Widihastuti, W., Supahar, S., & Hamdi, S. (2020). Measuring Creative Thinking Skills of Senior High School Male and Female Students in Physics (CTSP) Using the IRT-based PhysTCreTS. Journal of Turkish Science Education, 17(4), 578–590. https://doi.org/10.36681/tused.2020.46

Jacobs, L. C., & Chase, C. I. (1992). Developing and Using Tests Effectively. A Guide for Faculty. JosseyBass Inc.

Karelia, B. N., Professor, A., Pillai, A., & Vegada, B. N. (2013). The levels of difficulty and discrimination indices and relationship between them in four-response type multiple choice questions of pharmacology summative tests of Year II M.B.B.S students. IeJSME, 7(2), 41–46.

Khoshaim, H. B., & Rashid, S. (2016). Assessment of the assessment tool: analysis of items in a nonMCQ mathematics exam. International Journal of Instruction, 9(1), 119–132. https://doi.org/10.12973/iji.2016.9110a

Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. Internet and Higher Education, 27, 74–89. https://doi.org/10.1016/j.iheduc.2015.06.002

Lissitz, R. W., & Samuelsen, K. (2007). Further Clarification Regarding Validity and Education. Educational Researcher, 36(8), 482–484. https://doi.org/10.3102/0013189x07311612

Mayordomo, R. M., & Onrubia, J. (2015). Work coordination and collaborative knowledge construction in a small group collaborative virtual task. Internet and Higher Education, 25, 96–104. https://doi.org/10.1016/j.iheduc.2015.02.003

Nur Aini, D. F., & Sulistyani, N. (2019). Pengembangan Instrumen Penilaian E-Quiz (Electronic Quiz) Matematika Berbasis HOTS (Higher of Order Thinking Skills) untuk Kelas V Sekolah Dasar. Edumaspul: Jurnal Pendidikan, 3(2), 1–10. https://doi.org/10.33487/edumaspul.v3i2.137

Nursalam, N., & Rasyid, M. R. (2016). Studi Kemampuan Mahasiswa Mendesain Perencanaan Pembelajaran Matematika di Sekolah Menengah Pertama Berbasis Pendekatan Saintifik. MaPan: Jurnal Matematika Dan Pembelajaran, 4(1), 94–116. https://doi.org/10.24252/mapan.2016v4n1a8

O’Flaherty, J., & Phillips, C. (2015). The use of flipped classrooms in higher education: A scoping review. Internet and Higher Education, 25, 85–95. https://doi.org/10.1016/j.iheduc.2015.02.002

Panjaitan, R. L., Irawati, R., Sujana, A., Hanifah, N., & Djuanda, D. (2018). Item validity vs. item discrimination index: A redundancy? Journal of Physics: Conference Series, 983(1). https://doi.org/10.1088/1742-6596/983/1/012101

Postmes, T., Haslam, S. A., & Jans, L. (2013). A single-item measure of social identification: Reliability, validity, and utility. British Journal of Social Psychology, 52(4), 597–617. https://doi.org/10.1111/bjso.12006

Pradana, V. (2020). Penggunaan pendekatan saintifik untuk meningkatkan kemampuan menyelesaikan soal hots pada materi karakteristik geografi Indonesia. Didaktika Dwija Indria, 8(04). https://doi.org/10.20961/ddi.v8i04.39916

Quaigrain, K., & Arhin, A. K. (2017). Using reliability and item analysis to evaluate a teacherdeveloped test in educational measurement and evaluation. Cogent Education, 4(1). https://doi.org/10.1080/2331186X.2017.1301013

Redhana, I. W. (2019). Mengembangkan Keterampilan Abad Ke-21 Dalam Pembelajaran Kimia. Jurnal Inovasi Pendidikan Kimia, 13(1).

Retnawati, H. (2016). Proving content validity of self-regulated learning scale (The comparison of Aiken index and expanded Gregory index). Research and Evaluation in Education, 2(2), 155. https://doi.org/10.21831/reid.v2i2.11029

Sener, N., & Tas, E. (2017). Developing Achievement Test: A Research for Assessment of 5th Grade Biology Subject. Journal of Education and Learning, 6(2). https://doi.org/10.5539/jel.v6n2p254

Singh, A. S. (2017). Common procedures for development, validity and reliability of a questionnaire. International Journal of Economics, Commerce and Management, 5(5), 790–801. https://www.researchgate.net

Sugiyanta, S., & Soenarto, S. (2016). An evaluation model of educational quality assurance at junior high schools. Research and Evaluation in Education, 2(2), 194. https://doi.org/10.21831/reid.v2i2.11118

Sumintono, B., & Widhiarso, W. (2015). Aplikasi Pemodelan Rasch pada Assessment Pendidikan. Trim Komunikata.

Susantini, E., Faizah, U., Prastiwi, M. S., & Suryanti. (2016). Developing educational video to improve the use of scientific approach in cooperative learning. Journal of Baltic Science Education, 15(6), 725–737.

Thaneerananon, T., Triampo, W., & Nokkaew, A. (2016). Development of a test to evaluate students’ analytical thinking based on fact versus opinion differentiation. International Journal of Instruction, 9(2), 123–138. https://doi.org/10.12973/iji.2016.929a

Thiagarajan, S., Semmel, M. ., & Semmel, D. . (1974). Instructional development for training teachers of exceptional children: A sourcebook. Leadership Training Institute/Special Education University of Minnesota. https://eric.ed.gov/?id=ED090725

Thorndike, R. M., Cunningham, G. K., Thorndike, R. L., & Hagen, E. P. (1991). Measurement and evaluation in psychology and education. Macmillan Publishing Co, Inc.

Tooth, J. A., Nielsen, S., & Armstrong, H. (2013). Coaching effectiveness survey instruments: Taking stock of measuring the immeasurable. Coaching, 6(2), 137–151. https://doi.org/10.1080/17521882.2013.802365

Vishnumolakala, V. R., Southam, D. C., Treagust, D. F., & Mocerino, M. (2016). Latent constructs of the students’ assessment of their learning gains instrument following instruction in stereochemistry. Chemistry Education Research and Practice, 17(2), 309–319. https://doi.org/10.1039/c5rp00214a

Widodo, E., & Sudarsono, F. X. (2016). Developing an observation instrument for assessing the effectiveness of English teaching at vocational secondary schools. Research and Evaluation in Education, 2(2), 135. https://doi.org/10.21831/reid.v2i2.8648

Young, D. L., Estocado, N., Landers, M. R., & Black, J. (2011). A pilot study providing evidence for the validity of a new tool to improve assignment of national pressure ulcer advisory panel stage to pressure ulcers. Advances in Skin & Wound Care, 24(4), 168–175. https://doi.org/10.1097/01.asw.0000396304.90710.ea

Yuniar, M., Rakhmat, C. R., & Saepulrohman, A. (2019). Penggunaan Media Kartu Pecahan untuk Meningkatkan Pemahaman Siswa tentang Membandingkan Pecahan. Penggunaan Media Kartu Pecahan Untuk Meningkatkan Pemahaman Siswa Tentang Membandingkan Pecahan, 6(1), 90–100.

Zorlu, Y., & Zorlu, F. (2021). Investigation of The Relationship Between Preservice Science Teachers’ 21st Century Skills and Science Learning Self-Efficacy Beliefs with Structural Equation Model. Journal of Turkish Science Education, 18(1), 1–16. https://doi.org/10.36681/tused.2021.49