“I can’t search on Google for answersâ€: validity evidence of a developed computer-based assessment application
Abstract
While learning management systems have been increasingly used as innovative ways to teach students in the post-COVID-19 pandemic, academic dishonesty is still in crisis. The open-access facility has the potential intervention in academic dishonesty. This study reported validity evidence of a developed computer-based assessment application to combat academic dishonesty: 1) measuring the development of high-order-thinking-skill (HOTS) test items; 2) examining the test item's validity integrated into the application; and 3) reporting the program analysis of the anti-cheating application for English for Specific Purposes (ESP) class assessment. This mixed-method study used focus group discussion and reading tests to collect the data. Pearson product-moment and black box analysis were used to measure test-item validity and application. The quantitative result revealed that test items were developed according to authentic journal texts and HOTS question characteristics and were statistically proven valid and reliable. Simplifying the authentic text empirically enabled EFL students to have an easier ESP reading comprehension complexity level. The black box analysis showed that the auto-lock screen feature prevented suspicious cheating access. This study contributed to extending practical and theoretical research on technology-enhanced language learning (TELL) by providing an innovative anti-cheating technology application.
Keywords
Full Text:
PDFArticle Metrics
Abstract view : 250 times | PDF view : 80 timesReferences
Admiraal, W., Vermeulen, J., & Bulterman-Bos, J. (2020). Teaching with learning analytics: how to connect computer-based assessment data with classroom instruction? Technology, Pedagogy and Education, 29(5), 577–591. https://doi.org/10.1080/1475939X.2020.1825992
Alper, S., & Duygu, A. (2017). The effects of test length and sample size on item parameters in item response theory. Educational Sciences: Theory & Practice, 17(1), 321–325. https://doi.org/10.12738/estp.2017.1.0270
Alshamsi, A., Zahavich, A., & El-Farra, S. (2021). Why graded assessment for undergraduates during the COVID-19 lockdown? an experience introspection. IAFOR Journal of Education, 9(2), 55–75. https://doi.org/10.22492/ije.9.2.04
Arifin, R., Astuti, F., Baqiya, M. A., Winardi, Y., Wicaksono, Y. A., Darminto, & Selamat, A. (2021). Structural change of TiAl alloy under uniaxial tension and compression in the direction: a molecular dynamics study. Metals, 11(11), 1–14. https://doi.org/10.3390/met11111760
Ariyanti, A. (2020). EFL students’ challenges towards home learning policy during COVID-19 outbreak. IJELTAL (Indonesian Journal of English Language Teaching and Applied Linguistics), 5(1), 167–175. https://doi.org/10.21093/ijeltal.v5i1.649
Babaii, E., & Nejadghanbar, H. (2017). Plagiarism among Iranian graduate students of language studies: perspectives and causes. Ethics & Behavior, 27(3), 240–258. https://doi.org/10.1080/10508422.2016.1138864
Bahari, A. (2021). Computer-assisted language proficiency assessment tools and strategies. Open Learning: The Journal of Open, Distance and e-Learning, 36(1), 61–87. https://doi.org/10.1080/02680513.2020.1726738
Belarbi, F. M., & Bensafa, A. (2020). An evaluation of the Algerian EFL baccalaureate exam under the cognitive domains of Bloom’s taxonomy. Arab World English Journal, 11(4), 534–546. https://doi.org/10.24093/awej/vol11no4.34
Błachnio, A., Cudo, A., Kot, P., Torój, M., Oppong Asante, K., Enea, V., Ben-Ezra, M., Caci, B., Dominguez-Lara, S. A., Kugbey, N., Malik, S., Servidio, R., Tipandjan, A., & Wright, M. F. (2022). Cultural and psychological variables predicting academic dishonesty: a cross-sectional study in nine countries. Ethics & Behavior, 32(1), 44–89. https://doi.org/10.1080/10508422.2021.1910826
Buono, S., & Jang, E. E. (2021). The effect of linguistic factors on assessment of English language learners’ Mathematical ability: a differential item functioning analysis. Educational Assessment, 26(2), 125–144. https://doi.org/10.1080/10627197.2020.1858783
Chen, J., & Perie, M. (2018). Comparability within computer-based assessment: does screen size matter? Computers in the Schools, 35(4), 268–283. https://doi.org/10.1080/07380569.2018.1531599
Cherner, T., & Mitchell, C. (2021). Deconstructing EdTech frameworks based on their creators, features, and usefulness. Learning, Media and Technology, 46(1), 91–116. https://doi.org/10.1080/17439884.2020.1773852
Chirikov, I., Shmeleva, E., & Loyalka, P. (2020). The role of faculty in reducing academic dishonesty among Engineering students. Studies in Higher Education, 45(12), 2464–2480. https://doi.org/10.1080/03075079.2019.1616169
Clark, A., Goodfellow, J., & Shoufani, S. (2020). Examining academic integrity using course-level learning outcomes. The Canadian Journal for the Scholarship of Teaching and Learning, 11(2), 1–19. https://doi.org/10.5206/cjsotl-rcacea.2020.2.8508
Comas-Forgas, R., Lancaster, T., Calvo-Sastre, A., & Sureda-Negre, J. (2021). Exam cheating and academic integrity breaches during the COVID-19 pandemic: an analysis of internet search activity in Spain. Heliyon, 7(10), 1–8. https://doi.org/10.1016/j.heliyon.2021.e08233
Council of Europe. (2001). Common European framework of reference for languages: learning, teaching, assessment. Council of Europe.
Cui, R., & Teo, P. (2021). Dialogic education for classroom teaching: a critical review. Language and Education, 35(3), 187–203. https://doi.org/10.1080/09500782.2020.1837859
Delahunty, T., & Nà RÃordáin, M. (2022). Perspectives in contemporary STEM education research: research methodology and design (1st ed.). Routledge. https://doi.org/10.4324/9781003108122
Ding, H., Zhu, Q., & Lyu, H. (2020). Identification of contact stiffness between brake disc and brake pads using modal frequency analysis. Journal of Engineering and Technological Sciences, 52(4), 468–480. https://doi.org/10.5614/j.eng.technol.sci.2020.52.4.2
Downing, S. M., & Haladyna, T. M. (1997). Test item development: validity evidence from quality assurance procedures. Applied Measurement in Education, 10(1), 61–82. https://doi.org/10.1207/s15324818ame1001_4
Fan, J., & Jin, Y. (2020). Standards for language assessment: demystifying university-level English placement testing in China. Asia Pacific Journal of Education, 40(3), 386–400. https://doi.org/10.1080/02188791.2019.1706445
Finchilescu, G., & Cooper, A. (2018). Perceptions of academic dishonesty in a South African university: a Q-methodology approach. Ethics & Behavior, 28(4), 284–301. https://doi.org/10.1080/10508422.2017.1279972
Fluck, A., Pullen, D., & Harper, C. (2009). Case study of a computer-based examination system. Australasian Journal of Educational Technology, 25(4), 17–32. https://doi.org/10.14742/ajet.1126
Gestanti, R. A., Nimasari, E. P., & Mufanti, R. (2019). ESP issue in Indonesian tertiary context: what students need in learning English. PUPIL: International Journal of Teaching, Education and Learning, 3(1), 98–117. https://doi.org/10.20319/pijtel.2019.31.98117
Gestanti, R. A., Nimasari, E. P., Nurfitri, K., & Cobantoro, A. F. (2023). Designing prototype product of English test book materials for Indonesian tertiary students. International Social Sciences and Humanities, 2(1), 149–156. https://doi.org/10.32528/issh.v2i1
Güngör, B., & Önder, A. (2022). Development of English picture vocabulary test as an assessment tool for very young EFL learners’ receptive and expressive language skills. Early Education and Development, 33(3), 1–18. https://doi.org/10.1080/10409289.2022.2043134
Hamavandy, M. (2014). A historical overview on the concept of validity in language testing. Advances in Language and Literary Studies, 5(4), 86–91. https://doi.org/10.7575/aiac.alls.v.5n.4p.86
Harris, L., Harrison, D., McNally, D., & Ford, C. (2020). Academic integrity in an online culture: Do McCabe’s findings hold true for online, adult learners? Journal of Academic Ethics, 18(4), 419–434. https://doi.org/10.1007/s10805-019-09335-3
Hijazi, D., & AlNatour, A. (2021). Online learning challenges affecting students of English in an EFL context during COVID-19 pandemic. International Journal of Education and Practice, 9(2), 379–395. https://doi.org/10.18488/journal.61.2021.92.379.395
Hikamah, S. R., Rohman, F., & Kurniawan, N. (2021). Developing virtual communication skills in online learning based on modified PBL during the COVID-19 pandemic. International Journal of Education and Practice, 9(2), 323–339. https://doi.org/10.18488/journal.61.2021.92.323.339
Hill, K., & McNamara, T. (2015). Validity inferences under high-stakes conditions: a response from language testing. Measurement: Interdisciplinary Research and Perspectives, 13(1), 39–43. https://doi.org/10.1080/15366367.2015.1016334
Irshaidat, R. (2022). Interpretivism vs. Positivism in political marketing research. Journal of Political Marketing, 21(2), 126–160. https://doi.org/10.1080/15377857.2019.1624286
Jamieson, M. V. (2020). Keeping a learning community and academic integrity intact after a mid-term shift to online learning in chemical engineering design during the COVID-19 pandemic. Journal of Chemical Education, 97(9), 2768–2772. https://doi.org/10.1021/acs.jchemed.0c00785
Jannah, Z., Asih, R., Arifin, R., & Darminto. (2021). Study of electronic and magnetic properties of single layered graphene with vacancy and (-OH) adsorption by density functional theory calculation. Journal of Physics: Conference Series, 1951, 012011. https://doi.org/10.1088/1742-6596/1951/1/012011
Kuo, B.-C., Liao, C.-H., Pai, K.-C., Shih, S.-C., Li, C.-H., & Mok, M. M. C. (2020). Computer-based collaborative problem-solving assessment in Taiwan. Educational Psychology, 40(9), 1164–1185. https://doi.org/10.1080/01443410.2018.1549317
Kusumawati, A. J. (2020). Redesigning face-to-face into online learning for speaking competence during COVID-19: ESP for higher education in Indonesia. International Journal of Language Education, 4(2), 276–288. https://doi.org/10.26858/ijole.v4i2.14745
Leenknecht, M., Wijnia, L., Köhlen, M., Fryer, L., Rikers, R., & Loyens, S. (2021). Formative assessment as practice: the role of students’ motivation. Assessment & Evaluation in Higher Education, 46(2), 236–255. https://doi.org/10.1080/02602938.2020.1765228
Mariani, L., Pezze, M., Riganelli, O., & Santoro, M. (2012). AutoBlackTest: automatic black-box testing of interactive applications. 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation (pp. 81–90). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/ICST.2012.88
Maulana, H. S., & Sutantra, I. N. (2021). Design and analysis of ECVT on electric powered vehicles for determining the speed ratio. The International Journal of Mechanical Engineering and Sciences, 5(2), 17–29. https://doi.org/10.12962/j25807471.v5i2.8923
McCallum, S., & Milner, M. M. (2021). The effectiveness of formative assessment: Student views and staff reflections. Assessment & Evaluation in Higher Education, 46(1), 1–16. https://doi.org/10.1080/02602938.2020.1754761
Meng, Q., Jia, J., & Zhang, Z. (2020). A framework of smart pedagogy based on the facilitating of high order thinking skills. Interactive Technology and Smart Education, 17(3), 251–266. https://doi.org/10.1108/ITSE-11-2019-0076
Mongeau, M., Karsenty, H., Rouzé, V., & Hiriart-Urruty, J. B. (2000). Comparison of public-domain software for black box global optimization. Optimization Methods and Software, 13(3), 203–226. https://doi.org/10.1080/10556780008805783
Munaji, M., Buntoro, G. A., Purniawan, A., & Arifin, R. (2018). Molecular dynamics simulations of iron-joining using copper as a filler metal. Makara Journal of Science, 22(3), 137–141. https://doi.org/10.7454/mss.v22i3.8512
Munna, A. S., & Shaikh, M. S. I. (2020). Pedagogies and practice: online teaching during COVID-19. International Journal of Humanities and Innovation (IJHI), 3(4), 132–138. https://doi.org/10.33750/ijhi.v3i4.96
Nashruddin, W. (2013). Why muslim students plagiarize. Journal on English as a Foreign Language, 3(2), 93–103. https://doi.org/10.23971/jefl.v3i2.68
Nguyen, J. G., Keuseman, K. J., & Humston, J. J. (2020). Minimize online cheating for online assessments during COVID-19 pandemic. Journal of Chemical Education, 97(9), 3429–3435. https://doi.org/10.1021/acs.jchemed.0c00790
Nimasari, E. P. (2018). An ESP needs analysis: addressing the needs of English for Informatics Engineering. JEES (Journal of English Educators Society), 3(1), 23–40. https://doi.org/10.21070/jees.v3i1.1085
Nurcahyo, A. M. B., Arifin, R., Malyadi, M., Munaji, Ghulam, A. B., Ridwan, Alfiyan, S. H., & Frengky, V. D. A. (2020). Cooling times’ dependence on the glassy NiTi at extremely low temperatures: a result from rapid solidification using molecular dynamics simulations. Journal of Physics: Conference Series, 1428, 1–6. https://doi.org/10.1088/1742-6596/1428/1/012003
Onu, D. U., Onyedibe, M. C. C., Ugwu, L. E., & Nche, G. C. (2021). Relationship between religious commitment and academic dishonesty: is self-efficacy a factor? Ethics & Behavior, 31(1), 13–20. https://doi.org/10.1080/10508422.2019.1695618
Peimani, N., & Kamalipour, H. (2021). Online education and the COVID-19 0utbreak: a case study of online teaching during lockdown. Education Sciences, 11(2), 72–88. https://doi.org/10.3390/educsci11020072
Rets, I., Astruc, L., Coughlan, T., & Stickler, U. (2022). Approaches to simplifying academic texts in English: English teachers’ views and practices. English for Specific Purposes, 68(3), 31–46. https://doi.org/10.1016/j.esp.2022.06.001
Rojabi, A. R., Setiawan, S., Munir, A., Purwati, O., & Widyastuti. (2022). The camera-on or camera-off, is it a dilemma? Sparking engagement, motivation, and autonomy through microsoft teams videoconferencing. International Journal of Emerging Technologies in Learning (IJET), 17(11), 174–189. https://doi.org/10.3991/ijet.v17i11.29061
Shibuta, Y., Arifin, R., Shimamura, K., Oguri, T., Shimojo, F., & Yamaguchi, S. (2013). Ab initio molecular dynamics simulation of dissociation of methane on nickel(1 1 1) surface: unravelling initial stage of graphene growth via a CVD technique. Chemical Physics Letters, 565, 92–97. https://doi.org/10.1016/j.cplett.2013.02.038
Shin, D., & Cho, E. (2021). The national English ability test in Korea and its legitimising discourses. Journal of Multilingual and Multicultural D evelopment, 42(6), 537–550. https://doi.org/10.1080/01434632.2020.1745818
Swamy, M. K. C., & Bommanahalli, R. (2021). Experimental studies on mechanical properties of carbon nanotube reinforced aluminum 7075 composite material. Journal of Engineering and Technological Sciences, 53(6), 1208–1220. https://doi.org/10.5614/j.eng.technol.sci.2021.53.6.11
Thepseenu, B. (2020). Needs analysis for ESP course development: Thai civil engineering students’ perspectives. Asian Journal of Education and Training, 6(3), 433–442. https://doi.org/10.20448/journal.522.2020.63.433.442
Tosuncuoglu, I. (2018). Importance of assessment in ELT. Journal of Education and Training Studies, 6(9), 163–167. https://doi.org/10.11114/jets.v6i9.3443
Vidergor, H. E. (2018). Effectiveness of the multidimensional curriculum model in developing higherâ€order thinking skills in elementary and secondary students. The Curriculum Journal, 29(1), 95–115. https://doi.org/10.1080/09585176.2017.1318771
Xie, Q. (2020). Diagnosing linguistic problems in English academic writing of university students: an item bank approach. Language Assessment Quarterly, 17(2), 183–203. https://doi.org/10.1080/15434303.2019.1691214
Zhang, Y., Yin, H., & Zheng, L. (2018). Investigating academic dishonesty among Chinese undergraduate students: does gender matter? Assessment & Evaluation in Higher Education, 43(5), 812–826. https://doi.org/10.1080/02602938.2017.1411467
Zohar, A., & Agmon, V. A. (2018). Raising test scores vs. teaching higher order thinking (HOT): senior science teachers’ views on how several concurrent policies affect classroom practices. Research in Science & Technological Education, 36(2), 243–260. https://doi.org/10.1080/02635143.2017.1395332
DOI: https://doi.org/10.23971/jefl.v13i1.4496
Article Metrics
Abstract view : 250 timesPDF - 80 times
Refbacks
- There are currently no refbacks.
Copyright (c) 2023 THE AUTHOR(S)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
JEFL is published by Institut Agama Islam Negeri (IAIN) Palangka Raya in collaboration with the Association of Teachers of English Linguistics, Literature and Education (ELITE Association) in Indonesia [MoU manuscript].
Editor and Administration Address:
Department of English Language Education, Faculty of Teacher Training and Education, Institut Agama Islam Negeri (IAIN) Palangka Raya, Jalan G. Obos Komplek Islamic Centre Palangka Raya, Kalimantan Tengah, Indonesia, Postal Code 73111, Email: jefl@iain-palangkaraya.ac.id, Website: http://e-journal.iain-palangkaraya.ac.id/index.php/jefl
OAI address of JEFL is available at http://e-journal.iain-palangkaraya.ac.id/index.php/jefl/oai
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.Â