“I can’t search on Google for answersâ€: validity evidence of a developed computer-based assessment application

Elok Putri Nimasari, Rohfin Andria Gestanti, Khoiru Nurfitri


While learning management systems have been increasingly used as innovative ways to teach students in the post-COVID-19 pandemic, academic dishonesty is still in crisis. The open-access facility has the potential intervention in academic dishonesty. This study reported validity evidence of a developed computer-based assessment application to combat academic dishonesty: 1) measuring the development of high-order-thinking-skill (HOTS) test items; 2) examining the test item's validity integrated into the application; and 3) reporting the program analysis of the anti-cheating application for English for Specific Purposes (ESP) class assessment. This mixed-method study used focus group discussion and reading tests to collect the data. Pearson product-moment and black box analysis were used to measure test-item validity and application. The quantitative result revealed that test items were developed according to authentic journal texts and HOTS question characteristics and were statistically proven valid and reliable. Simplifying the authentic text empirically enabled EFL students to have an easier ESP reading comprehension complexity level. The black box analysis showed that the auto-lock screen feature prevented suspicious cheating access. This study contributed to extending practical and theoretical research on technology-enhanced language learning (TELL) by providing an innovative anti-cheating technology application.


academic dishonesty; computer-based assessment; online class; technology-enhanced language learning (TELL); validity evidence

Full Text:


Article Metrics

Abstract view : 250 times | PDF view : 80 times


Admiraal, W., Vermeulen, J., & Bulterman-Bos, J. (2020). Teaching with learning analytics: how to connect computer-based assessment data with classroom instruction? Technology, Pedagogy and Education, 29(5), 577–591. https://doi.org/10.1080/1475939X.2020.1825992

Alper, S., & Duygu, A. (2017). The effects of test length and sample size on item parameters in item response theory. Educational Sciences: Theory & Practice, 17(1), 321–325. https://doi.org/10.12738/estp.2017.1.0270

Alshamsi, A., Zahavich, A., & El-Farra, S. (2021). Why graded assessment for undergraduates during the COVID-19 lockdown? an experience introspection. IAFOR Journal of Education, 9(2), 55–75. https://doi.org/10.22492/ije.9.2.04

Arifin, R., Astuti, F., Baqiya, M. A., Winardi, Y., Wicaksono, Y. A., Darminto, & Selamat, A. (2021). Structural change of TiAl alloy under uniaxial tension and compression in the direction: a molecular dynamics study. Metals, 11(11), 1–14. https://doi.org/10.3390/met11111760

Ariyanti, A. (2020). EFL students’ challenges towards home learning policy during COVID-19 outbreak. IJELTAL (Indonesian Journal of English Language Teaching and Applied Linguistics), 5(1), 167–175. https://doi.org/10.21093/ijeltal.v5i1.649

Babaii, E., & Nejadghanbar, H. (2017). Plagiarism among Iranian graduate students of language studies: perspectives and causes. Ethics & Behavior, 27(3), 240–258. https://doi.org/10.1080/10508422.2016.1138864

Bahari, A. (2021). Computer-assisted language proficiency assessment tools and strategies. Open Learning: The Journal of Open, Distance and e-Learning, 36(1), 61–87. https://doi.org/10.1080/02680513.2020.1726738

Belarbi, F. M., & Bensafa, A. (2020). An evaluation of the Algerian EFL baccalaureate exam under the cognitive domains of Bloom’s taxonomy. Arab World English Journal, 11(4), 534–546. https://doi.org/10.24093/awej/vol11no4.34

Błachnio, A., Cudo, A., Kot, P., Torój, M., Oppong Asante, K., Enea, V., Ben-Ezra, M., Caci, B., Dominguez-Lara, S. A., Kugbey, N., Malik, S., Servidio, R., Tipandjan, A., & Wright, M. F. (2022). Cultural and psychological variables predicting academic dishonesty: a cross-sectional study in nine countries. Ethics & Behavior, 32(1), 44–89. https://doi.org/10.1080/10508422.2021.1910826

Buono, S., & Jang, E. E. (2021). The effect of linguistic factors on assessment of English language learners’ Mathematical ability: a differential item functioning analysis. Educational Assessment, 26(2), 125–144. https://doi.org/10.1080/10627197.2020.1858783

Chen, J., & Perie, M. (2018). Comparability within computer-based assessment: does screen size matter? Computers in the Schools, 35(4), 268–283. https://doi.org/10.1080/07380569.2018.1531599

Cherner, T., & Mitchell, C. (2021). Deconstructing EdTech frameworks based on their creators, features, and usefulness. Learning, Media and Technology, 46(1), 91–116. https://doi.org/10.1080/17439884.2020.1773852

Chirikov, I., Shmeleva, E., & Loyalka, P. (2020). The role of faculty in reducing academic dishonesty among Engineering students. Studies in Higher Education, 45(12), 2464–2480. https://doi.org/10.1080/03075079.2019.1616169

Clark, A., Goodfellow, J., & Shoufani, S. (2020). Examining academic integrity using course-level learning outcomes. The Canadian Journal for the Scholarship of Teaching and Learning, 11(2), 1–19. https://doi.org/10.5206/cjsotl-rcacea.2020.2.8508

Comas-Forgas, R., Lancaster, T., Calvo-Sastre, A., & Sureda-Negre, J. (2021). Exam cheating and academic integrity breaches during the COVID-19 pandemic: an analysis of internet search activity in Spain. Heliyon, 7(10), 1–8. https://doi.org/10.1016/j.heliyon.2021.e08233

Council of Europe. (2001). Common European framework of reference for languages: learning, teaching, assessment. Council of Europe.

Cui, R., & Teo, P. (2021). Dialogic education for classroom teaching: a critical review. Language and Education, 35(3), 187–203. https://doi.org/10.1080/09500782.2020.1837859

Delahunty, T., & Ní Ríordáin, M. (2022). Perspectives in contemporary STEM education research: research methodology and design (1st ed.). Routledge. https://doi.org/10.4324/9781003108122

Ding, H., Zhu, Q., & Lyu, H. (2020). Identification of contact stiffness between brake disc and brake pads using modal frequency analysis. Journal of Engineering and Technological Sciences, 52(4), 468–480. https://doi.org/10.5614/j.eng.technol.sci.2020.52.4.2

Downing, S. M., & Haladyna, T. M. (1997). Test item development: validity evidence from quality assurance procedures. Applied Measurement in Education, 10(1), 61–82. https://doi.org/10.1207/s15324818ame1001_4

Fan, J., & Jin, Y. (2020). Standards for language assessment: demystifying university-level English placement testing in China. Asia Pacific Journal of Education, 40(3), 386–400. https://doi.org/10.1080/02188791.2019.1706445

Finchilescu, G., & Cooper, A. (2018). Perceptions of academic dishonesty in a South African university: a Q-methodology approach. Ethics & Behavior, 28(4), 284–301. https://doi.org/10.1080/10508422.2017.1279972

Fluck, A., Pullen, D., & Harper, C. (2009). Case study of a computer-based examination system. Australasian Journal of Educational Technology, 25(4), 17–32. https://doi.org/10.14742/ajet.1126

Gestanti, R. A., Nimasari, E. P., & Mufanti, R. (2019). ESP issue in Indonesian tertiary context: what students need in learning English. PUPIL: International Journal of Teaching, Education and Learning, 3(1), 98–117. https://doi.org/10.20319/pijtel.2019.31.98117

Gestanti, R. A., Nimasari, E. P., Nurfitri, K., & Cobantoro, A. F. (2023). Designing prototype product of English test book materials for Indonesian tertiary students. International Social Sciences and Humanities, 2(1), 149–156. https://doi.org/10.32528/issh.v2i1

Güngör, B., & Önder, A. (2022). Development of English picture vocabulary test as an assessment tool for very young EFL learners’ receptive and expressive language skills. Early Education and Development, 33(3), 1–18. https://doi.org/10.1080/10409289.2022.2043134

Hamavandy, M. (2014). A historical overview on the concept of validity in language testing. Advances in Language and Literary Studies, 5(4), 86–91. https://doi.org/10.7575/aiac.alls.v.5n.4p.86

Harris, L., Harrison, D., McNally, D., & Ford, C. (2020). Academic integrity in an online culture: Do McCabe’s findings hold true for online, adult learners? Journal of Academic Ethics, 18(4), 419–434. https://doi.org/10.1007/s10805-019-09335-3

Hijazi, D., & AlNatour, A. (2021). Online learning challenges affecting students of English in an EFL context during COVID-19 pandemic. International Journal of Education and Practice, 9(2), 379–395. https://doi.org/10.18488/journal.61.2021.92.379.395

Hikamah, S. R., Rohman, F., & Kurniawan, N. (2021). Developing virtual communication skills in online learning based on modified PBL during the COVID-19 pandemic. International Journal of Education and Practice, 9(2), 323–339. https://doi.org/10.18488/journal.61.2021.92.323.339

Hill, K., & McNamara, T. (2015). Validity inferences under high-stakes conditions: a response from language testing. Measurement: Interdisciplinary Research and Perspectives, 13(1), 39–43. https://doi.org/10.1080/15366367.2015.1016334

Irshaidat, R. (2022). Interpretivism vs. Positivism in political marketing research. Journal of Political Marketing, 21(2), 126–160. https://doi.org/10.1080/15377857.2019.1624286

Jamieson, M. V. (2020). Keeping a learning community and academic integrity intact after a mid-term shift to online learning in chemical engineering design during the COVID-19 pandemic. Journal of Chemical Education, 97(9), 2768–2772. https://doi.org/10.1021/acs.jchemed.0c00785

Jannah, Z., Asih, R., Arifin, R., & Darminto. (2021). Study of electronic and magnetic properties of single layered graphene with vacancy and (-OH) adsorption by density functional theory calculation. Journal of Physics: Conference Series, 1951, 012011. https://doi.org/10.1088/1742-6596/1951/1/012011

Kuo, B.-C., Liao, C.-H., Pai, K.-C., Shih, S.-C., Li, C.-H., & Mok, M. M. C. (2020). Computer-based collaborative problem-solving assessment in Taiwan. Educational Psychology, 40(9), 1164–1185. https://doi.org/10.1080/01443410.2018.1549317

Kusumawati, A. J. (2020). Redesigning face-to-face into online learning for speaking competence during COVID-19: ESP for higher education in Indonesia. International Journal of Language Education, 4(2), 276–288. https://doi.org/10.26858/ijole.v4i2.14745

Leenknecht, M., Wijnia, L., Köhlen, M., Fryer, L., Rikers, R., & Loyens, S. (2021). Formative assessment as practice: the role of students’ motivation. Assessment & Evaluation in Higher Education, 46(2), 236–255. https://doi.org/10.1080/02602938.2020.1765228

Mariani, L., Pezze, M., Riganelli, O., & Santoro, M. (2012). AutoBlackTest: automatic black-box testing of interactive applications. 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation (pp. 81–90). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/ICST.2012.88

Maulana, H. S., & Sutantra, I. N. (2021). Design and analysis of ECVT on electric powered vehicles for determining the speed ratio. The International Journal of Mechanical Engineering and Sciences, 5(2), 17–29. https://doi.org/10.12962/j25807471.v5i2.8923

McCallum, S., & Milner, M. M. (2021). The effectiveness of formative assessment: Student views and staff reflections. Assessment & Evaluation in Higher Education, 46(1), 1–16. https://doi.org/10.1080/02602938.2020.1754761

Meng, Q., Jia, J., & Zhang, Z. (2020). A framework of smart pedagogy based on the facilitating of high order thinking skills. Interactive Technology and Smart Education, 17(3), 251–266. https://doi.org/10.1108/ITSE-11-2019-0076

Mongeau, M., Karsenty, H., Rouzé, V., & Hiriart-Urruty, J. B. (2000). Comparison of public-domain software for black box global optimization. Optimization Methods and Software, 13(3), 203–226. https://doi.org/10.1080/10556780008805783

Munaji, M., Buntoro, G. A., Purniawan, A., & Arifin, R. (2018). Molecular dynamics simulations of iron-joining using copper as a filler metal. Makara Journal of Science, 22(3), 137–141. https://doi.org/10.7454/mss.v22i3.8512

Munna, A. S., & Shaikh, M. S. I. (2020). Pedagogies and practice: online teaching during COVID-19. International Journal of Humanities and Innovation (IJHI), 3(4), 132–138. https://doi.org/10.33750/ijhi.v3i4.96

Nashruddin, W. (2013). Why muslim students plagiarize. Journal on English as a Foreign Language, 3(2), 93–103. https://doi.org/10.23971/jefl.v3i2.68

Nguyen, J. G., Keuseman, K. J., & Humston, J. J. (2020). Minimize online cheating for online assessments during COVID-19 pandemic. Journal of Chemical Education, 97(9), 3429–3435. https://doi.org/10.1021/acs.jchemed.0c00790

Nimasari, E. P. (2018). An ESP needs analysis: addressing the needs of English for Informatics Engineering. JEES (Journal of English Educators Society), 3(1), 23–40. https://doi.org/10.21070/jees.v3i1.1085

Nurcahyo, A. M. B., Arifin, R., Malyadi, M., Munaji, Ghulam, A. B., Ridwan, Alfiyan, S. H., & Frengky, V. D. A. (2020). Cooling times’ dependence on the glassy NiTi at extremely low temperatures: a result from rapid solidification using molecular dynamics simulations. Journal of Physics: Conference Series, 1428, 1–6. https://doi.org/10.1088/1742-6596/1428/1/012003

Onu, D. U., Onyedibe, M. C. C., Ugwu, L. E., & Nche, G. C. (2021). Relationship between religious commitment and academic dishonesty: is self-efficacy a factor? Ethics & Behavior, 31(1), 13–20. https://doi.org/10.1080/10508422.2019.1695618

Peimani, N., & Kamalipour, H. (2021). Online education and the COVID-19 0utbreak: a case study of online teaching during lockdown. Education Sciences, 11(2), 72–88. https://doi.org/10.3390/educsci11020072

Rets, I., Astruc, L., Coughlan, T., & Stickler, U. (2022). Approaches to simplifying academic texts in English: English teachers’ views and practices. English for Specific Purposes, 68(3), 31–46. https://doi.org/10.1016/j.esp.2022.06.001

Rojabi, A. R., Setiawan, S., Munir, A., Purwati, O., & Widyastuti. (2022). The camera-on or camera-off, is it a dilemma? Sparking engagement, motivation, and autonomy through microsoft teams videoconferencing. International Journal of Emerging Technologies in Learning (IJET), 17(11), 174–189. https://doi.org/10.3991/ijet.v17i11.29061

Shibuta, Y., Arifin, R., Shimamura, K., Oguri, T., Shimojo, F., & Yamaguchi, S. (2013). Ab initio molecular dynamics simulation of dissociation of methane on nickel(1 1 1) surface: unravelling initial stage of graphene growth via a CVD technique. Chemical Physics Letters, 565, 92–97. https://doi.org/10.1016/j.cplett.2013.02.038

Shin, D., & Cho, E. (2021). The national English ability test in Korea and its legitimising discourses. Journal of Multilingual and Multicultural D evelopment, 42(6), 537–550. https://doi.org/10.1080/01434632.2020.1745818

Swamy, M. K. C., & Bommanahalli, R. (2021). Experimental studies on mechanical properties of carbon nanotube reinforced aluminum 7075 composite material. Journal of Engineering and Technological Sciences, 53(6), 1208–1220. https://doi.org/10.5614/j.eng.technol.sci.2021.53.6.11

Thepseenu, B. (2020). Needs analysis for ESP course development: Thai civil engineering students’ perspectives. Asian Journal of Education and Training, 6(3), 433–442. https://doi.org/10.20448/journal.522.2020.63.433.442

Tosuncuoglu, I. (2018). Importance of assessment in ELT. Journal of Education and Training Studies, 6(9), 163–167. https://doi.org/10.11114/jets.v6i9.3443

Vidergor, H. E. (2018). Effectiveness of the multidimensional curriculum model in developing higherâ€order thinking skills in elementary and secondary students. The Curriculum Journal, 29(1), 95–115. https://doi.org/10.1080/09585176.2017.1318771

Xie, Q. (2020). Diagnosing linguistic problems in English academic writing of university students: an item bank approach. Language Assessment Quarterly, 17(2), 183–203. https://doi.org/10.1080/15434303.2019.1691214

Zhang, Y., Yin, H., & Zheng, L. (2018). Investigating academic dishonesty among Chinese undergraduate students: does gender matter? Assessment & Evaluation in Higher Education, 43(5), 812–826. https://doi.org/10.1080/02602938.2017.1411467

Zohar, A., & Agmon, V. A. (2018). Raising test scores vs. teaching higher order thinking (HOT): senior science teachers’ views on how several concurrent policies affect classroom practices. Research in Science & Technological Education, 36(2), 243–260. https://doi.org/10.1080/02635143.2017.1395332

DOI: https://doi.org/10.23971/jefl.v13i1.4496

Article Metrics

Abstract view : 250 times
PDF - 80 times


  • There are currently no refbacks.

Copyright (c) 2023 THE AUTHOR(S)

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

JEFL is published by Institut Agama Islam Negeri (IAIN) Palangka Raya in collaboration with the Association of Teachers of English Linguistics, Literature and Education (ELITE Association) in Indonesia [MoU manuscript].

Editor and Administration Address:

Department of English Language Education, Faculty of Teacher Training and Education, Institut Agama Islam Negeri (IAIN) Palangka Raya, Jalan G. Obos Komplek Islamic Centre Palangka Raya, Kalimantan Tengah, Indonesia, Postal Code 73111, Email: jefl@iain-palangkaraya.ac.id, Website: http://e-journal.iain-palangkaraya.ac.id/index.php/jefl

OAI address of JEFL is available at http://e-journal.iain-palangkaraya.ac.id/index.php/jefl/oai

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. 

stats View My Stats

Flag Counter