EVALUASI PROGRAM AI CHATBOT ANSIA DI PERGURUAN TINGGI: ANALISIS MODEL CIPP (CONTEXT, INPUT, PROCESS, PRODUCT)
DOI:
https://doi.org/10.51878/edutech.v6i4.16482Keywords:
Artificial Intelligence, Chatbot ANSIA, Evaluasi Program, Model CIPP, Perguruan Tinggi, Meylia Elizabeth RanuAbstract
This study aims to evaluate ANSIA (SIDIA Assistant), an artificial intelligence -based chatbot program at Universitas Negeri Surabaya, using the CIPP (Context, Input, Process, Product) evaluation model. A descriptive-evaluative qualitative approach was employed. Data were collected through in-depth interviews and document analysis involving developers from the Directorate of Digital Learning Innovation (DIPD) at UNESA as key informants, as well as six lecturers and 19 students from the Office Administration Education undergraduate study program, cohorts 2024 and 2025, as primary informants. Data credibility was examined through source triangulation, and the data were analyzed using the Miles and Huberman interactive model. The evaluation revealed four major gaps. In the context component, ANSIA was developed without a formal needs assessment, resulting in a gap in user awareness. In the input component, server infrastructure and WhatsApp integration were not adequately supported by a dedicated budget, formal socialization, or sufficient operational guidelines. In the process component, an increase in aggregate server-log access did not fully reflect student use of ANSIA at the study-program level. Interview findings indicated that preferences for commercial artificial intelligence tools and limitations in the internal knowledge base were factors associated with low utilization. In the product component, a discrepancy emerged between developers’ claims of 24-hour service availability and users’ experiences in accessing and using the system. The study recommends expanding the scientific knowledge base, providing visual user guides, allocating a dedicated budget, and integrating ANSIA into teaching and learning activities.
ABSTRAK
Penelitian ini bertujuan mengevaluasi program chatbot berbasis artificial intelligence, yaitu ANSIA (Asisten SIDIA), di Universitas Negeri Surabaya menggunakan model evaluasi CIPP (Context, Input, Process, Product). Penelitian menggunakan pendekatan kualitatif dengan desain deskriptif evaluatif. Data dikumpulkan melalui wawancara mendalam dan dokumentasi yang melibatkan pengembang dari Direktorat Inovasi Pembelajaran Digital (DIPD) UNESA sebagai informan kunci serta enam dosen dan 19 mahasiswa Program Studi S1 Pendidikan Administrasi Perkantoran angkatan 2024 dan 2025 sebagai informan utama. Keabsahan data diperiksa melalui triangulasi sumber, sedangkan analisis data menggunakan model interaktif Miles dan Huberman. Hasil evaluasi menunjukkan empat kesenjangan utama. Pada komponen context, pengembangan ANSIA tidak didahului asesmen kebutuhan formal sehingga ditemukan kesenjangan kesadaran (awareness gap) pengguna. Pada komponen input, dukungan infrastruktur server dan integrasi WhatsApp belum diimbangi anggaran khusus, sosialisasi formal, serta panduan operasional yang memadai. Pada komponen process, peningkatan jumlah akses berdasarkan log server agregat tidak sepenuhnya mencerminkan pemanfaatan ANSIA oleh mahasiswa di tingkat program studi. Temuan wawancara menunjukkan bahwa kecenderungan menggunakan artificial intelligence komersial dan keterbatasan basis pengetahuan internal menjadi faktor yang berkaitan dengan rendahnya pemanfaatan tersebut. Pada komponen product, terdapat perbedaan antara klaim pengembang mengenai ketersediaan layanan selama 24 jam dan pengalaman pengguna dalam mengakses serta memanfaatkan sistem. Penelitian ini merekomendasikan perluasan basis pengetahuan ilmiah, penyediaan panduan visual, pengalokasian anggaran khusus, dan integrasi ANSIA dalam kegiatan pembelajaran.
Downloads
References
Aldhafeeri, L., Aljumah, F., Thabyan, F., Alabbad, M., AlShahrani, S., Alanazi, F., & Al-Nafjan, A. (2025). Generative AI chatbots across domains: A systematic review. Applied Sciences, 15(20), Article 11220. https://doi.org/10.3390/app152011220
Alfred, P. A. P., Mabotha, P., & Ngcamu, B. S. (2026). Digital transformation in the higher education sector: A systematic literature review. Administrative Sciences, 16(1), Article 1. https://doi.org/10.3390/admsci16010001
Almatrodi, I., Li, F., & Alojail, M. (2023). Organizational resistance to automation success: How status quo bias influences organizational resistance to an automated workflow system in a public organization. Systems, 11(4), Article 191.
Andrews, T. C., Conaway, E. P., Zhao, J., & Dolan, E. L. (2016). Colleagues as change agents: How department networks and opinion leaders influence teaching at a single institution. CBE—Life Sciences Education, 15(2). [DOI atau URL artikel perlu dilengkapi].
Bond, M., Khosravi, H., De Laat, M., Bergdahl, N., Negrea, V., Oxley, E., Pham, P., Chong, S. W., & Siemens, G. (2024). A meta systematic review of artificial intelligence in higher education: A call for increased ethics, collaboration, and rigour. International Journal of Educational Technology in Higher Education, 21, Article 4. https://doi.org/10.1186/s41239-023-00436-z
Chan, C. K. Y., & Hu, W. (2023). Students’ voices on generative AI: Perceptions, benefits, and challenges in higher education. International Journal of Educational Technology in Higher Education, 20, Article 43. https://doi.org/10.1186/s41239-023-00411-8
Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). SAGE Publications.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
El-Sobkey, S. B., Kelini, K. I., ElKholy, M., Abdeldayem, T., Abdallah, M., Al-Amir Mohamed, D., Fawzy, A., Ahmed, Y. F., El Khatib, A., Khalid, H., Shaik, B. B., Anjos, A., Alharbi, M. D., Fathy, K., & Takey, K. (2025). Acceptance of AI-powered chatbots among physiotherapy students: International cross-sectional study. JMIR Medical Education, 11, Article e76574.
Ertmer, P. A., & Ottenbreit-Leftwich, A. T. (2010). Teacher technology change: How knowledge, confidence, beliefs, and culture intersect. Journal of Research on Technology in Education, 42(3), 255–284. https://doi.org/10.1080/15391523.2010.10782551
Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial intelligence in education: Promises and implications for teaching and learning. Center for Curriculum Redesign.
Kahneman, D., Knetsch, J. L., & Thaler, R. H. (1991). Anchors: The endowment effect, loss aversion, and status quo bias. Journal of Economic Perspectives, 5(1), 193–206. https://doi.org/10.1257/jep.5.1.193
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, Article 102274. https://doi.org/10.1016/j.lindif.2023.102274
Kotter, J. P. (1996). Leading change. Harvard Business School Press.
Liang, T. (2012). Matrix analysis of the digital divide in eHealth services using awareness, want, and adoption gap. Journal of Medical Internet Research, 14(1), Article e11. https://doi.org/10.2196/jmir.1670
Menzli, L. J., Smirani, L. K., Boulahia, J. A., & Hadjouni, M. (2022). Investigation of open educational resources adoption in higher education using Rogers’ diffusion of innovation theory. Heliyon, 8(7), Article e09885. https://doi.org/10.1016/j.heliyon.2022.e09885
Nugawela, S., & Sedera, D. (2022). Status quo bias in users’ information systems (IS) adoption and continuance intentions: A literature review and framework. Australasian Conference on Information Systems (ACIS) 2022 Proceedings, Article 65. https://aisel.aisnet.org/acis2022/65
Rana, M., Safaet, M., Sakib, N., & Ahamed, R. (2024). Assessing AI adoption in developing country academia: A trust and privacy-augmented UTAUT framework. Heliyon, 10(18), Article e37569. https://doi.org/10.1016/j.heliyon.2024.e37569
Ranu, M. E. (2026). Evaluasi pendidikan: Teori, praktik, dan kontribusi terhadap pencapaian pendidikan berkualitas. Pustaka Aksara.
Rogers, E. M. (1995). Diffusion of innovations (4th ed.). The Free Press.
Samuelson, W., & Zeckhauser, R. (1988). Status quo bias in decision making. Journal of Risk and Uncertainty, 1(1), 7–59. https://doi.org/10.1007/BF00055564
Selwyn, N. (2004). Reconsidering political and popular understandings of the digital divide. New Media & Society, 6(3), 341–362. https://doi.org/10.1177/1461444804042519
Strzelecki, A. (2024). To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interactive Learning Environments, 32(9), 5142–5155. https://doi.org/10.1080/10494820.2023.2209881
Stufflebeam, D. L. (1971). The relevance of the CIPP evaluation model for educational accountability. Journal of Research and Development in Education, 5(1), 19–25.
Stufflebeam, D. L. (2002). CIPP evaluation model checklist. The Evaluation Center, Western Michigan University. [URL publikasi perlu dilengkapi].
Stufflebeam, D. L., & Zhang, G. (2017). The CIPP evaluation model: How to evaluate for improvement and accountability. The Guilford Press.
Tian, W., Ge, J., Zhao, Y., & Zheng, X. (2024). AI chatbots in Chinese higher education: Adoption, perception, and influence among graduate students—An integrated analysis utilizing UTAUT and ECM models. Frontiers in Psychology, 15, Article 1268549. https://doi.org/10.3389/fpsyg.2024.1268549
Van Dijk, J. A. G. M. (2017). Digital divide: Impact of access. In The international encyclopedia of media effects (pp. 1–11). Wiley. [DOI atau URL bab perlu dilengkapi].
Van Dijk, J. A. G. M. (2020). The digital divide. Polity Press.
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540
Venkatesh, V., Thong, J. Y. L., & Xu, X. (2012). Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology. MIS Quarterly, 36(1), 157–178. https://doi.org/10.2307/41410412
Warju, W. (2016). Educational program evaluation using CIPP model. Innovation of Vocational Technology Education, 12(1), 36–42.
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education: Where are the educators? International Journal of Educational Technology in Higher Education, 16, Article 39. https://doi.org/10.1186/s41239-019-0171-0
Zhang, G., Zeller, N., Griffith, R., Metcalf, D., Williams, J., Shea, C. K., & Misulis, K. (2011). Using the context, input, process, and product evaluation model (CIPP) as a comprehensive framework to guide the planning, implementation, and assessment of service-learning programs. Journal of Higher Education Outreach and Engagement, 15(4), 57–84.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 EDUTECH : Jurnal Inovasi Pendidikan Berbantuan Teknologi
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
