Analysis of ChatGPT as a Question-Answering Tool

Pichappan, Pit, Krishnamurthy, M and Vijayakumar, P Analysis of ChatGPT as a Question-Answering Tool. Journal of Digital Information Management, 2023, vol. 21, n. 2, pp. 50-60. [Journal article (Paginated)]

[thumbnail of jdimv21i2_3.pdf]
Preview
Text
jdimv21i2_3.pdf

Download (1MB) | Preview
[thumbnail of jdimv21i2_3.pdf]
Preview
Text
jdimv21i2_3.pdf

Download (1MB) | Preview

English abstract

ChatGPT, in recent months, has made a significant impact and exposure in the information world. Many studies have been conducted within a shorter timeframe about its efficiency, reliability, ethics, accuracy and acceptance. In this study, the authors have looked at the ChatGPT as a question-answering tool using randomly generated prompts to solicit answers and analysed the results from a text analysis angle. The answers are compared with text analysers both manually and statistically. Authors suggest that ChatGPT still needs more precision for linguistic effects and fails to meet comprehensive users’ requirements.

Item type: Journal article (Paginated)
Keywords: ChatGPT, Question-answering tools, Text Analy- sis, AI tools, Natural Language Analysis
Subjects: L. Information technology and library technology > LL. Automated language processing.
L. Information technology and library technology > LM. Automatic text retrieval.
L. Information technology and library technology > LP. Intelligent agents.
Depositing user: Dr Pit Pichappan
Date deposited: 20 Jul 2023 15:49
Last modified: 20 Jul 2023 15:49
URI: http://hdl.handle.net/10760/44560

References

[1] https://fireflies.ai/blog/generative-ai-or-gpt-3-apps

[2] Number of parameters in notable Artificial Intelligence

Systems, Our World in Data. [Online]. Available: https://

ourworldindata.org/grapher/artificial-intelligenceparametercount. [Accessed: 22-Mar-2023].

[3] Hughes, A. (2023). CHATGPT: Everything you need to

know about OpenAI’s GPT-4 tool, ChatGPT: Everything

you need to know about OpenAI’s GPT-4 upgrade — BBC

Science, Focus Magazine, 16-Mar-2023. [Online]. Available: https://www.sciencefocus.com/futuretechnology/gpt3 [Accessed: 22-Mar-2023].

[4] https://towardsdatascience.com/gpt-3-a-complete-overview-190232eb25fd

[5] Yu, Caiyang., Liu, Xianggen., Tang, Chenwei., Feng,

Wentao., Jiancheng GPT-NAS: Neural Architecture Search

with the Generative Pre-Trained Model.

Lv.arXiv:2305.05351v1 [cs.CV] 9 May 2023]

[6] Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V.,

Gabr, M., Matsushita, H., Kim, Y. J., Afify, M., Awadalla,

H. H. (2023). How good are gpt models at machine translation? a comprehensive evaluation, arXiv preprint

arXiv:2302.09210, 2023

[7] Mathew L. Bindu, V. (2020). A review of natural language processing techniques for sentiment analysis using pre-trained models, In: 2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC). IEEE, 2020, p. 340–345.

[8] Bongini, P., Becattini, F., Del Bimbo, A. (2022). Is gpt3 all you need for visual question answering in cultural

heritage? In: Computer Vision–ECCV 2022 Workshops:

Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part

I. Springer, 2023, p. 268–281.

[9] Limna, Pongsakorn., Kraiwanit, Tanpat., Jangjarat,

Kris., KlayklungPrapasiri, Chocksathaporn, Piyawatjana

(2023). The use of ChatGPT in the digital era: Perspectives on chatbot implementation, Journal of Applied Learn-ing & Teaching. Vol.6 (1) 1-22.

[10] Cooper, K. (2021, November 1). OpenAI GPT-3: Everything you need to know. Springboard, https://

www.springboard.com/blog/data-science/machine-learning-gpt-3-open-ai

[11] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan,

J. D., Dhariwal, P., Amodei, D. (2020). Language models

are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901.

[12] Dehouche, N. (2021). Plagiarism in the age of massive Generative Pre-trained Transformers (GPT-3). Ethics

in Science and Environmental Politics, 21, 17-23

[13] Fyfe, P. (2022). How to cheat on your final paper:

Assigning AI for student writing. AI & Society, 1-11.

[14] Anson, C. M., Straume, I. (2022). Amazement and

trepidation: Implications of AI-based natural language production for the teaching of writing. Journal of Academic

Writing, 12 (1) 1-9.

[15] Anson, C. M. (2022). AI-based text generation and

the social construction of “fraudulent authorship”: A

revisitation. Composition Studies, 50, 37-46.

[16] Köbis, N., Mossink, L. D. (2021). Artificial intelligence

versus Maya Angelou: Experimental evidence that people

cannot differentiate AI-generated from human-written poetry. Computers in Human Behavior, 114, 106553

[17] Tate, T., Doroudi, S., Ritchie, D., Xu, Y. (2023). Educational research and AI-generated writing: Confronting

the coming tsunami. Pre-print. https://edarxiv.org/4mec3

[18] Moore, S., Nguyen, H. A., Bier, N., Domadia, T.,

Stamper, J. (2022). Assessing the quality of student-generated short answer questions using GPT-3. In: European

Conference on Technology Enhanced Learning (p. 243-

257). Springer.

[19] Elkins, K., Chun, J. (2020). Can GPT-3 pass a writer’s

Turing test? Journal of Cultural Analytics, 5 (2) 17212.

[20] Nguyen, H. A., Bhat, S., Moore, S., Bier, N., Stamper,

J. (2022). Towards generalized methods for automatic

question generation in educational domains. In: European

Conference on Technology Enhanced Learning p. 272-

284. Springer.

[21] Yu, Caiyang., Liu, Xianggen., Tang, Chenwei., Feng,

Wentao., Lv, Jiancheng.(GPT-NAS: Neural Architecture

Search with the Generative Pre-Trained

Model.arXiv:2305.05351v1 [cs.CV] 9 May 2023

[22] Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V.,

Gabr, M., Matsushita, H., Kim, Y. J., Afify, M. Awadalla,

H. H (2023). How good are GPT models at machine translation? a comprehensive evaluation, arXiv preprint

arXiv:2302.09210

[23] Mathew, L., Bindu, V (2020). A review of natural language processing techniques for sentiment analysis using pre-trained models, In: 2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC). IEEE, 2020, p. 340–345.

[24] Bongini, P., Becattini, F. Del Bimbo, A (2022). Is gpt3 all you need for visual question answering in cultural

heritage? In: Computer Vision–ECCV 2022 Workshops:

Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part

I. Springer, 2023, p. 268–281.

[25] Li, Chenglu., Xing, Wanli. (2021). Natural Language

Generation Using Deep Learning to Support MOOC Learners. International Journal of Artificial Intelligence in Education v. 31, p.186–214.

[26] Caballé, S., Conesa, J. (2019). Conversational Agents

in Support for Collaborative Learning in MOOCs: An Analytical Review. In F. Xhafa, L. Barolli, & M. Greguš (Eds.),

Advances in Intelligent Networking and Collaborative Systems. INCoS 2018. Lecture notes on data engineering

and communications technologies (Vol. 23). Cham:

Springer.

[27] Kumar, R., Rose, C. P. (2010). Architecture for building conversational agents that support collaborative learning. IEEE Transactions on Learning Technologies, 4 (1)

21–34. https://doi.org/10.1109/TLT.2010.41

[28] Mittal, A., Vigentini, L., Djatmiko, M., Prusty, G.,

Sharma, Y., King, M. E. (2018). MOOC-O-Bot: Using

Cognitive Technologies to Extend Knowledge Support in

MOOCs. Proceedings of the 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), 69–76. https://doi.org/10.1109/

tale.2018.8615453.]

[29] Shawar and Atwell Abu Shawar, B., Atwell, E. (2007).

Chatbots: Are They Really Useful? LDV-Forum Band, 22,

29-49.

[30] Indurthi, S. R., Raghu, D., Khapra, M. M., Joshi, S.

(2017). Generating natural language question-answer pairs

from a knowledge graph using a RNN-based question

generation model. In: M. Lapata, P. Blunsom, A. Koller

(Eds), Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers (pp. 376-385). Association for Computational Linguistics. https://doi.org/

10.18653/v1/e17-1036.

[31] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D.,

Sutskever, I. (2019). Language models are unsupervised

multitask learners. OpenAI Blog, 1 (8) 9. Retrieved 4 Jan

2020, from http://www.persagen.com/files/misc/radford2019language.pdf

[32] Zhang, Y., Sun, S., Galley, M., Chen, Y.-C.,

Brockett, C., Gao, X., Dolan, B. (2020). DialoGPT:

Large-scale generative pre-training for conversational

response generation. In: A. Celikyilmaz, T. Wen (Eds),

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (p. 270–278). Association for Computational Linguistics. https://doi.org/

10.18653/v1/2020.acl-demos.30

[33] Eva A. M., van Dis, Bollen, Johan., Rooij, Robert

van., Zuidema, Willem., Bockting, Claudi L. (2023).

ChatGPT: five priorities for research, Nature V. 614. 224-

2266. 9 February 2023.

[34] Rich, A. S., Gureckis, T. M.(2019). Nature Mach.

Intell. 1, 174–180. 2019.

[35] Else, Holly., Gao, C. A. et al (2022). ABSTRACTS

WRITTEN BY CHATGPT FOOL SCIENTISTS by. Nature V. 613, 19 January 2023 p. 423. Preprint at bioRxiv

https://doi.org/10.1101/2022.12.23.521610.

[36] ChatGPT: five priorities for research by Eva A. M.

van Dis, Johan Bollen, Robert van Rooij, Willem Zuidema

& Claudi L. Bockting- 224-2266, Nature, Vol 614 . 9 February 2023.

[37] JOUR Kenney, Noah A Brief Analysis of the Architecture, Limitations, and Impacts of ChatGPT DO -

10.5281/zenodo.7762245 2023/03/23

[38] Zhu, Yaoming., Lu, Sidi., Zheng, Lei., Guo, Jiaxian.,

Zhang, Weinan., Wang, Jun ., Yu, Yong. (2018). Texygen:

A Benchmarking Platform for Text Generation Models.

ACM SIGIR’18, July 8-12, 2018, p. 1097-1100. Ann Arbor,

MI, USA

[39] Sallam, Malik. (2023). ChatGPT Utility in Healthcare

Education, Research, and Practice: Systematic Review

on the Promising Perspectives and Valid Concerns,

Healthcare, 11 (6) 887; https://doi.org/10.3390/

healthcare11060887


Downloads

Downloads per month over past year

Actions (login required)

View Item View Item