Eighth-Day Freedom

Using AI as a research analysis tool

Using AI as a research analysis tool for social science data
As evidenced by some recent posts, we are using ChatGTP-4 as a research analysis tool, especially for the narrative responses to many of our survey questions. Our methodological approach on the task of pulling out the main themes is to read through each comment, taking notes on themes which emerge and then run the data through ChatGTP using the following prompt:

In a survey, participants were asked “[question]” Please group the following responses into a maximum of 5 themes with:

• • A summary for each theme.
  • 3 representative participant quotes for each theme.
  • The total number of responses to each theme, regardless of whether multiple themes were mentioned in the same response.

Use of AI and of ChatGPT in particular is increasing at an almost exponential level both inside and beyond academia. We would be remiss in our responsibility to make maximum use of the data were we to avoid the us of AI. The possibility for error exists, but so far we have found that the themes identified by ChatGTP analysis are spot on and in many cases has identified themes which we missed. One main goal of any research is to treat all data in the most objective manner possible, eliminating all conscious or unconscious bias. Given that one of our research team is ex-SDA and all four of us are atheists/agnostics, we have to grant the possibility of at least unconscious bias. Hence, using ChatGTP seems prudent.

One additional use of ChatGPT is that it can do language translation. We have a good number (n = 149) of respondents to the Portuguese version of our survey and will be able to present those data without the additional step of using a separate translation modality.

Resources
In an effort to stay current with the emerging literature on the use of AI in social science research we are scouring academic journals for relevant content. Below is our current list of sources, expertly amassed by Teresa LePors, master librarian at Elon University. Please contact us via email if you have additional resources for us.

Bano, M., Zowghi, D., & Whittle, J. (2023). Exploring Qualitative Research Using LLMs. arXiv preprint arXiv:2306.13298.

Chew, R., Bollenbacher, J., Wenger, M., Speer, J., & Kim, A. (2023). LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding. arXiv preprint arXiv:2306.14924.

Christou, P. A. (2023). A Critical Perspective Over Whether and How to Acknowledge the Use of Artificial Intelligence (AI) in Qualitative Studies. The Qualitative Report, 28(7), 1981-1991.

Christou, P. A. (2023). How to Use Artificial Intelligence (AI) as a Resource, Methodological and Analysis Tool in Qualitative Research?. Qualitative Report, 28(7).f

Chubb, L. A. (2023). Me and the Machines: Possibilities and Pitfalls of Using Artificial Intelligence for Qualitative Data Analysis. International Journal of Qualitative Methods, 22, 16094069231193593.

De Paoli, S. (2023). Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the model. arXiv preprint arXiv:2305.13014.

Fitkov-Norris, E., & Kocheva, N. (2023, August). Are we there yet? Thematic analysis, NLP, and machine learning for research. In European Conference on Research Methodology for Business and Management Studies (Vol. 22, No. 1, pp. 93-102).

Gamieldien, Y., Case, J. M., & Katz, A. (2023). Advancing Qualitative Analysis: An Exploration of the Potential of Generative AI and NLP in Thematic Coding. Available at SSRN 4487768.

Gao, R., Merzdorf, H. E., Anwar, S., Hipwell, M. C., & Srinivasa, A. (2023). Automatic assessment of text-based responses in post-secondary education: A systematic review. arXiv preprint arXiv:2308.16151.

Hayes, A. (2023). “Conversing” with Qualitative Data: Enhancing Qualitative Sociological Research through Large Language Models (LLMs).

Hitch, D. (2023). Artificial Intelligence (AI) Augmented Qualitative Analysis: The way of the future?. Available at SSRN.

Katz, A., Wei, S., Nanda, G., Brinton, C., & Ohland, M. (2023). Exploring the Efficacy of ChatGPT in Analyzing Student Teamwork Feedback with an Existing Taxonomy. arXiv preprint arXiv:2305.11882.

Koch, M. A. (2023). Turning Chaos into Meaning: A Chat GPT-Assisted Exploration of COVID-19 Narratives(Master’s thesis, University of Twente).

Li, D., Zhang, B., & Zhou, Y. (2023). Can Large Language Models (LLM) label topics from a topic model?.

Marshall, D. T., & Naff, D. B. (2023). The Ethics of Using Artificial Intelligence in Qualitative Research. d

Mesec, B. (2023). The language model of artificial inteligence chatgpt-a tool of qualitative analysis of texts.

Navigli, R., Conia, S., & Ross, B. (2023). Biases in Large Language Models: Origins, Inventory and Discussion. ACM Journal of Data and Information Quality.

Tai, R. H., Bentley, L. R., Xia, X., Sitt, J. M., Fankhauser, S. C., Chicas-Mosier, A. M., & Monteith, B. G. (2023). Use of Large Language Models to Aid Analysis of Textual Data. bioRxiv, 2023-07.

Törnberg, P. (2023). How to use LLMs for Text Analysis. arXiv preprint arXiv:2307.13106.

Tschisgale, P., Wulff, P., & Kubsch, M. (2023). Integrating artificial intelligence-based methods into qualitative research in physics education research: A case for computational grounded theory. Physical Review Physics Education Research, 19(2), 020123.

Zhong, Y., Lian, J., & Huang, H. Uncovering the Affordances of ChatGPT in Education from a Social-Ecological Perspective: A Data Mining Approach. Available at SSRN 4518523.

Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023). Can Large Language Models Transform Computational Social Science?. arXiv preprint arXiv:2305.03514.