Rethinking Qualitative Data Analysis: Do we truly want a faster horse?
You may have come across the quote: "If I had asked people what they wanted, they would have said a faster horse." Although often attributed to Henry Ford, there is no evidence that he actually said it. The essence of the quote is that true innovation goes beyond customer input.
While customer input is generally important for product development, there are instances where customers may be limited by their current perspective. Consider the example of the PC:
When asked in the early 1940s how many computers the world needed, IBM's president, Thomas J. Watson, responded, "I think there is a world market for about five computers."
Innovations such as the Mac, iPod, or iPhone wouldn't exist if a few individuals hadn't thought beyond the existing solutions and generated fresh ideas.
How does this relate to qualitative data analysis?
Recently, we have witnessed the integration of generative AI in various tools, including software designed to support qualitative data analysis. You can find related videos in this playlist on my YouTube channel
Generative AI has been implemented in three primary ways:
- Summarizing coded data.
- Summarizing categorized data and enabling users to ask the built-in AI assistant questions about the data.
- Coding qualitative text data, with the promise of replacing time-consuming manual coding.
Earlier this year, I wrote about the use of machine learning tools for coding in qualitative research and explored the potential usefulness of Chat-GPT for qualitative researchers. My conclusion was that machine learning-generated codes based on noun phrases or NLP learning could not fully replace human coding in qualitative research. For more details, take a look at the folliwng post.
However, I did find Chat-GPT useful for summarizing coded data. At that time, this feature was not integrated into any tools, so I conducted my own tests by copying and pasting content between applications. Subsequently, some tools integrated this feature. The results of my tests can be viewed in this video: Decoding AI Summaries: Which Qualitative Data Analysis Tool Performs Best?
To encapsulate my findings, the outcomes are nuanced. Summarizing coded data segments proves to be relatively effective, as it targets a specific subset of the data. However, summarizing entire documents tends to be less successful. The AI's approach to summarization is highly selective, revealing different facets of the data with each new summary requested. This phenomenon is linked to the inherent limitations in LLMs' ability to comprehend extended contexts. In February 2024, Google announced a significant advancement with its new Gemini 1.5 model, suggesting that, moving forward, generative AI tools may also improve in their ability to summarize longer documents more effectively.
During 2023, ATLAS.ti unveiled several AI-assisted coding features, promising to deliver "qualitative insights in minutes instead of weeks." I evaluated these features and found them lacking. In the following video, I provide an overview of my assessment of all the tools that were introduced in 2023:
My pursuit of this subject is primarily academic. Throughout my professional career, I have been dedicated to advancing qualitative data analysis through computing and writing about it. Now, I believe we are on the cusp of a new era, prompting the need to reconsider qualitative data analysis methodologies.
Promising a faster horse wouldn't have propelled us into the future 115 years ago, just as the utilization of generative AI for coding does today.
Envisioning the Future
In the spring of 2023, I was approached by researchers from MIT who were developing a new AI-based tool for qualitative data analysis called AI-LYZE. With their technical background, they sought to understand how social scientists work with qualitative data and came across my YouTube videos, leading to our connection.
As I began experimenting with the tool in the early summer on 2023, albeit in its rudimentary stage, I could already glimpse the future. Is it time to move beyond exclusively relying on data coding?
This method has been our backbone for the past three to five decades, evolving from the era of index cards and towering paper stacks. However, the moment has arrived for a paradigm shift, inviting us to reconsider and innovate beyond traditional practices.
For further ideas, check out this video:
Article update in February 2024.