Thinking, fast and slow. A Framework for Qualitative Data Analysis tools
Today, I found myself revisiting a paper I wrote in 2016 titled "Qualitative Data Analysis Software - The State of the Art." Although chatGPT wasn't yet on the horizon at that time, there were already tools available, leveraging machine learning, that promised to automate data analysis. In my paper, I employed Kahneman's framework of two modes of thinking - fast and slow - to categorize the diverse range of tools offered by various software packages for qualitative data analysis. According to Kahneman (2011), these two modes of thinking are described as follows:
"System 1 runs automatically, and System 2 is normally in a comfortable low-effort mode... System 1 continuously generates suggestions for System 2: impressions, intuitions, intentions, and feelings... If all goes smoothly, which is most of the time, System 2 adopts the suggestions of System 1 with little or no modifications... When System 1 runs into difficulty, it calls on System 2 to support more detailed and specific processing that may solve the problem of the moment." (p. 23)
Qualitative Data Analysis in Consumer Experience Research
In recent years, an increasing number of tools have emerged, particularly in the non-academic sector focusing on market and consumer research, that align with System 1 thinking. This translates to low effort, automatic, quick, and easy.
Here are a few quotes on how the benefits of the tools are presented:
"Qualitative Insights at Quantitative Scale.""Engage in real-time conversations with hundreds of people simultaneously. Understand your audience's sentiments through AI-driven organization and analysis of responses. Make confident business decisions with faster, richer insights.""Transform recordings into insights within minutes, not hours.""Our suite of AI tools expedites analysis, making it simple to uncover and present meaningful insights."
It seems like analysis is a dirty word, and any minute spent on data analysis is a waste of time: “High-definition output, low-effort analysis […] With just a few clicks, easily find and highlight key moments to include in your report.”
Kahneman found that the intuitive hunches of a human using System 1 thinking are often correct. The more skilled a person is in an area, the better the institution will be. Nevertheless, System 1 decisions are prone to errors as ambiguities are neglected and doubts are suppressed, there is a bias to confirm beliefs, a focus on existing evidence and an ignorance of absent evidence. WYSIATI – what you see is all there is.
Automation in Qualitative Data Analysis
I fear automated data classification will lead exactly to a WYSIATI bias contrary to the claims that it is less biased and more reliable than a human coder - unless System 2 is activated after AI tools have delivered automated results. This means reflecting on the outcome, taking it with a grain of salt, and taking the extra effort to take a deeper look into the data. On the downside, this destroys the claim that data analysis can be accomplished in no time.
Maybe, the 80/20 rule also applies here, meaning that such an analysis is good enough for 80% of all use cases. Still, I wonder.
Consequences of Shallow Data Analysis
Let's consider an example of a person named Sam working on their PhD thesis, spending several months conducting a qualitative study and analysing data. What would be at stake for Sam if he were to analyse data as many market researchers do? Could the consequence be losing hundreds of thousands or even millions of dollars due to a superficial analysis that only dedicates a few days to data analysis, relying on a Type 1 analysis without considering the context and providing only a handful of quotes as evidence?
The answer obviously is “No”. For Sam, the worst outcome might be failing his viva voce, which would affect him and possibly those close to him. However, the impact would be relatively minor compared to a major brand relying on such an analysis. Nonetheless, for Sam, there is a system of checks and balances in place. His supervisor would not let him get away with an analysis that appears to be standard in big business, even though there the stakes are much higher.
When I inquire about how market researchers analyse data, they often tell me how little time they can dedicate to it because clients demand quick results and are unwilling to pay for a more thorough analysis.
This makes me wonder why companies conducting or commissioning qualitative research do not care about a data analysis that goes beyond quick and superficial. Why do they consider it enough to cherry-pick a few quotes? Why do they consider some data organized by AI to be the analysis? Isn’t there anybody anymore who has a basic understanding of qualitative methods?
In a recently published video about a web-based data analysis tool, the protagonists say: “Imagine in about 10 hours of interviews, you might only have 30 minutes to an hour of sound bites that are really the crux of the information or the insights you are looking for. […] AI will be able to serve those moments up to you on a platter like a TikTok feed.”
A skilled person would ask: Why are you collecting all of this data if you think that only a few sound bites are valuable? I could help take a closer look at the other 95% of the data and show you the diamonds beneath the surface.
Do you truly want to make costly strategic decisions based on a few sound bites in your data? Or was the purpose of commissioning qualitative research merely to embellish the quantitative data?
The Added Value of Analysing Qualitative Data
Qualitative data can give you much more than that. It is just a question of knowing how. AI tools can be of great assistance, no doubt. But what they deliver are not insights on a platter nor deep understanding and meaning. The latter is what insights mean, at least if you ask the Oxford language dictionary: “the capacity to gain an accurate and deep understanding of someone or something.”
Allow me to provide an example. In an interview study on evaluating the implementation of a new organizational model, a respondent began by mentioning their European-Mediterranean background. Initially, I was surprised that she chose this as an opening statement, but it turned out to be crucial for interpreting the rest of the interview. For this respondent, "European-Mediterranean" signified a focus on people, ensuring the entire team was involved, listening to their experiences and suggestions during the implementation process, and hiring individuals who shared the same values and professional understanding.
Thus, her opening statement gave meaning and helped to interpret the rest of the interview to draw valuable conclusions for the continuous roll-out of the new model. An AI tool or sherry-picked statements would not have been able to deliver the same results as they are unable to understand connections within the data.
Against the Status Quo
I am aware that I am challenging the status quo. I do not expect to revolutionize an entire industry overnight, but I am not alone. There are others who acknowledge the need to address the elephant in the room. For instance, Kevin Gray wrote:
I think “it’s not the software to blame as much as the poorly trained humans using it. Again, the basics are missing. It’s not just quant, as many of my contacts who specialize in qualitative research will remind me. There almost seems to be the idea that if you are able to speak your native tongue, you can do qualitative research.”
“Both qual and quant reports are sometimes little more than collages with lots of buzz and client jargon printed on them. These reports look good, but many clients are still complaining and wondering where our recommendations (if even offered) came from. Reports often lack substance, in other words, and, worse yet, fact and speculation are frequently conflated.”
Is There a Way Out?
When it comes to marketing and consumer research, it requires foresight from C-level executives. They must understand the added value of properly analysing qualitative data and acknowledge that it takes time to generate useful results, along with individuals who possess expertise in qualitative research methods.
AI tools can expedite analysis, but a skilled qualitative researcher is still required to comprehend and work with the generated output. Key insights cannot be generated in minutes—unless you prefer to bet against chance.
References
Chandra, Suhbas (2019). Data Manipulation: The Elephant in the Market Research Room.
Gray, Kevin (2017). An elephant in our room.
Friese, Susanne (2016). Qualitative data analysis software: The state of the art. Special Issue: Qualitative Research in the Digital Humanities, Bosch, Reinoud (Ed.), KWALON, 61, 21(1), 34-45.