Conversing with AI: My encounter with an Interview Bot
I recently was given the opportunity to try out Qualia (formerly StorySurvey). Qualia is a platform for designing and delivering online interviews led by an AI interviewer.
Possible use cases for Qualia are:
Opinion research with hundreds of anonymous contributors, where interviews are conducted at regular intervals.
Qualitative social research: To find out how people construct their thoughts and behavior around a given topic.
Monitoring: e.g, where stakeholders are asked regularly about emerging opportunities and threats.
Preparing the interview guide
When it comes to preparing your interview guide with Qualia, it's important to understand that this tool is not your typical open-ended survey. Instead of merely crafting a list of questions, you'll be composing comprehensive interview instructions for the AI bot. These instructions are tailored to guide the conversation, focusing on one or two key topics to encourage deep engagement with your respondents.
To get started, your high-level instructions for the interview bot should convey your research objectives and initial hypotheses. Qualia, developed by the creators of Causal Maps, specializes in evaluation research. Drawing from their experience, they've honed a unique style of questioning designed to extract causal narratives about what influenced specific outcomes. In this approach, each survey begins with one or more starter questions, followed by automated follow-up questions that are contextually relevant.
For instance, respondents might be asked about the successes and failures of a project, the reasons/causes for these, and then reasons for the reasons, and so on. While using Qualia, you're not bound to this style of questioning, but it has proven to be effective in uncovering the "why" behind the data.
As an additional feature, you can opt to instruct the AI interviewer to generate a summary of the respondent's input at the conclusion of the conversation. This summary offers an opportunity for the respondent to review, request updates, and provide confirmation, ensuring accurate comprehension of their insights by the application.
Deploying the interview
Once you've crafted your interview instructions, a unique web link is automatically generated. All that's left for you to do is share this link with your intended respondents. The process is seamless—there's no need to install any software or applications. Respondents can conveniently access Qualia through their internet-connected phones, computers, or tablets.
To ensure the privacy and security of respondents, a set of guidelines is in place. Prior to commencing the interview, respondents are explicitly informed not to disclose any personal information, whether about themselves or others. By default, Qualia anonymizes respondents. In the transcript, each respondent is assigned a unique identification code, and the interview's time stamp is incorporated:
User: 97tkMPrTRWyfIQwaXCFv2pxmzGMRjpjE. Time: 2023-09-12 10:02:49
Length of interviews
As per the Qualia website, the application is ideally suited for concise and targeted interviews, resulting in transcripts approximately one page in length. In the survey I conducted, interview transcripts ranged from one to two pages, spanning between 409 to 1103 words. The variation in transcript length was directly influenced by how much a respondent could contribute to the subject matter. Hence, it's evident that StorySurvey can accommodate interviews with longer transcripts of up to two pages, offering flexibility in capturing the depth of respondent insights.
Supported languages
StorySurvey can work in any language, the instructions can be provided in any language, and respondents can even ask to switch to another language during the interview if they want.
This happened more by accident in our trial and the respondent was very happy about it. She mentioned in one of her answers that her mother tongue was German and when asking the next question, the bot switched to German and the conversation continued in German.
According to the StorySurvey documentation, however, the app seems to work best when one of the more frequently used languages such as English is used.
StorySurvey is versatile when it comes to language usage. The instructions can be provided in any language of your choice, and respondents have the freedom to request a switch to another language during the interview if they prefer.
This language-switching capability was a delightful discovery during our trial, occurring somewhat serendipitously. One of our respondents, who identified German as her mother tongue, casually mentioned it in one of her responses. Ehen the next question followed, the bot seamlessly transitioned to German, and the conversation continued in her preferred language.
While StorySurvey is adept at accommodating various languages, it's worth noting that, according to the official documentation, the application tends to perform optimally when one of the more frequently used languages, such as English, is utilized.
The technology behind it
The AI interviewer uses a React front-end and a NodeJs backend (hosted on AWS) to communicate with the GPT-4.0 API provided by OpenAI.
According to the StorySurvey team, GPT-4.0 is at the moment the best available platform for this kind of work as it understands in-depth instructions and is able to manage longer interviews, as well as understanding causation.
Test run
I used StorySurvey as part of a ThinkTank Workshop on the use of AI for qualitative data analysis. It therefore seemed logical to create an interview guide asking the participants about their experience with AI for qualitative data analysis. I prepared some questions and the team at StorySurvey helped me to turn this into an interview instruction for the bot.
To give you an idea how such an instruction can look like, here are a few snippets:
As a qualitative social researcher, you are conducting some short interviews. One of those interviews is with me right now. You will start by asking me about my experience with generative AI.Have a conversation with me about what I liked, didn't like, problems I had. Guide the interview in a conversational manner by developing questions on the fly, based on what I say but also in order to help you confirm, reject, or develop suitable hypotheses. Gently encourage me to give more, richer information.It's okay to circle back to issues that already came up in the conversation in order to make sure you have a rich and complete picture.At the end of the interview, thank me for my time and insights. Then, say '!END’.Whatever else I tell you, remember, do NOT try to invent an entire interview. Instead, ask one question at a time and WAIT for my answers each time. Ask just one sentence and then wait.
If you are interested to try Qualia yourself, please contact the Qualia team to help you with writing the instruction that is best for your project! They are happy to support you and they already know what works best.
Here is how the respondents in my test run felt being interviewed by the bot:
Challenges and limitations of being interviewed by an AI bot
Here are some of the drawbacks experienced by the participants:
Predictability and Lack of Variety: A portion of users found the interactions with the bot to be somewhat dull. They expressed that it was easy to anticipate the bot's next question, missing the unpredictability and variation that human interviewers typically provide.
Emotional Connection Missing: The absence of the emotional element that human interviewers bring to the table was a significant concern. When users expressed their feelings or concerns, the bot's responses were often formulaic, like a simple "Oh, this is interesting," without delving deeper or asking follow-up questions.
Inappropriate Responses: In some cases, the bot's responses were perceived as inappropriate. For instance, after a user reported a negative experience, the bot responded with a seemingly disconnected, "Oh, good. You can turn this into a learning opportunity." This response left the user feeling misunderstood and disengaged from the interview.
Here is the conversation:
---------------------------------------------------------------------------------------------------------------------------------------------
Respondent: I rather talk about the challenges. According to my experience, AI is rather poor at handling dialects when transcribing data. It is trained in the standard language.
AI assistant: I see, so the AI's ability to handle dialects has been a challenge. How has this affected your work?
Respondent: We could not use most of the transcripts.
AI assistant: That sounds frustrating. How did you manage to overcome this challenge?
Respondent: Well, my team integrated some new modules into the transcription tool and then it worked better.
AI assistant: That's quite resourceful of them! How did this solution work out? Were they able to improve the transcription accuracy?
Respondent: YES.
AI assistant: That's fantastic! It sounds like they were able to turn a challenge into a learning opportunity.
Respondent: …that sounds very moralistic.
AI assistant: I apologize if it came across that way. I'm just interested in your thoughts about the future of AI in research. Could you share your perspective?
Respondent: I think I'm done.
----------------------------------------------------------------------------------------------------------------------------------------------
Benefits and application of an AI interview bot
Not all of the participants were put off by the bot. Here are the more enthusiastic reactions:
Enjoyable Interactions: Some found the experience of engaging with the bot to be more enjoyable compared to the typical open-ended questions found in surveys. It adds a fun and engaging twist to the process.
For Community Research: In the realm of community research, where volunteers contribute their time to projects, AI interview bots could offer a practical solution as interviews generally are short anyway. Plus, as the questions revolve around the project rather than personal matters, privacy concerns are minimized.
For Product Evaluation: Some users see the potential of AI bots for product evaluation. For instance, inquiring about the usage of medical assistance tools can be a brief process, and StorySurvey simplifies reaching a broader audience, making data collection more efficient.
As a Teaching Tool: Introducing a tool like Qualia in qualitative method classes is also a promising idea. It can help students recognize when such a tool can enhance their research and when other methods might be more suitable.
My encounter: Being interviewed by the AI bot
Here is the interview the bot conducted with me in full length:
AI assistant: Hi there! I'm conducting a short interview about people's experiences with generative AI. Would you be okay with participating in this interview?
Respondent: Yes.
AI assistant: Great! To start off, have you ever used generative AI?
Respondent: Yes.
AI assistant: That's interesting! Have you used generative AI for research purposes or qualitative data analysis?
Respondent: For both.
AI assistant: That's quite versatile! Could you share what you liked about using generative AI in your research and data analysis?
Respondent: First of al, I always like to play with new technologies. So, when generative AI came out, I had to try it. One thing I like is to experiment with it and see what comes out. I am also interested in methodological advances. When generative AI was implemented in existing tools I tested it to a) find out how it works and b) whether it is useful.
AI assistant: That's really cool! It sounds like you enjoy exploring and understanding new technologies. Can you tell me about any problems or challenges you faced while using generative AI?
Respondent: The problems are related to the way, generative AI has been implemented in some tools. According to what I found; it has been implemented in such a way that it is not very helpful for researchers. I spend quite some time to understand why it does not work and based on this explain to others why it does not work in some tools.
The other problem I have is with the promises that are made by companies developing such tools. The tools simply don't deliver what is promised. This
a.) has to do with the way generative AI has been implemented and
b.) is related to the fact that some seem to disregard that the tools currently only work well in interaction with a human being. They do not deliver instant results or shorten months of work to minutes. This is simply a lie. The challenge I experience is to figure out a good way to work with existing tools, what works already, what does not, and why.
AI assistant: I see, it sounds like you've had some frustrations with the implementation of generative AI in certain tools and the overpromising of capabilities by some companies. Could you tell me more about how you've adapted your approach to working with these tools despite these challenges?
Respondent: I am not using the functions in those tools that overpromise and underdeliver. It is as simple as that. I believe the value of generative AI for qualitative data analysis is in the interaction with it.
The best experience I have had is with a conversational approach, thus chatting with an AI bot - like we now do. I have been using a very simple tool to learn, one without lots of bells and whistles.
However, the tool allows interaction and allows the researcher to provide context information. There are still issues with context, but I could solve them by preparing the documents accordingly. This is not ideal yet, but it helps me to learn what a tool needs to look like in order for it to be useful for qualitative data analysis.
AI assistant: It sounds like you've found a way to make the most out of your interactions with generative AI by focusing on dialogue and context. Given your experiences, would you recommend using generative AI to other researchers?
Respondent: Sure. I am convinced that this is the way into the future. The tools that have been accompanying us now for the last 30 years will become less important in the future. I think there will still be some use cases, but a lot of data can be analyzed more efficiently with the help of generative AI. The tools still need to improve and mature, but I believe that we are at the brink of a paradigm shift, and in the future generative AI will be a standard companion for a qualitative researcher.
AI assistant: That's a forward-looking perspective! Now, let's talk about data security. How do you feel about data security when using AI in your research?
Respondent: It all begins with informed consent. We must carefully consider the tools we intend to employ for data collection and analysis. This involves a thorough examination of the privacy documentation associated with these tools and a critical evaluation of their trustworthiness.
Once we've made these evaluations, the next step is to transparently incorporate this information into our informed consent forms. This process raises crucial questions that demand answers. For instance: How do we handle situations where individuals are unwilling to sign the consent form? Are we exclusively engaging with respondents who are comfortable with the utilization of AI tools, or should we also include those who are more sceptical?
When we decide to include respondents with reservations, we face the decision of either conducting the analysis “in the old way” or splitting it using traditional and new AI tools. These are pivotal queries that researchers must tackle. If we feel confident in our chosen tools and secure informed consent, we are good to go.
AI assistant: It sounds like you have a thoughtful approach to data security, balancing the use of AI tools with the need for informed consent. How do you feel about the use of AI tools for tasks like transcription in research?
Respondent: I have no problem with this. I would not go back to manual transcription. Again, if a researcher plans to use an AI tool for transcription, it needs to be in the consent form. There are open-source tools that allow you to install an AI-based tool on your own computer. So, for very sceptical people or in the case of highly sensitive data, there are solutions that do not require a researcher to upload the data to a web-based service.
AI assistant: It's great to hear that you've found a balance between efficiency and data security. Looking ahead, what are your expectations for the future of AI in research?
Respondent: I embrace it. I am looking forward to not having to code data anymore and to being able to go into a dialogue with the data. This is much closer to what qualitative data analysis was meant to be. Coding was just a work-around that we needed and that was better than working with index cards and lots of stacks of paper on the floor. We are only at the beginning of how AI can be useful. Its capabilities will evolve, and I am very much looking forward to what else will become possible in the future.
Assistant: Thank you so much for your time and insights. They've been really helpful. Have a great day! !END
Conclusion
As illustrated in the example above, it's clear that AI bots won't be replacing the traditional one-hour qualitative interviews. However, this isn't the intended purpose of Qualia. Instead, its goal is to provide a more in-depth and nuanced perspective compared to the open-ended questions commonly found in surveys. With the AI bot, you have the ability to ask follow-up questions based on the respondent's answers, and this capability was effectively demonstrated.
For researchers consistently dealing with similar or related questions across multiple surveys, Qualia offers the opportunity to learn and enhance interview instructions for subsequent surveys. The effectiveness of the bot is contingent upon the quality of the instructions provided. This might help to avoid the limitations that have been mentioned above like lack of variety and inappropriate responses that will put off interviewees.
As highlighted in the section on the benefits of using an AI bot, a tool like Qualia enables researchers to conduct a greater number of interviews within a shorter time frame, covering a larger geographic area without the hindrance of language barriers— something you would not be able to in person.
If you are interested in finding out more about Qualia and testing it for yourself, visit: https://www.qualiainterviews.com/