using ai to interpret user research

The university was preparing to roll out enterprise-grade AI tools to faculty and staff. Unlike a typical software launch, this one carried significant baggage: ethical concerns, confusion about data privacy, and skepticism from a research community that values intellectual rigor. Our task was to analyze early pilot feedback and develop a communication strategy that could build genuine trust. We decided to use AI to assist with analyzing the user research.

We first asked the AI platform to find common themes and patterns, using survey data and help desk tickets We then asked it to run a sentiment analysis, find out what the biggest issues and pain points were, and tell us what the common use cases were for staff and faculty.

Our Findings

We found that staff sentiment was generally positive but cautious. Faculty were more divided in sentiment, with some being enthusiastic adopters and others having ethical concerns. This distinction became the foundation of the communication strategy. The tools meant different things to different audiences, and the messaging had to reflect that.

Key points for communication

1. Emphasize security and data privacy

There was a lot of confusion about how the licensed university tools differed from the consumer versions. The primary motivation for users was to have a verified, secure alternative to consumer-grade AI.

Communication points:

  • Explicitly state that content is not used to train AI.  

  • Position the tools as a "closed system" that is safer for university business than public models.  

  • Provide clear guidance on what types of data should not be used with these tools.  

2. Lead with Responsible use and ethics

Addressing the ethical and environmental impacts of AI was essential for community buy-in, particularly among faculty.

Communication points:

  • Acknowledge Ethical Concerns: Proactively address concerns regarding environmental impact, algorithmic bias, and the potential for plagiarism.  

  • Equitable Access: Frame the rollout as a way to support equity, ensuring all staff have access to professional-grade tools regardless of their department's budget.  

  • Academic Integrity: For faculty, focus on how these tools can be "instructional partners" while maintaining intellectual development.  

3. Provide clear use cases and guidance

Early feedback showed that users needed help understanding which tool to use when. 

Communication points:

  • Be clear which tool is best for drafting content, brainstorming, and general conversational tasks, and which is best for "grounding" responses in specific source materials and summarize large, complex documents.  

  • Be transparent about known issues, such as hallucinations and the need for independent fact-checking, especially for web research and data analysis.  

Outcomes & Impact

The framework gave the rollout team a clear communication message. One that was grounded in ethics and knowledge. By grounding the communication strategy with actual user feedback rather than feature lists, we were positioned to build trust. And asking AI to analylze sentiment about AI was very meta.

AI greatly accelerated the timeframe it took us to do user research. We were able to analylze survey results in days instead of weeks. However, this didn’t save us humans from reading all the comments ourselves. AI can be a really useful tool, but it shouldn’t replace the human brain, especially when coming to conclusions about fellow humans.