Teaching Ava what she doesn't know she doesn't know

Bottom line, up front

Ava was only as useful as what she knew about the team using her. I designed a training experience that let sales teams teach Ava their organizational knowledge, and learned that transparency into what she knows matters as much as the training itself.

Ava is an AI agent for sales engineering and account executive teams. She connects to the tools these teams already use - Salesforce, Slack, Google Drive, Gong, and the web - and works autonomously across them to handle the research, preparation, and context-gathering that surrounds every deal.

Problem

Ava was smart, but she didn't know anything about your company specifically

Out of the box, Ava had general knowledge but no context about the team she was working with. She didn't know how you positioned against competitors. She didn't know what objections your sales engineers heard most often. She didn't know what a winning deal looked like for your specific team.

That's like hiring a smart new employee and never onboarding them. Useful in a generic way. Not yet the expert teammate she was supposed to be.

“I can’t just hand Ava our internal docs and hope she figures it out. How do I know what she understands? How do I know what she’s missing?”

Discovery

The knowledge existed. It just wasn't going anywhere useful.

11 out of 12 sales engineering leaders said insights from past deals weren't making it into new ones. Teams kept solving the same problems from scratch. Internal wikis, Slack threads, call recordings, shared docs - the knowledge was there, just scattered and hard to put in front of Ava in any systematic way.

And sales engineers were skeptical. They wanted to know what Ava had learned, where she got it, and how confident she was before they'd trust her output in a deal.

Key insights

Similar process, different context. Every sales engineering team followed a comparable deal lifecycle, but the specific knowledge that made a team effective (how they handled objections, what competitors they went up against, what their best customers had in common) varied dramatically from company to company. Ava needed to learn that context, not just general sales knowledge.

AI skepticism was real. Most sales engineers preferred traditional UI elements over AI-first interactions, expressing concern about reliability, control, and learning curve. Any training experience that felt opaque or unpredictable would lose users before they ever saw the value.

Knowledge exists but isn’t captured systematically. Sales engineers had internal wikis, shared documents, call recordings, and Slack conversations, but this knowledge was scattered and not easily accessible. They needed Ava to learn from existing materials without requiring them to restructure everything first.

Trust required transparency. Users were skeptical of AI black boxes. They needed to see what Ava knew, where her knowledge came from, and how confident she was in different areas. Without that visibility, they wouldn’t trust her output in deals.

Methods: User Interviews

Design

Three approaches, one core question: how do you know she actually understands?

Training Ava wasn't like configuring a tool. It was closer to onboarding a new hire. The challenge across every direction I explored was the same: how does a user know what Ava learned, and how do they know when she got something wrong?

Q&A testing let users upload documents and ask Ava questions to check her understanding. The problem was it put all the burden on the user. They didn't know what to ask, how thorough to be, or when they were done. No sense of progress, no visibility into gaps unless you kept probing.

“I don’t want to play 20 questions with an AI. Just tell me what you know and what you need.”

Topics and questions flipped the dynamic. Ava surfaced her own questions across organized topic areas like product, competitors, and customers. Users responded to specific gaps instead of trying to anticipate them. Testing showed users felt much more in control. The mental model made sense: here's what she knows, here's what she still needs.

“This actually makes sense. I can see she knows our product really well but needs more on competitive positioning. That’s exactly what I’d expect at this stage.”

Artifact generation had Ava generate summaries of what she learned, which users could review and edit directly. Users loved the idea but wanted to click into the artifact and change things inline, not ask Ava to fix it through conversation. We needed more time to figure out the right feedback loop before committing to the more complex implementation.

We shipped the simpler approach on purpose, to answer the most important question first

Q&A launched as the MVP. Not because it was the best experience, but because it let us answer the question that mattered most: can Ava actually ingest organizational knowledge and use it meaningfully in deals? If that didn't work, nothing else would matter.

It worked. Users who put in the time saw real improvements in how Ava performed. She became a better teammate with organizational context to draw on.

But most users didn't put in the time. Without visibility into what Ava knew or where the gaps were, there was no way to gauge progress. The black box problem from research turned out to be exactly as damaging as expected.

How Ava learns

Defining how Ava thinks about knowledge became a tool the whole team used

Working across three design approaches surfaced a problem that went beyond the interface. Design, engineering, and data science were using different mental models to describe how Ava learned, which made cross-functional conversations slow and imprecise.

I worked with the data scientists to define Ava's states of knowledge and learning modes as a shared framework. Unknown meant a gap Ava would proactively ask users to fill. Known and unvalidated meant information she'd learned but hadn't confirmed with a person yet. Known and validated meant something a user had confirmed accurate, and the only state she'd draw on without checking first.

Having this framework meant we could ask specific questions about any new feature: does this produce validated or unvalidated knowledge? Does it trigger independent or dependent learning? That precision made the whole team faster.

Impact

The capability is proven. The experience needs to catch up.

The Q&A approach answered the question that mattered most: can Ava actually learn organizational knowledge and use it in deals? It can. But without visibility into what she knows, most users never invested enough to see it.

Artifact generation fixes that. Users get something concrete to react to instead of something invisible to probe. That's what ships next.