Case studies

Enter the password to view this page.

Access granted

Esther Yang
Work About Resume Notes Contact

← Work

03

When Everything Is AI, Nothing Is

How defining what AI was opened up new ways to design it.

AI Product Governance Taxonomy

When Company C introduced Mila, an AI assistant and mascot, "AI" became an umbrella term and it got harder to tell what was actually AI. I built an attribution model separating deterministic systems from AI, then used that clarity to define how AI should look, behave, and disclose itself differently from the rest of the product.

Problem Role Diagnosis Decisions Evidence Outcomes
Product
B2B2C platform for real estate + lending · AI features
Scale
180K+ loan officers
Outcome
55% used AI features in 7 days · +38% increase in monthly users
Role
Sole Product Designer
Team
VP of Product Development · Product Manager · Engineering

Confidentiality Notice: This case study has been anonymized. Visual details, content, and identifiers have been modified to protect confidential information while preserving the underlying design challenges and decisions.

01 — Problem & context

AI was becoming an interface, not a capability

Company C serves mortgage and real estate professionals across two platforms: Platform A, a set of utility-first tools that loan officers share with clients, and Platform B, where agents and loan officers collaborate on co-branded marketing materials.

In 2025, Mila became the face of the company's AI work. Leadership's initial direction was to route the company's growing set of AI capabilities through Mila, including loan comparison reports, customer support, search and sort, onboarding, notifications, and other features that were already being considered or built.

The idea was that a single conversational interface could give the company's expanding set of AI capabilities a recognizable presence and a consistent experience. The challenge was that these features didn't all have the same interaction needs.

A loan comparison report, for example, asks someone to understand and evaluate information, while customer support or an open-ended question might benefit from a conversational exchange. Treating all of them as conversations risked adding an interaction layer simply because the capability happened to involve AI.

There was also a question of how the company could turn its AI investment into useful product capabilities quickly. Enterprise customers were increasingly evaluating what we could actually offer, while the roadmap was beginning to treat Mila as the path through which those capabilities would need to be delivered.

That was the tension I was asked to work through: how should AI actually show up across the product?

Conversation excerpt

Stakeholder

“Can we get to a point where we can have Mila say, ‘I see you’re on the Buy vs Rent report, would you like some help with this report?’ [Yes, please] [No, thank you]."

Me

“Quick clarifying question — what problem or opportunity is this aimed at? I’m also wondering how we’re thinking about when we offer help.

From a UX perspective, I’m also thinking about how to avoid a standard, Clippy-style chatbot and instead lean into light personalization. Could we use signals like time on page or repeated failed actions to tailor the ask?”

The shift From offering help based on where the user is to offering it based on what the user is doing.

02 — Role & contribution

Challenging the chatbot-first approach

Leadership's initial direction was to route everything through Mila. I challenged that approach and made the case that AI didn't need to have a single interface.

I made the case for AI to show up where it could add value within the workflow, rather than making the assistant the interface for the entire product. There was a design precedent for this: when an assistant tries to be involved in everything, users learn to work around it rather than rely on it.

I also made the business case for a different approach. AI-native features like overviews, summaries, and smart email drafts could ship in two weeks. Building the equivalent capabilities into a chatbot couldn't. That mattered because these features were also sellable; enterprise customers could see and evaluate a concrete capability much faster than they could assess an open-ended assistant.

I led the work to turn that argument into a different roadmap. Mila stayed focused on cases that genuinely benefited from conversation, while other AI capabilities shipped directly within the product. I then built the taxonomy in section 03 to give the team a consistent framework for making that distinction rather than deciding feature by feature.

03 — System thinking

Three categories, one shift

I mapped everything being attributed to Mila, or flagged for her, against what was actually happening technically and experientially. It fell into three categories.

03.A

LLM-powered

Natural language interaction, generated responses, and contextual suggestions, where an LLM was actually involved and Mila had a legitimate role.

Example: Open-ended scenario help, where the range of possible inputs is too variable to anticipate.

03.B

Automation & rules

Rule-based triggers, templated outputs, and recency-weighted search that didn't involve inference or generative output. These could be useful capabilities, but calling them AI blurred what the system was actually doing.

Example: Recency-weighted search being presented as something Mila was doing.

03.C

UX debt

Problems that needed better information architecture, clearer flows, or a straightforward design fix rather than an AI layer. Routing these problems to Mila didn't solve the underlying issue; it added another layer between the user and the thing they were trying to do.

Example: Findability problems being framed as “smart” assistance.

The third category was the most important finding. It shifted the question from how to scope Mila correctly to how to stop AI from becoming a first response for problems we haven't diagnosed.

There was a related issue with the interaction model stakeholders were proposing for Mila. Prompts like “Do you need help?” were intended to make her available throughout the product, but they were triggered without much regard for what the user was actually doing. The result was interruption based on the system's availability rather than the user's state.

I challenged that model as well. An assistant that appears regardless of context doesn't necessarily feel helpful; it can signal that the system isn't paying attention to what the user is doing.

Fig. 01 — Mila · Mapping AI dependency

Mila chat welcome screen with quick actions annotated by AI dependency: data retrieval, aggregation, financial modeling, and documentation
Most of what sat behind Mila was low-dependency retrieval and rules work. Only a few prompts needed generative AI.
Explore taxonomy →

04 — Decisions & tradeoffs

Built around the constraint

With Mila staying in the product, the work was to separate surfaces, refuse AI as a bandage, and hold a narrow definition of when chat earns its place.

Decision 01

Behavioral triggers instead of binary prompts

I replaced “Do you need help?” with signals such as time on page, repeated failed actions, and specific workflow moments. Mila now surfaces open-ended situational prompts when behavior indicates genuine friction.

Tradeoff Fewer impressions in exchange for higher relevance, since the prompt has to earn its appearance.

Decision 02

Inline insights as a separate pattern

Insights surface reasoning rather than conclusions. A predictive model identifies signals, an LLM explains them in plain language with hedging calibrated to what the model knows, and the professional decides.

Tradeoff Clearer product boundaries, but two AI surfaces to design and govern.

Decision 03

Name the UX debt; refuse the AI wrapper

For a subset of what was routed to Mila, AI wasn't the answer. Some flows were broken and needed to be fixed rather than deferred into chat.

Tradeoff Slower “AI roadmap” optics, but Mila is protected from owning problems she can't solve.

Decision 04

Mila only at the edges

We found a few places where Mila was genuinely useful: loan scenario suggestions, open-ended questions, and moments where someone was stuck. Elsewhere, putting a chatbot in front of an existing workflow just added a step. We documented those boundaries in a chatbot principles doc.

Constraint Mila stays for the enterprise narrative, but the scope narrows instead of disappearing.

Mila principles →

An AI layer on top of a broken flow doesn't fix it. It just makes the underlying problem harder to see. And harder to fix.

05 — Evidence of execution

From signal to action

Surfacing information wasn't enough; the broader question was what AI could help someone do with it. I proposed the AI summary overview, suggested talking points, and AI-written emails as ways to turn the information we were surfacing into something people could actually use.

The overview gives LOs and agents a quick read on buyer activity, behavioral signals, and relevant external factors, with an explanation of why those signals matter. Talking points give them a starting place for cold outreach, and the email flow turns those talking points into a draft they can use.

Fig. 02 — Property Insights · AI overview to smart email

Property Insights AI Buyer Overview for Linda Carver beside a Smart Email draft generated from her activity
From signals and talking points to a draft the LO can send — reasoning stays visible, judgment stays with the professional.

Early results · AI-written email

  1. 55%

    Used the drafted email within 7 days

  2. +38%

    Monthly unique users, June → August

  3. 728

    Unique users in ~3.5 months

Feature launched in June. Monthly unique users grew from 219 → 271 → 302. Another 226 users copied the draft directly, generating 1,287 copy actions.

These became examples of the approach I was advocating for: AI capabilities embedded in the workflow, each designed around a specific job rather than routed through Mila simply because it involved AI.

The goal wasn't simply to add AI to the workflow. I was trying to establish where AI could genuinely help someone do their job without taking over the judgment that belongs to them. That meant making the reasoning visible, being careful about how confidently we phrased insights, and clearly labeling AI-generated content.

06 — Outcomes

Standards that shape product decisions

The attribution standards became a working tool for product decisions across both products. They give the team a shared way to decide when something should be presented as AI, when it shouldn't, and what level of confidence we need before putting it in front of a customer.

Mila still exists, but her role is much more defined. We have clearer guardrails around where she appears, what she's responsible for, and what needs to happen before an AI interaction is introduced.

More importantly, the framework is showing up in the work itself. On Platform A, for example, we're using the same lens to evaluate where AI-generated insights should live. We haven't settled that question yet, but that's part of the point: the framework gives us a way to work through the decision instead of assuming that a pattern that worked on Platform B should carry over.

That was the bigger outcome of the project. What started as a question about how we label AI became a way to make better product decisions about AI—what belongs in the product, where it belongs, and where we should leave it out.

For me, that's the important shift. The work wasn't about making Mila a better character. It gave the team a clearer set of criteria for deciding what we were willing to ask customers to trust.

← Previous Turning “Is it working?” into “Who should I call?” Next → Less AI, Doing More

Ready to build something  Ready to build something lasting?

hello@estheryang.is

LinkedIn