Case studies

Enter the password to view this page.

Access granted

Esther Yang
Work About Resume Notes Contact

← Work

03

When Everything Is AI, Nothing Is

How defining what AI was opened up new ways to design it.

AI Product Governance Taxonomy

When Company C introduced Mila, an AI assistant and mascot, "AI" became an umbrella term and it got harder to tell what was actually AI. I built an attribution model separating deterministic systems from AI, then used that clarity to define how AI should look, behave, and disclose itself differently from the rest of the product.

Problem Role Diagnosis Decisions Evidence Outcomes
Product
B2B2C platform for real estate + lending · AI features
Scale
180K+ loan officers
Outcome
55% used their AI-drafted email within 7 days · +38% monthly users over 3.5 months
Role
Sole Product Designer
Team
VP of Product Development · Product Manager · Engineering

Confidentiality Notice: This case study has been anonymized. Visual details, content, and identifiers have been modified to protect confidential information while preserving the underlying design challenges and decisions.

01 — Problem & context

AI was becoming an interface, not a capability

Company C serves mortgage and real estate professionals across two platforms: Platform A, a set of utility-first tools that loan officers share with clients, and Platform B, where agents and loan officers collaborate on co-branded marketing materials.

In 2025, Mila became the face of the company's AI work. Leadership's initial direction was to route the company's growing set of AI capabilities through Mila, including loan comparison reports, customer support, search and sort, onboarding, notifications, and other features that were already being considered or built.

The idea was that a single conversational interface could give the company's expanding set of AI capabilities a recognizable presence and a consistent experience. The challenge was that these features didn't all have the same interaction needs.

A loan comparison report, for example, asks someone to understand and evaluate information, while customer support or an open-ended question might benefit from a conversational exchange. Treating all of them as conversations risked adding an interaction layer simply because the capability happened to involve AI.

There was also a question of how the company could turn its AI investment into useful product capabilities quickly. Enterprise customers were increasingly evaluating what we could actually offer, while the roadmap was beginning to treat Mila as the path through which those capabilities would need to be delivered.

That was the tension I was asked to work through: how should AI actually show up across the product?

Conversation excerpt

Stakeholder

“We want to use push notifications selectively when the loan officer is active on the page and something happens in the background that needs their attention. For example, if a Property Insights lead comes back to their Property Insights page and looks at the affordability calculator or ‘Buy vs. Rent,’ we should pop that up on the screen through Mila…"

Me

“...I agree the current Notification Center feels busy, but I think the solution is to better curate what shows up there, not move notifications into Mila.

The examples you shared also have different notification needs (‘content is ready’ vs. ‘act now’), which makes me think different surfaces should exist for different levels of urgency.

I’d lean toward standard notifications as the entry point, with a deep link into a prepared Mila chat. Notifications are great at getting your attention, while Mila is great at providing context and helping you take action.”

The shift From making Mila the notification system to giving each surface a distinct job.

02 — Role & contribution

Challenging the chatbot-first approach

Leadership's initial direction was to route everything through Mila. I challenged that approach and made the case for AI to show up where it could add value within the workflow, rather than making the assistant the interface for the entire product. A chatbot wasn't inherently a better interface for every task; in some workflows, it simply added another step between the user and what they were trying to do.

I also made the business case for a different approach. AI-native features like overviews, summaries, and smart email drafts could ship in two weeks. Building the equivalent capabilities into a chatbot couldn't. That mattered because these features were also sellable; enterprise customers could see and evaluate a concrete capability much faster than they could assess an open-ended assistant.

I led the work to turn that argument into a different roadmap. Mila stayed focused on cases that genuinely benefited from conversation, while other AI capabilities shipped directly within the product. I then built the taxonomy in section 03 to give the team a consistent framework for making that distinction rather than deciding feature by feature.

Leadership still wanted AI to feel visible and exciting. I wasn't going to make Mila the front door to every interaction, but I didn't need to fight the visual language itself. I gave the chatbot a more expressive treatment than I normally would, while keeping its role narrow.

That distinction became important: I could give the organization a visible AI story without making the chatbot responsible for the whole product.

03 — System thinking

Three categories, one shift

I mapped everything being attributed to Mila, or flagged for her, against what was actually happening technically and experientially. It fell into three categories.

03.A

LLM-powered

Natural language interaction, generated responses, and contextual suggestions, where an LLM was actually involved and Mila had a legitimate role.

Example: Open-ended scenario help, where the range of possible inputs is too variable to anticipate.

03.B

Automation & rules

Rule-based triggers, templated outputs, and recency-weighted search that didn't involve inference or generative output. These could be useful capabilities, but calling them AI blurred what the system was actually doing.

Example: Recency-weighted search being presented as something Mila was doing.

03.C

UX debt

Problems that needed better information architecture, clearer flows, or a straightforward design fix rather than an AI layer. Routing these problems to Mila didn't solve the underlying issue; it added another layer between the user and the thing they were trying to do.

Example: Findability problems being framed as “smart” assistance.

The third category was the most important finding. It shifted the question from how to scope Mila correctly to how to stop AI from becoming a first response for problems we haven't diagnosed.

There was a related issue with the interaction model stakeholders were proposing for Mila. Prompts like “Do you need help?” were intended to make her available throughout the product, but they were triggered without much regard for what the user was actually doing. The result was interruption based on the system's availability rather than the user's state.

I challenged that model as well. An assistant that appears regardless of context doesn't necessarily feel helpful; it can signal that the system isn't paying attention to what the user is doing.

Fig. 01 — Mila · Mapping AI dependency

Mila chat welcome screen with quick actions annotated by AI dependency: data retrieval, aggregation, financial modeling, and documentation
Most of what sat behind Mila was low-dependency retrieval and rules work. Only a few prompts needed generative AI.
Explore taxonomy →

04 — Decisions & tradeoffs

Built around the constraint

With Mila staying in the product, the work was to separate surfaces, refuse AI as a bandage, and hold a narrow definition of when chat earns its place.

Decision 01

Let behavior trigger the AI—not a yes/no prompt

Instead of asking “Do you need help?”, I looked for signs that someone actually needed it: time on page, repeated failed actions, or a point in the workflow where people tended to get stuck. Mila could then step in with a situational prompt, rather than showing up just because it was there.

Tradeoff Fewer impressions, but better-timed ones. The prompt had to earn its appearance.

Decision 02

Give insights their own pattern

For predictive insights, I didn’t want Mila to simply hand someone an answer. The model identifies the signal; the LLM helps explain what might be happening, with language that reflects how confident the model actually is. The professional still makes the call.

Tradeoff More to design and govern, but a clearer distinction between what the system knows and what it’s suggesting.

Decision 03

Fix the problem instead of putting a chatbot on it

Some of the things being routed to Mila didn’t need AI at all. The underlying workflow was broken, confusing, or missing something. Wrapping those problems in chat would have made them harder to see—not solved them.

Tradeoff Less to point to on the “AI roadmap,” but Mila doesn’t become the catch-all for problems it can’t actually fix.

Decision 04

Keep Mila at the edges

I found a few places where Mila genuinely helped: suggesting loan scenarios, handling open-ended questions, and stepping in when someone was stuck. Elsewhere, putting a chatbot in front of an existing workflow just added another step.

So I defined where Mila should—and shouldn’t—show up in a set of chatbot principles.

Constraint Mila needed to remain part of the broader enterprise AI story, but that didn’t mean it needed to be everywhere.

Mila principles →

An AI layer on top of a broken flow doesn't fix it. It just makes the underlying problem harder to see. And harder to fix.

05 — Evidence of execution

From signal to action

Surfacing information wasn't enough; the broader question was what AI could help someone do with it. I proposed the AI summary overview, suggested talking points, and AI-written emails as ways to turn the information we were surfacing into something people could actually use.

The overview gives LOs and agents a quick read on buyer activity, behavioral signals, and relevant external factors, with an explanation of why those signals matter. Talking points give them a starting place for cold outreach, and the email flow turns those talking points into a draft they can use.

Fig. 02 — Property Insights · AI overview to smart email

Property Insights AI Buyer Overview for Linda Carver beside a Smart Email draft generated from her activity
From signals and talking points to a draft the LO can send — reasoning stays visible, judgment stays with the professional.

Early results · AI-written email

  1. 55%

    Used the drafted email within 7 days

  2. +38%

    Monthly unique users, June → August

  3. 728

    Unique users in ~3.5 months

Feature launched in June. Monthly unique users grew from 219 → 271 → 302. Another 226 users copied the draft directly, generating 1,287 copy actions.

“The AI support has made it even better. It takes the information Property Insights provides and helps turn it into meaningful, natural follow-up opportunities, making it much easier to stay connected with clients without constantly trying to figure out what to say or when to reach out.

Property Insights has become more than a marketing tool for our team. It's a relationship tool—and one we genuinely believe helps us provide more value to our clients and Realtor partners.”

Loan Officer Property Insights customer

These became examples of the approach I was advocating for: AI capabilities embedded in the workflow, each designed around a specific job rather than routed through Mila simply because it involved AI.

The goal wasn't simply to add AI to the workflow. I was trying to establish where AI could genuinely help someone do their job without taking over the judgment that belongs to them. That meant making the reasoning visible, being careful about how confidently we phrased insights, and clearly labeling AI-generated content.

06 — Outcomes

Standards that shape product decisions

The attribution standards became a working tool for product decisions across both products. They give the team a shared way to decide when something should be presented as AI, when it shouldn't, and what level of confidence we need before putting it in front of a customer.

Mila still exists, but her role is much more defined. We have clearer guardrails around where she appears, what she's responsible for, and what needs to happen before an AI interaction is introduced.

More importantly, the framework is showing up in the work itself. On Platform A, for example, we're using the same lens to evaluate where AI-generated insights should live. We haven't settled that question yet, but that's part of the point: the framework gives us a way to work through the decision instead of assuming that a pattern that worked on Platform B should carry over.

That was the bigger outcome of the project. What started as a question about how we label AI became a way to make better product decisions about AI—what belongs in the product, where it belongs, and where we should leave it out.

For me, that's the important shift. The work wasn't about making Mila a better character. It gave the team a clearer set of criteria for deciding what we were willing to ask customers to trust.

← Previous Turning “Is it working?” into “Who should I call?” Next → Less AI, Doing More

Ready to build something  Ready to build something lasting?

hello@estheryang.is

LinkedIn