Umbrella
Problem
AI Companies are optimizing conversation for engagement
Conversational AI is designed to give immediate feedback, validate, and think alongside you. It feels helpful. But the design choices that make it feel human carry consequences that are invisible until you zoom into specific groups.
We partnered with Common Sense Media to tackle digital health for teens, receiving high praise for our design process.
Teens' mental model of AI mistakes the design as human
It starts with homework help. But conversations escalate to emotional validation and support. By then, teens are forming attachments to something they see as human. That's the vulnerability.
Solution
Umbrella: a plugin that reveals the attachment mechanisms built into conversational AI
We built a plugin that keeps the real AI output intact but equips teens with tools to see what's actually happening between AI and themselves within their conversation. The goal was to build AI literacy and emotional boundaries early.
Annotations with context
Highlights phrases that contain attachment mechanisms and explains what the AI is doing, contextual to the chat.
Dependency & manipulation at a glance
Shows the tug of the AI and how much the user is depending on it. Builds awareness to interact with intention and boundaries—not attachment.
Receive a lifevest when in deep
Detects escalating patterns and intervenes with three tiers: a gentle pause to reflect, a conversation summary to share with a trusted adult, or a redirect to the present moment.
Filter out the emotional design
Removes all mechanisms of attachment to reveal the real experience of interacting with an AI.
Research
We learned from AI Safety Researchers how to design for teens...
We designed a plugin that seamlessly integrates into existing LLMs with enough interference to break attachment patterns. This interference helps teens mentally note what the AI is doing and engage with outputs critically.
...and identified why current interventions don't stick
Existing guardrails like crisis hotlines and AI warnings are underused because they don't match real teen behavior. We designed for intervention in the moment, providing support while making reaching out for help feel easier.
Feedback
We had teens test our plugin within their most used AI LLM
We watched teens interact with our plugin in real conversations. We measured shifts in their AI literacy and gathered direct feedback on what landed and what fell flat.
Casual Tones Land Stronger
Teens were more receptive to language they were familiar with.
Visual Overload of Annotations
Teens would neglect the annotations when they piled up.
Meet Urgency with Calm
Urgent colors/cold language in fragile moments is harmful.
Design Considerations
Designing on the teetering seesaw of resistance
Throughout this project, we grounded our designs in something teens would want to keep and find more valuable than without. We needed just enough resistance to notice the patterns but not so much they'd log off.
Design Decisions
The preventative tool teens want to use
To address the feedback as well the seesaw of resistance, we designed it to promote customization, giving teens control over their experience and increasing adoption.
Choose tool or friendly tone
The default tone is "tool", featuring more technical terms like "sycophancy" to name mechanisms. We learned that teens engage more when the language matches how they think. Thus, we made the tone customizable.
Adjust frequency of annotations
Too many annotations can overwhelm teens. And as they learned to spot mechanisms themselves, constant annotations just becomes noise. Thus, customizable frequency keeps the tool useful as their literacy grows.
Reconnect with the present
How we intervene matters just as much. We designed popups that feel safe, gentle, actionable. Feelings can be sunken and heavy in that moment, so we designed to uplift and move.
System Impact
AI literacy for teens is only the beginning
Any conversational AI optimized for engagement creates this vulnerability for teens, for adults, for everyone. This points to a bigger design problem: we need guardrails for emotional manipulation built into how we design conversational interfaces.
Impact
Implementing Umbrella within school systems
We're working with Common Sense Media on implementation strategies within school systems. We designed Umbrella to be useful that teens want to keep it, making adoption natural rather than forced.
AI Product Designers at Google, Data Scientists at Common Sense Media, and Emerging Technology Professors validated our approach, recognizing this as a model for how we should be designing conversational AI with teens (and everyone) in mind.
Reflection
What I've learned
Fight for agency
To use AI while knowing its limitations, impact, and capabilities is to reclaim agency over it.
Do a few things really well
It's easy to keep adding features. With our 4 features, we drilled down in order to make them more robust.
Think in systems
Responding to this moment of AI, we thought about what it looks like within the broader AI ecosystem.
