A practitioner's guide to navigating the moral complexities of AI-powered design, from the frontlines of healthcare technology
Three months ago, I watched a veteran struggle with our VA healthcare portal. The AI-powered "smart scheduling" system kept recommending appointment times that conflicted with his dialysis schedule. Each suggestion felt helpful to the algorithm, but harmful to the human.
The veteran grew frustrated, then angry, then resigned. He closed his laptop and said he'd just walk into the clinic the next day and wait. The system had been optimized for efficiency, but it had failed at empathy.
This moment crystallized something I'd been grappling with throughout my career in government health tech: in our rush to make interfaces smarter, we risk making them less human. As AI becomes the invisible hand guiding user behavior, we face a fundamental question: How do we wield this power responsibly?
The stakes couldn't be higher. Netflix reports that more than 80% of content viewed on their platform is discovered through personalized recommendations, while shareholders are now demanding transparency reports about AI use and ethical guidelines. Meanwhile, Amazon discontinued use of a recruiting algorithm after discovering it exhibited systematic gender bias, and AI-generated music artists with over a million monthly listeners are flooding Spotify without users' knowledge.
The New Reality: Every Interface is an Influence Engine
Today's digital environments don't just present information; they also facilitate interaction. They shape behavior. Every recommendation, every default setting, every personalized interface element represents a choice about how technology should influence human decisions.
Consider your morning routine. Your phone's AI decides which notifications to show first. Your coffee app suggests your "usual" order based on past behavior. Your calendar automatically accepts meeting invitations that fit your patterns. Your news feed curates information based on your engagement history.
Each interaction may seem small, but collectively, they guide your attention, shape your preferences, and influence your choices in ways you might not consciously recognize. You're not just using these interfaces—they're using behavioral science to shape your behavior.
This transformation has happened gradually, but its implications are profound. We've moved from interfaces that respond to user intent to interfaces that anticipate and influence user intent. The question isn't whether this influence exists—it's whether we're wielding it ethically.
The Nudge-to-Manipulation Pipeline
The concept of "nudging" emerged from behavioral economics as a benevolent approach to influence. The idea is simple: small design changes can steer people toward better decisions without restricting their freedom. Auto-enrolling employees in retirement savings plans, for example, dramatically increases participation rates while preserving the option to opt out.
But in practice, the line between helpful nudging and harmful manipulation often blurs. What starts as user-centered design can evolve into user-exploitative design through a predictable progression:
Stage 1: Helpful Defaults The system suggests genuinely beneficial choices. A health app recommends tracking your daily steps. A banking app suggests setting up automatic savings. These nudges align with user goals and improve outcomes.
Stage 2: Optimization Pressure. Business metrics become the primary driver. The health app starts recommending premium features. The banking app suggests higher-fee investment products. The nudges still provide some user value, but they increasingly serve business objectives.
Stage 3: Engagement Maximization The system optimizes for user attention and time-on-platform. The health app sends more frequent notifications to maintain engagement. The banking app uses fear-based messaging to encourage product adoption. User well-being becomes secondary to engagement metrics.
Stage 4: Exploitation. The system actively works against user interests. The health app makes it difficult to cancel subscriptions. The banking app uses dark patterns to trick users into expensive services. The nudges have become manipulative traps.
This progression isn't inevitable, but it's common. Once organizations start optimizing for engagement over well-being, the slide toward manipulation becomes almost automatic. The key is recognizing when you're moving from Stage 1 to Stage 2 and having the courage to resist that progression.
