Channel: Sean's Stories | Published: 2025-11-28
Learning Notes
This video by Sean Chen shares 8 essential principles for building effective AI agent systems, based on feedback from AI founders, investors, and customers.
1. Smart Defaults over Infinite Customization
Instead of giving users unlimited customization options, provide smart defaults based on domain expertise. Users are not prompt engineers—wrong inputs can degrade performance.
Example: In a legal AI like Harvey, an immigration law expert with 10 years of experience sets the default prompt for green card applications, preventing beginners from making errors.
2. Meet Users Where They Are
Don't force users to change their existing workflows. Integrate AI into tools they already use (Notion, Google Sheets, etc.).
Example: If a startup uses Google Sheets, design your AI agent to work within Sheets rather than forcing them to adopt a new dashboard.
3. Vertical-First Strategy
Focus on a specific industry first to build high-quality examples that reduce the model's "foolish behavior." LLMs have a Mixture of Experts (MoE) architecture—specialize as one expert.
Example: Provide 20-30 domain-specific examples (e.g., JSON file processing) to your chatbot agent. Expand horizontally after proving success.
4. Examples Must Be Carefully Balanced
Too few examples make the model think narrowly; too many or wrong examples cause overfitting. Structure examples naturally and in balance.
5. Write Workflows as Explicit Natural-Language Specs
Minimize hallucination and increase predictability with step-by-step natural language instructions. Explanations work better than just throwing JSON.
Example: "Step 1: Check ABCD. In this case, use Stripe API. If authorized user, fetch data from Supabase."
6. Add Human-in-the-Loop Correction Loops
Add approval/rejection cycles for micro-training improvements. Especially useful in customer support (CX).
Example: Sierra AI and DoorDash trigger human intervention when dealing with angry customers. Good/bad examples improve future performance.
7. Give Your Agent a Single Memory Layer
In multi-agent systems, maintain shared context (chat history, highlights, calendar, etc.) in a single layer for consistency.
Example: In Notion AI, a paragraph polishing agent and a summary agent share memory access. Email agents reference previous calendar events.
8. Add Continuous Evals to Measure and Enforce Reliability
Agents regress easily, so maintain stability with automatic evaluations for workflow scenarios and edge cases.
Example: Before sending an email, verify subject, content, signature, and previous context. When Claude models update, verify performance with evals.
Key Takeaways
- AI agents should be example-rich, self-improving through human feedback, simple for customers, and verbose with natural language prompts
- Evals and Memory are the most important moats for AI products
- LLMs have superpowers; agent failures are prompt failures
- Focus vertically first, then expand after proving market fit
Simple Explanation
Think of an AI agent like a robot friend helping you run a pizza shop:
- Smart Defaults = Robot already knows a delicious pizza recipe
- Meet Users = Robot works with your existing toy box (tools)
- Vertical First = Master cheese pizza before trying all types
- Balanced Examples = Right amount of recipe examples
- Natural Language = Tell robot step-by-step like talking to a friend
- Human Loop = You correct weird pizzas
- Single Memory = Robot keeps one notebook for everything
- Continuous Evals = Daily taste tests to stay perfect