Self-Improving Loops: Agents That Learn From Outcomes
Track outcomes, mine the wins, update the strategy, guard the update. RL's shape without the gradients.
A self-improving loop lets Mailmind automatically refine its email drafting and meeting scheduling by tracking what worked, comparing outcomes to targets, then updating the prompts and few-shot examples it uses. This cycle of action, outcome capture, evaluation, and guarded strategy update gives agents an RL-shaped feedback loop without gradient descent.
Mailmind doesn't need a reinforcement learning algorithm. The same natural language prompts that control it also serve as the policy representation, so improvements are just edited text. You can open the updated prompt and see exactly why behavior changed.
After part 8 tackled context windows that never fit, we turn to the question of improvement: once the agent acts, how does it get better without human intervention?
Why does an agent need to look back at what it did?
Think of a salesperson who sends out a hundred emails every week. They keep a private note of which subject lines got replies. When they notice that “Quick question about [topic]” works 3× better than “Following up,” they switch. The next batch performs better. No expensive training run. Just a human reading the receipts and adjusting the playbook.
Mailmind does the same job mechanically. It drafts replies and meeting proposals every day. Some drafts the user sends without editing. Some get heavy rewrites. Some meeting slots get accepted within minutes; others sit unanswered. Those outcomes are the only signal Mailmind needs to tune its own behavior. The loop is not about backpropagating through a loss function. It's about mining the best-performing outputs, finding what they have in common, and making the next prompt a little more like them.
What are the four stages of Mailmind's self-improving loop?
Every pass through the loop has the same shape.
Action. Mailmind performs a task: draft a reply to a vendor, propose three meeting slots to a colleague, file a receipt. The action leaves a trace, the generated text and the context in which it was produced (recipient type, time of day, urgency signal).
Outcome tracking. For a draft, the outcome is the edit distance between what Mailmind wrote and what the user actually sent. A low edit distance means the draft was close to what the user wanted. For a meeting slot, the outcome is binary (accepted or declined) plus time-to-accept. These metrics are stored with tags that allow attribution: which prompt produced this draft, which example bank was active, which slot-selection rule was used.
Evaluation. Weekly, Mailmind pools all outcomes. It compares actual edit distances against an acceptance threshold (say, under 20 % of tokens changed). It also looks at meeting acceptance rates against a target (say, >70 % same-day acceptance). The wins are then mined: for high-quality drafts, what tone, length, and structure do they share? Are they grouped by recipient type? The evaluation surfaces patterns the previous prompt didn't explicitly capture.
Strategy update. The findings become concrete changes. The drafting prompt gets a new line: “For external vendors, keep the tone direct and under 120 words.” A few winning drafts become new few-shot examples. The slot-selection heuristic might be tweaked to prefer Tuesday 10 AM slots for timezone-sensitive recipients. These changes are versioned, recorded, and applied gradually.
How do you design a feedback loop that doesn't accidentally make things worse?
Self-modifying prompts are dangerous if the loop isn't careful. Four design rules keep Mailmind from drifting.
Explicit outcome capture. You can't improve what you don't measure. Mailmind always logs the exact output, the user action (sent as-is, rewritten, ignored), and the timestamp. For meeting proposals, it tracks the invitee, timezone, proposed slots, and acceptance status. Without this granularity, attribution is guesswork.
Attribution. Knowing which prompt version and example bank generated which output is critical. Mailmind tags every action with a strategy version ID. When evaluation later finds a drop in acceptance rate, the operator can instantly see which update caused it and roll back that single version.
Lag awareness. Outcomes take time. A meeting slot proposed on Friday might be accepted on Monday. Mailmind's evaluation windows are configurable: for meetings, a 72-hour wait before scoring. The loop never updates on an incomplete window. It also never punishes a proposal that gets a slow acceptance, it just records the latency and uses it to adjust slot-selection timing rules.
Guardrails on self-modification. Every update gets gated.
- Minimum sample size: no update fires with fewer than 100 observations for a given prompt template.
- Maximum change per iteration: the updated prompt may differ by no more than 20 % of the token count from the previous version. This prevents wild swings from anomalous batches.
- Human approval: if the evaluated delta exceeds 50 %, the update is flagged and held for review.
- Versioned strategies with rollback: every prompt and example bank change lives in a registry. Rollback is a single API (application programming interface) call.
- Staged rollout: new strategies go to 10 % of traffic first. If the outcome metrics hold or improve over 48 hours, they expand.
What failure patterns break self-improving loops in production?
Even with guardrails, loops can misbehave.
Metric drift. Optimize for opens and Mailmind learns to write clickbait subject lines that annoy recipients. The loop must optimize for the user's true goal (low-effort replies, accepted meeting slots, zero manual rewrites). Mailmind ties its primary metric to edit distance on drafts, not cosmetic signals like open rate.
Distribution shift. The user's email mix changes, suddenly 60 % of incoming messages are from a new client's French-speaking team. The loop keeps reinforcing patterns from the old English-dominant distribution. Mailmind runs separate loops per language segment and flags when segment proportions shift by more than 30 % over a month. It then re-evaluates whether the existing examples still apply.
Overfitting. All drafts start sounding identical: same opener, same structure, same sign-off. The diversity of the original training data gets squeezed out. Mailmind adds a diversity constraint: each updated example bank must include at least three distinct tones or lengths. A separate “freshness” score triggers re-seeding from the original base prompt if all drafts cluster too tightly.
Lag corruption. A batch of meeting proposals for a holiday period shows low acceptance. The loop interprets that as a bad strategy and downgrades the slot-selection rule, when the real cause was calendar absence. Mailmind marks known slow-response periods (company-wide holidays, weekends) and either excludes those windows or uses them to train a separate “seasonal” rule.
How does Mailmind handle rare email types when most feedback is from the common ones?
The loop naturally gets great at the majority cases (quick replies to internal colleagues) and gradually worse at rare ones (refund disputes in German). The feedback signal is dominated by common instances, so the strategy update slowly forgets what the minority needed.
Mailmind tackles this with three techniques. First, it stratifies outcomes: each email type (by topic label and language) gets its own mini-feedback pool. The update step weights minority-type feedback higher, a single German refund draft that the user sent unchanged counts as much as 20 regular internal drafts. Second, it keeps separate example banks per type, so the “refund-German” bank is never diluted by the “status-update-English” bank. Third, it seeds rare types manually: a human provides 5 to 10 high-quality examples that live as a fixed, protected bank. The loop can add to it but never remove or down-weight those seeds. This stops extinction.
How does this compare to reinforcement learning?
The shape is identical: act, receive reward, update policy. But instead of numerical rewards flowing through a gradient update over a neural network's weights, Mailmind's “policy” is a set of natural-language prompts and examples. The “update” is a text edit, grounded in human-interpretable evaluation rules.
That difference matters in production. You can read the updated prompt and understand the exact rule change: “We added a line that tells the model to ask for confirmation before proposing a slot to external attendees.” In a gradient-based system, you can't audit a weight change the same way. The RL loop is opaque; the self-improving loop is a decision record. For email drafting, where tone errors can damage relationships, auditability is non-negotiable.
- Numerical reward signals
- Gradient updates over neural network weights
- Policy is a black box
- Hard to interpret rule changes
- Natural language feedback
- Prompt edits based on outcome analysis
- Policy is a readable prompt
- Exact rule change is visible
| Property | Value |
|---|---|
| Minimum sample size before update | 100 observations per template |
| Maximum prompt token delta per iteration | 20 % |
| Human approval threshold | above 50 % expected change in outcome metric |
| Staged rollout initial traffic | 10 % |
| Rollback time | instant (versioned prompt registry) |
| Rare-type boost weight | up to 20× per instance |
| Meeting outcome window | 72 hours before scoring |
Frequently Asked Questions
Q: Can Mailmind write its own prompt improvements, or does a human still do it? Mailmind's evaluation step produces a structured diff: it identifies that drafts under 120 words got 30 % less editing, and that the phrase “let me know if that works” decreased acceptance. The actual prompt edit is generated by a carefully constrained “strategy writer” agent, but the final update is only applied after the human-approval threshold is met. Over time, the human sign-off can be relaxed for changes below 20 % delta.
Q: What stops the loop from amplifying a biased tone that a minority of recipients dislike? Segment-specific evaluation. If a tone change hurts a 5 % segment, the loop will see that segment's edit-distance metric drop. Because rare types are weighted higher, the signal isn't drowned out. The rollback mechanism also catches drops quickly.
Q: Could the loop accidentally learn to just copy the user's past replies and lose generality? Overfitting to exact past drafts is prevented by the diversity constraint and by the edit-distance metric itself: if the model merely memorizes a template, it will fail when a new recipient type appears. The loop measures generalization by tracking performance on held-out recent weeks, not the same data it trained on.
Q: Why not just use actual reinforcement learning with reward models? You could. But for an inbox assistant, the “reward” is complex (tone, timeliness, user edit friction) and hard to scalarize. The natural-language policy is easier to debug, modify, and explain. For many production agent systems, that operational clarity beats a black-box score.
Q: Does Mailmind ever revert its own updates automatically? Yes. If an updated prompt version runs in 10 % staging and the acceptance rate drops by more than 15 % relative to the control, the system automatically cuts over to the previous version and logs the event.
Test yourself
Scenario: Mailmind proposes meeting slots for a mixed team across three time zones. You want it to learn which slot characteristics lead to fastest acceptance, without breaking the existing rule that never proposes slots before 9 AM local time. Design the end-to-end outcome-tracking and update loop, including guardrails.
Answer: Capture per-invitee data: proposed slots, acceptance status, time-to-accept, invitee timezone, and day of week. Wait for a 72-hour outcome window before scoring. Compute acceptance rate and median time-to-accept for each slot position (e.g., slot 1, slot 2, slot 3) grouped by timezone and day. The evaluation mines patterns: Tuesday 10 AM slots for CET invitees see 85 % acceptance within 2 hours; Friday 4 PM slots drop to 45 %. The update step then rewrites the slot-selection rule to prefer those high-performing combinations, but it never violates the 9 AM hard constraint (that rule is locked and excluded from optimization). Guardrails: minimum 80 proposals per timezone before an update, max change of one slot preference per iteration, and human approval if the new rule would alter more than 20 % of proposals. Stage the new rule to 10 % of traffic and monitor same-day acceptance for 48 hours before full rollout.
If you want this kind of breakdown every week, how real AI systems actually learn from their own output, with production guardrails that stop them from quietly breaking, subscribe to Internals Decoded at internalsdecoded.com.
Next up: Part 10 examines how agents maintain alignment as they self-modify, building on the update guardrails we just introduced.