Human Review Escalation — When and How to Involve Humans

Core 7 min +30 XP

💡

THE ANALOGY

A bank's fraud detection escalation. Automated systems handle 95% of transactions. Unusual patterns escalate to a specialist. The specialist's decision feeds back to improve the automated system. The escalation path is designed in advance — not improvised when something goes wrong.

⚠️ EXAM TRAP — The Wrong Answer People Choose

Designing escalation triggers that are too broad (escalate everything) or too narrow (never escalate). The exam tests that you can identify valid escalation triggers vs invalid ones — confidence thresholds, scope changes, and ambiguous instructions are valid; routine decisions are not.

KEY POINTS

1 Valid escalation triggers: low confidence on critical fields, scope exceeds original instructions, ambiguous instructions, irreversible high-blast-radius actions.

2 Invalid escalation triggers: routine decisions within defined scope, recoverable errors, choices between equally valid options.

3 Escalation handoff quality: human reviewer receives enough context to make an informed decision quickly.

4 Escalation SLA: define maximum wait time before auto-escalating further up the chain.

5 Escalation feedback loop: reviewer decisions improve future automated routing.

Valid vs Invalid Escalation Triggers

class EscalationDecider:
    """Determines when to involve a human reviewer."""
    
    VALID_TRIGGERS = {
        "low_confidence_critical_field": {
            "description": "Confidence < high on amount, ID, or name fields",
            "escalation_type": "review",
            "priority": "high"
        },
        "scope_exceeded": {
            "description": "Action would exceed defined agent scope",
            "escalation_type": "authorization",
            "priority": "high"
        },
        "irreversible_high_blast": {
            "description": "Irreversible action affecting >100 entities",
            "escalation_type": "confirmation",
            "priority": "critical"
        },
        "business_rule_breach": {
            "description": "Requested action violates policy (e.g., refund > $500)",
            "escalation_type": "exception_approval",
            "priority": "standard"
        },
        "ambiguous_instructions": {
            "description": "Multiple valid interpretations of user intent",
            "escalation_type": "clarification",
            "priority": "standard"
        }
    }
    
    INVALID_TRIGGERS = {
        "validation_error": "Fix the input, don't escalate",
        "recoverable_error": "Retry, don't escalate",
        "routine_within_scope": "Auto-process, don't involve humans",
        "equally_valid_options": "Agent chooses based on policy, no escalation"
    }
    
    def should_escalate(self, situation: dict) -> dict:
        for trigger_name, trigger_config in self.VALID_TRIGGERS.items():
            if self._matches_trigger(situation, trigger_name):
                return {
                    "escalate": True,
                    "trigger":  trigger_name,
                    "type":     trigger_config["escalation_type"],
                    "priority": trigger_config["priority"]
                }
        return {"escalate": False}

Teach-Back Answers

# Scenario 1: $600 refund (over $500 auto limit)
# → ESCALATE: business_rule_breach, exception_approval
#   Policy defines $500 as auto limit. $600 requires human exception approval.

# Scenario 2: 3-year-old refund request
# → ESCALATE: ambiguous_instructions
#   Refund policy for 3-year-old purchases is likely not defined — need clarification.
#   Or business_rule_breach if policy says 30-day return window.

# Scenario 3: validation error on order ID
# → DO NOT ESCALATE: validation_error
#   The order ID format is invalid. Fix: ask the customer to provide the correct order ID.
#   This is a recoverable input error, not a decision requiring human judgment.

Escalation Handoff Quality

def build_escalation_request(situation: dict, trigger: str) -> dict:
    """Build a context-rich escalation request for the human reviewer."""
    return {
        "escalation_type": trigger,
        "priority":        situation.get("priority", "standard"),
        "sla_hours":       situation.get("sla", 4),
        
        # What the reviewer needs to make a quick, informed decision
        "summary": f"Agent needs authorization for: {situation['requested_action']}",
        "context": {
            "customer":         situation.get("customer_summary"),
            "requested_action": situation["requested_action"],
            "why_escalated":    situation["escalation_reason"],
            "relevant_policy":  situation.get("applicable_policy"),
            "recommendation":   situation.get("agent_recommendation"),
        },
        "decision_options": [
            {"option": "approve",  "consequence": situation["approve_consequence"]},
            {"option": "deny",     "consequence": situation["deny_consequence"]},
            {"option": "modify",   "consequence": "Reviewer specifies modified parameters"},
        ],
        "source_documents": situation.get("relevant_documents", [])
    }

Key Takeaways

Valid triggers: confidence, scope, irreversibility, policy breach, ambiguity
Invalid triggers: validation errors, recoverable errors, routine decisions
Handoff quality: reviewer gets context to decide quickly, not just an alert
SLA per trigger type — high-priority escalations have shorter SLA
Feedback loop — reviewer decisions improve future escalation thresholds