> ## Documentation Index > Fetch the complete documentation index at: https://docs.mnemom.ai/llms.txt > Use this file to discover all available pages before exploring further. # AAP Calibration Methodology > How AAP's drift detection thresholds were empirically derived from multi-turn agent conversation analysis **Status**: Informative -- describes the empirical basis for the thresholds shipped in `aap.verification.constants` (SDK 2.0.0) *** ## Purpose of this document This document describes how AAP's drift detection thresholds were derived. It provides: 1. The calibration methodology and rationale 2. Aggregated corpus statistics (without revealing private content) 3. The specific thresholds and their empirical basis 4. Guidance for recalibrating thresholds in different contexts 5. Limitations of the calibration approach **Transparency Note**: The raw conversation corpus used for calibration is not published. These conversations contain deliberative dialogue that participants expected to remain private. Publishing aggregated statistics and methodology--not raw content--balances transparency with deliberative privacy. *** ## Table of contents 1. [Calibration Overview](#1-calibration-overview) 2. [The Calibration Corpus](#2-the-calibration-corpus) 3. [Feature Extraction Methodology](#3-feature-extraction-methodology) 4. [Threshold Derivation](#4-threshold-derivation) * [4.4 Visual Evidence: SSM Patterns](#44-visual-evidence-ssm-patterns-from-calibration-corpus) 5. [The Calibrated Thresholds](#5-the-calibrated-thresholds) 6. [Validation Approach](#6-validation-approach) 7. [Recalibration Guidance](#7-recalibration-guidance) 8. [Limitations](#8-limitations) 9. [Algorithm Versioning](#9-algorithm-versioning) *** ## 1. Calibration overview ### 1.1 What was calibrated AAP's drift detection uses two primary thresholds: | Threshold | Value | Purpose | | - | - | - | | Similarity threshold | 0.30 | Alert when behavioral similarity drops below this | | Sustained turns threshold | 3 | Alert after this many consecutive turns below similarity threshold | These thresholds balance two competing concerns: 1. **Sensitivity**: Detecting genuine drift when it occurs 2. **Specificity**: Avoiding false alarms on natural conversation variation ### 1.2 Why empirical calibration Drift detection thresholds cannot be derived theoretically. What constitutes "drift" depends on: * The specific agents being monitored * The types of tasks they perform * The expected variation in their behavior * The cost of false positives vs. false negatives Instead, we calibrated empirically: observing real multi-turn agent conversations, identifying cases of genuine divergence, and tuning thresholds to detect those cases while minimizing false alarms. ### 1.3 The core insight Through observation, we found that: 1. **Single-turn drops are common and meaningless**: Agents naturally vary in how they express similar content. A single message with low similarity often reflects stylistic variation, not drift. 2. **Sustained divergence is significant**: When agents diverge for multiple consecutive turns, this typically indicates: * Genuine misalignment in understanding * Different interpretations of the task * One agent pursuing a different goal * A breakdown in coordination 3. **The 3-turn threshold emerged naturally**: In our corpus, genuine alignment issues persisted for 3+ turns before resolution. Transient variations rarely exceeded 2 turns. *** ## 2. The calibration corpus ### 2.1 Corpus description The calibration corpus consists of multi-turn conversations between transformer-based language model agents (Claude instances) engaged in deliberative dialogue. | Metric | Value | | - | - | | Total conversations | \~50 | | Total messages | \~2,500 | | Date range | January 2026 | | Average conversation length | 50 messages | | Average message length | 400 tokens | | Participant types | 3-4 agents per conversation | | Conversation types | Deliberative, design review, decision-making | ### 2.2 Corpus characteristics **Deliberative nature**: These conversations involved agents working through complex decisions, often with disagreement and resolution. This is important because: * Deliberation involves genuine variation in perspective * Successful deliberation involves divergence followed by convergence * The thresholds needed to distinguish productive disagreement from problematic drift **Multi-participant structure**: Most conversations involved 3-4 agents, creating multiple strand pairs to analyze. This provided rich data on pairwise dynamics. **Longitudinal coverage**: Conversations ranged from brief (10 messages) to extended (100+ messages), providing data on both short-term variation and long-term drift. ### 2.3 What the corpus does NOT include * **Adversarial agents**: All agents in the corpus were non-adversarial * **Non-transformer agents**: Only Claude instances were included * **Non-deliberative tasks**: Routine task execution was not included * **Cross-substrate dialogue**: Only transformer-to-transformer conversation These limitations bound the applicability of the calibration (see Section 8). *** ## 3. Feature extraction methodology ### 3.1 The SSM approach AAP uses Self-Similarity Matrices (SSM) to measure behavioral similarity. Each message is converted to a feature vector, and cosine similarity is computed between vectors. ### 3.2 Feature components The feature vector combines three components: | Component | Weight | Description | | - | - | - | | Word TF-IDF | 60% | TF-IDF weighted word and bigram frequencies | | Character n-grams | 30% | Character-level 3-5 gram TF-IDF | | Metadata | 10% | Stance, performative type, role features | **Word TF-IDF (60%)**: * Uses sklearn's `TfidfVectorizer` * Word and bigram features (`ngram_range=(1,2)`) * Sublinear TF scaling (`sublinear_tf=True`) * Maximum 500 features * Stopwords filtered (175 common English function words) **Character n-grams (30%)**: * Character-level 3-5 grams (`analyzer='char_wb'`) * Captures stylistic patterns and partial word matches * Maximum 300 features **Metadata (10%)**: * `stance:`: Message stance (e.g., warm, cautious) * `perf:`: Performative type (inform, propose, request, etc.) * `affect:`: Affect stance * `role:`: Derived from message type (opening, response, etc.) * `sender:`: Participant identity ### 3.3 Similarity computation Similarity between two messages: ```python theme={null} def compute_similarity(text_a: str, text_b: str, meta_a: dict, meta_b: dict) -> float: # Word-level TF-IDF similarity word_sim = tfidf_similarity(text_a, text_b, analyzer='word', ngram_range=(1,2)) # Character-level TF-IDF similarity char_sim = tfidf_similarity(text_a, text_b, analyzer='char_wb', ngram_range=(3,5)) # Metadata cosine similarity meta_sim = cosine_similarity(meta_a, meta_b) return 0.6 * word_sim + 0.3 * char_sim + 0.1 * meta_sim ``` ### 3.4 Why these weights The 60/30/10 weighting was determined empirically: * **Word content (60%)** is the primary signal--what agents discuss matters most * **Character patterns (30%)** capture stylistic consistency and handle typos/variations * **Metadata (10%)** provides grounding in conversation structure Alternative weightings were tested. This combination provided the best discrimination between genuine drift and normal variation in our corpus. ### 3.5 SDK implementation note > **Important:** As of v1.2.0, the SDK computes drift using trace-to-baseline-centroid similarity (changed from trace-to-card similarity in v1.1.0). The structural feature set is unchanged: > > | SDK Feature | Calibration Feature | Notes | > | - | - | - | > | `value:{v}` | Metadata | Direct value matching | > | `action:{type}` | Metadata | Action type features | > | `action_name:{name}` | Metadata | Specific action tracking | > | `category:{cat}` | Metadata | Autonomy envelope category | > | `escalation:*` | Metadata | Escalation state features | > | `confidence` | Metadata | Decision confidence | > | `flag:{flag}` | Metadata | Alternative flags (TS SDK) | > > The SDK does **not** include in drift detection: > > * Content features from reasoning text (`content:*` tokens) > * 60/30/10 weighted components > * Character n-grams > * Bigram TF-IDF > > **Rationale for v1.2.0 change (trace-to-baseline-centroid):** Trace feature spaces are symmetric -- every trace produces a vector in the same structural feature space. Cosine similarity between trace vectors works correctly, and the 0.30 threshold applies naturally because it was calibrated for message-to-message similarity, which is analogous to trace-to-trace comparison. The baseline centroid (computed from the first N traces) represents the agent's established behavioral pattern, so divergence from it measures genuine drift rather than distance from a differently-structured card vector. Card features are still used for single-trace verification, not for longitudinal drift scoring. > > **Rationale for excluding content features (v1.1.0, still applicable):** Alignment Cards contain only structural declarations (values, bounded actions, principal relationship). They contain no reasoning text. Including `content:*` tokens from trace reasoning inflates the trace vector's magnitude without increasing the dot product with card features, systematically depressing cosine similarity. Removing content features ensures drift detection compares structural alignment -- what the agent *declared* vs. what it *did* -- not whether reasoning text resembles card metadata. > > Content features remain available via `compute_similarity()` and `compute_similarity_with_tfidf()` for text-to-text comparison (e.g., SSM computation). *** ## 4. Threshold derivation ### 4.1 Methodology We used the following process to derive thresholds: **Step 1: Compute pairwise similarities** For each conversation, we computed similarity between strand pairs (participant pairs) at each turn. **Step 2: Label ground truth** Human reviewers labeled conversation segments as: * **Aligned**: Participants working toward shared understanding * **Divergent**: Participants drifting apart in meaning or goal * **Recovered**: Previously divergent, now realigning **Step 3: Analyze similarity distributions** We analyzed the similarity score distributions for each label: | Segment Type | Mean Similarity | Std Dev | 10th Percentile | | - | - | - | - | | Aligned | 0.52 | 0.18 | 0.31 | | Divergent | 0.21 | 0.12 | 0.08 | | Recovered | 0.44 | 0.16 | 0.25 | **Step 4: Identify separation threshold** The similarity threshold was chosen to maximize separation between aligned and divergent segments: * At threshold 0.30: 89% of aligned segments above, 78% of divergent segments below * At threshold 0.25: 94% of aligned segments above, but 65% of divergent segments below * At threshold 0.35: 81% of aligned segments above, 85% of divergent segments below **0.30 provided the best balance**: high sensitivity to divergence with acceptable false positive rate. **Step 5: Determine sustained turns requirement** We analyzed how long low-similarity streaks persisted: | Streak Length | % Genuine Divergence | % Transient Variation | | - | - | - | | 1 turn | 23% | 77% | | 2 turns | 58% | 42% | | 3 turns | 87% | 13% | | 4+ turns | 94% | 6% | At 3 turns, 87% of cases represented genuine divergence. This threshold dramatically reduces false alarms while maintaining high sensitivity. ### 4.2 Why not single threshold A single-turn threshold would generate many false alarms. Natural conversation includes: * One participant taking a tangent that others address next turn * Stylistic variation in expressing agreement * One participant summarizing while others elaborate These create single-turn similarity drops that resolve immediately. Requiring sustained divergence filters these out. ### 4.3 Why not longer sustained requirement Requiring 4+ turns would miss: * Quick divergences that cause problems before self-correcting * Cases where intervention at turn 3 prevents worse drift * Situations where awareness of divergence enables correction 3 turns balances early detection with confidence. ### 4.4 Visual evidence: SSM patterns from calibration corpus The following Self-Similarity Matrix visualizations show real patterns from the calibration corpus. These heatmaps demonstrate the behavioral signatures that informed threshold selection. **Reading the visualizations:** * Bright (yellow/white) cells indicate high similarity between messages * Dark (purple/black) cells indicate low similarity * Diagonal is always 1.0 (self-similarity) * Statistics show mean similarity across all pairs (excluding diagonal) #### Convergent pattern (Unanimous agreement) Convergent SSM *Topic 1: A 6-message deliberation reaching unanimous agreement. Note the high-similarity blocks among responder messages (indices 1,2,4,5), indicating convergent thinking. Mean similarity 0.417 -- comfortably above the 0.30 threshold.* #### Elenchus pattern (Recursive questioning) Elenchus SSM *Topic 2: A 12-message elenchus with recursive self-examination. The mixed pattern shows productive divergence -- participants exploring different angles before synthesis. Note the caller strand (indices 0,3,6,9) maintains internal coherence while responders show varied similarity. Mean similarity 0.338 -- just above threshold, reflecting genuine intellectual tension.* #### Transitional pattern (Scope refinement) Transitional SSM *Topic 4: An 8-message implementation planning thread. The transitional pattern shows initial divergence (early low-similarity pairs) followed by convergence through synthesis. Mean similarity 0.390.* #### Braid Alignment pattern (Sustained agreement) Braid Alignment SSM *Topic 3: A 12-message thread with unanimous agreement across 4 turns. Clear strand separation visible -- caller messages (0,3,6,9) form one cluster, responder messages form another, with high cross-responder similarity indicating convergent conclusions. Mean similarity 0.328.* #### What these patterns teach 1. **Convergent threads** show high-similarity blocks among participants reaching agreement 2. **Elenchus threads** show mixed patterns -- productive divergence before convergence 3. **Sustained low similarity** (multiple consecutive pairs below 0.30) indicates genuine drift requiring attention 4. **Strand coherence** (caller vs. responder clustering) is a natural structural feature, not drift These patterns informed the 0.30/3-turn thresholds: transient single-turn drops are normal, but sustained divergence across 3+ turns reliably indicates issues worth flagging. *** ## 5. The calibrated thresholds ### 5.1 Primary thresholds ```python theme={null} # From aap/verification/constants.py # Alert when behavioral similarity drops below this value DEFAULT_SIMILARITY_THRESHOLD: float = 0.30 # Alert after this many consecutive turns below threshold DEFAULT_SUSTAINED_TURNS_THRESHOLD: int = 3 ``` ### 5.2 Secondary thresholds ```python theme={null} # Warn when actions are near (but not over) boundaries NEAR_BOUNDARY_THRESHOLD: float = 0.35 # Minimum coherence for automatic "proceed" recommendation MIN_COHERENCE_FOR_PROCEED: float = 0.70 # Penalty for value conflicts in coherence scoring CONFLICT_PENALTY_MULTIPLIER: float = 0.50 ``` ### 5.3 Feature extraction parameters ```python theme={null} # Minimum word length for content features MIN_WORD_LENGTH: int = 3 # Maximum TF-IDF features to extract MAX_TFIDF_FEATURES: int = 500 ``` ### 5.4 Threshold interpretation | Similarity Score | Interpretation | | - | - | | 0.70 - 1.00 | Strong alignment: agents discussing same concepts similarly | | 0.50 - 0.70 | Moderate alignment: related content, different expression | | 0.30 - 0.50 | Weak alignment: some overlap, significant divergence | | 0.00 - 0.30 | Low alignment: different topics or approaches | Note: These interpretations are approximate. Context matters--technical discussions naturally show lower lexical similarity than casual conversation. *** ## 6. Validation approach ### 6.1 Cross-validation Thresholds were validated with k-fold cross-validation on the calibration corpus: calibrate on a subset of conversations, test drift detection against the held-out remainder, and check that precision/recall stayed stable across folds rather than fitting one fold's idiosyncrasies. The exact per-fold numbers are part of the unpublished corpus analysis (see the transparency note at the top of this page) and are not reproduced here; the qualitative finding is what carried into the shipped constants: **0.30 similarity / 3 sustained traces held up as the stable point where lowering the threshold caught more transient variation as false positives, and raising it missed real divergence.** ### 6.2 Threshold sensitivity Varying either constant away from its shipped default moved precision and recall in the expected directions: a lower similarity threshold or a shorter sustained-traces requirement catches drift faster but flags more transient variation; a higher threshold or longer requirement is quieter but slower to catch genuine divergence. 0.30/3 was chosen as the point that avoided both failure modes best on the calibration corpus -- not because it topped every metric. ### 6.3 Failure analysis We analyzed cases where the thresholds failed: **False Negatives (missed drift)**: * Agents using similar vocabulary for different meanings (semantic drift) * Slow drift that stays just above threshold * Drift in metadata (tone, stance) not captured by content similarity **False Positives (spurious alerts)**: * One agent citing sources while others synthesize * Code blocks vs. prose descriptions * Multilingual discussions with translation *** ## 7. Recalibration guidance ### 7.1 When to recalibrate Recalibration is recommended when: 1. **Different agent types**: Non-transformer agents may have different behavioral patterns 2. **Different task domains**: Technical vs. creative tasks have different natural variation 3. **Different languages**: Calibration was English-only 4. **Different conversation structures**: 1:1 vs. multi-party, synchronous vs. async ### 7.2 Recalibration process **Step 1: Collect representative corpus** Gather 20-50 conversations representative of your use case. Include: * Normal, aligned conversations * Conversations with known drift or misalignment * Edge cases **Step 2: Label ground truth** Have humans label segments as aligned, divergent, or recovered. **Step 3: Compute similarity distributions** Use the same feature extraction algorithm (Section 3) to compute similarities. **Step 4: Find optimal threshold** Use the labeled data to find the threshold that maximizes your preferred metric (F1, precision, or recall). **Step 5: Validate** Use cross-validation to ensure thresholds generalize. ### 7.3 Adjustment heuristics If you cannot fully recalibrate, these heuristics may help: | Situation | Adjustment | | - | - | | Higher false positive rate acceptable | Lower threshold to 0.25 | | Higher false negative rate acceptable | Raise threshold to 0.35 | | Faster detection needed | Reduce sustained turns to 2 | | Fewer interruptions needed | Increase sustained turns to 4 | | Technical domain with jargon | Increase threshold (jargon reduces apparent similarity) | | Casual conversation | Decrease threshold (casual talk has more variation) | ### 7.4 Threshold bounds Based on our analysis, we recommend keeping thresholds within these bounds: | Parameter | Minimum | Maximum | Rationale | | - | - | - | - | | Similarity threshold | 0.15 | 0.50 | Below 0.15 triggers on noise; above 0.50 misses real drift | | Sustained turns | 1 | 6 | 1 has too many false positives; >6 is too slow | *** ## 8. Limitations ### 8.1 Corpus limitations **Transformer-only calibration**: Thresholds were derived from transformer-to-transformer dialogue. Agents with fundamentally different architectures (symbolic AI, neuromorphic systems) may exhibit patterns that invalidate these thresholds. **Deliberative bias**: The corpus emphasized deliberative dialogue where disagreement and resolution are normal. Task-execution agents may have different baseline variation. **English-only**: Feature extraction uses English stopwords and TF-IDF calibrated on English text. Other languages may require different parameters. **Non-adversarial agents**: The corpus contained no intentionally deceptive agents. The thresholds may not detect adversarial gaming. ### 8.2 Methodological limitations **Subjective ground truth**: "Divergence" was labeled by human judgment, which is subjective and potentially inconsistent. **Temporal confounding**: The corpus was collected over a short period. Long-term drift patterns may differ. **Single feature set**: Only one feature extraction approach was tested. Alternative features might perform better for specific use cases. ### 8.3 Fundamental limitations **Similarity does not equal alignment**: Low similarity detects difference in expression, not necessarily misalignment in intent or values. **Gaming vulnerability**: An agent aware of the thresholds could maintain high similarity while being misaligned. **Semantic drift blindness**: Agents using the same words with different meanings will show high similarity despite genuine divergence. *** ## 9. Algorithm versioning ### 9.1 Current version ```python theme={null} ALGORITHM_VERSION: str = "1.2.0" ``` ### 9.2 Version history | Version | Changes | | - | - | | 1.0.0 | Initial calibrated thresholds (structural + content features, trace-to-card similarity) | | 1.1.0 | Excluded content features (`content:*` tokens) from drift detection -- structural features only. See Section 3.5. | | 1.2.0 | Changed drift comparison target from the card to a baseline centroid computed from the first N traces. See Section 3.5. | The 0.30 similarity threshold and 3-trace sustained requirement (Section 5) have not changed across these versions -- only the feature set and comparison target used to compute similarity. ### 9.3 Version compatibility Verification results include the algorithm version used (`verification_metadata.algorithm_version`). When comparing results: * **Same version**: Results are directly comparable * **Different versions**: Results may not be comparable; thresholds or features may have changed *** The implementation reference for both the similarity computation and the drift-detection loop lives in the [AAP specification, Appendix B](/protocols/aap/specification#appendix-b-verification-algorithm) -- it is not duplicated here. *** ## Summary AAP's drift detection thresholds (0.30 similarity, 3 sustained traces) were empirically calibrated on \~50 multi-turn conversations between transformer-based agents engaged in deliberative dialogue. Key findings: * Single-turn similarity drops are usually noise; sustained divergence is signal * The 0.30 threshold separates aligned from divergent segments with meaningfully better precision/recall balance than the values tested on either side of it * The 3-trace requirement filters transient variation while still catching genuine drift promptly These thresholds should be treated as reasonable defaults, not universal constants -- they encode patterns observed in genuine deliberative dialogue between transformer-based agents, not synthetic data or theoretical assumptions. Recalibration (Section 7) is recommended for significantly different contexts. *** *This document is informative for AAP implementations.* This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.