# PHENIX AI Agent Architecture

## System Overview

The PHENIX AI Agent is an automated crystallographic workflow system that:
1. Analyzes experimental data and logs
2. Decides which program to run next
3. Executes the program
4. Tracks results and quality metrics
5. Repeats until structure is solved or workflow completes

## Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────────────┐
│                              CLIENT                                      │
│  ┌─────────────────────────────────────────────────────────────────────┐│
│  │                         ai_agent.py                                 ││
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌────────────┐ ││
│  │  │   Session   │  │    Agent    │  │   Command   │  │    Log     │ ││
│  │  │   Tracker   │  │  Interface  │  │  Executor   │  │   Parser   │ ││
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └────────────┘ ││
│  │        │                │                │                │        ││
│  │        ▼                ▼                ▼                ▼        ││
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌────────────┐ ││
│  │  │ BestFiles   │  │ Local/Remote│  │  easy_run   │  │ Metrics    │ ││
│  │  │ Tracker     │  │   Agent     │  │  subprocess │  │ Extraction │ ││
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └────────────┘ ││
│  └─────────────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    │ v2 JSON Request
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         DECISION ENGINE                                  │
│  ┌─────────────────────────────────────────────────────────────────────┐│
│  │                       run_ai_agent.py                               ││
│  │  ┌─────────────────────────────────────────────────────────────────┐││
│  │  │                      LangGraph                                  │││
│  │  │  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌───────┐│││
│  │  │  │PERCEIVE │─▶│  PLAN   │─▶│  BUILD  │─▶│VALIDATE │─▶│OUTPUT ││││
│  │  │  └─────────┘  └────┬────┘  └─────────┘  └────┬────┘  └───────┘│││
│  │  │                    ▲                          │                │││
│  │  │                    └──────── retry < 3 ───────┤                │││
│  │  │                                               ▼                │││
│  │  │                                          ┌──────────┐          │││
│  │  │                                          │ FALLBACK │──▶OUTPUT │││
│  │  │                                          └──────────┘          │││
│  │  └─────────────────────────────────────────────────────────────────┘││
│  └─────────────────────────────────────────────────────────────────────┘│
│                                                                          │
│  ┌─────────────────────────────────────────────────────────────────────┐│
│  │                        Knowledge Layer                              ││
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌────────────┐ ││
│  │  │   YAML      │  │   Rules     │  │  Templates  │  │  Workflow  │ ││
│  │  │  Programs   │  │  Selector   │  │   Builder   │  │   State    │ ││
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └────────────┘ ││
│  └─────────────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────────────┘
```

## Unified Request Flow

Both LocalAgent and RemoteAgent use the same v2 JSON API and **identical transport encoding**:

```
┌─────────────────────────────────────────────────────────────────┐
│                         Client Side                              │
├─────────────────────────────────────────────────────────────────┤
│  LocalAgent / RemoteAgent                                        │
│       │                                                          │
│       ▼                                                          │
│  build_request_v2()                                              │
│       │                                                          │
│       ▼                                                          │
│  prepare_request_for_transport()                                 │
│       │                                                          │
│       ├── 1. sanitize_request() [YAML-driven]                    │
│       │       ├── Remove ZZxxZZ markers                          │
│       │       ├── Truncate long quoted strings                   │
│       │       ├── Replace tabs with spaces                       │
│       │       └── Remove control characters                      │
│       │                                                          │
│       ├── 2. json.dumps()                                        │
│       │                                                          │
│       └── 3. encode_for_rest() [text_as_simple_string]           │
│                    │                                             │
│     ┌──────────────┴──────────────┐                              │
│     │                             │                              │
│  LocalAgent:                 RemoteAgent:                        │
│  process_request_from_      send encoded to                      │
│  transport() locally        REST server                          │
│     │                             │                              │
│     └──────────────┬──────────────┘                              │
└────────────────────│────────────────────────────────────────────┘
                     ▼
┌─────────────────────────────────────────────────────────────────┐
│                      run_ai_agent.py                             │
├─────────────────────────────────────────────────────────────────┤
│  process_request_from_transport()                                │
│       │                                                          │
│       ├── 1. decode_from_rest() [simple_string_as_text]          │
│       └── 2. json.loads()                                        │
│                    │                                             │
│                    ▼                                             │
│              LangGraph Execution                                 │
│                    │                                             │
│                    ▼                                             │
│  prepare_response_for_transport()                                │
│       │                                                          │
│       ├── 1. sanitize_response()                                 │
│       ├── 2. json.dumps()                                        │
│       └── 3. encode_for_rest()                                   │
└─────────────────────────────────────────────────────────────────┘
```

**Key Design Decision**: LocalAgent performs the full encode/decode roundtrip locally, even though it could skip encoding. This ensures that transport bugs are caught during local testing rather than only appearing in production server scenarios.

## Component Responsibilities

### Client Components

#### ai_agent.py
Main entry point and execution loop:
- Manages the iterative workflow loop (prepare state → call graph → execute → repeat)
- Coordinates GUI callbacks and session management
- Handles command execution via PHENIX subprocess
- Post-execution result routing (`_handle_execution_result`, `_handle_failed_execution`)
- Duplicate command detection and retry (via `_handle_duplicate_check` with graph re-query)
- Client-only file injection (`_inject_missing_required_files`, needs `os.path.exists`)

**Not responsible for** (moved to graph in v112.66–112.69):
- Command sanitization, user param injection, crystal symmetry injection (→ BUILD)
- Stop decisions: hard stops (after_cycle, metrics targets) in PERCEIVE; after_program minimum-run guarantee in PLAN; consecutive-program cap (→ PERCEIVE)
- Duplicate retries bypass (deleted `_retry_duplicate`, now uses graph)

**GUI mode execution caveat (v112.78):** In GUI mode, `_execute_sub_job_for_gui`
runs each program in a dedicated subdirectory (`sub_NN_program/`) and restores CWD
to the parent agent directory afterward.  Any code that runs after the sub-job
returns MUST NOT use `os.getcwd()` to locate output files — it will point to the
parent directory, not the sub-job output.  `_execute_sub_job_for_gui` returns
`gui_output_dir` as the 4th element of its return tuple; callers must use this
for file scanning (`_record_command_result`, `_track_output_files`) while keeping
`os.getcwd()` only for writing the agent's own log files.

#### command_postprocessor.py
Server-safe command transforms called by the BUILD node (new in v112.66):
- `sanitize_command()` — Rules A–D: strip placeholders, blacklisted params, hallucinated cross-program params, bare unscoped params. Rule B2 (v112.72): validates `space_group=` values and strips non-space-group words.
- `inject_user_params()` — append user key=value params missing from command (scope-matched for dotted keys, strategy_flags-validated for bare keys)
- `inject_crystal_symmetry()` — append unit_cell/space_group from directives (validates with `_is_valid_space_group()`)
- `inject_program_defaults()` — append defaults from programs.yaml if missing (safety net)
- `postprocess_command()` — single entry point calling all four in order
- `_is_valid_space_group()` — validates that a value is a plausible space group symbol (H-M notation or IT number 1-230), rejecting English words like "determination"

All functions take explicit data arguments (no `self`/class dependencies), making
them callable from both the graph (server-side) and `ai_agent.py` (client-side
replay path).

**Rule D consistency (v112.70):** `inject_user_params` now validates bare (undotted)
keys against the program's `strategy_flags` allowlist before injection, mirroring
Rule D in `sanitize_command`. Without this, `sanitize_command` would strip a
hallucinated param like `d_min=2.5` and then `inject_user_params` would re-add it
from the user advice text.

**Strategy-flag alias awareness (v112.75):** `inject_user_params` builds an alias
map from `strategy_flags` key→flag mappings.  When `wavelength` maps to
`autosol.lambda={value}`, the alias leaf is `lambda`.  The duplicate check now
verifies both the bare key (`wavelength`) AND the alias leaf (`lambda`) against the
command string.  This prevents re-injection of `wavelength=0.9792` when
`autosol.lambda=0.9792` is already present — the prior check only looked for
"wavelength" and missed the aliased form.

**Autosol atom_type dedup (v112.75):** `postprocess_command` validates that
`autosol.atom_type` and `mad_ha_add_list` differ after injection.  When both are
the same element (e.g., both `S`), the duplicate `mad_ha_add_list` is stripped to
prevent the secondary scatterer from being silently lost.

**Heavier-atom-wins rule (v112.76):** After the dedup check, a deterministic
validation compares atomic numbers (Z) of `atom_type` and `mad_ha_add_list`.  If
the primary has lower Z than the secondary, they are swapped — the heavier element
is always the primary scatterer in SAD/MAD.  Uses `_ANOMALOUS_Z` table covering
27 common anomalous scatterers.  Skips swap when either element is unknown (do no
harm).  Handles multi-element `mad_ha_add_list` by swapping with the heaviest
secondary.

**Rule D design tension (v112.77):** Rule D ("fail closed") strips bare params not
in a program's `strategy_flags` allowlist.  This is safe against hallucination but
blocks legitimate LLM error recovery — e.g., `rebuild_in_place=False` correctly
identified by the LLM as the fix for a sequence mismatch was silently stripped
from autobuild.  The catch-all blacklist (v112.76) handles the reverse case (bad
params that *cause* errors) but Rule D has no feedback loop for good params that
get stripped.  The current mitigation is to expand `strategy_flags` for programs
where recovery params are known.  Autobuild expanded from 3 to 6 flags
(`rebuild_in_place`, `n_cycle_build_max`, `maps_only` added in v112.77).  A
future "warn but keep" mode could rely on PHIL validation + catch-all blacklist
as safety nets.

#### Session Tracker (session.py)
Persists workflow state across cycles:
- Experiment type (X-ray/cryo-EM)
- Resolution
- R-free MTZ path
- Cycle history
- **User Directives** (extracted structured instructions)
- **Supplemental file discovery** — `_rebuild_best_files_from_cycles` (session load)
  and `record_result` (live path) call `_find_missing_outputs` to discover companion
  output files (e.g., map coefficients MTZ) and evaluate them through the best_files
  tracker. This ensures `best_files["map_coeffs_mtz"]` is populated even when the
  client doesn't track all output files.
- **Duplicate detection** — `is_duplicate_command()` uses a two-tier check: exact
  match against all prior commands, then 80%-token-overlap against successful
  commands. The overlap heuristic now compares file tokens (basenames with
  crystallographic extensions) separately — different input files means a different
  computation, regardless of parameter overlap.

#### Directive System
Extracts and enforces user instructions:
- `directive_extractor.py`: Parses natural language → structured JSON
- `directive_validator.py`: Validates LLM decisions against directives
- Integrated into graph nodes for consistent enforcement
- See [USER_DIRECTIVES.md](../guides/USER_DIRECTIVES.md) for details

#### BestFilesTracker (best_files_tracker.py)
Tracks highest-quality files:
- Scores files using YAML-based criteria
- Maintains current best by category (model, data_mtz, map_coeffs_mtz, etc.)
- Provides files for server decisions
- `STAGE_TO_PARENT` maps stage names to parent categories (e.g.,
  `refine_map_coeffs` → `map_coeffs_mtz`, `original_data_mtz` → `data_mtz`)

#### file_utils.py
Shared file classification and pattern matching:
- `classify_mtz_type()` — classifies MTZ files as `data_mtz` or `map_coeffs_mtz`
  based on filename patterns
- `matches_exclude_pattern()` — word-boundary-aware pattern matching for
  `exclude_patterns` and `prefer_patterns` in slot definitions. Patterns match at
  start-of-string or after separators (`_`, `-`, `.`), preventing false positives
  like "noligand" matching "ligand"

#### workflow_state.py
File content analysis for slot guards:
- `_pdb_is_small_molecule()` — reads first 8KB, returns True for HETATM-only PDB
  files (ligands, cofactors). Used to reject small molecules from model slots.
- `_pdb_is_protein_model()` — positive protein check, returns True for PDB files
  with ATOM records. Used to reject protein models from ligand slots. Returns
  False for non-existent files (safe for use as rejection filter).

#### Agent Interface (LocalAgent/RemoteAgent)
Both agents use identical interface, v2 JSON format, **and transport encoding**:
- `LocalAgent`: Full encode/decode roundtrip, then calls `run_ai_agent.run()`
- `RemoteAgent`: Encodes and sends to REST server
- Same `prepare_request_for_transport()` for encoding
- Same `history_record` response format
- Transport module: `agent/transport.py`
- Configuration: `knowledge/transport.yaml`

**Error propagation (v112.78):** `RemoteAgent._send_request()` is wrapped in a
generic `except Exception` handler.  Since `Sorry` inherits from `Exception`,
fatal server errors (e.g., daily usage limit from `rest/__init__.py`) were
silently caught, logged, and returned as `None`.  Fix: `except Sorry: raise`
before the generic handler lets fatal errors propagate to the GUI.

#### Log Parsers (phenix_ai/log_parsers.py)
Extracts metrics and output files from program log output:

**Key functions:**
- `extract_all_metrics(log_text, program)`: Main entry point for metric extraction
- `detect_program(log_text)`: Identifies which program generated a log
- `extract_output_files(log_text, working_dir)`: Finds output file paths
- `format_metrics_report(metrics)`: Generates "FINAL QUALITY METRICS REPORT"

**Note**: Metric extraction patterns are defined in `programs.yaml` and loaded via `metric_patterns.py`. Hardcoded extractors in log_parsers.py only handle complex cases (tables, multi-line context) that YAML patterns can't express.

**YAML `log_parsing` pattern conventions** (see `programs.yaml`):

| Field | Values | Usage |
|-------|--------|-------|
| `extract` | `first` (default), `last` | `last` required when a program emits one metric line per cycle (RSR, autobuild, phaser) |
| `pick_min` | `true` | After collecting all matches, take the minimum value. Used for xtriage `resolution` (multiple lines; smallest value = highest resolution) |
| `type` | `float`, `int`, `string`, `boolean` | Controls return type |

**Important audited patterns** (fixes applied in v112.14):
- `phenix.xtriage` `resolution`: uses `pick_min: true`. Pattern anchored with `^\s*` and negative lookbehind to prevent "Completeness in resolution range: 1" from matching. Skip group `(?:[0-9.]+\s*[-]\s*)?` handles "50.00 - 2.30" format.
- `phenix.real_space_refine` `map_cc`: uses `extract: last`. RSR emits one CC_mask line per macro-cycle; `last` gives the final value.

**Adding new program support:**
See [ADDING_PROGRAMS.md](../guides/ADDING_PROGRAMS.md) for the complete guide.

### Centralized Configuration Modules

The agent uses a centralized YAML-driven architecture that reduces program configuration from 7 files to 2-3 files.

#### Metric Patterns (knowledge/metric_patterns.py)
Centralizes metric extraction patterns from programs.yaml:

```
programs.yaml          metric_patterns.py
     │                        │
     ▼                        ▼
log_parsing: ────────▶ extract_metrics_for_program()
  pattern:                    │
  type:              ┌────────┴────────┐
  display_name:      ▼                 ▼
                log_parsers.py    session.py
```

**Key functions:**
- `get_all_metric_patterns()`: Load all patterns from YAML (cached)
- `extract_metrics_for_program(log_text, program)`: Extract metrics using YAML patterns
- `format_metric_value(program, metric, value)`: Format using YAML config

#### Program Registration (knowledge/program_registration.py)
Reads `done_tracking` blocks from programs.yaml for workflow done flags:

```
programs.yaml             program_registration.py
     │                            │
     ▼                            ▼
done_tracking:           get_program_done_flag_map()  ← ALL programs
  flag: "xxx_done"       get_trackable_programs()     ← strategy: "run_once"
  strategy: "run_once"            │
  # or "count"           ┌────────┴────────┐
  # or "set_flag"        ▼                 ▼
                 workflow_state.py   workflow_engine.py
                 (_set_done_flags)   (context building)
```

**Key functions:**
- `get_program_done_flag_map()`: All programs → done flag names (from YAML)
- `get_trackable_programs()`: Programs with `strategy: "run_once"` (filtered from valid list after completion)

**`_is_program_already_done` scope (v112.75):** `_apply_directives` in
`workflow_engine.py` calls `_is_program_already_done()` before re-adding programs
from `program_settings` directives.  This function now checks two conditions:
(1) `run_once` programs whose done flag is set (original), and (2) any non-count
program with a program-specific done flag (e.g., `autosol_done` contains `autosol`).
Without check (2), `_apply_directives` could re-add completed programs like autosol
at the front of `valid_programs`, causing the LLM to re-run them.  Count-strategy
programs (refine, rsr, phaser) are excluded since they intentionally repeat.
Shared flags (e.g., `validation_done`) are excluded since the flag name doesn't
contain the program short name.

#### Summary Display (knowledge/summary_display.py)
Configures session summary display from metrics.yaml:

```
metrics.yaml           summary_display.py
     │                        │
     ▼                        ▼
summary_display: ─────▶ format_quality_table_rows()
  quality_table:              │
  step_metrics:        ┌──────┴──────┐
                       ▼             ▼
                session.py     session.py
             (Final Quality)  (Steps table)
```

**Key functions:**
- `get_quality_table_config()`: Load quality table row configs
- `get_step_metrics_config()`: Load per-program step metric configs
- `format_quality_table_rows()`: Format Final Quality table
- `format_step_metric()`: Format a step's key metric

#### Program Validator (agent/program_validator.py)
Validates that programs are fully configured:
```bash
python agent/program_validator.py phenix.map_symmetry  # Check one program
python agent/program_validator.py --all                 # Check all programs
```

### Decision Engine Components

#### run_ai_agent.py
Single entry point for all requests:
- Parses v2 JSON requests
- Validates request schema
- Creates graph initial state
- Invokes LangGraph
- Returns v2 JSON response

#### LangGraph Pipeline
Decision-making workflow:

```
PERCEIVE ──┬──▶ PLAN ─▶ BUILD ─▶ VALIDATE ──┬──▶ OUTPUT ─▶ END
           │     │        │         │    │    │
           │     │        │         │    │    │
        Analyze Select   Build   Validate │  Format
        inputs  program  command response │  decision
           │                        │     │
           │               retry <3 │     │ retry >=3
           │                   ┌────┘     │
           │                   ▼          ▼
           │                  PLAN     FALLBACK ─▶ OUTPUT ─▶ END
           │
           └──▶ OUTPUT ─▶ END  (if red flag abort)
```

#### Decision Flow Architecture

The decision flow follows a clean, layered architecture where each component has a single responsibility:

```
┌─────────────────────────────────────────────────────────────────┐
│                     SESSION START                                │
│  directive_extractor.py extracts structured directives          │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                 LAYER 1: WORKFLOW ENGINE                         │
│  workflow_engine.py determines valid_programs                    │
│                                                                  │
│  Input: phase, context, directives                              │
│  Output: valid_programs list (includes STOP if appropriate)     │
│                                                                  │
│  Responsibilities:                                               │
│  - Get programs from workflow phase definition                   │
│  - Add directive-required programs (after_program target)        │
│  - Add start_with_program to front of list (multi-step flows)   │
│  - Add STOP if skip_validation=true                             │
│  - Apply workflow preferences (skip/prefer)                      │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                 LAYER 2: LLM DECISION (PLAN)                     │
│  graph_nodes.py plan() function                                  │
│                                                                  │
│  Input: valid_programs, context, history                        │
│  Output: chosen program (must be in valid_programs)             │
│                                                                  │
│  Responsibilities:                                               │
│  - LLM selects best program from valid_programs                 │
│  - Simple validation: is choice in valid_programs?              │
│  - If invalid choice, use first valid program                   │
│                                                                  │
│  NOT responsible for:                                            │
│  - Directive logic (already handled in Layer 1)                 │
│  - Stop condition evaluation (handled in Layer 4)               │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                 LAYER 3: BUILD & EXECUTE                         │
│  graph_nodes.py build(), ai_agent.py execute                    │
│                                                                  │
│  Input: chosen program, files, context                          │
│  Output: command result, output files                           │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                 LAYER 4: POST-EXECUTION CHECK                    │
│  ai_agent.py _run_single_cycle() - SINGLE PLACE                 │
│                                                                  │
│  Input: completed program, cycle number, directives             │
│  Output: should_stop (bool), stop_reason                        │
│                                                                  │
│  Responsibilities:                                               │
│  - Check if after_cycle condition met (hard stop)               │
│  - Check if metric targets met — r_free, map_cc (hard stop)    │
│  - This is the ONLY place hard stop conditions are evaluated    │
│                                                                  │
│  NOTE (v112.78): after_program is intentionally NOT a hard      │
│  stop here.  It is a minimum-run guarantee: PLAN suppresses     │
│  auto-stop until the target program has run, but the LLM        │
│  decides when to actually stop.  This prevents premature        │
│  termination on multi-goal requests where the directive          │
│  extractor can only name one program.                           │
└─────────────────────────────────────────────────────────────────┘
```

**Key Design Principles:**

1. **Single Responsibility**: Each layer handles one aspect of decision-making
2. **Clear Data Flow**: Directives flow from extraction → workflow engine → validated programs
3. **Post-Execution Stop**: Stop conditions are only checked after program execution, not during planning
4. **Simple Validation Gate**: The plan() function validates that LLM choice is in valid_programs list

#### Knowledge Layer

**YAML Programs** (`knowledge/programs.yaml`):
- Program definitions
- Input/output specifications
- Invariants (rules that must be satisfied)
- Scoring rules

**Rules Selector** (`rules_selector.py`):
- Matches workflow state to applicable programs
- Filters by experiment type
- Applies priority rules

**Command Builder** (`command_builder.py`):
- **Single entry point** for all command generation
- 4-stage pipeline:
  1. `_select_files()` - Priority-based file selection
  2. `_build_strategy()` - Strategy with output_prefix
  3. `_apply_invariants()` - Auto-fill resolution, R-free flags
  4. `_assemble_command()` - Final command string
- Uses `CommandContext` dataclass for all parameters
- Replaces fragmented logic from template_builder and graph_nodes

**Template Builder** (`template_builder.py`):
- Legacy interface (delegates to CommandBuilder)
- YAML-to-slot mapping
- Invariant checking

**Workflow State** (`workflow_state.py`):
- Determines current workflow position
- Categorizes available files
- Detects experiment type

## LLM Provider Architecture

The agent supports three LLM providers: **google**, **openai**, and **ollama**. The
provider is set via `params.communication.provider` (or the `LLM_PROVIDER` env var)
and flows consistently through every LLM call point — there is no cross-provider
fallback.

### LLM Call Points

The system uses three distinct LLM roles, each with provider-specific model defaults:

| Activity | Function | ollama | google | openai |
|---|---|---|---|---|
| **Agent decisions** (PLAN node) | `get_planning_llm()` → `get_expensive_llm()` | qwen3:32b (json_mode) | gemini-2.5-pro | gpt-5 |
| **Log summarization** | `get_cheap_llm()` | qwen2.5:7b | gemini-2.5-flash-lite | gpt-5-nano |
| **RAG analysis** | `get_expensive_llm()` | qwen3:32b | gemini-2.5-pro | gpt-5 |
| **Directive extraction** | `call_llm_simple()` | direct ollama HTTP | gemini-2.0-flash | (provider default) |

### Provider Flow

```
params.communication.provider = "<provider>"
         │
         ├─► LocalAgent.decide_next_step()
         │       └─► request["settings"]["provider"] = "<provider>"
         │            └─► run_ai_agent.py: create_initial_state(provider=...)
         │                 └─► graph_nodes.py PLAN: get_planning_llm(provider)
         │                      └─► get_expensive_llm(provider)
         │
         ├─► _extract_directives()
         │       ├─► ollama: forces run_on_server=False (local ollama not on server)
         │       └─► run_directive_extraction(provider=...)
         │            └─► call_llm_simple(provider=...)
         │
         └─► run_ai_analysis.run() [summarization + RAG]
                 └─► setup_llms(provider=...)
                      ├─► expensive_llm = get_expensive_llm(provider)
                      ├─► cheap_llm = get_cheap_llm(provider)
                      └─► embeddings = provider-specific embeddings
```

### Key Files

| File | Role |
|------|------|
| `core/llm.py` | `get_llm_and_embeddings()`, `get_expensive_llm()`, `get_cheap_llm()` — model creation |
| `agent/graph_nodes.py` | `get_planning_llm()` — cached LLM for PLAN node decisions |
| `agent/api_client.py` | `call_llm_simple()` — lightweight direct calls (directive extraction) |
| `utils/run_utils.py` | `setup_llms()` — creates all three LLM roles for analysis |
| `phenix_ai/local_agent.py` | Reads provider from `params.communication.provider`, passes to request |
| `phenix_ai/run_ai_agent.py` | Reads provider from settings, passes to `create_initial_state()` |

### Ollama-Specific Behavior

- **Directive extraction** forces `run_on_server=False` because the Phenix REST
  server doesn't have access to the user's local ollama service
- **Planning LLM** uses `json_mode=True` (sets `format="json"` in ChatOllama)
  because ollama models need explicit JSON formatting; Google and OpenAI handle
  structured output without this flag
- **GPU requirement**: `rest_server.requires_gpu = True` is set for ollama to
  ensure server dispatch targets GPU-capable machines

### Adding a New Provider

1. Add provider case to `get_llm_and_embeddings()` in `core/llm.py`
2. Add validation check in `validate_provider()` in `agent/graph_nodes.py`
3. Add `_call_<provider>_llm()` function in `agent/api_client.py`
4. Add rate limit handler in `agent/rate_limit_handler.py` (optional)

## Data Flow

### Request Building (Client)

```python
# Both agents build requests the same way
session_state = {
    "resolution": 2.5,
    "experiment_type": "xray",
    "rfree_mtz": "/path/to/data.mtz",
    "best_files": {"model": "/path/to/model.pdb"},
    "bad_inject_params": {"phenix.refine": ["ignore_symmetry_conflicts"]},
    "advice_changed": False,
    "unplaced_model_cell": None,
}

request = build_request_v2(
    files=available_files,
    cycle_number=5,
    log_content=last_log,
    history=cycle_history,
    session_state=session_state,
    user_advice=guidelines,
    ...
)
```

**`bad_inject_params` flow** (added in v112.66): Parameters that previously
caused PHENIX errors are blacklisted and propagated to BUILD for stripping:

```
ai_agent.py: session.data["bad_inject_params"]
    → session_info["bad_inject_params"]
    → build_session_state() → session_state
    → build_request_v2() → normalized_session_state
    → transport encode/decode
    → run_ai_agent.py → create_initial_state(bad_inject_params=...)
    → graph state["bad_inject_params"]
    → BUILD: postprocess_command(bad_inject_params=set(...))
```

**Error pattern expansion (v112.75):** The learning system originally only triggered
on "unknown command line parameter" and "no such parameter" errors.  PHIL
boolean-type errors ("True or False value expected, scope.path.param="value" found")
are now also caught — the full PHIL path and all components ≥ 6 characters are
blacklisted.  Without this, `inject_user_params` could loop indefinitely injecting
a parameter that causes a type mismatch rather than an "unknown parameter" error
(observed as 9 wasted cycles in run 107).

**Catch-all injection blacklist (v112.76):** A supervisor pattern that handles
"unknown unknowns" — PHIL error formats not covered by pattern-based learning.
`postprocess_command` surfaces the list of params added by inject_* steps via
`return_injected=True` (opt-in, default False for backward compat).  The replay
path in ai_agent.py stores this list on `session.data["last_injected_params"]`.
`_update_inject_fail_streak()` tracks consecutive same-program failures via error
fingerprint (first 120 chars, normalized, digits stripped).  After N=2 failures
with matching fingerprints and non-empty injected list, all injected params are
blacklisted.  Recovery retries (`force_retry_program`) are excluded.

```
_get_command_for_cycle()
  ├─ session.data["last_injected_params"] = []        (init)
  ├─ _query_agent_for_command()                        (server path)
  │    └─ graph BUILD node
  │         └─ postprocess_command(return_injected=True)
  │              └─ state["last_injected_params"] = [...]
  │    └─ history_record["last_injected_params"] → session.data
  └─ postprocess_command(return_injected=True)         (replay path)
       └─ session.data["last_injected_params"] = [...]

_record_command_result()
  └─ _update_inject_fail_streak(program, error, session)
       ├─ SUCCESS → clear streak
       └─ FAILURE → fingerprint match?
            ├─ yes → count++ → count>=2? → blacklist injected → reset
            └─ no  → reset streak to count=1
```

**`best_files` value type contract:** Most entries are plain strings (`category → path`). However, multi-file entries such as `half_map` may be stored as a list:

```python
"best_files": {
    "model":    "/path/to/model.pdb",       # single file → str
    "half_map": ["/path/map1.mrc", "/path/map2.mrc"],  # two files → list
}
```

All server-side code that reads `best_files` values must handle both types. Use `CommandBuilder._best_path(value)` to safely extract a single path from either a string or a list. Do **not** pass `best_files` values directly to `os.path` functions — this caused a cycle=2 crash (`TypeError: expected str, bytes or os.PathLike, not list`) that only surfaced when `session_state` was re-sent from a client that stored `half_map` as a list.

**Programs that need both `full_map` and `half_map` simultaneously:** Several programs (`phenix.mtriage`, `phenix.predict_and_build`, `phenix.map_to_model`) accept both a full map AND half maps at once — the half maps are not redundant, they enable FSC-based resolution calculation or density modification. These programs are marked `keep_half_maps_with_full_map: true` in `programs.yaml`. The post-selection validation in `CommandBuilder._select_files()` checks this flag before removing half maps. When adding a new program that needs both simultaneously, add this flag to its YAML entry.

### Request Processing (Server)

```python
# run_ai_agent.py processes all requests uniformly
def run(request_json):
    request = _parse_request(request_json)
    response = _process_request(request)
    return _build_group_args_response(response)
```

### Response Handling (Client)

```python
# Both agents receive the same response format
result = agent.decide_next_step(...)
history_record = result  # Contains program, command, reasoning, etc.
command = history_record["command"]
```

## File Organization

See the [README.md](../README.md#directory-structure) for the complete directory tree.

Key directories:
- `agent/` — Core agent logic (session, graph nodes, command builder, command postprocessor, workflow engine, etc.)
- `knowledge/` — YAML configuration files and supporting Python modules
- `phenix_ai/` — Runtime entry points (local/remote agent, log parsers)
- `programs/` — PHENIX program integration (main entry point `ai_agent.py`)
- `analysis/` — Post-run log analysis and session evaluation
- `core/` — LLM provider abstraction
- `validation/` — Command validation framework
- `tests/` — 34 test files with 900+ tests

## Key Design Decisions

### 1. Unified v2 JSON API

**Rationale**: Consistency between local and remote execution.

**Implementation**:
- Both agents use `build_request_v2()` to create requests
- Single `run_ai_agent.run(request_json)` entry point
- Identical response format for both paths

### 2. Client-Server Separation

**Rationale**: Allows server-side updates without client reinstallation.

**Implementation**:
- Client gathers inputs, executes commands, and handles results
- Server makes all decisions (program selection, command construction, stop logic)
- All command post-processing (sanitize, inject params, inject symmetry, inject
  defaults) runs in the graph's BUILD node (server-side)
- Only `_inject_missing_required_files` remains client-side (needs `os.path.exists`)
- Clean API boundary via JSON protocol

### 3. Configuration Over Code

**Rationale**: Domain experts can modify behavior without code changes.

**Implementation**:
- Program definitions in YAML
- Scoring rules in YAML
- Workflow transitions in YAML

### 4. Session State Tracking

**Rationale**: Ensures consistency across workflow cycles.

**Implementation**:
- R-free MTZ locked after first refinement
- Experiment type locked after detection
- Best files tracked with quality scores
- All state transmitted in `session_state`

### 5. LangGraph Pipeline

**Rationale**: Modular, testable decision-making.

**Implementation**:
- Discrete nodes (PERCEIVE, PLAN, BUILD, VALIDATE, OUTPUT)
- State passed between nodes
- Easy to add/modify nodes

### 5a. Single Source of Truth (v112.66–112.69 refactor)

**Rationale**: The agent had accumulated overlapping decision layers — "should we
stop?" was answered in 5 places and "fix the command" in 2 places. This caused
real bugs: 20× refine loops, premature stops, crystal symmetry mismatches.

**Implementation**: The graph is the single authority for decisions:
- **"Should we stop?"** → PERCEIVE + PLAN only
- **"Fix the command"** → BUILD only (via `postprocess_command()`)
- **"Execute and recover"** → `ai_agent.py` only

`ai_agent.py` is a pure execution loop: prepare state → call graph → execute
command → handle result. No parallel decision paths remain. Duplicate retries
flow through the graph (via `duplicate_feedback` parameter) rather than bypassing
it.

**Command post-processing pipeline** (runs in BUILD after command construction):

```
postprocess_command():
  1. sanitize_command()         — Rules A–D (strip bad params)
  2. inject_user_params()       — from user advice text
  3. inject_crystal_symmetry()  — from directives
  4. inject_program_defaults()  — from programs.yaml defaults
```

**Sanitization rules:**

| Rule | Fires when | Strips |
|------|-----------|--------|
| Probe | Program in `_PROBE_ONLY_FILE_PROGRAMS` | All key=value except file paths |
| A | Always | Blacklisted params (`bad_inject_params`) |
| B | key contains `space_group` or `unit_cell` | Placeholder values ("Not specified", etc.) |
| C | Program has zero strategy_flags | Any key=value not in `_UNIVERSAL_KEYS` |
| D | Program HAS strategy_flags, key has no dots | Bare params not in program's allowlist |

**Rule D consistency (v112.70):** `inject_user_params` mirrors Rule D's allowlist
check for bare keys extracted from user advice. Without this, a param stripped by
Rule D (e.g. `d_min=2.5` for autobuild) would be re-injected from the advice text.

### 6. Automation Modes

**Rationale**: Different users need different levels of control over the workflow.

**Implementation**:
Two modes controlled by `maximum_automation` parameter:

| Mode | Setting | `predict_and_build` Behavior |
|------|---------|------------------------------|
| **Automated** | `maximum_automation=True` (default) | Runs complete workflow (prediction → MR → building) |
| **Stepwise** | `maximum_automation=False` | Stops after prediction only |

The `automation_path` is set in workflow state and propagated to all decision points:

```python
# workflow_engine.py
context["automation_path"] = "automated" if maximum_automation else "stepwise"

# workflow_engine.py (_check_program_prerequisites)
# Block after full workflow has completed (both automated and stepwise)
if context.get("predict_full_done"):
    return False
# Block in stepwise mode after prediction-only step
if automation_path == "stepwise" and context.get("predict_done"):
    return False
```

**Stepwise workflow example**:
```
xtriage → predict_and_build(stop_after_predict) → process_predicted_model → phaser → refine
```

### 7. Fallback Program Tracking

**Rationale**: When LLM planning fails 3 times, the fallback mechanism must correctly report which program it selected.

**Implementation**:
- Fallback node now sets `state["program"]` when returning a command
- Response builder uses `state["program"]` if available, falls back to `intent["program"]`
- Prevents mismatch where PLAN shows one program but command is different

**Fallback diagnostics (v112.70):**
When the fallback cannot produce a command, it now provides specific stop reasons:

| Stop Reason | Meaning |
|------------|---------|
| `cannot_build_any_program` | No program could be built (missing required inputs) |
| `build_failures_and_duplicates` | Some programs can't build, rest are duplicates |
| `all_commands_duplicate` | All built commands are actual duplicates of prior runs |

The `abort_message` includes per-program diagnostics showing exactly which required
input slots could not be filled (via `CommandBuilder._last_missing_slots`).

## Workflow States

### X-ray Crystallography Workflow

```
xray_initial → xtriage → xray_analyzed
                              │
              ┌───────────────┼───────────────┐
              ↓               ↓               ↓
       predict_and_build    phaser         autosol†
              ↓               │               ↓
    xray_has_prediction       │        xray_has_phases
              ↓               │               ↓
  process_predicted_model     │          autobuild
              ↓               │               │
    xray_model_processed ─────┘               │
              ↓                               │
           phaser ────────────────────────────┘
              │
              ├── [anomalous data] → xray_mr_sad
              │                         ↓
              │                 autosol (partpdb_file=PHASER.pdb)
              │                         ↓
              │                  xray_has_phases
              │                         ↓
              │                    autobuild ──┐
              ↓                               │
       xray_has_model                         │
              ↓                               │
         refine (loop) ←──────────────────────┘
              ↓
       xray_refined
         ↓    ↓    ↓
    molprobity refine STOP
              │
              ↓
       [if ligand] ligandfit → pdbtools → refine
```

† Prioritized when strong anomalous signal detected (measurability > 0.10)
‡ MR-SAD: phaser places model first, then autosol uses it as partpdb_file

| State | Description | Valid Programs |
|-------|-------------|----------------|
| `xray_initial` | Starting point | xtriage |
| `xray_analyzed` | After data analysis — includes `probe_placement` phase (maps to same external state) | predict_and_build, phaser, autosol, model_vs_data (probe only) |
| `xray_has_prediction` | Have AlphaFold model | process_predicted_model |
| `xray_mr_sad` | After phaser + anomalous data (MR-SAD) | autosol (with partpdb_file) |
| `xray_has_phases` | After experimental phasing | autobuild |
| `xray_has_model` | Have placed model | refine |
| `xray_refined` | After refinement **or** validation (`refine` and `validate` phases share this external state; internal `phase_info["phase"]` distinguishes them) | refine, molprobity, autobuild, STOP |

### Cryo-EM Workflow

```
cryoem_initial → mtriage → cryoem_analyzed
                                │
                ┌───────────────┼───────────────┐
                ↓               ↓               ↓
      predict_and_build    dock_in_map    [half-maps only]
                │               │               ↓
                └───────┬───────┘         resolve_cryo_em
                        ↓                       ↓
                cryoem_has_model          map_sharpening
                        ↓                       │
              real_space_refine (loop) ←────────┘
                        ↓
                 cryoem_refined
                   ↓    ↓    ↓
              molprobity RSR  STOP
```

| State | Description | Valid Programs |
|-------|-------------|----------------|
| `cryoem_initial` | Starting point | mtriage |
| `cryoem_analyzed` | After map analysis — includes `probe_placement` phase (maps to same external state) | predict_and_build, dock_in_map, map_correlations (probe only) |
| `cryoem_has_model` | Half-map optimisation / model check (legacy name — no model yet; `check_map` and `optimize_map` phases) | resolve_cryo_em, map_sharpening |
| `cryoem_docked` | Model docked, ready for first real-space refinement (`ready_to_refine` phase) | real_space_refine |
| `cryoem_refined` | After refinement **or** validation (`refine` and `validate` phases share this external state; internal `phase_info["phase"]` distinguishes them) | real_space_refine, molprobity, STOP |

## Model Placement Detection

When a user supplies an atomic model together with reflection data or a cryo-EM
map but no session history, the agent must decide whether the model is already
positioned in the unit cell / map before choosing the next program.  A three-tier
framework resolves this automatically, with each tier more expensive but more
definitive than the last.

### Tier 1 — Unit cell comparison (free, instant)

`agent/placement_checker.py` reads the unit cell parameters from each file and
compares them with a 5% fractional tolerance.

| Source | Reader |
|---|---|
| PDB CRYST1 record | `read_pdb_unit_cell()` |
| MTZ file | `read_mtz_unit_cell()` (iotbx.mtz, falls back to mtzdump) |
| CCP4/MRC map | `read_map_unit_cells()` — returns **two** cells: full-map and present-portion |

- If any read fails → **fail-safe: no mismatch declared** (workflow falls through to Tier 2)
- Definitive mismatch (> 5% on any parameter) → model cannot be placed →
  route immediately to **MR** (X-ray) or **docking** (cryo-EM)
- Cryo-EM: model is compatible if it matches *either* the full-map cell or the
  present-portion cell (partial maps are common)

### Tier 2 — Existing heuristics (`_has_placed_model`)

Checks history flags (`refine_done`, `dock_done`), file subcategory (`positioned`),
and user directives.  When any of these give a clear signal the framework is done.
When the result is still ambiguous, Tier 3 runs.

The `build_context()` method in `WorkflowEngine` computes a `placement_uncertain`
flag that is `True` exactly when all of the following hold:

- `has_model` and (`has_data_mtz` or `has_map`)
- `has_placed_model` is False (Tier 2 found no evidence)
- `cell_mismatch` is False (Tier 1 found no mismatch)
- `placement_probed` is False (probe has not run yet)
- `has_predicted_model` is False (predicted models always need processing/docking)

### Tier 3 — Diagnostic probe (one program cycle)

When `placement_uncertain` is True the workflow engine routes to the
`probe_placement` phase, which runs a single quick diagnostic:

| Experiment | Program | Threshold | Placed | Not placed |
|---|---|---|---|---|
| X-ray | `phenix.model_vs_data` | R-free < 0.50 | → refine | → molecular_replacement |
| Cryo-EM | `phenix.map_correlations` | CC > 0.15 | → refine | → dock_model |

After the probe cycle, `_analyze_history` in `workflow_state.py` detects the
result **positionally**: the first occurrence of `model_vs_data` or
`map_correlations` that appears *before* any refinement or docking cycle is
identified as the probe.  This requires no schema change to history entries.

`build_context()` sets `placement_probed = True` and
`placement_probe_result = "placed" | "needs_mr" | "needs_dock" | None` (None if
the result could not be parsed — fail-safe: falls through to obtain_model).
When the result is `"placed"`, `build_context` overrides `has_placed_model = True`
so normal refinement routing takes over.

The probe never repeats: `placement_probed = True` in `history_info` prevents
`placement_uncertain` from being set on subsequent cycles.

### Routing summary

```
model + data, no history
        │
        ▼ Tier 1
   Cell mismatch? ─── YES ──▶ MR / docking (skip probe)
        │ NO
        ▼ Tier 2
  Has heuristic evidence? ─── YES ──▶ normal routing
        │ NO (placement_uncertain = True)
        ▼ Tier 3
   probe_placement phase
    (model_vs_data or map_correlations)
        │
   R-free / CC result
        ├── placed    ──▶ refine
        ├── not placed ──▶ MR / docking
        └── unparseable ──▶ obtain_model (fail-safe)
```

### Key files

| File | Role |
|---|---|
| `agent/placement_checker.py` | Unit cell readers and comparison |
| `agent/workflow_engine.py` | `build_context()` new keys; `_detect_*_phase()` routing |
| `agent/workflow_state.py` | `_analyze_history()` probe detection |
| `knowledge/workflows.yaml` | `probe_placement` phase in xray and cryoem |
| `knowledge/programs.yaml` | `done_tracking` for `map_correlations` (pre-existing gap fixed) |
| `agent/yaml_tools.py` | `if_placed` / `if_not_placed` added to `valid_transition_fields` |

### Implementation notes

**Import paths**: `placement_checker` is importable via both `libtbx.langchain.agent.placement_checker`
(production) and `agent.placement_checker` (tests / local dev). `_check_cell_mismatch`
tries both paths — libtbx first, bare `agent` path as fallback — matching the pattern
used throughout the codebase.

**Short-circuit**: `build_context()` computes `cell_mismatch` by calling `_check_cell_mismatch()`
on every cycle. For cryo-EM this involves running `phenix.show_map_info` as a subprocess.
After the context dict is built, a post-processing block overrides `cell_mismatch = False`
when either `has_placed_model` or `placement_probed` is `True` — both conditions mean
placement is already resolved and the check cannot change the outcome. The cell check
still runs unconditionally on the first cycle when placement is genuinely unknown.

**YAML validator**: `if_placed` and `if_not_placed` are registered in
`valid_transition_fields` in `yaml_tools.py` so that `_validate_workflows()` does not
emit spurious "unknown field" warnings for `probe_placement` phases.

---

## Workflow History and Done Flags

`workflow_state.py` maintains two complementary systems for tracking
completed programs:

### _analyze_history: done flag extraction

`_analyze_history(history)` reads the history list and sets done flags
(`refine_done`, `resolve_cryo_em_done`, …) based on marker strings in
each record's `result` field. Key invariants:

- **Failure blocks flags**: `_is_failed_result(result)` is checked
  first. If the result is a failure, all done-flag extraction for that
  record is skipped. Failure signals (in priority order):
  1. Shell exit code (caught at the subprocess layer before this function)
  2. Specific Phenix terminal phrases: `FAILED`, `SORRY:`, `SORRY `,
     `*** ERROR`, `FATAL:`, `TRACEBACK`, `EXCEPTION`
  - Generic `ERROR` words without those prefixes are **not** failure
    signals — Phenix logs contain "Error model parameter", "Expected
    errors: 0", etc. and these must not suppress done flags.

- **predict_and_build cascade**: `predict_full_done` additionally sets
  `refine_done=True` and increments `refine_count` (Python-only; no
  YAML `history_detection` entry) because a full `predict_and_build`
  run includes refinement internally.

### _clear_zombie_done_flags: crash recovery

After `_analyze_history`, `detect_workflow_state` calls
`_clear_zombie_done_flags(history_info, available_files)`.

A **zombie state** occurs when the agent crashed mid-cycle or the user
deleted output files: history records `done_flag=True`, but the output
file is missing from disk. The phase detector sees `done=True` and
skips the program; the workflow becomes stuck.

The function checks each crashable done flag against its expected
output filename pattern:

| Done flag | Expected output | Also clears | Any-PDB fallback |
|-----------|----------------|-------------|-----------------|
| `resolve_cryo_em_done` | `denmod*.ccp4/mrc` | `has_full_map` | No |
| `predict_full_done` | `*_overall_best.pdb` | `has_placed_model` | Yes |
| `dock_done` | `*docked*.pdb`, `*dock*map*.pdb`, `placed_model*.pdb`, `*_placed*.pdb` | `has_placed_model` | Yes |
| `refine_done` | `*_refine_NNN.pdb` | decrements `refine_count` | Yes |
| `rsr_done` | `*_real_space_refined*.pdb` | decrements `rsr_count` | Yes |

**Any-PDB fallback**: when `True`, the presence of any `.pdb` file on disk suppresses
zombie detection even if the pattern doesn't match. Used for model-producing programs
where output may be renamed. `resolve_cryo_em_done` uses `False` because `denmod_map.ccp4`
is referenced by name downstream.

Flags are cleared **in-memory only** — history is never rewritten.
The function returns a `zombie_diagnostics` list which `detect_workflow_state`
attaches to the state dict under the key `"zombie_diagnostics"`. The PERCEIVE
node logs each entry prefixed `"PERCEIVE: ZOMBIE STATE — "`. This makes
crash/restart re-runs self-explaining: the user can see exactly which done flag
was cleared and why the program appears again.

## Best Files Tracking

The agent maintains a **Best Files Tracker** that identifies and tracks the highest-quality file of each type throughout a session, ensuring programs always receive optimal inputs.

### Categories Tracked

| Category | Description | Example |
|----------|-------------|---------|
| `model` | Best atomic model (PDB/mmCIF) | `refine_002_001.pdb` |
| `map` | Best full cryo-EM map | `denmod_map.ccp4` |
| `mtz` | Best reflection data | `data_with_rfree.mtz` |
| `map_coefficients` | Best map coefficients | `refine_001_001.mtz` |
| `sequence` | Sequence file | `sequence.fa` |
| `ligand_cif` | Ligand restraints | `LIG.cif` |

### Scoring System

Each file is scored based on **processing stage** (0-100 points) and **quality metrics** (0-100 points). All scoring parameters are configurable in `knowledge/metrics.yaml` under `best_files_scoring`.

**Model stage scores:** refined=100, rsr_output=100, autobuild_output=80, docked=60, processed_predicted=50, predicted=40, phaser_output=30, pdb=10

**Model metric scores:** R-free (40 pts, linear_inverse), Map CC (30 pts, linear), Clashscore (30 pts, linear_inverse)

**Map stage scores:** optimized_full_map=100, sharpened=90, density_modified=80, full_map=50, half_map=10. Plus resolution bonus (0-30 pts).

### R-free MTZ Locking (X-ray Only)

Once an MTZ with R-free flags is identified, it is **locked for the entire session**. This ensures consistent cross-validation statistics throughout refinement. The locked MTZ has absolute priority (Priority 0) over all other file selection mechanisms.

### File Selection Priority

When building commands, files are selected in this order:

0. **Locked R-free MTZ** — For MTZ inputs in X-ray refinement (highest priority)
1. **Best Files** — From the BestFilesTracker scoring system
2. **Categorized Files** — From workflow_state file categorization
3. **Extension Matching** — Search available_files by extension

### Key Files

| File | Role |
|------|------|
| `agent/best_files_tracker.py` | Core tracker class with scoring and STAGE_TO_PARENT mapping |
| `agent/session.py` | Integration with session persistence; supplemental file discovery |
| `agent/template_builder.py` | Uses best_files for command building |
| `agent/file_utils.py` | `classify_mtz_type()` for MTZ classification; `matches_exclude_pattern()` for word-boundary pattern matching |
| `agent/workflow_state.py` | `_pdb_is_small_molecule()` and `_pdb_is_protein_model()` for content-based PDB analysis; MTZ categorization safety net in `_categorize_files()` |
| `knowledge/metrics.yaml` | Scoring configuration (best_files_scoring section) |

### Companion File Discovery

Some clients only track a subset of program output files. The agent discovers
missing companion files in multiple layers:

**Layer 0: `session._discover_cycle_outputs()` (v112.73)** — The foundation
layer.  All other layers depend on files appearing in `available_files`.  This
method resolves output files for any cycle using three strategies:

| Strategy | When it helps |
|---|---|
| Try stored `output_files` paths as-is | Normal operation |
| Resolve relative paths against `_get_session_dir()` | cwd changed between runs |
| Scan `sub_{NN}_{program}/` by cycle number + program name | output_files completely empty |

`_get_session_dir()` returns `os.path.dirname(session_file)` — always known,
never depends on stored file paths.  The `get_available_files()` Step 3
directory scan is also seeded from the session directory, not just from
already-tracked files.  This means even if Steps 1-2 find nothing, the scan
still runs.

Without this layer, all downstream recovery (categorization safety nets,
best_files tracking, etc.) is irrelevant because the files were never in the
working set.

**Layer 1: `session._find_missing_outputs()`** — Runs in
`get_available_files()` after Layer 0.  Derives companion files from
known output file names (e.g., if `refine_001_data.mtz` is found,
looks for `refine_001.mtz`).  Supplements Layer 0's directory scan
with pattern-based inference.

**Layer 2: Best files evaluation (v112.70)** — Both
`_rebuild_best_files_from_cycles` (session load) and `record_result`
(live cycle completion) call `_find_missing_outputs` and evaluate
supplemental files through the best_files tracker. This ensures
`best_files["map_coeffs_mtz"]` is populated even when the client
only tracked `refine_001_data.mtz` in `output_files`. Without this
layer, programs with `require_best_files_only: true` (like
ligandfit's map_coeffs_mtz slot) would fail to build because the
map coefficients MTZ was never evaluated.

**Layer 3: MTZ categorization safety net (v112.71)** — Runs at the
end of `_categorize_files()` after both YAML and hardcoded
categorization paths. Cross-checks every MTZ file against the
authoritative `classify_mtz_type()` regex and corrects three types
of misclassification:

| Failure Mode | Detection | Correction |
|---|---|---|
| File in `data_mtz`, should be `map_coeffs_mtz` | `classify_mtz_type()` returns `map_coeffs_mtz` but file not in that category | Move to `map_coeffs_mtz` + subcategory, remove from `data_mtz` |
| File in BOTH `data_mtz` and `map_coeffs_mtz` | YAML Step 1 extension match + Step 2 pattern match create dual membership | Remove from `data_mtz` (prevents `exclude_categories` rejection) |
| File in `map_coeffs_mtz`, should be `data_mtz` | `classify_mtz_type()` returns `data_mtz` but file in `map_coeffs_mtz` | Move to `data_mtz`, remove from `map_coeffs_mtz` and subcategories |

All corrections are logged at `WARNING` level via Python's logging module.

**Layer 4: Self-contained MTZ classification (v112.73)** — The Layer 3 safety
net depends on `classify_mtz_type()` from `file_utils.py`.  Three deployment
failures can disable it: (a) `get_mtz_stage` not deployed (joint import kills
`classify_mtz_type` too), (b) `file_utils.py` entirely missing, (c) YAML
patterns incomplete.  `_import_mtz_utils()` eliminates all three by **always
returning working functions**.  It tries importing from `file_utils.py` first,
then falls back to inline implementations that embed the refine-output regex
directly in `workflow_state.py`.  No external dependency can break it.

**Layer 5: Principled exclusion rule (v112.73)** — Defense-in-depth in the
command builder for dual-categorization (file in both `data_mtz` and
`map_coeffs_mtz`).  `_should_exclude()` implements the rule: **exclude only
if the file is in an excluded category AND NOT in any desired category**.
With Layer 4 working, dual-categorization is cleaned up before it reaches the
command builder.  This layer catches partial failures where the safety net adds
a file to `map_coeffs_mtz` but doesn't remove it from `data_mtz`.

### MTZ Categorization Diagnostics

Two logging points help diagnose map_coeffs_mtz failures:

1. **`perceive()` node**: After file categorization, logs all MTZ category
   contents: `PERCEIVE: MTZ categories: data_mtz=[...]; map_coeffs_mtz=[...]`.
   Warns if refinement is in history but `map_coeffs_mtz` is empty.

2. **`_categorize_files()` safety net**: Logs `WARNING` when it corrects a
   misclassification (e.g., `MTZ safety net: moved refine_001_001.mtz from
   data_mtz to refine_map_coeffs/map_coeffs_mtz`).

### Intermediate File Filtering

`graph_nodes._filter_intermediate_files()` removes
temporary/intermediate files before categorization.  Runs in the
perceive node after history injection.

**Filtered patterns:**
- Files in `/TEMP`, `/temp`, `/TEMP0/`, `/scratch/` directories
- Files with `EDITED_` or `TEMP_` prefixes

These are internal working files from programs like ligandfit that should never
be used as inputs to other programs.

### Best Files Exclusion Check

When `best_files["model"]` is applied as a model override for refinement
programs, it is first checked against the program's `exclude_categories`. If
the best model is in an excluded category (e.g., `ligand_fit_1.pdb` in
`ligand_fit_output` → parent `ligand`), it is skipped and category-based
selection runs instead. This prevents ligand fragments from being used as the
protein model for refinement.

Applied in `command_builder.py` at two points: (1) pre-population of the model
slot from best_files, and (2) LLM override where best_files would normally
take precedence over the LLM's model choice.

### Content-Based File Selection Guards (v112.70)

Three layers of defense prevent files from being assigned to wrong slots:

**Layer 1: `exclude_patterns` (YAML-driven, word-boundary matching)**
Slot definitions in `programs.yaml` specify `exclude_patterns` to reject files
by name. Uses `matches_exclude_pattern()` with word-boundary semantics: `ligand`
matches `atp_ligand.pdb` but NOT `nsf-d2_noligand.pdb`. Applied to both
auto-fill and LLM-selected files.

**Layer 2: Content-based PDB analysis**
After pattern matching, PDB files are checked by content:
- **Model slots** (`model`, `protein`, `pdb_file`): `_pdb_is_small_molecule()`
  rejects HETATM-only files (ligands like `atp.pdb`)
- **Ligand slot**: `_pdb_is_protein_model()` rejects files with ATOM records
  (protein models like `refine_001.pdb`)

Uses `_pdb_is_protein_model()` for the ligand slot rather than
`not _pdb_is_small_molecule()` because the latter returns False for unreadable
files, which would incorrectly reject valid candidates.

**Layer 3: LLM selection validation (v112.70)**
LLM file hint assignments are now validated against the slot's `exclude_patterns`
before acceptance. Previously, the LLM could bypass exclusion rules by explicitly
assigning a file. Now the same guards apply to all file sources:

```
File selection pipeline (all three apply at each stage):
  1. LLM hint → validate against exclude_patterns + content guards
  2. Auto-fill → apply exclude_patterns + content guards + prefer_patterns
  3. Safety net → apply exclude_patterns + content guards + prefer_patterns
```

Applied in:
- `CommandBuilder._find_file_for_slot` (server-side auto-fill)
- `CommandBuilder` LLM hint validation loop
- `_inject_missing_required_files._find_candidate_for_slot` (client-side safety net)

### Ligand Category Content Validation (v112.74)

The YAML categorizer (`file_categories.yaml`) may misclassify protein PDB files
as `ligand_pdb` when broad filename patterns match names like `1aba.pdb` or
`3gx5.pdb`.  A protein with a few HETATM ligand/cofactor atoms is still a
macromolecular model, not a ligand coordinate file.

Post-processing guard in `_categorize_files()` validates every `ligand_pdb`
entry using `_pdb_is_protein_model()`.  If a PDB file has >150 coordinate
records and majority ATOM records, it is a false positive and gets moved from
`ligand_pdb`/`ligand` to `unclassified_pdb`/`pdb`/`model`.  Same defense
pattern as the half-map validation guard.

Only runs when `files_local=True` (file content readable on disk).

| File | ATOM | HETATM | Total | is_protein_model | Action |
|------|------|--------|-------|------------------|--------|
| 1aba.pdb (protein+ligand) | 729 | 20 | 749 | True | Rescue → model |
| atp.pdb (pure ligand) | 0 | 31 | 31 | False | Keep in ligand |
| hem.pdb (cofactor) | 0 | 43 | 43 | False | Keep in ligand |

**Defense layers for PDB ligand classification:**

| Layer | What it catches | Where |
|-------|----------------|-------|
| YAML patterns | Name-based (e.g. `*lig*`) | `_categorize_files_yaml` |
| Hardcoded name check | Word-boundary `lig`/`ligand` | `_categorize_files_hardcoded` |
| Content: unclassified→ligand | HETATM-only in unclassified_pdb | Post-processing (existing) |
| Content: ligand→model | Protein in ligand_pdb | Post-processing (v112.74) |
| best_files_tracker | Content + name check | `_is_ligand_file` |

### Duplicate Command Detection

`session.is_duplicate_command()` prevents the agent from repeating commands.
Two-tier detection:

**Tier 1 — Exact match:** Compares normalized commands against all prior
commands (both successful and failed). Catches verbatim retries.

**Tier 2 — Overlap heuristic:** For same-program commands, computes
basename-level token overlap. >80% overlap flags as duplicate. However, file
tokens (basenames with crystallographic extensions) are compared separately:
if the input files differ, the commands are NOT duplicates regardless of overall
token overlap. This prevents iterative refinement (same program, new model)
from being blocked.

When a duplicate is detected, the agent retries through the normal graph path
with `duplicate_feedback` appended to guidelines (up to 3 attempts).

| File | Role |
|------|------|
| `agent/session.py` | `is_duplicate_command()`, `get_all_commands()`, `get_all_failed_commands()` |
| `programs/ai_agent.py` | `_handle_duplicate_check()`, `_build_duplicate_feedback()` |

## Metrics and Stop Conditions

### Quality Assessment

Metrics are evaluated against YAML-defined thresholds:

| Quality | R-free (2.0Å) | Map CC | Anomalous Measurability |
|---------|---------------|--------|-------------------------|
| Good | < 0.25 | > 0.80 | > 0.10 |
| Acceptable | < 0.30 | > 0.70 | > 0.05 |
| Poor | ≥ 0.30 | ≤ 0.70 | ≤ 0.05 |

### Stop Conditions

The workflow stops when:

1. **Success**: Quality metrics reach targets AND validation done
2. **Plateau**: < 0.5% improvement for 2+ consecutive cycles
3. **Hopeless bailout**: R-free > 0.50 after 1+ refinement cycle
4. **Hard limit**: 3+ refinement cycles regardless of R-free
5. **Validation Gate**: Must run molprobity before stopping if R-free is good

### Refinement Loop Enforcement

When `_is_at_target()` returns True (conditions 2-4 above),
`get_valid_programs()` actively **removes** `phenix.refine` and
`phenix.real_space_refine` from valid programs in both `validate` and
`refine` phases, and adds `STOP`. This prevents the LLM from selecting
refinement even when it appears as a phase-preferred program.

**Exception:** `needs_post_ligandfit_refine` always allows refinement.
After ligand fitting changes the model, re-refinement is scientifically
required regardless of the current cycle count or R-free value.

The validation gate prevents stopping without validation: if R-free is below the success threshold or 3+ refinement cycles have completed, STOP is removed from valid_programs until validation runs.

## Client-Server Update Model

Understanding which code runs where is essential for planning fixes and knowing
whether a deployed fix reaches existing users without requiring them to reinstall.

### The Core Principle

**Decisions and knowledge → server. Execution and I/O → client.**

The client's job is: receive user input → serialize it → send to server → receive
a command string → run that command string locally. Everything in between happens
on the server.

### Execution split diagram

```
┌──────────────────────────────────────────────────────────────────┐
│                        CLIENT (user install)                      │
│                                                                   │
│  run()  ──→  run_job_on_server_or_locally()                      │
│                     │                                             │
│                     ├── run_job_on_server()  ─── serialize ──────┐│
│                     │   _inject_user_params()  ← RUNS CLIENT-SIDE││
│                     │   _run_single_cycle()    ← RUNS CLIENT-SIDE││
│                     │   phenix program execution                  ││
│                     │   GUI components (wxGUI2/)                  ││
│                     │   .eff / .pkl file generation               ││
│                     │                                             ││
│                     └── run_job_locally()  ─── (has local DB) ───┘│
└────────────────────────────────────────────────────────────────── │
                                           HTTP / REST               │
                                               ▼                     │
┌──────────────────────────────────────────────────────────────────┐│
│                        SERVER (your install)                      ││
│                                                                   ││
│  run_ai_agent.run()                                               ││
│     └── LangGraph pipeline                                        ││
│          ├── PERCEIVE  (graph_nodes.py)                           ││
│          ├── PLAN      (graph_nodes.py + prompts_hybrid.py)       ││
│          ├── BUILD     (command_builder.py + program_registry.py) ││
│          ├── VALIDATE  (graph_nodes.py)                           ││
│          └── OUTPUT    (graph_nodes.py)                           ││
│                                                                   ││
│  Knowledge layer                                                  ││
│     ├── knowledge/programs.yaml                                   ││
│     ├── knowledge/workflows.yaml                                  ││
│     ├── knowledge/prompts_hybrid.py                               ││
│     └── agent/workflow_engine.py, workflow_state.py, session.py  ││
└───────────────────────────────────────────────────────────────────┘│
```

### Always server-side (no user action needed)

**LLM decision-making — the entire graph**
Any change to how the agent thinks: PERCEIVE, PLAN, BUILD, VALIDATE, ACT nodes,
workflow routing, phase detection, placement logic, error recovery. Users get
this immediately.

**Prompts and knowledge**
`prompts_hybrid.py`, `programs.yaml`, `workflows.yaml`. Any improvement to how
programs are described to the LLM, new invariants, strategy flag fixes, or stop
conditions — effective immediately.

**Command construction**
`command_builder.py`, `program_registry.py`. Changes to passthrough filtering,
invariant application, file selection — all server-side.

**Session and history logic**
`session.py`, `workflow_engine.py`, `workflow_state.py`. How history is
interpreted, context built, S2c promotion, zombie detection — all server-side.

**API schema and transport encoding**
Changes to the request/response format take effect on the server. The client
passes opaque encoded strings through without inspecting them.

### Always client-side (requires user to update their install)

**Post-command injection — `_inject_user_params()`**
This function runs *after* the server returns a command, on the user's machine.
It can corrupt an otherwise-correct server-built command. This is the most
dangerous client-side code path because server fixes alone cannot protect against it.

**The local execution loop**
`_run_single_cycle()`, `_get_command()`, the `for cycle in range(...)` loop,
`iterate_agent`. These control how commands get run, retried, and logged locally.

**Phenix program execution**
`_try_native_execution()`, `_run_easy_run()`, subprocess handling, output
capture. The server does not run phenix programs; the client does.

**GUI components**
`wxGUI2/Programs/DockInMap.py` and all other `wxGUI2` files. Restoration,
display widgets, and result panels are entirely client-side.

**The .eff and .pkl generation**
`generate_program_eff()`, DataManager PHIL mapping. File writing happens on the
client.

**The top-level branching**
`run()`, `run_job_on_server_or_locally()`, `run_job_on_server()`. The logic that
decides whether to call the server at all. A bug here means users might not
reach the server.

### The gray area: `programs/ai_agent.py`

This single file straddles both worlds. The **top half** (through
`run_job_on_server()`) runs client-side. The **bottom half** (when the server
receives `run_on_server=False`) runs server-side. Changes in this file need user
updates if in the client path, and don't if in the server path.

**Practical rule:** if logic can live in `agent/`, `knowledge/`, or
`phenix_ai/run_ai_agent.py` rather than in the client path of `ai_agent.py`,
prefer that location — it turns a required user update into a free server update.
The `_inject_user_params` bug (S2k) is a textbook example: if post-injection
filtering were done server-side before returning the command string, no user
update would ever be needed for that class of bug.

### Practical decision rule for new fixes

When writing a fix, ask: *does this code run before or after
`run_job_on_server()` is called?*

- **Before** (serialization, call site, result handling) → client-side → user must update
- **After** (on the machine that received `run_on_server=False`) → server-side → free

### Summary table

| Component | Runs on | Update required? |
|---|---|---|
| `knowledge/prompts_hybrid.py` | Server | No |
| `knowledge/programs.yaml` | Server | No |
| `knowledge/workflows.yaml` | Server | No |
| `agent/graph_nodes.py` | Server | No |
| `agent/command_builder.py` | Server | No |
| `agent/program_registry.py` | Server | No |
| `agent/workflow_engine.py` | Server | No |
| `agent/workflow_state.py` | Server | No |
| `agent/session.py` | Server | No |
| `phenix_ai/run_ai_agent.py` | Server | No |
| `programs/ai_agent.py` — `_inject_user_params()` | **Client** | **Yes** |
| `programs/ai_agent.py` — `_run_single_cycle()` | **Client** | **Yes** |
| `programs/ai_agent.py` — `run_job_on_server()` | **Client** | **Yes** |
| `programs/ai_agent.py` — `run()` top-level | **Client** | **Yes** |
| `wxGUI2/Programs/DockInMap.py` | **Client** | **Yes** |
| `phenix_ai/remote_agent.py` | **Client** | **Yes** |

---

## Extension Points

### Adding a New Program

See [ADDING_PROGRAMS.md](../guides/ADDING_PROGRAMS.md) for the complete guide. In summary:

1. Add program definition to `knowledge/programs.yaml` (inputs, outputs, log_parsing)
2. Add to appropriate workflow phase in `knowledge/workflows.yaml`
3. Add file categories to `knowledge/file_categories.yaml` (if new file types)
4. Add hardcoded extractor to `phenix_ai/log_parsers.py` (only if YAML patterns insufficient)

### Adding a New Workflow

1. Define states in `workflow_state.py`
2. Add transitions in workflow YAML
3. Update rules selector if needed

### Adding New Session State

1. Add to `session_state` in request schema
2. Map to `session_info` in `_process_request()`
3. Use in graph nodes as needed

## Performance Considerations

### Graph Caching

```python
_cached_graph = None

def _get_graph():
    global _cached_graph
    if _cached_graph is None:
        _cached_graph = build_agent_graph()
    return _cached_graph
```

### Request Size

- Log content can be large (truncate if needed)
- History grows with cycles (summarize old cycles)
- Files list should only include relevant files

### Response Time

- Graph invocation: ~1-5 seconds
- LLM calls: ~2-10 seconds (if used)
- Rules-only mode: <1 second

## Command Builder Architecture

The CommandBuilder provides a unified, single-entry-point system for generating
PHENIX commands. It replaces the previous fragmented approach spread across
multiple modules.

### Pipeline

```
┌──────────────────────────────────────────────────────────────────────────────┐
│                         CommandBuilder.build()                               │
├──────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌──────────┐  ┌──────────┐  ┌───────────┐  ┌──────────┐  ┌─────────────┐  │
│  │_select_  │─▶│_build_   │─▶│_apply_    │─▶│_assemble_│─▶│ _inject_    │  │
│  │ files()  │  │strategy()│  │invariants()│  │command() │  │ recovery()  │  │
│  └──────────┘  └──────────┘  └───────────┘  └──────────┘  └─────────────┘  │
│       │              │              │              │              │          │
│       ▼              ▼              ▼              ▼              ▼          │
│  Priority order:  Auto-fill:    Auto-fill:    Template-      Append any     │
│  1. LLM hints    - output_pfx  - resolution   based cmd      recovery-     │
│  2. Locked rfree - from hist   - R-free flags  string         sourced       │
│  3. Best files                 - twin_law                     strategy      │
│  4. Categories                                                entries not   │
│  5. Extensions                                                in command    │
│                                                                              │
│  ↕ _apply_recovery_strategies() runs between _build_strategy and            │
│    _apply_invariants, adding recovery flags to the strategy dict.           │
│    Recovery entries are tagged with strategy_sources[key]="recovery".        │
└──────────────────────────────────────────────────────────────────────────────┘
```

The recovery injection step is necessary because `registry.build_command()` only
emits strategy entries matching `strategy_flags` in programs.yaml.  Recovery
params use fully-qualified PHIL paths (e.g., `scaling.input.xray_data.obs_labels`)
that don't match short strategy_flags names.  The post-assembly injection ensures
they reach the final command.  See "Automatic Error Recovery" section for details.

### CommandContext

All parameters are passed via a `CommandContext` dataclass:

```python
@dataclass
class CommandContext:
    cycle_number: int = 1
    experiment_type: str = ""
    resolution: float = None
    best_files: Dict[str, str] = None
    rfree_mtz: str = None
    categorized_files: Dict[str, List[str]] = None
    workflow_state: str = ""
    history: List[Dict] = None
    llm_files: Dict[str, str] = None      # Optional LLM hints
    llm_strategy: Dict[str, Any] = None   # Optional LLM hints
    log: Callable = None
```

### File Selection Priority

1. **LLM hints** - If LLM specified files and they exist
2. **Locked R-free MTZ** - For MTZ slots in X-ray refinement
3. **Best files** - From BestFilesTracker (quality-scored)
4. **Categories** - From `input_priorities` in YAML
5. **Extension fallback** - Most recent file with matching extension

**Best files exclusion:** Before applying `best_files["model"]` as a model
override, the system checks if the file is in a program's `exclude_categories`.
E.g., `ligand_fit_1.pdb` (in `ligand_fit_output` → `ligand`) is excluded from
refine's model slot, falling through to category-based selection.

### Command Validation

The `validate` graph node checks that all file references in a generated command
exist in `available_files`. Output arguments (`output.file_name=X.pdb`,
`output.prefix=Y`) are stripped before extraction since they reference files
that don't exist yet. Duplicate commands and empty commands are also rejected.

### Integration with Graph Nodes

```python
# In graph_nodes.py build() function:
if USE_NEW_COMMAND_BUILDER:
    return _build_with_new_builder(state)

# _build_with_new_builder creates CommandContext from state
# and delegates to CommandBuilder.build()
```

## Automatic Error Recovery

When a PHENIX program fails with a recognized error pattern, the agent can automatically
recover and retry with corrected parameters. This is handled by `agent/error_analyzer.py`
with patterns defined in `knowledge/recoverable_errors.yaml`.

### How It Works

1. **Detection**: After a program fails, the error output is matched against
   `detection_patterns` in `recoverable_errors.yaml`
2. **Extraction**: Regex patterns extract the parameter name and available choices
   from the error message
3. **Resolution**: A strategy selects the best choice based on program type and
   workflow context (e.g., anomalous vs merged data)
4. **Storage**: The resolution is saved as a file-keyed recovery strategy in the
   session (`session.set_recovery_strategy`), and `force_retry_program` is set
5. **Retry**: On the next cycle, `force_retry` bypasses the LLM's program choice,
   the command builder applies recovery flags, and the command is re-executed

### Recovery Parameter Injection (v112.74)

Recovery strategies add fully-qualified PHIL parameters to the command builder's
strategy dict (e.g., `scaling.input.xray_data.obs_labels=I(+)` for ambiguous
data labels).  These must survive two stages to reach the final command:

**Stage 1: build_command template expansion.**  `registry.build_command()` only
emits strategy entries matching `strategy_flags` keys in programs.yaml.  Recovery
params use fully-qualified PHIL paths that don't match short flag names.  Fix:
after `_assemble_command` returns, any strategy entry sourced from recovery
(`strategy_sources[key] == "recovery"`) whose key is not already present in the
command is appended as `key=value`.

**Stage 2: probe-only sanitizer.**  Programs in `_PROBE_ONLY_FILE_PROGRAMS`
(xtriage, model_vs_data, etc.) have all non-file-path key=value tokens stripped.
Data-label selection parameters (`obs_labels`, `labels`, `data_labels`,
`anomalous_labels`, `r_free_flags_labels`) are whitelisted because they come
exclusively from error recovery for ambiguous MTZ arrays.

### Recovery Loop Guard (v112.74)

Before storing a new recovery strategy, the system checks if one already exists
for the same file (`session.get_recovery_strategy(file)`).  If so, the previous
recovery attempt didn't work — re-triggering would create an infinite loop.
The recovery is skipped and the failure falls through to the terminal diagnosis
path.

### Duplicate Check Bypass for Recovery (v112.74)

Recovery retries re-run the same program, often with commands that look like
duplicates of the failed command.  When `forced_retry` is detected in
`decision_info` (or fallback: reasoning text contains recovery marker), the
duplicate check in `_handle_duplicate_check` is skipped entirely.  The
`forced_retry` flag propagates from `state['intent']` through the BUILD node's
return dict to the top-level graph output.

### Currently Handled Errors

- **`ambiguous_data_labels`**: MTZ file contains multiple data arrays (e.g., both
  merged intensities and anomalous pairs). The agent picks the right array based on
  whether the program needs anomalous or merged data.
- **`ambiguous_experimental_phases`**: MTZ file contains multiple phase arrays.
  The agent selects standard vs anomalous HL coefficients based on context.

### Configuration

`knowledge/recoverable_errors.yaml` defines:
- **`errors`**: Detection patterns, extraction regexes, resolution strategies, max retries
- **`label_patterns`**: How to classify data labels as anomalous or merged
- **`program_data_preferences`**: Which programs need anomalous vs merged data
- **`context_keywords`**: Workflow hints from user advice (e.g., "SAD" → anomalous)

### Adding a New Recoverable Error

1. Add an entry under `errors:` in `recoverable_errors.yaml` with detection patterns
   and extraction regexes
2. Add a resolution handler in `error_analyzer.py` (`resolve_error()` method)
3. The fallback node in the graph will automatically use the new pattern

### Recovery strategies persist intentionally

After a program succeeds with a selected label (e.g., `obs_labels=I(+)`), the
recovery strategy for that MTZ file is NOT cleared.  The ambiguity is a property
of the MTZ file, not the program.  Downstream programs (phaser, refine) that use
the same MTZ also need the label selection.

### Key Files

- `knowledge/recoverable_errors.yaml` — Error patterns and resolution config
- `agent/error_analyzer.py` — Detection, extraction, and resolution logic
- `agent/graph_nodes.py` — Fallback node triggers error analysis on failure
- `agent/command_builder.py` — Recovery param injection after `_assemble_command`
- `agent/command_postprocessor.py` — Label param whitelist in probe-only sanitizer
- `programs/ai_agent.py` — Loop guard, duplicate check bypass, `force_retry` handling

---

## Diagnosable Terminal Errors

Some program failures are categorically unrecoverable — for example, a crystal symmetry
mismatch between input files, or a model entirely outside the cryo-EM map. Rather than
stopping silently or showing a generic error, the agent detects these, asks the LLM to
diagnose the specific cause, and presents a self-contained HTML report to the user.

### Overview

When a recognized terminal failure occurs the agent:

1. Identifies the error type via `DiagnosisDetector` (pattern matching against
   `knowledge/diagnosable_errors.yaml`)
2. Calls the LLM on the server (`analysis_mode=failure_diagnosis`) with the error
   excerpt, a domain hint from the YAML, and the tail of the failing program's log
3. Writes `ai_failure_diagnosis.html` to the job's working directory
4. Opens the report in the user's browser automatically
5. Returns `True` from `_run_single_cycle` to stop the cycle loop cleanly —
   **no `Sorry` exception is raised**; the ai_agent job is considered successful
   because it correctly identified and diagnosed the sub-job failure
6. `_finalize_session` runs unconditionally (saves the session, populates
   `self.result`), but **skips the Results summary page** — the diagnosis HTML
   is the user's sole output window, so a second page would only bury it

### Error flow diagram

```
Program fails
      │
      ▼
DiagnosisDetector.detect(result_text)
      │
      ├── No match → normal failure handling (agent may retry or move on)
      │
      └── Match found: (error_type, description, excerpt)
                │
                ▼
       _diagnose_terminal_failure()
                │
                ├── 1. Read log tail (client-side, 150 lines / 4 000 chars)
                ├── 2. LLM call: analysis_mode=failure_diagnosis
                │         └── fallback: rules-only text if LLM unavailable
                ├── 3. Build & write ai_failure_diagnosis.html
                │         ├── Heading: "Error diagnosis"
                │         ├── Meta bar: program, cycle, job name, working dir
                │         ├── Error excerpt (red box)
                │         ├── AI diagnosis (three paragraphs: what/cause/fix)
                │         └── Footer: full path to the saved file
                ├── 4. Open HTML in browser (load_url)
                ├── 5. Store path in session.data["failure_diagnosis_path"]
                ├── 6. Print diagnosis to CLI log
                └── 7. return True  ← cycle loop breaks; no Sorry raised

_finalize_session()
      │
      ├── Saves session JSON
      ├── Detects failure_diagnosis_path in session.data
      └── Skips _generate_ai_summary() and display_results()
              (Results page suppressed; diagnosis page is the output)
```

### HTML report content

The report (`ai_failure_diagnosis.html`) is self-contained — no CDN dependencies —
so it renders correctly when opened directly from a local path. It includes:

| Section | Content |
|---------|---------|
| Header | "⚠ PHENIX AI Agent — Error diagnosis" |
| Meta bar | Failed program, cycle number, job name, working directory |
| Error | Human-readable error type label + the raw error excerpt |
| AI Diagnosis | Three plain-text paragraphs: WHAT WENT WRONG / MOST LIKELY CAUSE / HOW TO FIX IT |
| Footer | Full path where the file is saved |

### Configuration

`knowledge/diagnosable_errors.yaml` defines each recognizable error type:

```yaml
errors:
  crystal_symmetry_mismatch:
    display_name: "Unit cell or space group mismatch between input files"
    detection_patterns:
      - "crystal symmetry mismatch"
      - "incompatible crystal symmetry"
    diagnosis_hint: |
      Check that all input files were processed from the same crystal form.
      ...
```

Fields:

| Field | Purpose |
|-------|---------|
| `display_name` | Human-readable label shown on the error page and in the log |
| `detection_patterns` | Strings that, if found in the failing program's output, trigger diagnosis |
| `diagnosis_hint` | Domain knowledge injected into the LLM prompt to guide the diagnosis |

### Adding a new diagnosable error

1. Add an entry in `knowledge/diagnosable_errors.yaml` with `display_name`,
   `detection_patterns`, and a `diagnosis_hint`
2. No code changes required — `DiagnosisDetector` and `_diagnose_terminal_failure`
   handle all new entries automatically
3. Add a detection test in `tests/tst_audit_fixes.py` following the `test_s3a_detect_*`
   pattern

### Key files

| File | Role |
|------|------|
| `knowledge/diagnosable_errors.yaml` | Error definitions and LLM hints |
| `agent/error_analyzer.py` | `DiagnosisDetector` — pattern matching and hint lookup |
| `agent/failure_diagnoser.py` | Prompt builder, Markdown sanitiser, HTML report builder |
| `programs/ai_agent.py` | `_diagnose_terminal_failure()` — orchestrates steps 1–7 |
| `tests/tst_audit_fixes.py` | `test_s3a_*` tests covering detection, HTML output, UX flow |

---

## Session Summary Generation

The agent generates structured summaries of completed sessions with optional LLM assessment.

### Summary Components

1. **Header**: Run name derived from the working directory basename
2. **Session line**: Session ID, cycle count (successful / total), and **working directory path**
3. **Input Section**: Files, user advice, experiment type, resolution
4. **Input Data Quality**: Metrics from xtriage/mtriage (resolution, completeness, twinning)
5. **Workflow Path**: High-level description of strategy taken
6. **Steps Performed**: Table of all cycles with key metrics
7. **Final Quality**: Final metrics with quality assessments
8. **Key Output Files**: Output files from the session with relative paths
9. **Failure Diagnosis** *(only on fatal error)*: Path to `ai_failure_diagnosis.html` — shown when the session ended due to a diagnosable terminal failure; see [Diagnosable Terminal Errors](#diagnosable-terminal-errors)
10. **Assessment** (optional): LLM-generated evaluation

> **Note**: When a fatal error diagnosis was produced, the Results summary page is suppressed entirely — `_finalize_session` detects `session.data["failure_diagnosis_path"]` and skips `_generate_ai_summary()`. The diagnosis HTML is the user's sole output window.

### Quality Assessments

Metrics are automatically assessed based on resolution-dependent thresholds:

| Metric | Good | Acceptable | Needs Improvement |
|--------|------|------------|-------------------|
| R-free (2.5Å) | ≤0.25 | ≤0.30 | >0.30 |
| Map CC | ≥0.80 | ≥0.70 | <0.70 |
| Clashscore | ≤5 | ≤10 | >20 |

### LLM Assessment Prompt

When enabled, the LLM evaluates:
1. **Input Data Quality**: Resolution, completeness, issues
2. **Goal and Strategy**: User's goal and agent's approach
3. **Strategy Assessment**: Was it appropriate? Goal achieved?
4. **Current Status**: Ready for deposition?
5. **Next Steps**: Recommendations

### Usage

```python
# Generate summary without LLM
result = session.generate_agent_session_summary(include_llm_assessment=False)
print(result["markdown"])

# Get concise summary for LLM assessment
llm_input = session.get_summary_for_llm_assessment()
```

### Files

- `agent/session.py`: `generate_agent_session_summary()`, `get_summary_for_llm_assessment()`
- `analysis/agent_session_analyzer.py`: LLM assessment integration
- `phenix_ai/run_ai_analysis.py`: `run_agent_session_analysis()`
- `knowledge/prompts_hybrid.py`: Assessment prompt template

---

## Advice Preprocessing

The agent preprocesses user input to create structured, actionable guidance, with
built-in protection against prompt injection attacks and automatic change detection
for mid-session updates.

### Components

| Component | File | Function |
|-----------|------|----------|
| README discovery | `agent/advice_preprocessor.py` | `find_readme_file()` |
| README reading | `agent/advice_preprocessor.py` | `read_readme_file()` |
| Advice gathering | `agent/advice_preprocessor.py` | `gather_raw_advice()` |
| Input sanitization | `agent/advice_preprocessor.py` | `sanitize_advice()` |
| LLM preprocessing | `agent/advice_preprocessor.py` | `preprocess_advice()` |
| Change detection | `programs/ai_agent.py` | `_preprocess_user_advice()` |
| Server routing | `phenix_ai/run_ai_analysis.py` | `run_advice_preprocessing()` |

### Sources

1. **Direct advice**: `project_advice` PHIL parameter
2. **README files**: Found in `input_directory` if specified

### Process

1. **Gather**: Collect advice from all sources
2. **Hash**: Compute MD5 hash of raw advice for change detection
3. **Compare**: Check if advice changed from previous run
4. **Sanitize**: Remove potential prompt injection patterns
5. **Combine**: Merge with clear labeling
6. **Preprocess**: Use LLM to structure (optional)
7. **Store**: Save raw, processed, and hash in session

### Advice Change Detection

The agent detects when advice has changed between runs and automatically reprocesses:

- **Same hash** → reuse cached processed advice
- **Different hash** → reprocess advice AND re-extract directives
- **No new advice** → reuse cached version

This enables mid-session advice updates - users can provide new instructions
(like "stop" or "focus on the ligand") and have them take effect immediately.

### Advice Change Detection

The agent detects when advice has changed between runs and automatically reprocesses:

- **Same hash** → reuse cached processed advice
- **Different hash** → reprocess advice AND re-extract directives
- **No new advice** → reuse cached version

This enables mid-session advice updates - users can provide new instructions
(like "stop" or "focus on the ligand") and have them take effect immediately.

#### Extending a Completed Workflow (Q1)

A special case arises when the workflow has already reached the `complete` phase
and the user resumes with new advice. Two independent mechanisms both need to be
active to allow follow-up work:

**Wall 1 — AUTO-STOP in PLAN (pre-existing):**
When `metrics_trend.should_stop` is True, the PLAN node normally terminates
immediately. The `advice_changed` flag suppresses this for exactly one cycle,
letting the LLM plan before reverting to normal termination.

**Wall 2 — `valid_programs = ['STOP']` in PERCEIVE (Q1 fix):**
The `complete` phase returns `['STOP']` from `get_valid_programs`. Even with
Wall 1 down, the LLM is presented with a menu of only STOP and cannot select
any follow-up program. Fix: when `advice_changed=True` AND `phase=='complete'`,
PERCEIVE steps back to the `validate` phase, which contains:

```
['phenix.polder', 'phenix.molprobity', 'phenix.model_vs_data',
 'phenix.map_correlations', 'STOP']
```

After one successful cycle, `advice_changed` is cleared in the post-execution
check and normal termination logic resumes on the next cycle.

**Complete flow on resume with new advice:**

```
_preprocess_user_advice()
  ↓ new hash ≠ stored hash → session.data["advice_changed"] = True

PERCEIVE (graph_nodes.py)
  ↓ phase == 'complete' AND advice_changed
  ↓ valid_programs ← get_valid_programs(exp, {"phase":"validate"}, ctx)
  ↓ phase_info ← {"phase": "validate", "reason": "advice_changed: stepped back"}

PLAN (graph_nodes.py)
  ↓ metrics_trend.should_stop AND advice_changed → skip AUTO-STOP this cycle
  ↓ LLM: sees new advice + validate-phase program menu → chooses phenix.polder

BUILD / VALIDATE / OUTPUT → polder command executed

Post-execution (iterate_agent.py)
  ↓ session.data["advice_changed"] = False
  ↓ next cycle: normal AUTO-STOP and complete-phase logic resume
```

**Usage:**
```bash
# Completed workflow: xtriage → phaser → refine ×3 → ligandfit → molprobity
phenix.ai_agent \
    log_directory=AIAgent_run1 \
    restart_mode=resume \
    project_advice="also run polder on chain B residue 100"
```

**Key design note:** `phenix.polder` intentionally has no `run_once` strategy
in `programs.yaml`. Different residues and ligands may each require a separate
omit map, so `polder_done=True` does not gate polder out of `valid_programs`.

### Session Management Keywords

Two Phil parameters allow inspecting and modifying an existing session without
running new crystallographic cycles.

#### `display_and_stop`

```bash
phenix.ai_agent log_directory=AIAgent_run1 display_and_stop=basic
phenix.ai_agent log_directory=AIAgent_run1 display_and_stop=detailed
```

Prints the session history (`basic`: one line per cycle; `detailed`: full
reasoning + command), then exits. Populates `self.result` via
`_finalize_session(skip_summary=True)` so `get_results()` / `get_results_as_JSON()`
work identically to a normal run.

#### `remove_last_n`

```bash
phenix.ai_agent log_directory=AIAgent_run1 remove_last_n=2
```

Removes the last N cycles from the session JSON, clears the stale AI summary
(since the cycle set has changed), rebuilds `active_files.json` and `best_files`
from the remaining history, then saves and exits. Useful for pruning a failed
refinement run before re-running.

#### Auto-set `restart_mode=resume`

Both parameters operate on an existing session directory and therefore always
require resume semantics. `run()` automatically forces `restart_mode='resume'`
before `set_defaults()` when either parameter is set, so the user does not need
to remember the flag.

#### `get_results()` safety

`run()` sets `self.result = None` as its very first statement. `get_results()`
uses `getattr(self, 'result', None)` as a defensive fallback. This prevents
`AttributeError` on any early-exit path (session management, red-flag abort,
bad parameters) that would otherwise bypass the normal result-assignment code.

### Input Sanitization

The `sanitize_advice()` function removes potentially malicious patterns:

- Instruction override attempts ("ignore all previous instructions")
- System prompt manipulation (`<system>`, `[system]`)
- Role manipulation ("you are now a...", "act as a...")
- Hidden text (null bytes, control characters)
- Excessive repetition (potential buffer overflow)

```python
from agent.advice_preprocessor import sanitize_advice, is_suspicious

# Check if input contains suspicious patterns
if is_suspicious(user_input):
    clean_input = sanitize_advice(user_input)
```

### README Discovery

Searches for these files (case-insensitive):
- `README`, `README.txt`, `README.dat`, `README.md`
- `notes.txt`, `NOTES.txt`

Files are truncated at `max_readme_chars` (default: 5000).

### LLM Preprocessing

When `preprocess_advice=True` (default), the LLM extracts:

1. **Input Files Found**: Data files mentioned in the text
2. **Experiment Type**: SAD, MAD, MR, cryo-EM, etc.
3. **Primary Goal**: What the user wants to accomplish
4. **Key Parameters**: Wavelength, resolution, sites, heavy atom type
5. **Special Instructions**: Ligands, quality targets, etc.

### PHIL Parameters

```phil
input_directory = None
  .type = path
  .help = Directory containing input files and optional README

preprocess_advice = True
  .type = bool
  .help = Use LLM to preprocess and clarify user advice

readme_file_patterns = README README.txt README.dat README.md notes.txt
  .type = strings
  .help = Filenames to look for when extracting advice

max_readme_chars = 5000
  .type = int
  .help = Truncate README files longer than this
```

### Usage

```bash
# With direct advice
phenix.ai_agent data.mtz seq.fa project_advice="Solve by MR, R-free < 0.25"

# With README in directory
phenix.ai_agent input_directory=/data/project/

# Both combined
phenix.ai_agent input_directory=/data/project/ project_advice="Prioritize geometry"

# Update advice mid-session (on restart)
phenix.ai_agent session_file=session.json project_advice="stop"
```

### Testing

```bash
python tests/tst_advice_preprocessing.py
```

Tests cover sanitization, README discovery, advice combination, file extraction,
and change detection logic.

---

## Event System

The agent uses a structured event system for transparent decision logging.

### Event Flow

```
┌────────────────────────────────────────────────────────────────┐
│                        Graph Nodes                              │
│  perceive() → plan() → build() → validate()                    │
│       │          │         │          │                        │
│       │          │         │     ┌────┴─────┐                  │
│       │          │         │     │fallback()│                  │
│       │          │         │     └────┬─────┘                  │
│       │          │         │          │                        │
│       │          │         │     output_node()                  │
│       ▼          ▼         ▼          ▼                        │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │               state["events"] (list of dicts)            │  │
│  └──────────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌────────────────────────────────────────────────────────────────┐
│              Response Building (run_ai_agent.py)               │
│  response["events"] = state["events"]                          │
│  response["events_as_simple_string"] = json.dumps(events)      │
└────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌────────────────────────────────────────────────────────────────┐
│                    Display (ai_agent.py)                        │
│  formatter = EventFormatter(verbosity=params.verbosity)        │
│  output = formatter.format_cycle(events, cycle_number)         │
│  print(output, file=self.logger)                               │
└────────────────────────────────────────────────────────────────┘
```

### Event Types

| Type | Level | Description |
|------|-------|-------------|
| `cycle_start` | quiet | Cycle beginning |
| `cycle_complete` | quiet | Cycle finished |
| `state_detected` | normal | Workflow state determined |
| `metrics_extracted` | normal | R-free, CC, resolution |
| `metrics_trend` | normal | Improvement/plateau analysis |
| `sanity_check` | normal | Red flag or warning detected |
| `program_selected` | normal | Decision with reasoning |
| `program_modified` | normal | Program changed by rules/validation |
| `stop_decision` | normal | Whether to continue |
| `directive_applied` | normal | User directive enforced |
| `user_request_invalid` | quiet | User request unavailable |
| `files_selected` | verbose | File selection details |
| `file_scored` | verbose | Individual file scoring detail |
| `command_built` | normal | Final command |
| `thought` | verbose | LLM chain-of-thought/reasoning |
| `error` | quiet | Error occurred |
| `warning` | quiet | Non-fatal warning |
| `debug` | verbose | Internal debug information |

### Verbosity Levels

```phil
verbosity = normal
  .type = choice(quiet, normal, verbose)
```

- **quiet**: Errors, warnings, cycle summaries only
- **normal**: Key decisions and metrics (default)
- **verbose**: Full details including file selection, LLM traces, debug info

Note: `debug` is accepted as an alias for `verbose` (3 levels total).

### Implementation Files

- `agent/event_log.py` - EventType, Verbosity, EventLog class
- `agent/event_formatter.py` - EventFormatter class
- `agent/graph_nodes.py` - Event emission (`_emit()` helper)

See [TRANSPARENCY_LOGGING.md](../project/TRANSPARENCY_LOGGING.md) for full details.
