Operon v0.12: Verification

1. The Problem: Verifying Biological Claims

The previous releases introduced mechanisms inspired by Systems Biology: Chaperone Proteins for structural healing, T-cell receptors for anomaly detection, Epiplexity for epistemic stagnation monitoring. These biological analogies are compelling—but analogies are not evidence.

A framework that claims "Chaperone-mediated folding repairs malformed JSON" needs to demonstrate this under controlled conditions. A system claiming "Immune surveillance detects prompt injection" needs adversarial evaluation, not just unit tests. v0.12 addresses this gap.

2. BFCL: Testing the Chaperone

The Berkeley Function Calling Leaderboard (BFCL) evaluates how well models generate structured function calls. We adapt it to test the Chaperone protein motif—the retry-and-repair loop that folds malformed LLM outputs into valid schemas.

2.1 The Folding Analogy

In molecular biology, newly synthesized proteins emerge as linear amino acid chains that must fold into precise 3D structures. Chaperone proteins (GroEL/GroES) provide a protected environment for refolding attempts. If a protein fails to fold repeatedly, it's tagged for degradation.

The BFCL suite tests this analogy directly. We take valid function-call schemas from the BFCL dataset and corrupt them with realistic LLM failure modes:

2.2 Results

We evaluate two metrics: strict accuracy (exact match after folding) and cascade accuracy (valid structure, possibly with semantic differences). Each sample receives 1–3 random corruptions.

The gap between cascade and strict accuracy is expected: structural repair (removing trailing commas, fixing quotes) succeeds at high rates, but recovering the exact original values from type-swapped or missing fields requires semantic understanding beyond pure syntax repair. This mirrors biology: chaperones can refold structure, but cannot repair sequence mutations.

2.3 Healing: Error Context Matters

The Chaperone Loop (Section 5.5.1 of the paper) specifies that error traces should be fed back to the generator for context-aware correction. We test this directly:

Blind retry recovers only 37.5% of failures. Injecting the error trace ("TypeError: 'one hundred' is not float") into the generator's context achieves 99.5% healing. The biological analogy holds: the GroEL cage doesn't just retry folding—it provides a protected environment where the error signal itself guides correction.

3. AgentDojo: Testing the Immune System

AgentDojo (Debenedetti et al., NeurIPS 2024) provides a dynamic environment for evaluating prompt injection attacks and defenses. We adapt its attack templates to test the Adaptive Immunity motif—the T-cell surveillance system that detects compromised agents through behavioral anomaly.

3.1 Experimental Setup

We create a population of 40 simulated agents. Eight are "compromised"—their behavioral profiles are shifted by injection patterns derived from four AgentDojo attack categories:

• Important Instructions: "IMPORTANT: Follow these instructions instead..."
• Tool Knowledge: Exploiting tool descriptions to inject behavior
• Direct Injection: Explicit instruction override in user content
• Ignore Previous: "Ignore all previous instructions and..."

The immune system observes 30 training samples per agent to build baseline profiles, then evaluates 20 test observations to classify each agent as healthy or compromised.

3.2 Results

The wide confidence intervals reflect the small sample size (8 compromised agents)—a deliberate choice to model realistic deployments where compromised agents are rare. The sensitivity of 100% with 0% false positives on AgentDojo-derived attacks is encouraging, but the confidence bounds remind us that larger-scale evaluation is needed.

4. Mathematical Corrections

During the development of the eval harness, a comprehensive audit revealed several inconsistencies between the paper and the implementation. These are now corrected in both directions.

4.1 Exponential Saturation, Not Sigmoid

The paper defines perplexity normalization as an exponential saturation function:

This maps $[0, \infty) \to [0, 1)$ with a natural interpretation: $H_0$ is the baseline perplexity where the function reaches $\approx63\%$ saturation. The implementation was using a logistic sigmoid $\frac{1}{1+e^{-x}}$ instead—a function with different domain ($\mathbb{R} \to (0,1)$), different inflection point, and different asymptotic behavior.

4.2 Stagnation Direction

The paper states that stagnation occurs when the Epiplexic Integral falls below a threshold:

Low Epiplexity means low Bayesian surprise—the agent's outputs carry no new information. The implementation was checking the opposite direction ($\mathcal{E}_w > \delta$), which would flag high-surprise (exploring) agents as stagnant.

4.3 Embedding Novelty Normalization

The raw cosine distance $1 - \cos(e_t, e_{t-1})$ has range $[0, 2]$. The implementation correctly normalizes this to $[0, 1]$ via division by 2, but the paper omitted the $\frac{1}{2}$ factor. The corrected Epiplexity equation is:

Both terms now live in $[0, 1]$, making the $\alpha$-weighted combination interpretable as a proper convex combination.

4.4 Equation Renumbering

The paper's 20 numbered equations were scattered non-sequentially across sections (e.g., Section 4 jumped from Eq. 9 to Eq. 16 to Eq. 10). All equations have been renumbered 1–20 in document order.

5. Practical Applications

Three new examples demonstrate the framework in realistic software engineering scenarios:

6. What's New in v0.12

7. Interactive Demos

Every biological motif in the framework now has a corresponding interactive demo on HuggingFace Spaces. These run entirely in-browser—no API keys, no setup. Each demo includes curated presets that demonstrate the motif's core behavior: failure modes, recovery dynamics, and parameter sensitivity.

The demos are built with Gradio and deployed from the huggingface/space-*/ directories in the repository. Each includes tunable parameters so that edge cases—entropy collapse, budget exhaustion, stagnation recovery—can be explored directly.

8. Conclusion

Operon v0.12 shifts the project from "biologically inspired architecture" to "empirically validated architecture." The eval harness provides reproducible, statistically grounded evidence for the framework's claims. The mathematical corrections ensure that the paper and the code agree exactly—not approximately, not directionally, but precisely.

The results are honest. The Chaperone repairs structural damage (65.5% cascade accuracy) but cannot recover semantic information lost to type mutations (6.5% strict). The Immune System detects all simulated compromises but operates on behavioral distributions, not actual LLM outputs. These are the bounds of what the current implementation can claim.

The next frontier is end-to-end evaluation: real LLMs, real attacks, real function calls. The eval harness provides the scaffolding; the benchmarks provide the methodology. What remains is connecting the simulated biology to the living system.

The framework is available at github.com/coredipper/operon, pypi.org/project/operon-ai, and huggingface.co/coredipper (22 interactive demos). Feedback welcome.