Operon v0.15: Diagram Optimization

1. The Problem: Diagrams Without Economics

v0.14 gave us composable coalgebras, morphogen diffusion, and optic-based routing. The wiring diagram system can describe complex multi-agent topologies with typed ports, integrity labels, prism routing, and traversal transforms. But it has a blind spot: all modules are treated as equally cheap.

The DiagramExecutor uses naive topological sort. It doesn’t know that one module costs 5 ATP and another costs 500. It can’t identify that two modules could run in parallel. It doesn’t notice that a prism wire can never fire. And it certainly doesn’t adapt its behavior when the system is running low on energy.

Biology solved this problem long ago. Cells don’t blindly run every metabolic pathway at full throttle. They perform Flux Balance Analysis—identifying rate-limiting enzymes, pruning vestigial branches, parallelizing independent pathways, and downregulating non-essential reactions under ATP depletion. The pathway structure itself reveals optimization opportunities.

2. Cost-Annotated Diagrams

The foundation is a simple dataclass:

@dataclass(frozen=True)
class ResourceCost:
    atp: int = 0           # Energy cost (ATP units)
    latency_ms: float = 0  # Time cost
    memory_mb: float = 0   # Memory footprint

    def __add__(self, other):
        return ResourceCost(
            atp=self.atp + other.atp,
            latency_ms=self.latency_ms + other.latency_ms,
            memory_mb=self.memory_mb + other.memory_mb,
        )

ResourceCost composes additively—the cost of a path through the diagram is the sum of the costs along it. Formally, this is an enrichment of the wiring category over the commutative monoid $(\mathbb{R}_{\geq 0}^3, +, \mathbf{0})$.

Costs are attached at two levels:

• Module cost—ModuleSpec gains an optional cost: ResourceCost field. This is the cost of running the module (enzyme catalysis).
• Wire cost—Wire gains cost: int = 0 (ATP units). This is the transmission cost (metabolite transport between compartments).

A second new field, essential: bool = True, marks whether a module can be skipped under metabolic stress. Essential modules (like the Validator in a pipeline) always execute; non-essential modules (like an Enricher or Logger) can be dropped when ATP is scarce.

diagram.add_module(ModuleSpec(
    "Validator",
    inputs={"data": pt}, outputs={"valid": pt},
    cost=ResourceCost(atp=10, latency_ms=20.0),
    essential=True,  # always runs, even when starving
))
diagram.add_module(ModuleSpec(
    "Enricher",
    inputs={"data": pt}, outputs={"enriched": pt},
    cost=ResourceCost(atp=30, latency_ms=50.0),
    essential=False,  # skipped under ATP depletion
))

All annotations are backward-compatible. Modules without cost annotations default to zero cost. Modules without essential default to True. Existing code is unaffected.

3. Static Analysis

The analyzer examines a WiringDiagram without executing it, extracting structural information that guides optimization.

3.1 Dependency Graph and Parallel Groups

dependency_graph() builds the adjacency list of module dependencies. find_independent_groups() uses topological layering to identify sets of modules with no mutual dependencies—these can execute concurrently.

groups = find_independent_groups(diagram)
# [{'Parser'}, {'Validator', 'Enricher', 'Logger'}, {'Output'}]
#                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#                These three can execute in parallel

3.2 Dead Wire Detection

A wire is “dead” if it carries a PrismOptic whose accepted types have no intersection with the source port’s DataType. Such a wire can never transmit data—the prism will always reject.

# Source outputs JSON, but the prism only accepts ERROR
dead_prism = PrismOptic(accept=frozenset({DataType.ERROR}))
diagram.connect("Parser", "out", "ErrorSink", "err", optic=dead_prism)

dead = find_dead_wires(diagram)
# [Wire(Parser.out -> ErrorSink.err)]

3.3 Critical Path

The critical path is the longest cost-weighted path through the dependency DAG. It determines the minimum execution time under maximal parallelism—no schedule can complete faster than the critical path cost.

path, cost = critical_path(diagram)
# path = ['Parser', 'Enricher', 'Output']
# cost = ResourceCost(atp=38, latency_ms=62.0)
#
# The Enricher branch (30 ATP) dominates over
# Validator (10 ATP) and Logger (2 ATP)

3.4 Optimization Suggestions

suggest_optimizations() returns human-readable suggestions combining all analyses:

suggestions = suggest_optimizations(diagram)
# [parallel]   Modules ['Enricher', 'Logger', 'Validator'] can execute concurrently
# [dead_wire]  Wire Parser.out -> ErrorSink.err can never transmit
# [cost_hotspot] Module 'Enricher' consumes 30 ATP (58% of total)

4. Rewriting Rules as Endofunctors

Each optimization pass is an endofunctor $F : \mathbf{WDiag} \to \mathbf{WDiag}$ on the category of wiring diagrams. The critical property: $F$ preserves input-output behavior. An optimized diagram is bisimilar to the original—no observer can distinguish them.

Three concrete passes are implemented:

Passes compose via the optimize() pipeline:

optimized = optimize(diagram)
# OptimizedDiagram(
#   passes_applied=['eliminate_dead_wires', 'parallel_grouping', 'cost_order_schedule'],
#   eliminated_wires=[Wire(Parser.out -> ErrorSink.err)],
#   parallel_groups=[{'Parser'}, {'Logger', 'Validator', 'Enricher'}, {'Output'}],
#   schedule=['Parser', 'Logger', 'Validator', 'Enricher', 'Output'],
# )

Note the schedule: within the parallel group, Logger (2 ATP) comes before Validator (10 ATP) comes before Enricher (30 ATP). Cheaper modules first, so if a budget cap is hit mid-group, the expensive modules are the ones that get skipped.

5. Resource-Aware Execution

The ResourceAwareExecutor replaces the naive topological-sort executor with metabolic-state-gated scheduling. It takes an ATP_Store and adapts its behavior based on energy availability:

# Normal state: full parallel execution
store = ATP_Store(budget=200, silent=True)
executor = ResourceAwareExecutor(optimized, store, max_workers=4)
# ... register handlers ...
report = executor.execute(external_inputs={...})
# report.execution_order = ['Parser', 'Logger', 'Enricher', 'Validator', 'Output']
# report.skipped_modules = []
# report.total_cost = ResourceCost(atp=50)

# Starving state: non-essential modules skipped
store2 = ATP_Store(budget=100, silent=True)
store2.consume(92, "drain")  # 8% remaining -> STARVING
executor2 = ResourceAwareExecutor(optimized, store2)
report2 = executor2.execute(external_inputs={...})
# report2.skipped_modules = ['Logger', 'Enricher']
# Only essential modules (Parser, Validator) execute

The executor also deducts wire transmission costs from the ATP store. Each module execution and wire transmission is metered, producing a complete ResourceAwareReport with total cost and skipped module tracking.

6. BudgetOptic

The BudgetOptic provides wire-level cost capping. It tracks cumulative cost across transmissions and blocks further data flow once the budget is exhausted:

budget = BudgetOptic(max_cost=10)
budget.can_transmit(DataType.JSON, IntegrityLabel.VALIDATED)  # True

budget.add_cost(8)   # 8/10 spent
budget.remaining     # 2

budget.add_cost(5)   # 13/10 spent -> over budget
budget.can_transmit(DataType.JSON, IntegrityLabel.VALIDATED)  # False

This is allosteric feedback inhibition at the wire level: once a pathway has consumed its energy budget, the pathway shuts down. BudgetOptic composes with existing optics via ComposedOptic, so you can combine cost capping with prism routing or traversal transforms on the same wire.

7. Paper Section

A new paper section (§7, Diagram Optimization via Categorical Rewriting) formalizes the above. Key definitions:

The section also positions operon’s approach relative to Abbott & Zardini: where they optimize IO transfer costs across GPU memory hierarchies, operon optimizes coalgebraic systems with internal state. The key differentiators are optimization target (IO costs vs. agent orchestration), dynamic state (runtime metabolic gating vs. static analysis), and scope of composition (multi-agent workflows vs. single-algorithm pipelines).

8. Test Coverage

9. Example

10. What’s Next

With cost-annotated diagrams and resource-aware execution in place, several directions open up:

• Learned Rewriting Rules—The current optimization passes are hand-crafted. Learning rewriting rules from execution traces—identifying recurring subdiagram patterns and their optimized replacements—would connect diagram optimization to program synthesis and equality saturation techniques.
• Cost Profiling—Automatically measuring actual module execution costs and feeding them back as annotations, replacing manual cost estimates with empirical data.
• Multi-Objective Optimization—Balancing ATP, latency, and memory simultaneously rather than using the current single-objective critical path. Pareto-optimal schedules for different resource trade-off preferences.
• Tissue-Level Optimization—Extending diagram optimization across tissue boundaries, where inter-tissue wires carry different cost profiles than intra-tissue wires.

Code and examples: github.com/coredipper/operon