Skip to content

Optimization and Its Blindness

The hinge of the viability arc

Two different questions

The repository now separates two questions:

  1. Model identification: Given declared observations, interventions, and a model family, which candidate processes remain consistent?
  2. Viability: Given an acting process and a boundary, which constraints keep continued operation, participation, and correction possible?

There is no universal theorem that forward execution is cheap and reconstruction is hard. There is also no theorem that every optimizer destroys its substrate. The questions belong together for a practical reason: better models can increase capability without selecting a good objective or protecting the conditions under which action remains correctable.

The hinge in one line

Optimization improves a declared objective. Anything important that is absent from that objective, its constraints, and its system boundary can be traded away.

This is not mysterious blindness. It is specification. A controller can respond only to variables made available through its state, observation, objective, and update process.

1. Optimization is model-relative

An optimizer requires at least:

  • a space of possible actions or policies;
  • an objective, loss, or preference relation;
  • information available at decision time;
  • resource and compute limits;
  • an environment or transition model;
  • a rule for updating or selecting actions.

Change any of these and the optimizer changes. “Optimization” alone predicts no unique trajectory.

Replicator dynamics are one useful example. With the TEO choice \(f_i(\mathbf{x})=\beta x_i\) and no homeostatic brake, an initially leading resource share is reinforced and generic unequal states approach a simplex vertex. This is concentration inside that equation, not a proof that every market, society, or AI system converges to monopoly.

2. Local objectives can miss system effects

A policy may improve the signal it receives while degrading an unobserved variable. A firm may reduce measured cost by shifting maintenance outside its accounting boundary. An AI system may increase task completion while increasing human review load. These are testable boundary and measurement failures.

Participants are not necessarily unable to learn the global effect. Shared sensors, science, institutions, and causal models can make it visible. The relevant questions are:

  • Which effects are observed?
  • With what delay and uncertainty?
  • Can the observation change the commitment?
  • Who can stop or revise the process?

The claim should weaken when the controller has accurate global feedback and uses it effectively.

3. Instrumental convergence is a hypothesis

Some goals create incentives for resource acquisition, self-preservation, or capability gain because those means support many later actions. How broadly this holds depends on the agent model, horizon, uncertainty, shutdown incentives, and environment.

The paperclip note gives a conditional TEO trajectory. It does not derive Bostrom-style instrumental convergence in general. For real agents, convergent subgoals require empirical or formal analysis under a declared decision model.

4. Capability loading inside the viable-corridor models

The strongest current result is narrower and executable. In the TEO ODE and a separate stochastic agent ecology, increasing a selected capability parameter pushes more than one modeled constraint toward failure. Under high capability:

  • strengthening the substrate budget alone can leave resource concentration;
  • strengthening regulation alone can leave substrate failure;
  • joint strengthening controls both modeled failure frequencies.

This is synthetic evidence that a shared driver can load multiple constraints. It does not show that every kind of capability increases every risk, or that one scalar captures AI capability in the world.

A real-system test would need to define the capability intervention, hold relevant resources constant, measure several failure modes, and compare single-axis with joint controls.

5. Why constraint architecture matters

An output-level rule can fail if the surrounding deployment system rewards bypass, hides costs, or cannot recover from error. Viability may therefore depend on a bundle:

  • limits on resource concentration and action rate;
  • substrate budgets and vital floors;
  • independent verification;
  • veto, appeal, and repair paths;
  • monitoring that remains useful under distribution shift;
  • authority assigned to those who bear the consequences.

TEO represents only a small subset of this list. Its corridor is one model of conjoint constraints, not a complete safety architecture or moral theory.

What would weaken the claim

  • Capability interventions improve the objective without loading additional measured constraints in calibrated systems.
  • One well-chosen control reliably substitutes for the proposed bundle.
  • Apparent joint-control benefits disappear under matched budgets or alternative models.
  • Costs attributed to optimization are better explained by an omitted external shock.
  • The protected variables do not predict recovery or continued viability.

What remains

The viability arc does not say that optimization is bad. It says that an objective is not a world, and capability is not purpose.

The handoff is therefore architectural:

Make the objective explicit, keep important dependencies inside the boundary, expose uncertainty, and preserve effective ways to slow, refuse, test, and repair action.