Gamed the eval

The failure

A long run keeps only so much in context. After enough compaction the original constraints stop being re-read, and the run drifts toward whatever moves the score — not what the score was standing in for. The metric climbs while the actual goal recedes. Because the loop optimizes the number in front of it, the gaming looks like progress right up until someone reads the output.

Watch it happen — then get refused

How it works

The goal is held outside the agent’s context, where compaction can’t erode it. Two contexts track the run, and a gate reads their verdict at turn-end.

A goal-tracking context records the declared goal when it is written, and stays active until the goal is met — so the target survives across turns and compactions.
A recording context requires every evaluation run to be documented, checked across all runs so far, not just the latest.
A gate at Stop refuses to let the run end while a declared goal is still unmet — and skips entirely when no goal was declared, so a goal-free session is free to stop.

A context can only track; only the gate refuses a turn.

The actual configuration

This is the real example shipped at examples/eval-loop-maxing/. Three natures, each a directory under .sloprail/ — a *.yaml declaration plus the scripts it names:

Directory.sloprail/
- Directorycontext/
  - Directorygoal-tracking/
    context.yaml
    enter.sh
    exit.sh
  - Directoryrecording/
    context.yaml
    enter.sh
    exit.sh
- Directorygate/
  - Directorygoal-verify/
    gate.yaml
    run-verify.sh

context

Two contexts. goal-tracking wakes when a goal is written and holds it active until it’s met. recording wakes on each evaluation invocation and checks that every run so far is documented.

# Wires goal.yaml into the engine. PostFileWrite so it reads settled content
# (the `enabled` flag exists only once the write lands). Tracks active/inactive
# only; the goal-verify gate is what refuses a Stop.
on:
  - event: PostFileWrite
    match: event.path startsWith "goal/" and event.path endsWith "goal.yaml"
enter: ./enter.sh
exit: ./exit.sh

#!/usr/bin/env bash
# enter: a goal.yaml was written (settled). Read its `enabled` and activate
# only if the goal is currently in force.
set -uo pipefail

input="$(cat)"
goal_path="$(printf '%s' "$input" | jq -r '.event.path // ""')"

# Content by event kind. Post runs in practice; the Pre branches keep the script
# correct for any kind. On a Pre write with resultKnown false, the content is not
# derivable yet — defer to the Post kind rather than mistake it for an empty file.
kind="$(printf '%s' "$input" | jq -r '.event.kind // empty')"
case "$kind" in
  PostFileCreate|PostFileUpdate)
    # The engine declares newContentKnown on PostFileCreate and PostFileUpdate
    # (internal/filemod/module.go FieldNewContentKnown): false when it could not
    # read the settled goal.yaml — a link to a FIFO or a device, or past the
    # read cap. Whether it is enabled is unknown, so the goal counts as in force
    # (activate: goal-verify then holds the turn to it) — never as switched off.
    if [ "$(printf '%s' "$input" | jq -r '.event.newContentKnown // false')" != "true" ]; then
      jq -n --arg name "$(basename "$(dirname "$goal_path")")" --arg path "$goal_path" \
        '{goal: $name, goal_path: $path}'
      exit 0
    fi
    content="$(printf '%s' "$input" | jq -r '.event.newContent // ""')"
    ;;
  PreFileCreate|PreFileUpdate)
    known="$(printf '%s' "$input" | jq -r '.event.resultKnown // false')"
    if [ "$known" != "true" ]; then
      # Result not derivable ahead of the write: defer to the Post kind.
      exit 0
    fi
    content="$(printf '%s' "$input" | jq -r '.event.newContent // ""')"
    ;;
  *)
    # No kind, or one this context is not about: nothing to activate on.
    exit 0
    ;;
esac

if [ -z "$content" ]; then
  exit 0
fi

enabled="$(printf '%s' "$content" | grep '^enabled:' | awk '{print $2}')"
goal_name="$(basename "$(dirname "$goal_path")")"

if [ "$enabled" != "true" ]; then
  # Written but not enabled — a goal can be authored and left off.
  exit 0
fi

jq -n --arg name "$goal_name" --arg path "$goal_path" \
  '{goal: $name, goal_path: $path}'

#!/usr/bin/env bash
# exit: reads the goal-verify gate's verdict from `gates` and reflects pass/fail
# into active/inactive. Does not run verify or refuse the Stop itself.
set -uo pipefail

input="$(cat)"
status="$(printf '%s' "$input" | jq -r '.gates["goal-verify"].status // "fail"' 2>/dev/null)"

if [ "$status" = "pass" ]; then
  # Target met — this context deactivates.
  exit 0
fi

# Not met (or the gate hasn't run yet this cycle) — stay active.
exit 1

# Separate from goal-tracking: this tracks every eval run being documented, not
# any one target. Re-enters per eval; exit checks completeness across all runs.
on:
  - event: PreCommandInvoke
    match: any(event.invocations, .bin == "eval")
enter: ./enter.sh
exit: ./exit.sh

#!/usr/bin/env bash
# enter: an eval command is about to run. Just activate — the exit check reads
# the trajectory for every run this session produced, so nothing needs to be
# carried in the payload.
set -uo pipefail
cat >/dev/null
jq -n '{}'

#!/usr/bin/env bash
# exit: reads the recording-verify gate's verdict from `gates` and reflects
# pass/fail into active/inactive. Does NOT re-run the completeness check or
# refuse the Stop itself.
#
# The original version of this script ran the trajectory scan and tried to
# refuse the Stop directly, via a non-zero exit carrying a {"decision":"block",…}
# body — the OLD context contract. Under the current engine (nature_context.go's
# runContextExits: "exit is pure lifecycle... nothing here contributes to a turn
# block"), a context's exit verdict only ever flips `active`; the engine does not
# read its stdout as a refusal at all. So that refusal was silently discarded —
# an undocumented eval run was NEVER actually blocking a Stop, contradicting the
# README's "must be documented... before the turn can end." The fix mirrors
# goal-tracking/exit.sh: the real check moves to a paired gate
# (gate/recording-verify/run-verify.sh, bound to Stop, which DOES have a refusal
# channel), and this script becomes a thin read of that gate's own verdict.
set -uo pipefail

input="$(cat)"
status="$(printf '%s' "$input" | jq -r '.gates["recording-verify"].status // "fail"' 2>/dev/null)"

if [ "$status" = "pass" ]; then
  # Every run so far is documented — this context deactivates.
  exit 0
fi

# Not documented (or the gate hasn't run yet this cycle) — stay active.
exit 1

gate

The piece that blocks. It wakes at Stop only while a goal is active, and refuses the stop until the goal is verified met.

# The "don't stop until target met" verdict lives here (a gate refuses; the
# context only tracks active/inactive).
#
# match skips this when no goal is active, so a goal-free Stop is permitted
# (`require` alone would block it). `require` orders the context's enter first,
# making the active read meaningful.
on:
  - event: Stop
    match: context["goal-tracking"].active
require:
  - context: goal-tracking
checks:
  - script: ./run-verify.sh

#!/usr/bin/env bash
# Runs verify and decides the Stop. `require` guarantees goal-tracking ran first,
# so just read what it left behind.
set -uo pipefail

input="$(cat)"
goal_name="$(printf '%s' "$input" | jq -r '.context["goal-tracking"].payload.goal // empty' 2>/dev/null)"

# GateCheckPayload carries `context` at top level.
if [ -z "$goal_name" ]; then
  # goal-tracking is not active — nothing to verify, permit the Stop.
  exit 0
fi

goal_dir="${SR_WORKSPACE:-.}/goal/$goal_name"
script_name="$(grep '^script:' "$goal_dir/goal.yaml" | awk '{print $2}')"
verify_script="$goal_dir/${script_name:-verify.sh}"

if [ ! -x "$verify_script" ]; then
  echo "goal '$goal_name' is active but its verify script is missing or not executable at $verify_script" >&2
  exit 1
fi

if "$verify_script"; then
  # Target met — permit the Stop.
  exit 0
fi

echo "Goal '$goal_name' target not yet met. Do not stop: keep iterating until verify.sh passes." >&2
exit 1