---

DEVLOG #5 (2026-07-02, FIRST EVALUATION AND NO-GO)

Project: SL-LLM-R

The proposal engine did not outperform random. That is the headline.

I built the evaluation runner and compared the engine with four baselines over
four seeds and two budget levels. The result contained 991 data rows, and the
engine’s HitRate was indistinguishable from random proposal.

My first reaction was to blame the engine, so I built a separate no-go analysis
with a failure taxonomy, kernel contribution matrix, and oracle upper bounds.
The oracle result changed the interpretation: even a system with knowledge of
the hidden rule could barely solve the tasks above chance.

The benchmark was not discriminative. Tasks were mostly either trivial for
every system or impossible for every system. There was no useful intermediate
band where structured search could beat random search.

I also found a split-integrity bug. One split-ID path depended on Python hash
randomization and could change between versions. I replaced it with canonical
JSON and SHA-256, then documented the root cause.

I defined remediation hypotheses H1-H5 and tested the V2 development changes.
They also ended in NO-GO. At that point, changing the proposal engine further
would have been tuning against a broken ruler.

---

What Works

• Deterministic evaluation runner
• Failure taxonomy and oracle-bound analysis
• Split-integrity bug fixed and tested
• Negative result preserved instead of hidden

---

Tests

make stage1a-test
make stage1a_v2-test

The recorded evaluation and remediation tests passed. The scientific result was
still NO-GO.

---

Changed Files

Commits: 4de0165, 268c508, 33e64e2, 5ee6178

---

Next Steps

Build a V3 benchmark with real intermediate difficulty and measurable
separation between structured and random search.

---

Summary (AI)

Stage 1A did not beat random on the original benchmark. Oracle analysis showed
that the benchmark lacked discrimination, so the correct result was NO-GO and
a benchmark redesign.