DEVLOG #46 - Some repair scores could not be trusted

Every possible repair SL-LLM-R produces should be linked back to the original proposal it came from so the search can be scored correctly. I found cases where that link was ambiguous, which can make the score describe the wrong search. I changed the scoring path so ambiguous pairs are rejected instead of guessed, and the earlier zero-hit result stays zero.

DEVLOG #45 - Fixing one broken link still gave zero hits

When SL-LLM-R fails to find a repair that I know should be reachable, I need to know whether the search is bad or the test case is broken. One case had a broken link between the original proposal and the repair I expected the search to find, so I fixed only that link and ran the case again with the same number of attempts. The search still found the target zero times, so that broken link was not the whole problem.

DEVLOG #44 - I can explain 306 misses, but 1,224 still look the same

I am trying to understand why SL-LLM-R’s repair search fails to find fixes that should exist. Right now I can explain 306 misses, while another 1,224 look identical in every field I am allowed to compare. Two different explanations for the source-signal problem also match the same 306 cases, so the available data still cannot separate those explanations for the larger group.

DEVLOG #43 - I built one failure where I already knew the cause

SL-LLM-R’s repair search was missing many fixes that I expected it to find, but I did not know why. I built one small synthetic failure where the component reading the source signal gets the wrong input, so I can reproduce that miss on purpose. At least one failure now has a known cause, which I can compare with the unexplained failures without assuming they are all the same problem.

DEVLOG #42 - The first real learning run is still blocked

The repaired SL-LLM-R comparison produced 1,530 completed results, but I still did not understand the repair-search failures well enough to start changing real model weights. Training therefore stayed off while I investigated why targets were being missed. I would rather delay the first learning update than call an unclear result progress.

DEVLOG #41 - If the model grows, the extra capacity has to help

I do not want SL-LLM-R to add parameters just because a larger model sounds better. If the system ever needs more capacity, my current idea is to add a small module in a copy and test whether that addition actually improves the model. If it does not help, the extra module should be deleted just like any other failed update.

DEVLOG #40 - What happens if the model runs out of room?

SL-LLM-R is meant to keep learning over time, which creates a long-term problem if a fixed-size model eventually runs out of useful capacity. My current idea is a stable core, fast memory, small learnable parts, and possibly extra specialist modules only when the existing capacity is not enough. The project cannot do that yet, so this is still future research.

DEVLOG #39 - The repair-search failures are finally visible

With a completed comparison, I can now inspect why SL-LLM-R’s repair search misses targets. Some misses come from repeated ideas, some repairs need more steps, and some targets are never reached by the current search. Separating those cases is more useful than treating every miss as one generic failure.

DEVLOG #38 - The retry produced 1,530 completed results

After the first SL-LLM-R comparison crashed, I fixed only the problems that prevented the test from running and repeated the same comparison. This time all 1,530 runs completed and produced scores. I could finally inspect failures in the repair search itself instead of only debugging the test setup.

DEVLOG #37 - All 1,530 comparison runs crashed

I ran a large SL-LLM-R comparison to measure the repair search, but all 1,530 runs crashed before producing a useful score. That means the run tells me nothing about whether the repair method was good or bad. The test setup failed, so I kept the result as a failed experiment instead of treating the crashes as model evidence.

DEVLOG #36 - The learning loop is connected, but training is still off

The main SL-LLM-R learning path is now connected. It starts with the original model, makes a temporary changed copy, checks whether the change helped and whether older abilities got worse, then either keeps the copy or throws it away. Real training is still disabled, so this is not yet a finished self-learning model.

DEVLOG #35 - A better retry is still not model learning

SL-LLM-R is meant to change future model behavior, not just get a better answer on the second attempt. Many AI systems can retry with more context and look like they learned even when the model itself stayed the same. I only care about a correction as learning if it affects later behavior, does not break other abilities, and can still be undone.

DEVLOG #34 - Replay and consolidation fit the project together

I realized two ideas from continual-learning research fit SL-LLM-R quite well. Replay means keeping older knowledge present while learning something new, while consolidation is about deciding which new experiences deserve to last longer. Together they match the goal of turning some verified experiences into lessons without making every observation permanent.

DEVLOG #33 - Brain wiring gave me one idea to test

I looked at connectomics, which is research on how brain cells are connected, because SL-LLM-R needs a way to keep model changes local. One idea is that connection structure might help choose which parts are safe to change together. It is only inspiration for an experiment, not a claim that transformers work like brains.

DEVLOG #32 - The first real update has one clear path

I wanted the first real SL-LLM-R model update to follow one understandable sequence. The system loads a parent model, copies it, changes only the child, tests the parent and child, then either keeps the child or deletes it. Training is still disabled, but the update path itself is now connected instead of being spread across separate pieces.

DEVLOG #31 - A real base model is now part of the setup

SL-LLM-R started with placeholder model inputs while I built the safety and comparison logic around them. I replaced that placeholder with a real open model that the system can identify and verify exactly. Training is still off, but future experiments can now start from a real model rather than a fake name.

DEVLOG #30 - I can now undo a model update through the full practice loop

SL-LLM-R needs a complete way to undo a model change before real learning updates can be trusted. I can now save the original model, make a changed copy, test it, switch to it, and return to the original in one practice run. I have not yet used that path to undo a genuinely learned LLM update, so the mechanism works but the real learning case is still untested.

DEVLOG #29 - The cause finder is allowed to say it does not know

Before SL-LLM-R changes a model, it tries to estimate what caused the original failure. I tested that part on simple cases where the cause was already known, especially to see what happens when the signal becomes weak. It can stop instead of inventing a confident answer, which is the behavior I want before any model update depends on that guess.

DEVLOG #28 - Loading a model should not enable training

SL-LLM-R needs access to a real base model before I can run realistic experiments. I made that access read-only so loading and checking the model does not automatically give the system permission to change its weights. Training can stay completely off while I verify exactly which model and data the project is using.

DEVLOG #27 - Failed repairs still tell me something

SL-LLM-R generates repair ideas after model mistakes, and many of those ideas get rejected. I started keeping track of why they fail instead of treating every rejected repair as the same kind of failure. Duplicates, impossible repairs, multi-step repairs, and bad ideas reveal different weaknesses in the search even when the model learns nothing from them.

DEVLOG #26 - Every model change needs a history

If SL-LLM-R keeps a model update, I want to be able to trace exactly why that change exists later. Each kept update should point back to the original mistake, the proposed repair, the parent model, and the checks that approved it. Otherwise a future model problem could turn into a mystery with no clear way to undo the right change.

DEVLOG #25 - I need to know what caused a mistake before changing weights

SL-LLM-R is supposed to learn from a model mistake without changing more of the model than necessary. A single bad result only tells me that something went wrong, not what caused it. The system first needs a likely cause before it can choose a small place where a repair might make sense.

DEVLOG #24 - I had to keep the final test separate

I need to improve SL-LLM-R without slowly tuning it to the exact tasks used to judge it at the end. If I keep looking at those final tasks while changing the repair search, I can accidentally learn the test itself. Development tasks and final test tasks therefore stay separate.

DEVLOG #23 - Random search was harder to beat than I expected

SL-LLM-R uses a structured search to find possible repairs after model mistakes, and I compare it against random repair ideas. I expected random search to be an easy baseline to beat, but it was surprisingly competitive under the same budget. The structured search still has to prove that its extra complexity is actually worth using.

DEVLOG #22 - More attempts can make a weak search look better

I compare SL-LLM-R’s repair search against simpler search methods. If one method gets more attempts or more compute, it can look better just because it had more chances to get lucky. I now keep the search budget equal across methods so the comparison is about the method rather than the amount of compute.

DEVLOG #21 - The learning controller mostly blocks updates

SL-LLM-R has a controller that decides whether a possible model update is safe enough to try or keep. I originally imagined it mostly choosing how to learn, but most of its job is actually refusing weak updates. If the signal is weak, the change is too large, or forgetting risk is too high, the model stays unchanged.

DEVLOG #20 - Memory can change fast, weights should change slowly

SL-LLM-R uses two different kinds of adaptation. Temporary memory can change quickly because it can be thrown away, while real model weights should change much more slowly and only after stronger checks. This keeps a random observation from becoming permanent too easily.

DEVLOG #19 - New lessons can overwrite old ones

A continual-learning system like SL-LLM-R can forget older knowledge if it trains only on the newest mistake. I use replay, which means mixing some older verified examples back in while a new lesson is learned. The goal is to make new learning fit alongside old knowledge instead of pushing it out.

DEVLOG #18 - Fixing one mistake can break something else

SL-LLM-R cannot judge a learning update only on the mistake it was meant to fix. A model can improve on that one case and quietly get worse at older abilities. I therefore check for forgetting too, so an update is rejected if the local gain hides a larger loss elsewhere.

DEVLOG #17 - A changed model has to beat the unchanged one

SL-LLM-R creates a temporary child model when it tests a learning update. The child is not kept just because it is newer. It has to perform better than the unchanged parent under the same checks, otherwise the update is discarded.

DEVLOG #16 - I wanted rollback before trusting any update

SL-LLM-R should be able to undo a model change if the new version turns out worse. Every update starts from a saved parent model and happens in a temporary child copy. If the child fails the checks, it can be deleted and the original parent stays untouched.

DEVLOG #15 - Which experiences should the model learn from?

SL-LLM-R eventually needs to choose which experiences are worth using for learning instead of treating every case equally. I tested selecting problems near the model’s current ability limit rather than choosing practice cases randomly. In development tests that found useful cases more often, so task selection became part of the learning design too.

DEVLOG #14 - I made the first real weight change in a test copy

Most early SL-LLM-R tests only simulated what a learning update would do. This was the first time I changed actual model parameters with one tiny training step on an isolated copy while the original model stayed untouched. It does not prove self-learning works yet, but the real weight-update path itself did run.

DEVLOG #13 - Context can look like learning when it is not

SL-LLM-R needs to separate temporary memory from actual model learning. A model can answer better simply because the correct information is still in its context window, even though its weights never changed. I keep those two kinds of improvement separate so short-term memory does not get reported as permanent learning.

This part of the project was mostly about making the final test harder to fool. I preregistered the analysis, froze the generator and sealer, and sealed the held-out split before looking at any confirmatory outcome. The important result here is a constraint rather than a score: if I can still change the test after seeing results, the result is not trustworthy. The evaluation itself was not run at this point, so this was preparation for evidence, not evidence that SL-LLM-R works. I would rather keep the experiment locked than make the final result easier to obtain.

DEVLOG #12 - One mistake should only allow a small change

SL-LLM-R should not get permission to change the whole model because of one mistake. I want each repair to affect the smallest useful place, starting with temporary memory when possible and only touching learnable parameters when there is enough reason. A small amount of evidence should only allow a small intervention.

DEVLOG #11 - The model cannot grade its own repair

In SL-LLM-R the model can suggest a repair after it makes a mistake. I do not want the same model to also decide whether its own repair was successful, because that could turn into self-approval. The repair is checked separately from the model that proposed it.

DEVLOG #10 - Sometimes the best update is no update

SL-LLM-R is meant to learn from verified mistakes, but changing the model after every interesting result would probably teach it a lot of bad lessons. The system needs to be allowed to say there is not enough information and leave the model unchanged. One bad permanent update can be worse than learning nothing from that case.

DEVLOG #09 - A working repair still does not tell me what broke

SL-LLM-R can replay a model mistake with a repair and see whether the result gets better. The problem is that a better replay still does not tell me which part of the model caused the mistake. I split those jobs apart so the system first estimates the likely cause and only then decides what small part could be changed.

DEVLOG #08 - I needed an A/B test for model mistakes

The main idea in SL-LLM-R is to test a possible repair from the exact state before a model mistake happened. I replay the situation once normally and once with only one repair changed. Comparing those two runs is much more useful than seeing a better answer later and guessing that the repair caused it.

I lost track of posting while I was building the first search loop, so this entry covers more than one session. I tried five simple ways of generating possible fixes under the same fixed budget and gave every candidate a reproducible identity and lineage. The annoying part was budget accounting: duplicates and rejected candidates still consume real search effort, which matters when I compare the method against a baseline. At this point the engine was implemented, but I still did not know whether the extra structure was actually better than random search. The next experiment was to test exactly that instead of assuming the machinery was useful.

DEVLOG #07 - The new benchmark can finally separate methods

I rebuilt the SL-LLM-R repair benchmark so it has more tasks that are difficult but still solvable. Different search methods finally started producing different results instead of all looking the same. That does not prove my search is good, but at least the benchmark can now show when one method is worse.

DEVLOG #06 - Different searches kept finding the same repairs

SL-LLM-R can try different ways of searching for a repair after a model mistake. I expected those methods to explore different ideas, but many of them kept finding the same repairs. I started tracking duplicates directly because a large attempt budget is not useful if most attempts repeat the same idea.

DEVLOG #05 - My first benchmark could not tell the searches apart

I need a benchmark to tell whether SL-LLM-R’s repair search is actually better than simple alternatives. The first one made structured search and random search look almost identical. Most tasks were either very easy or almost impossible, so I threw that benchmark away and rebuilt it with more tasks in the middle.

DEVLOG #04 - The repair search was wasting attempts

When SL-LLM-R sees a mistake, it searches for possible repairs to try. My first version kept spending attempts on duplicate fixes and repairs that were not valid in the first place. I changed the budget so every attempt counts, including the useless ones, because otherwise the search would look better than it really is.

DEVLOG #03 - A crash cannot leave the model half changed

SL-LLM-R compares a model before and after a possible learning update. If that update crashes halfway through, a half-changed model would make every later result impossible to trust. So each update is all or nothing: the system keeps either the old state or the complete new state.

DEVLOG #02 - What should count as real learning?

SL-LLM-R is supposed to make a model learn from mistakes, so I first had to decide what actually counts as learning. A better second answer is not enough. I only count it if the improvement still helps later, does not break other things, and can be undone if the change was bad.

DEVLOG

#1 - Why I Started SL-LLM-R

Project: SL-LLM-R

Introduction

I originally just wanted a project for the summer.

I had a few ideas, but most already existed or sounded like something I would abandon after a few days.

Then I started thinking about AI.

I use AI a lot for coding. It can understand large projects and solve difficult problems, but it can also repeat the same simple mistake several times. Even when it notices that something failed, it usually does not actually learn from it.

The context becomes longer, but the model itself stays almost the same.

Current models are trained on human-created data and updated through processes designed by humans. They can generate answers, but they normally cannot decide what to learn next, which decision caused a failure, or whether a change would damage another capability.

So I started thinking about a system that could learn from its own verified experiences.

That became SL-LLM-R.

The Idea

The core mechanism is called VGCP:

Verifier-Grounded Counterfactual Plasticity.

The simplified process is:

1. Save the state before a decision.
2. Let the model act.
3. Check the result with an external verifier.
4. Generate possible repairs.
5. Replay the same situation with and without each repair.
6. Compare the outcomes.
7. Keep only changes supported by evidence.

In simplified form:

Δ = E[Y(change) - Y(original)]

If the repaired version performs better across controlled tests, the change may be useful. If the evidence is weak, the system should not update.

The goal is controlled, local, reversible, and verifiable learning.

The PDF

I first planned to write a short document.

That did not go as planned.

Every answer created more questions: How do I save the exact state? How do I compare runs fairly? How do I prevent leakage, forgetting, broken rollbacks, or the system approving its own changes?

The document became the SL-LLM-R V1.2.2 specification.

It is now 135 pages long and contains the architecture, mathematics, safety rules, algorithms, experiments, and failure conditions.

The PDF does not prove that the system works. It defines what I need to build and what would count as failure.

Completed Changes

• Defined the project idea.
• Created the VGCP concept.
• Wrote the V1.2.2 specification.
• Started the initial implementation.

Next Steps

• Continue the deterministic foundation.
• Implement snapshots and rollback.
• Build the proposal system.
• Compare it against simple baselines.

Summary

I started with a few random project ideas, got annoyed by how intelligent and stupid AI can be at the same time, and ended up designing a system that should learn from verified mistakes.

It might become something important.

It might also be another stupid idea that takes over my entire summer holiday.

Right now, I do not know.

That is why I am building it.

Summary (AI)

SL-LLM-R is a pre-experimental project about controlled, verifier-grounded continual learning. No validated self-improvement result is currently claimed.

Note

I spent way too long writing this devlog and the PDF.

I also started coding parts of the project, and it was… different from what I expected.

More soon.