PLANETARIUM LLM BENCHMARK

Devlog #6

Today I changed the website theme from black to white.

• added open ai adapter so someone can test the benchmark and llm with their own api key
• made a clear readme with insturctions on how to run, with other things.
• organised the codebase so it becomes modular, easy to grasp
• cleaned up stale code from each file
• added and connected things to config.yml so every change dont need a deep dive in codebase
• refactored some python files manually

Here is sneak peak of how new dashboard looks like

Devlog #5

Today i finally evaluated gpt 5.6-sol on the benchmark and found many things that we can change and tweak to improve the benchmark more
and its detailed trajectory for research use!
after that i cleaned up the codebase added docs images etc. for production release

Here’s a sneak peak of Gpt-5.6-sol walked path.

Devlog #4: the live progress site is up, and a real model is on screen
Date: 2026-08-13

Short one today. We finished a website that shows a model’s progress through the world as it happens. It’s working. You can watch the path take shape live instead of waiting on a final score.The run you’re looking at right now is being walked by GPT-5.6 Luna, set to medium reasoning. That’s our first model actually moving through the world on the live view.The site does its job, but it’s rough in spots. We’ll keep polishing it slowly rather than trying to rewrite everything at once.

We also know the benchmark isn’t balanced yet. We need to tweak some settings so every part of what a model can do gets a fair chance to show. Right now some things are easier to see than others, and that’s not good enough.

A bunch of smaller changes and fixes went in since the last update. Nothing flashy, just steady work.
This is still a development run. Nothing here counts as an official result yet. But seeing a real model move on the screen is a long way from a few days ago.

Devlog #4

Today i dont have much progress because we evaluated model and several gaps were identified so all the time gone in fixing them!

Wait for 2 or more days thanks

Devlog #2:
our test world is ready to try, first real run tomorrow
Date: 2026-08-11
Hey everyone, here’s a update on what we’ve been building with Planetarium.

What we’re building:
We’re making a benchmark. That’s a pretend little world where we put an AI in charge and see if it can survive and get useful stuff done on its own. No remote control, no holding its hand. Just the AI, the world, and a job to do.

What got done lately:
We locked in the size and shape of the world. It’s now a real, proper size, about a thousand squares by a thousand squares, and we’ve confirmed it’s actually built that way and not just sketched on paper.We finished the safety plumbing too. All the behind the scenes machinery that lets us safely drop an AI into the world and watch what it does is in place now. The AI’s private login details stay out of the saved results, and everything we produce is clearly marked as a practice run rather than an official score.We also cleaned up our own decision notes so the record of what we agreed on is tidy and signed off.On top of that, we ran a quick practice round with a fake stand-in, basically a scripted dummy and not a real AI, just to check the world and the recording system work from start to finish. They did.Where things standThis is still a practice version. No real AI model has been tested in it yet, and nothing we make counts as an official result. We call it “not yet certified” on purpose. We don’t want to claim a score we haven’t actually earned.

What’s next:
The benchmark needs a bit more finishing, but it’s close. Tomorrow we’re running our very first real AI model through it, and we’re paying for it ourselves with our own credits, not a free trial. That’ll be the first time a genuine model tries to survive and make progress in our world.After that there are still a few owner-only steps before any result can be called official. Proper sign-off, locking the setup so results can’t be messed with.
First real run.

I am building Planetarium, a benchmark for autonomous AI agents that measures something harder than answering questions. Can an agent actually operate in a world, discover things, build, survive, trade, and adapt over a long horizon, reproducibly, without gaming the score?

It is a deterministic, content-addressed simulation. An agent gets observations, takes actions, and the world reduces them through a sealed, hash-chained event store. Same inputs produce byte-identical outputs every time, with no floating-point flakiness and no hidden randomness.
It has a built-in anti-cheat with two worlds:
a public sample world and a hidden scoring world. In the hidden world only the agent starting position is randomized per run, so a model that memorized coordinates from a past attempt gains nothing. The layout is frozen by the world seed.

13 roadmap phases are done at the dev-fixture level: deterministic kernel, observation/action, scientific discovery, engineering, survival/ecology, production/logistics, population/organizations, markets/economy, civilization/adaptation, long-horizon memory, evaluation/scoring, dashboard, and adversarial release validation. 2287 tests pass with 0 failures.
My favorite part is the honesty guardrail. The release validator is structurally incapable of claiming verified without evidence.
I am very excited to release this benchmark!