Stardance Stats dev log #1

New project! Stardance Stats, a better way to browse Stardance.

The crawler is polite. It’s rate-limited, retries when things fail, and backs off entirely if the site starts struggling. Pages that haven’t changed don’t get re-parsed.

Parsing hooks onto stable attributes instead of layout, so a restyle won’t break it. It reads devlogs and ships off the feed, then checks the numbers against the page’s own totals. If those drift too far, something broke.

Everything goes into MongoDB as time series. Counters like devlogs or stardust can’t go down, so if they do, it’s a bad parse, and it gets rejected. First crawl also backfills old history from dates already on the page.

oh btw all testing is done on @The_Craw project, so shoutout to him.