Stardance Stats dev log #9

Ask is done.

You type a question in plain English, and a model writes the database query for it. You get a table back, and bars if it makes sense to chart.

It only gets to write queries out of a list I picked. No writes, its own read-only database user, a time limit, and a rate limit on how many questions you can ask an hour. If it writes something outside that, it just gets refused.

Also made API docs with examples for every endpoint, a README, and a license. There’s a docs button in the sidebar now.

Stardance Stats dev log #8

SDS website is already up! Check it out at https://stardancestats.xyz/

Devlogs page. Every devlog on the site in one feed.

Cards with the author, the project, the numbers, the text, and the images or videos. Filter by how recent, sort by newest, likes, comments, or views, or search.

Search reads the body. Type “cat” and you get everyone’s cat posts, or put quotes around something to match it exactly. Whatever you searched gets highlighted in the card.

Also put the little star creatures around the pages. (the sillies)

Stardance Stats dev log #7

Projects page is done.

Same podium as People but with the banners behind the top three. The list under it includes a thumbnail, the owner, and hours and devlogs in a single row. Buttons to filter to Super Stars or hardware.

Click one, and you get everything about a project. Banner, tiles, a chart from 1 to 30 days, every ship, and all the devlogs. Sort them by newest, longest, most liked, most discussed, or most viewed.

Stardance Stats dev log #6

People page: Leaderboard with a podium for the top three; click through to anyone’s profile.

Search goes through the API since whoever you’re looking for usually isn’t on the page you’re on. It tries handles that start with what you typed first, then ones that just contain it.

Added a blacklist. Some people don’t want to be tracked, so their ID goes in, and everything of theirs gets deleted.

Fixed missions. Some pay a fixed amount instead; others go to rating; and none of that shows up as a payout on the page. Now each ship gets credited properly.

Also added those silly creatures to the global page.

Stardance Stats dev log #5

(almost done for ship!)

Comments. A thread gets queued when its count moves, and they run in their own lane so they don’t starve behind everything else. Deleted ones get marked gone instead of vanishing.

I got the shop too. Prices are per region, and the region is just a cookie, so it just grabs the catalogue seven times. That gives you the spread: how much more the same thing costs depending on where you are.

Devlogs have their own history now, so you can watch one post’s likes over time.

Fixed a demotion bug. Crawling a page more often used to make it go cold faster, which is backwards. Now it just checks how long it’s actually been quiet.

Frontend started too. One page with the totals and a chart. The rest of the tabs are greyed out. And that’s what I’m going to be working on right now.

Stardance Stats dev log #4

History endpoints.

Projects, users, and the whole platform all run through the same thing. Pick your metrics, pick hourly, daily or weekly, get points back.

Points only get written when a number moves, which means quiet stretches look like gaps. There’s a fill option that carries the last value forward so a flat line renders as a flat line. Ask for too wide a range, and it says no instead of trying.

Every response now says when the data was last seen and whether that’s old. Responses also carry an ETag, so asking again for something that hasn’t changed costs nothing.

Also added a platform endpoint for the site-wide totals, and a single devlog one.

Stardance Stats dev log #3

Little differently today. I fetched all of Stardance today, and if you have any query requests, make a comment and I will answer!

It can be something like, “What’s the best rated project on SD?” or “How many users have over 20 followers?”

There’s a queue of everything worth crawling, and each page gets a tier. Pages that keep changing get checked often; pages that never change drop to the slow lane.

It reads the sitemap to find pages and see what changed recently. But the sitemap misses a lot, so it also just walks id ranges past the highest one it knows is real. Most of those 404s, that’s fine. Owners that never got listed get found through their projects instead. Hidden and unverified profiles still parse.

Stardance Stats dev log #2

Users and leaderboards.

It reads profiles now, by handle or ID. Followers, streaks, achievements, what projects you’re on. Renames are fine, the old handle still finds you.

Project pages only show handles, no IDs, so those rows just wait until the user gets crawled and then link up. Order doesn’t matter.

Ships only pay after review, so shipped hours and paid hours aren’t the same. Payouts also use capped hours that aren’t shown anywhere, so I work those out from the payout and multiplier. Unpaid hours get an estimate.

Super Star badges too.

Stardance Stats dev log #1

New project! Stardance Stats, a better way to browse Stardance.

The crawler is polite. It’s rate-limited, retries when things fail, and backs off entirely if the site starts struggling. Pages that haven’t changed don’t get re-parsed.

Parsing hooks onto stable attributes instead of layout, so a restyle won’t break it. It reads devlogs and ships off the feed, then checks the numbers against the page’s own totals. If those drift too far, something broke.

Everything goes into MongoDB as time series. Counters like devlogs or stardust can’t go down, so if they do, it’s a bad parse, and it gets rejected. First crawl also backfills old history from dates already on the page.

oh btw all testing is done on @The_Craw project, so shoutout to him.