I wrote a program that completely scraps Stardance. It starts with the projects (from 1 to 20000) that it can find on the page. It analyses them and stores the information in a database. It also searches for references to users, devlogs, followers, comments, etc. So for every project I get also the creator of it and then the creator gets also scraped. So in the end I have users, projects, devlogs, followers, following and comments.

But was do I want with does data

The name of this projects reveals it: I want to create a graph, of every possible connection between objects. I want to host it on a website, so you can search for users like @Fantamomo and see how I am connected with other projects and users, like I repost a devlog, if someone comments one one of my devlogs etc.

Image

The image show a prototype (coded by AI, because I have no clue how to write something like that). The problem is that the program constantly uses around 139% CPU and around 2 GB RAM. Even with that much CPU usage, it is still laggy and I can barely use it.  
So now I have to figure out how to make the graph faster so that I can host it as a website on nest.

Stats

Scrapping Stardance took 3 hours (I paused between request to avoid the non existing rate limits)
During the time:

• 23654 requests where made
• To serve my requests the Stardance server executed 388157 database queries (174135 queries had been cached)
• I received 0.6 GB HTML text as responses.

Fun fact

My first devlog was reposted by someone.  
At that time, the notification system did not exist yet, so I didn’t know who reposted it.
Through my scraping I found out: It was @neon (btw thanks)