I worked on the Site Checker System, and it is now ready enough.

The Programm checks every site in the hackclub/dns repo. I makes a simple GET Request to it.

I watch the request and wait for it to fail, than I check what exactly went wrong.
Or if it was successful if read the response and check if it is maybe the site of a hosting provider which tells that the deployment could not be found (example). I also want to do this with other hosting providers like Orchard, the problem is Orchard does not give me enough information to be sure that it is orchard and not the deployment.

When I figured that a site has a problem (or not) I do three things:

1. I write it in the database (or delete it)
2. I create/update/delete the row in the slack list (view it in slack)
3. I write a message to the #dns-alerts channel

The messages I write to #dns-alerts contains:

1. the type of the message (new problem, updated problem or problem resolved)
2. the link to the site
3. the problem type (and I am really proud on this one, because it is colored and has the same style like the links or list items, see the screenshot, or join the channel, this is done vie the slack tag block)
4. The link to the list record unless it was resolved (because than the record does no longer exist)

Since the last devlog, I have made some changes: Hosting Emojis, Memory improvements, better Proxy. And I am currently working on Site-Checker.

Hosting Emojis

When the target of a record matches the following hosting providers an emoji is displayed next to it in brackets:
Orchard
🅰️ Coolify A
🅱️ Coolify B
Vercel
GitHub Pages
Netlify
Nest
Cloudflare Pages

This is done with regexes (see them here)

Memory improvement

In my last devlog, I wrote that the program is always killed by the OOM-Killer because it uses too much RAM, that is now fixed.

The problem was that for each commit I cached all records that exists at that commit, which was not only complete waste of resources, but also completely unnecessary because I didn’t use the cache afterwards.

better Proxy

There were also some improvements for the Proxy, such as blocking Permanent Redirects for unsafe Location headers and caching the HTTPS state for a site.

Site Checker

That is my new big feature for this project. It is going to check all the sites in the dns zones. And if it finds a problem it is going to send it to a special channel and write it in a slack list (see the attached image). The checker will check for problem in the DNS or server deployment.

I already working on it. See the feature/site-chechker branch

Previews

I have a really cool new feature (actually it is really old, the first version was already working on the 26. 6. (two months ago), but I kinda forgot to write about it):

Previews

The problem is pretty simple: Imagine someone is adding a new record like super-cool-site.hackclub.com.

The record is not merged yet, so the DNS record does not exist. Because of that you normally can’t just open the domain in your browser to check if the site is working.

So how do I solve this?

Well, to understand this you need to understand how the hostname actually works.
When you open a site in your browser like hackclub.com, it resolves to an IP address, in this case 216.150.1.1 (owned by Vercel).
But on Vercel many sites are hosted on the same IP, so how does Vercel know which site to serve.

The Host Header

The Host Header is a HTTP header which is automatically added by the browser and contains the original hostname that was requested in this case hackclub.com

So the server can look at this header and then serve the correct page.

With HTTPS it is a little bit more complicated, because the hostname is needed before the HTTP request even exists. This is where SNI comes in. I did some tricky and hacky stuff to make it work.

How the preview works

When a new proposal is detected, the Slack message now has a link next to the name of the proposal:

https://dns-tracker.fantamomo.com/preview/<host>

When the link is opened, my server looks in the database for the target IP or CNAME.
Then it makes a request to the target, but instead of using the target hostname as the Host header, it uses the hostname from the proposal.
So even tho super-cool-site.hackclub.com does not exist in DNS yet, the upstream server gets a request which looks like it came for super-cool-site.hackclub.com.
Nearly all headers and the full body are passed through to the upstream server.

When the response comes back, I check what it is.
If it is a redirect and it targets the upstream server, I rewrite it so that it goes through the preview server.
If it is anything besides HTML, I just pass it through.

But if it is HTML, I parse the HTML and do two things:
First I rewrite all links, relative or absolute, so that they point to the preview server instead of the real domain. I also inject the /preview/<host> path so my server knows what it is supposed to preview.
Second I inject a small JS script which overrides the fetch function.

This is needed because otherwise the website could load correctly, but JavaScript could still make requests directly to the real domain (which does not exist).

The script is here if you are interested: proxied_fetch.js

There is one problem

While this allows the client to basically pretend that the DNS record already exists, it doesn’t work with every hosting provider.

The problem is that the hosting provider itself may expect the domain to be configured before it serves the website.

My server can make the requests like if the DNS record exists, but I can’t make the hosting provider believe that the domain really exists.

More problems

Another problem I have is that the program uses to much RAM, so much that it gets all the time killed by the Out-of-Memory-Killer.
So I am going to investigate what exactly takes so much RAM.
If I can’t find out or it actually needs that much, then I am going to request more RAM for my nest server.

So until then, it is possible that the server gets killed, or change detection gets delayed. If you really need it, I recommend you to just wait 1-2 days until I fixed that, or ping me in Slack so that I can restart the server.

What changed

In short, I added some new code to scrap some other things like:

• missions on the project page, so if a project is attached to a mission or has shipped with one
• source urls on the project page
• the daily random number pages (including the past)
• banner urls on users and projects
• if a ship has been returned or is pending

I also added timeline tables of users, projects, devlogs, superstar and ship events. So that I now how projects have changed over the time.

Problem

I also run in a problem, while the program scraps the site, I cannot access it too, because I wrote it so that the requests are exactly calculated so that I can fully utilize the rate limit. I tried to avoid it by running the program on nest but the container just kills my program after 50,000 requests (I don’t know why, and the logs don’t help me and it wasn’t the OOM-Killer). So I am now running it on my Raspberry Pi 4b and because it is at my house, it uses the same IP)

Renaming

I am also going to rename this project from Stardance-Graph to Stardance-Scraper, because the program is now more then just a graph. I am still going to write the graph, but in a second project.

Scraping

Well I scraped Stardance 4 times, since the last devlog:

1. The first time was on the Raspberry PI for testing (and because I couldn’t run it on my pc), after around 40000 request it stopped due to a death lock (two threads waiting on each other, but on my case it were 4 threads), the chance that this happened was so low, but it did, I tried to fix it but I failed. So a just killed the program.
2. I tried to run it on nest but after 50,000 requests it killed it so not the right option
3. Then I rerun it on my raspberry pi (with the fix) and it worked pretty well
4. I added the timeline feature and since then it just does its work

In total I have done like 400,000 requests to Stardance.

Image

The image shows a line chart that shows the number of users who joined on each date. (Yes someone joined on 19.04.2026: @/cskartikey)

What changed

I have used HTTP/1.1 for my request to stardacen, then I found out that I can use HTTP/2. I tried it and it so much faster, like I could send 600 Requests under 1 minute.  
It was so fast that my Database Writer, couldn’t keep up and I needed to slow down, so that everything is stored in the DB. But then Stardance got (D)DOSed (which wasn’t me). So they added ratelimits. And now I can only do 120 Requests per minute.  
Which slows my down.
I also updated the DB so that now every request I do is stored in it, for better debugging and analytic.

Some cool statistics

Here are some statistics about my scraping and Stardance

Scraping

Yesterday I started scraping around 6 AM EST and it took 10 hours and 21 minutes.

• I send 63,364 requests.
    - 6,820 returned with 404
    - and 98 requests failed due to other reasons
• I received a total of 1.22 GB HTML
• To serve my requests the Stardance server executed 746,089 database queries

Stardance

And I found some really cool things
Like:

• 30,239 users are on Stardance
    - 113 users got achivments
    - And the most common one is „Presentable Hardware Project„
• 19,171 projects have been created
    - There are currently 17 project that have been marked with superstar
    - And 1,167 ship events
• 11,614 devlogs have been posted
    - With a total time of 26,182 hours
    - 837 devlogs have been reposted
    - And a total number of 10,981 likes have been given.
    - there are 741 comments.
    - I also run a (very) simple local AI detection tool on every devlog, which can check for some obviously Ai and it only found 35 devlogs.

Images

I haven’t worked on the graph, so I don’t have pictures of it. But I have some diagrams about the data I have scraped.

1. The first (left) image is about the sign in users over the time I scraped
   2  The second (right top) image shows the visitors over the time
2. And the last (right bottom) one shows the project id to the total hours in the project, and the interesting thing about it is that most of the hours came from the projects with small ids, where the users of those projects have joined near the start of the event. I wrote I small test program which takes all projects with more then zero hours, then take the first 355 users. Then request Slack for there start date. Of those 355 users, 165 started before the 2026-5-20, 107 after, and by 83 I couldn’t find the start date.

Please note that this data is from yesterday and can be outdated

I wrote a program that completely scraps Stardance. It starts with the projects (from 1 to 20000) that it can find on the page. It analyses them and stores the information in a database. It also searches for references to users, devlogs, followers, comments, etc. So for every project I get also the creator of it and then the creator gets also scraped. So in the end I have users, projects, devlogs, followers, following and comments.

But was do I want with does data

The name of this projects reveals it: I want to create a graph, of every possible connection between objects. I want to host it on a website, so you can search for users like @Fantamomo and see how I am connected with other projects and users, like I repost a devlog, if someone comments one one of my devlogs etc.

Image

The image show a prototype (coded by AI, because I have no clue how to write something like that). The problem is that the program constantly uses around 139% CPU and around 2 GB RAM. Even with that much CPU usage, it is still laggy and I can barely use it.  
So now I have to figure out how to make the graph faster so that I can host it as a website on nest.

Stats

Scrapping Stardance took 3 hours (I paused between request to avoid the non existing rate limits)
During the time:

• 23654 requests where made
• To serve my requests the Stardance server executed 388157 database queries (174135 queries had been cached)
• I received 0.6 GB HTML text as responses.

Fun fact

My first devlog was reposted by someone.  
At that time, the notification system did not exist yet, so I didn’t know who reposted it.
Through my scraping I found out: It was @neon (btw thanks)

Slack id finding

The Programm now checks the follows the links in the slack profiles to check if the may redict to GitHub (because some hackclubbers habe www.github.com instead of GitHub.com, and some other use there own url shortener so that my program broke). And due this the number of found slack ids is the first time higher then the not found (310 to 276).

Also if a found some new records and the author/commiter hasn’t been indexed, the program prioritize them so that I can hopefully send the message with the mentions.

Link

If the records I an A, AAAA, CNAME or ALIAS the link in the slack message is clickable so that you can directly open the page.

I also added a link message to the GitHub repository where the change came from.

Deleted records

In my last devlog I said that I want to detect deleted and removed records. I tried that, but it is easier to say then to implement.

Changes

Slack id finder

I updated the GitHub to slack id finder, it now also search for messages, because the chance that someone sendet a link to his/her GitHub account is really high. Before this change I found from the 603 users only 70. After the change I found from 488 users, 234 which is near at 50% (some users are missing, because I couldn’t check does due the rate limiting from Slack).

Records parsing

Due a bug in the parsing logic more then half of the records in the repo are ignored. Like this one:

test:
- type: CNAME
   value: hackclub.com.

The bug is -, the parser has read it as an list instead of an map so I missed 1018 records, it is now fixed.

Ignoring

I also added the possibility to ignore specific repos. Like this one. I use this to test my code locally without spamming the #dns-tracker channel.

Next

The next thing is to detect deleting records on any possible way (removing the lines, reseting head, merging, branch deleting, etc.)

The image contains an proposals from @parth one of the Stardance developers, who adds an new domain staging.stardance.hackclub.com

I have done a lot in the 13 hours

What I added

I build a database migration system so that if I change some tables (which I do more then a should), a migration script is generated and on the next start executed so that I don’t have to manually update the db.

Mentions

Since the start of this project, I always wanted to mention the user, in my dns update message, which has done the changes. Which is easier to say than to make. Before this devlog I tried to do this by Slack APIs which didn’t work, so I thought where all of Hackclubs Users have there Slack Id and there GitHub account: Hackatime. I experimented which there API to find a way to get the slack id from an GitHub-account, which is sadly not possible. So I got back to Slack API and found a way:

1. If I want to get an User I request the Slack API with the account name
2. Then I go through every user id that I found and check the profile of it, if there GitHub profile matches my requested name, then I have the user id
   When a new changes is detected I look in the DB and check if there is a connection.

There is only one problem: The api I use is heavily rate limited, so that I can currently check one user every 5 minutes, but it works.

(Yeah I know 13 hours is to much, but I needed to finish this feature)

Since my last devlog the bot found some new interesting records (before #hackclub-leeks knew them), like the one in the image.

What I added

I changed the logging so that it also logs to a file and is colorful in the console.

What I tried to add

I wanted to make it so that in the message the person is mentioned that added/changed a record. The problem is: There is no way to for my to get a Slack user from its GitHub account. I know there is a profile field for the GitHub account, but I am not able to search for it. So I have some options:

1. I could index all users on Slack, by 124.265 users with 100 requests per minute it takes around ~21 hours (and I don’t think that the FD would like that)
2. Then I found a API. It is called “Slack SCIM API”, and it is perfect, because I can request all users that match a rule (like field GitHub == something). Until I found out that only Admins can use it.
3. I could do it manually. It will take some time (currently 664 users have contributed or forked the hackclub/dns repo) and that is not the only reason why I will not do it

Other things

Also due to some unknown reasons the bot sometimes gets killed by the OOM Killer and I couldn’t figure out why it needs so much ram.

Btw, 3h 10m where coded on my new MacBook Air (from flavortown) and because of the new keyboard layout it takes four times longer to write every line of code and the @ is killing me

Updated the slack message so that it is more clear what changes.
Also made the app more efficient in checking if the main repo or fork had been updated. (It makes 2 http requests which together only take around 1 second). And if nothing has changed, nothing will be added to the github rate limit.

It also runs on Nest and post messages in #dns-tracker.
There are still some bugs with detecting deleted records (in forks)

I created a program that scans the Hackclub/DNS repository, including its forks, to look for new entries or updates.
(Also i think i have the first devlog of stardance) (not the first, but the second)