Updates

Bunch of API related things this time

• Added api docs
• API key section in account tab and APIT key account endpoints
• Added CSRF support (lmao i dont know why i didnt have it before)
• Made the api key response be a popup in account page and centered it in a later commit
• Added Stats API endpoint
• Added Max Age to cookies
• Website and slack bot now share 1 source of truth for stats (stupid that i didnt have it like this already)
• Added granting perms to other api keys
• New user joins are now picked up in real time.

Thats about it

Updates

Really short devlog this time.

• Added Error logging for session_query
• Resumable channel sweeps
• Use a new action for build & deploys

Workflows are also greatly increased with speed

Alot of updates this time

Updates

Brand new scraper crate separate from app, dk if I have mentioned it before in devlogs.
It does slack and hackatime scraping all in one so we can have the main app deploys not interrupt scrapes.

Updates^2

• Added “a” keyword - Alias for “all time”
• Updated settings.rs and example.env to fit new defaults
• Fixed refresh_daily_stats failing on a second
• Fixed a bunch of ::bigint needing to be everywhere a sum() was decoded
• switched scraper to mimalloc allocator
• Replaced ts with real message_tis / 1000000 (I dont know if i put enough/too many zeros here)
• Fixed slack time recomputing in new postgredb
• Precomputes page stats now
• Removed a useless half-hourly job
• ON CONFLICT now only fires when counts actually change
• Added DB available probe to tests
• Better logging (disabled timestamps in logger)
• Added a mark_seen dedupe
• Updated hackatime sync logging to be less taking up all logs
• Added zstd + drop duplicate tables
• Removed zstd as postgre18-alpine does not come with it and I am not rescraping slack
• Only the scraper can init tables now

I don’t think im missing anything else i did,
If I am though then sorry gois

Also see in one of the images below is total ram and cpu usage. somehow the postgre db is using over 4.5G of ram

Updates

Mighta lowkey forgot to be doing devlogs again…
Anyway

Updates^2

• Basically any keyword for time works now
• Cargo Chef is now used for docker
• Updated AGENTS.md
• No longer requires hackatime data to display slack time data.
• Ported everything from clickhouse to postgresql but havent deployed that yet
• You can now mention other people to get their stats

I think thats it
Might be wrong on that tho.

Anyway I got better from being sick last week and im sick again

Also used @mahad in screenshot below, def not an excuse just for a random pung

Updates

We still sick (not much better today) but i got some work done

Updates^2

• Split a bunch of existing code into separate files for maintainability
• added font caching
• parallel resyncs (hackatime)
• More memory/CPU optimizations in the code (I lowkey forgot what i changed for this

Thats it for now

A decent length devlog this time

So recently I have been sick so i got less time.
(Wish me to get better soon)
Anyway enough of that, on to the changes:

Updates

• Fixed authless hackatime syncing
• Using the spans api of hackatime so we get all data for time specific commands
• fix SQL and rust helper dividing by 1000000
• Wrapped the entire sum(if(…)) in toUInt64(), which fixed the slack bot reporting wrong time from public hackatime syncing change. It makes it so clickhouse sees a single output type regardless of what it infers inside.

Thats about it

Updates

• We swapped aws-lc-rs for ring
• futures swapped for futures-util
• nuked tower crate & lowered what is pulled for tokio
• Brought back dependabot auto merge
• Added cool charts
• Nuked cool charts
• Stats page changed again (wip)
• Broke hackatime time syncing when i tried to make it not need auth (tmrw me problem)
• Added page streaming and incremantal word totals

Updates

improving my devlog to hour ratio

Anyway..
New db scrape is completed!

Major changes

• Top words leaderboard
• Leaderboard search system
• Updated stats page (see below)
• Added a combined leaderboard (hackatime + slack time)
• Added backups in coolify
• Minor repo restructure (moved shiptalkers-app to its own shiptalkers-app/ dir)
• Added gzip support for slack api scrapers

im lowkey running out of things to add…
Drop any suggestions in the comments

Updates

• Tad bit of time that i forgot to log progressively.
  Anyway
• DB got corrupted
• We rescraping dis shi rn
• Added a whole new formula (multiple times actually)
• Added tests for formula
• Did a bunch of fixes to try and save corrupted db
• Afterwards i just nuked it and we started rescraping yesterday
• Made ini_tables probe slack_messages_by_user first
• Added default clickhouse values into the rust app
• Now recomputes stats page cache every 30s
• Fixed UNKNOWN_IDENTIFIER: ‘chars’ (Yeah i had claude do this one)
• Added a better system to add new slack keys to the env, instead of 1 big env var, its now, SLACK_BOT_TOKEN_X=xoxb….
  Where X is token number
• made it a shared work queue instead of static round robin sharding for scraper
• Switched to a mimalloc allocator from glibc
• It now does backfills for channels and users in parallel
• Added a system for incase hackatime goes down
• Did a commit for one new keyboard being “last 2 months”
• It also now scrapes reactions

Updates

• Added proper test system
• More keywords
• Heavily sped up per user pages & stats page
• Updates stats ui to mostly show 5 decs for db size in gib
• added a display load time in ms to footer
• Nuked the admin panel
• Dependabot & PR labeler system(s)
• We now use alpine base for docker
• We got a tests system now
• I think thats it, idk tho

Updates

• May have forgot to be doing devlogs
  Anyway
• Actual thread scanning and actually scraping its replies
• Recconect system for run_socket so the slack bot part stops dying
• Added an admin panel
• Is bot and Is Deleted acc tags to make leaderboard better
• Slack_app_creation script (to scrape with more tokens.. faster)
• Added error support from slack
• Proper slack stats time computation for in channel image
• Scan older threads support (up to 30 days)
• If slack stops sending periodic pings then slack bot auto reconnects
• Displays coding time as hours and minutes instead of just minutes
• DB schema updates

Oops, forgot to devlog for a bit

Anyway

Updates

• its online
• it works
• Round robin support for multiple tokens for > speed
• it replies in channel (ship-talkers channel)
• downgraded clickhouse to 26.4 for my older server
• added a proper health check endpoint
• Optimized slack messages daily
• Realized conversations.history supports 999 per request instead of just 100 msgs
• Auto deploy system
• Limited resources for clickhouse
• Consistant rescrape checks w/ round robin support ofc

Thats about it i think

Oh yeah it finished the full slack scrape

Next up

• Changing scrape system for privacy reasons (maybe)
• more scrape accs
• QOL features
• Idk gang

Updates

As usual we got a bunch of features

• Search system
  • Channel search
  • User Search
• Per user stats page
• Per channel stats page
• Hackatime/HCA login systems
• Header & Footer
• Added a cool banner at top of the page to show its scraping for initial scrape
• Fix docker failing due to forgeting to COPY templates for it
• Added scopes needed for everything to readme
• Switched out the purple for a soft pink
  Still not done scraping

Updates

• Docker stuff
• Better concurrency system
• Replaced hand rolled bucket
• Two queue system for need to be scraped and already scraped checking
• Show total db size in stats page
• Backfilling for already scraped channels
• Top chatters and top shiptalkers tables in stats page
• Move all styling to a css file
• Add the css file to the repo as i forgot to include it in the move all styling to css file commit
• Use askama crate for server side rendering to remove javascript in website
• remove dead code and dead crates
• Create AGENTS.md
• Syslink CLAUDE.md to AGENTS.md
• Create clippy.toml
• Create a justfile
• Create CONTRIBUTING.md
• Add expected environment to contributing file
• Realize slack api sucks for the millionth time
• Fix things that i was ignoring and just lint needed me to fix to not error outs

Scraping is coming along good too
over 1k channels done
14.8k to go

Updates

• Archived Channel ID scraping
• I then reverted that change 2 days later after how stupid it was
• No more infinite scan loops
• Message scraping support
• Auto dedupe
• Better scraping
• Better logging
• Thread Scraping
• Then better thread scraping

Finally

• Thread reply scraping in background

Updates

We got it scraping total channel count so far

Proper list

• Link up stats page to clickhouse
• Fix it not pushing to clickhouse often enough
• Start off with scraping total channel count
• Docker compose stuff

Fun Fact: There is a total of 15,958 channels in Hack Club Slack!

Updates

Basic project setup