Spent most of the time on the GitHub side of ingestion-service. Went with GraphQL instead of REST, mainly to actually learn something new — turned out to have its own quirks (no file diffs available, rate limiting works completely differently, cursor-based pagination instead of the usual page numbers).
Built the GitHub client with three methods for commits, PRs, and issues, all going through one GraphQL endpoint. Tested it live against a few real repos, including our own Devault repo, and it’s working correctly along with pagination and rate limit tracking.
Sorted out secure token storage too - separate table for credentials, encrypted with AES-GCM, and nothing else in the codebase touches the raw token except one dedicated service. Also added a check so nobody can use someone else’s token, which wasn’t there at first and was a real gap I almost shipped.
Backfill is working now - runs async on virtual threads, loops through pagination for each resource type, saves to the DB with idempotency so restarts don’t duplicate anything. Source status goes PENDING -> SYNCING -> ACTIVE/ERROR depending on how it goes.
Last thing I got done was the service and controller for creating sources. Still need to test the whole flow end to end through the API, add the workspace role check I skipped for now.