So I worked on many things these 5 hours and made a lot of progress. Also in one of my ships I was told to make the devlog a bit technical as here it is.

The projects main pipeline lives in the src/core/ folder. It contains a ‘types.ts’ file which is the shared vocabulary basically it has RepoRef, FileEntry, Chunk, Citation, ChatMessage etc. The src/core also contains a folder called***’state/store.ts*** its a dead-simple global store using**’useSyncExternalStore** from react is just a plain object with a setState function and a listener set.

The ingest pipeline (ingest/pipeline.ts*) is where the magic happens. It checks IndexDB first, if you’ve already indexed a repo, it loads the cache version instantly. If not, it fetches the zip from Github’s codeload endpoint, unzips it with fflate, runs a tree walk to build a manifest of usable text files, chunks each file, then kicks off embedding in the background. The treewalk (treewalk.ts) strips the top-level directory from codeload paths and filters out the binaries, ignored folders and lockfiles using simple extension checks in ‘langs.ts’. The unzip step (‘unzip.ts’) is just 10 lines wrapping fflate’s unzipSync, GitHub’s codeload endpoint doesnt support CORS for arbitrary origins, so githubFetch.ts handles that by routing through Vite’s dev proxy locally and through a CloudFlare worker in production (in the deployed version). I have attached the pics of the code below so feel free to check them out.