RepoSense
By
@CassianWray
- 4 Devlogs
- 13 Total hours
A website where you put in a github link and chat with an AI that knows about the code.
Ship #1
Pending review
I made a project named RepoSense which lets you learn about a specific repository. Basically you put in the giuhub link of the project and the site automatically fetches it and embeds it and you can ask any question specific to the repo. It will cite the repo and tell you about the project and why any decision was made.
What was challenging?
The most challenging part was getting the embedding right during deployment. Since the embedding is quite an intensive process I had to figure out how to get it done in relatively short timespan. Previously when I first created and tested it locally it was so slow like 10 minutes for a project with 2 files (~200 lines of code) but after lots of tweaking I brought it down to about a minute for most repos unless you decide to put in massive repos then well it will take some time.
The thing I am most proud of is that I somehow got it to work. Originally I had not idea what I was doing and just created the UI and made it pretty but as I learnt about embedding and stuff I somehow figured it out.
One thing necessary to try the project is an API key if you run it on the web. I have added the CORS_PROXY with CloudFlare workers from my side but you will need a free API to get it to work (don’t worry its a client side project so nothing is stored). I recommend using Groq API keys since its quite good for this.
The screenshot below for example is of my own repo (this repo same project which is deployed I am using as an example)
3h 12m 44s logged
I competed the project for the most part. The one thing though is that deploying is so damn difficult like you do one thing and another thing breaks and you do another thing another breaks.
Anyway for the project I am using Cloudflare workers for CORS_PROXY since its free tier is quite generous and enough for me right now.
5h 12m 17s logged
So I worked on many things these 5 hours and made a lot of progress. Also in one of my ships I was told to make the devlog a bit technical as here it is.
The projects main pipeline lives in the src/core/ folder. It contains a ‘types.ts’ file which is the shared vocabulary basically it has RepoRef, FileEntry, Chunk, Citation, ChatMessage etc. The src/core also contains a folder called***’state/store.ts*** its a dead-simple global store using**’useSyncExternalStore** from react is just a plain object with a setState function and a listener set.
The ingest pipeline (ingest/pipeline.ts*) is where the magic happens. It checks IndexDB first, if you’ve already indexed a repo, it loads the cache version instantly. If not, it fetches the zip from Github’s codeload endpoint, unzips it with fflate, runs a tree walk to build a manifest of usable text files, chunks each file, then kicks off embedding in the background. The treewalk (treewalk.ts) strips the top-level directory from codeload paths and filters out the binaries, ignored folders and lockfiles using simple extension checks in ‘langs.ts’. The unzip step (‘unzip.ts’) is just 10 lines wrapping fflate’s unzipSync, GitHub’s codeload endpoint doesnt support CORS for arbitrary origins, so githubFetch.ts handles that by routing through Vite’s dev proxy locally and through a CloudFlare worker in production (in the deployed version). I have attached the pics of the code below so feel free to check them out.
1h 56m 2s logged
I have finally completed some of the core logic of the project. Namely:
- Chat (the chat logic for communication with the repo)
- llm (the llm integrations to enable chatting with the BYOK or local model)
- retrieve (the thing that retrieves the data from the Github repo and gives context to the app)
2h 41m 7s logged
I am trying to build a developer tool that makes it possible to chat with any github repo.
The plan is to have the repo be able to answer questions about itself so that its easier to debug and find flaws and also to help understand concepts better.
For Now I have completed the landing page and theme toggle and now am gonna start working on the rest.
