My summer break has ended so development is going to be a lot slower from now. I added mocha outdated so you can list outdated apps before upgrading them to a newer version.
Besides that I tweaked a lot of stuff related to CLI output. For example I changed search to output one table, and simply outputting exact matches at the top of that table.
I also changed a lot of messages which were using LogOutput to LogInfo so they use the correct color and output to stderr and not stdout.
Not sure why I didn’t do this earlier, but I’ve added success messages to commands where it makes sense. Also added a success color to the config so you can change what the success color is.
I do still need to improve CLI output quite a bit, but it’s slowly getting there.
I’m writing this one right before F1 Qualifying at Spa! Continuing with the trend of some of my recent work, I improved what table outputs look like to be a little bit more readable.
We now use proper separators in the form of Unicode Box Drawing Characters. I also made column alignment configurable, which improves legibility for some commands like mocha bucket list and mocha cache show.
I’ve attached an image of the scoop and mocha versions of bucket list so you can compare for yourself.
I love refactoring!!!! I’ll probably end up spending another 20 hours just on refactoring because it’s so fun!!!!
This time around I wanted to refactor the code so that our commands orchestrate stuff instead of the helper functions. For example, pkg.Install would first download a package, then install it, and then link it through shims and shortcuts all by itself.
This causes issues because it means error messages become slightly worse, and it creates a class of bugs due to edge cases where we don’t necessarily want this order of orchestration. Hence I did a lot of refactoring in the pkg module.
I also ended up adding a way to skip verification of downloads when downloading, installing or updating a package through --skip-verify. It’s a useful option in situations where there is a malformed manifest or simply no hash to begin with.
mocha now supports a user defined theme for output colors. In the process I ended up doing a lot of stuff related to the config. First thing I did was swap to fatih/color for colored output, since it handles stuff like colors in command prompt and removing color when output is piped.
Then I refactored the config module. Earlier I would pass around a MochaConfiguration object through every function. Now I use a config singleton that’s accessible from any function. Why? Because it means I don’t need to pass the object every time I want to log information. Also means I don’t need to pass mochaDir through almost every function in the codebase.
Getting user defined themes was a bit complicated because I needed a good way to handle a default theme so output doesn’t look bad if parsing fails. I handled this by only using the configured theme if every color resolves properly. Otherwise output falls back to the already loaded default theme. Due to this you still get colored output even if there are errors or warning before the configuration and theme is loaded.
So I’ve been doing a bit of refactoring, a bit of bug fixing and a bit of optimization. Essentially just polishing mocha before I work on the next feature.
Here’s some of the stuff I did:
ParseRefString to validate behaviormanifest/info.go
Besides that I fixed the uninstall flow since it was really bugged after the latest changes. Essentially mocha would uninstall apps with incorrect info. It would also try to remove shims/shortcuts for apps which don’t have any active shims/shortcuts.
I then added a bit of logging since the uninstall command didn’t give any user output for whatever reason (i forgot to earlier).
This is going to be the equivalent of 2 devlogs since I thought I made one earlier for persists but I didn’t.
New feature! We now handle persist items in order to store data between versions. These are files which are persisted between versions of an app.
Lets take git as an example to explain why persist exists. Say you want to make sure your git config isn’t deleted every time you update. The issue with this is that the config file is saved in the version specfic directory.
To resolve this we can keep the config in $MOCHA_DIR/persist and then symlink the config to where it needs to be, which is essentially what persist tells us to do.
Since we handled persist items last time we can now implement mocha upgrade to update apps. The way scoop handles updates is by first removing anything that references the app, like shortcuts or shims.
Then we just follow the install process from scratch. This leaves the old version available in case a user wants to revert versions due to a bug, while also updating to the latest available version.
This can be seen in the video below, which shows me updating llama.cpp-hip-radeon.
We now create shortcuts! This means any GUI apps should show up properly in the start menu, just like they do when you install with scoop. All I really had to do was a bit of orchestration since I decided to vendor in an implementation from https://github.com/jxeng/shortcut
Besides that I expanded the number of apps that are supported by implementing a bit of code that handle architecture specific bin fields, along with bin fields which store 2D arrays.
I also cleared up some of the TODO items. I swapped the junction implementation to an implementation from https://github.com/nyaosorg/go-windows-junction since we were previously shelling out to CLI. That and I added a bit more zip bomb protection
For the purposes of the video below I changed the location which shortcuts get put to my desktop. It’ll probably be configurable, and match scoop by putting shortcuts in the startup menu by default
As I mentioned in the previous devlog I added good help information to every bit of the CLI. It’s surprisingly difficult to write good help messages which explains everything to users in a few words
I also changed the behaviour of the mocha cache clear command to not clear all cache by default. That’s now gated behind the --all flag in order to match scoop a bit more
Typically I don’t want to make devlogs for small stuff like this, but this is something I found really cool.
So I was working on help messages for all the commands when I realized that mocha bucket list felt a bit slower than pretty much every other command.
This is because we need to shell out to git 2 times per bucket, and I had 4 buckets added. While 8 git commands might not sound like a lot, it was enough to make mocha bucket list feel slow.
So what I did is I parallelized the metadata fetching through an errgroup. Instead of fetching the metadata for each bucket in order, we fetch the metadata for all 4 buckets simultaneously.
This led to a big performance uplift. About 3 times faster than the old version of mocha, and just over 12 times faster than scoop for this operation. See the video below.
I finished up mocha uninstall and mocha list, which means you can actually manage installed apps instead of manually deleting them.
There’s not much that goes into mocha uninstall. You just need to remove the relevant shims and app directories since the apps that mocha supports are all portable.
mocha list is similarly simple. It’s just loops over the directories in $MOCHA_DIR/apps and looks for apps along with the versions installed for each app.
So I added support for more apps by implementing more extractors. This means in theory any apps which don’t need a pre-install or post-install should work.
To be more specific, I handle a few more properties on manifests. Those being extract_dir and innosetup, which inform how an app needs to be installed. I also implement 7zip and innounp as extractors to support more file formats which applications are distributed in.
Of course there’s still a lot of work before I consider this ready for public use. I need to create commands for some basic operations like uninstalling, updating and listing apps. I also need to handle shortcuts otherwise GUI apps can’t be accessed from the start menu.
Besides that I need to improve the user experience. As you can see in the video below mocha just sort of dumps everything to the console even if it’s not relevant. Help menus are also critical to get working properly.
Finished up the code cleanup in commands, fileops and manifests. I did catch a few edge cases and some incorrect behavior in the process, which is why I always spend so much time reviewing code.
I did use AI to assist with code review because I think it’s an area where AI shines. AI is really good at finding small errors or inconsistencies in the codebase which I probably wouldn’t notice.
For example, it identified that the ManifestReferences argument was not a required argument for the download command, which meant no help message was displayed on incorrect usage like the other commands.
One big thing I will work on eventually is console output. Right now errors are well formatted but a lot of the time commands just run stuff without providing a lot of insight into what actions they are taking which isn’t great. Even the help menu is pretty useless right now (see the image below).
Extra context: A manifest reference is a pointer to an app (e.g. main/[email protected])
Before working on more features and expanding the amount of apps mocha actually supports I wanted to go through the codebase and do a lot of cleanup since the codebase is quite messy. There are silly mistakes in some files and bad implementations in others.
Hence the time I spent during this period was focused on moving code to where it should be and cleaning up some edge cases, specifically related to the bucket, config and shim modules.
A good example of this is moving the code to get the active config to the config module instead of leaving it in main.go. Next up will be refactoring the commands, fileops, and manifest modules.
I don’t have much to show so I figured I would present a feature I added but didn’t show earlier: the ability to skip cache hits with the --force flag when installing or downloading an app.
Some context I forgot to add initially: I renamed the app module to manifest since I felt it was more accurate to what it actually handles scope wise.
Huge milestone!!! I’ve gotten mocha install working for very basic apps! This means the entire process of downloading a file, then extracting it, then creating a junction and finally creating a shim now works for basic pieces of software!
So in terms of what I actually did this devlog, I added cache hits to the download command since it’s relevant to the install command. I then implemented the aforementioned .zip extraction and junction creation.
Then all that was left was a bit of orchestration, getting everything to work together properly. See a working install of innounp using mocha install below!
mocha update is another command which does very little but is very important. All it does is update the known_buckets file, and run git pull on all added buckets to update them. It can also update just the buckets you specify if you pass in an argument.
I do need to improve the output of the command since it outputs raw git right now, but at least it’s functional. See the demo below.
Spent a bit longer between devlogs this time, but it was worth it.
So I realized the error messages were really bad. I was pretty much just passing errors up the call stack without adding any context, which isn’t great since it means our error messages will be useless to people reading them
I therefore went through every single file of the codebase and wrote good error messages which add a bit of context every time we pass it up the call stack.
I also did a bit of refactoring where I thought it would be worth it. For example in the shim module I now set the PE subsystem of the shim binary while it’s in memory instead of writing to disk twice.
A few good examples of what the new error messages look like can be seen in the video below.
Shims are simple in concept, complicated in reality. So essentially a shim in mocha is just an executable which calls another executable. It’s similar to a shortcut in that sense, except it shows up in $PATH
Thankfully the scoop project already has a repository with 4 different implementations for the shim binary, so I can just have mocha shim download one of those binaries.
The complicated part was the PE Subsystem. The PE subsystem is basically a bit of info in each binary which tells Windows whether it’s a console application, or a GUI application.
We need to set the PE subsystem of the shim exe to the PE subsystem of the binary we are calling for Windows to execute the binary properly. Hence I need to read and write raw binary data.
All of this is just the implementation for exe shims. Scoop itself contains implementations for .bat/.cmd, .ps1, .jar, .py and linux binaries, all of which have wildly different implementations compared to .exe shims.
The video below shows the working set of mocha shim commands (those being add, remove and list).
mocha cache is a simple command. It shows you what files are currently in $MOCHA_DIR/cache directory and it clears them out.
One thing I noticed was that scoop cache show actually displays raw bytes instead of human readable file sizes for each cache entry, which is a bit odd since the total they display is human readable.
Therefore I made my implementation mocha cache list display human readable sizes since that’s a more reasonable default.
Fuzzy search is really really good.
While implementing mocha search I noticed the results returned by scoop search were in alphabetical order, which doesn’t make a lot of sense. If I do scoop search git the first result should probably be git
As a result my implementation of search uses github.com/sahilm/fuzzy which is a small library which provides a fuzzy search implementation.
Scoop also checks for binary name matches, e.g. package biodiff can provide a binary called git-biodiff and show up in search results. I decided to remove this for now, since I personally find it a bit unnecessary.
scoop search also lists all search results, which is okay when results are sorted alphabetically but doesn’t make sense when searches are relevance based.
Therefore, by default mocha search will list exact matches plus the 20 most relevant search results. This can be overridden by the --count argument, where setting it to 0 displays all results.
A demonstration of mocha search vs scoop search can be seen in the video below.