rasterminal
- 69 Devlogs
- 449 Total hours
A cross-platform 3D model viewer in the terminal from scratch https://github.com/PavolUlicny/rasterminal
A cross-platform 3D model viewer in the terminal from scratch https://github.com/PavolUlicny/rasterminal
I’m finally ready to release v0.1.0-alpha.2. Only notable thing I changed here is that I changed the default thread count: It now defaults to hardware concurrency everywhere. Other than that, I of course did final documentation checks and fixes and now I’m just waiting for CI to pass so I can tag it and let the release workflow do the last step. So this is hopefully the last devlog before my ship!
Did final polish and cleanup of the platform layer since I want to actually finally cut a release relatively soon and I just did a lot of work there. Fixed things like useless tests, outdated or wrong comments and some platform specific quirks. Probably actually gonna cut a release soon (this time for real). Also added early rejections for BVH and CSM files, since they contain no model geometry (they can’t currently be displayed at all) and documented the change. The code is basically done (hopefully, I really don’t want to find another big gap that’s gonna take a week to close), I just want to correct anything wrong in the README, changelog, manpage or help menu and maybe make them look a little better before cutting a release and shipping.
SIGSTOP and SIGCONT are POSIX signals for stopping and resuming a running process. I’m working on adding handling for these, so you can safely pause and unpause rasterminal at any time. This is a lot of work, since until now, we just set up the terminal on program start and cleaned up when the program exited. There’s a lot of quirks and weird scenarios that I have to handle, for example if another signal gets sent while cleaning up the terminal for SIGSTOP. I’m trying to catch as many edge cases by just doing random things while using the program and also with the help of AI to come up with potentially uncaught scenarios.
Well, would you look at that! That took significantly less time than the Windows fix. It is partly because it was a smaller fix and only part whole POSIX fix, while the Windows one was the whole thing and also the foundation had already been laid out from the Windows work and I just integrated the POSIX path into that, but it is also very much because Microsoft is a horrible, terrible company that can’t make good software. I mean, look at GitHub, they have been going downhill ever since they got acquired by Microsoft.
Now that I got that out of my system, here’s what was fixed:
If the program cannot enable raw mode at startup, it safely fails instead of continuing and corrupting the terminal state
If the cleanup gets interrupted, it retries cleanup so it doesn’t leave the terminal state corrupted after one failed attempt
Onto the next POSIX fix: handling SIGQUIT and SIGHUP.
My nightmare is over. I hopefully won’t have to deal with Windows for a while now. Summary of what is now fixed/handled on Windows:
Ctrl+C, Ctrl+Break and Q exits fully recover the terminal’s state
Ctrl+C and Ctrl+Break don’t leak mouse escape sequences into the next prompt if you terminate while actively dragging a model
A bunch of other minor Windows or cmd.exe specific quirks
Only thing that isn’t handled (didn’t find a good way to handle it) is task manager “end process”. It kills the program without cleaning up and sometimes freezes the shell.
Windows just doesn’t work. It doesn’t work. Nothing works on Windows. Everything breaks. I don’t get it.
THE UNIT TESTS PASS ON WINDOWS CI BUT WHEN I ACTUALLY RUN THE TESTS ON WINDOWS IT FAILS????? HOW IS THAT EVEN POSSIBLE.
The release is now definitely getting delayed by at least a week because I keep running into more and more and more and more and more and more and more and more and more issues. I hate Windows. Microslop is the only company that could mess up a basic terminal emulator.
And I also have to reboot my PC every time I fix something and push it and then reboot to get back to linux. this is torture.
At one point I tried to do it on a Windows VM on linux but that was too slow so I just gave up.
Most of them are Windows or cmd.exe specific (nothing ever works on Windows), but some are present on POSIX too. The Windows specific ones are:
cmd.exe for some unknown reason doesn’t turn off the raw input sequences, so arrow keys don’t work like they should, even though I explicitly turn it off (????????) (I hate Windows, they always have to do something unexplainable for no reason just so your code doesn’t work as it should)
when you terminate the program with ctrl+c or ctrl+break while actively dragging the model around, it sometimes leaks mouse escape sequences into the next prompt
bunch of other weird small stuff that nobody cares about
And the few on POSIX are mostly ctrl+z suspend/resume behaviour not recovering the terminal and the same thing with ctrl+\ (SIGQUIT).
I’m still working on fixing these. Most of the windows ones are already fixed, but I haven’t even started with the linux/POSIX ones.
I have postponed the planned release to when I fix these + any other issues I find.
As a prize for your great work, look out for a bonus prize in the mail :)
The repo is in a pretty complete and polished state, so I’ll cut a release here. This will also probably be the last release before I ship, so I’m making sure that nothing is broken and everything works on every platform.
I’ve still got a lot of work to do though, even after the end of stardance, like animations, glTF PBR, format completeness for every format, GPU acceleration, etc.
I really don’t have much to say here, I’ve already put all the info about the branch into the past devlogs. All the formats work and the wrapper is very sturdy. I’ve found more bugs in assimp itself than my code (devlog on that soon). So, rasterminal can now load basically any 3D model you throw at it. Some of the formats assimp supports are so niche, I’ve never even heard of them.
I’m trying a bunch of hand-crafted models from all the formats that I can load and trying to find things that dont load or render properly. Also had AI write me a simple fuzzer in python and I’m just letting it run for a few hours. Also auditing the assimp loader paths against my wrapper to find any format specific behaviour that should be handled. I’ve already found a bunch of things and I’m still finding more.
Assimp has a lot of weird quirks and issues that I keep running into. Honestly kinda regretting the decision to integrate it, but since I’m already so deep in, I’ll just keep going.
Currently trying various models from all the different model formats that assimp loads. So far found a segfault, some weird behavior and integration mistakes that I made. Also found a small bug in assimp itself, for which I’ve submitted the PR with the fix to their GitHub repo. For anyone interested, here’s the PR: https://github.com/assimp/assimp/pull/6799
Yop, I’m adding the Open Asset Import Library as a fallback for any model formats that aren’t natively supported with their own dedicated loader (only ones with dedicated loaders are glTF2, PLY, STL and OBJ). I’m still keeping the dedicated loaders, because they are more feature-complete, to-spec and wayyy faster (especially the glTF one).
I added it because I feel like for a 3D model viewer, support for many formats, even niche and obscure ones, is pretty important, even though nobody might use them.
This roughly doubled compile time and almost quadrupled binary size, which also made CI take way longer (of course). It somehow made the MacOS runners take 50 minutes to compile (from the previous 3 minutes), but I fixed that by bounding the number of processes CMake can spawn to the number of threads on the runner (previously was using -j which spawns as many as it wants and it was spawning about 230 of them).
I’m still testing how it handles various model formats and trying to find any reachable segfaults.
After I just merged three big feature branches (kitty and sixel graphics protocols and the high-res performance branch), I needed to do some overdue cleanup and refactoring. And so I did:
Collapsed the super long changelog entries into way more readable and legible points
Restructured the src directory into multiple subdirectories because it had like 50 files already
Sadly completely removed the Make build system, so now you can only build with CMake (CMake is very much evil and even more evil but it’s the only one of the two that supports windows and I was tired of managing two build systems)
Removed the –graphics-scale flag, so now everything renders at native resolution
And some small docs and comment cleanup
It’s finally time to merge the performance branch. I basically redesigned the whole renderer and rasterizer around high resolution workloads, where it would still choose the old high-poly paths where they were faster.
Even snuck in some SSE2 SIMD code (picture), although I will probably be switching to a dispatch SIMD library like Google’s Highway. Not entirely sure it will be Highway or even if I will do that yet, that depends on the code complexity and before/after benchmarks.
Spent the first ~7 hours trying to find microoptimizations and each one was either a wash or worse somehow. The compiler works in mysterious ways.
At some point, I just gave up and let Claude have a go at the optimizing. Surprisingly it actually came up with a lot of interesting optimization techniques and solutions that I definitely couldn’t have come up with on my own. But it found so much, that the whole diff adds 5k lines of code. So now I’ll be cleaning up, testing, and maybe further optimizing that diff…
Just got an email from one of the maintainers that the crash bug i reported a month ago was fixed in the new release. libtermkey is a terminal input parsing library that I considered using and found a crash bug in. The bug was already fixed in neovim’s fork of the library, so i just pointed the maintainers at that fix. For anybody interested, here’s the issue link:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1142891
With the addition of the high-res protocols, the target resolutions quickly went from hundreds or thousands of pixels to millions. So I now desperately need to improve performance. So far it’s looking promising:
A whole model covering the camera 6 moving at 4k went from about 8 fps to 20-30 fps. Not ideal yet, but I think we’re on the right track.
Super dense meshes (more than 2 million polygons) improved 2-3x
Meshes with transparent parts improved about 2x
While these wins may seem really big (they kinda are), the jump in resolutions still means they are barely usable interactively. I do want to add gpu acceleration later (with the cpu path as a fallback), but until then I want to make it as fast as I possibly can.
Working on adding support for the other mainstream terminal pixel protocol of the main two - Sixel. Already did fully implemented the kitty protocol last time, so once I’m done with this one, I’ll be done with terminal protocols (at least for the near future, hopefully). There are some other ones like the iterm2 protocol, but that one’s pretty rare and I won’t add it unless I really don’t have anything better to do.
It’s basically done, just need to polish it and test it on a bunch of terminals and operating systems to make sure everything works before merging.
Finally completed the kitty protocol implementation. It is kinda slow, since the resolution is so much bigger, so I will try to optimize the high-res performance even more. So now the next two big milestones are:
Add Sixel on top of kitty, that’ll be a lot of work
Optimize the rasterizer even more
I was thinking of maybe adding gpu rendering as an option for super large meshes or high-res scenarios, but idk. I’ll definetly keep the pure cpu renderer for computers without gpus, but I’m not sure about a gpu implementation. It would be hard to do, less portable and it might not even work that well.
Ghostty is a modern and fast terminal emulator that i personally use. While testing the kitty protocol feature in rasterminal, I found a Ghostty bug that lagged my whole computer after a few minutes. I tested it in Kitty (terminal emulator) and there it worked fine.
I looked at Ghostty’s code vs Kitty’s, found the bug and reported it to the Ghostty maintainers. Mitchell himself (the creator) fixed it in like 2 hours.
Honestly I’m kinda sad I didn’t get to become a contributor and fix the bug, but since it was a small fix and I already described it in the issue, I get why Mitchell just fixed it in main.
Here’s the full issue: https://github.com/ghostty-org/ghostty/discussions/13719
Some modern terminals like Kitty and Ghostty have support for the Kitty protocol. It’s a feature that allows you to display high resolution content like images and videos in the terminal without it looking super pixelated. There’s also the Sixel protocol, which is basically the same thing but a little worse and more widely supported (even Windows terminal has it). I’ll add Sixel after i merge Kitty to main.
It is very clearly a worthwhile feature and I have been planning to add it from the start of the project. Today I finally decided that it’s time. Most of the work is already done on the branch and i’m just polishing out the bugs and issues before I merge.
In the photos, you can see the comparison between using just the block characters and using Kitty. It is a night and day difference. Although the performance isn’t great at that high of a resolution, but I’m still working on optimizing that.
Added templates and a config for submitting GitHub issues, for both bug reports and feature requests.
Also added a CONTRIBUTING.md for clear instructions on how to make, structure and audit code contributions. (tests, linters, formatting, etc.)
I tried to make some optimizations to the renderer by auditing the compiler generated assembly and looking for inefficiencies, but everything i tried turned out to actually hurt performance.
And GitHub actions decided to have a major outage right when i pushed the commits, so that’s awesome
The HUD used to be just a single line at the bottom of the screen aligned to the left. It looked pretty bad, but it did tell the user everything they needed to know. [picture 1]
First, I added model name sanitization, so models with control bytes like newline (\n), line feed or other symbols like emojis can’t break the whole program when trying to display them in the HUD.
Then, I completely redesigned the whole HUD to look better and be more robust, while still telling the user everything they need to know.
I went with this design [picture 2]:
There can also be optional indicators:
I also made each “island” cleanly disappear when the terminal’s width is too low instead of just slowly getting cut off letter by letter.
I think this design looks better, less cluttered and is pretty easy to understand at a glance.
I would very much appreciate a star on GitHub!
First, let’s talk about the small fix:
And now the big feature:
In the photos, you can see some angles taken in the first person mode that weren’t possible in the default orbit mode.
The current input handling is subpar at best; it interprets a lot of random keys as escape, which currently quits the program (This will also be changed to Q and Ctrl+C only. Kind of a no-brainer since this is the more standard choice for TUIs - for example btop - and it also means I don’t have to deal with the headache of distinguishing function keys and others from escape) and has a lot of other quirky behavior that needs fixing. I’ll also try to improve robustness against malformed input like a leaky SSH session. There are a lot of various edge cases and situations where you have no way to distinguish real input from malformed input. This is all a very major pain in the ass. And I thought figuring out the raw mode output would be the hard part. And after all this, I’ll end up with way less attention than strictly slower, less robust competitors, just because they already have an audience from other projects. I bet nobody will even read this. I’m basically talking to myself right now. Or maybe, just to spite me, the algorithm will push this more than my other devlogs.
I was researching terminal input libraries to use for this project because my current approach was so buggy that it wasn’t worth fixing. Both of them are memory bugs in the libtermkey library. There are forks of the lib with this fixed, but since many projects still rely on this lib, i submitted them anyways. I’ve also confirmed that the bug is present on my machine on Ubuntu 24.04 LTS on Neovim, because it links the vulnerable shared library. I didn’t mark them as CVEs (hopefully that isn’t a problem). If anyone want’s to look at the bug reports, they are at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1142891 and https://bugs.launchpad.net/ubuntu/+source/libtermkey/+bug/2161916
Fixed a Windows bug that crashed the program on model sizes bigger than 2GB, improved the orbit usability (fixed mouse drag behavior when models are turned upside down), and did a few cleanups and fixes in the CLI parsing.
I’m planning on adding a few CLI flags that I think can be very useful for some people, specifically better spin settings, starting camera angle/zoom control and a few more.
The branch is finally merged. I’ve accumulated a bunch of usability and code quirks and issues that i can finally get to now that the branch is merged. So now I’ll be reworking all the CLI flags, camera and other components. And we also passed the 700 commit mark with this merge!
Until now, the conversion picked the closest color possible mathematically. But from testing on a bunch of different models, I’ve realized that it doesn’t actually look very close to the human eye. So I swapped it out for a different method, called CIELAB distance. Now it still doesn’t look perfect, but as a fallback for older terminals it’s close enough.
The color terminal compatibility thing I was working on has landed in the branch, now I’m adding a –color flag to override the automatic capability detection for situations like ssh, where we almost always degrade to the 256 color pallete for maximum compatibility. So now you’re gonna be able to use the right color mode even if the automatic detection is wrong for your situation.
Some older terminals can’t display full 24bit RGB colors and use a 256 color palette instead. I’m adding a fallback so the older terminals can still use rasterminal. It still calculates the full old RGB colors, but then just maps them to the nearest 256 color with LUTs.
On a side note, I switched my terminal emulator from Kitty to Ghostty.
Did a bunch of small fixes and improvements in the last ~week. Right now I’m on vacation for the next couple of days, so I won’t be active. I am currently working on broader terminal and platform support, so older systems and terminals will be able to run cleanly with some fallbacks or will fail loud with a clear message about what isn’t supported.
Before, rasterminal built a shadow map on model load and displayed it during runtime. I now completely removed it, because it was slower during load, slower during runtime, sometimes looked broken on curved surfaces and added essentially no value.
Here, the first picture is with the shadow map and the second is without.
I did a bunch of small things alongside it, but the main feature for this devlog is the pararelization of the wireframe shading mode. It was previously fully single threaded, but it wasn’t that hard to pararelize, i just had to make the function it used to write into the framebuffer (draw_line) CAS atomic, to avoid race conditions. Now wireframe is the fastest shading mode in basically all scenarios.
Below, you can see a performance benchmark (built into rasterminal using a flag) with wireframe mode on a super dense mesh and how the wireframe mode looks like (you can also cycle the colors).
Gouraud was previously the main shading mode, but now I’ve just completely removed it. It was by far the slowest, while not adding any actual value to the project. Blinn-phong lighting is faster AND it supports more features (normal maps, etc.), so there was just no point in keeping gouraud. Now that I’ve removed it, the project has less code to maintain and a faster default shading mode with more features.
OBJ bump maps are textures for simulating bumps and wrinkles on models. Blender used to author bump maps as normal maps for some reason, so I made it automatically detect if it’s a normal map disguised as a bump map and handle it well, since there are a lot of old Blender OBJ models in the wild. But it also handles bump maps as bump maps normally, so it’s basically the best of both worlds.
Not that hard, just read it, pass it to the rasterizer and then display it. I did have a very sneaky bug in the loader that made the whole feature render wrong, where I just had to negate one value to fully fix it, but now everything fully works.
Below you can see the Khronos test asset for texture transforms. (tick means fully working, clearcoat is not yet implemented so it doesn’t show as working)
glTF sometimes authors a second TEXCOORD set (first one is TEXCOORD_0, 0 indexed numbering). The official spec only requires support for the first two TEXCOORD sets, even though there can theoretically be any number of them in a model. That said, major model exporters don’t author any more than the two sets, so to avoid extra work, I only added support for 0 and 1.
(not merged into main yet, I want to polish it a little and maybe add KHR_texture_transform support on top of it first)
After rendering the model in the transparent path, we have to resolve the transparent pixels, which basically just pastes in the pixels behind it and sometimes tints it based on how dark and what color the transparent material is (tinted glass will make the things behind it darker, normal glass will be mostly completely untinted). Before, the resolve function looped over every single pixel in the frame, even non-transparent and background ones. This is obviously wasteful. So I made it find the smallest box, in which all the transparent pixels of the whole frame fit and just looped over that. It recovered the overhead of the transparent resolve on models with smaller transparent parts, but it also made fully transparent or mostly transparent models slightly slower (it still basically loops over almost the whole screen on these types of models, so keeping track of the smallest bounding box while also still doing the same work as before made it a little slower). But the performance benefits for the average model outweighed the losses on rare fully transparent models. In the image, you can see the benchmark results versus the baseline.
Did the same template if constexpr (compile time if branches) thing that we already do for opaque/transparent models with the shading mode (was a runtime per triangle if branch before), which improved performance. It basically just makes a new copy of the function for each shading mode x each transparency mode, so 4x2 which is 8 slightly different functions and uses the correct one on each call. That increases binary size and icache pressure, but drops per triangle or per pixel if branches that hurt performance, so it’s an overall performance win. Benchmarked across a ton of different resolutions, models, shading modes, etc. (the benchmark by itself took like 2 hours to run)
Still a work in progress and I might not merge it into main, because it’s a lot of work for not that much benefit.
I have to catch a lot of bad_allocs and things like that for super large SSAA sizes, which is really annoying and can affect performance.
SSAA works by rendering at a higher resolution and then downscaling for better edges (makes edges smooth and fade-in instead of jagged)
Picture with 2x SSAA and without SSAA:
After two unsuccessful and unmerged branches, I finally did something useful on main.
I used to deduplicate the vertices in stl_reader and then reduplicate them in the loader. But now, we just dedup in stl_reader and use those. That also allowed me to enable ambient occlusion, which was disabled for stl before. And now crease smoothing fully works too. And it’s faster (runtime wise, load time is slower because of ambient occlusion and crease smoothing that weren’t there before).
So an all-round very positive change, and also an end to my non-merging dry spell.
But after actually replacing it and integrating it into the source code, i realized it’s not worth it, because rapidobj just has too many downsides and isn’t very actively supported anymore.
For example, it doesn’t support multiple mtl files, which is against the spec. So if i want to keep it spec compliant, I need to find a workaround for it, which just isn’t worth the speed increase of rapidobj, even if it’s like 4x faster (from my testing on a large obj)
The graph is from this article: https://aras-p.info/blog/2022/05/14/comparing-obj-parse-libraries/
Transparency works correctly, but transmission, which is completely different, is simplified to transparency. So for spec-complete gltf, I would need to implement the whole transmission pipeline, which I’m way too lazy to do right now. So for now, we just simplify it to transparency and call it a day.
For that, I have to completely rework the rendering loop and the rasterizers, so basically the whole core of the project. It’s going very badly.
And since performance is a big factor here, I’m going for a template approach so the transparency overhead doesn’t affect non-transparent meshes.
I did a hot-loop performance cleanup run, because the hot loops were getting slower with new features, and I found a big optimization opportunity in clip-near call sites:
clip_near slices off the part of a triangle that pokes behind the camera’s near plane, but almost every triangle is fully in front of it with nothing to clip. The pipeline still called it for all of them, and in the compiled binary that common case just copied all three vertices to memory and back, millions of times a frame. The fix was to check inline whether the triangle is fully in front and skip the call entirely if so. Result on a 2.77M-triangle model: median frame time dropped from 21.1 ms to 19.2 ms, about 8.7% faster.
Because of all the libraries I added, the binary size increased significantly.
I don’t use most of the vendored code, so I added build flags to ignore unused code in the final binary.
Other than that, I added some other performance flags and also a CMakePresets.txt.
It took longer than it should’ve, because I tested and benchmarked a bunch of other configs and flags.
I put off PGO for now.
I wired in Webp image format support by vendoring in google’s libwebp decoding library.
It wasn’t that hard to wire into the source code, the hard part was the vendoring, dealing with licenses and professional vendoring practices.
Only thing that remains to add for full gltf extension support is avif image support, but I’ll do that later.
I just added unlit material support, which means materials that are supposed to stay intentionally unaffected by lighting render correctly.
Pretty straightforward change, just read it at load time and then check if that material is unlit during the lighting pass and skip it.
Reworked the whole readme with better instructions, screenshots and a gif.
The gif was super annoying to polish, because i used asciinema which records all the setup escape sequences so there were gaps at the start and end that i had to manually read and remove.
And I also had to perfectly close it exactly when it did one full rotation and that took like an hour to perfect.
At least now I don’t have to touch the readme for a while.
Well, I tried to refactor the main rendering loop for like the third time, because it’s horrifyingly indented, but it has defeated me once again.
At this point, I’m confident that it’s impossible to do without a noticeable drop in performance, or g++ is just ragebaiting me.
I installed a rainbow indentation neovim plugin just to show the absurdity of this.
I just implemented KTX2 and Basis texture support, which are special compressed texture formats.
So now any models with these textures are viewable in rasterminal.
rasterminal is a full software 3D rasterizer and model viewer that runs entirely in your terminal. The whole rendering pipeline is implemented from scratch. I only rely on third-party libraries for load-time things like 3D model parsing, image parsing, and decompression.
It doesn’t touch the GPU at all, running purely on the CPU. That makes it completely safe for SSH usage, though it does come with a performance cost. I use multithreading to recover some of what’s lost, and the result is still significantly faster than any other terminal 3D viewer I’ve come across.
Right now I’m focused on broader 3D format feature support and transparent material rendering. If you want to try it out, I’d love to hear what you think, feedback and issues are very welcome over on GitHub.
https://github.com/PavolUlicny/rasterminal
Thanks for any support! 🙏