Github Actions and try-it-out scripts

I’m now finished preparing for shipping the kernel on stardance! :)

Github Actions

I’ve added a workflow which runs on tagged pushes in the v* format (anything starting with v), so that a tag like v0.0.1 is recognized as release version 0.0.1. The workflow builds a debug and release ISO, and also ships scripts to easily try the kernel out.

(and yes, the workflow and scripts were written with claude code, i really don’t wanna spend hours on that kinda stuff)

Exception Table done!

Finally, bad userspace pointers don’t bring down the kernel anymore.

What the Exception Table does

Only a few instructions in a few places in kernel code can actually produce a pagefault which is the user’s fault, not the kernel’s. If we handle those normally, we’d panic. The exception table however gets built at compile time and contains every place in the kernel that could produce userspace-caused page faults. We simply check if the page fault happened at one of these instructions, and if it did, skip the instruction and return a BadAddress error instead.

Memory Mapping and Paging Stuff

With the stack no longer exploding, I finally got to refactoring the page fault handler. Before, it operated based on whether the fault happened in ring 3 or ring 0, when it should actually mainly distinguish between kernel and user addresses.

Why distinguish between addresses instead of rings?

Imagine this: we’re handling a syscall, attempting to read userspace-provided memory, and we page fault. The fault happens in ring 0 but we’re actually working with userspace memory. Now, if we handle it like a kernel page fault, where our mindset is “Either there’s a valid VMA and we can map this or there’s a kernel bug so we panic”, a malicious userspace process could just provide a wrong address and panic the whole system.
Even worse: a valid pointer also brings down the system if it points into lazily allocated memory.

Now, the page fault handler looks at the address (and for user address page faults also at the current ring) and decides based on that what to do.

As proof that this works, the user binary now uses the stack for printing a text smiley, which works, even with the userspace stack now being lazily allocated.

What’s still left to do

Currently, bad user programs can still bring down the system because the page fault handler is among others still missing the so called “Exception Table” but more on that next devlog.

whoops

Turns out, when testing the new debug write syscall, it was slightly broken when passing wrong params, specifially when passing an unmapped or invalid address as a pointer to the string to write. Because of that, a page fault was triggered in the syscall, which blew up the stack, which was for some reason already filled more than halfway (around 9 KiB)?!?

Side note: in a syscall handler, there are only 16 KiB of stack memory available, and therefore 9 KiB before any error handling is already kind of a lot (normally it should be less than half in this application)

What was actually wrong

Turns out, the error was the freaking kprint stuff. Yes, the funny kdebug!() and knotice!() logging macros.. The main issue is that they’re macros, which expand in place.

Why that’s a problem

In Syscall::dispatch, there’s two kdebug!() calls. That blows the method’s needed stack to almost 5 KiB for a simple dispatcher function with 10 lines, which is unacceptable. The problem is that the stack memory for all the kdebug!() writes gets reserved as soon as the method is called, and each kprint macro call has a 2 KiB buffer and more behind it that it needs.

The Solution

To fix this, I replaced the internal __kprint macro with a function. This means that now the needed stack for the logging action is only reserved during the logging, not during the whole parent function.
By doing this, I managed to cut the debug write system call’s memory footprint from around 8 KiB to just 1.5 KiB, which is a huge improvement.

Final Touches

Simple Chat is now basically finished after some final touches. Lots of fixes have been made, along with adding more documentation.
Here’s a quick changelog:

• Fix issue where invalid timestamps would panic all clients which received them
• Fix key inputs being registered twice on windows
• Check user name length properly
• Fix multi-line message drawing logic
• Add a timeout for the initial handshake (5s)
• Validate chat messages and user names for control characters
• Write the timestamp on the server-side to prevent spoofing
• Fix CTRL+C not quitting the TUI
• Add a TUI message backlog limit
• Fix the TUI text input box (wide chars now work properly)
• Update documentation
• Fix a few other minor inconsistencies

Packet Types

After a (way too long) pause, I’m finally working on Simple Chat again. I’ve added a packet enum, which wraps all possible kinds of data communication.

What this means

Basically, now everything that’s being sent has the same interface: serialize and deserialize. You don’t need to worry about what is in this packet, just serialize it, encrypt it and it works.

This means handshakes aren’t loose bytes being sent between client and server and held together by pure hopes and dream (im praying somebody gets the reference), but instead a proper packet, which can also now be encrypted.

Heartbeats

Previously, there was also a big design flaw. If no message was being sent, you couldn’t tell if the others were just not sending anything or had already (uncleanly) disconnected. The TCP stream looked the same in both scenarios. Because of this, I’ve added a Packet::Heartbeat type. It contains no data, and if it gets received by any party, (almost) nothing happens. It’s sole purpose is to keep the stream alive.

Now with this, lets say we’re the server: a client doesn’t send a heartbeat twice. We now know that they’ve likely crashed or not properly disconnected in some other way. Because of this, we can now kick them instead of keeping a connection live.

Server Rewrite!

I’ve rewritten the server using Tokio instead of Threads for improved performance and less server-side overhead.

What Tokio is

It utilizes Rust’s async implementation with futures to create a much more lightweight alternative to threads. Basically, you can summon multiple tasks and instead of them being full threads with seperate 8MB stacks and so on, they have way less context switching overhead and a much smaller stack per instance.

Other Server-Side Changes

In the initial handshake, the client provides its name to the server. If the server receives a message with the wrong name from that client, it will replace the name with the originally given one.

Encryption

I’m still working on finishing up encryption for all messages and the initial handshake.

im almost done already what

So… it kinda works????
The client AND server, the TUI, the new message structs and all of that work now.

Changelog

• Reworked Message Struct(s) and multiple message types (Chat, System)
• Proper connect handshake
• Rework server to use SystemMessages to broadcast client connect/disconnect events

What’s planned

• A config file to store your name (~/.lan-chat)
• Encryption (between clients)

TUI Time! :)

The TUI works now, I only need to polish it (stuff like scrolling when message box is full)

Additonally, I’m planning to add a special kind of message that the server sends on client join/leave events.

New Project!!

This is LAN Chat, a TUI chat messenger to use in the same LAN as others.

Build using Rust with ratatui (a TUI library)

Currently, I have a working client/server system, which sends serialized message structs. Next up, I’m gonna work on getting the TUI running.

DebugWrite syscall!

The userspace process is now able to ask the kernel to write log messages! This means that finally, you can actually properly do something from the userspace program.

Next, I’ll be implementing userspace page fault handling

Syscall Dispatching, Syscall returning and Exit Syscall

Finally, after all this time: User programs can now interact with the kernel!! :)

Changelog

• Added Syscall enum which represents all possible syscalls and their respective IDs
• Added SyscallArgs whichs contains syscall number and args in an architecture-independent format
• Added SyscallResult and SyscallError
• Added syscall dispatching logic
• Fixed kernel data leaking into userspace via initial_userspace_jump by clearing CPU registers before the jump

the end of stardance is approaching… i wish all of you a lot of fun for the last 9 days.

Syscalls!

Changes so far:

• Added gs_info struct, which saves the user stack and holds a pointer to the kernel stack
• Added syscall::init which enables the syscall/sysret instructions and sets up the STAR, LSTAR and SFMASK registers
• Added syscall_entry which builds a UserFrame (a snapshot of the CPU registers on syscall entry) and calls syscall_dispatch
• Added syscall_dispatch, which currently logs the received syscall number and halts (the system call return path is not implemented YET)
• Added UserFrame which holds a CPU register snapshot which gets taken upon entering the kernel

ELF Loading Refactor and improved build system

After a week of basically no progress, I’ve finally finished moving the ELF init sequence from main.rs into the proper methods under the sched subsystem.

I’ve also simplified and improved the build system (build.py/docs.py). They now use a shared lib.py to avoid code duplication. Also, build.py got a few changes:

• Removed build cache (cargo already does it)
• Swapped console restore for proper serial control char output filtering

Bug Fixes

The docker permissions have been fixed, as under Linux, root would own parts of the build files which is of course annoying as you can’t easily interact with those files then.

The github-hosted rustdoc was failing to build as it could not find USER-BINARY, so I just faked user-binary via touch user-binary.

time to go to bed yippeeeee :D

Finally… after all that work, we’re back!!

After almost 30 hours of work over the last week, I’ve finally finished the big refactor.

Changelog of the refactor

• Rewritten kprint backend
• Restructured KSTATE
• Hit my head on the wall A LOT
• Added a few new datatypes and macros
• Optimizations all over the place
• Added quite a few TODOs (more work 🥲)
• Restructured the source tree
• Changed the entrypoint (kernel now starts in an arch-specific kernel_entry, which builds a BootInfo and hands to kernel_main)
• Moved the userspace/ELF stuff from main.rs into the sched subsystem
• Rewrote the UART driver

… and so much more.

and of course i organized it all into one slightly large 144 file commit :)

Big changes are still happening

oh boy this is taking way too long

Rewritten kprint

kprint, the kernel print macro has been fully rewritten. The public macros have not changed, but internally, the macros now:

• Construct a kernel writer
• Call core::fmt::write on the kernel writer

The kernel writer collects the text of all calls made to it using core::fmt::write in an internal buffer, and on drop, it pushes a new entry into the kernel log ringbuffer in a format that is (currently only partially) syslog-compatible. Then, the kernel writer dispatches a log header and the full log text to all active log recievers (like UART or the basic framebuffer).

New Types

• PackedU8: A type which wraps all the bitwise logic for packing multiple pieces of data which are less than one byte into a byte
• UncheckedCell: A datatype with interior mutability, which can be uninitialized via MaybeUninit. When the cargo feature debug-checks is enabled, this type also checks for double initialization and uninitialized access
• NicheCell: A datatype with interior mutability, which can be uninitialized via Option. This datatype is designed to be used with values that can take advantage of niche optimization (where the datatype T has a value which is guaranteed to be invalid, like a NonZeroU64 being zero)

and oh boy, the big refactor: redid kernel source tree, ... commit is gonna be one hell of a mess…

Big changes upcoming…

I’ve decided to change what the kernel aims to be and how it will work. I’ve dropped full linux drop-in compatibility in favour of creating a clean, modern and optimized kernel which tries to fix design mistakes other kernels have already made.

New Design

The kernel will be POSIX-compatible, feature a namespaced VFS to seperate virtual files or devices like /dev/sda, /sys/class/hwmon, /proc/self/ and more from actual files. It will have namespaces like fs:/ for the actual file system, dev:/ for device files (like dev:/sda), vdev:/ for virtual devices (like vdev:/urandom) and more, including hw:/ for a modern hardware info interface, log:/ for a standardized logging interface and reg:/ for a global registry-like configuration system. Internally, the kernel will also feature KSTATE, a central static struct for storing everything the kernel uses. Also, ther kernel will move from strictly limine to a dynamic bootloader format, where each architecture has its own kernel_entry, which prepares the CPU and kernel into a common state (long mode, paging, …) and builds a standardized BootInfo struct, which then gets handed to kernel_main when it is called.

New Source Tree

I’m also currently working on massively reorganizing and cleaning up the kernel source, by for example standardizing where arch-specific code lands, categorizing and organizing the code better and more. Also, the kernel print system (kprint) is currently being rewritten to improve performance and integrate it into KSTATE.

Refactoring hell is here again!

Before I start on syscalls for the userspace binary, I’ve decided to first get most TODOs done, like updating the safety comments and rustdoc, refactoring the ELF loading stuff to properly utilize KSTATE::procs, proper error delegation and more.

I’ve finally finished refactoring the TSS, GDT and IDT properly by removing the struct wrapper for GDT and IDT, getting rid of all the Onces and more. Additionally, I refactored KSTATE::cpu to be an arch-independent container of arch-specific structs.

holy crap ive got so much left to refactor… and the new bodycam update just came out i wanna play but am stuck doing kernel stuff and some other stuff :’)