Finally… after all that work, we’re back!!

After almost 30 hours of work over the last week, I’ve finally finished the big refactor.

Changelog of the refactor

• Rewritten kprint backend
• Restructured KSTATE
• Hit my head on the wall A LOT
• Added a few new datatypes and macros
• Optimizations all over the place
• Added quite a few TODOs (more work 🥲)
• Restructured the source tree
• Changed the entrypoint (kernel now starts in an arch-specific kernel_entry, which builds a BootInfo and hands to kernel_main)
• Moved the userspace/ELF stuff from main.rs into the sched subsystem
• Rewrote the UART driver

… and so much more.

and of course i organized it all into one slightly large 144 file commit :)

Big changes are still happening

oh boy this is taking way too long

Rewritten kprint

kprint, the kernel print macro has been fully rewritten. The public macros have not changed, but internally, the macros now:

• Construct a kernel writer
• Call core::fmt::write on the kernel writer

The kernel writer collects the text of all calls made to it using core::fmt::write in an internal buffer, and on drop, it pushes a new entry into the kernel log ringbuffer in a format that is (currently only partially) syslog-compatible. Then, the kernel writer dispatches a log header and the full log text to all active log recievers (like UART or the basic framebuffer).

New Types

• PackedU8: A type which wraps all the bitwise logic for packing multiple pieces of data which are less than one byte into a byte
• UncheckedCell: A datatype with interior mutability, which can be uninitialized via MaybeUninit. When the cargo feature debug-checks is enabled, this type also checks for double initialization and uninitialized access
• NicheCell: A datatype with interior mutability, which can be uninitialized via Option. This datatype is designed to be used with values that can take advantage of niche optimization (where the datatype T has a value which is guaranteed to be invalid, like a NonZeroU64 being zero)

and oh boy, the big refactor: redid kernel source tree, ... commit is gonna be one hell of a mess…

Big changes upcoming…

I’ve decided to change what the kernel aims to be and how it will work. I’ve dropped full linux drop-in compatibility in favour of creating a clean, modern and optimized kernel which tries to fix design mistakes other kernels have already made.

New Design

The kernel will be POSIX-compatible, feature a namespaced VFS to seperate virtual files or devices like /dev/sda, /sys/class/hwmon, /proc/self/ and more from actual files. It will have namespaces like fs:/ for the actual file system, dev:/ for device files (like dev:/sda), vdev:/ for virtual devices (like vdev:/urandom) and more, including hw:/ for a modern hardware info interface, log:/ for a standardized logging interface and reg:/ for a global registry-like configuration system. Internally, the kernel will also feature KSTATE, a central static struct for storing everything the kernel uses. Also, ther kernel will move from strictly limine to a dynamic bootloader format, where each architecture has its own kernel_entry, which prepares the CPU and kernel into a common state (long mode, paging, …) and builds a standardized BootInfo struct, which then gets handed to kernel_main when it is called.

New Source Tree

I’m also currently working on massively reorganizing and cleaning up the kernel source, by for example standardizing where arch-specific code lands, categorizing and organizing the code better and more. Also, the kernel print system (kprint) is currently being rewritten to improve performance and integrate it into KSTATE.

Refactoring hell is here again!

Before I start on syscalls for the userspace binary, I’ve decided to first get most TODOs done, like updating the safety comments and rustdoc, refactoring the ELF loading stuff to properly utilize KSTATE::procs, proper error delegation and more.

I’ve finally finished refactoring the TSS, GDT and IDT properly by removing the struct wrapper for GDT and IDT, getting rid of all the Onces and more. Additionally, I refactored KSTATE::cpu to be an arch-independent container of arch-specific structs.

holy crap ive got so much left to refactor… and the new bodycam update just came out i wanna play but am stuck doing kernel stuff and some other stuff :’)

@everyone… WE HAVE USERSPACE!!! 🎉 :D

This is something I’ve been waiting for sooo long… a binary is actually executing in USERSPACE now (ring 3 and stuff)!!!

How it works

Currently, the code is extremely hacked together, or “gepfuscht” as we like to call it in Austria ;). Basically, what should be cleanly integrated with KSTATE::procs, should use a proper userspace and kernel stack and so on, doesn’t. All of this of course won’t stay like that, but it currently is like that to just get something in userspace running. The one thing that is actually done cleanly is the ELF validation and definitions, which live under src/elf. There is a struct representing the ELF header, an ELF PHDR and also an enum for ELF validation errors. All other logic was just thrown into main for now, like creating the processes address space, mapping its memory, copying the binary into the mapped memor, setting up its stack and actually jumping to it.

The current testing binary

Written in good old assembly of course, the test binary currently has the job to cause some sort of CPU exception, as syscalls or any other way to interact with the kernel or output something doesn’t even exist yet.

The current program manages that via just running the hlt instruction, which, when run from userspace (Ring 3) causes a general protection fault (#GP). This is because a userspace program shouldn’t just be able to halt the CPU or disable interrupts (via the cli instruction). That would defeat the whole purpose of rings and kernel/userspace.

The userspace binary currently lies under src/user-binary.asm, next to src/kernel/. It is build via running docker exec ferrite_os /ferrite_os/build-user-binary.sh. NASM was also added to the Dockerfile to make building the assembly file possible.

Virtual Memory Management is complete! (for now)

I’ve added methods for mapping pages and VMA together. I’ve also improved the kernel VMA granularity and added a kernel paging remap step to mm_init() which applies the kernel VMA permissions to the kernel pages. This is only intermediate until I switch to the linux boot protocol.

Next up, I’ll be adding a (very basic) ELF loader!

Lots of stuff

 
Changelog for the last two days

• Fixed the clean option in build.py
• Add kernel VMA setup method
• Update rustdoc in several places
• Add panic types to kernel_panic output
• Move SIMPLE_STATE out of main
• Improve per-arch code layout

… and an INSANE amount of optimizations, including:

• Marking certain code paths as cold (which signals the compiler to expect that the code path is unlikely to occur and optimize accordingly)
• Remove a lot of overhead introduced by Once and similar by swapping them for UnsafeCell<MaybeUninit<T>> and others and taking advantage of things like niche optimization
• Using bit-packing instead of seperate booleans for state tracking

Next up, I’ll be doing some more work on the VMM and finally add the full mapping methods, which map pages and VMAs together.

AddressSpace and VMA data structs done!

I’ve introduced a struct called AddressSpace which contains the virtual address of the root page table (the PML4 on x86_64) and a BTreeSet of VMAs (virtual memory areas).

A BTreeSet is basically a type of set which organizes data in a tree-like structure (in a B-Tree structure to be exact) with a time complexity of O(log n) for inserting, removing and searching for elements.

A VMA (virtual memory area) is an area in memory with specific flags, such as READ, WRITE, EXEC or USER. One of their main uses is determining what to do on a #PF (page fault): should we allocate a page here (lazy-mapping)? Or was this truly a faulty operation like trying to execute READ-only memory?

Also, I briefly attempted to switch over to the linux boot protocol, but I decided to do that sometime later, because it is slightly complicated and definitely did not break everything ;)

Finally finished fixing the PMM free method!

After needing to quickly relearn the logic of the free method because I somehow forgot it, it finally works again!

Additionally, I’ve added a little safeguard to the bootloader memory reclamation. Next, I’ll be continuing on the VMM and refactoring it to support userland page tables too.

Also, I’ve added a few core::fmt traits to PhysAddr and VirtAddr and also added a constructor which creates a new address type (PhysAddr or VirtAddr) with an address of zero. In addition, PhysAddr got a few new getters like as_ptr or as_mut_ptr.

Memory Stuff and General Fixes

Currently working on the PMM and VMM a little more, also cleaned up the main for better readability.
I also moved the PMM and VMM into the mm subsystem of the KState struct.

Additionally, I switched from rust 2021 to rust 2024, which also needed a little fixing to get working again. Why? Well to use let chains in if statements of course cause its like a little bit nicer to write! :)

Also, I partially upgraded scripts/x86_64/build.py to log to a log file and restore your terminal config on termination.

Currently working on getting a bug in the PMM fixed, getting bootloader memory reclaim to work and then I’ll do the VMM’s VMAs.

Everything finally works!!

I added support for linux and tested on WSL and native Ubuntu. Everything works perfectly (except AMD GPUs on Linux, didn’t have any hardware to test that…)

In the last two days, a lot of changes happened, which are listed below.

Changelog

August 19th:

• Window immediately opens on app start now
• App now works without a tray icon
• Remove fan speed display
• Add CPU temp sensing for linux
• Make Disk and Net Panels grow on more entries that can fit
• Add filter for unwanted mounts/filesystems on disks view
• Hide middle panel (GPUs) if no valid GPU is present (mainly happens on linux + intel gpu)
• Implement GPU stat reading for NVIDIA and AMD GPUs in Linux
• Switch to JBR (Jetbrains Runtime) to fix scaling issues on Linux
• Create & debug CI/CD pipeline (a sh*t ton of debugging mostly)
• Add the logo
• Add file lock for only one instance
• Add PawnIO install prompt

August 20th:

• Overhaul the tray icon behaviour and window controls
• Switch RAM graph to percentages from absolute values
• Add EMA filter for CPU temperature
• Add temperature data to CPU and GPU graphs
• Add color coded font for stats corresponding to graph datasets
• Fix up popups and the main class
• Clean up and optimize the whole app a little
• Fix Linux app launch
• Hide the AppImage mount under Linux

… and of course tons of tiny tweaks and fixes everywhere too. :)

All Sensor Data works now! :)

(on windows at least)

Disks

Per disk, you get its name and mountpoint, its R/W and its current capacity (used/total GB, progress bar and percentage).

Network Adapters

Per adapter, you get name and up/down speeds.
Names above 40 chars are shortened to avoid text of the speed and name colliding.

Next Up

I’m working on getting Linux working too. Will probably have that done by tomorrow.

RAM Panel and ADLX works!

(yes, i did the adlx part partially using claude because i don’t hate myself enough to work on the adlx stuff for 5h when i have to get up tomorrow)

CPU and GPU data working!

The Data Sources

For CPU, mostly OSHI. For CPU frequency, I’ve also decided to go back to OSHI for various reasons. CPU temperature is queried using PawnIO.
For GPU(s), D3DKMTHK is used for a list of GPUs and GPU temperature. For all other metrics, NVAPI (NVIDIA API) is used. AMD GPUs will also get support in the near future (via ADLX), Intel GPUs won’t be supported though.

The Architecture of the PawnIO Reader Data Path

The Hardware Manager calls the method requestData in WindowsReader which hands it a SensorData object, which contains the CPU temperature and a list of GpuData object, which look like this:

public record GpuData(
    String name,
    double tempC,
    double usagePct,
    long vramUsedMb,
    long vramTotalMb
) {}

This data is then passed to all subclasses of the hardware manager that need it (Cpu, Gpu).

Other Improvements

The hardware manager now has a new method called addUpdateListener with which you can register a class that implements the HwUpdateListener interface. The hardware manager will then call onHwUpdate on all registered classes as soon as it has refreshed its data via OSHI and PawnIO-Reader.

Additionally, the fan panel will not be displayed on windows because of reasons mentioned in the last devlog (too time-consuming to implement properly)

f*ck microsoft

After MANY hours of research and wanting to launch this project into orbit WAY too many times, I’ve finally managed to query almost all of the missing and/or incorrect data in windows (CPU frequency and temp and all GPU stuff) except fan speeds i just dont care enough anymore to implement that too (why later).

Why this is SO complicated

In contrast to linux (which has hwmon), windows has no standardized way of getting proper sensor data reliably. CPU temp (accessible via WMI) is often not properly exposed by the manufacturer causing it to return either zero or garbage. CPU frequency is exposed, but it kinda lags behind and is measured/calculated badly. GPU stuff is basically not exposed at all, expect via their half-working API called “D3DKMTHK” (no i did not smash my keyboard) over which you can pretty reliably get GPU temp and memory frequency, but usage and fan speed not so much. For that, I used NVAPI (NVIDIA’s GPU query API).
 

What? AMD and Intel GPUs don’t work with NVAPI?
i know and i will not fix it, i don’t have the energy or time for that (i’m sorry). you’re welcome to give me a PR tho

 
For the CPU stuff (frequency and temperatue) I used PawnIO, a kernel driver that can be scripted to do a lot of things. I use premade “AMX Modules” which basically are such scripts to get these CPU metrics.
The AMX modules in use are:

• IntelMSR
• AMDFamily17
• RyzenSMU

(you can find these here)

The Final Solution

A C++ shim which takes the data from PawnIO, NVAPI, “D3DKMTHK” and all the others and hands it to the java app via stdout. The java process runs the .exe everytime it needs the values, parses the returned json and saves the values.

What were u talking about fan speeds too difficult again?

You see, I could just do the same as with the CPU stuff (AMX modules). The problem is, the module LpcIO which exposes the EC (embedded controller) only exposes a direct path to it, meaning I’d have to implement and parse everything myself. The huge problem with this is: on laptops the EC is vendor-specific and undocumented, and on desktop, there are ~71 supported chips that require different implementations, which is probably the biggest waste of time, so I’ve decided to just skip fan speeds entirely.

VMM: Major Refactor aswell as Huge Pages!

I’ve refactored the VMM, aswell as added support for huge pages (2 MiB & 1 GiB) to it. Also, I’ve added vmm-debug-logging.

Also, I’ve updated the README to reflect the most up-to-date information and to look nicer.

Memory Management Updates! (oh and a logo too)

VMM Changes

I’ve added remap_page aswell as translate to the VMM (virtual memory manager). Also, I’ve added alignment-checking to ensure the VMM doesn’t corrupt memory when an incorrect virt/phys address is given to one of it’s methods.

Additionally, I’ve cleaned up the code a little bit by making things like the raw pointer stuff more readable.

LOGO!!! :)

I’ve painstakingly (hope i spelled it right) put together a handmade logo for the kernel. It’s pretty simple but I still think it looks pretty nice.

PMM Buddy Allocator done!

The PMM now uses a buddy allocator which gives us many advantages, including:

• Tracking overhead doesn’t grow with RAM size
• Faster (bitmap is always O(n), buddy allocator is O(1) most times, worst case O(log n))
• Order-based allocation (group a lot of free space together into one 4 MiB region for order 10 or 16 KiB for order 2 for example)

Next is adding a few methods to the VMM, and then I’ll work on the heap allocator to finally be able to work on the kstate struct :)

funny stuff

I’ve added a marker byte to the ring buffer that makes it possible to later properly parse log levels from the buffer. Also, I’ve finally figured out how buddy allocators work meaning I can start work on redoing the PMM :)