Assembly os x64 custom kernel and bootloader

Ok i have successfully made detokenizator which goes trough the token id book that many times as a token because token id =id book entries untill the token, and also i have made matrix–vector dot product calculator which goes through all token ids and translates them into a big amount of vectors which then in another step i will be doing comparison hbuf with all these vectors to find a token with lovest diff and that will be our output token, but big but it still isn’t tested i will be testing it when i will have the full command done for now i just want to live in an illusion that my code is error free

There was a problem with pointers in rnsm found it when i did double check and from there errors just piled up but at least its fixed now

Today i got help from my great teacher and was able to do more than usual, today i did hbuf setuper it basically just calls fcon 576 times to translate the values from 16 bit quantization to 32 then i have written a function RMSNorm which uses hbuf and weights you give it and calculates rescaled feuture vector

I have no idea how to fix it so i retreated ro the ol reliable method by first hardcoding it and after some time actually fixing it

Today i am a bit better so i did the harder part that is needed to be done, i have made tensor info gatherer, basically each layer has 9 tensor info, so for my model 9*30 tensor infos, also the string i am comparing in the function needs to be like [blk.][number of current layer in decimal(use bintrn)][specification string], it still doesn’t work but it will be something like infinite inc loop to memory or division basically smth that generates general protection, also because I was smart i saved my previous error where i used tensor info praser so i didn’t Neded to reerite it.

Ok so uuh those 5l of Kofola wasn’t a great ide today i got a fever or flu or whatever the thing that makes you feel hot but i made some progress even though it was hard cuz i had a headache and couldn’t do anything so what i did i made a f16 to 32 translation and you might say hmm but this doesn’t seem too vomplex first of all i had to split registers inro 3 sections do calculation on them and the glue them together to xmm0 but what you might not know is if you don’t allow xmm is 32 bit mode it will throw gp an fuck the whole code and you cant know why it crashed because whole runtime in crash log looks absolutely normal for some reason also I don’t actually care about the time anymore i just want to get this working so i can get a laptop to not code on my phone

Ok bad News today i drank 5l of Kofola and now me attention me hyperactive me cant concentrate but did token and bos token enumetion there was a problem with division because of no xor rdx,rdx but ez fix if boestoken add 1 it adds tokwn

Made the metadata praser work i was using tensor format and nkt metadata Format also verifyed it works on pc

Tokenizer done now for vector context calculator . The tokenizer was somwhat refusing to work but one rewrite later it subdued to my power it is split into teo parts the text decoder and encoder decoder formats the words for encoder and encoder finds tokens for the words

Ok i gave lost the war against the matadata praser i will not fight it now i will fix it in second ship for now i have hard coded the data needed and also made the core of the tokenizer which was relatively easy compared to the praser

Ok all metadata entry data prased but when i delete the print at the bottom it for some reason triple faults going to report every ±9 hrs because i need to speedrun this because the stardance is ending soon now onto tensor data praser

Ok bigg devlog todat i decided to speedrun this so i can ship this project as fast as possible also changed the model to a smaller native f16 quantization and It works which is a proof that anyone can put their model into sofaos and run it rn it just loops troug metadata to find general architecture

Hello good news today have started on setting up the data from GGUF rn it is just getting version metadata entryes and tensor entry, I Also started a loop for the metadata entryes but just started this one was easy cuz it has prefixed everything 4 bytes gguf magic 4 version 8 metadata entryes and 8 tensor info entry which makes 24 bytes in total

I have successfully loaded whole LLM problem was that It was loading 1 gb by 512 bytes and that takes very long and thats why it seemed it hanged I fixed it by giving it a counter so you can see that it doesn’t hang

Ok sooo I made the code buuut it hangs when I do the loading idk what happening will have to debug

Ok I give up on writing it, cuz GGUF file structure is ass and I can’t write it in fucking the way it should be I am going to pull a cheap trick by putting a control string at the end of the file to know when the file is ending below is structure of how the whole file should look like

Ok I was figuring out how to load the gguf file which I think I got right but I can’t find a way to know how many sectors to pull out of disk because the metadata in gguf aren’t always the same nor size of the entries are the same so there is now way to know when they end and I need them to find the size of the file in picture you can see how the metadata is structured

Ok I have made a new command inout it let’s user chose number port in and port to listen from and it spits out response from the port as you can see I tried corrupting time register and it worked

Ok so I half rewritten the code for sector read I had to drop hard coded 0 and look for actuall devices to get the data from as you can see it works still

Ok so I was working on gluing gguf file at the end of my iso you can see image it was successful also needed to generalize ahci driver to work on all PCs now writing the gguf retrieve from disk code

Ok so uuuh….. Somehow secr fucked itself up and I had to fix it the thing is that from 0xffffff… Somehow one of the f magically dissapeared and I had to basically trace whole Drs sector to find what was wrong also waking aps for some reason isn’t working even I am using textbook code to wake it up I now have no idea why is it not working mameybie I will gat back to it at the end and now do the special function because everyone on stardance for some reason started doing X64 os so I had to come up with something radical, I am going to make a full local assembly llm chat engine like llama.cpp fully in assembly for sofaos all things from scratch tokenizer network loading calculation and other stuff I have nothing really to show so just enjoy some random pic from my gallery

Aps not responding should put 67 to px67 but aps not doing idk if because of qemu or whatever I don’t want to debug this piece of shi

Spent I don’t even know how much time I lost track of time doing this shit I did it whole night I am happy that I am going to sleep at 12:33 the error was that I wasn’t sending enough sipis so the aps didn’t actually wake up but I didn’t knew that so I was tracing the whole os to find the error then I actually checked on internet how should it be done and that’s when I was finally able to fix this simple ass problem

Finished the core wakeup now I can wakeup any numver of cores that I have and now I am going to make some serious multitasking i still need to do the trampoline to launch the cores to 64 bit mode but they will be fast also the limit of cores for legacy PCs is 255 but it is still engouh to wake up and epyc

Ok I have found the number of cores which is a huge milestone now I need to find what is the bsp to not wakeup already woken up core number of cores is in a memory buffer 256 bytes big cous the CPU doesn’t have enough IDs for more than 255 cores so even In future it will be ok

Found Madt table finally going to get number of cores to wakeup

I did some preparations for init sipi sipi trampoline to wake up more cores for multitasking it just finds acpi rsdp in memory I also want to be able for user to chose how many cores to wakeup

Added the ability to read what’s on the disk there was a problem with R14 cuz one function that I called used it but push and pop fixed it after I traced the problem without it would print 20 endlessly till it crashed

Ok I have found the solution instead of looking trough all ports it stops on the firt found idk how it works but it works now I can extract data from disk

Something is very wrong here. Every debug marker along the boot/init path prints exactly like it should, and it still never makes it to the shell. Going to need a proper trace through this instead of guessing.

PCI bus enumeration, done by brute-force — loop over every bus, device, and function number, write a config address to IO port 0xCF8, read whatever comes back from 0xCFC. All ones means nothing’s there; anything else gets recorded. Once every device on the bus is found, it scans the results for a class code matching AHCI (SATA storage), grabs that device’s BAR5, and starts bringing the controller up — reset bit set, poll until it clears, enable global host control. This is what opens up actual disk read/write instead of relying on BIOS

Added whatindr just copied function whatincr and changed registers from control to debug

I added so when the user types it automatically moves screen to VGA buffer so it doesn’t overwrite bad memmory and implemented this thing for cpritn too so if you are near end of the screen it automatically moves up (number of rewrites this function took 3) in pic you can see how cpritn automatically moved up

Added translate which translates number from hex to dec and also the other way and command to see what in memory I had to make a new function to accept hex and Dec and figure out which one is which than grab byte from memory not 8 bytes was stuck on that then output which was easy cus I already have function for that then added translate which just uses hexbin function for Dec and bin for hex and outputs in the other format also I don’t have much time to code because I am a slave on a construction now and I get paid absolute minimum

Ok so today I have made saythis command which is basically echo (system says what you tell it to) and whatincr which basically spits out what is in control registers which might be useful for some repairman or whatever but I had to make new functions like bintrn which turns normal pc number into decimal ASCII string but I had the loop wrong it started at wrong point but now it should work

Added scroll up, just the reverse of scroll down — but still needed its own direction logic and its own set of variables reset correctly, and the first version was so badly written that i just rewrote the thing

Limited coding time cuz i am on vacation in Berlin but made progress on scrolling — the screen now scrolls down properly when text overflows the bottom. Scroll up still doesn’t exist the first attempt bookkeeping was so tangled it wasn’t worth saving, so that one got scrapped rather than patched.

Added more text commands and started on ones that actually do something, like echo. Also started work on moving text up and down the screen, which is lowk hard

Completed the command system — a compare routine (cmper) checks whatever’s typed against a table of known command strings and jumps straight to that command’s handler on a match. First command wired up was cls, mostly because the screen clearer routine already existed. Everything else gets added the same way from here.

Terminal UI is functional now — Enter does something, and extended interrupts (arrow keys) get decoded properly instead of coming through as garbage. Also added a distinct sys response visual state versus just characters The color choices right now are rough — need to actually look into color theory instead of guessing.

Forgot to say this last time, but there’s now a basic (not working) concept of user input reaching the system layer — just the shape of how it should work, not the real thing yet. Finishing that up next so it actually looks like a terminal. Worth sayi the cursor (curzor) tracking and the cls screen-clear command both went in a little earlier and are what everything since has been built on top of.

Text now behaves like a normal shell — typing lands where it should, and added a working delete/backspace key plus some general UI cleanup. Starting to lok like an actual terminal instead of a proof of concept.

Keyboard handler is fixed. Can freely type now. thought: there’s currently nothing stopping unrestricted writes from going past where they should, which means someone could in theory type their way into overwriting parts of the running system. Needs boundaries before this goes further.

Fixed yesterday’s error. Now testing on real hardware to check whether the machine actually virtualizes PS2 keyboard controller the same way QEMU does, since the handler logic depends on that

Tried implementing a keyboard press handler. Sadly, it does not work. Something in the interrupt handling isn’t right — more debugging needed before typing does anything useful.

Finally transitioned into 64-bit mode, so the actual OS can start getting written now. Not going to explain every step in detail since it’s always the same , but simple: built page tables using 2MB pages so the fourth paging level (the 4KB page table) could be skipped entirely, set up the stack, and did the rest of the long-mode setup (PAE, the EFER long-mode-enable bit, then paging itself) before the far jump into 64-bit code.

Big milestone — got it running on an actual physical machine instead of just QEMU. Had to go back into stage 1 and loosen a couple of assumptions that only held up in an emulator; real BIOSes are pickier than QEMU’s about a few things. Nothing structural changed, just less forgiving case to handle.

Ok found the bug it was bad cmp flag check still a bad Habit I have from arm to instead of je is Zero try to do arm syntax and do jz, (BZ branch if zero) but now it works ok

Pulled an all-nighter and finally got a boot working — turned out QEMU was set to -nographic, so my VGA output was being generated correctly but never actually displayed. Once I fixed that, i found a new problem up: once in 32-bit mode, my code started printing from the top of the screen and overwriting text the BIOS had already written. Progress, but a new bug was right behind the old one. Still it doesn’t want to move forward it should print k at the end so something is wrong I am going to investigate It.

I hate this stupid code , literally the only thing I do is find an error, find what could possibly cause it, fix it, and it still doesn’t work. Do that a hundred times, and it turns out the BIOS — which promises not to touch DL — corrupts it anyway during interrupt 0x16. Whatever. Still not fully working, but at least I’ve got the core structure for moving into 32-bit mode. I hate this — why do all x64 processors have to wake up going “uhm I think I’m an IBM processor from 1967” instead of just dropping into 64-bit like ARM does? Also my PC screen broke, so now I’m SSH’d in through Termux to code this thing. I hate assembly but low-key need a new PC — that old Compaq 6510b with 2GHz and two cores isn’t cutting it anymore.

Getting close to 32-bit mode. Added a memory tester using INT 0x15, EAX=0xE820 so the OS actually knows how much RAM exists and where, instead of assuming. Also built two different A20 gate openers — the fast method through port 0x92 first, and a manual read/write test on mirrored addresses as a fallback if that doesn’t take. If one fails, the other starts

Went back and leaned stage 1 more heavily on the shared print routine (cpritn) instead of duplicating printing logic, and tightened up the handoff into stage 2. Stage 2 now loads the kernel using a precomputed Disk Address Packet instead of recalculating the sector count and offset from scratch every single time it runs.

Added LBA (logical block addressing) table implementation — sadly no love for old hard drives that don’t support it. Also added a test so if the computer’s too old to do extended reads, it just refuses to boot instead of silently corrupting stuff. Auto-retry if something goes wrong, and it jumps to address 0x7E00, where the end of the 2# stage starts.

Did some testing on the 1# stage bootloader today. Now I need to link it up to the 2# stage and then to the actual OS — right now the 1# stage basically just saves what the BIOS hands us (bios present) and tells the system where the 2# stage lives.