Devlog 8 - Will it Survive The Winter?

A lot of time, this time went on fixing mistakes, which I did by making this application just for macOS, but I was building this to support different platforms, I can’t survives other platforms

1. Just had one JSON file

• for the complete configuration of the layouts.
• I realised that certain layouts which I am giving pre built were going to have different keys bindings for different platforms
• so I just separated my layouts into different files for Mac, Linux and Windows.

2. While talking to Claude,

• I got to know that the data of the applications which are edited are supposed to be saved in a different location other than the installation location.
• It is different for Mac and Windows, I was actually saving my data where my application is installed, so I fixed this issue and stored layout configs data at their appropriate location in the different system

3. So the special function keys, if you reading my devlogs, you are aware about this

• special function, such as increasing or decreasing volume and brightness, skipping and pausing and playing media
• these functions cannot be triggered by virtual keyboard. They are bounded to pressing keys in physical keyboard.
• to trigger these special function, I used Apple script on macOS, but Apple script are still failing on some cases
• so I used a swift language code , I did not understand how I could write swift inside rust but, stack overflow works. I got the code from stack overflow.

I also added some pre-layouts for various tasks and helping in application such as VS code, Photoshop, fusion 360 and while giving presentations.

Devlog 8 - Polishing is really Boring

• The project is near completion on terms of features
• Now i am just Polishing the app doing things like improving UX, Text Content, adding minor features.

Despite that here are two major things i did

1. Special Function Support

• remember my earlier devlog where i mentioned that, increasing and decreasing the brightness or vol is not as simple as just some keypress on keyboard
• These comes under special fucniton and although you can trigger then via physical keyboard, but can’t via the virtual ones.
• to do that you need to run some special code, on macos via Applescript and on window vis winShell, i am still figuring it out for linux.

2. Appearance settings

• user can now customize button’s color, bg, add icon and images (custom images are under dev)
• i did all this with backward compatibility in mind (Big Brains)

CI/CD Pipeline

• Used Github Action to build the application for major platforms (macos, windows, linux)
• yeah all that, thousand build erros , waiting 10 min for each github action just to see a red X, i will not rant i am past that now

Future

i will be straight forward

• add custom image support
• verify and improve the cross platform functionality
• add some pre built layouts
• maybe add a guide/tour for the layout builder

Devlog 7 - There is always a Solution

Focus V/S Feature Dilemma

• I thought it would be a cool feature addition to display the current resource consumption (CPU/RAM) and some other system details of the computer right on the mobile app.
• but my concern was that these features are basically "extras" and I really didn’t want to slow down my core functionality.
• I was about to drop the idea entirely, but at the last minute I decided to ask Gemini about it and guess what? There is always a solution I mean, what did I expect? Its engineering after all

Smart Subscription Architecture - the solution

• well, my main concern was the wastage of resources and WebSocket bandwidth, which might bottleneck my main selling point: ultra low latency under 5ms control command communication
• But this architecture perfectly solves that.
  • so basically, the server will only blast data at me if I am explicitly subscribed to it. Otherwise, it sleeps.
  • and since this project uses WebSockets on a local network, there is generally only one user. So, if I unsubscribe from the service (like closing the telemetry component), that particular data broadcast will completely pause, saving precious resources on both the mobile client and the computer server
• forgot to mention : the server updates this system data every second if user is subscribed, else sleepyy

The Scalability FOMO

I am well aware my app won’t probably be used by more than 10 people for 10 minutes, but my Senior Software Engineer came shouting -

“Your current method to render different views, that too without using react-router, it will become scalability issue as application grow”

• in my initial HL system design i was going to use a single page and conditional rendering for different layouts on same page to avoide a package i decided not to use the react-router for my client web page

• but now i have like 5 pages and as i am building i keep getting more and more ideas…

• I fixed this issue by

  • creating a standard state for different viewModes and changing props/types to support it (nothing just some painful refactoring)
  • implementing a component mapping (view registry) for all the views. Now I can just plug in any number of views without writing spaghetti if/else statements.

The important thing is as i am not using the react-router, i will to keep an eye on memory and component lifecycle to free up the space in memory from components which are not in view.

I should have used react-router from beginning 😭

I am thinking to add a button to force kill a process from the phone, but again i am standing at a point where these features are kind of feelign like extras and note my core application

• attached image of the QR code that i use to pair the mobile with the computer, will add some security in future

Devlog 6

• My this session was so exciting that i only stopped because I reached the 10 hour limit of the hackclub.
• Well, I think throughout this project, I have been doing things which I don’t understand, things which were boring
• but right now this is the point when everything clicked, finally got the hang of RUST

Its gonna be long Devlog read at your own risk(10 hours dump. Note: the maximum limit is 4000 characters so I cannot write everything here just this much)

1. Crazy things this time

So if you may know that in this project, one of the things is to simulate the actual keyboard through the application

Here is a interesting thing i found about special fucniton keys

• those special functionalities keys work strangly, on MacBook, we press F1 to decrease brightness, so i coded my applicaiton to to press the F1 but nothing happened
• in reality, most OS do not registers the F1 and F2 key for changing the brightness but it actually uses some code tied to hardware to trigger the special fucitionality

How did i solve this?

• well, I read about how actually a Operating systems register the keys, and trigger those special functions.
  And this was the reason pressing the shortcuts were not triggering those special functionalities into the computer. As the OS was not looking for me to press F1. it looks for a special code which is tied with the keyboard itself, which was not in virtual keyboard,
• solution is simple. You have to find out what code is used by a particular operating system to increase or decrease the brightness, and then you will have to trigger that code when the key is pressed.

2. Macros

I implemented macros, you can basically press a sequence of keys with control delay. Just on a single click. It is spamming a key running some keyboard shortcuts on a single click.

• I created different thread to run this macro so that it does not freeze my main application
• this is the point which like frustrated me for very long as it is been like three years when I learned concurrency and threading in Python and I did it again in rust this time. I used MPSC (Multi-Producer, Single-Consumer) here. I don’t fully understand it, though, but yeah, I know why I used it and how I used it

3. App Lanucher

So I just liked a button, which upon clicking will open up any application on your computer

• but the main thing was not actually opening the application, it was which application to open, Open an application by running a command on operating system, you have to know their exact official names, and a normal user does not know that
• so, I scanned the whole application folder onto the operating system and found out the official names of each application. Then I showed the list in a dropdown to user, from where user can select the names of applications.
  -well, it took me a lot of time because this was my third approach which worked my initial implementaion were like I would force the user to open up that application and then my application can register it, but obviously, the above approach is better than this, so I dropped it.

Jupiter (idk, maybe because its big, and who doesnt like sloar system?)

4. Context Aware Layouts

This is it. The big one. This was the most important, most time-consuming, and absolute flagship feature of the entire project.

• so, what is context aware layouts? So in the application, one of the parts is to create layouts, which helps with your work aespecially in the productivity/professional applications
• so lets say you are coding or working, constantly shuffling between multiple desktop apps, with your phone sitting on your desk acting as your custom macro pad. This feature actively extracts the currently focused application on your computer and instantly switches the layout on your phone to match it.

Devlog 5

Loking back and accepting

• this time I saw what happens when you build a project without any proper system design,
• things were so clunky in my application that I was not in mood of creating another groundbreaking feature
• i just spent time improving existing features and make them actually usable

What went down

So basically, I had done a dumb thing very dumb thing. I did not realize that the buttons on the actual keyboard have different states, you know when they are being held down OR they’re being pressed OR they are being released, this info matters and I completely missed it.

1. so now, basically, my key presses have a state

   1. Down (pressing down)
   2. Up (releasing the key)
   3. Pressed (holding the key)

2. So the SHIFT button which I had implemented earlier was only clickable, user couldn’t hold it, but the shift button is meant to be held, so I fixed this and other similar buttons like alt, control and command

3. Improved the portrait and landscape switching between the views in layout builder. This is just normal UI improvement Nothing special here

4. Other than these two improvements above, there has been numerous other small little improvements which took me a lot of time, so I don’t really have much to write on this devlog, changes are like

   1. I improved the whole conditional rendering thing, it was messy as I decided not to use react router for this project as there weren’t any actual pages or URLs, just bunch of different control layouts.
   2. Improve my state management to handle menu overlay properly
   3. Overall code quality improvement, creating utility functions and type safety.

and a lot of my time is invested in learning Rust as this is like my second project in rust (i leanred rust like a year ago and now i remember nothing, thanks to my good c++, i have held my ground so far)

Bottom line → Better UI and UX, button can now be held and tracked weather they are being pressed or released

Devlog 4

Control Layout

first i would like to define what i call a control layout, as i reliased i never relly defined it properly

Control Layout: It is a layout of buttons and other control elements like

For Example

• control layout for playback control will have button/controls like volume up/down, play/pause, fast forwarding and controling playback
• Maybe some specifi button to help while editing video
• Or controls for some particular Applications

So basically, these controls can trigger and functionality or a shortcut or a macro onto the computer that the client is connected with

Single click on the phone screen will be able to run a complex keyboard shortcut

User can map shortcuts based on particular applications, and he does not need to remember keyboard shortcut.

They don’t even need to click anywhere on their screen. The phone will act as a secondary keyboard or a small macropad.

Customizable Control Layout

The application contains some prebuilt control layouts

But different user user different application and have different workflow, so i created a featrure to let users

• create his own custom layouts where he can place different buttons anywhere on the screen and assign what work each button will be doing.

As of now the current active applicaiton detection is not present but i will add this feature soon.

Details

• basically, I have an interface which tries to mimic a mobile screen, and I have divided that screen into grids with CSS

• I can move the elements around and resize them.

• I track the position, height and width of those elements and store them in a JSON config file upon saving these layouts. This JSON config is sent over local web socket to the Rust server.

• Along with just the information about the UI, there is also working information, like name of button, its functionality etc

• I have a component which is quite opposite of a builder which is a renderer.

• This renderer takes up JSON config and renderes the different components and maps the functionality, in smple word it takes json and create UI.

Devlog 1

A Dynamic Macropad

I want to make a macropad which will be able to change its nature based on the user activity

video Editing -> rotation knob can control timeline and zoom

playing media -> knob now controls playback and volumne

And user can set custom key functionality based on which Apps are currently user is using and can also set modes for the macropad

I have a good coding experience i can easily build the companion app, but writing the firmware and hardware is completely new for me.

• I followed the stardance documentation and with AI help and designing the schemantics was fairly easy

• Now i am learnign about PCB designs, and it also looks simpler than i thought

My plan for 11 pins

• 3x4 keyboard [4 pins]
• Screen [2 pins]
• Two Rotatory Knobs (no clicks) [4 pins]
• Backlit RBG Lightning (I don’t know how to do the backlight thing) [1 pin]

Japanesse Duplex Matrix

I used something called Japanese Duplex Matrix instead of standared row and column design

• this design takes up only 4 pins for 12 keys, where as the row and col design will take up 8 pins for 12 keys

Dont mind the PCB, i have no idea of PCB right now, see you till the next dev log

Devlog 3

Trackpad Control

• Cursor Movement: Simply dragging on the mobile phone, moves the pointer on the desktop
• Clicks: supports Single Click, Double Click, Two-Tap Secondary Click
• Gestures: As of now only Two Finger scroll gesture is s supported

Used a blocking thread to offlaod the hardware control operations (60 times per second)

The hardest part now is supporting the ability to drag and supporting more gestures, It will requires complex math and numerous trial & error iterations.

Live Typing

• a simple textarea where user can type with his native keyboard of the phone and the text gets typed live on the on computer.

• Buttons like Backspace, Enter required different approach as they cant be clubed with normal text typing feature of Enigo.text()

several other improvements

• trachpad data sending is more optimized to send cursor cordinates efficiently
• scroll has kind of accelaration detection for smooth scrolling
• detecting the difference between drag, scroll and just basic move

NEXT

1. Implement a fully customizable control layout
2. Implement PWA
3. Improve connectivity
4. settings for the rust server

Devlog 2

Magic

well in my last devlog in mentioned i wanted to improve code quality and modularity, but guess what its still spaghetti code.

But i did some other cool things

actully too cool

1. I impelmented a simple interace in client, and create button controls for the Arrow Keys
2. The client can now talk with the server and send data
3. server recieves the data and use Enigo to simulates key presses

So basically now i can contorl the keys of my laptop with phone over local network and that under 5ms latency

the flow is very simple

Client Sends data over websocket –> Rust appliaction listening to the socket recieves it –> formats the incoming data to decide the appropirate action –> uses Enigo to simulate Key presses

Next Target

• Improve code quality and modularity
• make the communication between client and rust app based on certain standards
• make more layouts for the client
• implement trackpad

Below gif shows a demo

Devlog 1

Making a Remote Human Interface Device (HID): cmdRelay

basically a controlling a device with another device, here controlling a computer with mobile phone

Idea

Problem: On my desk i have a windos pc (with top tier monitor) and i also have a macbook, and i lie on the bed and play movies on the monitor, but then i cant control the playback, volume, and cant scroll youtube.

• some of the soltions are platform dependent.
• some of them are paid
• only one solution kde connect, was usable, but it uses an ios app on the iphone and due to app store restriction, it offers limited functionlality on ios, compares to android, also its complex to use and consumes more battery and lack modern features and UI.

how cmdRelay solves these problems

• Modern features and Better UI.
• No need to install an app, a PWA(Progressive Web App) is used in my idea, which solves the app store’s restrictions
• a very lightweight and efficient background process written in Rust, cross platform support. (an installation is required on the desktop, though its under 10 MB)
• lesser battery consumption and sleep mode feature.

Details??

• it uses an conection over LAN, to reduce latency to about 5ms
• User can control trackpad, keyboard, run big keyboard shortcuts on just a click, i will try for `gesture₹₹ support as well can’t promise anything right now.
• communicaiton will happen via a local websocket connecion.

Client on mobiles phones

• A React based PWA

Background process on mac and windows

• Rust and Tauri based background process, with minimal settings

Current Status

• I have make a basic websocket server in Rust and React js client, and establised the connection successfuly

wahts next

1. improve code quality and modularity
2. build the client side and send the json data to the computer via web sockets

Devlog

Updated Readme files, and wrote down some technical details

Devlog 5: Making it Beep

Last time, I talked about how Continuous Integration was eating my time, Windows was demanding my soul with missing DLLs, and Apple Gatekeeper was holding my app hostage for $99.
This time, I fought back. The emulator is now completely polished, actually sounds like a retro console, and distributes like a real, professional application.

Here is what went down.

It Beeps now

The CHIP-8 doesn’t have a sound card. It basically has a timer attached to a buzzer. If the soundTimer register is greater than 0, it beeps. It ticks down at exactly 60Hz.

I basically wrote some code in c++ to generate a square wave, and used raylib’s audioPlayer to support classic beep sound of chip 8.

Production (The Illusion of Polish)

Originally, I hated that a black developer console popped up behind the Emulator. It screamed unfinished project.
So, I spent hours engineering it out. On Windows, I added the -mwindows flag to the compiler to change it from a Console App to a pure GUI App. On Mac, I completely rewrote my CI pipeline to generate a standard Apple .app folder structure so it would look like a commercial application.

the result? Total disaster. Because my newly polished Windows .exe was a standalone GUI application without a digital signature, Windows Defender immediately flagged it as a virus. Meanwhile, Apple’s Gatekeeper still blocked the .app bundle because I refuse to pay Tim Cook $99 a year.

The Real Solution: I reverted it all. I am keeping the console.

Git always has one more feature (Isn’t god’s another name linus?)

This was my first time actually using the tag feature of git in practical life, i after git push v1.1.0 i was wondering the new releases are created, code editor says everything is uptpdate with remote, but why the hell my gihub does not shows new code, and guess waht branch and tags need to be updated seperately.

Checkout latest release

Devlog - Almost completed this time (almostt)

This time my goal was to convert this basic chip8 emulator in to something any user can use

coming from web development background, creating ui in C++ raylib was completely new.

No concept of ui elements here, cant just attach event listeners to button like we do in javascriot

What happens here is -

• you create a rectangle, put some text in it, this becomes your button
• then there is a constant loop running, which erases and redraws everything on the screen 60 times per second, just like a game loop.
• now to detect mouse hover, or click, we need to constantly tract the mouse position 60 times a second, and check if it overlaps the coordinates of a button

But

• i found an interesting piece of code raygui, written by the original creator of the raylib
• it basically removes the manual need of tracking the cursor position and checking them with the coordinates of the ui elements, it handles click and hovers.

custom ROM import support

• I wanted users to be able to run any .ch8 program they wanted to run.
• I simply opened up a native dialog to select a .ch8 file and run it on the emulator.
• I used AppleScript on Mac to trigger the file picker, but on Windows, I had to pull up a shell in the background

The Final Boss: CI and distribution

• This is actually a big time eater, while implementing Continuous Integration i did countless iterations solving dumb errors, adjusting platform depended configs and so on…
• but results are just as fin, now automatic releases are created for every tag.

distribution

• macos blocks applications from unknown developer (i need to pay $99/yr to get verified, Best of luck with that Tim Cook), to fix this i am thinking to create standard .app structure not sure but it might tame the apple gatekeeper.

Apple wanted my money ($99), Windows won’t be happy without taking my soul

• Like hell, every here and there windows throws some error be it missing dll or path problems or linker issue.
• the biggest hurdle was that I don’t have access to a windows machine, I had to ping my non-tech friend with new .exe file each time. (aah friends are a blessing, aren’t they, my friend? huh?)

Checkout lateset release (supports mac and windows, can’t say about linux for sure)

running on mac

1. open the executbale file inside the zip (i wont run, but it is required to open at least once)
2. Open System Settings go to privacy and security scroll down there you will se option to open anyway
3. each time you will have to run the application this way in mac (sorry i dont really have $99)

Can anyone with linux please let me know if it works (i am to lazy to install VMWare)

one more thing supported keys are (inspired from original chip-8 keys)

1 2 3 4
q w e r 
a s d f
z x c v

use thses keys to give input, nothing else will work (other keys only work in launcher interface like backspace to get back to menu form emulator)

Devlog 3.

The emulator is almost completed and and can run .ch8 binaries

not fully complete it still requires some polishing like right now the graphics is flickering a bit, probably due to a bug in graphics engine logic or I might need to change how fast cycles are being executed.

Checkout my previous devlogs to know what is chip8?

What this time?

This time I felt like nobody in the world can explain the logic to translate those opCodes. Coming from high level low level is absolute new, i thought I might find some resemblance (guess what?? I was wrong)

I still can’t say in understand it 100%, but the struggle is on

Here’s what I learnt and built.

we load the ch8 binary file -> Save instruction in the RAM -> a continuous while loop which executes each instruction code

How does the program even runs?

• Basically there is a while loop, which keeps on running unless user closes the program or program ends
• This is where we write logic to
• find which instruction code to execute next
• Control the execution speed (clock speed of cpu)
• Map keyboard events (more on this below)

How do we know what code to run next?

• There is a function executeCycle, inside this funciotn we have -
• A variable programCounter which tracks the current instruction code position and calculates next one too.
• Then we have a huge switch case flow, which basically maps all the 35 standard instruction codes form chip8 to modern computer.
• Then based upon the code, we perform tasks like painting a pixel or change some data in RAM.

To keep this short, I will not get into the details about translating the chip8 instruction.

Biggest man in the room - Graphics Engine (XOR)

• we basically have black and white screen, 64px length and 32px height (we use multiplier so the size is not exactly 64x32, as that will be two small)
• we then have 1s, 0s to represent black and white pixels
• the engine basically deleted every pixel and re draws it again, this causes a flashing effect as moder screens are faster and also show the transition.

Collision detection

• the value of a pixel is set to 1 (white) and 0 (black)
• We take XOR of the Current value of the pixel and the to be updated value
• if two pixels are white (1) the xor will be 0, and when this happens we set the Vf to 1, indicating a collision has occured.
• Vf is the 16th register which is left empty and use only to show collision.

Keypad

keypad

The original chip8 keypad was like this

Chip8                                             Modern keyboard
1   2   3   C					1    2   3   4
4  5   6   D	        ---> 	        Q  W  E   R
7  8   9   E					A   S   D   F
A  0   B   F					Z   X   C   V

What’s left

1. implement UI to control chip 8 settings like speed, screen size etc.
2. Add a list of software that user can run, as well as option to run his own

Decades old Low level concepts are still haunting.. (Chip 8 Interpreter)

• This is my first time building a proper low level project.(Strong Man Crying 😭)
• its tough i had to study new architechures, understand memory bottlenecks, i spent aout 10% time writing code and rest learing , reading Chip8 technical references, banging head to AI to explain the low level memory concepts (So Exciting its an Automatic All-Nighter puller)

And .. heres what i learnt

waht is chip 8?

chip 8 basically a a interpreted programing language of ancient times (70s). It kind of mimics a small computer (8 bit), limiting its max memory to 4KB. But the special thing is, its not just a language, it mimics a CPU architecture, making it like a different computer system which doesnt exists phycally, but uses other physical computer to run its own acrhitecture.

What is Emulator

Well think of emulator as a extra added layer between the software and low level computer systems. which does some magic to run a software which is not ment to run on that system.

Windows Application –> macOS (Won’t Run)

Windows Application –> Emulator –> macOS (Will Run)

Flow

• Lets say we have a software S, designed to run on System of type C1, but now we want to run the same software on a different system lets say C2,
• S –> C1 RUNs properly
• S –> C2 Fails as C2 cant understand the instruction that software S is trying to give to C2

Emulator in the picture

• Software S (instruction codes for C1 ) –> Emulator (Translates Instruction code for C1 to C2) –> now C2 can run (Translated instructions).

Now Chip8 Emulator

Lets say you have a program which was designed in and for Chip8, but now you dont have a chip8 system, how do you run this program?

We basically use an emulator to trick the software to thing that its running in the old Chip8 system but its acctually running on moder computers.

Howwwwww? well Chip8 has 35 standard instructions and only 4096 bytes of memory, so its faily easy to map each and every of its instruction and instead do it in the modern langauges (like i have used C++).

We store the data in modern C++ Data Structures, but we trick the target software to thing that it has dorect acces to the RAM of the computer.

Current Status

• implemnted the basic template to mimic real RAM for chip 8, which is basicaly an array of size 4096, weher first 200 are reserved for the chip8 system use, rest is opent to softwares running on it.
• I have translated some basic codes, more on this in next devlog
• A black window with raylib for now

Building an Chip 8 Emulator

This is like my first proper project in low level programing

I wanted to take on the low level for a long time, took the chance this time

I know only the basics of chip 8 interpreter , I read cowgod’s chip8 technical reference last night

this is gonna be a very learning and building like thing,

been into the high level for over 5 (Full Stack) years now, hope I can find some similarities and help from my past experiences in to the low level

is anyone of you guys (gods) know about low level or specifically chip8 , please drop a comment because I thing I am too going to need babysitting 😭

#update number 0.0000033

The old man, Terminal

• Now terminal supports file system operations, can create and delete folders and files

Textpad

• now it can open text files and edit those and save, can create new files as well

Activity Manger , task manager

• Shows the currently running processes in the os, also gives and option to forcefully terminate any process, shows CPU and RAM usage (dummy but mindful of currently running processes)

Menu bar

• menu bar now has specific and dynamic context menu options for each app
• although its not available yet for each app, but the logic is implemented, I just need to write json config for the ui to display

Personalization

• Right click on desktop and click on Personalization to change wallpaper,
• will support more customization soon

Boot

• added a cool mock up to mimic shutdown and booting process of OS,

DevLOG FOURRR

What do we got this time

well well I now have a complete basic file management system build, I can create text files, edit them, open them, save them in any specific location , delete it,

terminal now can run more commands

‘’‘ls, mkdir, rm. touch, echo, clear, etc’’’’

context menus

It was time taking and exhausting as every section of the operating system required different type of context menus. It was so much messed up. I ended up creating something central where a single config can decide and make a whole context menu,

Desktop is ALIVE

Long awaited something I wanted to do is to display what there is in the desktop directory directly onto the desktop after making the complete file system.