Devlog 9 - Near completion Optimizations

Linux

• i had almost forgotten that i got to handle linux as well, mac and windows does not mean corss platform
• although linux is completely optional and i will try as i have promised the cross platform like 100 times
• i will try, i cant promise, as i dont have access to linux so i will code, do AI analysis to check if its gonna work on linux
• and that is all linux gonna get from me.

I will test the application on windows soon, (i have mac, i am gonna borrow from a friend)

Bug Fixes

• Adding new pre built layouts won’t nullify user editing on the layouts
• changed the server port from 3000 to more random port, as 3000 is commonly busy
• Special Function can now be executed on the Linux
• Moved the Layout Builder/Settings to the main application from web client to make it feel like a native app instead of a web UI
• Handled some of the observed edge cases in key mappings for cross platform support
• assets were not loading in the producition build, moved the assets to the public directory for direct access after produciton build

Devlog 8 - Will it Survive The Winter?

A lot of time, this time went on fixing mistakes, which I did by making this application just for macOS, but I was building this to support different platforms, I can’t survives other platforms

1. Just had one JSON file

• for the complete configuration of the layouts.
• I realised that certain layouts which I am giving pre built were going to have different keys bindings for different platforms
• so I just separated my layouts into different files for Mac, Linux and Windows.

2. While talking to Claude,

• I got to know that the data of the applications which are edited are supposed to be saved in a different location other than the installation location.
• It is different for Mac and Windows, I was actually saving my data where my application is installed, so I fixed this issue and stored layout configs data at their appropriate location in the different system

3. So the special function keys, if you reading my devlogs, you are aware about this

• special function, such as increasing or decreasing volume and brightness, skipping and pausing and playing media
• these functions cannot be triggered by virtual keyboard. They are bounded to pressing keys in physical keyboard.
• to trigger these special function, I used Apple script on macOS, but Apple script are still failing on some cases
• so I used a swift language code , I did not understand how I could write swift inside rust but, stack overflow works. I got the code from stack overflow.

I also added some pre-layouts for various tasks and helping in application such as VS code, Photoshop, fusion 360 and while giving presentations.

Devlog 8 - Polishing is really Boring

• The project is near completion on terms of features
• Now i am just Polishing the app doing things like improving UX, Text Content, adding minor features.

Despite that here are two major things i did

1. Special Function Support

• remember my earlier devlog where i mentioned that, increasing and decreasing the brightness or vol is not as simple as just some keypress on keyboard
• These comes under special fucniton and although you can trigger then via physical keyboard, but can’t via the virtual ones.
• to do that you need to run some special code, on macos via Applescript and on window vis winShell, i am still figuring it out for linux.

2. Appearance settings

• user can now customize button’s color, bg, add icon and images (custom images are under dev)
• i did all this with backward compatibility in mind (Big Brains)

CI/CD Pipeline

• Used Github Action to build the application for major platforms (macos, windows, linux)
• yeah all that, thousand build erros , waiting 10 min for each github action just to see a red X, i will not rant i am past that now

Future

i will be straight forward

• add custom image support
• verify and improve the cross platform functionality
• add some pre built layouts
• maybe add a guide/tour for the layout builder

Devlog 7 - There is always a Solution

Focus V/S Feature Dilemma

• I thought it would be a cool feature addition to display the current resource consumption (CPU/RAM) and some other system details of the computer right on the mobile app.
• but my concern was that these features are basically "extras" and I really didn’t want to slow down my core functionality.
• I was about to drop the idea entirely, but at the last minute I decided to ask Gemini about it and guess what? There is always a solution I mean, what did I expect? Its engineering after all

Smart Subscription Architecture - the solution

• well, my main concern was the wastage of resources and WebSocket bandwidth, which might bottleneck my main selling point: ultra low latency under 5ms control command communication
• But this architecture perfectly solves that.
  • so basically, the server will only blast data at me if I am explicitly subscribed to it. Otherwise, it sleeps.
  • and since this project uses WebSockets on a local network, there is generally only one user. So, if I unsubscribe from the service (like closing the telemetry component), that particular data broadcast will completely pause, saving precious resources on both the mobile client and the computer server
• forgot to mention : the server updates this system data every second if user is subscribed, else sleepyy

The Scalability FOMO

I am well aware my app won’t probably be used by more than 10 people for 10 minutes, but my Senior Software Engineer came shouting -

“Your current method to render different views, that too without using react-router, it will become scalability issue as application grow”

• in my initial HL system design i was going to use a single page and conditional rendering for different layouts on same page to avoide a package i decided not to use the react-router for my client web page

• but now i have like 5 pages and as i am building i keep getting more and more ideas…

• I fixed this issue by

  • creating a standard state for different viewModes and changing props/types to support it (nothing just some painful refactoring)
  • implementing a component mapping (view registry) for all the views. Now I can just plug in any number of views without writing spaghetti if/else statements.

The important thing is as i am not using the react-router, i will to keep an eye on memory and component lifecycle to free up the space in memory from components which are not in view.

I should have used react-router from beginning 😭

I am thinking to add a button to force kill a process from the phone, but again i am standing at a point where these features are kind of feelign like extras and note my core application

• attached image of the QR code that i use to pair the mobile with the computer, will add some security in future

Devlog 6

• My this session was so exciting that i only stopped because I reached the 10 hour limit of the hackclub.
• Well, I think throughout this project, I have been doing things which I don’t understand, things which were boring
• but right now this is the point when everything clicked, finally got the hang of RUST

Its gonna be long Devlog read at your own risk(10 hours dump. Note: the maximum limit is 4000 characters so I cannot write everything here just this much)

1. Crazy things this time

So if you may know that in this project, one of the things is to simulate the actual keyboard through the application

Here is a interesting thing i found about special fucniton keys

• those special functionalities keys work strangly, on MacBook, we press F1 to decrease brightness, so i coded my applicaiton to to press the F1 but nothing happened
• in reality, most OS do not registers the F1 and F2 key for changing the brightness but it actually uses some code tied to hardware to trigger the special fucitionality

How did i solve this?

• well, I read about how actually a Operating systems register the keys, and trigger those special functions.
  And this was the reason pressing the shortcuts were not triggering those special functionalities into the computer. As the OS was not looking for me to press F1. it looks for a special code which is tied with the keyboard itself, which was not in virtual keyboard,
• solution is simple. You have to find out what code is used by a particular operating system to increase or decrease the brightness, and then you will have to trigger that code when the key is pressed.

2. Macros

I implemented macros, you can basically press a sequence of keys with control delay. Just on a single click. It is spamming a key running some keyboard shortcuts on a single click.

• I created different thread to run this macro so that it does not freeze my main application
• this is the point which like frustrated me for very long as it is been like three years when I learned concurrency and threading in Python and I did it again in rust this time. I used MPSC (Multi-Producer, Single-Consumer) here. I don’t fully understand it, though, but yeah, I know why I used it and how I used it

3. App Lanucher

So I just liked a button, which upon clicking will open up any application on your computer

• but the main thing was not actually opening the application, it was which application to open, Open an application by running a command on operating system, you have to know their exact official names, and a normal user does not know that
• so, I scanned the whole application folder onto the operating system and found out the official names of each application. Then I showed the list in a dropdown to user, from where user can select the names of applications.
  -well, it took me a lot of time because this was my third approach which worked my initial implementaion were like I would force the user to open up that application and then my application can register it, but obviously, the above approach is better than this, so I dropped it.

Jupiter (idk, maybe because its big, and who doesnt like sloar system?)

4. Context Aware Layouts

This is it. The big one. This was the most important, most time-consuming, and absolute flagship feature of the entire project.

• so, what is context aware layouts? So in the application, one of the parts is to create layouts, which helps with your work aespecially in the productivity/professional applications
• so lets say you are coding or working, constantly shuffling between multiple desktop apps, with your phone sitting on your desk acting as your custom macro pad. This feature actively extracts the currently focused application on your computer and instantly switches the layout on your phone to match it.

Devlog 5

Loking back and accepting

• this time I saw what happens when you build a project without any proper system design,
• things were so clunky in my application that I was not in mood of creating another groundbreaking feature
• i just spent time improving existing features and make them actually usable

What went down

So basically, I had done a dumb thing very dumb thing. I did not realize that the buttons on the actual keyboard have different states, you know when they are being held down OR they’re being pressed OR they are being released, this info matters and I completely missed it.

1. so now, basically, my key presses have a state

   1. Down (pressing down)
   2. Up (releasing the key)
   3. Pressed (holding the key)

2. So the SHIFT button which I had implemented earlier was only clickable, user couldn’t hold it, but the shift button is meant to be held, so I fixed this and other similar buttons like alt, control and command

3. Improved the portrait and landscape switching between the views in layout builder. This is just normal UI improvement Nothing special here

4. Other than these two improvements above, there has been numerous other small little improvements which took me a lot of time, so I don’t really have much to write on this devlog, changes are like

   1. I improved the whole conditional rendering thing, it was messy as I decided not to use react router for this project as there weren’t any actual pages or URLs, just bunch of different control layouts.
   2. Improve my state management to handle menu overlay properly
   3. Overall code quality improvement, creating utility functions and type safety.

and a lot of my time is invested in learning Rust as this is like my second project in rust (i leanred rust like a year ago and now i remember nothing, thanks to my good c++, i have held my ground so far)

Bottom line → Better UI and UX, button can now be held and tracked weather they are being pressed or released

Devlog 4

Control Layout

first i would like to define what i call a control layout, as i reliased i never relly defined it properly

Control Layout: It is a layout of buttons and other control elements like

For Example

• control layout for playback control will have button/controls like volume up/down, play/pause, fast forwarding and controling playback
• Maybe some specifi button to help while editing video
• Or controls for some particular Applications

So basically, these controls can trigger and functionality or a shortcut or a macro onto the computer that the client is connected with

Single click on the phone screen will be able to run a complex keyboard shortcut

User can map shortcuts based on particular applications, and he does not need to remember keyboard shortcut.

They don’t even need to click anywhere on their screen. The phone will act as a secondary keyboard or a small macropad.

Customizable Control Layout

The application contains some prebuilt control layouts

But different user user different application and have different workflow, so i created a featrure to let users

• create his own custom layouts where he can place different buttons anywhere on the screen and assign what work each button will be doing.

As of now the current active applicaiton detection is not present but i will add this feature soon.

Details

• basically, I have an interface which tries to mimic a mobile screen, and I have divided that screen into grids with CSS

• I can move the elements around and resize them.

• I track the position, height and width of those elements and store them in a JSON config file upon saving these layouts. This JSON config is sent over local web socket to the Rust server.

• Along with just the information about the UI, there is also working information, like name of button, its functionality etc

• I have a component which is quite opposite of a builder which is a renderer.

• This renderer takes up JSON config and renderes the different components and maps the functionality, in smple word it takes json and create UI.

Devlog 3

Trackpad Control

• Cursor Movement: Simply dragging on the mobile phone, moves the pointer on the desktop
• Clicks: supports Single Click, Double Click, Two-Tap Secondary Click
• Gestures: As of now only Two Finger scroll gesture is s supported

Used a blocking thread to offlaod the hardware control operations (60 times per second)

The hardest part now is supporting the ability to drag and supporting more gestures, It will requires complex math and numerous trial & error iterations.

Live Typing

• a simple textarea where user can type with his native keyboard of the phone and the text gets typed live on the on computer.

• Buttons like Backspace, Enter required different approach as they cant be clubed with normal text typing feature of Enigo.text()

several other improvements

• trachpad data sending is more optimized to send cursor cordinates efficiently
• scroll has kind of accelaration detection for smooth scrolling
• detecting the difference between drag, scroll and just basic move

NEXT

1. Implement a fully customizable control layout
2. Implement PWA
3. Improve connectivity
4. settings for the rust server

Devlog 2

Magic

well in my last devlog in mentioned i wanted to improve code quality and modularity, but guess what its still spaghetti code.

But i did some other cool things

actully too cool

1. I impelmented a simple interace in client, and create button controls for the Arrow Keys
2. The client can now talk with the server and send data
3. server recieves the data and use Enigo to simulates key presses

So basically now i can contorl the keys of my laptop with phone over local network and that under 5ms latency

the flow is very simple

Client Sends data over websocket –> Rust appliaction listening to the socket recieves it –> formats the incoming data to decide the appropirate action –> uses Enigo to simulate Key presses

Next Target

• Improve code quality and modularity
• make the communication between client and rust app based on certain standards
• make more layouts for the client
• implement trackpad

Below gif shows a demo

Devlog 1

Making a Remote Human Interface Device (HID): cmdRelay

basically a controlling a device with another device, here controlling a computer with mobile phone

Idea

Problem: On my desk i have a windos pc (with top tier monitor) and i also have a macbook, and i lie on the bed and play movies on the monitor, but then i cant control the playback, volume, and cant scroll youtube.

• some of the soltions are platform dependent.
• some of them are paid
• only one solution kde connect, was usable, but it uses an ios app on the iphone and due to app store restriction, it offers limited functionlality on ios, compares to android, also its complex to use and consumes more battery and lack modern features and UI.

how cmdRelay solves these problems

• Modern features and Better UI.
• No need to install an app, a PWA(Progressive Web App) is used in my idea, which solves the app store’s restrictions
• a very lightweight and efficient background process written in Rust, cross platform support. (an installation is required on the desktop, though its under 10 MB)
• lesser battery consumption and sleep mode feature.

Details??

• it uses an conection over LAN, to reduce latency to about 5ms
• User can control trackpad, keyboard, run big keyboard shortcuts on just a click, i will try for `gesture₹₹ support as well can’t promise anything right now.
• communicaiton will happen via a local websocket connecion.

Client on mobiles phones

• A React based PWA

Background process on mac and windows

• Rust and Tauri based background process, with minimal settings

Current Status

• I have make a basic websocket server in Rust and React js client, and establised the connection successfuly

wahts next

1. improve code quality and modularity
2. build the client side and send the json data to the computer via web sockets