Ship
DEVLOG 6 - CONTROL
With the final addition of UI and controls, the simulation is now complete. While I could continue to add more detail and more realism, I think this is more than enough for what I set out to do.
If you don’t know, I’ve been working on a simulation of a decentralised robot swarm that I plan to build soon. My goal was to recreate every relevant aspect of the real life robots, so I could explore how they would behave in different scenarios. It also allowed me to plan out how the robots are going to communicate in the real world.
Changes
UI
The major addition of this update was the UI. You can now configure different aspects of the simulation to test the robots. Specifically the IR range, the noisiness of IR and UWB, the precision of the motors, the amount of IR receivers and emitters and of course the size of the swarm. As well as settings, you can now send formation commands to the swarm to dynamically change their formation. I also added a new REMOTE state that lets you select a robot and control it directly. Finally, you can toggle on and off different visual elements.
Walls
The addition of walls was the final piece to test my communication system. As I said in the last devlog, I designed it to accommodate for robots not always being able to see each other. Now, with the ability to block lines of sight, I can finally test it.
Communication
While messing with the controls I discovered a lot of oversights with the communication. One example is I was only establishing the network once when a new robot joined. This meant that even if the robots moved, they would still read from the old distance values when the formation changed, causing them to choose inefficient paths and in some cases get tangled or stuck. I fixed this by making them reestablish the network whenever the formation is changed.
Findings
To be completely honest, I didn’t discover that much that I didn’t already know. One thing that stood out was that the IR noise didn’t affect the formations as much as I thought it would. Outside of a few erratic movements, they remain fairly stable. The UWB noise had a much larger effect, as UWB is used to determine distance. The formations also hold with both sliders up to max which was reassuring.
DEVLOG 5 - PERFECTION
The simulation is now fully functional and perfect in a sense. The only thing left is adding noise for realism, and UI to make it interactive.
Changes + Additions
Drivetrain
The robots now have simulated drivetrains. They are modeled off 3 wheel omnidrives, and allow them to convert their desired velocity into wheel speeds. This leaves room for me to manipulate the accuracy/precision of the wheels, to see how well the robots can adapt
Formations
The robots can now form a circle as well as a line. For the circle formation, the robots first find a suitable cyclical path through the swarm, and assign positions in the path. Next, they find where they should be in relation to their 2 neighbours based on a regular polygon of size n. I struggled a bit with this part because the robots couldn’t consistently tell which way to go, and would sometimes bend inwards towards the center of the circle instead of out. I fixed this by making them take all visible robots into account not just the closest 2. This allowed them to find which direction the center of the swarm was in, and they choose the position furthest from that center as their target.
Optimization
The algorithm the robots used to find the most efficient path through the swarm would was causing the simulation to freeze, or even crash with larger swarms. This is because the robots were checking every possible path, the amount of which grows exponentially as the swarm’s size increases. To solve this, instead of checking every path they now build a chain from each potential starting point and choose the shortest one. They first pick a starting point, A. robot can see B and C, but C is closer so C is added to the chain. Robot C can see A and B, but A is already in the chain, so B is added making the final order ACB. This removes the delay entirely allowing it to run comfortably on ESP32, at the cost of finding slightly worse paths in very rare cases.
What’s Next
With the core functionality in place, the next Devlog will likely be the last before I ship the project. I want to add UI and controls to the simulation, to make it more interactive. This will allow me to stress test the swarm in lots of different scenarios. A feature I’m particularly looking forward to is walls. The reason designing the communication took so long is because I had to account for cases where all the robots couldn’t see each other. Adding walls will let me see if the system works in practice.
DEVLOG 4 - COMMUNICATION
1 full rewrite later, and the robots can form a line again, but better this time! When I started this project I didn’t have a clear vision of how it was going to work. At a certain point, it became very hard to add improvements to the simulation, because it was built on such a shaky foundation. The new version is cleaner and more organised, as well as being more realistic.
Changes
Signals
Instead of solely relying on IR for angle, distance and communication, it’s now split between IR, UWB and WiFi. IR and UWB now only transmit tiny signals for detecting each other. All the heavy communication and networking is done over WiFi.
Communication
I overhauled the flow of communication between the robots. When each robot is turned on, it sends out a message attempting to establish a network. This message contains what other robots it can see, and who goes next. The robots take turns, keeping track of who has already spoken and building the network. By the end, they all have a shared map of who can see who, which they use for formations, and the last robot to speak is designated the temporary leader. I had to make the robots take turns to avoid the signals interfering with each other.
Next I added a controller class to tell the robots when to change formation. When the robots hear this signal, the leader uses the network to find the most optimal line from the current positions. It then broadcasts this signal to the rest of the swarm, and they enter motion.
What’s Next
Right now, the robots never fully settle in a line, they continue to oscillate for a long time. The position estimates (magenta dots) are also quite far off from reality. Right now, the simulation is still ‘perfect’ in the sense that there is no noise or interference. When I start adding noise, these problems are only going to get worse. To combat this, in the next update I plan on implementing more techniques to make the estimates accurate. I also want to add more formations, to test how fast they can alternate, as my goal with this whole project is speed and precision.
Video: Big circles are robots, red dots are IR LEDs, blue dots are IR receivers. Thin green lines are lines of sight, thick lines represent the intended path. Magenta dots are where the highlighted robot believes other robots are located
DEVLOG 3 - GRAPHS
After a lot of thinking, I decided to base the new line algorithm on graphs.
High level process:
- Robots negotiate shape of the physical network (who can see who).
- Robots search for a Hamiltonian path through the network. If there is none, they use the path that connects the most robots.
- Robots at the end of the path move inwards, internal robots move to the midpoint between their two neighbours. Robots outside the path move inwards until they are connected, and the path is reevaluated to include them.
I like this method, because it avoids gossiping position information which is susceptible to noise. This also means that once the path is decided, the robots only need to rely on what they can see with their IR receivers.
The Problem
While I was implementing this solution, I realised how much data I was transmitting. Each packet contained a full JSON representation of the network, alongside IDs, timestamps, etc. Although this could be compressed in the final version, it’s still too much data if I want to reliable real time communication. This means instead of purely IR, I also need to make use of the ESP32’s builtin radio. Despite it seeming more complex at first, it actually makes everything a lot simpler.
With the integration of radio, the robots can now hear as well as see. Instead of passing information between robots, they can simply broadcast it globally, making negotiation much faster.
I’ll go into more detail about the specific implementation in the next devlog, but that’s the update for now. No visual changes right now because the code doesn’t run, but hopefully by tomorrow they should be able to form lines
DEVLOG 2 - SENSORS
Ok this is a lot harder than I thought it would be. Today I added IR transmitters and receivers to my robot swarm simulation, to bring it closer to reality. Now the robots only receive signals they can actually see, and compare the signal strengths across the receivers to get the direction of the source.
This is great, but now I have a big problem. With line-of-sight added, it’s possible for a robot not to be able to see the whole swarm. This means if they try to form a line, they can’t agree on how to do it.
Negotiation
To solve this, the robots need to distribute information throughout the swarm and agree on a formation, in a process called negotiation. There are a few ways to do this with varying levels of complexity, and I’m still doing research, so I won’t go into detail in this devlog. Whatever method I choose, I need to make sure it will work in real life with imperfect signals, and it can adapt to the swarm losing members.
Notes
I haven’t changed too much since the last devlog, but I felt like I should post this as context so the next one isn’t super long. AI Declaration: I used AI to help with implementing the emitters/receivers (occlusion, field of view, some helper functions etc.)
RED = Emitter
BLUE = Receiver
DEVLOG 1 - LINE
My goal is to make a fully decentralized robot swarm, fast and precise enough to form complex patterns, which to my knowledge has never been done before. Before I make the physical robots however, I want to simulate them.
How it works
Communication
All the robots communicate via an external Network class that passes messages between the robots. Because I’m trying to simulate real IR emitters and receivers, the only information the network gives is the distance and direction of the signal. Every tick, the robots emit a heartbeat-like signal to their peers, and use the signals they receive to construct an internal map of the swarm.
Formation
Once they know where their neighbours are they can start moving in formation. Initially, I had them organise in a line from 0-4. This worked but it was very slow, because they would go to spots that were further away than necessary. After spending some time working it out on a whiteboard (and consulting chatgpt), I decided on a new method.
Using their internal map, they find the 2 robots with the largest distance between them and this becomes the line they use. They then compare every robot’s distance from the start of the line to determine the order, and move to their assigned point on the line. With this new algorithm, they can arrange themselves a lot faster with less collisions
Notes
Right now my goal is to make the simulation fully functional with as much detail as possible, down to the individual motors powering the wheels. Once this is complete, the simulation will be theoretically perfect. After that, the next step is implementing noise to make it a true simulation of real life. This will also be very helpful when I eventually build the real robots.
Until then, the next thing I want to add is real sensors. Right now, the robots magically know which direction the signal came from, but real ones will have a ring of IR receivers, and compare the signal strengths to determine the angle of origin.
Ship
FINISHED
Its finally finished! If you don’t know, I’ve been making an app to track information about how I use my computer, and a companion website to display the data. Right now, it logs every click, as well as the positions and dimensions of apps every 5 minutes. The dashboard website then displays that data in a variety of interesting ways.
The collector is now (in theory) compatible with the majority of Linux desktops (sorry Windows users, maybe next year). I also added a small config file. Right now this only stores the window-snapshot interval, which you can change through the UI. I originally planned on adding a list of apps to ignore, but there were too many problems. Primarily, I wasn’t sure how to reliably handle the focused app being blocked without making it null, which is very messy, and the whole thing seemed too complicated for a feature I’m don’t really plan on using. I also considered making it a frontend filter, but chose not to for a similar reason.
Once again, you can try it yourself here with my data. If you want to collect your own data, you can download the collector here (note: you need to give your user input permissions for this to detect clicks, more details in the README).
I’ve only tested this on my machine (KDE Wayland) so if you have a different setup, tell me how it goes!
Afterword
I had a lot of fun making this. This was my deepest dive into web development yet and I learned so much. This is also the first time I’ve shipped a project for public use, which was a has been a valuable experience. I could definitely do more with this (more config, more charts, more data), and I do plan on revisiting it in the future, but I think this is a good place to leave it for now. Thanks for reading, see you next project :)
Day ?? - New Features!
After 15 hours (got a bit carried away) the next feature is (mostly) finished! You can now monitor and display window data. Every 5 minutes, the collector logs what apps are open and their positions on screen, which you can then visualize on the dashboard.
I thought it would be interesting to combine the click data and the window data in some way, but I decided against it. The reason being that I technically don’t know what app the user clicked since it could have changed in the 5 minutes.
To collect the data I had to run a separate KWin script to get around the restrictions, same as with the clicks. This script gave me the windows as JSON, which I then stored in 2 SQLite tables. One containing general info (one entry per snapshot), and the other containing info about specific windows.
The second major change is the collector’s new UI. It was a relatively small addition, but it makes the whole experience a lot smoother. It also uses the same UI library I was already using to get the monitor configuration which was nice.
Finally, the dashboard now stores the database in cache so everything is preserved when you reload. This change was mostly for me, because having to re-upload the file every time I changed the code got really annoying.
What’s Next
There’s a few finishing touches I need to add before shipping, Specifically:
- Compatibility (KDE only right now)
- Config options (Snapshot frequency, apps to ignore, etc)
- Possibly more window features
yap
I definitely could have done all this way faster but I spent a lot of time fiddling, trying to fully understand everything. Now between the KWin scripts and the dashboard I have a much deeper understanding of Javascript than before which will help in the future. Hopefully I can ship this today or tomorrow so I can start working on my next project
Days 5 + 6?
Changes
I added filters for time and button type to make it more interactive. I also added new charts, specifically pie charts for button and monitor usage, and bar charts for hourly and daily activity. I used chart.js to make the charts, because it makes the programming way simpler while still giving me full control. The final change was reworking the heatmap again. instead of each click being an individual element, they’re now drawn on a canvas which boosts performance significantly.
What’s Next
Now that the dashboard is mostly finished. I want to add some quality of life things (namely caching the database so I don’t have to reupload it every time) before adding the next feature. Speaking of, the next feature is going to track what applications are open at what times, which I think will make for some really interesting data.
Day 4 - Cleanup
Changes
Cleaned up the UI a lot. I like the new heatmap visualization a lot, but it’s a bit slow so I’ll continue to experiment with it.
I optimized the backend a lot, it used to block a full CPU core but now it sleeps properly when idling.
I also expanded compatibility to other Linux environments. This was actually way simpler than I expected, I just happened to start with KDE which is really complex for some reason.
Notes
The dashboard is now live on my website, which you can test with this sample data.
My aim right now is to get the project ship-ready, and then add features one by one.
Day 3 - Dashboard
Today I started making the dashboard, to present the data. I decided to make it a website and have the user upload their data. This makes demoing a lot easier, as well as being good for compatibility. Right now it’s a very bare bones prototype with only a heatmap, but all the set up for reading the data should be finished, which means I can focus on polishing the UI and adding features.
I also fixed some bugs in the backend. Previously the code would fail if your mouse disconnected, but now it can handle disconnects and reconnects, as well as detect multiple mice at the same time.
Day 2
Continuing work on my data collection project. Yesterday finished the core of the first feature — a click heatmap. Today was mostly polish and fixing the structure to make shipping easier when the time comes.
Changes
Organisation
Instead of having everything in one big ugly python file, I created a module and separated all each part into it’s own file. This will make things much easier as I continue to add new features.
As well as organising the python code, the project uses a JavaScript script to get the position of the mouse, and a bash script to run it. Neither of these can be packaged, so I updated the code to generate the JS file in the user’s share directory and run it directly from python.
Compatibility
The original code was highly specific to my setup, and couldn’t be shipped. The fix came in two parts.
- I previously had my exact model of mouse hard-coded, so I added the ability to detect what mouse the user has
- The code only worked for my specific Window Manager. This is still the case, but I added the structure to detect the WM and adjust for it. I plan on adding support for other’s in the coming days
Storage
All data is now stored persistently in an SQLite database. In the future this will be read and displayed by the dashboard program.
What’s Next
- Create the dashboard to start displaying data
- Expand compatibility to other Linux systems (might take a while)
