O.T.I.S
Hardware- 12 Devlogs
- 46 Total hours
This is my AI desk assistant. He will be able to answer questions and move around. He is powered by an ESP32-S3 sense and my PC.
DEVLOG #12
What I did
I know this is a lot of time for a single devlog, so I will try to post them more often moving forward. First, I worked on allowing the AI to control the movement of the treads. My first Idea was to create a JSON schema with all 4 directions that the AI could enter values into. The values would dictate the time (in milliseconds) for which the action should be executed. Initially, I decided the 4 directions should be forward, backward, left, and right. But I knew that this would not work, since the tank treads can really only move forward and backward, as well as turn. So I then decided to replace the directions with “Right tread forward”, “Left tread forward”, “Right tread backward”, and “Left tread backward”. I decided on this because I felt that it would allow the AI to reason through my commands, which would allow for much more abstract movement. But the model was not smart enough to map out its movements, and I didn’t really have any other ideas for movement except for forward, backward, and spinning. But that gave me the idea to just use those elements in the JSON scheme. So I set the directions to “forward”, “backward”, “spin left”, and “spin right”. This way, the AI would be able to move exactly how I wanted it to move, without overcomplicating the whole thing, giving me more accurate results. After this, I began working on the APP for OTIS’s controller. I have not decided on a layout, but I do know I want buttons for the 4 directions, a chat box, and a button to send voice commands and photos.
Challenges
422 Unprocessable Content
So while trying to find a way to get the AI to respond as JSON, I remembered Ollama allows you to send format requests as part of the API POST request to the AI. So after reading the docs, I was able to whip up a simple JSON format using Pydantic and BaseModel with the 4 directions. But when I tried to send the request, I kept getting the 422 error response code. So then I tried a bunch of different stuff, like removing the requirements and removing the nested JSON with the format data. But none of that worked. That’s when I did some research on the docs, and I found that it was the Pydantic JSON class that was causing all the problems. You see, Pydantic is used to check whether incoming data is formatted properly. So when I used it to create a JSON schema, it would constantly look for that data from the Arduino, which it wouldn’t find. This caused the 422 error. So I removed the class and created a JSON schema as a regular variable, and this solved the issue
What is next
The next thing on my agenda is going to be to create the OTIS app that will allow me to manually control OTIS like a mini rover. I am currently using Tkinter to create the GUI, and will likely use an ESP web server to allow the GUI to interact with the Arduino. Thank you.
DEVLOG #10
What I did
In this devlog, I really just updated some of the things I had already implemented. First, I remodeled the wheels from gears to regular smooth wheels. This is because I had the idea of using those silicone wristbands as the tank treads instead of timing belts. This would not only be more accessible, but it would also make OTIS much more customizable and, in my opinion, look much better. This is because the silicone bands are thicker than the timing belts, which means the tank treads are larger. I also installed another AI model, qwen3-VL:4b, because I believed it would have better image analysis. I tested out all the models to find the best one, and I think I will use qwen3-vl:4b for this project, as it was able to recognize pictures and follow instructions more accurately than Llava. Lastly, while testing the models, I realized the prompting was a bit tedious when using the camera, since I would have to take a photo each time I prompted the AI. So I revised the code so that it would ask if you wanted to take a photo. If you say “photo”, it will ask you to position the camera and send the prompt. If you don’t want to take a photo, you just put the prompt in the first question, and it will default to no-photo mode for that prompt.
Challenges
Modeling the wheels
This wasn’t really one large problem, more so multiple smaller ones. You see, I had already completed the 3d model with the gears instead of the wheel. This meant that changing the wheels would alter the whole model. For this reason, I was contemplating whether this would be the right choice, as I would also have to fix the animation. But I decided to do it anyway. The main problem was that the wheels have a much larger diameter than the gears. So if I were to just replace them, it would look extremely disproportionate. So, I had to remodel the servo slot and the front axles by placing them closer together and moving them up. Once I completed this, I had to redo the whole animation because the old one did not fit the new wheels well. But after all this, I was able to reupload the new files to the GitHub repo and begin fixing the code.
What is next
The next thing I am going to do is test out the tool. I wrote up a simple tool in Python that would allow the AI to send data that controls the servos, but I have not tested it yet, which is what I will be doing next. I will also likely wire up the servos and continue testing other models to try to find a good setup. Thank you.
DEVLOG #10
What I did
So for this devlog, I mainly spent all of my time working on the camera. I looked up an example script from the Seeed Studio docs to learn about the main system. I then spent quite some time working on a way to convert the picture into a format that I could place into the JSON. I also shifted the old code a bit to better fit the way I wanted to use the camera. In the end, I was able to get the camera working and send a photo to the AI to reference in its response.
Challenges
Getting the conversion of the photo
So I went into this knowing literally nothing about how cameras like this one worked. So I started with the docs and was able to write the code that took the photos. Although this was a bit tedious, the main challenge came when I tried to convert the photos to a format that was compatible with the AI. You see, when you take the photo, it is saved as a frame buffer JPEG. But I needed to convert it to Base64, since that is what Ollama uses when integrating images in the POST request. So I did some digging and came across mbedtls_base64_encode() from the mbedtls/base64.h library. This was really annoying to work with because there are a lot of required parameters, and I didn’t know what any of them meant, let alone how to use my code with them. After an hour of searching, I asked Gemini what each of the parameters meant and what to include in them, and I was able to get the code completed. But when I tried to run it, my ESP kept rebooting. After some more research, I learned that the cause was insufficient memory to store the Base64 string. I was able to fix this by adding if statements as well as memset() to manage the memory space better. After this, I added some code so that it would work regardless of whether the camera was connected.
What is next
The thing I am going to work on next is to create a way for the AI to use the servos. I already started working on a tool to handle that, but I have not tested it yet. I am also thinking about remodeling the wheels because I noticed that rubber wrist bands might be a better alternative to the timing belts. Thanks.
DEVLOG #9
What I did
For this block, I fixed minor details to make the prompting more seamless. First, I fixed the deserialization so that only the response would be printed. I also used Serial.readString() so the user can send multiple prompts without having to reupload the code with a new prompt
Challenges
Getting the Deserialization and print to work
So this was a really annoying problem that I should not have had this much trouble with. When you receive the response from the AI, it comes back as JSON, with the response as well as other information like the date created, context, etc. I only wanted to print the response, so I tried using deserializeJson() from the ArduinoJson.h library. But for some reason, I kept getting null when trying to print the response element. I tried messing around with the data types, switching between string and const char*, but those didn’t work. I also tried basing my code off of an example from the ArduinoJson website, but it still would not work. Then I created an if statement that checked if there was an element in the JSON labeled response. This returned false, which meant that the data was not deserializing properly. At that point, I felt pretty lost. But then I thought of deserializing the JSON in the Python script and then sending just the response. So I looked into how to deserialize JSON strings through Python, and I came across json.loads() from the json library. There are two main versions of this method: json.loads() and json.load(). .load deserializes a file, while .loads deserializes a string. Because the response from the AI came back as a string, I was able to use the latter to separate the response from the rest of the data, which I was then able to send back and print successfully.
What is next
Now that I pretty much have the prompting working, my next course of action is likely going to be to figure out how to use the camera and send the feed to the AI. This will definitely be challenging, as I have no idea how I want this part of the project to work. Thank you.
DEVLOG #8
##What I did
These 7 hours, I basically spent trying to get the HTTP POST request to work. I had a lot of trouble when trying to get it to work, but I was able to find a way to get the prompt to the AI and the response back to the ESP.
##Challenges
####Getting the POST to work
Initially, I had decided to scrap the MCP idea and to just send an HTTP request directly to Ollama. This plan should have worked, but I could not get it to function. For some reason, I kept receiving a response code of -1 when I tried to send the request. This code means the connection was refused, and this could happen for a multitude of reasons. After some simple troubleshooting, I deduced that the problem was likely related to my firewall. You see, Ollama runs on port 11434, which is protected by the firewall. It also only listens to 127.0.0.1, which meant that I could not send data from the ESP32. To fix the IP problem, I tried setting an environment variable to allow it to listen on 0.0.0.0, but this didn’t work. Then I tried exposing port 11434 on my firewall to the IP of my ESP32, but that was also unsuccessful. I even tried connecting the ESP and my computer to my phone’s hotspot, but that also yielded -1. After consulting Google’s Gemini for any other solutions, I decided that I should just create a Python file that would handle all the requests. So I created a file, installed FastAPI with Uvicorn, and used those libraries along with requests to send an API request to Ollama’s REST API, which I could then return to the ESP. The ESP sent the prompt to the Python script via HTTPClient and ArduinoJSON. After creating all of this, I was finally able to get a response from the AI in serialized JSON, and I am currently trying to figure out why I can’t deserialize it and print it.
##What is next
As I mentioned above, I will continue trying to find a way to print only the response and to send prompts seamlessly. This part of the project was definitely very frustrating, but I am so happy I was able to make it possible. Thanks.
DEVLOG #7
What I did
So this whole time, I basically spent trying to get the web server of the ESP32 to work, which I had a hard time understanding. I also tried using the Python script to send data to the web server, but that doesn’t seem to work either. So I began setting up the AI model with Open Web UI and Docker. I decided that I will probably scrap the whole MCP plan since i’m pretty sure I can just send and receive data as HTTP requests directly between Open Web UI and the ESP32 as JSON.
Challenges
Getting the data to send
The main problem I have with all of this is that I don’t know what I’m doing a lot of the time. Even though I have worked with Arduino, I have never worked with an ESP32, and I don’t know how I can utilize the wifi to send and receive data. I tried understanding an example script for a basic web server that is built into the IDE, but I couldn’t wrap my head around the part where you create the server. I was able to get the ESP32 to connect to my WiFi, though, which I guess is progress. I also tried sending a POST request from the Python script, but I don’t think that is going to work since I have no idea how to get the ESP32 to receive the data. It’s even harder to find answers to these problems since I don’t even know what to look for. I decided that I should scrap the MCP plan, since I could just read JSON data that the ESP32 can receive through a GET request. In conclusion, I just ran in circles trying different things that all yielded the same results.
Getting the AI to work
For some reason, the model that I decided to use can only complete tasks, which are defined by a set of tools. I am able to use it in the CLI just fine, but this problem occurs when I use the model in Open Web UI. I’m using Qwen3-vl:8b because it can supposedly analyze images, though I will likely try another model if this persists.
What is next
The next thing is likely going to be to try other AI models to see if I can get one to work. I will also do more research on an alternative to creating a web server. This project has been very tedious, so wish me luck.
DEVLOG #6
What I did
So today, I decided that I should finish up my design before I apply to get funded. But there were a few things that I had yet to do. First, I created the top panel for OTIS. I don’t exactly know why I put it off until now, but I made a simple design that will just sit on top and be held in place by little stoppers. I also did a bit of research beforehand on what battery i should purchase, but a lot of them are too big. So I also increased the size of OTIS from 77mmx77mmx77mm to 80x80x78.5 (78.5 because the lid is 1.5 mm thick). Lastly, I went and updated the assembly video and drawing, and uploaded all the changed parts to the GitHub repo.
Challenges
Resizing the robot
When I initially changed the size of the robot, I thought everything was ok. I used joints to attach everything, so all the parts just kinda “shifted” into the right spot. But the challenges came when I had to edit the parts of the robot that were now disproportionate. One example of this was the LCD. The screen is supposed to sit right in the middle of the robot, but it was not in the middle of the LCD board, which means all the dimensions got messed up when I resized the enclosure. To make matters worse, the project tool that I used to make the holes kept bugging out and offsetting the projection to where it was BEFORE I shifted the design. Eventually I got it to work by creating separate sketches from the projections, which is a setting on the pop-up. This way, the projections would stay exactly where I needed them instead of trying to auto-correct to some past position. I also fixed some other parts; the spacer near the servo also got messed up, but I was able to fix those problems much more easily.
What Is Next
The microcontroller and wires will arrive today, so I will begin working with them and the AI to figure out how to synchronize them all. I also applied for funding, so I will wait and see where that goes. Thank you.
DEVLOG 5
What I did
I began by updating the 3d model by updating the position of the screws. I did not like the visibility of the screws in the earlier design, which is why I decided to put them in from the back. I had placed some spacers between the front of the robot and the screw holes on the LCD, which meant I could screw it in from the back since the spacers created a place for the screws to screw in. This meant two things. The screws were now invisible from the exterior, and I didn’t need to use nuts. Then I created the assembly. I created my own animation for the assembly by manually transforming each component. I also created a drawing of the exploded view with a parts list. Lastly, I updated the schematic because someone suggested adding net labels to make it look cleaner. I also did quite a bit of research trying to wrap my head around MCPs and other options. But I landed on using Ollama to install a local AI model, Open Web UI to use the model with MCPs, and Python that will handle the MSPs as well as the communication between the AI and the microcontroller.
Challenges
Creating the exploded view.
Creating the animation was hard enough, especially since the animation system is really annoying to work with. But I decided that an animation of the disassembly guide would not suffice, so I wanted to make a drawing from the animation as an exploded view. But the problem came up when I created the drawing, and not all of the parts were visible. So I solved this by creating a separate animation where all the parts are moved far enough that they are all visible from the same angle, but still in their relative locations.
Figuring out the architecture workflow
This has been the worst part of this project, especially because I have practically no knowledge whatsoever regarding this kind of stuff. The main problems I was having were trying to find a way to allow the AI to interact with the microcontroller. I had no Idea where to start, and was losing faith in this project. I also considered scrapping the whole movement idea because I didn’t know how I would integrate it effectively. But I stumbled on MCPs, which I think will work. They basically work by creating tools that an AI can use to interact with other software, and they are very versatile. I want to make an MCP server (a custom set of tools) that controls the code that the microcontroller runs on. Only problems are that I don’t know how to use Python or MCPs, and therefore have no idea what my options are. After some extensive research, I have a rough idea that I think might work. The prompt and camera feed will be sent to a web server created by the esp32, which a Python script will be able to access via an API. This prompt is then fed into the AI, which will use the data to generate an answer. I will also create tools in the Python script that the AI can use that will turn the servos on and off through an API if it decides it needs to move. This data, along with the response from the AI, will be sent back to the web server, which the microcontroller will display on the LCD.
What is next
Currently, I am waiting on my microcontroller, which is said to arrive later this week. While I wait, I will probably continue researching ways to get this to work and try to get a better idea of what I expect of OTIS. Any help or guidance with this matter would be greatly appreciated. Thank you.
Devlog #4
Hey. This is my fourth Devlog for O.T.I.S, my personal AI desk robot. I know this is a lot of time to be putting into one Devlog, but I will try to go into as much depth as possible.
What I did
So, I started by creating a little notch for the camera module. I am going to buy an extension for the camera because it allows me to mount it separately, and because it comes with two heatsinks. Next, I made the gears that would hold the treads using a model of an SG90 attachment for dimensions. I also remodeled the top of the servo because the model had more comprehensive dimensions for the servo as well. After that, I began working on the front wheels. I created a model of a 2mm rod for the axle and made a cylindrical guide for the axle and gear. I also made a small stopper that I will attach to keep the axle from sliding out. Then I figured I should start on the assembly guide. I decided that an exploded view of the robot with a disassembly animation would suffice, but then I realized that I had not created a model for the LCD or the camera, so I began with the LCD. Once I completed that, I put the LCD on the robot and modeled holes for the screws. Lastly, I moved on to the camera module itself. I decided to model just the camera and the heatsink, since I didn’t think the extended wire would look accurate or contribute to the design. I also created models for the screws and nuts so that I could include them in the assembly guide. Lastly, I ordered the microcontroller, 22 AWG wire, and the servos.
Challenges
Modeling the gears
I had quite a bit of trouble when trying to model the gears. I wanted them to have the grooves inside similar to other common servo attachments, but I couldn’t find dimensions for the attachments that included them. I did find a 3d model online that had dimensions, so I based my design off of that. I haven’t modeled the grooves, but I will likely do so if I can find some reference. I also had a little bit of trouble finding rubber treads for the robot, since they were all too large. But I came across timing belts, which are usually used in printers or small CNC machines, and they seemed like the perfect solution. This is because they were small enough, were made of rubber, and had teeth.
Creating the screws
When looking for ways to mount the LCD to the robot, I decided that screws would work well. So I measured the size of the holes on the LCD, but they were 3mm big. This meant that a standard M3 (3mm) screw would not fit. So I looked for one slightly smaller and found M2.5. This size should work, since it is slightly smaller than the hole, but not too small. I also had a bit of trouble finding dimensions for the threads, since I thought that I would need to specify all of that when creating the screw. But I realized that Fusion only needed the size of the screw (M2.5), which I was able to provide. I did the same for the nut, using a reference I found on a listing. I will likely buy these screws from a local hardware store, so I will update my design after I buy them.
Deciding on a board
This was one of the biggest problems I had when creating this robot. Buying an unsoldered version would mean I could make stronger connections by twisting wire or soldering (I don’t have a soldering iron and don’t know how to solder). But it would make testing harder, and I would also be risking causing a short circuit if the wires shift around too much. Buying the board with presoldered pins would mean easier and more reliable testing. But jumper wires are not very secure, which means I would have to learn to solder anyway. In the end, I decided to go for the unsoldered version. My plan is to wrap the wire around the pins and to add hot glue or shrink wrap to make the connection safer while I get comfortable with soldering.
What is next
I am going to do research on the best AI for OTIS’s brain, as well as finish the 3D model and assembly guide.
DEVLOG #3
What I did
So, I started with the Schematic on KiCad. I created symbols for the microcontroller and the LCD screen (since there weren’t ones on KiCad) and used the premade servo symbols for the continuous servos. I put them all on the page and wired them up in a way that made sense to me. I have never made a schematic before and did not know if my wiring was accurate. After that, I started on the BOM. I don’t have much on there yet, just details about the main components like the microcontroller and servos, but I will expand on it later.
Challenges
Making the schematic
I had quite a bit of trouble when making the schematic. This was mainly because I had never done this before and didn’t know what to expect. One challenge I faced was creating the symbol for the Microcontroller. I started by going to the product page on Seeed Studios website. I saw a diagram with all the pins and their functions, but I didnt know what to include. So I googled an image of a schematic using the microcontroller and found that people would often just label the external GPIO pins, so thats what I did. Another problem I faced was trying to find the pin numbers. I was able to find the name of each pin on the diagram from the website, but they didn’t list the pin numbers. So I decided to look at the data sheet. So I googled for the data sheet, found one from Seeed Studio, and copied the pin numbers from a table on the sheet. But I noticed that they didn’t label all the pins on there, specifically the 5V (VBUS) pin and GPIO43 (which turned out to be TX), which is when I began to look elsewhere. Then I noticed that other schematics were just labeling them as 1 - 14 along the pins, so that is what I did. I faced similar problems with the LCD, but I decided to exclude the pin numbers and to just name them depending on their function. After I had made both symbols, I moved onto the schematic. This was fairly simple, since I just had to connect the right pins. But I was a little skeptical as to whether I had wired up the power correctly, since I didn’t know how I was supposed to wire the battery to all the components. However, I looked at some simple example schematics with batteries to see how they were implemented, and based mine off of them. I decided to power the LCD from the microcontroller since it runs off of 3.3V and the battery is set to 5V, with the servos and microcontroller running directly off of the battery.
What is next
Next, I will probably order the parts that I know I will need and begin working with them. Sadly, the microcontroller is said to arrive a week from now, while all the other parts are going to arrive much sooner unless I purchase an unsoldered version of the board. I was thinking about buying an unsoldered version and then just tying wire around the pins to make connections since I don’t have a soldering iron, but I don’t know how good of an Idea that is. Any suggestions would be greatly appreciated.
DEVLOG #2
What I did
I created a rough 3d Model of OTIS’s body. I did so in Fusion360, and also used dimensions of servos and a sample LCD screen to model those parts as well. The servos are going to be MG90S continuous servos, and the LCD is most likely going to be a 2.4-inch LCD. I had some trouble when trying to decide how I was going to run the AI. But i decided that I will use an ESP-32 and run the AI on my PC. I also decided that instead of using YOLO or OpenCV, I will probably just use an LLM. This is because I want the robot to be able to answer more abstract questions about what it sees, and I don’t know how YOLO works with LLMs since it is designed to recognize a collection of specific items, but I will do research. I also updated my plan for what I am going to do. Also, I decided to go for a Seeed Studio ESP-32S3 with the camera module instead of the ESP-32S3 cam because its smaller and has a built in mic, which will not only make the robot more compact, but will also save me some pins.
Challenges
One of the biggest challenges I had was trying to find the right parts. This is the first time I had every done research to find parts for my own personal project, so it was very time consuming and tedious.
Finding the right board
One concern I had was whether I would even be able to use the ESP 32 for this project. While doing research, I knew that I would use SPI for the LCD, but I also assumed the camera module would use SPI as well. So, I was worried about whether I would be able to use both modules at the same time, especially since there was only one set of SPI pins on the board. This would have been possible regardless, but it didn’t matter because the camera module ended up not actually needing the SPI pins, so I think I am ok.
Finding Dimensions
Another big challenge I faced was while I was modeling the robot. I had never modeled something from scratch before, so I was really lost at the beginning. But I decided to just start somewhere and model a servo. I pulled up a random photo of the dimensions of the MG90S (I was fairly confident that the dimensions would be pretty accurate since its a popular model) and began modeling. Then I modeled the robot’s size based off of the Servos and then chose the size of the LCD by modeling it and seeing how it fit.
What is next
I will do more research on LLMs and whether this is the right option for my project, as well as the ways I can use them in my project. I will also update and create more designs and work on making a README.md file for GitHub.
