“Hey Tiny Jarvis. where’d I put my keys again?”

• Tiny_Jarvis now has memory!
• I’ve implemented a simple sqlite local database in Python to store every question the user asks. (first time doing this ! very satisifying)
• This means it can help you remember timings, dates, locations… anything!
• However, this also means that all these cumulatively added memories make the prompt for the AI longer and longer
• Which is why for the moment it only remembers the last 5 prompts. This may seem like a small amount, but for the most it is sufficient for remembering recent prompts.
• Should I make it remember more stuff? tell me

“Hey Tiny Jarvis, what’s going on in the world?”

• New feature: Tiny Jarvis automatically fetches the news from BBC World, chooses the most impactful headline, reads the articles and generates a summary!
• This happens in the background, every hour, to ensure that the news is recent.
• I discovered smth called RSS feeds, which is basically a tool that feeds website updates back to you in the form of XML files.
• Then I learnt about the feedparser python library, as well as trafilatura, for extracting text from HTML (for getting the text from the articles), which made the job ten times easier.
• However, this process does take some time:
  • Step 1: script parses the BBC World feed
  • Step 2: local AI chooses between the thirteen daily headlines
  • Step 3: script fetches the associated articles text
  • Step 4: local AI summarizes ALL THREE articles.
• On a Raspberry Pi 5, obviously, this can take at least 5 minutes.
• Now for the moment this is a bit biased, because I chose the “Technology” category for the RSS feed 😅 but i’ll add the option to change it later
• Is this a good feature? Please tell me

“Hey Tiny Jarvis, play Smooth Criminal”

• mopidy-youtube integrated! and it works! All you have to do is press the button, say out loud “Play (song) …” and it will search for it in Youtube music (FREE MUSIC!!!) and play it.
• You can then adjust the volume with the rotary encoder, which is satisfying asf.
• This was one hell of a debugging session, holy
• I had to search for forked repos on Github that fixed a obscure yt-dlp issue, and somehow it works.
• Anyways apart from that, i restructured the project code repo entirely, everything is now separated into different sections like firmware and pcb to distinguish hardware from software.
• New: the install.sh was finally tested and it works perfectly after a few tweaks! woohoo!
• Oh and finally: rewrote my README.md as recommended by the Stardance guides, and yh I agree it looks better
• Tell me if you have any other ideas of features to implement

Music handler + system metrics idle screen

• Time to make Tiny_Jarvis a little more like the real Jarvis….
  SYSTEM METRICS SCREEN!

• Displays in real time: battery life, cpu temperature, current time on the OLED screen.

• Finished up the Mopidy part of the music player, can’t test it yet sadly (i don’t have access to my rpi5 right now) it was way more difficult than I thought

• AIM: “Play …” -> searches through free music online database like Youtube music for query and plays the music if it finds it.

• Added another quirky feature: flip a coin

• Tell me if you have ideas of other new useful features

New timer feature implemented, plus more

• Created a handle_prompt() function in main.py to handle any task-specific prompt given by the user.
• For example: if I say “Play …” or “Set a timer of …” then it will recognize this specific pattern and handle those actions perfectly, and if it’s something else than the AI will just answer it.
• It’s scalable, so I can add more features in the future, such as “Definition of” or idk (if you have any ideas lemme know in the comments)
• Created a timer.py file which handles the timer feature, which is sort of complicated but now works (it displays on-screen in real time, which is pretty cool)
• All in all, main.py is now very clean and readable.

Cleaned up main.py and cooking_helper.py

• It’s been a while since I last worked on this project, I’m still waiting for my hardware funding request to be approved, which made me lose motivation.
• But I’m back, so here’s what’s new:
  • I separated the main function in cooking_helper.py Unix-style, so that everything is cleaner and easier to debug or scale (in case the CSV file gets bigger)
  • Now it doesn’t need to open the csv everytime, instead it just opens it once upon startup.
  • In main.py, created an initialize() function which separates the tasks into an easier format. The AssistantDisplay class and the list of recipes is then passed onto the while True loop.
  • The main() function is now much simpler: it calls the initialization function, then in the loop it checks whether there are new recordings thanks to an imported function, if there are not it waits 0.2 seconds before checking again, and if there is then it runs the process_recording() function.
• Very clean!

Upgraded the cooking helper + critical bug fixes

• Translated the recipes to english - will add more later
• Fixed a critical issue with initializing and importing different python files, now main.pyhas the real control of initialization thanks to a listener.init_listener() function.
• Added a speak(['Ingredients']) in main.py so Tiny_Jarvis will speak the ingredients of the chosen recipe out loud
• Switched from hardcoding random.randint(0,33) to random.choice(list) way simpler and easier to read, and created a format_cooking_sentence(recipe_object) function so that main.py doesn’t have to manage that, way better
• Future changes I might implement: apparently Raspberry Pi OS manages audio through the Advanced Linux Sound Architecture (ALSA) which is responsible for the hardware drivers, the audio routing and control, and the low-level utilities such as the commands I often use, such as alsamixer, aplay and arecord
• So when two python libraries that need to either record audio or output audio call ALSA to do so, it can lead to conflicts.
• In this case, sounddevice and pyaudio which are both wrappers of the C library PortAudio. So a future change would be only using sounddevice for example, or vice versa.
• This has another hidden advantage: removing a possibly finnicky dependency and making the whole process smoother, since compiling C on a Raspberry Pi can be difficult at times.

Added a cool silkscreen logo

• Made an entry for the 23rd of July in JOURNAL.md
• Found and added this logo from The Noun Project.
• Really cool website btw. This icon clicked with me bc it really represents what Jarvis (or AI chatbots are in general) is, that being SIMULATED intelligence, and makes us remember that’s its all just complex math equations and not real intellect.
• Fixed some DFM issues, and rerouted a lot of traces, some were going over the mounting holes
• Tried to make more space for the logo but broke the whole thing and had to Ctrl+z my way out of there (help)
• Looks cool no?

PCB + updates on the cooking feature

• Wrote an entry in JOURNAL.md for today’s work
• Worked a lot on the PCB in EasyEDA Pro, mostly clearance issues but also a lot of struggling with the GND pour.
• Updates on the cooking feature:
  • Thought it would be cool to be able to show the ingredients on the OLED screen
  • So updated the AssistantDisplay class and created a neat function to show text using the python library luma.oled
  • Problem is it would go off screen, so I implemented the built-intextwrap module and it can now display 5 lines.
  • Will get to translating everything to english later, promised
• Anyways if anyone has any recommendations for the PCB don’t hesitate to leave a comment!

Added a cooking recipes feature for no reason

• Created a cooking_helper.py in a cooking/ folder
• OCR’d half of a cooking book with help from Gemini and put everything in a beautiful recipes.csv
• Used the csv.DictReader object to access the info and organize it into a list, very useful.
• Updated main.py so that when the user’s prompt starts with “Cooking”, it takes a random recipe from the csv file (using the random.randint(a, b) function) and returns a formatted sentence with all the info for cooking that recipe.
• Made this cos I’m really bad at cooking and it could also benefit others, you never know.

Schematic and PCB tweaking

• Made an entry for today in JOURNAL.md
• I’ll keep this short, but basically: huge issues with my schematic, electrically incorrect for the most part, some useless components, which then translated into the actual PCB, having to rewire traces manually, making tons of mistakes along the way and spending waay too much time repairing them.
• Still waiting for the funding request to be approved before I can order these.
• Huge thanks to M-Mrk in the Slack electronics channel for all his help, he saved me

Error handling time! + bug and dependency fixing

• Now that I have a clean slate to test my project on, I ran python main.py and somwhow got three errors in a row.
• Spent an hour adding trys and except Exception as e everywhere to prevent this.
• Now debugging is way easier and more user-friendly.
• Fixed some bugs such as ssd1106 not being a real OLED screen part name
• As well as gpiozero missing the rpi-lgpio library, which needs system-wide packages swig and liblgpio-dev so yh dependency hell.
• Almost everything should be error-proof now. I tested it, and no errors popped up, just friendly reminders that I2C wasn’t enabled, or that my buttons weren’t detected, or no screen.
• Update install.sh with these changes, now works perfectly.

RPI5 OS Switch + Tiny_Jarvis easier to put in place

• Worst day in my life. Tried to flash a Raspberry Pi OS Lite (64-bit) on the raspberry pi 5 (which btw has an NVMe SSD), and it did NOT go well:
  • First flash using the Imager: didn’t work, nothing happened, the Pi started speaking in freaking morse code
  • Spent an hour trying to decode that morse code according to documentation, thought it was an EEPROM bootloader issue, made a diagnostic card, made a factory reset card, nope.
  • Realized I could plug it through HDMI to my projector; did that: it was asking me for a username.
  • Didn’t have a USB keyboard so couldn’t type it in. Almost threw the Pi out of the window cos I configured all that in the fricking Imager.
  • Realized I was using the Imager from Flathub and not rpi-imager
  • Retried the whole thing with that Imager: still didn’t work.
  • Spent two hours doing the same thing over and over: flashing the card with the Imager, going into the files, editing user-data, cloud-init-ouput.log, etc… realized the Imager wasn’t writing the information in those files but others, booting it up into HDMI, checking the logs, managing to bypass username but not keyboard layout, etc… I EVEN TRIED LEGACY VERSIONS OF THE OS AND CREATING A wpa_supplicant.conf.
  • Atp I was going to crash out but then I checked my rpi-imager version (even though I had updated it) and turns out my laptop just couldn’t get the latest version : it was 1.8.0, not 2.1.0 (the latest one).
  • Downloaded the newest version, ran it, and it worked. I have officially lost faith in humanity.
• Anyways I read my README.md and decided that it was getting way too complicated (the installation part) so I pasted that whole part into Kimi K3 and he made me a install.sh which is basically fire since you just run

chmod +x install.sh
./install.sh

and the whole system-wide-packages-and-dependencies are all installed in the right order, easily.

• This is tested and proved (i did it with the newly flashed OS).

Major software update + PCB testing

• Found out yesterday that Ollama is just the whole UI and simplificator of use for the C library llama.cpp
• And turns out there’s a python wrapper for that library, llama-cpp-python, which has two advantages over Ollama:
  • Faster
  • Way more flexible, we can get every last bit of performance from the rpi5
• So I updated ai.py with this new feature, there are more settings than Ollama for optimizing performance for CPU-only (4 cores) which is great, however I had to download a .gguf from Hugging Face and accidentally ended up being given access to a repository for the model?! so i can now propose changes to that repo, for some reason…
• Fixed a typo which would’ve raised a TypeError in main.py
• Made it so the second button shuts down the Pi safely
• Wrote another entry in JOURNAL.md, where I soldered and tested the prototype PCBs I ordered from JLCPCB

JOURNAL.md new entries!

• Added some entries for the 7th and 8th of July, where I delved into PCB making from scratch!
• As well as an entry for the 17th of July, when I received my first PCBs - spoiler alert: they do not work as intended
• Made a lot of mistakes, but managed to make a rather good first try, which will hopefully lead to a better final PCB (i’m working on that)
• This time I’ll check the DFM before ordering
• Hope you appreciate my storytelling!

JOURNAL.md started + Hardware time!

• Created a JOURNALmd file
• Documented everything I did on the 5th and 6th of July, with photos and some explanations as to what I was doing and trying to achieve
• To make it short: prototyping and creating some visuals for the OLED screen/UI, as well as assembling everything together.
• Connected README.md to JOURNAL.md
• You can read it here

Finishing up the software part

• Took on the task of making my code more readable, and as such easier to debug
• Created a AssistantDisplay class to better manage the OLED threads - way cleaner now
• Put sentence_buffer() in ai.py and get_lastest_recordings() in listener.py, now the only function in main.py is main(). Clean!
• Squeezed a bit more performance out of the faster-whisper model by setting cpu_threads to 4 and beam_size to 1, even if we’ll lose a tiny bit in quality. Still a good tradeoff
• Detected an avoided a critical issue: if glob detected the file while it was being written to and passed it on to faster-whisper, then it would be corrupted and be mashed garbage. So I read some documentation about the queue module and used that, it’s way better and easier in the end.

Faster response speed thanks to streaming audio

• Ok so ollama’s ChatResponse class allows, when creating an instance of it, to write stream = True.
• So I could stream the outputted response directly into Piper TTS and into the speakers.
• Except it would sound very weird, very robotic-like since the chatbot outputs tokens, not words. I then created a function called sentence_buffer() that seperates them into a list of words, that are then given to Piper TTS in speaker.py
• Then I used a library called pyaudio to stream the Piper TTS output into the speakers! it’s actually just a Python wrapper for a C library called portaudio, so I ran into a LOT of dependency issues, but that’s a battle for another day.
• Moved a few functions that were in main.py into other files for better management
• Other small modifications

Threading implementation for better speed

• Today I decided to learn about the python module threading, and realized it wasn’t that difficult to implement in my project.
• The issue was: when I ran oled("thinking_animation.gif") it ran sequentially, so the whole script had to wait for the animation to finish, which obviously can’t happen.
• So I created different threads for each animation/png, and made a “switch”, better known as a threading.Event() which enabled me to manage the “workers” way better (sorry, i’m getting used to the vocabulary)
• Also I added comments everywhere in main.py for better explainability.
• Finally I adjusted the code in OLED.py with the switch implemented and added some small user-friendly checks throughout my project.
• Hopefully everything works!

Note: sadly it will be difficult to use threads in the rest of the code, since the process is, at its core, sequential. Faster-whisper needs audio to proceed, the ollama AI chatbot needs a prompt to continue, etc…

README.md first commit

• Spent a LOT of time on the README.md, I want my project to be reusable so made a good How to install section, there are tons of different commands to run to ensure it works first try
• Didn’t write anything in the Hardware section, but will soon. (my first PCBs don’t work yet)
• In OLED.py: it now supports gifs, which is great because of Piskel which can turn simple animations into gifs.
• Made some visuals in Piskel for the screen (thinking_animation.gif and button_pressed.gif)
• Simplified drastically listener.py for future debugging.

One of the biggest issues with this project so far: the difficulty to test it! I don’t have all the required materials yet. I might use the pytest module? we’ll see

Linking everything together

• Updated the project structure with more python scripts so that it now looks like this:

Tiny_Jarvis/
├── .venv/
├── piper_voices/
├── recordings/     # New
├── assets/    # New
├── ai.py
├── listener.py    # New
├── main.py
├── OLED.py
├── speaker.py
├── transcriber.py    # New
├── README.md
└── requirements.txt

• What’s new:
  • Added an assets/ folder for the OLED screen visuals
  • Added a recordings/ folder for the audio captured by the mic when the user is speaking
  • Created listener.py: actively listens for a button press; when pressed, records audio through the usb mic until the button is released, then that audio is sent to transcriber.py
  • Created transcriber.py: takes that audio and passes it as input into faster-whisper to “translate” into text !
• And finally, first time linking everything together in main.py: there is still a lot to be done if we are talking speed/efficiency, but the core pipeline is done.
• The user presses a button to talk, releases it when finished, the audio is captured and translated into text by faster-whisper, then that text is given as prompt to the AI chatbot (like Qwen3-0.6B. The response is then turned into a voice using Piper TTS and outputted through the speaker!

Possible future implementation of the threading module to make it faster, even though I have no idea how it works.

Initializing the project

• Created a virtual environment .venv/ so that all necessary python modules and packages are cleanly installed together.
• Created a requirements.txt, where I put all the python modules I import in the project so that people can reuse it.
• Created a piper_voices/ directory and downloaded a medium voice via python -m piper.download_voices en_US-lessac-medium
• Created 4 different files:

Tiny_Jarvis/
├── ai.py   # the AI chatbot
├── main.py   # the main script
├── OLED.py    # The visuals!
└──  speaker.py    # the audio

• Wrote the basic code for each of those files. For the moment, I concentrate on making importable functions that function indepedently, and will link them together in main.py later on.
• Created a README.md. It’s empty for now, will fill it in later.