Ship
Pending review
JADE pretty much complete!
Its been a while since the last devlog, I know. I’ve been really busy these past couple weeks of summer and have not been able to post, though I was able to work on JADE a little bit every day until its pretty much as done as it is now.
To start, the pipeline of JADE is finished, tested, and functional. For hardware, using a mini microphone, mini amp, mini speaker, a breadboard, and lots of jumper wires, I was able to make my first “hardware project” connected to a raspberry pi. The pipeline of waiting for a wakeword (“JADE”), then activating once hearing the wakeword and using a speech to text engine to process audio (all using the microphone), then sending the request to an agent, which will output a message that will be played using text to speech on the mini speaker, all works!
Some things I tweaked were, for one, the agentic architecture. Originally, I had a global agent that controlled 5 sub-agents that specialized in specific tasks (e.g. calendar agent, weather agent, etc…) but ended up switching to a one-agent structure that uses an embedding model beforehand to filter for the 5 most relevant tools before passing the query and appropriate tools to the agent. I did this because running multiple agents using the strands API on a raspberrypi is extremely computationally intensive and makes the pi reboot itself.
Additionally, the slack tools and slack agent were deleted because they were too complicated and did not fit into the scope of this small project. The calendar tools were also refactored to interface with Google Calendar.
All in all, the pipeline works, and this was a great project wherein I learned how to write code and set up a raspberry pi, use agents, and use audio I/O, all which definitely improved my abilities as a programmer. Expect a ship soon!
Updates on JADE!
So! A lot of stuff has happened since the last devlog. I went to Boston last week which is why I wasn’t able to do too much, but I’ve done a lot with the actual hardware component the last couple days…
I finished the first iteration of my tools. I have five mini-agents that control tools on the weather, web, calendar, writing (journals, notes, todos), and slack. These are controlled by a kind of global agent. Basically, it takes in the query from the user and selects one of the mini-agents to use, which will then select the tool to use. This was a kind of experiment to see how well agents would interface and control other agents, but it worked well from my testing!
I was able to solder pins onto a mini-amplifier, connect a speaker to it, and connect that to my raspberry pi 4 using jumper cables. I wrote a test of the microphone in python and it makes noise! However, I think I must’ve messed up the soldering or something because the audio quality is pretty bad…
I started on the Alexa/Siri-like component of the project by training an openwakeword wake-word model on the word “Jade” and am experimenting with having it work better in noisier environments. It does detect when I say “Jade”, which is awesome! Once I solder my mini-microphone and connect it to the pi, I should be able to run the full pipeline. Here are my next steps:
- Finish “Jade” Wake Word Model + test
- Train or implement speech to text model, test connection with global agent
- Connect + test microphone to see if speech to text and wake word run on rpi from mic
- Fix speaker or solder new speaker and test audio
- Connect agent response to speaker and test full pipeline
Writing tools complete- the agent is able to make, edit, and delete todo lists, notes and journals! Mostly tested so far. On the hardware side, I finished soldering everything so I am now able to connect amplifier and speaker to the pi, though not the microphone bc i dont have anough jumpers :( . will start writing software on the pi for connecting and using the speaker and amp.
Finished slack agent/slackbot + tested. Able to read messages from channels and DMs, write and send messages in channels and DMs, reply to threads, get latest messages, search for a keyword.
Web Tools class done + tested! Agent is able to search for a definition (webster dictionary), topic (wikipedia), current news headlines (news API), stock prices (yfinance) and summarize results!
Weather class completed + tested- agent able to tell current weather conditions and forecasts
Calendar class for my desktop assistant. Uses Strands API to create classes and methods whose tools the agent that connects to the rpi can use. This first class/tool the global agent can use is finished, a basic calendar to keep track of important stuff.
Making a game about statistics (regression, distributions, correlation, etc.) inspired by Guess the Correlation for a student-led class I’m teaching with my friend next year. I am able to generate questions for users to guess the type of distribution and the correlation of a randomly generated scatterplot, will add more theory-related questions later.
