@justin_m_baratta

Updates on JADE!

So! A lot of stuff has happened since the last devlog. I went to Boston last week which is why I wasn’t able to do too much, but I’ve done a lot with the actual hardware component the last couple days…

I finished the first iteration of my tools. I have five mini-agents that control tools on the weather, web, calendar, writing (journals, notes, todos), and slack. These are controlled by a kind of global agent. Basically, it takes in the query from the user and selects one of the mini-agents to use, which will then select the tool to use. This was a kind of experiment to see how well agents would interface and control other agents, but it worked well from my testing!

I was able to solder pins onto a mini-amplifier, connect a speaker to it, and connect that to my raspberry pi 4 using jumper cables. I wrote a test of the microphone in python and it makes noise! However, I think I must’ve messed up the soldering or something because the audio quality is pretty bad…

I started on the Alexa/Siri-like component of the project by training an openwakeword wake-word model on the word “Jade” and am experimenting with having it work better in noisier environments. It does detect when I say “Jade”, which is awesome! Once I solder my mini-microphone and connect it to the pi, I should be able to run the full pipeline. Here are my next steps:

1. Finish “Jade” Wake Word Model + test
2. Train or implement speech to text model, test connection with global agent
3. Connect + test microphone to see if speech to text and wake word run on rpi from mic
4. Fix speaker or solder new speaker and test audio
5. Connect agent response to speaker and test full pipeline

Writing tools complete- the agent is able to make, edit, and delete todo lists, notes and journals! Mostly tested so far. On the hardware side, I finished soldering everything so I am now able to connect amplifier and speaker to the pi, though not the microphone bc i dont have anough jumpers :( . will start writing software on the pi for connecting and using the speaker and amp.

Finished slack agent/slackbot + tested. Able to read messages from channels and DMs, write and send messages in channels and DMs, reply to threads, get latest messages, search for a keyword.

Web Tools class done + tested! Agent is able to search for a definition (webster dictionary), topic (wikipedia), current news headlines (news API), stock prices (yfinance) and summarize results!

Weather class completed + tested- agent able to tell current weather conditions and forecasts

Calendar class for my desktop assistant. Uses Strands API to create classes and methods whose tools the agent that connects to the rpi can use. This first class/tool the global agent can use is finished, a basic calendar to keep track of important stuff.

Making a game about statistics (regression, distributions, correlation, etc.) inspired by Guess the Correlation for a student-led class I’m teaching with my friend next year. I am able to generate questions for users to guess the type of distribution and the correlation of a randomly generated scatterplot, will add more theory-related questions later.