This session completes my base project.

Last time I logged, I’d gotten a chatbot to speak sign language through the robot hand using serial. This means basically half of the interface was done. The other half was for the chatbot to interpret my sign language so it could adequately respond.

In one of my earlier logs, I mentioned finding a repo that supported fingerspelling, so you could spell out any word. This session I forked that repo, and edited the script such that, signs it interprets were spoken to a chatbot, and that chatbot would communicate outward through serial.

However finger spelling is easier said than done. The first difficulty was that letters were often misinterpreted, which was frustrating, but also a limitation beyond my control. The second difficulty was repeating consecutive letters. If I held my hand too long, the model assumed I intended for the letter to be repeated, which lead me to incorporate a cooldown. Finally to adress spacing between words, I set a condition where both hands must be present for over 2.5s, to include a space character.

With this the bridge was effectively complete. What I have left really is optimizing this process, by perhaps improving the hardware design of the hand to grant more freedom, or maybe training a more robust sign language interpreting model.

But all of these options sound boring to me. Instead I want to learn something new, so the second half of this work session, I spent deliberating ideas, and I came across IsaacSim, a simulation software for robots developed by NVIDIA. I plan to learn more about this topic, to allow for the simulation of this hand, such that I can attempt a more sophisticated approach to a sign language robot, than just static recorded symbols.

Anyways, that’s all for this session folks…