You are browsing as a guest. Sign up (or log in) to start making projects!

Aura — Sub-Second ESP32 AI Voice Appliance & VoIP Telephony Bridge

By @shravanmali_main

  • 2 Devlogs
  • 5 Total hours

Aura is an ultra-low-latency (<800ms) Smart Voice Assistant that can be used as an intelligent bridge between Landline Telephony.

It allows your home broadband connection to be used with your mobile by using native SIP VoIP over JioFiber IMS. This means you can make hands-free landline calls by simply saying “Alexa call Mom”.

Current Build (v0)
ESP32-S3 N16R8 - 16kHz I2S with INMP441 MEMS microphone and MAX98357A Class-D amplifier.
Wake Word Detection (on-device), Groq LPU (Whisper Turbo + Qwen-27B), Text-to-Speech (on-edge). (<800ms).
Acoustic Echo Suppression (AES), Sound Radar, 8-Bit LED RGB Engine (WS2812).
Multi-Turn Recipe Memory (with missing ingredient handling and prioritised timers).

Upgrade Roadmap (v1)
Host engine will move to local raspberry pi with DietPi (Less than 260mb ram). Computer Vision, ArUco Automatic Docking, and pose tracking for mobile rover.

Source
Ship #1 Pending review

@shravanmali_main on Aura — Sub-Second ESP32 AI Voice Appliance & VoIP Telephony Bridge

I didn’t just make a voice assistant which gives simple AI replies .
what i actually made is the sub-second smart voice assistant and real landline telephone appliance from scratch on an ESP32-S3 microcontroller.

there was most of this three challangeing things for me which i faced to reach the full and final V0 of the project which were

i have given them ranked as-

1.Latency: Commercial assistants take 3 to 4 seconds to respond; optimizing cloud LPUs (Groq Whisper Turbo + fast TTS) to get sub-second (~800–1200ms) conversational turns was a huge optimization puzzle.

2.Audio Glitches & ESP32 Watchdog Crashes: Dropped Wi-Fi packets caused the audio to stutter and crash the ESP32, which ifixed by separating the mic and speaker into two separate I2S hardware buses.

3.Acoustic Echo & Feedback (on which i am still working on): The microphone kept hearing its own speaker and looping into feedback, forcing me to design a software echo suppression window (ducking) to mute the mic while the speaker talks.

And the things I am proud of are -

:Getting a real SIP landline phone call to trigger and connect smoothly through a bare ₹850 microcontroller.

:The sub-1100ms speed approx —it feels like an instant human conversation rather than a robotic delay.

: Memory - It remembers memory Also after it is rebooted (Eg - , recipe ingredient limits, and project identity)

Here are direct links of the video which shows the latency and some of the features including Calling , memory …

Video 1: Sub-Second Multi-Intent Voice Query (Live Weather & Bitcoin): https://youtube.com/shorts/PmZ0z8Imm-w

Video 2: Live ESP32 VoIP Telephony Call (Voice-Activated SIP Call): https://youtube.com/shorts/HGMN75dKRuE

Video 3: Autonomous Long-Term Memory & Identity Recall Registered on Stardance
▶️ Watch on YouTube Shorts

the people who will make or let it run in IRL - feel free to ask questions and can also suggest .

thanks .

GitHub repo link - https://github.com/shravanmali/aura-esp32-voice-assistant

The repository includes a local web dashboard (http://localhost:8080) where all live WebSocket events, timers, and transcripts can be viewed.

  • 2 devlogs
  • 5h
Try project → See source code →
Open comments for this post

@shravanmali_main on Aura — Sub-Second ESP32 AI Voice Appliance & VoIP Telephony Bridge

1h 21m 20s logged

Devlog Update #3: Adding Long Term Memory to Aura + Stardance Identity! 🧠⚡
Hi everyone! Shravan here.

Most voice-assistants you create on a breadboard will have very short term memory and will forget once they are asked a question. Today I’ve added autonomous long term memory to our minimal ESP32-S3 hardware!

Aura now has the ability to pick up natural conversation, detect what the user’s identity is, and write it into the persistent storage with no delay.
Today I was able to do the following:

“I am Shravan and I have registered you on Stardance platform.” I said to the hardware.
The hardware picked up my name and that I had registered it to my project. It saved this into the persistent storage with no delay. I could see it being added to the dashboard via WebSockets.
I then said “Where did I register you?” and it automatically remembered this fact in under 600ms. It said “You registered me on Stardance platform.” via the speaker.
Watch the hardware in action:
https://youtu.be/rkv8eTRfRxI

Read the full part 3 article and code on GitHub:
https://github.com/shravanmali/aura-esp32-voice-assistant#part-3-autonomous-long-term-memory-stardance-demo

0
0
1
Open comments for this post

@shravanmali_main on Aura — Sub-Second ESP32 AI Voice Appliance & VoIP Telephony Bridge

3h 57m 33s logged

[ [ Aura ] Building a sub-second Voice Assistant & Full Duplex VoIP Telephony System using Bare Silicon 🚀 ]
About
This is a low-latency, full-duplex Smart Voice Assistant and VoIP Telephone system using only 5 components using only a bare ESP32-S3 microcontroller. Not only can you ask questions using this like you would with a Smart Speaker, but we also go to extremes using the microcontroller silicon! Making actual landline calls across the country using a one voice command using the ESP32.

[Note: Using the pre-trained Alexa firmware models until the wake-word neural network model has been trained.]

Hardware
ESP32-S3 DevKitC-1 (Microcontroller: Dual-Core Xtensa LX7 @ 240MHz, 16MB Flash, 8MB Octal PSRAM)
INMP441 Ominidirectional MEMS Microphone (Digital: I2S Sound In)
MAX98357A 3.2W Class D Mono Amplifier & DAC (Digital: I2S Sound Out)
4Ω 3W Full Range Speaker
WS2812B 8-Pixel RGB LED Ring (Live Visual Feedback)
Built onto a basic Breadboard using jumper wires. Custom 2-Layer PCB being made.
Development Time (~65hrs spent coding optimising firmware & DSP)
There were many challenges when trying to make a fully working full-duplex system using the raw development silicon:

Audio Cleaning & Removing Noise
Problem: Large noise from the 50Hz hum from mains along with DC offset was being captured by the MEMS Microphone due to the breadboard jumpers.
Solution: Designed a digital signal chain with a high pass filter (85Hz) to block out mains hum. A peak filter (3.2kHz) was used to ensure there was no distortion in the voice.
I2S Bus Splitting & Preventing Watchdog Crashes
Problem: When both the microphone and speaker were capturing/sending sound, the DMA was sharing resources causing audio distortion and watchdog crashes with FreeRTOS.
Solution: Split the audio into two separate I2S devices (I2S0 for sending sound, I2S1 for receiving sound). This prevented any deadlocking and allowed full-duplex.
Reducing Latency (< 800ms)
Problem: Most DIY Voice Assistants have a latency between 3-5 seconds.
Solution: Used the Groq LPU to handle processing (Whisper Turbo & Groq LLM). Used Cartesia Sonic for streaming TTS. Clauses are sent to the DAC in milliseconds after detecting silence.
Live Voice Assistant with AES
A custom SIP client was made to talk to JioFiber’s IMS exchange. This allows the ESP32 to make live calls. Ducking ensures that the microphone does not echo back the sound from the speaker.
Autonomous Memory System
Captures information about the user and what platforms they’re registered too. All data is stored in JSON format. Dashboard can be viewed using WebSockets.
Live Demo Videos
[Video 1] Sub-Second Multi Intent Voice Assistant (Live Weather & Bitcoin) 📺:
https://youtube.com/shorts/PmZ0z8Imm-w
[Video 2] Live ESP32 VoIP Call 📞:
https://youtube.com/shorts/HGMN75dKRuE
Full Documentation & Deep Dive
View the full documentation, schematics & Milestones for Part 2/3:
https://github.com/shravanmali/aura-esp32-voice-assistant/blob/main/PROJECT_STORY.md#part-2

0
0
2

Delete project?

Are you sure you want to permanently delete this project? This action cannot be undone.

All devlogs, followers, and associated data will be removed.

Followers

Loading…