Note: This one is kinda boring, mostly plumbing stuff to get things to work, feel free to skip it.

Sephyr (the mod)

I’m working on setting up the the logic for registering and casting spells. This is quite hard, as the design of the mod is far from complete, which means I don’t even know what exactly do I need. I created a registry of elements, which are a source of a color, and are going to have other effects in the future. The first word of a spell is its element. For example in the spell ʔalɪˈvɑn vɛɾtɛɾ ɛvi the word ʔalɪˈvɑn means wind. I also made a basic system for figuring out the spell the player is casting. For now its very simple and can’t handle more complex spells, for example, with optional words.
The entirety is a big mess as I’m figuring things out, and the fact I’m making this mod for both Neo and Fabric doesn’t make it any easier.
I also updated the mod to Minecraft 26.1.2, which means there is going to be no 1.21.1 release.

Sepple (word detection system)

I keep working on collecting data to finetine the speech to ipa ml model. After discovering a better way of getting the audio files while writing last devlog I scraped the download script, and instead replaced it with a new one to extract needed files from the 20GB Wiktinary audio dump tar.
I also cleaned up the entire code, turning it back to human standards, after a bit of Opencode usage. (Since those are one-time scripts to extract data I decided using a bit of AI in this area won’t hurt. I don’t use AI for the rest of the project.).
I also prepared a fourth script, that will use the existing multipa model, to generate IPA transcription of the files. Then files with generated IPA being too different from original IPA will be discarded, as a way to detect files, where the IPA doesn’t match the audio (because the audio contains more text for example).

Sephyr (the mod)

• A proper lifecycle of the detection system was added. The IPA model is loaded in background during game start, but the audio capture and processing start waits until you join a world, and stops once you leave it.
• The Rust code now has a logging hook, so that it can do civilized logging using the Java logger, instead of blatantly writing to stdout.

Sepple (the voice casting detection system)

• I made the pipeline system cancellable, so that the mod can cleanly stop the detection, which was not possible before.
• Support for running the speech detection inference on GPU was added as a feature flag. For now it is not used by default, as it showed no measurable improvements over running on CPU on my PC, but it might be useful on other, weaker devices.

Collecting data for training

As I said too many times, the speech to IPA model used for magic speech detection is not quite the quality I would like it to be. I’m hoping to improve it by fine-tuning it on its weak areas, but for that I need some data.
My idea is to use wiktionary as a data source. It is a free online dictionary run by the same people who run Wikipedia. Most words on it have the pronunciation attached both as an IPA and an audio file, which makes it ideal for the task.
I downloaded a preprocessed jsonl dump file from kaikki and wrote a script to extract just words and ipa from it. Unfortunetely it was missing urls of the audio files. So I had to download a raw xml dump of the entire english wiktionary, and spend few hours writing a script to extract them. Then I used deepseek a generate a third script, to download all the audio files from a list. There is going to be almost 20k of them, and the script is running right now as I’m writing this, and it is going to take multiple hours as I can only download around 1 file per second to respect rate limits and not overburden wikimedia. And just now while writing I realized I’m an idiot, because kaikki provides a 20GB tar file with all the audio, downloading which would be more efficient.

Integrating Rust IPA word detection with Minecraft

With the IPA based magic keyword detection (which is written in Rust) working I spent all of this devlog integrating it into Minecraft. I created a new repo for the mod, using the Jared’s template. I decided to go with supporting both Fabric and Neoforge, and picked the 1.21.1 Minecraft version, though in the future I’m planning to move to 26.2 or later. I set up an additional gradle subproject and a rust package, that both together act as a Java to Rust bridge, using the j4rs crate.
Using it I managed to initialize Sepple (the detection system), and get it running. The tricky part was going the other direction, to pass detected words from Rust to Java. For some reason Rust wasn’t detecting the JVM and was trying to spawn a new one!
After a bit of experimentation I concluded, that I can only call Java from Rust only during the period when Java calls Rust, and only on the same thread. In the end I managed to get it working, by spawning a new thread on the Java side, and calling a blocking Rust function, that kept calling a Java callback passed to it. Originally I tried to use a static function as a callback, but for some reason (probably some kind of classloading isolation) the system was failing with some crazy null errors.

I also had to adjust Sepple a bit, to not require a bunch of random json files at specific paths, and to provide an mpsc chanell directly outside its pipeline, for less boilerplate.

With that I have a proof of concept working! Minecraft is receiving the detected words and printing them to chat.

Note that those are all custom words, created by me specifically for this mod. This is why I had to create an entire new system for detecting them, and why they are written in the International Phonetic Language, which is also why they look a little weird in the minecraft’s font.

The Java<->Rust setup is also not very good yet. For example the packaging of compiled Rust is very basic, and you can’t control the word detection from Java - no way to stop it or restart. There is also no good error handling, which means if a file is missing it will just crash everything.
But it works, and I can continue working on the mod, and making it better!

About project

Sephyr is a magic mod, with spell casting powered by speech live translated into International Phonetic Alphabet, to detecting custom-made spell words in it.

Adding Voice Activity Detection - part 2

After the refactor I was able to finally add the VAD. But after few tests I noticed that the VAD was making things worse, not better!
That was as a part of the previous devlog (which ran out of character limit).
I added a pipeline part that writes every chunk of audio that passes through it to a file, so it can be inspected.
With it I found the main problem: It took few hundred milliseconds before the VAD detected speech, so a small chunk was always cut off from the start. To fix this I added context keeping: A bunch of previous chunks are kept, and when speech is detected they are also taken into consideration.
I also made the sliding window chunking aware of VAD - the VAD detector sends a signal when it detects an end of speech. The chunker then flushes its buffer into the pipeline, so that it doesn’t get stuck until next speech, and replaces it with precise amount of silenece. This way the start of next audio doesn’t get cut off later due to the sliding window mechanism.
The author of the bunsen crate (providing the VAD model implementation) also released version 0.29.0, which fixed all the issues I encountered in the last devlog, and updated its API slighly, which meant that after adjusting my code I could stop using my own fork of it.

Improving word detection from IPA

My friend found a crate named phonetics-rs and suggested using it to improve the algorithm. This crate calculates the difference between two IPA strings, taking into account how similar they sound. This is much more accurate than just comparing strings on grapheme level. I modified my algorithm to use it, and it turned out to be a great idea, that significantly improved the detection rate of magic words.
I also changed the dictionary to be sorted by longest words first, so that they take priority over short ones.

So far, testing and improving the word detection algorithm has been the most fun out of the entire project. Looking at IPA strings, and figuring out why some speech has or hasn’t been classified as some word is quite fascinating. For example, I encountered a string pɾilifɛra which the model output after me saying prizim and fɛra, the two magic words that I made up for the spell system. Surprisingly, my algorithm said the string contained just prizim despite there being a literal fɛra in it.

As I found out, pɾilif is closer then pɾili to prizim, and it fell under the threshold, so the algorithm took that part and left out ɛra in the buffer which was too far from fɛra. After seeing this, I made the algorithm scan the string for exact matches first, and then do fuzzy phonetic matching, also ensuring the order is preserved. I also tuned the thresholds, adding more and more tests (there is now 8 of them) to ensure the algorithm behaves well.
Finally, I noticed the IPA output from the model changes based on audio’s offset in time. So I wrote a script that lets me test this, and see the effect of different offsets.

What’s next

There is a saying in data science: garbage in - garbage out. This is also true for my project. The detection works decently, but it also quite often outputs false positives during normal speech. I believe I did I all could to compensate for the unsatisfying quality of the speech to IPA translation model.
Now I will be shifting to the Minecraft part of the project - creating a mod, integrating the detection system into it and creating spells. In paralell, I will be exploring the possibility of finetuning the model, to improve the accuracy of it and hopefully get better results.

Watch the video with audio, that’s the part that matters right? :)

Finding words in long IPA strings

For the magic system I need to find individual words in the ever accumulating string of IPA characters. To do that I hold a list of valid words in memory. When there is some text, I iterate over the dictionary, calculating a levenshtein distance between each word and start of the string. Then the word with the lowest score is taken and if the score is under a set threshold, the word is considered a match, and the matching part is remved from the buffer. To cover more cases different lengths of the string are checked. If no match is found, one character is removed from the string, and the process repeats, until all words are found.

Fun fact: When comparing IPA and other non-ascii strings you need to make sure you are considering graphemes and not characters, because in UTF-8 what we perceive as one symbol, can be encoded as many characters in memory.

So far this algorithm works quite well, though in the future it might need some performance optimization, or tuning, if it starts yelding too many false positives.

Other misc changes:

• Migrated main fn to use clap for parsing arguments
• Added flake.nix with dev env so I can run this code on my NixOS laptop
• Added time information to the pipeline - Together with the audio from the mic, a timestamp is captured, and passed through entire pipeline up to the end, where it is used to discard text older then 200ms from the buffer, so that stale data doesn’t mess up new words.

Adding VAD

I noticed that the speech to IPA model likes to output random data when no one is talking, so I decided to add VAD - Voice Activity Detection - a system that detects if someone is talking, and only then passes the audio on. I estimated this to take me around 2 hours, and oh, how wrong I was?
It took 10x that, which is why this devlog documents 3x the normal amount of time. And it still isn’t fully ready! So let me tell you what I did and why it took so long.
In the Burn (ML/AI lib for Rust) discord I noticed a solution just dropped out of the sky: Merely few days before a Silero VAD model implementation was added to a library named bunsen. ML models take input as Tensors - multidimensional arrays, which do not say in what form or shape the data is expected. So I had to figure that out myself. Fortunetely it was just 512 elements long chunks of 16khz audio with no additional pre-processing. Knowing that I fed the microphone input to the model and began to watch it output complete nonsense unrelated to whether I was speaking or not. So I started debugging - checked my code, wrote an equivalent in python using the official silero implementation to compare results - completely different. So I cloned the bunsen library, wrote an unit test there and ran off a Opencode agent. I also contacted the library author. He found one major issue, and the agent identified a second problem and wrote a solutuion - Silero VAD expects 64 samples of the previous chunk to be passed along the 512 to work correctly. I cleaned up the code, which was quite a pain because rust-analyzer was super slow on that repository, and made a PR, but it wasn’t merged because the author went with his own way of implementing it.

The big refactor

Coming back to my project I realized that my code strucutre makes it very hard to integrate VAD into it. So I spent the next 6 hours of so refactoring almost every file in the codebase into a new pipeline system. It lets you compose a pipeline out of a producer, many processors, and a consumer. It makes use of async, which saves on the amount of threads used. Static type checking ensures you can’t compose parts that wouldn’t work together, and all elements can also define contracts on the size of data chunks they produce and consume.

I ran out of the character limit, so the rest of VAD story will be in next devlog.

Live speech to IPA transcription in Rust using ML

I finished my rough implementation of a live speech transcription pipeline. It captures my speech from the mic, and processes it with the Multipa model using the Burn framework.
To make it work live I implemented a sliding window that chunks the audio into 2 second fragments, each of which is processed separately, and then assembled into a single string. The chunks overlap, and are then sliced, so that audio that was on the borders of a cut (which will reduce the quality) is thrown away, and the text for it is taken from the next chunk, for which it is at the center.

The inference is running on cpu for simplicity and compatiblity, so by sacraficing a bit of pain I added threading (where each chunk is processed on a separate thread) to make sure the pipeline can keep up. I’m slightly worried about how the perfromance will be when this is going to run along Minecraft, but hopefully I will be able to make it as lightwieght and unintruisive as possible.

Finally I reworked how the text is accumulated, switching from a vector of logits to a string. Since IPA has no concept of words, just phonemes, everything I say is appened to one long string.
So I added a very simple silence detection, where if a captured chunk returns no text a space is added to the string.

For easier teasting I also improved the main function, so that the program can:

• take a file and process it all at once
• take a file and process it using the pipeline
• run live mic capture on the pipeline

What’s next - improving the detection

In theory this is all I need, and I can start working on the Minecraft mod. However I’m not satisfied with the quality of the transcription, and at the level it currently is most spell detection will probably fail.
The quality of the model itself is pretty bad. If the description on github of the Multipa model is correct, the model was trained on merely 9 hours of speech data, which probably explains it. The model is also sensitive to noise, and distractions like other people talking, and sometimes outputs garbage.
So the next devlog I will be experimenting with various techinques of filtering and cleaning up the audio the model is receiving, and potentially even training a new model myself on a bigger dataset.

Attached video

Working demo of live capture and transcription of me talking.

Sephyr - project overview

Sephyr is an unique magic mod for Minecraft. It aims to differentiate itself by:

• Introducing a voice powered spell casting system, that’s both fun to use and intricate. You will be reading spells out loud into your mic in a magic language, and what will you say will affect the result of the spell.
• Having stunning, detailed visuals unique to each spell. This is going to be achieved by spending way too much effort on rendering. I want spell casting to be something satisfying, and the look is a key part of it.

The project consits of two parts:

• Sephyr - The minecraft mod itself containing the spells, mechanics and visuals, written in kotlin.
• Sepple - Speech processing engine, responsible for transforming your babble into something Sephyr can work with and figure out what have you just said.

Sepple overview

Sepple is a foundation of the voice powered spell casting. It is an audio processing library written in rust, that Sephyr will depend on and embed. It captures your mic live and pre-processes the speech. Then it uses the Burn framework to locally and quickly run a speech recognition model called Multipa. The model transcribes speech into the International Phonetic Alphabet (IPA), a textual notation of various sounds that make up the spoken language. Finally it tries to match the result against a custom dictionary of magic words, which will form sentences - the spells you will be casting.

This devlog

Before Stardance launched I have written a rough skeleton for Sepple. With some trouble I managed to import the Multipa model to Burn via ONNX export from Pytorch, and implement the required post-processing to decode model’s output into IPA. I also started working on a pipeline for mic capture and sliding window processing, so that the audio can be transcribed live.
After Stardance started I noticed that the same input audio file gives quite different result in Pytorch then in Rust. I spent few hours debugging why that is, and found a missing piece in my implemenation. For the Multipa model to work correctly the samples need to be normalized using a so-called z-score normalization. That means calculating the mean of the input and its standard deviation (a statistics measure), and applying a formula that rescales the data, so that it has a mean of 0 and standard deviation of 1. After implemenating that the model started to work correctly and give the same output for the same input on both sides.

Picture: “Hello Stardance. Sepple is working now” translated to IPA using pytorch and the fixed Rust implementation. It is not veeery accurate, but it is good enough.