v0.1.0

Well Stardance told me this:

Warning
You have 1h 14m logged that you haven't posted a devlog for. If you ship without posting a devlog, this time will be lost!

IDK why, but i was planning to devlog my silly anyways so:

five commits. all tagged v0.1.0. all fixing things that should have been done before I said “the project is ready.” the classic “wait I forgot” sequence.

“Oops” - CI was trying to build docs without running npm ci first. added the install step. three words of actual code change. (this was wrong)

“Huh. where did that come from.” - the pages deploy was creating a docs/ directory but the docs copy was still commented out. the demo was deploying fine but docs were going nowhere. fixed the mkdir.

“Oooops guess who forgot to uncomment some code” - uncommented the actual docs copy in static.yml. also fixed the docs URL from nisoku.org (wrong domain) to nellowtcs.me/UTAU.js/docs. details.

“You never saw that coming did you /sarc” - added all the npm publish metadata to package.json. license, author, repository, homepage, bugs URL, and keywords. the keywords are: utau, synthesis, singing, voice, speech, audio, dsp, lf-glottal, source-filter, formant. so if you npm search for any of those things, maybe you’ll find this. someday. (NPM release failed lol, that’s why I did this)

“Badges hehe” - npm version badge, CI status badge, MIT license badge at the top of the README. because a project isn’t real until it has badges. everyone knows that. (/j)

five commits to go from “done” to actually done. four of them are CI fixes. like i just said, this is software development.

CI is green. docs are deployed. npm metadata is set. badges are shiny. v0.1.0 is tagged.

so. that’s UTAU.js.

six days. from “idk my voice tho” in an Obsidian doc to a parametric singing synthesizer with three languages, 14+ import/export formats, a Svelte demo app, a docs site, 160+ tests, and a v0.1.0 tag.

it sounds mid. like, genuinely. the vowels are recognizable but the consonants are strange and the whole thing has this uncanny valley quality where you can tell it’s trying to be a voice but isn’t quite there. it’s really adorable when it tries :3

but it makes sound. from math. in a browser. and that’s the whole point of v0.1.0. it doesn’t have to be good. it has to exist.

it exists now.

:)

the docs exist and the README isn’t “Web-Template” anymore

so uh. the README has said “Web-Template: My typical web dev starter project” for the entire life of this repo. six days. dozens of commits. a full synthesizer. and the README was still introducing it as a starter template.

fixed that. also wrote an entire documentation site. in one sitting. because apparently that’s how I do things now.

the README

133 lines replacing the 3-line placeholder. features list, quickstart code (Node.js WAV rendering + browser streaming in like 10 lines each), API overview table with every public export, development instructions, and a Mermaid architecture diagram.

the mermaid diagram is a mindmap and it renders SO PRETTY on GitHub. multicolored branches for Core DSP, Languages, Synth, Voices, and IO. each branch fans out into its components.
it’s AMAZINGGGGGGGGGGG
I spent way too long picking which mermaid diagram type to use and I regret nothing.

the docs site

ten markdown pages built with DocMD. custom Ruby theme with blue/cyan gradient accents, dark mode support, semantic search via HuggingFace transformers, and a proper navigation structure:

getting started: quickstart (5-step guide with code), installation (npm/CDN/source), core concepts (source-filter model explanation with a Mermaid flowchart showing glottal source -> formant cascade -> output)

guide: architecture (module layout, data flow diagram, design decisions), renderer (6-step pipeline walkthrough from lyric to audio), languages (built-in language docs + how to register custom languages + cross-language aliasing)

API reference: full export table, DSP primitives with code examples, VoiceConfig reference (264 lines of parameter documentation with perceptual effect tables for every single field. “openQuotient 0.3 = pressed tense voice, 0.5 = neutral modal voice, 0.7 = breathy soft voice.” that kind of detail.), import/export format support table

the VoiceConfig page alone is probably the most thorough documentation of a formant synthesizer’s parameter space I’ve ever seen in a JS library. I might be biased. I’ve not seen many others. We don’t talk about that.

also

added UST and TSSLN to the export format list (they were import-only before). fixed the export function to handle generators that return arrays instead of single Uint8Arrays. CI workflows now build and audit the docs workspace too.

the TODO is almost empty. just “README: quickstart, API overview, architecture diagram” which is now done. the project has: a synthesizer, three languages, 14+ import formats, 14+ export formats, a demo app with a piano roll, voice presets, undo/redo, settings, file import/export, WAV export, tempo control, buffer management, 160+ tests, JSDoc on everything, a full documentation site, and CI that’s green.

six days :O

the documentation commit (part 1)

JSDoc everywhere

this is the kind of commit where the line count looks impressive and the actual behavior changes are zero. 118 lines became 216 lines in types.ts alone. every interface, every field, every function got documented.

some highlights:

(don’t ask why i did a double dash. i was being silly and just noticed somehow idk how)

• openQuotient: “fraction of the glottal cycle the vocal folds are open. Higher values produce a breathier, softer sound (0.2–0.9)”
• speedQuotient: “ratio of the opening phase to the closing phase. Higher values give a faster closure, increasing brightness (0.3–3.0)”
• FormantCascade: “cascade of anti-resonators followed by resonators, modelling the vocal-tract transfer function. Anti-resonators (zeros) come first, followed by resonators (poles), matching the source-filter model: S(f) = G(f) * Z(f) / P(f)”
• LFGlottalSource: “Liljencrants-Fant (LF) glottal pulse source with jitter, shimmer, aspiration noise, and DC blocking. See Fant (1986), Liljencrants (1985)”
• renderNote: documents the full pipeline: “lyric -> phoneme symbols -> phoneme lookup -> LF glottal pulse -> formant cascade -> noise mix -> amplitude envelope -> normalisation”

every DSP class, every language module, every player method, every import/export function. the person reading this code in six months (probably me) will thank me.

docs soon

Docs next time, I will not be long heheeee

TODO cleanup

the languages section is fully checked off now. English CMUDict (124,911 words), Japanese EDICT2 (224,808 kanji-to-kana entries), Mandarin with full pinyin decomposition, and phoneme alias maps for cross-language compatibility. all done. all in previous commits I apparently forgot to devlog. oops.

reorganized the remaining items: Documentation (JSDoc done, README and docs site still needed) and Future (scroll-to-playhead, seek, loop, Web Worker rendering).

the TODO is getting short. that’s a weird feeling for a project that’s six days old.

567,211 lines of “code”

so about that line count. I added the CMU Pronouncing Dictionary as a git submodule. 134,000+ English words with their ARPABET pronunciations. the cmudict repo is… large. like, genuinely enormous. the diff says 567k lines and it’s almost entirely a text file of words and their phoneme sequences.

the actual code changes across these four commits? probably 280 lines. the submodule? 567,131 lines. software engineering.

the real G2P

Build/data/cmudict is now a submodule pointing to cmusphinx/cmudict. this is THE pronunciation dictionary. the one that every TTS and speech recognition system uses. “BEAUTIFUL” -> “B Y UW1 T AH0 F AH0 L”. “SQUIRREL” -> “S K W ER1 AH0 L”. every word, every pronunciation variant, every stress marker.

the 100-word hand-rolled dictionary was cute. this is the real thing. (chunk size warning in vite went from 500 to 13000 because bundling the dictionary is… yeah.)

glottal pulse fix

the LF glottal source had an audible click on every single cycle. the return phase (after the glottis closes) was a bare exponential decay: -exp(-epsilon * x * period). the problem: it starts at -1 with non-zero slope, so at the boundary between cycles there’s a discontinuity. click. every. cycle.

fix: multiply by sin(pi * x). the sine term makes the slope match the open phase at t=te (C1 continuity) and reach 0 at t=tc. smooth transition. no click. this is how the actual LF model paper describes it but I’d skipped the sine modulation the first time around because I thought it was just cosmetic. it was not cosmetic.

also killed the 0.8/0.2 smoothing filter. it was supposed to prevent clicks but was actually just low-passing the glottal pulse and adding phase distortion. the DC blocker handles drift on its own. coefficient went from 0.995 to 0.99 for faster response.

renderer performance

formant coefficient updates were happening every sample. that’s setFormants() with trig functions 44,100 times per second. formants don’t actually change that fast. throttled to every 5ms (221 samples at 44.1k). no audible difference, meaningful CPU savings.

cascade reset came back at phoneme boundaries. I removed it two commits ago thinking filter state carryover was smoother. it is smoother, but it also causes wideband clicks when the coefficients jump. the reset is the right call.

aspiration lowpass

aspiration noise on vowels was full-spectrum white noise. real breathy voice has most of its noise energy below 4kHz. added a one-pole lowpass at 4kHz on the aspiration path. sounds warmer and less harsh. also re-added parallel filter resets at phoneme boundaries (matched the cascade reset fix).

signal quality tests

new test category: signal quality. a long 8-beat note must stay in [-1, 1], be finite throughout, and not be dominated by white noise. the “not white noise” check uses a crude HF proxy: ratio of RMS(sample-to-sample difference) to RMS(signal) in the steady state portion. pure white noise scores ~1.4, a clean tonal signal scores ~0.2. test asserts between 0.05 and 0.8.

wrote these tests because the glottal fix could have easily introduced NaN propagation or DC drift and I wanted to catch it automatically.

four commits, and failure

I did not intend to do all of this in one sitting. and yet.

player batching + buffer pool

the player was creating a new AudioBuffer for every single chunk. for a 100-note score that’s 200 buffer allocations (stereo) plus 200 BufferSourceNodes. the garbage collector was not happy.

rewrote the scheduling layer. chunks now accumulate into batches (~0.5s each) and get merged into a single AudioBuffer per channel before scheduling. acquireBuffer() checks a pool of previously used buffers by sampleRate and length before allocating new ones. onended releases them back to the pool. pre-buffer threshold bumped to 1.0s for more runway.

the result: way fewer Web Audio nodes, way less GC pressure, smoother playback on longer scores.

japanese UTAU compatibility

imported UST files from real UTAU projects were breaking because Japanese has a bunch of special lyric conventions I didn’t handle:

• っ (sokuon): geminate consonant marker. maps to a new sil (silence) phoneme.
• ※ (pronunciation alias): text after ※ is the actual pronunciation. あ※ka uses “ka” not “あ”.
• dot-prefixed lyrics (.sil, .S): UTAU rest/silence commands. treated as empty.
• ＠ and ％: stripped. ＠ also acts as a note-join marker in the renderer: if present, the initial consonant gets suppressed so the vowel carries through from the previous note.

also removed the cascade.reset() between phonemes. filter state now carries through for smoother formant transitions instead of clicking at every phoneme boundary.

settings panel

new SettingsPanel.svelte. modal overlay, dark theme, closes on Escape or click-outside. currently has one setting: auto-scroll toggle (for a future scroll-to-playhead feature). settings gear icon in the header bar.

undo/redo

integrated undora for undo/redo history. 50-state capacity. snapshots are saved on mouse-up in the piano roll, delete key, pitch point changes, and lyric/note-number edits (on blur, not on every keystroke).

Ctrl/Cmd+Z to undo. Ctrl/Cmd+Shift+Z or Ctrl/Cmd+Y to redo. Undo2 and Redo2 icons in the header bar, disabled when there’s nothing to undo/redo. history clears on language switch and file import.

the formatting commit

ran prettier again. removed a dead MIN_POOL_SIZE constant and an unused import. cleaned up a few TODO items. the usual.

scrollToNotes also got a fix: it was centering on the midpoint of all notes which looked weird when the first note started late. now it clamps to the start of the earliest note so you always see the beginning.

also I tried to fix the hissing and static and clicking but to no avail. i tried fixing a LOT of stuff. but. sadly ir only made things worse, so I had to retry. and retry. and retry. and retry. with no luck

I’m not sure what the problem is tbh. Let me use Satori to debug some stuff ig

making it sound less terrible (two commits, one mission)

these two commits are about one thing: the output was robotic and buzzy and I was tired of it. every change here is about making the synthesizer sound more like a voice and less like a modem.

the renderer got smarter

cross-note co-articulation. renderNote() now returns { chunk, finalFormants }. the stream passes the previous note’s final formants into the next note’s renderer, and the first phoneme interpolates from those formants instead of jumping cold. gaps between notes reset the chain. notes that follow each other seamlessly now blend their formants across the boundary.

per-phoneme envelopes. the old global 5ms attack / 10ms release is gone. replaced with getPhonemeEnvelopeSamples() which gives each phoneme type its own envelope: plosives get 2ms attack / 15ms decay (sharp burst), consonants and vowels get 5ms / 3ms. every phoneme segment fades independently.

diphthong formant sweeping. PhonemeDef got an endFormants field. if a diphthong has both formants and endFormants, the renderer sweeps between them over the phoneme duration. AY now actually glides from /aa/ to /ih/. EY glides from /eh/ to /ih/. OW glides from /oh/ to /uh/. they sound like diphthongs now instead of static vowels.

vibratoOverride. was defined on Note but never read. now it is. per-note vibrato control works.

everything got retuned

glottal source. added shimmer (per-cycle amplitude variation driven by jitter, so the volume wobbles slightly like a real voice). aspiration noise is now high-pass filtered (subtract a lowpass from the raw noise) so it’s airy instead of muddy. aspiration gain bumped from 0.1 to 0.15.

plosive bursts. noise envelope for plosives changed from symmetric fade to a fast 12ms exponential decay. “pa” now sounds like a burst instead of a pop.

formant data for everything. Z, ZH, V, DH, Y, W, HH, JH in English all got formant targets. same for z, h, y, w, j in Japanese. consonants that were previously just noise bursts now resonate through the vocal tract. the difference is huge.

vowel bandwidths tightened. defaults went from 80/100/120 to 70/90/130 Hz. narrower bandwidths = sharper resonant peaks = more vowel-like quality.

voice presets retuned. male voice: lower open quotient (0.4), lower speed quotient (0.65), higher tenseness (0.65), less aspiration (0.05). sounds less breathy, more chest voice. female: formant scale 1.18. gender slider in scaleVoice now affects speed quotient and has a gender-dependent tenseness base.

pitch accent. Japanese got resolveAccents() implementing heiban pattern (low first mora, high rest). the stream groups consecutive notes into phrases, calls resolveAccents, and applies the offsets as constant pitch shifts. it’s basic but it makes Japanese phrases have some melodic contour beyond what the score provides.

four TODO items checked off in one go: co-articulation, phoneme envelopes, diphthong sweeping, vibratoOverride. pitch accent too.

it still doesn’t sound human. but it’s starting to sound like it’s trying. and that’s a big step from where it was.

if you can identify the song in the editor image, good job, you’re cool :D

CI arc (three commits, one story)

three commits that are really one story: getting CI from “permanently red” to green.

the setup

“wrote” (aka copied and modified) six GitHub Actions workflows in one go:

ci.yml: lint + test + build on push/PR. matrix tests Node 20 and 22. builds the library first, then typechecks the demo. (docs build is commented out because docs don’t exist yet. they will. eventually.)

test.yml: dedicated test runner. same Node 20/22 matrix. runs jest in the Build workspace.

release-npm.yml: publishes to npm on GitHub release. strips private/scripts/devDependencies from package.json, copies README and LICENSE into Build/, publishes with --provenance. has a workflow_dispatch with a dry-run option so I can test without actually publishing.

security-audit.yml: runs npm audit --audit-level=high on all three workspaces (root, Build, Demo). daily cron plus on push when package files change.

static.yml: reworked the GitHub Pages deployment. it was pointed at Build/dist (wrong, that’s the library output). now it builds the library, builds the demo, and deploys Demo/dist as the pages root. docs will go in pages-root/docs/ when they exist.

single-file.yml: was still referencing Web-Template (the old template name). fixed to point at Demo, output file is now UTAUjsEditor.html.

also integrated Updato (my own auto-updater library) into the demo. on load it checks the current build hash against the latest commit on main and shows an update notification if there’s a newer version. the build hash gets injected at build time via __BUILD_HASH__ in the vite config.

cleaned up the TODO: removed all the completed checkboxes (they were cluttering the file), added detail to the remaining items.

the fixes

CI was 0/3 passing. then 1/8 passing. then 2/8. then eventually 8/8. the classic experience.

the single-file and updato workflows needed the library built before the demo (workspace dependency). added npm ci at root level and a “Build Library” step before the demo build. also added vite-plugin-singlefile and cross-env for the build:single script.

second fix commit added ts-node and unrun as dev deps because the ESM config loading was unhappy without them.

three commits to go from red to green. could be worse honestly but whatever

the prettier commit (and a tiny bugfix)

two commits. one has 19 lines of actual code. the other touched every single file in the project.

the bugfix

the piano roll’s resize handle wasn’t working for already-selected notes. you could resize on first click, but if you clicked a note to select it and THEN tried to drag the right edge, it would move the note instead of resizing. added a resize zone check that fires before the drag-to-move handler when a note is already selected. 19 lines.

the formatting pass

ran prettier on the entire codebase. every file. the diff is enormous and the actual logic changes are: zero.

added .prettierrc (semicolons, double quotes, trailing commas, 140 char width, svelte plugin), .prettierignore, and eslint.config.ts. bumped eslint to 10.5 and typescript-eslint to 8.61. added jiti for ESM config loading.

the one real improvement buried in here: replaced the Function type in ufdata.ts with a proper ParseFn type alias. eslint was right to yell at me for using bare Function. everything else is semicolons and line breaks.

the codebase has a consistent style now. that’s the whole commit. sometimes you just gotta.

open any vocal synth file ever made

so you know how the TODO said “MIDI file import” as one little checkbox? I may have slightly exceeded scope on that one.

UTAU.js can now import UST, USTX, VPR, VSQX, VSQ, SVP, MIDI, MusicXML, PPSF, S5P, TSSLN, CCS, DV, and UFData files. that’s UTAU, OpenUTAU, Vocaloid, Synthesizer V, Piapro Studio, CeVIO, DeepVocal, and standard MIDI. basically every vocal synth format that exists.

this is thanks to utaformatix-ts by sevenc-nanashi, which is a universal parser for vocal synth project files. it converts everything into a common UfData format. I wrote a 145-line adapter (ufdata.ts) that converts UfData into UTAU.js Scores. lazy-loaded so the parser only gets pulled in when you actually import a file.

pitch bends

this was the hard part. vocal synth files have pitch curves. track-level arrays of tick-value pairs that describe how the pitch deviates from the written note. the importer splits the track-level curve into per-note pitch bends, handling absolute-to-relative conversion, null value filtering, start/end padding, and extrapolation from preceding points.

the renderer now reads note.pitchBend and interpolates it per-sample alongside vibrato. the math: f0 = baseF0 * 2^((bendSemitones * 100 + vibratoCents) / 1200). pitch bends and vibrato stack correctly in cents space.

piano roll overhaul

the piano roll was hardcoded to 3 octaves (ik bad, but will fix soon) and a fixed width. now it covers the full MIDI range (0-127) with virtual scrolling. wheel scrolls vertically, shift+wheel or trackpad scrolls horizontally. viewport culling so only visible notes and key labels get drawn. auto-scrolls to center on notes when you import a file.

also: pitch curves render as yellow lines overlaid on note blocks. you can see the imported pitch data right there on the piano roll.

other stuff

added await setTimeout(0) in the streaming loop so the UI doesn’t freeze during long scores. the demo has an “Open” button that accepts all 14 supported file extensions. TODO got updated with a lot of checkboxes ticked.

Some amount of lines of import tests covering note mapping, tempo conversion, pitch splitting (absolute, relative, null filtering, extrapolation, out-of-range skipping, pitch:false opt-out), and edge cases.

uhhh tests (that was fast huh)

the commit message says it all. 961 lines added. 10 test files. plus a WAV encoder, a bunch of bug fixes, and the entire player got an upgrade. in one sitting.

tests

jest config, ts-jest, ESM mode. ten test files covering everything that exists:

• envelope.test.ts: attack/release shape, mixBuffers offset and gain, edge cases
• filter.test.ts: FormantFilter resonator, FormantCascade
• noise.test.ts: NoiseSource output
• oscillator.test.ts: LFGlottalSource waveform shape
• wav.test.ts: encodeWav RIFF header
• english.test.ts: lexicon hits and fallback
• japanese.test.ts: hiragana, romaji, special cases
• renderer.test.ts: renderNote output shape, sample count, non-zero output
• stream.test.ts: streamScore chunk boundaries, mixChunks
• voices/index.test.ts: buildVoice, scaleVoice parameter ranges

writing tests found bugs. writing tests always finds bugs.

the bugs tests found

OW was missing. the English phoneme table had every ARPABET vowel except OW. “HELLO” ends with OW. the demo word was literally broken and I didn’t notice because the fallback produced silence instead of crashing. added it. F1=470 F2=1000 F3=2400.

anti-resonator formula was wrong. the pole radius was hardcoded to 0.99 with a random 0.95 frequency offset. now it derives the pole from bw * 1.5 like a real anti-resonator should. nasals sound less terrible.

jitter was per-sample. it was recalculating a random f0 every single sample, which made the pitch wobble chaotically instead of naturally. moved it to recalculate once per glottal cycle. much more realistic.

noise fadeout could go negative. phDur - segPos - 1 can be negative at the boundary. clamped to 0.

gain normalization could divide by zero. clamped overallPeak to 1e-6 and gain to max 100.

buildVoice spread order was wrong. ...overrides was before the sub-objects, so the glottal/formant/vibrato defaults always overwrote user values. flipped the order.

WAV encoder

(i’ve written manual WAV encoders before, I just copy-pasted that, it’s not that bad tbh)

62 lines. encodeWav() takes AudioChunks and writes a proper RIFF/WAVE file. 16-bit PCM, little-endian, handles mono and stereo. float-to-int16 conversion with clamping. exported from the barrel file.

player upgrades

setVolume() API. volume parameter on play(). progress events actually fire now (the type existed but was never emitted). stop does a 50ms gain ramp to zero before closing the AudioContext so it doesn’t click. scheduling uses Math.max(ctx.currentTime + 0.01, ...) to prevent scheduling in the past if rendering falls behind.

stream tempo handling

streamScore() was using a single currentTempo for the whole note. now it has tempoAt() for point lookups and noteSampleDuration() that integrates across tempo changes within a note. a note that spans a tempo change gets the right duration now.

also added a cascade reset between phonemes in the renderer (was carrying filter state across phoneme boundaries causing ringing), added a missing phoneme console.warn so you can actually debug G2P failures, and fixed the Japanese romaji parser to skip spaces instead of treating them as unknown consonants.

the TODO list is getting shorter. slowly but surely.
…and i forgot to update it oops

it makes sound now

okay so. I may have blacked out and written an entire synthesizer in one commit. 210 lines of renderer, 65 lines of streaming, a full Svelte demo app with a piano roll, and a dozen DSP fixes. this is flo-era hyperfocus energy except I can HEAR it this time. (well I could hear flo, it’s a audio format, so like, duh, but yk what i mean loll)

the renderer

renderNote() takes a Note + VoiceConfig + LanguageModule and produces actual audio. per-sample processing: vibrato with attack ramp, 30ms smoothstep formant interpolation between phonemes, glottal pulse through the cascade, shaped noise for consonants with per-segment fade in/out, smoothstep attack/release envelope, peak normalization. consonants get their default duration, vowels split the remaining time. if consonants would eat more than 40% of the note they get compressed.

every filter, oscillator, and cascade got refactored from batch to per-sample. slower but I can morph formants sample-by-sample for smooth transitions.

DSP fixes (there were several)

the LF oscillator’s open phase was inverted. added a DC blocking filter because the pulse was making everything drift. added jitter for natural-sounding pitch variation.

the resonator gain formula was wrong (b0 = 1 - r*r should be b0 = 1 - B - C). FormantCascade now runs anti-resonators before resonators (correct order for nasals). added setPassthrough() for unused filter slots.

female formant scale went from 0.88 to 1.15. I had it backwards. scaling down shrinks the tract and sounds childlike. scaling up is what you want.

streaming + demo

streamScore() is an async generator. walks the score note by note, handles tempo changes, yields chunks. player starts playing before the score finishes rendering.

the demo is a full Svelte 5 app. canvas piano roll (click to create, drag to move/resize, delete to remove), voice panel with easy sliders + expandable advanced params, transport bar, language switcher. Japanese demo says “ka na ta shi i ne”. English says “HELLO WORLD THIS IS A TEST”.

press play and it synthesizes through Web Audio in real time. from math.

everything else

Japanese plosives got formant data (were noise-only). added “l” as an r-alias for loanwords. barrel file exports the full public API. wrote a comprehensive TODO.md because the list of things that aren’t done is very long.

it sounds terrible. robotic and buzzy and the consonants are more like clicks. but it’s SOUND. generated from MATH. in a BROWSER. Peterson and Barney would be proud. (or horrified.)

voices, a player, and oh look a demo app

three things in one commit because I couldn’t decide which to work on so I did all of them.

voice presets

two built-in voices: Male and Female.

the male voice has a lower open quotient (0.45, vocal folds close faster), lower aspiration (0.08, less breathy), and formant scale of 1.0. the female voice has higher open quotient (0.55), more aspiration (0.12), and formant scale of 0.88 which shifts all the formant frequencies up to simulate a shorter vocal tract.

vibrato differs too. male is 5.5 Hz rate with 30 cents depth. female is 6.0 Hz with 40 cents. these are rough averages from the singing voice literature. real vibrato varies wildly between singers but you have to start somewhere.

buildVoice() lets you construct a custom voice with partial overrides. scaleVoice() is the fun one. it takes a voice config and five intuitive sliders: gender (-1 to +1), breathiness, tension, brightness, vibratoAmount. the gender parameter interpolates formant scale between 0.88 and 1.0. breathiness maps to open quotient and aspiration. tension maps to glottal tenseness. brightness maps to formant bandwidth (narrower bandwidths = brighter, more resonant sound). vibrato amount scales depth.

the idea is you start with a preset and then tweak it with human-readable parameters instead of raw acoustic values. “make this voice breathier” is easier to think about than “increase open quotient to 0.55 and aspiration to 0.12”.

registry pattern same as languages. getVoice("male"), registerVoice("my-voice", config).

stream player

this is the thing that actually makes sound come out of your speakers.

StreamPlayer takes an AsyncGenerator<AudioChunk> and schedules the audio through the Web Audio API. each chunk gets turned into an AudioBuffer, wired through a GainNode (fixed at 0.8 for now), and scheduled at its exact start time using ctx.currentTime + chunk.startSample / sampleRate. the generator can yield chunks as fast or as slow as it wants. the player just keeps scheduling them.

async generator as the interface is the key design decision. the synthesizer can stream audio chunk by chunk as it renders each phoneme, and the player starts playing before the whole score is done. no waiting for the full render. just start.

pause suspends the AudioContext. resume resumes it. stop aborts the generator via AbortController and closes the context. event system emits stateChange, progress, done, error. on() returns an unsubscribe function. clean lifecycle.

the demo app

svelte 5 + vite. the Demo/ workspace finally has code in it. just scaffolding for now, no UI components yet. but the package.json is wired up: utaujs as a workspace dependency so it pulls from the Build/ output, @sveltejs/vite-plugin-svelte, vite 8.

I picked svelte because it’s the lightest framework that still gives me reactivity and components without a virtual DOM. for a music app where audio timing matters, I don’t want React’s reconciliation cycle anywhere near my render loop. svelte compiles to vanilla JS. no runtime overhead. (also I just like svelte.)

the engine is almost wirable end to end now. language module produces phonemes, voice config provides the acoustic parameters, the DSP layer renders audio chunks, the stream player schedules them through Web Audio. the only missing piece is the actual synthesizer that takes a Score + Voice + Language and yields AudioChunks. that’s next.

getting close to hearing actual sound.

two languages walk into a synthesizer…

so I said “next step is a basic ARPABET phoneme dictionary” and then I just… did both English AND Japanese in one sitting. because apparently my brain doesn’t know how to do things incrementally.

english

every phoneme in ARPABET, with real formant data from real papers. I’m going to be responsible and cite my sources (gasp):

vowel formants are Peterson & Barney 1952 male speaker means. the classic dataset. 76 speakers, 10 monophthongal vowels. /IY/ is F1=270 F2=2290 F3=3010. /AA/ is F1=730 F2=1090 F3=2440. these numbers are from a 74-year-old paper and they’re still the standard reference. wild.

consonant noise centres follow Jongman et al. 2000 for fricatives (sibilant spectral peaks), Stevens 1998 for plosive burst loci, and Fujimura 1962 for nasal formants/antiformants. I feel like an actual phonetician typing these citations. I am not an actual phonetician.

the full set: 10 vowels, 4 diphthongs, 6 plosives, 9 fricatives, 3 nasals, 4 approximants, 2 affricates. every consonant has its noise shaping config. every nasal has antiformant data. every vowel has 5 formant targets (F1 through F5).

there’s also a grapheme-to-phoneme dictionary with like 100 common English words. “HELLO” -> [“HH”, “EH”, “L”, “OW”]. “BEAUTIFUL” -> [“B”, “Y”, “UW”, “T”, “IH”, “F”, “UH”, “L”]. it’s a smol subset but it covers the words you’d actually want a singing synthesizer to say. sun, moon, star, dream, love, forever, together. very anime opening core vocabulary. I should expand it eventually but it’s fine for testing.

the fallback for unknown words is fun: first it checks if the input is already ARPABET symbols separated by spaces/underscores. if not, it falls back to a dead simple single-character mapping where each letter gets one phoneme. it’s terrible but it won’t crash.

japanese

honestly? mapping Japanese to phonemes is SO much easier than English. kana are basically a syllabary. each character maps to exactly one consonant-vowel pair (or just a vowel). no ambiguity. no “through” being pronounced nothing like it looks. English is a disaster and Japanese is a joy.

(I still don’t know Japanese. sighhh. but the internet is very helpful.)

vowel formants are from Yazawa & Kondo 2019, specifically the short-vowel midpoint averages for male speakers from their ICPhS paper. that Japanese vowel formant displacement paper I was reading earlier today. /a/ F1=687 F2=1283, /i/ F1=301 F2=2154, etc. F3 values come from Kitamura et al. 2009, the ATR MRI vocal tract study. different paper, different research group, but the F3 data fills a gap that Yazawa didn’t cover in detail.

the lyric parser handles: raw romaji (“ka”, “shi”, “tsu”), hiragana (あ, き, しゃ), and compound kana (きゃ, しゅ, ちょ). hiragana gets converted to romaji via a lookup table, then romaji gets split into consonant-vowel pairs via romajiToPhonemes(). special cases for し -> “shi”, ち -> “chi”, つ -> “tsu”, ふ -> “fu”. word-final ん becomes the moraic nasal N. geminate consonants (double letters) get handled. it’s not perfect but it covers standard Hepburn romanisation.

the registry

a Map of language IDs to modules. getLanguage("en") or getLanguage("jp"). registerLanguage() for future additions. both “jp” and “ja” point to Japanese because people use both and I’m not going to pick a side.

the LanguageModule interface from types.ts is earning its keep already. both languages implement the same lyricToPhonemes() contract. the synthesizer won’t know or care which language is active. plug in English, plug in Japanese, plug in anything. the architecture handles it.

I love this.

new project. yes, another one.

(penumbra is on pause btw)
so uh. hi. I’m starting a new thing.

penumbra is on pause. not abandoned, just… paused. the backend is solid, the architecture is clean, and the UI needs to catch up but I need to do the HTML mockup thing first and I’m not in the headspace for that right now. it’ll come back. I promise.

in the meantime: UTAU.js.

the idea: a browser-based singing synthesizer inspired by UTAU. but here’s the thing. it’s not sample-based like the original. it’s parametric. formant synthesis. generate the entire voice from math. no voicebank files, no platform dependencies, no 200MB sample libraries. just DSP and physics and the human vocal tract modeled in TypeScript.

why TypeScript and not Rust this time? because.

I scaffolded the repo from my Web-Template and then immediately ripped out everything that made it a web template.
I replaced it with a proper library setup: tsdown for bundling (ESM + CJS + type declarations), npm workspaces (Build for the engine, Demo for a future demo app), TypeScript strict mode.

renamed to utaujs. added jest, eslint, prettier, typescript-eslint. the foundation is there.

also brought in @nisoku/satori as a dependency because I’ll want observability later and I might as well wire it up now. And like, it’s such a great observatory library like smh my head, why wouldn’t I use it

the research rabbit hole (aka I read way too many papers)

okay so today was one of those days where hackatime probably says like less than a hour but reality says 4-5.

I spent most of half of today reading. papers. reference tables. phonetics Wikipedia articles. I have 20 browser tabs open right now (thank goodness for tab groups) and they’re all about formants and audio and human vocal behaviors.

here’s the reading list:

• Peterson & Barney 1952 (a classic formant frequency dataset, 76 speakers, 10 American English vowels, F0 through F3)
• the Stanford CCRMA formant table (Peterson’s data averaged by gender and age group, the numbers everyone cites)
• ARPABET (the phonetic notation system CMU uses, 39 phonemes for General American English)
• CMU Pronouncing Dictionary (134,000+ words mapped to ARPABET, sheesh)
• a paper on Japanese vowel formant displacement (short vs long vowels have different formant targets, which matters a lot)
• Kitamura et al on vocal tract transfer functions from MRI-derived solid models (they literally 3D printed vocal tracts and measured the acoustic response)

the MRI one is wild. they took volumetric MRI scans of people saying Japanese vowels, built physical 3D models of their vocal tracts via stereolithography, and then measured the frequency response by pumping sound through the models.

but I also wrote code

I remember a LOT of this from working on flo.

types.ts is the big one. 91 lines of the full type stuff

oscillator.ts is the glottal source. an LF (Liljencrants-Fant) model. this is the buzzing sound your vocal folds make before your throat and mouth shape it into speech. lots of math that i don’t want to talk about.

filter.ts is the formant cascade. second-order IIR resonator (biquad) that can operate as either a resonator or anti-resonator.

noise.ts is simple. white noise generator plus a function to shape it through formant resonators. this is how you get consonants like “s” and “sh”. they’re just filtered noise.

envelope.ts is attack/release shaping with smoothstep curves plus a buffer mixing utility.

Fun!