I watched a Kotlin talk about Flows and I was inspired to rewrite my program using them.

This made my program more efficient, idiomatic and also helped me fix the annoying progress bar refresh bug (multiple old bars shown). It also let me actually properly sort by position across all 4 status types (added, removed, moved, errored).

I also made the downloading functions use one underlying common function to clean up the codebase. I also removed sqlx4k because it wasn’t actually parallel and was causing issues with init.

Here, in the screenshot I am showing the new elegant code for the logging of track statuses. It used to be many individual async calls, but flows allow me to group them up and sort them.

I now detect invalid playlist ids and print out an error telling the user how to get a valid playlist id.

I made position diffing aware of insertions and deletions, so that output is less bloated with useless move info such as 0->1, 1->2, etc.

I also did some optimization by using database transactions wherever possible. I can’t do it in the main sync body because it is bounded to its own coroutine scope.

I added duration tracking to the database. The M3U generation now doesn’t hardcode 0 as the duration.

I made it so that thumbnails should now be even higher quality, it does take more storage but the difference is quite noticeable on some music players.

Also small hotfix, if you accidentally pass in another unrelated SQLite database it will now not run migrations on it before erroring.

To help users migrate from the old Rust implementation, I made it so that a sync will re-tag the track if the track’s thumbnail does not exist on disk.

The Kotlin implementation always gets higher resolution images, so this allows for much higher resolution track covers.

In my attempt to make the streamExtractor lazily loaded, I caused a deadlock. I have fixed the deadlock and this fixed version is at v0.2.7.

The fix was pretty simple, it was to not use CoroutineStart.Lazy because a lazy coroutine may never run, meaning the parent coroutineScope will hang forever.

I also switched the backend driver to sqlx4k as it provides a connection pool but more importantly makes the suspend code flow cleaner.

I wont switch to Exposed (the official SQL driver) because it doesn’t have the same parallel capabilities as sql4k.

I forgot to add error handling from the original Rust project, so I did that.

I also fixed the parallelism by using a Semaphore as limitedParallelism still wakes all the coroutines. Using a Semaphore only allows N permits to be handed out at once. This means that only N tasks would be running at once, given that each task takes one permit. I set N to 8 as that should not cause any throttling but still allow multiple tracks to be downloaded at once.

I also implemented better lazy fetching than the old Rust implementation. Now only pieces that must be fetched are fetched, and if they don’t need to be fetched, a StreamExtractor is never fetched. This causes a huge speed up, allowing my ~400 track playlist to be incrementally synced in 8 seconds.

Other notable quality of life features (taken from the git log):

- refactor: make output use terminal.theme.*
- refactor: make output more human readable by using title instead of youtube_playlist_id
- fix: reorder items in subtasks so that the text doesn't move everything else

Shortly after I shipped, I found a better library for progress bars and displaying to the terminal in general.

I also made sure that the exists locally log line would be printed out in order.

I decided to use it, and I am really happy with the result!!!!

Preparing to ship the project, so here’s a list of everything I did:

• added back an edited readme
• added back the apache 2.0 license
• fixed the github actions auto release
• recorded an asciinema cast and put it in the readme
• made the cli say music-player-kotlin instead of cli
• …other small changes to make sure its ready for release

added a progress bar and actually actually fixed the deadlock this time

I fixed position tracking, so that moved tracks will output a line telling you they have moved.

I attempted to fix a deadlock that was happening, and fixed some database queries targetting the wrong table.

I cleaned up the code, bounded parallelism to 8 with limitedParallelism, and tried to optimize the detection of moved tracks

the rewrite is almost finished

thumbnails are now way higher in resolution because we can now get youtube’s webp images. unfortunately jaudiotagger does not support webp images, so i force all images from youtube to turn into a lossless png

kotlin is pretty cool, especially coroutines, its like lua’s coroutines but actually backed by a thread pool

i need to limit the concurrency but otherwise this is great! very fast even on a debug build

what’s left:

• currently unlimited concurrency, this will definitely crash the jvm on large playlists
• better terminal output with colors and a diff generator
• use a temporary table in sqlite to make diffing more memory efficient
• packaging and ci should be fine since “java runs anywhere”

i begun porting the project to kotlin

while i love rust i also want this project to actually work

rustypipe will always be a bit behind youtube’s protections so i decided to just rewrite the whole thing in kotlin so i can use newpipeextractor directly

what’s left:

• tag audio with jaudiotagger
• use clikt to make a nice command line interface
• (maybe) add unit tests

there is no actual screenshot because i didn’t actually implement the commands yet

i got rustypipe-botguard working with Nix

i still can’t figure out why some videos don’t work sometimes but other videos work all the time

i updated the test suite to reflect that

i finally did the ci pipeline for windows, linux, and macos

i wrote a readme but i’ll need to add an asciinema cast to it to make it look better and allow me to ship the project

thumbnails were really low quality, this change allows for higher quality thumbnails. one example would be 60x60 -> 120x120 still low quality but a higher low quality.

other notable changes:

• unification of all track identifiers into one
• refetch thumbnails if not existing & do not fetch thumbnail if its {id}.{png,jpeg,jpg} exist
• m3a -> m3u, it is more compatible with music players (this happened in 4c337f491dd949fdc3f1f5d51363b0342defaac1 on aug 15, i forgot to log it)

next thing to do is packaging . this is my worst nightmare so the whole program will be statically linked

progress bars with tracing-indicatif

might have to increase tick rate, it flickers a bunch on release mode

tracks that are only available as m4a’s can now be downloaded ffmpeg won’t be spawned to remux

screenshot is forcefully required so heres the main fix

/tmp is now only hardcoded in the singular unit test

errors are more friendly, m3a generation has been refitted to be m3u generation so that more music players are supported, fixed a bug where songs would be downloaded with audio/{id}..m4a which is an invalid path, and updated the user agent to be a firefox extended support release so i dont have to update it more

downloads were broken by the rustypipe switch, so i fixed them by forcing the AndroidVr client. some tracks are weird and still don’t work, but most do

i fixed the nix build

this should allow me to cross compile to windows (maybe with mingw, and after i don’t hardcode /tmp), mac, and linux

i switched from kopuz-server to rustypipe

it is much faster and more accurate which may allow me to bypass youtube music’s weird quirk of some tracks only being playable directly on youtube

the playlist is now synced in 3.2 seconds, and not 5

i made tracks that cause errors to show up in the diff form

i added the base diffing feature i wanted

i still need to make it filter out tracks that are unfetchable that are currently marked as “removed”

and i need to rename the project to music-syncer

i added configurable concurrency

it doesn’t seem to improve download or sync times

but i think now it has to be stdio performance thats holding it back, logging is not free

i shrunk the number of crates being built from 700-800 to 500 by stripping my fork of kopuz-server of all bloat

it depended on lots of crates and most importantly it doesn’t rely on rusqlite 0.31 and sqlx 0.8

the screenshot shows how much smaller the build is now

parallelism!!!

i was able to download my whole playlist in ~30 seconds (with unbounded parallelism that lagged the rest of my laptop)

next steps:

• maybe change parallelism from hardcoded 16 to num_cpus::get() or ask tokio how many threads we have
• use ratatui and display the status of each song being downloaded, since we’re using tracing we can use https://github.com/gin66/tui-logger
• use color_eyre https://ratatui.rs/recipes/apps/color-eyre/
• maybe add login, some songs are age restricted (even if they have no lyrics)
• definitely add configurable parallelism; i think this software would rip through playlists if it had more sqlite connections and more threads

not really any screenshots to show anymore……. but i can try and show how many tracks were downloaded successfully and how long it took

the screenshot is bounded by 16 futures at a time, which is probably too small and the sqlite connection pool is capped at 5. 2 minutes is a lot. but on a good note incremental syncs only take ~5.7 seconds

i switched to sqlx and the code is much cleaner but now i have to hack around how sqlx builds offline with SQLX_OFFLINE=true cargo build but i only have to do it once then its cached

using sqlx is much more developer friendly and it taught me that a connection pool can harm the application because temp tables are only kept per connection not per entire database

since nothing really changed but the code became cleaner i will show another music player that can display the generated playlist

audio metadata

for this dev log you will see 2 hours for one command spawn and a small refactor. but what you have not seen was 2 hours fighting ffmpeg-next, ffmpeg-sys-next, bindgen, libclang

all that to say i decided to not link against ffmpeg, and i am much happier with the result

audio players get the title, artist, even cover art all from the track’s metadata

unfortunately, youtube music does not attach metadata to the tracks we scrape, so we need to attach metadata

attaching metadata to audio files is actually pretty simple thanks to the lofty crate unless the file is a webm. webm does not have any sort of audio metadata (?) so you will need to transcode webm into another audio format to put metadata into the file. i decided to just pick m4a because everything supports it.

lastly i did a small refactor because main.rs was pushing 600 lines of code, and i plan to:

• separate the db functions from the Playlist struct
• make the code truly asynchronous, which involves moving to sqlx
• pipeline everything

the end result is pretty metadata in Gapless (or any other modern music player)

I switched from directly printing to stderr to using tracing.

The m3a playlist generation works quite well, as you can see in the screenshot.

after a lot of thinking i decided to not make a music player and instead decided to instead make a really fast youtube music playlist syncer

the upstream library i was gonna use anyway does not expose YoutubeMusicClient, so i forked it and added a patch so i can use that

everything was working until i realized downloads were really slow and the fix was to add a Range header with a value of “bytes=0-” and now downloads are very fast. i suspect that this makes my code look like a real browser to youtube/google which lets me get the normal download speed.

i used rusqlite (sqlite rust wrapper) and each playlist is its own database file. i might switch to sqlx because it has compile time sql checking and has a very good async sqlite wrapper

everything should work except m3a playlist generation, there are no good m3a playlist libraries so i hand coded it myself and have not tested it yet; UPDATE: it works i tested it now

i plan on adding thumbnail deduplication, songs from the same album can have the same thumbnail

i added support for loading custom tinted-theming themes such as from https://github.com/tinted-theming/schemes

the screenshot is using ayu_dark: https://github.com/tinted-theming/schemes/blob/spec-0.11/base16/ayu-dark.yaml

i added the base code for base16 themes and the skeleton sidebar (no buttons or actual pages yet)

i plan for the sidebar to do something with navigation so like a playlists button and because i want the music player to be minimal it will probably become floating and will probably become a small menu icon that when clicked will expose the Playlists button

i will probably make some sort of abstract button wrapper since i still need to make the actual full screen player ui which will at least have 3 icons (pause/play, backwards, forwards)

the slow resizing issue was fixed by using gpui’s git repository and not crates.io

the next steps are to add custom theme support which means i will need to also locate a config folder; i plan on using https://codeberg.org/dirs/directories-rs; colors will probably be in yaml so that you can copy and paste tinted-theming schemes from https://github.com/tinted-theming/schemes other config settings will likely be in toml

oh also i will need to set up automatic binary releases for amd64 and arm64 on linux and macos with github actions, which should be easier because it already uses Nix

gpui at least with the crates.io version has very slow resizing so i might switch to iced

i have some ideas for how data will be stored:

• sqlite for database engine via sqlx or pure rust library (depends on which one allows to decode into structs)
• migrations can be done as a map to sql string (version -> appropriate migration)
• each playlist can be its own database & can store its own downloaded songs because fetching from db is faster: https://sqlite.org/fasterthanfs.html
• settings (theme) can be done in toml

i have also disabled bounds on the gpui window, so thats the screenshot

I set up the flake at first using rust-overlay but realized crane is probably better so I asked opencode to refactor it to use crane

The default gpui example now runs