@techs-sus

I watched a Kotlin talk about Flows and I was inspired to rewrite my program using them.

This made my program more efficient, idiomatic and also helped me fix the annoying progress bar refresh bug (multiple old bars shown). It also let me actually properly sort by position across all 4 status types (added, removed, moved, errored).

I also made the downloading functions use one underlying common function to clean up the codebase. I also removed sqlx4k because it wasn’t actually parallel and was causing issues with init.

Here, in the screenshot I am showing the new elegant code for the logging of track statuses. It used to be many individual async calls, but flows allow me to group them up and sort them.

I now detect invalid playlist ids and print out an error telling the user how to get a valid playlist id.

I made position diffing aware of insertions and deletions, so that output is less bloated with useless move info such as 0->1, 1->2, etc.

I also did some optimization by using database transactions wherever possible. I can’t do it in the main sync body because it is bounded to its own coroutine scope.

I added duration tracking to the database. The M3U generation now doesn’t hardcode 0 as the duration.

I made it so that thumbnails should now be even higher quality, it does take more storage but the difference is quite noticeable on some music players.

Also small hotfix, if you accidentally pass in another unrelated SQLite database it will now not run migrations on it before erroring.

To help users migrate from the old Rust implementation, I made it so that a sync will re-tag the track if the track’s thumbnail does not exist on disk.

The Kotlin implementation always gets higher resolution images, so this allows for much higher resolution track covers.

In my attempt to make the streamExtractor lazily loaded, I caused a deadlock. I have fixed the deadlock and this fixed version is at v0.2.7.

The fix was pretty simple, it was to not use CoroutineStart.Lazy because a lazy coroutine may never run, meaning the parent coroutineScope will hang forever.

I also switched the backend driver to sqlx4k as it provides a connection pool but more importantly makes the suspend code flow cleaner.

I wont switch to Exposed (the official SQL driver) because it doesn’t have the same parallel capabilities as sql4k.

I forgot to add error handling from the original Rust project, so I did that.

I also fixed the parallelism by using a Semaphore as limitedParallelism still wakes all the coroutines. Using a Semaphore only allows N permits to be handed out at once. This means that only N tasks would be running at once, given that each task takes one permit. I set N to 8 as that should not cause any throttling but still allow multiple tracks to be downloaded at once.

I also implemented better lazy fetching than the old Rust implementation. Now only pieces that must be fetched are fetched, and if they don’t need to be fetched, a StreamExtractor is never fetched. This causes a huge speed up, allowing my ~400 track playlist to be incrementally synced in 8 seconds.

Other notable quality of life features (taken from the git log):

- refactor: make output use terminal.theme.*
- refactor: make output more human readable by using title instead of youtube_playlist_id
- fix: reorder items in subtasks so that the text doesn't move everything else

Shortly after I shipped, I found a better library for progress bars and displaying to the terminal in general.

I also made sure that the exists locally log line would be printed out in order.

I decided to use it, and I am really happy with the result!!!!

Preparing to ship the project, so here’s a list of everything I did:

• added back an edited readme
• added back the apache 2.0 license
• fixed the github actions auto release
• recorded an asciinema cast and put it in the readme
• made the cli say music-player-kotlin instead of cli
• …other small changes to make sure its ready for release

added a progress bar and actually actually fixed the deadlock this time

I fixed position tracking, so that moved tracks will output a line telling you they have moved.

I attempted to fix a deadlock that was happening, and fixed some database queries targetting the wrong table.

I cleaned up the code, bounded parallelism to 8 with limitedParallelism, and tried to optimize the detection of moved tracks

the rewrite is almost finished

thumbnails are now way higher in resolution because we can now get youtube’s webp images. unfortunately jaudiotagger does not support webp images, so i force all images from youtube to turn into a lossless png

kotlin is pretty cool, especially coroutines, its like lua’s coroutines but actually backed by a thread pool

i need to limit the concurrency but otherwise this is great! very fast even on a debug build

what’s left:

• currently unlimited concurrency, this will definitely crash the jvm on large playlists
• better terminal output with colors and a diff generator
• use a temporary table in sqlite to make diffing more memory efficient
• packaging and ci should be fine since “java runs anywhere”

i begun porting the project to kotlin

while i love rust i also want this project to actually work

rustypipe will always be a bit behind youtube’s protections so i decided to just rewrite the whole thing in kotlin so i can use newpipeextractor directly

what’s left:

• tag audio with jaudiotagger
• use clikt to make a nice command line interface
• (maybe) add unit tests

there is no actual screenshot because i didn’t actually implement the commands yet

i got rustypipe-botguard working with Nix

i still can’t figure out why some videos don’t work sometimes but other videos work all the time

i updated the test suite to reflect that

i finally did the ci pipeline for windows, linux, and macos

i wrote a readme but i’ll need to add an asciinema cast to it to make it look better and allow me to ship the project

thumbnails were really low quality, this change allows for higher quality thumbnails. one example would be 60x60 -> 120x120 still low quality but a higher low quality.

other notable changes:

• unification of all track identifiers into one
• refetch thumbnails if not existing & do not fetch thumbnail if its {id}.{png,jpeg,jpg} exist
• m3a -> m3u, it is more compatible with music players (this happened in 4c337f491dd949fdc3f1f5d51363b0342defaac1 on aug 15, i forgot to log it)

next thing to do is packaging . this is my worst nightmare so the whole program will be statically linked

progress bars with tracing-indicatif

might have to increase tick rate, it flickers a bunch on release mode