Builded entire job

I built the entire job that retrieves data from Open Library. The biggest constraint was that it had to run on my Raspberry Pis; however, the compressed Open Library dump is 12 GB (expanding to nearly 50 GB), while I had barely 10 GB of free space. Consequently, I had to stream the download of the compressed file, decompress it on the fly, parse the content, and output it into CSV batches of up to 500 MB. Then, leveraging the power of PostgreSQL, I was able to rapidly load these batches sequentially into a staging table without running into constraints. I subsequently deduplicated the data using a second table with a UNIQUE constraint on the ISBN, followed by the final table—which also featured an ISBN index—enabling searches in under 100 ms. My connection wasn’t very reliable, and I faced major issues with the download constantly dropping, so I had to rig up a system to maintain the stream despite these interruptions.
I also created the GitLab pipelines that build a versioned Docker image (using semantic versioning) of the application.