alright, final bit of work for today. since the last log, i put some more logging messages and files to diagnose different issues. i caught one where sometimes goBILDA didn’t redirect through an HTTP code but rather through <meta http-equiv="refresh"> which was annoying. and some category pages had a link to other category pages given the SAME SET OF ATTRIBUTES AS PRODUCTS, or took from an entirely different place in the DOM because the product list is CLIENT SIDE RENDERED. i’m sorry but i am NOT about to pull out puppeteer for a (presumably) 1,000+ SKU website.

i also added a cache to fetched URLs so that i didn’t bombard goBILDA with duplicate requests as i worked. also on that note, i put in a custom user agent while i was at it so that they hopefully don’t block what i’m doing 😊

tomorrow i’ll also probably work on deduplicating DOM queries by keeping track of a list of scanned product pages and filtering out pages that are irrelevant for the catalog’s purposes.