I focused on building the first stage of the web app by coding the first way discover legal document links – scanning page footers using Requests and BeautifulSoup, and filtering them against a list of keywords such as “privacy”, “terms”, and “cookies” and removing duplicates.
I also began implementing a second discovery method using robots.txt, which can now be loaded.
This didn’t take me 19 minutes – over 2 hours, in fact – but I only discovered Stardance later.
