I added a bunch of stuff to my app!
To explain simply:
-> I scrape the content off all the links using Beautiful Soup
-> I push this content into AI to be summarised using Hack Club’s AI API
-> Summarised terms and conditions are returned!
This means I can work on my frontend now yayyyy!!!
Added scanning of robots.txt and sitemap.xml
CHECKPOINTTT
I finished building the first stage of the web app by coding the second and third ways way discover legal document links – scanning robots.txt and sitemap.xml using Requests, Python and BeautifulSoup, and filtering them against a list of keywords such as “privacy”, “terms”, and “cookies” and removing duplicates. Now I have three ways of finding links, and can move onto actually reading what’s inside them!
Yay :)
Adding Scraping Robots.txt Functionality
To get a more comprehensive link check, I made it so that my link searcher also scraped robots.txt of a domain, where more potential links may be lurking (hehehe).
This is important because it will help me find more terms to scan, making my tool better!
This was done in a similar process to the scraping footers function. However, I used response/request to get the robots.txt file instead of scanning as page with BeautifulSoup.
Just as before, I scanned the links and checked for duplicates before adding to the main array.
Yay!
Below, you can see it in action (I combined the two functions).
I focused on building the first stage of the web app by coding the first way discover legal document links – scanning page footers using Requests and BeautifulSoup, and filtering them against a list of keywords such as “privacy”, “terms”, and “cookies” and removing duplicates.
I also began implementing a second discovery method using robots.txt, which can now be loaded.
This didn’t take me 19 minutes – over 2 hours, in fact – but I only discovered Stardance later.
