Adding Scraping Robots.txt Functionality
To get a more comprehensive link check, I made it so that my link searcher also scraped robots.txt of a domain, where more potential links may be lurking (hehehe).
This is important because it will help me find more terms to scan, making my tool better!
This was done in a similar process to the scraping footers function. However, I used response/request to get the robots.txt file instead of scanning as page with BeautifulSoup.
Just as before, I scanned the links and checked for duplicates before adding to the main array.
Yay!
Below, you can see it in action (I combined the two functions).