pywikiscrape
- 1 Devlogs
- 5 Total hours
A user-friendly but powerful open-source Wikipedia scraper written in Python that lets users export articles in any language to an SQLite3 database using Beautiful Soup.
Ship #1
Pending review
I recently created pywikiscrape, a powerful, easy-to-use, accessible open-source Wikipedia scraper written in Python that lets users export articles in any language to an SQLite database using BeautifulSoup. This project was challenging because it required moving from scraping one Wikipedia article with predefined variables to scraping as many articles as the user chooses, with user-defined variables that may be incorrect or require special handling. Essentially, all the edge cases. I am very proud of myself due to the fact that I was originally thinking “That project would be way too hard! I’ve scraped some Wikipedia articles before, but something like this? It’s so much harder,” but now I have a working 440-line project with a beautiful README. I even created an X/Twitter account while making this project to promote my coding, which I am still working on to this day. The project is very straightforward to use; it will automatically prompt you for any option it needs user input for, and has defaults you can access by simply skipping the question and clicking the Enter key without any text entered.
5h 28m 27s logged
I think that this project, pywikiscrape, has been a very fun project to make so far and has helped me resolve many new types of errors I haven’t had much experience correcting. I’ve added many features, such as custom seed input, support for scraping in any language (as long as it has a Wikipedia), and many, many more things. For each and every feature added, the number of variables skyrockets, making it very hard to debug and work in any circumstance. For example, I was stuck for so long trying to solve an error that was caused by an unintentional slip of my keyboard, removing the f before an f-string. The issue wasn’t solving the error, it was finding where it was. I was considering such big things, when the reality was much smaller. I also accidentally slipped in a backslash in the version number, which was very fast to fix in the code after I realized it, but I had already pushed and tagged it. That was another one of my very big keyboard slip-ups. Encountering this taught me how to handle catastrophic errors once they’ve already been pushed. And, outside of the coding aspect, I think I really grew in my documentation skills. This was my first ever project with an image, database schemas, or tables for installation in the README. And, I created an X/Twitter account to try and bring more traction to my code for the purposes of this project, my first time using social media to publicize my code. I still have more things in plan for this project.
