Recipe Extractor

Hello, I forgot to log this hour before shipping, but I just worked on the java code a bit and tried fixing some of the underlying issues in the code that I discussed in previous dev logs. Some issues still persist, but the website does work for a significant portion of websites.

Hello Humans. I have been working on my recipe extractor but I keep on hitting walls. I have updated the coad to work on a broader set of websites. Regardless, even though I spent like 2 hours debugging, I just can not get it to work on food network websites when running on the nest server. The simple solution for this would getting a proxy server… But I don’t know if I can do that. Here is the website working on a recipe that was flagged for not working when originally shipping the project so yaay!

Basically, 3 hours of work and little progress accomplished.

Hello! This will probably be my last dev log for this project. I am quite happy with the result. Though the website does not work with all websites, it does with a lot of them. Some of the bigger recipe websites such as food Network and Allrecipes do not allow this kind of scraping and thus the website will not work.

So What have I done since the last dev log?

Well, I updated the UI to be a bit more nicer to look at. This includes removing the brown background as it kinda looked tacky. Secondly, I alted the ENTER key to also act as an extract button; rather than having to move your mouse all the way to the button, you can just click ENTER! Also, I fixed a couple of bugs in the java code, including more error checking to ensure no random exceptions are thrown.

Errors With Some Recipes Continue to Persist

I can not seem to fix the issue where some recipes redirect the user to their local home page via popupos. I tried setting the chrome drivers language and country to the US and english, but it seems like they are using IP to determine location. Thus, due to the usage of a nest server where the IP is likely not in the US I am unable to access some websites. If anyone has a solution that would be much appreciatecd. I consulted an llm to see if their is a solution but their was no luck finding a simple solution. The only solution that i think would work is the use of a proxy but that is to complex for me.

SO MANY BUGS!

So, my scraper is working perfectly on my laptop but sometimes when I run it on the nest server it does not work for specific websites. For example, for some food network recipes it seems to be redirecting me to an alternative link. My hypothesis for this is that it thinks i’m coming from the uk (perhaps server location) and is trying to redirect me to the uk website. Hence, my logs show for a lot of the failed attempts the page title being “Recipe and Global Dishes | Food Network UK”. Next steps for me may include retrieving the url to see if a redirect occured, and perhaps checking if I can configure the headless chrome i’m using to show as being a static country such as always being in the us… We will see…

Got Nest Server Request Handling Working!

Instead of running the java program on my local computer I got it to run in a terminal window on my nest container. To accomplish this I used tmux. Thus on my website, I can send requests to the admin1.hackclub.app website rather than local host. This is a big step in the right direction, as it allows me to run requests 24/7.

The next steps for this project is to polish up the website and get a domain for it. After that, the project will be ready to ship.

Basic website complete!

I have finished a prototype website. Right now the website looks like hot garbage, but I plan to change that. The way it works is it asks the user for a url, witch they put in the textbox, than after the input is error checked it is sent to the localhost server, with the input attached to the end of the url. Then, the recipe information is sent back to the website, where it is than displayed as plain text.

In the future I plan on running the java application on my nest serve, and I will also look to find a way to host the website. Also, I plan on making the website look nicer as my next step.

Recipe now sent via http!

Through the use of the spring boot and Spring java frameworks, I was able to take my preexisting recipe extractor, and send the json data locally via http. Right now this only works locally, so my plan is to have this running on a server where it can be called on at any given time by the front-end. Learning how to use the Spring framework in java was quite hard, and this was the first time I worked on back end work like this. As a result, I did use AI to assist with spring boot implementation.

the way you can run the program with any recipe is by, when the program is running, go on your browser, and enter the assigned port with the following attatched:
/api/extract?url=RECIPE_URL

This is truly a big step for me now that the back-end is completely done. Now, I will be moving on to creating the actual website that the user will interact with.

Scraping & Parsing logic fully complete!

After a lot of work, I converted the code from printing the recipe onto the console to returning it as a JsonObject. This will ideally allow me to use an api to fetch it from a frontend website.

I tested the program with various recipe websites and they all seem to work! Originally, I was having trouble with website security features but through the use of selenium I was able to get past to the website.

For the next devlog I hope to be done API integration, and some basic front end testing.

Ingredients can now print on the console

I have just finished a prototype version where, when given a recipe link, the ingredients of the recipe are printed. This took a lot of time, and I had to change what package i used to make it work. The trick to make this work really boiled down into the fact that most recipe websites store their data in json. As a result, this is really easy to scrape and extract. Here is an example of the ingredients of some chocolate brownies printing in the console.

Website Scraper

Through the use of the JSoup java library I tried implementing a website scraper, witch takes all of the elements on a website and displays it as plain text on the console. This method worked for less secure sites such as wikipedia, but on most recipe sites that may use cloudflare it did not work. This is likely due to the anitscraping measures cloudflare has set. To work around this I may play around with other java libraries that usually have a higher success rate.

Also, I don’t know why my coding time saved under the ‘user’ project in hackatime rather than a frictionless project. If anyone can help with that, that would be lovely.