@Tawhidulh
on
tyse
I created a meta search-engine. Initially I wanted it to have a full fledged crawler, but I realized I needed lots of resources (money) t make it actually usable. Instead, I decided to transition to a searxng wrapper. I am proud that I was able to make the backend work as I had very little knowledge of spring and spring-boot.
@Tawhidulh
on
tyse
6h 21m 53s logged
The basic search engine is complete, I added css and made it look nicer. Still need to polish the backend and add more features.
@Tawhidulh
on
tyse
6h 40m 58s logged
The backend for the search engine is mostly done, I need to fix some issues with elasticsearch database and make it look good. The current index is being fetched and saved into elasticsearch through searxng as I do not have the resources to run the crawler/fetch the indexes myself, though I did add a prototype crawler.
Currently this is how it works:
User searches a query -> find the best results with the given query through elasticsearch, if it does not exist it fallbacks to searxng from port 8888 -> then display the result to user.
This is so that I could save the search results as continue to use it, I plan on adding some sort of timer or something to update the index for outdated searches.
As seen in the video, there are clearly some issues with overlapping searches.
@Tawhidulh
on
tyse
5h 1m 41s logged
I added a single threaded basic web crawler for my search engine. I have yet to implement a queue system to act as the Url Frontier, instead it is a single List of String(s) to represent Url’s. This was my first time making a web crawler as well as my first time working with JSoup. Im glad it only took 5 hours 😀
