Devlog #5 - Search first, DMs second, twice a day max
Last log was crew DMs landing on the desk. That worked. I tried adding a scraping capability to the collection worker.
Then Chrome started skipping niches, camping in Direct, and stealing the Reels feed from the upload tabs. I also wanted to scrape every time the worker restarted because the crew (my friends) stopped sending it reels and gave excuses =_=
The collector is supposed to grab three unique reels per niche (sports, anime, memes), then unread DMs. Early runs were too fast: tiles were not loaded, or it clicked the left-nav Reels path and harvested the homepage.
Fix: wait for real result tiles, copy hrefs, stay off /reels/. No niche word still goes to Unsorted cargo. I can confirm or reject so junk does not come back. Chrome is muted so upload tabs stay quiet. Still never upload from ig-collector.
Running Twice a day, without killing the worker
Same triggers: boot and every 3 hours. At most two real scrapes per local day. Waiting on FFmpeg does not count. Opening Chrome does, even if Instagram fails later.
Unsorted cargo has an On/Off switch. Off keeps the worker and upload queue running; it just does not start the scraper. Laptop .env is still the kill switch.
Also locked overlapping cycles, re-check the switch after idle, and skip Chrome if the daily count cannot be saved.
Rn I have got many reels scraped due to inital runs, when it wasnt capped properly 
btw It still has manhy bugs and random breakpoints to fix to do, Thats what I will be doing it next, This one is a tedious task tho, it took lot of tries to get it to work correctly 
Comments 0
No comments yet. Be the first!
Sign in to join the conversation.