TESS and K2 missions dont work, i am sorry

its been 5 days i didnt get time to code properly but what i know is that i cannot add support for TESS and K2 for now.
_
i tried everything which i can for now, K2 fires its thrusters every 6 hours to correct its position and that thruster fire overpowers the star transit, i tried increasing period_min and increasing it enough did lead to getting correct period but that was only posible cuz i already knew the period of the star i was testing on, from what it looks like i think just rolling median in detrending isnt enough for K2 mission stars, and i am not planning to implement the corrections it requires for now
_
TESS had 2mins candence so it had over 1 million data points and in multiple loops it lead to taking hours to analyse, and my BLS double pass scales too much with data points, but i binned it like i did for the plot but even with that it took around 18 minutes to run and produced a 24x period which was an alias, idk how the alias array didnt even include the normal alias, maybe because the period is less than 1 day and my algo searches between period_min and max and min was set to 1.
that tells about the problem but it kinda proves that my system finds period based on what you feed into it in some values which i need to work on, cuz some stars can be weird, and wont lie i am giving up on it for now.

Advanced options in search

finally added period min, period max, and mission selection in search, ez
but looks like i should use shadcn buttons instead the advanced options button doesnt look like a button lowk

Result card rework

shiii i made it hella good now
it has a nice border and tooltips for each metric, and a verdict explaination in a details summary tag
also remved the basic confidence display cuz it wasnt telling anything, now it has explaination

Better scoring and Confidence levels

pain in ASS bro

so i started from a normal scoring system which worked fine for finding periods but it wasnt good for confidence factors cuz it scales with dataset, so a bigger data set despite having no period will still have better score than a smaller data set with a period, so i need a better scoring system

Signal-to-Noise Ratio

this idea came while i was soldering a jammer i am working on lol
so i measured the ratio and replaced the basic system with depth / (noise / sqrt(n_in)) it is not the perfect SNR system but it works nice enough.
_
so i tested it against KIC 8462852 which does not have any exoplanet, and the signal was still wrong cuz this star (Tabby’s star) is a bit weird and it has large periodic dimming events and it was long enough on a single candence on various points to score high enough on SNR and look like a strong signal, so that made it clear that SNR isnt alone enough

Coverage-Ratio check

THIS finally passed every test, it clusters in transit points between time gaps so a bigger time gap will make a new occurence, and it counts those occurances, the tabby’s star had only 2 so i made the limit 5, the “transit” must occur more than 5 times to be qualified for SNR check, which worked good and finally i got WEAK signal on that star check.

Minimal UI work (functional)

made this very simple search thingy, it was easy asf
i used recharts to plot that, which was the real pain cuz the data has 60k+ points to plot and it was taking too long but most of those points werent nececssary so i made smaller bins of those points and plotted them which worked fine
__
i am yet to work on the confidence score, i dont have enough observations to decide on a “good enough high score” so thats a todo for now
__
and variables like mission, period min, and period max are also a todo for now :3

/api/search made API route for analysis

it runs the pipeline scripts in order: ingest -> detrend -> analysis
and all that takes ton of time, so there is a cached/ file and it creates files of star’s ingestion and detrended data. analysis however is re-run everytime cuz period min and max can change

Initialized nextjs app

just that, initalized nextjs app and picked a nice color scheme from tweakcn.com

Analysis script

holy rupnil judgement

so if you dont know in this project i am just getting the light curve
data from a library, other than that i am doing EVERYTHING from scratch,
so this is the 7 hour dev log of what i did in my analysis script.

bls.py

most time was spent here trying to write Box Least Square algorithm
from scratch, it will be used to find the orbital period of the planet
from the folded data, i wrote various versions in order to find the
correct and best way to do it and had to fix shit ton of bugs, which i
wrote down for this dev log (check below)

analysis.py

after lot of rough work focused around Kepler 8 star in bls.py i
transfered the BLS and other refining functions to analysis script for a
proper cli, it takes detrended data json file and outputs:

1. orbital period of planet
2. confidence score (calculated from depth of transit)
3. duration of transit (calculated two different ways)
4. phase start

hella problems i faced

1. performance in BLS algo: the very first version i wrote had a triple
   loop, it was like 2+ billion operations so it was taking like 7 minutes
   to complete and give wrong orbital period, then i switched to numpy to
   use vectorization and it made it hella fast, dropped the time to just 60
   seconds
2. “aliases”: later i found out that i was not getting wrong orbital
   period it was just different multiples of the period, so i had to make a
   function check_aliases which gets the first 4 values of period it
   could find and the aliases[0] is the best one with highest score
3. precision: the values were never precise and the reason was that the
   BLS seach wasnt enough, what i needed to do was a coarse search and get a
   general idea of transt and do a more precise search on top of the
   coarse values which i did with a function i made bls_search_two_pass
   who’s period was then correct but with lower score and another multiple
   was still with higher score, so after hella digging i found that it was a
   like chicken and egg problem yk, see the BLS search needs duration to
   calculate stuff better and we dont know that duration, we can get the
   duration by doing some calculations on period but to get period we need
   to do BLS search, so at first i used a constant value 0.4 of duration
   after some manual experimenting, which works as a decent enough
   heuristic at first, then after we get a nice period i use the
   refine_duration function which then digs in and calculates the best
   duration value, which is THEN used to run another BLS search this time
   with proper suitable duration. so it went like: bls coarse search
   -> bls precise search -> bls alias calculation -> duration refining -> anoher precise search and then we come to proper
   orbital period with highest score, FINALLY.

this dev log was MUCH bigger but got deleted cuz it wasnt saved and this fucking error deleted it, now i am too frustrated and idc enough to write it completey again this is all i had saved

Data cleaning and Detrending

removing outliers

made a simple z score calculation, just removes values 3 standard deviations away, got a “nice enough” result and removed like ~2000 values from ~64000 values (4 years of data) lmao

detrending

why: this was important, cuz the telescope moves around to protect its instruments from the sun and various other reasons, so there can be gaps which look like a transit but they arent, they are slow and long in transition unlike a real exoplanet between the star and the telescope, that transit will be like an instant sharp dip, so i had to smooth out those false values
__
used a basic moving window median to divide out those values, you can see the before and after of detrending in first two pics

and this is how the script runs:

python detrend.py --input lightcurve.json --output detrended.json --window 50

Some exploration and Light Curve ingestion

first dev log

lightkurve library

so i spent some of the time messing around with this library exploring and plotting random things, this library can fetch public data from telescope missions like Kepler, K2, and TESS and it makes getting that data and normalising some values a bit easy so i tested those things
and i plotted this in matplotlib this is one light curve of Kepler-8, you can see the gap in the data that MIGHT be an exoplanet lmao, who knows

ingestion script

so i will make different scripts for differnet stuff and make a single pipeline later ig, the current script runs like:

python ingest.py --star "KIC 6922244" --output lightcurve.json

it takes the star name output as file name or json output in stdin, and then it downloads a HUGE file (check screenshot lightcurve.json) full of the timestamp and flux values (which are basically brightness of the star at that timestamp)

further analysis will be done on the output of ts script.