OrbitLens Devlog #8: Scaling the 3d globe and some other stuff

In the last update, i finally implemented the 2d map so this time i
want to take this website a step further since the foundation is stable
enough. I decided to go from tracking 5 hardcoded satellites to over
16,000 space objects. Here is what i changed, how i changed it and the
challenges i faced:
Change 1:

Originally, i just added the starlink satellites to what i already
had which was my initial goal. but the thing was, i was requesting the
API for thousands of satellites, which did work once and all of them
were stored in localStorage but for ‘testing purposes’ (i was just
stupif), i cleared the localStorage and re-requested the TLE which it
then instantly rejected cus of its ‘anti-bot firewalls’ and i got ip
banned for like a couple of hours and it gave a 403 error aswell.
To fix this, i read some CelesTrak documentation and i realised that
there is a way to ismply request all the satellite (every satellite that
it had) data in one request. Apparently there was a special search
query (‘active’) that i could’ve used but instead i individually fetched
everything. So once i used that query, i got all of the satellitse (a
bit more than 16000) rendered on the earth but it was laggy, which leads
to my second change.
Change 2

To fix this massive lag caused by the 16000+ satellites, I completely
replaced the standard 3D mesh pipeline with THREE.InstancedMesh so that
instead of creating 16,000 individual sphere objects (which would like
instantly crashes the browser), i can feed the GPU a single geometry and
material, along with an array of 16,000 matrixes. That means now the
gpu like renders everything in one go.
But, I hit a massive 5-second freeze whenever a user clicked the ISS
(which makes sense cus this was a 4k extremely detailed 3d model). The
thing was, WebGL delays uploading heavy 3D textures to the GPU until an
object is actually rendered on screen. And because the ISS model was
hidden on startup, the compiler skipped it, so the website was stuck
trying to render it when i selected the ISS. I solved this by forcing
the renderer to draw one “dummy frame” with all models visible during
the loading screen, which, in a sense, ‘warmed up’ the GPU.
Change 3:

The lag was better but it was still there and i realised the problem
was that each of the dots representing a satellite were smoothly moving
in space. This smooth-moving physics applied to over 16000 dots was what
slowed down the website.
To fix this, I rewrote the loop that propogates the satellite dots so
physics calculations wouldn’t choke the main JavaScript thread. To do
this i built a system to process the satellites in batches of 4,000
satellites per frame instead of all 16,000. Second, i scrapped geodetic
trigonometry (latitude/longitude conversions; see one of the earlier
devlogs) and replaced it with direct Earth-Centered, Earth-Fixed (ECEF)
3D vector rotation.
Initially, our batches were too small, causing satellites to visually
‘teleport’ instead of smoothly gliding, which looked really glitchy.
But by upgrading to matrix rotation, i managed to get a huge increase in
speed. This allowed me to crank the batch size up and get that smooth
motion across the entire thing.
Change(s) 4:

There were still some small bugs in the visual aspects like the dots
disappearing or the transition jump (see previous devlog(s) to
understand) and stuff but i took care of them.