Readme, Docker Deployment

FINALLY SOS CLOSE

ReadMe:

Updated read me with instructions and features

Got the deployment up to date:

Check it out YAY

Excited to see what you guys think about it

Weather Emails, Monthly Summaries, Registration Logic, and readme/project restructure

Okok I swear this is the last one. It has to be ;-;

Monthly Summaries

A little page that holds a summary of the past 30 days for all of the stations. High/low temperature, average temperature, average dew point, precipitation, average wind speed, and maximum wind gusts for each station. You can download the data as a csv. If you look a little below you can see where I am going with this…

Weather Emails:

Once a month (same time as the admin report) the monthly weather report summary is sent out to whoever signs up. (Whole other email model SQL table with its own CRUD in the admin settings). It also includes links to the website.

Registration Logic

Now the registration has a toggle in the settings for the page to be on or off. If the env variable BOOTSTRAP_REGISTRATION=True then the page is unlocked for the next account (which will be a admin account)

Weather Emails, mobile tweaks, map zoom

Soooooooooooooo close to shipping.

Map Zoom:

The Map will now automatically zoom based the center of the points.

Humidity and Dew point data collection:

The place the data was being scraped from got changed so I had to update the logic.

Weather Emails:

Created another CRUD for another email list. This list will get a monthly email summarizing the weather for the past 30 days for each of the stations. I created the web page for it, but need to finish cleaning it up, allowing people to sign up for the service and make the emails send automaticcally.

Mobile Tweaks:

The mobile version of the website did not look good. I spent a lot of time trying to fix it before I was able to get it to look like this!

Map! And Develop mode

Map

I used ArcGIS’s JS to create a map tab that show cases all stations that have a latitude and longitude value. I am working on getting them on the first ever scrape but you can also edit them within the settings page. There is a link that takes you to the station page as well.

Develop Mode

Created a toggle within the .env file. If on, the program will use the SQlite db and be able to run on VS Code for example. When it is off, it needs the PostGRESql db to function.

Docker

Docker is really infuriating lmao. Spent most of my time fixing tiny bugs.

Final Devlog Before Ship

Hey guys! I am so excited to get this macropad built. I finished setting up the gihub repo and added pictures and I am going to submit my design for review. Wish me Luck!!!

Prepare for Shipping

CAD:

Man this project has taught me how much I hate fusion 😭. But I had to learn it eventually right?
I added a 45 degree one inch feet to match with my keyboard. I also added small insert holes for me to put rubber feet (or more realistically hot glue).

PCB:

I spent some time error checking and making the pcb production ready. I created my own footprint for the XAOI to get it to fit on my pcb. I wanted the USB to stick out to reduce the length of cable required to go in.

Github:

I started putting all of the files on GitHub, check it out! My Github

Docker Deployment (ALMOST finished)

Hey guys!

Docker:

I am using Docker for deployment of this system to make it easy for anyone who has a weather station or network of them to showcase and monitor their own data. Docker is a type of virtualization (similar to VMs but instead sharing the kernel) that is a lot less resource intensive and is used for deploying a lot of small projects as well as any type of home labbing work flow (hosting your own server etc.). (Yes I pulled that explanation from my last devlog lmao). I will have a tutorial in the ReadME.
Within the docker stack, there are several containers with different jobs and responsibilities:

Web:

Hosts the FastAPI webserver. It uses a endpoint (/health) to monitor the servers responses. It is hosted on port 8000 by default (configurable in the .env file). I would forward this through Nginx or a Cloudflare tunnel to host it under your own domain.

Scraper:

Runs the status and weather scrapers every 15 and 5 minutes (offset by two mins) respectively through a crontab file (scraper.crontab). It is run using Supercronic. The scraper waits for the web server as well. Flock is used to prevent two instances of a program.

DB:

PostgreSQL server that holds all of the data. Weather data, users, etc. All of the other containers wait till the DB is healthy to start.

Db_backup:

A container created by Prodrigestivill to automate backups of Postgres databases. I have it configured to keep the last month and 14 days of backups. It saves all backups to the /opt/mesonet-backups folder.
I wrote scripts in order to test restoring backups (creates test table, restores using latest backup, and deletes it), create a manual backup, and restore the last backup.

Email Alerts:

Fairly similar to how emails are sent in the status-scraper, it obtains admin recpients from the web sever uses enviroment vairbales to send an email. It asks for subject and text using arguments in the terminal, so the user can send emails manually to admins as well.

Health-tests:

Runs a couple of health based scripts on a Supercronic schedule. Monthly restore test (runs the restore test mentioned before) and a daily backup health (checks how old the backup is). Emails will be sent to admins if the restore fails or the backup is more than 26 hours old.

Persistent Storage:

All the containers used volumes that continued even if the containers were deleted to be updated, the data (and the backups) are fine.

Docker and Environment Variables

Hey guys! It has been a couple of days…

Time Crunch:

So I looked on my calendar a week ago and realized that Stardance was almost over, and thought “this is plenty of time”. Then a couple of days ago I really felt the pressure. I wanted to get this out before the program finished so I started on the final bits and deployment to turn this into a usable system.

Docker:

If you don’t know what docker is, count your lucky stars. Just kidding, its very cool when it works. It is a type of virtualization (similar to VMs but instead sharing the kernal) that is a lot less resource intensive and is used for deploying a lot of small projects as well as any type of home labbing work flow (hosting your own server etc.). I have worked with docker with pre-built images and containers before and never really went into the DIY portion. This was my first intro into it and I had to do a lot of googling and the such. It was really frustrating and required quite a few head banging.

NOTE:

I did use AI sometimes (tried as little as I could) to get docker working. You can see me sometimes using it in my lapse recordings, but I tried to keep it closed as long as I could to stop me from using it :D.

Environment Variables:

I was told that hard coding my config values in a config file was a bad idea…and it makes sense. It took a bit, but once I setup environment variables, oh man did it get so much better. This also makes it a lot easier in docker to setup the container. Very happy about this.

Help from a friend!

I told this story a couple of weeks ago, but one of my friend’s dad found my repo and left a couple of issues on my page and these were my last two. I really gotta thank him for these ideas.

API Email Calls, Hardware specifications, Docker?

Hey guys, School started two weeks ago, so my posts will be a little delayed lol…

API Email Calls:

I originally hard coded my email lists for Reports, station-specific, admin, and global emails, but I wanted to unify them with the rest of the API. I spent a lot of time creating columns in SQL, endpoint CRUD, settings HTML page CRUD, as well as changing the scrapers to pull the lists and use them.

Hardware specifications:

I added columns in SQL to add Hardware metadata for the stations. It can be seen in all dashboards (see photo)

Docker:

I was thinking of creating a docker container for this setup to make deployment easier. Would that be something you guys would like to see? Let me know.

Case work!

Hey hey hey, I know it has been like three months but I can explain…

Why so long?

Well to be honest, I do NOT like fusion. I hate geometry and by extension cad. This project is my first time ever doing it so it took a lot of youtube videos, advise, and procrastination to get here.

The Case:

The case is pretty simple, it has the top piece with cutouts for the OLED, rotary encoder, keys, as well as holes for the LEDs to shine through. (I will fillet everything at the end) It also has holes for the heat inserts to melt into. The bottom case has the actual walls of the case, holes for the screws, standoffs for the pcb, as well as the USB C cutout.

This is definitely not my cup of tea, so please let me know if anyone has any ideas.

Robots.txt, User login error, API security, credit footer, station hardware

Whats up?

Robots.txt:

For those who don’t know, a “robots.txt” tells scrapers and any bots where they allowed to go on your website. I only allowed them to go on public html pages. However this is only a recommendation, the big guys can do whatever they want.

User Login Error:

I had this error where any public accounts (or not admins) would get a error when logging in. I was searching for permissions with the wrong column in the database 😅.

API Security:

I made sure there were no open routes and everything was secured.

Credit Footer:

This was pretty cool. At the end of ever page, I added this little footer with my name and a link to the github repo.

Station Hardware:

I added a new column to store station hardware info. This is mainly for what weather station model it is. Ex: Davis Vantage Pro2. I am working on making the update form in the settings page allow to update this info as well as more station info.

Some Help and ReadMe!

The project is almost done! (why do I feel like I will be saying that a lot…)

Some help!

Today I saw interesting post on Facebook about a website someone my age built and I commented it on it asking if I could see the code cause I was curious if they used Flask, FastAPI, etc. I also posted my project and to my surprise, the poster is actually my friend’s dad (small world am I right) and he reviewed my code and left a few issues on it. He even downloaded it to update my requirements.txt. I then merged his request. Thank you for all of your help!

ReadMe:

I spent all of today working on the readme, you can read it here. However here is a little sneak peak:

Hey! My name is Krupam, and I am a nerd. If you are reading this, you are most likely one as well. I love weather, and learning about it. Other people who love weather will create and host Personal Weather Stations (PWS) and post the data to websites like Weather Underground. I have used these stations for my own research projects. Check one out!. In my area, there is a mesonet (regional network of PWS’s) from the Save Barnegat Bay non-profit (check them out, they are awesome!). I have used plenty of their stations in my research. I was given the amazing opportunity to present my research at one of their meetings regarding their mesonet. It was eye-opening! As I listened into the meeting, they discussed one of the current issues; they could never tell when a weather station went down. The station owners would find out days after if not weeks. This program is the solution to that and more.

The system is a weather-station data platform and monitoring system. The system collects data from Weather Underground (WU), turns it into usable information, and alerts station owners and administrators when a station needs attention. All data is shown on the website in a easy, user-friendly approach. This includes dashboards, graphing, analysis, scientific tests, and admin controls (accessed through JWT-based auth). The system is coded mostly in Python and there are three main systems: The weather and status scrapers, backend, and frontend website.

Settings Page, API Stations List Call for Scraping System

Hey! Here is what I have been up to…

Settings Page:

This is a Admin only page (will pop up in nav bar when admin) that configures, deletes, creates, the main functionalities. This includes stations and users. After every function is run a error/success message will show with detailed intomation depending on the function.

These Functions include:

• Modify User: Change role, username, password
• Register User: Create
• Delete User
• Read User Assigned Stations: Choose a station, the page will reload and show all stations the user can access.
• Grant User Access: Choose a user, and give them access to a station. This is view only by default
• Revoke Access: Remove User access to a specific station. On reload after choosing a user it will show any assigned stations
• Update User Access: Choose a user, on reload it wil show all available stations and when selected, will show what access it has. You then check/uncheck the access you want it to have and submit.
• Modify Station: Currently only allows to change station name.
• Create Station: Enter new id and name as well as options to be public and in maintenance mode (default off).
• Delete Station: Deletes station only from the current stations. Does not delete data. I need to rewrite this to make it so it stops collecting data instead.
• Read all Stations: Lists all stations and if they are in maintenance or not

API Stations List Call

I realized around 11 at night that I did something very dumb. All of the stations in the configs for the Weather and Status scrapers were hard coded values and were NOT synced with the current stations database. To combat this, I added a new column to the Stations database table. This column was called collect_active or something of the sort. I plan to migrate the delete function to turning off the collection. I then changed the scrapers to pull a new current list at every run and used that as their truth. I finished this around 3 AM last night 😭 .

If you guys have any other ideas, let me know!

Admin Settings Page (WIP)

Settings Page:

Now that the main functionality of the site is finished, I started making HTML representations of all the methods I have predefined. This includes CRUD (create, read, update, and delete) functions for several pieces of the website. Here are the ones I started to make:

• Reset/Update User password, username, role.
• Register Users
• Delete User
• Read User Assigned Stations
• Grant User Access to a station
• Revoke User Acccess to a station
• Modify User Access to a station.

Everything uses HTLM forms and when they submit, they will return a pop up success or error message.

There are a couple of other features I forgot that I made (Such happens when coding at ungodly hours) but anyone can try out the website (if I am working on it at the same time) at https://sbb.krupamlab.com

CSV Downloads, Weather Table, Error Details on pages, and a profile indicator!

Let’s jump into it!

CSV Downloads:

For graphing, analysis, tests, and table pages, I created CSV downloads.

• Graphing: Downloads raw data
• Analysis: Downloads analysis summary and raw data
• Tests: Downloads test results and raw data
• Weather Tables: Downloads raw data

Weather Tables:

In the public dashboards, there is now a table at the bottom with all data from the current day. There are quick buttons to go forward and back a day as well as a day input and go button.

Profile Indicator:

It is just a blank profile picture icon with your username next to it. It only shows when logged in. When clicked it sends you to the my-stations page.

Error Details:

If there are specific error details in the backend, they will be sent to the error page. Otherwise a generic 40X error will be show

Testing Page Finished! (without csvs…) HTML Form Bug Fixes:

Sup guys, here is what I have been up to…

Testing Page:

Oh man this was pretty hard. I actually had writers block but for code. I just couldn’t think of a solution. I just sat there trying to think of how to implement the system. With enough brute force however and some breaks, I did manage to get it working.

What is it?

This page allows the user to run scientific tests. These tests include Linear Regressions, ANOVAS, and T-Tests. Linear Regressions look for a relationship between variables. ANOVAS and T-tests determine if there a difference between groups of data (single variable).

Functionality:

The form will ask the user to choose a test, station, variable, unit, and time range, as well as test-specific inputs to run a test. Once the user submits the test, it will either return a graphical result (for the Linear Regression) or stats of the T-Test or ANOVA. The Tests will only make groups depending on the same time frame but different stations. (I only decided to do one for simplicity) This all works but I need to create the ability to download the csv of the data used for the tests.

General Inputs:

• Test Type (T-test, ANOVA, Linear Regression)
• Range Type: (Relative or Date)
  • Last ___ (hours, months, etc)
  • Start and end date
• Units (Imperial or metric)

Test Specific Inputs:

When a test is chosen, test-specific inputs appear. (this happens for other forms such as graphing and analysis pages. I just forgot to mention it) The appeared inputs are then required (like the other default inputs like time, etc) but when a different test is choosen the required inputs are updated with the new ones. These include:

Linear Regression:

• Station
• X Variable
• Y Variable
• Regression

T-Test:

• Outcome Variable
• Station A
• Station B

ANOVAS:

• Outcome Variable
• Station A
• Station B
• Station C

HTML Form Fixes:

I went through all of the Forms that have inputs that switch and make them work nicely with the backend. I make inputs that switch be required and seen at the right time.

PARTS!!! and coding…AND ASSEMBLY!!!

Oh Oh Boy have I had a lot of fun haha.

Overall

I started assembling my parts, more unboxing, but I also started testing individual components with the picos. I took the code I had created already and modified it in seperate files to only test one sensor at a time. I have had a jolly good time. I did most of my work over laspe so take a look! My Lapse Account

Soldering

I was unable to get this on camera but I did most of y soldering for the MicroSD card. It was my first time but fairly simple, I just connected a few Dupont connectors into the holes in the board (for lack of a more technical term). I still have more soldering to do but I will try to get it on camera next time!

Pico Troubles

I have never used a microcontroller before so this was a crazy thing to try and learn. I spent a lot of time trying to mess around with libraries and find them on the internet.

Battery Drivers

The UPS that I have for the Pico has drivers and a program along it to collect battery info…but it only works on actual linux. It connects through I2C however, so I just reverse engineered it for the pico! I am still working on it however.

(I also spent like 2 hours trying to figure out my own diagram and plug the MicroSD card board in…)

Testing Page (Still WIP), General Bug Fixing, and Site Cleanup

Yikes, I got reaaaal close to hitting the 10 hour limit for these devlogs 😅…

Site Cleanup

I am really proud of this one. I finally took time to go around and get most of the small detailed squared away. I got the site favicon (small icon in the web tab) and other meta data information fixed (like the .webmanifest) working. I also spent a good chunk of time creating the home page and making it look professional.

General Bug Fixing

I took some time to go through the website and found some tiny bugs. I also had a few of my friends test out the website (they didnt break it!!!!). Small stuff like making it easier to see the graphs, titles on the tabs, etc.

Test Page

This page is like the ultimate boss for me haha. I would like this page to contain linear regressions (LR), ANOVAS, and T-tests. These are scientific tests used to find statistical differences in data. (see the below screen shot for a better explanation of all of these tests. The user chooses a station, range mode, range, units, and the type of test. Then based on that test, a few options are given:

LR:

• X Variable
• Y Variable
• Title

Mode A:

Choose 1 station but two time ranges (same variable)

Mode B:

Choose 2 stations with one time range (same variable)

ANOVA:

• Mode a/b
• Three or more time ranges/station depending on mode

• Two ranges/station depending on mode

A Variable Analysis Page and Graphing with a date system!

Hey guys! Here is what I have been up to…

Graphing Date System:

As well as a “last _ ____” system (relative range), I created a version that allows me to choose “Date Range.” Date Range allows you to choose between two dates. It sounds simple but was a lot harder to code 😅. As I wrote most of the code while being really sleep deprived, take a look over at @app.get("/graph/weather/{station_id}") on the project’s github page. (I recommend using ctrl + f to find it)

Variable Analysis Page:

This page is pretty similar to the graphing page (see previous devlogs) where it uses a HTML form with query variables to create calculations. The user selects a weather station, variable, range mode (relative or date), inputs for that mode (hours/date, etc) and the units. Then the latest, minimum, maximum, mean, median, range, sample value (n value), and the overall trend (falling, steady, rising, insufficient data). The time will also be shown if applicable (latest value). You can then download a csv of all the data the calculations were derived from. Under that is another box showing the full time the data is collected from as well as the units

Future:

I still need to add a tab for more advanced analysis such as linear regressions, anovas, t-tests, etc.

If you guys have any more ideas, Let me know!!!