Owner Dashboard (finished), Maintenace and Public Toggles, Dashboard Graphs, and Graph Page (WIP)

A lot has happened, lets dive into it:

Owner Dashboard:

This dashboard may not look very different from the public dashboard but it hold a lot more features:

• Status widget with the status time and last status. A green check if online or a red X if not.
• If offline, no weather data/widgets are shown and instead a gear widget with more information is shown. This includes if a alert was sent, last time connected, first time offline, and consecutive offline checks
• Maintenance and Public toggles
  • Maintenance toggle: If in maintenance mode, no emails will be sent about the status of the station but it will still be logged.
  • Public toggle: If off, the station will not be shown in the public dashboard list. If you try to go to the url directly it will show a “Station Not available” Page.

Dashboard Graphs

Graphs are created using Matplotlib and show cased on both dashboards. This includes solar, precipitation, pressure, and temperature graphs.

Graphing Page (and analysis)

This page is currently being worked on but will allow anyone to choose the time range, variables (yes multiple!), station, and unit. I also want to have a section for analysis like regressions and so on.

Lmk how this is looking so far!

Owner Dashboard (WIP) and Maintenance Toggle

Welcome back to a episode of me going insane! :D (jk)

Owner Dashboard (WIP)

The owner dashboard is a dashboard that is separate from the public dashboards. You must log in to access a page called “My Stations”. This page lists all stations that your account has permission to view. The account that you are using must have the station added by a administrator (rn all of those methods are in the docs. I plan on moving them to html soon)

Purpose of Owner Dashboard

The owner dashboard holds some extra functionalities based on the status:

Online

• It will show a green or red icon based on the status (along with the circle up by the units toggle) and display the exact time and status of the station.

  • The time of status is separate from weather.

• If you have the permissions to do so, you can then toggle the maintenance and public modes. (Only maintenance mode has been implemented thus far)

  • The maintenance mode stops any emails from being sent about the station
  • Public mode will allow the station’s public dashboard to be listed on the website.

Offline:

• Shows everything in online*.
  • Does not show weather data
• Displays how many times the station has been shown to be offline. (consecutive offline)
• The time of last connection.
• First offline (this is when the station is set as offline)
• If an alert email was sent

Let me know if you guys have any ideas!

Public Dashboard and dashboard widgets!!!

I had a lot of fun with this…

Public Dashboard

This is a public webpage anyone (no need to log in) for each station, listing the current weather conditions and status in a nice gui. In here are some widgets I made by myself all in CSS and HTML. (I don’t like JS if you couldn’t tell…)

These weather values include:

• Temperature
• Dew Point
• Humidity
• Precipitation Rate
• Precipitation Accumulation
• UV Index
• Solar Radiation
• Atmospheric Pressure
• Wind Speed
• Wind Gusts
• Wind Direction

Dashboard Widgets:

I made a couple of widgets that update based on the values:

• Temperature
• Pressure
• Wind

Temperature:

A small thermometer that fills up as the temperature gets higher. (I have to rescale it for Celsius)

Pressure:

A small barometer that changes based on the pressure. The center pressure (29.92) is the atmospheric pressure at sea level.

Wind Speed:

A little compass shows the cardinal through tertiary wind directions (16 directions). The current wind is represented by the blue arrow. The arrow is where the wind is coming from (how wind is properly notated)

Let Me know if you guys have any other ideas, I would love to hear em!

Working Error Pages, User Updates, Logouts, Weather Post/Weather History and a (partial) public dashboard

Been a couple of days guys, here is what I have been up to:

Error Pages:

Whenever there is some sort of error such as 404 (not found), 401 (unauthorized), 403 (Forbidden), you will get forwarded to error pages. These pages will prompt you to return or log in, depending on the error.

User Updates:

Admins can know easily update usernames, passwords, roles, etc. However this is still in the API and not a webpage yet.

Logouts:

When you sign in, your menu bar will no longer have a sign in option (shocker) and has an option to log out. This “new” menu bar is created using jinja2 if-statements in the layout.html.

Weather/Weather History:

A Weather Underground Scraper (I made this for a seperate project, check it out: https://github.com/Krupamc/weather_underground_datascraper) gathers weather data for each station (returns a null filled json if station is offline) and posts it to the API. The API stores it in the database. We can access the current weather for each station as a well as the history. The default history is 20 entries but can be extended with queries.

Public Dashboard:

Now this is the fun one. This pulls the current status, weather values, time (as a date and timestamp in a readable format), as well as wind direction in label format (e.g. NNE) and sends it to the public dashboard. The dashboard then takes the values and puts them in an easy format. It is not finished, I plan to have pressure, temperature, and wind widgets that change based on the current values.

PARTS!!!

Hey guys! Long time no see. I have been waiting on parts but I finally got them! (All but the LoRa Radio but it is on it’s way.

Parts I got:

• Raspberry Pi Pico 2x
• 18650 3200 mAh batteries (Have to send these back cause they sent the wrong type 😭)
• NEO-6M GPS Module
• DS18B20 Temperature Sensor
• 4.33” x 2.36” 5V 200mA Solar Panels x4
• Waveshare DS3231 RTC
• Pico UPS (Uninterruptible Power Supply)
• DC5521 Male cable Solar Panel Charging Module TP4056
• Micro USB to USB
  -C1N5819 Schottky Diodes
  -MicroSD Card Module
• MicroSd Card x2

I am missing the Waveshare SX1262 LoRa Node Module. I can start building everything else in the meantime and start testing out code.

If you guys would like links to my parts, either leave a comment or wait until I update the github.

Here are some photos, I was too busy being excited and taking inventory to get any 😅.

Registration Form, Posting Status, Status History

Hola guys!

Registration

There is now a registration form! It takes in a username and password and creates a user. However this user has no permissions and the person has to contact the admin to give them permissions. (in the future, a regular base user will still be able to do more stuff, see more detailed data maybe? But I don’t want to block data based on accounts. Maybe emails of your favorite stations?)

Posting Status

When the status checker runs, it will send the results over HTTP post to the database (DB) to be saved. The “current” Status DB model is overwritten by new data, but if there is a change in the status in any way, the change will be saved to the Status History DB Model.

Time

Another tiny detail is that before the status checker would store time in local time, but I changed that to be UTC time. UTC time is now universal and I plan to display the time in the local timezone based on the browser time zone.

Easter Egg

I added a easter egg in the website, I wonder if someone will find it…

Thanks for reading these devlogs, this is not necessarily a fancy/flashy project and it means a lot when someone does ❤️.

LOGIN FORMS!!! And Station Lists…

Hey guys! Whats up? I’ll tell you whats up, the moon. 😅 Dad jokes aside, I am really excited to update you.

Login forms…how exciting

Wait wait I can explain. It is cool I swear. When you input your credentials into the form and submit it, it will check the database (DB) if you have the right ones. If you don’t you will get the invalid screen shown below (I definitely didn’t forget my own password…). But if you put in the right credentials, you will be given a access_token cookie. This cookie will let you stay logged in till the cookie expires (set for 30 mins). Once you are logged it, it will direct you to your account dashboard and show you all the stations you have access to. (I didn’t make that one yet but shhhhh)

Station List

When you click on the Weather Station tab, it will show you a list of every single weather station in the network. Something else cool, whenever you hover/click a button, it will change color. Just fun for me.

Save Barnegat Bay

I am building this system as a idea I had for the Save Barnegat Bay (SBB) non-profit organization. I utilize their weather stations all of the time for my projects that I love giving back to their mesonet! Once I am finished building this, you will be able to access the SBB weather stations and see weather points as well as graphs and stuff. Oop can’t spoil all of my planned features

Website Backend, JWT, and HTML Front End

Ok, these dev logs keep getting exponentially longer and longer ;-;

What the hell is JWT?

JWT (JSON Web Token) is a way for my authenticate a user. It essentially is how I can do logins. When a user is registered, their username and hashed password is stored in the SQL database (DB). When someone would like to authenticate, their password is hashed and sent with the username. The DB check if the username and hashed password match and send back a JWT token that expires after a set time (30 mins but configurable). During this time the user would be able access their station dashboard, etc. Admin accounts have the same log in procedure. They however have the power to give each user (based on their user id in the DB) access to specific stations. This includes the ability to see the station dashboard, toggle maintenance mode, and toggle if the station is public (on our site).

Backend:

Added functions for creating accounts and logging in. Most functions now require either a general account or a admin account. If this is not fulfilled a 401 error is sent (Unauthorized error). As of right now, only admins can register accounts. The access to each station can be updated by a admin at any time. I did some more stuff but I honestly forgot :)

HTML Front End

This was honestly my favorite part so far. I started by getting the Jinja2 template system to work as well as mounting the static directory. The layout.html file allows me to change it slightly for every page. That is a bad explanation but it is a really cool system. I got the home page done (needs some text and description but thats a later me issue). I am working on getting the general station page. It will list every station and allow you to click on them to get forwarded to that stations public dashboard.

Sanity Check?

I am having so much fun with HTML. It gives a certain gratification that SQL can’t give. I feel very happy :D

Any ideas pls let me know!

SQL Database, Website Backend, Dashboard

When I said a lot last time…that was nothing compared to now ;-;

What am I going on about?

Well, I wanted a way for station owners to be able to be able to toggle maintenance mode without the admin (whoever deploys this for their own mesonet) directly editing a json file every time. One thing led to another and now I am making a website…

Website Plan:

This website will run on a FastAPI backend with either Jinja2 or Htmx front-end. There will be a public “dashboard” where anyone can take a look at general station stats and current weather conditions. The weather conditions will also be made graphical with Matplotlib. Historical data will also be accessible. However on the same site, people will be able to log in. Logged in users will be able to access only certain stations they are allotted. They will have a more detailed dashboard with more stats. Certain users will be able to toggle maintenance mode and some won’t. Admin accounts will be able to see and toggle for all stations. Usernames and passwords will be hashed and saved in the database. The log in system will also make use of cookies/sessions to keep the user logged in.

Planned SQLite Models

users: (DONE)

• id
• username
• password_hash
• role (admin, station staff, etc)

user_station_access

• user_id
• station_id
• can_view
• can_toggle_maintenance

stations:

• id
• station_id
• is_in_maintenance
• is_public

station_status:

• station_id
• last_status
• consec_offline
• first_offline
• last_conected
• alert_sent

station_current_weather:

• temp
• dew
• humd
• wind_speed
• wind_gust
• wind_dir
• pressure
• precip_rate
• precip_accum
• uv
• solar

Progress?

I have made all the models and created basic routes. There is a CRUD (Create, Read, Update, and Delete) for the stations based on another config file. On start, the program will automatically create database entries for new stations and delete ones not in the config file.

Monthly Reports, CSVs, and a little bit of Insanity

Hey guys, I have done so much, I might go insane haha!

Reports?

Every first of the month at 8 AM, a email will be sent out to any emails configured for them. They contain various stats such as, which stations are down, which have been down for the longest, uptime percentage, etc.

Emails

So I just realized that I never explained all the different emails haha.

Recipients:

• Admin: Email(s) configured to recieve only system error emails
• Global Recipients: Email(s) configured to receive all emails sent by system concerning status.
• Report Recipients: Email(s) configured to receive monthly emails. E.g. Mesonet maintainers
• Station Recipients: Email(s) configured separately for each station. E.g. station owners.

Email Types:

• Offline: Sent after a station is recorded off 3 (default) times in a row.
• Recovery: Sent once a station is detected to be online after being considered offline
  Reminder: Sent after 7 days (default) to remind of the status.
• Monthly Reports: Sent at the first of the month at 8 AM (default) with statistics such as which stations had outages, the station with longest, uptime percentage for each station, etc.
• System Errors: Sent to system admins for any errors detected, HTTP, Requests, Time, etc.

CSVs

All of the data is logged on a csv as well. Not much to say

Insanity

Something about sitting around staring at a screen doing like 5 projects on and off makes you a little crazy…on the brightside it is my birthday tomorrow. That means more coding time! ;-;

Monthly Reports, Maintenance mode, and Reminder Emails:

Yolo!

Reports?

Every month (Rn configured to be the first of the month at 8 AM) a report will be sent for all of the stations with a bunch of stats such as:

• Total downtime
• Uptime Percentage

• of outages

• Table of outages
• Longest outage
• Station with most outages
• etc.

Reminder Emails:

Once a station goes down and it has not been resolved in a set amount of days, a reminder email will be resent to the owner

Maintenance Mode:

Once switched on, emails about status will not be sent for that weather station. Checks will still occur in the background however. In the future, I will create an FastAPI app that will let owners trigger this mode on/off for their station.

Suggestions?

If you guys have any other ideas, please let me know!

Error Protection and CSV Logging

Long time no see haha!

Progress

• Error Protection through try/excepts and error codes
  • Error codes were creates for the failure points.
  • Any time there is an error, it is emailed to the “admin” email.
• Every whenever there is an error/email sent/status the data like the station id/name, time, status, etc. is saved (appended) to a csv.
• All error retry logic is modifiable through the config.py.

IT IS WORKING!!!!

Oh my god man. The email notifications were a little annoying to get finished but they are working. The image is more redacted than a Call of Duty game but it is fine.

Progress

• Recovery emails have also been made.
• Everytime a station is checked, it’s “stats” (such as when their last offline was etc) are updated using a seperate json file.
• I have mostly spent my time on bug fixing and smoothing out the app.

Start!

Hey guys!

This project is all about improving the experience of personal weather station (PWS) owners. It is a webscraper checking for the status of these weather stations. After it detects it has been off for a while it will email the administrator of the network as well as the owner. (This is a summary, take a look at the project page for a better explanation ;)

Progress?

• The station are scraped and their status is recorded
• Json files for the current and past status as well as other things is updated/created when the program is run
• Email notifications (in work)
• Timestamps in the right timezone (config).
• Several stations can be checked at once
• And other stuff I kinda forgot ;-;

Weather Underground Scraper

Hola! I swear I’m not going insane…

What is this?

Weather Underground is a place to get very local and high detailed weather data. It is a network of 250,000+ personal weather stations (PWS) used to map weather world wide. In my previous project, I used weather underground stations in my local area of Barnegat Bay. Most of them actually came from the non-proffit Save Barnegat Bay’s Mesonet of stations. Huge shoutout to them and Dr. Michael Folmer, “a dedicated meteorologist and forecaster for the NWS Ocean Prediction Center, Weather Prediction Center, Tropical Analysis, and the Forecast Branch of the National Hurricane Center.” This year I will continue using data from their mesonet

Soooo why are you scraping

It is a bit of a sore topic for me, but Weather Underground discontinued their public API a few years back. This led me to collecting EVERY. SINGLE. DATA POINT by hand. I typed them all up in a spread sheet. All 40,000+ of them. Lovely. This year I do not plan on wasting that much time and instead I created a webscraper using the beautifulsoup and requests libraries (as well as some others)

Features?

• Uses a config file for the list of weather station ids, API key for the PostgreSQL website, and error catching variables
• Scans HTML of the weather stations and picks out the data (for all variables not just what I need. This makes it so you can use it as well)
• Converts all imperial measurements into metric
• timestamps in UTC for the API to then turn back into ETC
• Send it to the API

Future:

This scraper will run every hour to collect the data from the weather stations. Scraping will begin once the buoy has started collecting data as well.

PostgreSQL Update:

• Added another SQL table for the Weather Underground data
• Added functions for it

Links:

• As much as clowned on Weather Underground, they are a great place to get weather data: https://www.wunderground.com/
• Link to the scraper repo:

PostgreSQL API Backend for Buoy

Hey guys! Let me tell you what I have been up to:

PostgreSQL:

PostgreSQL is an back-end database that will hold the data collected from the buoy. In the future it will also hold data collected from local Weather Underground weather stations using web scraping.

API Backend:

A FastAPI back-end was coded in python (running in my home lab) to use a PostgreSQL server running on a separate Raspberry Pi 5.

My homelab may or may not be at 95% storage used ;-;

Functions

• Ability to use HTTP commands (most) to send, delete, and view data.
• Communication with the PostgreSQL database
• Use of a “api-key” to only allow the buoy to POST data

Back-end for Buoy API

Today I started working on the FastAPI backend for the buoy to send data using HTTP POST…FastAPI doesn’t really like me that much. It took a while lmao.

Buoy!

Plan

This buoy will be built with:

• Raspberry Pi Pico 2x
• 18650 3200 mAh batteries
• NEO-6M GPS Module
• DS18B20 Temperature Sensor
• 4.33” x 2.36” 5V 200mA Solar Panels x4
• Waveshare SX1262 LoRa Node Module x2
• Waveshare DS3231 RTC
• Pico UPS (Uninterruptible Power Supply)
• DC5521 Male cable 
• Solar Panel Charging Module TP4056
• Micro USB to USB-C
• 1N5819 Schottky Diodes
• MicroSD Card Module
• MicroSd Card x2

Function:

The buoy will be anchored right next to a marina where every hour, it will colect the water temperature (among other parameters), record the data to a local SD card, and send the data over LoRa to the reveiver pico at the marina to send the data over WiFI to a API on my homelab server. The website to predict wind speed will pull the latest weather points from weather stations nearby as well as these buoy points to predict the wind speed based on the approach choosen (which model)

Buoy Hardware Plan

• Every hour, the Pico is reset by the real-time clock module (RTC) to restart the program.
• Upon restart, the program will set the next wake-up alarm and note the data collection number.
• Water temperature, GPS data, buoy temperature, sequence, (data collection number), and battery voltage will be collected.
• The data is then saved to an onboard MicroSD card before being sent by LoRa radio at 915 MHz (North America Standard)
• The buoy will be put in a plastic mooring buoy.
• All electronics will be in an IP68 (or greater) rated box with epoxied holes for wires inside of the buoy.
• The top of the buoy will be sliced at the seam with a 3D printed replacement (Brightly colored) that is flat and has slots on the inside for the solar panels to attach. 
• A LoRa antenna will come out the top.
• It is epoxied close with a designed opening latch using gaskets on the top to provide access in case of emergency.
• Water Temperature Probe will come out of a drilled hole in the bottom of the buoy and be epoxied close.
• Local Township Dive Team Logo will be put on the buoy to deter anyone from interacting with it.

Code status:

Most of the buoy and receiver code is coded, however it is untested as parts haven’t arrived yet.

Parts? (WHEN):

I am still waiting on parts ;-;
Once I receive them I will be able to assemble the parts and start testing my code.
Then, I will 3D design a top case to go on top of the buoy

All finished with WebOS 1!

My website hold a welcome page that can be dragged, opened (from icon), closed and can shift layers. The other app is how to contact me in about 5 ways!