Robots.txt, User login error, API security, credit footer, station hardware

Whats up?

Robots.txt:

For those who don’t know, a “robots.txt” tells scrapers and any bots where they allowed to go on your website. I only allowed them to go on public html pages. However this is only a recommendation, the big guys can do whatever they want.

User Login Error:

I had this error where any public accounts (or not admins) would get a error when logging in. I was searching for permissions with the wrong column in the database 😅.

API Security:

I made sure there were no open routes and everything was secured.

Credit Footer:

This was pretty cool. At the end of ever page, I added this little footer with my name and a link to the github repo.

Station Hardware:

I added a new column to store station hardware info. This is mainly for what weather station model it is. Ex: Davis Vantage Pro2. I am working on making the update form in the settings page allow to update this info as well as more station info.

Some Help and ReadMe!

The project is almost done! (why do I feel like I will be saying that a lot…)

Some help!

Today I saw interesting post on Facebook about a website someone my age built and I commented it on it asking if I could see the code cause I was curious if they used Flask, FastAPI, etc. I also posted my project and to my surprise, the poster is actually my friend’s dad (small world am I right) and he reviewed my code and left a few issues on it. He even downloaded it to update my requirements.txt. I then merged his request. Thank you for all of your help!

ReadMe:

I spent all of today working on the readme, you can read it here. However here is a little sneak peak:

Hey! My name is Krupam, and I am a nerd. If you are reading this, you are most likely one as well. I love weather, and learning about it. Other people who love weather will create and host Personal Weather Stations (PWS) and post the data to websites like Weather Underground. I have used these stations for my own research projects. Check one out!. In my area, there is a mesonet (regional network of PWS’s) from the Save Barnegat Bay non-profit (check them out, they are awesome!). I have used plenty of their stations in my research. I was given the amazing opportunity to present my research at one of their meetings regarding their mesonet. It was eye-opening! As I listened into the meeting, they discussed one of the current issues; they could never tell when a weather station went down. The station owners would find out days after if not weeks. This program is the solution to that and more.

The system is a weather-station data platform and monitoring system. The system collects data from Weather Underground (WU), turns it into usable information, and alerts station owners and administrators when a station needs attention. All data is shown on the website in a easy, user-friendly approach. This includes dashboards, graphing, analysis, scientific tests, and admin controls (accessed through JWT-based auth). The system is coded mostly in Python and there are three main systems: The weather and status scrapers, backend, and frontend website.

Settings Page, API Stations List Call for Scraping System

Hey! Here is what I have been up to…

Settings Page:

This is a Admin only page (will pop up in nav bar when admin) that configures, deletes, creates, the main functionalities. This includes stations and users. After every function is run a error/success message will show with detailed intomation depending on the function.

These Functions include:

• Modify User: Change role, username, password
• Register User: Create
• Delete User
• Read User Assigned Stations: Choose a station, the page will reload and show all stations the user can access.
• Grant User Access: Choose a user, and give them access to a station. This is view only by default
• Revoke Access: Remove User access to a specific station. On reload after choosing a user it will show any assigned stations
• Update User Access: Choose a user, on reload it wil show all available stations and when selected, will show what access it has. You then check/uncheck the access you want it to have and submit.
• Modify Station: Currently only allows to change station name.
• Create Station: Enter new id and name as well as options to be public and in maintenance mode (default off).
• Delete Station: Deletes station only from the current stations. Does not delete data. I need to rewrite this to make it so it stops collecting data instead.
• Read all Stations: Lists all stations and if they are in maintenance or not

API Stations List Call

I realized around 11 at night that I did something very dumb. All of the stations in the configs for the Weather and Status scrapers were hard coded values and were NOT synced with the current stations database. To combat this, I added a new column to the Stations database table. This column was called collect_active or something of the sort. I plan to migrate the delete function to turning off the collection. I then changed the scrapers to pull a new current list at every run and used that as their truth. I finished this around 3 AM last night 😭 .

If you guys have any other ideas, let me know!

Admin Settings Page (WIP)

Settings Page:

Now that the main functionality of the site is finished, I started making HTML representations of all the methods I have predefined. This includes CRUD (create, read, update, and delete) functions for several pieces of the website. Here are the ones I started to make:

• Reset/Update User password, username, role.
• Register Users
• Delete User
• Read User Assigned Stations
• Grant User Access to a station
• Revoke User Acccess to a station
• Modify User Access to a station.

Everything uses HTLM forms and when they submit, they will return a pop up success or error message.

There are a couple of other features I forgot that I made (Such happens when coding at ungodly hours) but anyone can try out the website (if I am working on it at the same time) at https://sbb.krupamlab.com

CSV Downloads, Weather Table, Error Details on pages, and a profile indicator!

Let’s jump into it!

CSV Downloads:

For graphing, analysis, tests, and table pages, I created CSV downloads.

• Graphing: Downloads raw data
• Analysis: Downloads analysis summary and raw data
• Tests: Downloads test results and raw data
• Weather Tables: Downloads raw data

Weather Tables:

In the public dashboards, there is now a table at the bottom with all data from the current day. There are quick buttons to go forward and back a day as well as a day input and go button.

Profile Indicator:

It is just a blank profile picture icon with your username next to it. It only shows when logged in. When clicked it sends you to the my-stations page.

Error Details:

If there are specific error details in the backend, they will be sent to the error page. Otherwise a generic 40X error will be show

Testing Page Finished! (without csvs…) HTML Form Bug Fixes:

Sup guys, here is what I have been up to…

Testing Page:

Oh man this was pretty hard. I actually had writers block but for code. I just couldn’t think of a solution. I just sat there trying to think of how to implement the system. With enough brute force however and some breaks, I did manage to get it working.

What is it?

This page allows the user to run scientific tests. These tests include Linear Regressions, ANOVAS, and T-Tests. Linear Regressions look for a relationship between variables. ANOVAS and T-tests determine if there a difference between groups of data (single variable).

Functionality:

The form will ask the user to choose a test, station, variable, unit, and time range, as well as test-specific inputs to run a test. Once the user submits the test, it will either return a graphical result (for the Linear Regression) or stats of the T-Test or ANOVA. The Tests will only make groups depending on the same time frame but different stations. (I only decided to do one for simplicity) This all works but I need to create the ability to download the csv of the data used for the tests.

General Inputs:

• Test Type (T-test, ANOVA, Linear Regression)
• Range Type: (Relative or Date)
  • Last ___ (hours, months, etc)
  • Start and end date
• Units (Imperial or metric)

Test Specific Inputs:

When a test is chosen, test-specific inputs appear. (this happens for other forms such as graphing and analysis pages. I just forgot to mention it) The appeared inputs are then required (like the other default inputs like time, etc) but when a different test is choosen the required inputs are updated with the new ones. These include:

Linear Regression:

• Station
• X Variable
• Y Variable
• Regression

T-Test:

• Outcome Variable
• Station A
• Station B

ANOVAS:

• Outcome Variable
• Station A
• Station B
• Station C

HTML Form Fixes:

I went through all of the Forms that have inputs that switch and make them work nicely with the backend. I make inputs that switch be required and seen at the right time.

Testing Page (Still WIP), General Bug Fixing, and Site Cleanup

Yikes, I got reaaaal close to hitting the 10 hour limit for these devlogs 😅…

Site Cleanup

I am really proud of this one. I finally took time to go around and get most of the small detailed squared away. I got the site favicon (small icon in the web tab) and other meta data information fixed (like the .webmanifest) working. I also spent a good chunk of time creating the home page and making it look professional.

General Bug Fixing

I took some time to go through the website and found some tiny bugs. I also had a few of my friends test out the website (they didnt break it!!!!). Small stuff like making it easier to see the graphs, titles on the tabs, etc.

Test Page

This page is like the ultimate boss for me haha. I would like this page to contain linear regressions (LR), ANOVAS, and T-tests. These are scientific tests used to find statistical differences in data. (see the below screen shot for a better explanation of all of these tests. The user chooses a station, range mode, range, units, and the type of test. Then based on that test, a few options are given:

LR:

• X Variable
• Y Variable
• Title

Mode A:

Choose 1 station but two time ranges (same variable)

Mode B:

Choose 2 stations with one time range (same variable)

ANOVA:

• Mode a/b
• Three or more time ranges/station depending on mode

• Two ranges/station depending on mode

A Variable Analysis Page and Graphing with a date system!

Hey guys! Here is what I have been up to…

Graphing Date System:

As well as a “last _ ____” system (relative range), I created a version that allows me to choose “Date Range.” Date Range allows you to choose between two dates. It sounds simple but was a lot harder to code 😅. As I wrote most of the code while being really sleep deprived, take a look over at @app.get("/graph/weather/{station_id}") on the project’s github page. (I recommend using ctrl + f to find it)

Variable Analysis Page:

This page is pretty similar to the graphing page (see previous devlogs) where it uses a HTML form with query variables to create calculations. The user selects a weather station, variable, range mode (relative or date), inputs for that mode (hours/date, etc) and the units. Then the latest, minimum, maximum, mean, median, range, sample value (n value), and the overall trend (falling, steady, rising, insufficient data). The time will also be shown if applicable (latest value). You can then download a csv of all the data the calculations were derived from. Under that is another box showing the full time the data is collected from as well as the units

Future:

I still need to add a tab for more advanced analysis such as linear regressions, anovas, t-tests, etc.

If you guys have any more ideas, Let me know!!!

#Graphing Page!!! As well as other bug fixes…

Graphing Page

This page is created by making a HTML form and submitting as a POST request to the same method that creates the dashboard graphs. All the graph information is a query request (the infomation after a question mark in a url, see photo). This means you can bookmark a graph to see the most recent data with your own personal preset. The user must select the station (from a list), type a title, the time range, (last __ unit), and the variables they would like. They can choose as many variables as they would like. There are also variable type presets a user can choose. It will preset everything to make the graph including the time range and title. Once a graph is generated, it will be displayed with the options to download it as a png, download the csv data, or the clear the graph. I am working on a date range next as well as the analysis page with linear regressions, t-tests, ANOVAS, etc.

Bug Fixes

I had a couple of friends test out the site to find some bugs, and boy did they deliver. They managed to point out some logical errors I had, and I fixed them.

Owner Dashboard (finished), Maintenace and Public Toggles, Dashboard Graphs, and Graph Page (WIP)

A lot has happened, lets dive into it:

Owner Dashboard:

This dashboard may not look very different from the public dashboard but it hold a lot more features:

• Status widget with the status time and last status. A green check if online or a red X if not.
• If offline, no weather data/widgets are shown and instead a gear widget with more information is shown. This includes if a alert was sent, last time connected, first time offline, and consecutive offline checks
• Maintenance and Public toggles
  • Maintenance toggle: If in maintenance mode, no emails will be sent about the status of the station but it will still be logged.
  • Public toggle: If off, the station will not be shown in the public dashboard list. If you try to go to the url directly it will show a “Station Not available” Page.

Dashboard Graphs

Graphs are created using Matplotlib and show cased on both dashboards. This includes solar, precipitation, pressure, and temperature graphs.

Graphing Page (and analysis)

This page is currently being worked on but will allow anyone to choose the time range, variables (yes multiple!), station, and unit. I also want to have a section for analysis like regressions and so on.

Lmk how this is looking so far!

Owner Dashboard (WIP) and Maintenance Toggle

Welcome back to a episode of me going insane! :D (jk)

Owner Dashboard (WIP)

The owner dashboard is a dashboard that is separate from the public dashboards. You must log in to access a page called “My Stations”. This page lists all stations that your account has permission to view. The account that you are using must have the station added by a administrator (rn all of those methods are in the docs. I plan on moving them to html soon)

Purpose of Owner Dashboard

The owner dashboard holds some extra functionalities based on the status:

Online

• It will show a green or red icon based on the status (along with the circle up by the units toggle) and display the exact time and status of the station.

  • The time of status is separate from weather.

• If you have the permissions to do so, you can then toggle the maintenance and public modes. (Only maintenance mode has been implemented thus far)

  • The maintenance mode stops any emails from being sent about the station
  • Public mode will allow the station’s public dashboard to be listed on the website.

Offline:

• Shows everything in online*.
  • Does not show weather data
• Displays how many times the station has been shown to be offline. (consecutive offline)
• The time of last connection.
• First offline (this is when the station is set as offline)
• If an alert email was sent

Let me know if you guys have any ideas!

Public Dashboard and dashboard widgets!!!

I had a lot of fun with this…

Public Dashboard

This is a public webpage anyone (no need to log in) for each station, listing the current weather conditions and status in a nice gui. In here are some widgets I made by myself all in CSS and HTML. (I don’t like JS if you couldn’t tell…)

These weather values include:

• Temperature
• Dew Point
• Humidity
• Precipitation Rate
• Precipitation Accumulation
• UV Index
• Solar Radiation
• Atmospheric Pressure
• Wind Speed
• Wind Gusts
• Wind Direction

Dashboard Widgets:

I made a couple of widgets that update based on the values:

• Temperature
• Pressure
• Wind

Temperature:

A small thermometer that fills up as the temperature gets higher. (I have to rescale it for Celsius)

Pressure:

A small barometer that changes based on the pressure. The center pressure (29.92) is the atmospheric pressure at sea level.

Wind Speed:

A little compass shows the cardinal through tertiary wind directions (16 directions). The current wind is represented by the blue arrow. The arrow is where the wind is coming from (how wind is properly notated)

Let Me know if you guys have any other ideas, I would love to hear em!

Working Error Pages, User Updates, Logouts, Weather Post/Weather History and a (partial) public dashboard

Been a couple of days guys, here is what I have been up to:

Error Pages:

Whenever there is some sort of error such as 404 (not found), 401 (unauthorized), 403 (Forbidden), you will get forwarded to error pages. These pages will prompt you to return or log in, depending on the error.

User Updates:

Admins can know easily update usernames, passwords, roles, etc. However this is still in the API and not a webpage yet.

Logouts:

When you sign in, your menu bar will no longer have a sign in option (shocker) and has an option to log out. This “new” menu bar is created using jinja2 if-statements in the layout.html.

Weather/Weather History:

A Weather Underground Scraper (I made this for a seperate project, check it out: https://github.com/Krupamc/weather_underground_datascraper) gathers weather data for each station (returns a null filled json if station is offline) and posts it to the API. The API stores it in the database. We can access the current weather for each station as a well as the history. The default history is 20 entries but can be extended with queries.

Public Dashboard:

Now this is the fun one. This pulls the current status, weather values, time (as a date and timestamp in a readable format), as well as wind direction in label format (e.g. NNE) and sends it to the public dashboard. The dashboard then takes the values and puts them in an easy format. It is not finished, I plan to have pressure, temperature, and wind widgets that change based on the current values.

Registration Form, Posting Status, Status History

Hola guys!

Registration

There is now a registration form! It takes in a username and password and creates a user. However this user has no permissions and the person has to contact the admin to give them permissions. (in the future, a regular base user will still be able to do more stuff, see more detailed data maybe? But I don’t want to block data based on accounts. Maybe emails of your favorite stations?)

Posting Status

When the status checker runs, it will send the results over HTTP post to the database (DB) to be saved. The “current” Status DB model is overwritten by new data, but if there is a change in the status in any way, the change will be saved to the Status History DB Model.

Time

Another tiny detail is that before the status checker would store time in local time, but I changed that to be UTC time. UTC time is now universal and I plan to display the time in the local timezone based on the browser time zone.

Easter Egg

I added a easter egg in the website, I wonder if someone will find it…

Thanks for reading these devlogs, this is not necessarily a fancy/flashy project and it means a lot when someone does ❤️.

LOGIN FORMS!!! And Station Lists…

Hey guys! Whats up? I’ll tell you whats up, the moon. 😅 Dad jokes aside, I am really excited to update you.

Login forms…how exciting

Wait wait I can explain. It is cool I swear. When you input your credentials into the form and submit it, it will check the database (DB) if you have the right ones. If you don’t you will get the invalid screen shown below (I definitely didn’t forget my own password…). But if you put in the right credentials, you will be given a access_token cookie. This cookie will let you stay logged in till the cookie expires (set for 30 mins). Once you are logged it, it will direct you to your account dashboard and show you all the stations you have access to. (I didn’t make that one yet but shhhhh)

Station List

When you click on the Weather Station tab, it will show you a list of every single weather station in the network. Something else cool, whenever you hover/click a button, it will change color. Just fun for me.

Save Barnegat Bay

I am building this system as a idea I had for the Save Barnegat Bay (SBB) non-profit organization. I utilize their weather stations all of the time for my projects that I love giving back to their mesonet! Once I am finished building this, you will be able to access the SBB weather stations and see weather points as well as graphs and stuff. Oop can’t spoil all of my planned features

Website Backend, JWT, and HTML Front End

Ok, these dev logs keep getting exponentially longer and longer ;-;

What the hell is JWT?

JWT (JSON Web Token) is a way for my authenticate a user. It essentially is how I can do logins. When a user is registered, their username and hashed password is stored in the SQL database (DB). When someone would like to authenticate, their password is hashed and sent with the username. The DB check if the username and hashed password match and send back a JWT token that expires after a set time (30 mins but configurable). During this time the user would be able access their station dashboard, etc. Admin accounts have the same log in procedure. They however have the power to give each user (based on their user id in the DB) access to specific stations. This includes the ability to see the station dashboard, toggle maintenance mode, and toggle if the station is public (on our site).

Backend:

Added functions for creating accounts and logging in. Most functions now require either a general account or a admin account. If this is not fulfilled a 401 error is sent (Unauthorized error). As of right now, only admins can register accounts. The access to each station can be updated by a admin at any time. I did some more stuff but I honestly forgot :)

HTML Front End

This was honestly my favorite part so far. I started by getting the Jinja2 template system to work as well as mounting the static directory. The layout.html file allows me to change it slightly for every page. That is a bad explanation but it is a really cool system. I got the home page done (needs some text and description but thats a later me issue). I am working on getting the general station page. It will list every station and allow you to click on them to get forwarded to that stations public dashboard.

Sanity Check?

I am having so much fun with HTML. It gives a certain gratification that SQL can’t give. I feel very happy :D

Any ideas pls let me know!

SQL Database, Website Backend, Dashboard

When I said a lot last time…that was nothing compared to now ;-;

What am I going on about?

Well, I wanted a way for station owners to be able to be able to toggle maintenance mode without the admin (whoever deploys this for their own mesonet) directly editing a json file every time. One thing led to another and now I am making a website…

Website Plan:

This website will run on a FastAPI backend with either Jinja2 or Htmx front-end. There will be a public “dashboard” where anyone can take a look at general station stats and current weather conditions. The weather conditions will also be made graphical with Matplotlib. Historical data will also be accessible. However on the same site, people will be able to log in. Logged in users will be able to access only certain stations they are allotted. They will have a more detailed dashboard with more stats. Certain users will be able to toggle maintenance mode and some won’t. Admin accounts will be able to see and toggle for all stations. Usernames and passwords will be hashed and saved in the database. The log in system will also make use of cookies/sessions to keep the user logged in.

Planned SQLite Models

users: (DONE)

• id
• username
• password_hash
• role (admin, station staff, etc)

user_station_access

• user_id
• station_id
• can_view
• can_toggle_maintenance

stations:

• id
• station_id
• is_in_maintenance
• is_public

station_status:

• station_id
• last_status
• consec_offline
• first_offline
• last_conected
• alert_sent

station_current_weather:

• temp
• dew
• humd
• wind_speed
• wind_gust
• wind_dir
• pressure
• precip_rate
• precip_accum
• uv
• solar

Progress?

I have made all the models and created basic routes. There is a CRUD (Create, Read, Update, and Delete) for the stations based on another config file. On start, the program will automatically create database entries for new stations and delete ones not in the config file.

Monthly Reports, CSVs, and a little bit of Insanity

Hey guys, I have done so much, I might go insane haha!

Reports?

Every first of the month at 8 AM, a email will be sent out to any emails configured for them. They contain various stats such as, which stations are down, which have been down for the longest, uptime percentage, etc.

Emails

So I just realized that I never explained all the different emails haha.

Recipients:

• Admin: Email(s) configured to recieve only system error emails
• Global Recipients: Email(s) configured to receive all emails sent by system concerning status.
• Report Recipients: Email(s) configured to receive monthly emails. E.g. Mesonet maintainers
• Station Recipients: Email(s) configured separately for each station. E.g. station owners.

Email Types:

• Offline: Sent after a station is recorded off 3 (default) times in a row.
• Recovery: Sent once a station is detected to be online after being considered offline
  Reminder: Sent after 7 days (default) to remind of the status.
• Monthly Reports: Sent at the first of the month at 8 AM (default) with statistics such as which stations had outages, the station with longest, uptime percentage for each station, etc.
• System Errors: Sent to system admins for any errors detected, HTTP, Requests, Time, etc.

CSVs

All of the data is logged on a csv as well. Not much to say

Insanity

Something about sitting around staring at a screen doing like 5 projects on and off makes you a little crazy…on the brightside it is my birthday tomorrow. That means more coding time! ;-;

Monthly Reports, Maintenance mode, and Reminder Emails:

Yolo!

Reports?

Every month (Rn configured to be the first of the month at 8 AM) a report will be sent for all of the stations with a bunch of stats such as:

• Total downtime
• Uptime Percentage

• of outages

• Table of outages
• Longest outage
• Station with most outages
• etc.

Reminder Emails:

Once a station goes down and it has not been resolved in a set amount of days, a reminder email will be resent to the owner

Maintenance Mode:

Once switched on, emails about status will not be sent for that weather station. Checks will still occur in the background however. In the future, I will create an FastAPI app that will let owners trigger this mode on/off for their station.

Suggestions?

If you guys have any other ideas, please let me know!

Error Protection and CSV Logging

Long time no see haha!

Progress

• Error Protection through try/excepts and error codes
  • Error codes were creates for the failure points.
  • Any time there is an error, it is emailed to the “admin” email.
• Every whenever there is an error/email sent/status the data like the station id/name, time, status, etc. is saved (appended) to a csv.
• All error retry logic is modifiable through the config.py.

IT IS WORKING!!!!

Oh my god man. The email notifications were a little annoying to get finished but they are working. The image is more redacted than a Call of Duty game but it is fine.

Progress

• Recovery emails have also been made.
• Everytime a station is checked, it’s “stats” (such as when their last offline was etc) are updated using a seperate json file.
• I have mostly spent my time on bug fixing and smoothing out the app.

Start!

Hey guys!

This project is all about improving the experience of personal weather station (PWS) owners. It is a webscraper checking for the status of these weather stations. After it detects it has been off for a while it will email the administrator of the network as well as the owner. (This is a summary, take a look at the project page for a better explanation ;)

Progress?

• The station are scraped and their status is recorded
• Json files for the current and past status as well as other things is updated/created when the program is run
• Email notifications (in work)
• Timestamps in the right timezone (config).
• Several stations can be checked at once
• And other stuff I kinda forgot ;-;