Making Exporting Functional

This will be the final session (for now) on the archive generator. There are still plenty of bugs, but after this session, it is at the point where it is actually functional, so I merged the generator into the main branch. According to GitHub, this is a Python project now.

Part 1: object Message has no attribute “person”

Most of the time in this session was spent trying to fix this attribute error. This is where using textual makes debugging slightly more difficult, as there is no terminal to print to. There is a debug terminal that can be attached, but it still is more complicated. I tried to make Labels pop up with debug information, but the never appeared or caused more errors.

Part 2: Logging

I got so frustrated with the attribute error that I added logging to the entire application. I started to write my own logging script, but then learned that Python comes with a logging library. I initalised it to use all logging level, although I only use Debug and Info. This helped fix a few other bugs, such as not actually writing messages to the output (once the attribute error was fixed). I used logging to print all the vars in the messages being processed; from this I found that none of the instance variables existed.

Part 3: The solution

I turned to Google to try to fix my problem. After Googling attribute errors, I saw a suggestion that passing the class, not an initalised instance, could be the cause. As it turns out, when adding a new message to the messages list, I had capitalised the “M”, adding the object not the instance. I spent quite a lot of time for a single character mistake. I’m glad I found it, but it took a frustratingly long time to find for how simple it was.

Finished?

The archive generator is not done. There are still bugs, it hasn’t been stress tested, the UI looks bad, and only basic messages can be created. But it is functional enough so that I can work on the actual application for the time being.

Producing Output Zips

It is about time to wrap up the archive generator, so it is time to actually generate archives. The basic steps are as follow:

1. Parse the Archives

2. Write the archives to the zip file.

Obviously the basics steps are not that complicated; Parsing the archives in the complicated part…

Part 1: Parsing Dates

I wrote a quick datetime parsing tool for converting to/from strings. The chat archives have a fairly standard format for dates, so it was pretty easy to implement. The only really weird thing is there is a single narrow no-break space character (U+202F) for some reason. All other spaces are normal spaces.

Part 2: Parsing Archives

parsing archives is split into creating group_info.json and creating messages.json. group_info.json contains the details for every person in the group. This includes their name, email, and if they are Human or a Bot. messages.json contains the data for every messages. For each message, there is a json object detailing who created the message (name, email, and user type), and further data of when it was created, the message text, and some ID values.

Message IDs

Each message in the main group gets its own topic ID; messages in threads use the topic ID of the message the thread is linked to. The message’s own ID is created using the Group ID, the Topic ID, and an ID unique to the message (if it is a top level message, this is the same as the topic ID).

#Part 4: Writing to Archives

This is a fairly simple step; the most interesting part is generating the archive name. I based it on the actual exported zip names by making the name chronovisor-generator_takeout- + the time it was exported.

Message Adding Menu

In this session, I added the menu for adding messages. This went surprisingly smoothly; I had to learn how to use the Select widget, but most of the other parts of the UI I had already used, or could piece together from other menus. The biggest UI section, the data and time editing, can simply be pulled from the group editing menus. Overall, this session went smoothly, and I was able to focus on how to do make person selection work.

Part 1: Person Selection

Each message obviously needs a person who sent it; for this I use a Select widget, which is essentially a drop down menu. After my previous problems with passing the wrong type into a field, I wasn’t exactly sure how to make the selection box work; I really wanted to pass Person objects in, as it would remove needing to use a person’s name as a key to try to find the right Person object afterwords. I tried it and… it totally worked. After overriding the str method to return the person’s name and email, the Select widget was functional. Sometimes it is nice when code just works.

Group Editing

I was going to start doing messages in this session, and realised that there was an important element that I was missing: group settings. This is important for one sort of sneaky reason: message history is dated. At the beginning of chat history, there is a small informational message that says when the chat was created, and then all messages must come afterwards. Without defining a set start time, there would be no way to validate the dates set when new messages are added. To go along with this, I also added the ability to set the name for spaces. I am also storing if the archive is a DM or Space in Archive objects so I know what fields to ignore when saving occurs.

Part 1: Date and Time Entry

Date and time entry is split into a box for each Month, Day, Year, Hour, Minute, Second. Each of these has a corresponding label to detail what each field is for. All of these are arranged in a Horizontal container so that they line up. I also experimented with data validation; I at first started to right my own validation classes, then realised that textual has a built in validator for numbers. There are a few potential bugs, the biggest is that days are not validated based on the selected month, so entering dates like February 31st is possible. Also, a year that is too late causes an error, it may be something to do with python’s datetime, but it isn’t a priority to experiment with at the moment.

Part 2: Name Entry

The type (DM or Space) is passed to the container; this is used to enable or disable editing the group. The was I have set this up, the name editing is below the save button name, but I really am trying to get features in and not waste my time on UI for the moment. It works, and for now that is what matters. And speaking of the Save button…

Part 3: The Save Button

The Save button is fairly simple; while editing, values are stored locally in the container. When the save button is pressed, these values are actually written to the correct archive.

Part 4: Silly Mistakes

I don’t know what I was thinking today; I made a bunch of silly mistakes. The worst was trying to pass an integer where a string was required. I was setting up the base for the date & time inputs, and as I was thinking about time, so I just tried to input numbers. I got back an error about translation; it took a bit to figure out what was causing the error.

People Editing!

In this session, I implemented the menus for adding, removing, and editing people from a DM or Space. I decided to make the editing menus popup menus to avoid having to redo the sidebar with more menu switches. To make the popups work better, I separated the styling for popups into their own tcss file from the licence popup styles.

Part 1: Editing Buttons

The first order of business was to add the buttons for adding, editing, and removing people. They are setup so that the first person cannot be edited or removed, as they are the main user in all DMs and Spaces. A maximum number of people is passed to the people editing widget- two people for DMs, and 400 people for Spaces. 400 is the limit for personal use, and I don’t think any more than that is really required.

Part 2: Adding People

The people adding popup has two labels, one for the name and one for the email. The default values are generated based on the number of items so there are hopefully no duplicates. The UI design has each input labelled, and buttons to cancel or save the new person. Designing this was really interesting, as I found that nesting containers using their constructor, not a with statement, allows event bindings to bind correctly to the main container. This nested structure is surprisingly quite a bit like the Flutter child/children structure.

Horizontal(
                Label("Name: "),
                Input(placeholder=self._name, id = "name")
            ),

In order to make the table reload, I just switch back to the menu with a switch command; this works, although that fact that there is no default menu so it returns to the void menu means it isn’t a perfect solution.

Part 3: Removing People

This was fairly simple; the index for the person is passed into the constructor, so the person’s name can be referenced in the confirmation message. An option to remove the person or cancel is provided.

Part 4: Editing People

This popup is in part a modification of adding people and removing people. The person’s index is passed into the constructor so they can be found, and text boxes are provided to modify the person’s name.

Part 5: Adding people to Spaces

I was originally going to stop at adding people to DMs, but I thought that the larger number of people in Spaces would make a better demo. I unfortunately hadn’t actually made it easy to add this to spaces; I just forgot I would need to make this work for the separate Spaces and DMs. Making this separate is required as I store DMs and Spaces in separate dictionaries, and the command to return to the correct menu needs to be different for spaces and menus. To fix this, I pass a type to the constructor, which is used to determine which dictionary to use, and which call to make when closing the popup

Adding the People Viewer

The purpose of this session was to add a way to view the people currently in the Space/DM.

Part 1: Person Storage

Up to this point, there was no way to actually store the created people. To fix this, I created a Person class, a simple object to store the name and email. I then added a list to the Archive class to store the people.

Part 2: Dynamically Adding Widgets

I am using a more complicated system than with User Creation to dynamically switch what you can edit; I am hoping that this will give me more control, and hopefully make it easier to make the UI look good. The new method is to dynamically add a container with the right widgets inside when a button is pressed. To this end, I created a class based on the VerticalScrollBar to control the person editing. I didn’t fully understand how to dynamically add widgets at first; my first idea was to have a instance variable containing a container that would be yielded in the compose method; I could then update the variable to update the UI. As it turns out, this is not how texual works; I did try a bunch of methods to refresh the screen, but then I learned that you can call a mount method to add new widgets dynamically.

Part 3: The DataTable

The final step to getting this to work was to add a DataTable. I originally tried to generate the table in the compose method, but I switched to generating it upon creation of the container, as that appears to be the intended way to do it. I am a bit disappointed that the data table is rendered far to the right; Another UI detail for later (actually for later this time) I guess.

Adding Licence Information

The main feature of this session was to add a popup box bound to “l” that displays the licence info. I opted to use the short licence info text that was designed to be at the top of source files, as I won’t have a popup on startup, and having multiple commands is not very user friendly. To go along with another bound command, I added the Footer widget to all menus, as it lists avaliabel commands. I also added licence comments to the top of my source files, including the dart files on the main branch.

The Licence Box

The texutal docs explain that the ModalScreen class can be used as the base of a popup message. Some tcss is used to centre and align everything; this took multiple attempts to figure out where in the container hierarchy to put the align properties. You can see that the button on the bottom is still not centred, but after failing for a bit, I decided that it looks fine. Another UI element to revisit later?

Binding the Box

I bound the box to “l”, as it makes sense for “licence”. i spent a little bit trying to figure out why the hotkey didn’t do anyhting; turns out it was a typo. It appears textual does not throw an error if you bind a function that does not exist to a key, maybe because you pass a string to it not ana actual function. Another feature I added was to make “l” close the popup box; whatever screen is on top takes priority.

DM Creation Menu Base

It is time to start on the menus for actually creating archives. Spaces and DMs have the same basic elements, so I am planning to create a few custom widgets that I can reuse for both of them. I decided that I would start with creating the DM creation, as DMs are a bit simpler than spaces.

Hub Menu

The first thing I created was a hub menu that will be used to create new DMs, Spaces, or Export the archive. It is fairly similar to the Main Menu, except that I used tcss instead of a container to centre the buttons. i think that this makes the code cleaner, and makes more sense with how textual works. I had a small problem where nothing was happening after I added content-align to my tcss file for the Hub Menu; as it turns out I needed normal align to align buttons.

DM Creation Menu

This menu has a basic docked sidebar just like the User Creation Menu. I am hoping that the plan of using single widgets that “plug in” will prevent UI issues like the user Creation Menu has.

ID and Archive Handling

I added a scripts folder with two small scripts:

1. id_handeler to create the IDs for each message archive

2. archive_handeling for classes to store archive, and, in the future, handle exporting

The ID handler contains a single Enum with values for DMs and Spaces. It also has a single static function to create a custom ID using Python’s uuid and base64 libraries, adding the correct DM or Space value based on the passed type. The archive handler currently contains two stub classes, one for storing archives, and another for storing messages. As more features are added, these classes will grow as well.

Attempting to fix layout issues

In my last devlog, I created the Create User Menu, and stated that I wasn’t a fan of how the UI looked, but would work on it after the rest of the app was done. Then I thought I knew how to fix it, so I spent some time working on the UI. I also added a text label for the input box, and docked the sidebar on the left side so other widgets won’t affect it.

Attempt 1

My idea was to take advantage of Python’s multiple inheritance and add the Horizontal container as a parent class; this would allow the Screen class to still be used for rendering, but everything would be aligned horizontally. This worked okay; it looked much like the attached screenshot. Unfortunate, the input was pushed over to the right.

Attempt 2

I didn’t like having everything pushed over to the right, so I thought that “maybe I can wrap the label and input in a Vertical container to align the label and input box; maybe that would allow them to move over and align with the docked sidebar. While the input and label did line up, it had two problems:

1. The input and label were still pushed over on the right side

2. Wrapping with a Vertical container is done with the Python “with” statement. This makes the Vertical widget capture all the events the input boxes generate

Attempt 3

Next I tried to use the grid layout. I have used the grid layout in Python Tkinter, so thought that it might be the way to go here. Using textual tcss, I setup a grid with one row and two columns, and didn’t dock the sidebar on the left. I attempted to make one column fill more of the screen than the other with grid-columns: 1fr 2fr;, unfortunately this didn’t work; this maybe could be a problem with how I read the docs; but I thought I understood how this works. They have an example of how to use the grid layout after all.

Conclusion

At this point, I have spent 49 minutes failing to make the layout look better. I was feeling frustrated, and realised that I was wasting my time. My goal is to make a Google Chat archive time machine in Dart Flutter, this is only a side part to make it easier to use and demonstrate. I fixed it up so it works, it is functional, and I am going to leave it. For real this time. Once the app is functional I might have a better idea of how to use textual to make a good UI, but my main focus should be making the main viewer.

Add Create User Menu UI

I added all the required elements for the CreateUsemenu, the menu for creating the “user” who owns the chats in the archive. The menu has:

• A side bar that contains buttons for selecting to edit the email or name, or selecting to create the user
• A input box for the actual editing

ContentSwitcher

I spent most of my time on using a ContentSwitcher to switch which editor can be used based on which button was clicked. I completely missed how their demonstration worked the first time I tried to implement it in my app.

Input Boxes

My second problem was in saving the input boxes’ values so they can be used later. I was reading the API and completely missed how you were supposed to implement the on_input_changed method. They listed the variables (and their types) which were available when receiving the input. i thought that I had to create a method in my class with those inputs, then got very confused why I got so many argument errors. It turns out that you use event: Input.Changed parameter just like the button.

Organisation

The organisation is not entirely how I would like; the input widget is at the bottom not next to the buttons. I tried to wrap them in a Horizontal container, but this messed up using the self parameter. I might come back to this later after the rest of this app is done.

Starting the Archive Generator

I started making an application for generating archives that will be able to be used with the main app. This solves two problems:

1. I will want to show the app working further into development without showing my personal chats

2. It will allow people who don’t use Google Chat to test the app

Design

I am using the textual library for Python to create this helper app. I am using Python not Dart as I am faster with it, and the main focus of this project should be a Dart app. I am using textual not just a simple console app because I saw textual and thought it looked like a really cool library

Progress

In this session I made the first two menus- the main menu and the create player menu. I was doing some experimenting with how I wanted to design the app, and eventually settled on separate Screens for each menu. When implementing the Screens, however, I didn’t fully understand the api, and couldn’t figure out why my buttons were not rendering. It turned out, I just had to install and switch to the screen, not just push to it or switch to it.

Setting up basic GUI

I made a mistake in my initial plan for this project. I wanted to get all the data loading parts done first as I have done json handling in other languages. This would have worked were I familiar with flutter. However, as I am still very new to flutter, I have been spending time trying to figure out how to process the data so it can easily be converted to flutter GUI; I am not sure what tools I have available as I just don’t have a lot of experience. Therefore, I decided I need to work on the UI along with the data processing. I created an app bar that moves the upload button onto it with an IconButton, and I am using the resizable_widget library to create the three panel structure of Google Chat.

Load archive from flutter app

In this session, I finished the basic button UI to trigger loading the archive. The first thing I did was to setup a MVVM structure for the app. I chose this as the flutter tutorial uses it for http requests, and I figured that file io posed a similar problem. I first tried to use async/await to prevent the UI from freezing while it loads the quite large zip file; I spent quite a while messing with Future, async, and await trying to get it to work. After my attempts failed, I turned to research, and learned that I actually needed to use compute to run the file processing in a background thread. this works quite well, as is demonstrated in the attached video. I also uploaded the Google Sans and Robot fonts to the project, selecting Google Sans as the default font. I want the project to look a bit like Google Chat in the end, so Google fonts are necessary

Linking archive loading to GUI

I first setup the archive_handeler package to be importable. I made a mistake and accidentally setup the package imports in the chronovisor folder, not in archive_handeler. It was fortunately not a difficult fix, but it makes the git history a bit of a mess.

After fixing the import, I added file_picker as a dependency, and added a basic button to use to create a file picker popup and linked the popup with the function to load the selected archive. This involved reading the file_picker docs; fortunately, the picker selection works on the web, as I am currently testing the app in a web environment.

Finish basic archive parsing

I finished the basic parsing of archives to retrieve the chat archives. Using the archive library, the archive is loaded, the entries in the Google Chat folder are located, and then they are split into each separate chat.

I struggled a lot trying to figure out the most efficient way to sort the files. Each file has a name that includes the entire path inside, and I needed to extract the files from the folder for each specific space/dm, separate the two data json files from other file data, and wrap the data and files in a MessageGroup object. My problem was that the easiest way to get all the files is to use an enhanced for loop, which makes it hard to sort out the files they way I set up the object. Eventually, I settled on storing the files for each dm or space in a List, which could be looped through later; I want to get the program up and running, and it can be optimised later.

I also spent a lot of time seeing what methods I had, especially on the ArchiveFile class that the extracted files are stored in, and just looking up APIs as I am new to Dart.

At the moment there is not much to see here, as i am doing the basic message parsing before starting the actual gui app.

Setup Chronovisor project

I setup the basic project structure; I have a folder for the flutter app, and one for the package to handle all the archive details. This did not take that long, so I also prepared the archive_handeling package to have archive as a dependency, and setup basic loading of a zip file.

Note: the time for this devlog is too high; I didn’t see the project show up of Hackatime at first, and so spent some time messing with the files trying to get it to show up, which inflated the time quite a bit. Oops.