HTML parsing is now complete

its been nearly 2 weeks and i have great news: html parsing is complete
what this means is that html blocks get processed into token streams like the one’s markdown-it makes. now, i must clarify that these html blocks were already in a token stream but they were distinctly html, having the type html_block, and if i didn’t convert those html blocks into markdown-it’s _open/_close tokens, i’d have to parse it in the file that styles the tokens, and many html tags do the exact same thing as their markdown counterparts so i could reuse some functions. i must also clarify that i have NOT done anything for stylizing html in the stylize.js file, which deals with processing text formatting and image processing

next up: stylizing html!

Changelog

• (c5a5ce0) Finish HTML parsing
• (841ba42) Start work on HTML parsing

improvements and fixes to inline rendering

this is way earlier than my normal schedule, but that’s because i’ve got something to show. basically, inline rendering was bugged, and not only did i fix it, i also made it so that it works better! that’s all you need to know, do NOT read the rest of this if you don’t want to be bored to death, its a really lengthy explanation of this

the initial bug was that inline rendering wouldn’t work at all. i figured out the problem was that i wasn’t pushing "inline" to the state array somewhere. i realized that it wasn’t working when called in the table rendering function, and that was because i wasn’t pushing "inline" to the state array.
‎
the reason for this is because the renderInline function slices the state array at the index after the "inline" string, so if its something like ["paragraph", "inline", "bold", "italic"], the function slices at index 2 because "inline" is at index 1, and we get ["bold", "italic"]. the problem is, if "inline" doesn’t exist, running indexOf("inline") returns -1, so it slices at -1 + 1, which is 0, so its just the whole array. but anything before “inline” isn’t a valid style, and since the function for that is undefined, it goes kaboom and throws an error saying that it’s not a function.

also gpt found 5 other bugs but they’re kinda irrelevant

the fix was making it so that the renderInline function itself pushed "inline' to the state array instead of depending on it being already there. this also removed the need for making sure that the inline string was already there. i might do this with the rest of the functions too actually also removed unnecessary state.pop() occurrences since the pop method is destructive and can mess up everything

as for the improvement, you might see in the first screenshot that there’s a bunch of text tokens together. yk, they could be squished together into one token… oh right, that’s what i did! i changed the renderInline function to make a variable where the text accumulates until there’s no more text or if there’s an image, on which it gets pushed to the contents array

now the most important thing is that there’s only one more thing left to do: html rendering. this is the last thing before i get to actually rendering stuff on the screen! this is exciting!

Changelog

• (9de477d) Fix, improve inline rendering

please like this took more than an hour to write

THAT WAS SO LONG WHAT ON EARTH

in short, tables now work and the inline rendering function is fixed! this took a while because i genuinely had no idea what i was doing. now the long stuff:

tables go brr

i got distracted while writing this and realized everything’s actually borked, but tables do somewhat work! its still borked tho–

also i realized that my whole code is borked, but…

inline rendering includes links now!

still have no idea how i missed that

other

i just changed image function so that it returns an object with a prototype function instead of unnecessarily repeating the function as a property of all the image objects, so if there’s 10 pictures in a file, there will only be one function to render each as opposed to 10 instances of that function

Changelog

• (d8feb46) Add table support, fix inline rendering

oh fuck, my code is COMPLETELY borked

PEAK CODE

gng TS IS PEAK!!!!

i added a function that renders code AND ITS PEAK JUST SEE THE SS

now some of this time came from trying to figure out why my code was joining the contents of a function, and it was because i forgot to add the argument that gave the coloring function (chalk library) the text to color. hey, at least i found a way to read the contents of functions time to go steal code from closed-source packages

leaving that aside, my next step is thematic breaks. this shouldn’t be too hard, in fact, i’ll do it rn since it’s that easy. rn its 8:49 PM GMT+0300, i’ll tell in the next paragraph when i finish it.

[8:54 PM] just did it, it only took like 2 mins, the other 3 were because i found a bug that never removed codeBlock from the stack after it was added.

that’s about it, and this devlog was quite short. well, comparatively, to my other devlogs, after all there’s people posting 4 word devlogs. you should follow this project tho

Changelog

• (a17bd2b) Add thematic break support
• (e28f1b1) Add code block styling

when the FUCK did this become 4 hours

so uhh i spent like all of this time debugging and improving my code.

first thing is, i replaced these “microfunctions”, as i call them, with
an object that had keys whose values were arrow functions. these arrow
functions did the exact same thing and took up less space and was even
more readable!

second, i spent some time on a stupid bug where the main function in
stylize.js would hang. i thought it was an issue with the image
function, but after half an hour or so, when i added console.log(done)
right before the return, i realized it had to do with the main function
that was calling the image function, and sure enough, the second i
looked at it, i realized the problem: it was a while loop and i forgot
to add a line that incremented the index, so it just kept doing the
same thing forever until i killed the process oh well, more hours for me

third, i improved (or rather cleaned) my image function. it returns
promises and doesn’t use terminal-image directly. instead, i made it
so that it would return an object with a bunch of keys, one of which is a
function that takes the whole object as an input. this function
actually renders the image, since i plan to write the function to
display it in a way that deals with image rendering after rendering
text, and while the images get rendered, a spinner takes their place
temporarily, but the images replace those spinners once they are
processed

fourth thing is, i made it so that the main function calls the
renderInline function instead of the paragraph function. in fact,
this paragraph function is completely useless, so i went ahead and
removed it too.

lastly, i upgraded a bunch of packages:

• upgraded typescript from version 5 to 5.9.3
• upgraded got from version 15.0.7 to 15.1.0
• upgraded markdown-it from version 14.2.0 to 14.3.0
• upgraded shiki from version 4.3.0 to 4.3.1

if you wonder what these packages are

• typescript is, well, the typescript language, not sure why i have
  it, might be a dependency for another package.
• got is some package that does pretty much the same thing as the
  default fetch() but it can give me a buffer, which i need for images
• markdown-it is the markdown parser
• shiki is for coloring code. fancy, right?

Changelog

• (9118913) Improve code
• (5b9c953) Remove test file
• (ab3af08) Test image URL parsing on GitHub

aight this took me 2 hours to write and i started coding again in the
middle of writing this

oof! that was a lot of work, but now we’re REALLY getting somewhere!!!

• first thing, headings actually work now, and if there’s a link in the
  heading, it gets displayed right below the heading! tbf i did rip the
  heading thing straight out of the original index.js file when i was
  trying to make a markdown parser using regex

• secondly, and the more important thing, image rendering now works. pretty big milestone. i achieved this through the terminal-image npm package. however, this took a while because i couldn’t figure out why it wouldn’t render. the problem was tmux–it was stripping the codes it didn’t recognise so i implemented another system to detect if the terminal had color and/or image support to decide whether to use ansi blocks or actual images. now, it works! ts so peak fr

tl;dr

1. headings work (i ripped them out of my original implementation in
   index.js)
2. images work (took me a while because it didn’t work properly in tmux
3. we getting somewhere with this one 🔥

Changelog

• (9941b1b) Finish image rendering
• (007b3f4) Work on image rendering
• (609be18) Make heading work

holy this shit is so long

heyo! this is a long one, so i’ll go over this commit by commit

• (0ad2cb2) Start paragraph parsing

  here, i did these:

  • i made a global string state to maintain the nesting while styling. managing this was quite tedious and useless
  • i added a helper function to get the style of a token called getStyle(). i’ll explain the use for it below.
  • i continued working on the heading() and paragraph() functions as well as the stylize() function

• (eb218d2) Add markdown extensions
  as for this one, i added a bunch of packages, mostly extensions for markdown-it. these are:

  • markdown-it-footnote
  • markdown-it-github-alerts
  • markdown-it-sanitizer
  • markdown-it-table
  • markdown-it-task-lists
  • shiki (for code highlighting)
  • terminal-image
    i removed a few of these later because they were either not really working properly or just weren’t necessary.

  besides these, i also configured the plugin to use the plugins, enable the autolink extension and also replace all HTML line breaks (<br>) with double newlines (\n\n)

• (5ecef2c) Start working on image rendering and colors/styles

  now here is where i found a gem in my dependency tree. you see, i was searching for a package to handle color rendering since i sure as hell wasn’t gonna deal with the hex codes shiki, the code-highlighting plugin, was giving me for the colors of the tokens. so i saw a package called chalk and it sounded familiar. turns out, shiki had chalk as a dependency, so it was already installed!

  for that reason, i added a chalk import in this commit. i also started working on an image function to render images, and named it renderImage(). also added a great error message if two requirements weren’t met:

  throw new Error("WRONG TOKEN IDIOT DEV");
  

  but that was all i really did in that function

• (0dc393d) Remove unnecessary object processing and dependencies

  this is where i removed markdown-it-footnote and markdown-it-table because i realized there just wasn’t a point in those since they weren’t letting me test my function, they were erroring out for some reason and i found out markdown-it-table already had support for markdown tables. i also removed the mapping function in the parser stage that i made to give myself only a few fields. it was pretty useless since i kept randomly finding that i’d need some new property.

• (8a5200c) Continue work on inline parsing, add terminal-link

  latest commit, this is where i did a lot of work:

  • i removed terminal-image as a dependency since i found that bun already provided a class for it (Bun.Image) and it also had webp support unlike terminal-image, so no point in another package when a built-in one works better
  • i made the getStyles() function better
  • i added a bunch of micro-functions like bold(), italic(), etc. i also renamed renderImage() to image()
  • i combined two if statements into one in this image() function because why 2 if statements when 1
  • i did a lot of work in the renderInline() function. now it uses computed function names that call those micro-functions.
  • i also changed that global string state to a global array, made things much easier.
  • i worked on the heading() and paragraph() functions further

that’s about it gng, also its 12:28 am for me rn this is why we devlog more often gng ✌️

toiling away…

toiling away at this code…
today was a bunch of stuff

• i added a renderInline() function to render stuff inline. i still need to add images and links, but i’d need a lib for links, and maybe for the images too
• i worked more on the main function. right now, im trying to get paragraphs actually rendered, which will take a while.
• i also added a heading() function that will render headings, emphasis on will because its not being called yet
  it took me a lot of thinking to get here so that part wasn’t too fun (a person who thinks all the time, has nothing to think about 🔥)

changelog

• (e24a192) Work more on the stylizer
• (1a02163) Start actual work on stylizer

revamp!

i revamped the parse-input.js file! now, it has two functions:

1. giveFields(obj)
   this is a function called by the second function. since markdown-it
   (the parser im using now!) has a lot of fields that i don’t need. so
   this function returns a mapped array with the fields type, tag,
   nesting, content, and children. as for this children field, the
   children field is an array of objects, where the objects are just like
   the parent object, with lots of unnecessary fields. for those, i used
   recursion! (it’s my first time using recursion
   )

2. parse(input):
   this is the function that i’ll be using to get the tokens i need to
   style! what it does is pretty simple:

• initialize a new instance of the MarkdownIt constructor (? i’m not
  too sure if it’s a constructor)
• get markdown-it’s tokens using their parsing function
• make a constant whose content is an array of objects returned by the
  first function (giveFields() if your memory is that terrible). again
  its a mapped array that gives only select fields from the markdown-it
  tokens
• return that constant

in other news, i wrote 2 more test markdown files. one tests the
behaviour of headings and the other tests the behaviour of inline stuff like bold and italic text!

Changelog

• (88bad121) Revamp parse-input.js
• (3eda61e5) Add more stuff to inlines.md

FUCK

fuck this rewrite, i’m abandoning this shithole of a parser i’m trying
to make, i’ll just go use an already existing parser, its just not worth it
it’s too much hassle to make an in-house parser, so here’s my plan:

• i install a markdown parser that returns a tree of text, not html, and as a dependency
• in the styling part, i’ll go ahead and read the tree of text that the parser returns
• it gets passed to the renderer
  hopefully this will move much faster now!

rewrite

i started a rewrite! what i was doing earlier was honestly too messy to actually use.
my earlier way was making a bunch of regexes and using those to match the remaining text. my new method is still using regexes, but i split the whole thing into two parts: block parser and inline parser.

block parser

• the block parser splits the markdown doc on newlines and checks what it starts with or ends with.
• when it finds that some line is not a paragraph, it pushes a token to the tokens array indicating that it is the start of a block. when it detects that the block has ended, it pushes a token to the tokens array indicating the end of the block.
• this is currently only implemented for headings, more are coming. it took this long because me dumb

inline parser

• the inline parser loops through the existing tokens array made by the block parser and looks for characters that indicate the start of an inline token like bold, italic and so on.
• like the block parser, it will push start and end tokens to the tokens array
• the implementation of the inline parser has not started yet

new character

i added a backslash to the giant block letters i made for headings!

Changelog

(bac8c5a) Start parser rewrite
(664713e) Add backslash in heading

help me what on god’s green earth is this

so after dealing with a shit ton of regex and doing pretty much nothing with it besides some stuff that would improve performance, i have decided that regex is not the way. instead, i’ll just set a bunch of flags with empty content and make a separate function that processes everything. i think the styling function will handle tokens because that’s its job, to style the tokens. at least i have something to follow now. (i did not implement any of this yet)

Changelog

fc8de72: Fix some regexes
fe6eeef: Change some regexes
f67c3b5: Change matching system in parser
8a80409: Add some HTML support in lexical parser
0727d82: Finish lexical parser (almost)
b7bc45b: Make lexical parser work

i got bombed by my own regex

anyways, so i started making a lexical parser and uhh its not going all sunshine and rainbows. the blockquote regex doesn’t really work right and also i checked the gfm spec since i want to adhere to it and uhh fuck i might be cooked. i thought ts was going to be a 20h project help meeeeeee

also yes it took me 6 hours for 74 lines of code sob this is just great

HEADINGS NOW RENDER!

headings now render! as you can see in the screenshot, it now renders the headings as big boys also, me very happy.
this took a while because i lowk didn’t know what to really do. i didn’t think about how i would do it, i was only thinking about doing it anyways gn gng

took only this long to make a bunch of letters
you see, i had to make a bunch of custom glyphs to render headings in the terminal. making that for pretty much all the characters on your keyboard isn’t quite easy but hey its done now. the thing is, this is only for level one headings. once i finish the actual parser, i’ll add glyphs for the rest of the headings. in the meantime, i’ll have to do the rest. i did not expect to have spent this long on the project already yk

a terminal markdown reader that has actual headings? never could have thought of it huh
well this is what i’m doing. literally a markdown reader that prints text the way you might have written it.
anyways so this is gonna use bun since bun is ultra fast. also this might be a 10-20h proj, shouldn’t be too hard.
well i like more hours, so i might also add stuff like:

• browsing all the markdown files in a directory (and subdirectories)
• editing (very unlikely!)

well i don’t really know what else to put so ig that’s it? anyways i have actually done some work, that is, my file currently ouputs some text. literally the raw original text. what’d you expect from 24m of coding