After I made my rows, I started to work and learn about rotary encoders, initially, I had made a 6x6 matrix, but I realized that I only need a 4x5 matrix, and I leave the other 2 rows for my rotary encoders.
It took me a while to figure it out, because I was new to making schematics, but I luckily got some help on the slack, and then made a schematic like so: