Even more abstractions please

Added generics support, which allows the user to define generic functions/structs that apply with every type, for example:

warp <T> void forEach(T[] a, (T) -> void action) {
    for(T t in a) {
        action(t);
    }
}

Depending on what type you used to call this, a new version of forEach gets generated, if you called it with an integer:

warp void forEach@int(int[] a, (int) -> void action) {
    for(int t in a) {
        action(t);
    }
}

Of course I also allow this on structs:

struct Triangle<Num>(Num x1, Num y1, Num x2, Num y2, Num x3, Num y3);

Also did some small changes:

• Added an FPS counter to my test code
• Better optimizations that actually detect not(not(x))
• Optimized triangle::fill even more (+2 fps)
• Made DynamicDispatchHandler actually use the apply return properly (it was always returning true)

More abstractions please!

Added dynamic function dispatch which allows for functions to take in
other functions as arguments, of course, this isn’t possible at all in
normal scratch, so actually FunctionLiteral gets turned into an
integer and a custom dispatch function gets created for every function
literal’s signature which allows for a binary search to call the actual
function

Also made HeapConversion reuse stack slots instead of allocating one
for every variable(mostly by AI)

Refactored the entire type system to use a sealed interface instead of an unsafe single class

Nothing much to say, this doesn’t affect the input code or the output project, I did just cleanup

Did more syntax improvements

Added ++ and – support:
index++;

Added type inference support:

auto a = Triangle(5, 15, 60, -120, 240, 67); //type automatically becomes Triangle
auto a = func(); //type automatically becomes the return type of func

Scratcher is at 100 commits! (Now 104 cause all of these refactors are different commits)
Completely refactored all translation steps to use the ASTVisitor.
Also did type resolving cleanup.

Function Reachability: Went from 117 to 32 lines
Function expression lowering: Went from 439 to 293 lines
Re Parse Local Variables: Went from 34 to 27 lines
Remove Empty Allocations: Went from 97 to 79 lines

ConvertToHeapAccess: Biggest refactor of this whole devlog, went from 500 lines to 418, split into 5 files

Fixed a lot of bugs with when expressions, also got deepseek to fix some of them

Added If expressions, which allow using if statements inside other expressions, for example, here is some previously invalid code that is now valid:

int max = if (a > b) a else b;
str  message = if (score >= 90) {
    looks::say("something");
    "You win!"
} else {
    looks::say("something");
    "You lose :("
}

 triangles[15] = Triangle(
        utils::random(-240, 240),
        utils::random(-180, 180),
        utils::random(-240, 240),
        utils::random(-180, 180),
        utils::random(-240, 240),
        if(mode == TriangleRenderMode.OFF) {
            5
        } else {
            triangles[14].y3
        }
    );

These still get lowered properly and just look like normal if statements in the final scratch code

Added more syntax improvements!

Else if: Added support for else if, which replaces big blocks

//Before:
if(triangles[14].y1 >= 15) {
        looks::say(">=15");
    } else {
    if(triangles[14].y2 >= 155) {
        looks::say("<=15");
    } else {
        looks::say("Both false!");
    }
}
//After:
if(triangles[14].y1 >= 15) {
        looks::say(">=15");
    } else if(triangles[14].y2 >= 155) {
        looks::say("<=15");
    } else {
        looks::say("Both false!");
    }

Compound assignment: Added support for compound assignment

//Before:
triangles[15].y3 = triangles[15].y3 + 5;
//After:
triangles[15].y3 += 5;

Enums: Added support for enums, which are just fancy integer wrappers

enum TriangleRenderMode(OFF, RENDER, ITHINK);

TriangleRenderMode mode = TriangleRenderMode.OFF;

When: Added support for when expressions/statements that allow for matching a subject:

//Before:
if(answer == "off") {
    mode = TriangleRenderMode.OFF;
} else if(answer == "render") {
    mode = TriangleRenderMode.RENDER;
} else {
    mode = TriangleRenderMode.ITHINK;
    looks::say("If thinking is your power, what are you without it?");
    return;
}
//After:
TriangleRenderMode mode = when(answer) {
    "off" -> TriangleRenderMode.OFF
    "render" -> TriangleRenderMode.RENDER
    else -> {
        looks::say("If thinking is your power, what are you without it?");

        TriangleRenderMode.ITHINK;
    }
};

Tried to do turbowarp only return optimization, wasn’t able to make it very good… reverted it

Improved the structure of CompilationConstants
Added string::split and string::substring
Made some of the list functions inlined (itemAt, clear, length)

Made the garbage collector treat top level variables as roots, this prevents top level variables from getting freed

I added 2 modes to this

• Reflect: This uses a hacked block to read the variables by name and reduce script size if theres a lot of variables(this doesn’t render properly inside the editor but it works)
• NoReflect: Get all the top level variables at compile time and mark them one by one

Garbage collection

Added a mark and sweep garbage collector to my language, it walks the roots, marks every used object, then frees all unused objects!

Most impressive part of this(in my opinion) is that most of the garbage collector is actually written in my language

Misc

• The garbage collector gets called automatically every 1 second, but you can turn it off (CompilationConstants.AUTOMATIC_GC)
• If you want to manage memory manually, you can turn off the garbage collector entirely (CompilationConstants.MANUAL_MEMORY)

Added a name parameter to alloc which acts as runtime type metadata, soon to be used.

Added more compiler sugar for lists:
list[index] replaces list::itemAt(list, index)

list[index] = ...; replaces list::replace(list, item, index)

for(type name in list) {} replaces

int len = list::length(list);
int i = 0;
while (i < len) {
    type name = list::itemAt(list, i);

    i = i + 1;
}

(i am not proud of the shitcode that was used to get for(type name in list) working)

Added list support! A very barebones implementation that allows the user to create, add to, read from and delete items from lists!

Made “PromoteToGlobals” which is a more powerful version of OptimizeToGlobals which allows for recursive functions to have their locals turned into globals aswell

This is something called liveness analysis, the compiler looks at the code paths and decides at which points a variable is “live”, which means it can’t get overwritten by a recursive call
Example:

warp int fib(int n) {
    return 0 if n <= 0;
    return 1 if n == 1;
    if (n == 2) {
        int b = n * 2;
        return otherFunc(b);
    }
    return fib(n - 1) + fib(n - 2);
}

In this function the “b” variable can be turned into a global because it cannot get overwritten by a recursive call, its used too early for that to happen

Did some compiler testing
Fixed function inlining bugs
Fixed a single use assignment bug
Created a triangle library that allows for triangle rendering using the pen

Borrowed the “return expr if(cond)” syntax from ruby
Implemented Tail call optimization which optimizes away recursion on tail calls

Example:
Before:

str count(int amount) {
    return "Done!" if amount == 1;
    return count(amount - 1);
}

After:

str count(int amount) {
    int tco@argument@amount = amount;
    while (true) {
        if (tco@argument@amount == 1) return "Done !";
        tco@argument@amount = tco@argument@amount - 1;
    }
}

Source code of the image:

on GreenFlag {
    if(a() == 5) {
        looks::say("yo!");
    }
    int index = 0;
    index = index + 1;
    rgb(index, 5, 5);
    index = 5;
    while(true) {
        pen::eraseAll();
        render();
    }
}

The image is of the entire flow working perfectly:
The index variable had their value propagated to “1” by the Sequential constant propagation optimization
The variable got its uses removed by Inline single-use assignments
Then the variable got fully removed by Dead store elimination
The rgb function got inlined.
Then while it was inlined, because the rgb function’s return wasn’t being used, it got fully removed(the inlining preserves side effects while removing its return value, but because it didn’t have any side effects it got fully removed)

Same with the a() function:
It got inlined
Then the if statement became

if(5 == 5) { looks::say("yo!"); }

Then Constant folding turned 5 == 5 into true
Then Dead code elimination turned if(true) into just the then block

I know most optimizations look like they aren’t that powerful, but when combined with other optimizations and ran in a loop, they can pile up and actually optimize code

Optimizations

Simplify boolean equality: Simplifies x == true to x, simplifies x == false to !x
Simplify double negation: Simplifies !!x to x, simplifies --x to x
Constant folding: Simplifies true == true to true, simplifies 5 + 5 to 10
Repeat to While: Converts repeat(amount) to a while loop
Dead code elimination: Removes provably unreachable code blocks, eg: if(true) { ... } else { .. } turns into ..., removing the if and the else block entirely

Now the big ones

Sequential constant propagation: Analyzes the control flow of your program, removing useless variable assignments and inlining their expressions
Inline single use assignment: Self explanatory
Dead store elimination: Removes useless “set variable to x” blocks that don’t get used afterwards
Function inlining: The compiler evaluates a cost for every function call is inside a function, if that total cost is less than 10,000, it gets inlined inside that function, removing call overhead and allowing for further optimization

Other misc stuff:

Fixed the while bug (if you called a function inside a while statement, it would only get called once)
Made function lowering not allocate a stack slot if we’re ignoring return
Made the OptimizeToGlobals optimization be called last to allow variable optimizations to happen first

Optimizations

OptimizeToGlobals: If the compiler can prove that a function is not recursive and runs atomically(single threaded), it gets rid of the function’s primitive local variables, which improves performance as scratch’s list operations are more expensive than local variables
InlineSingleUseVariables: Pretty self explanatory, if a variable is only used once, it will be inlined

ConvertToHeapAccess: I did optimizations on the normal heap converter so that it doesn’t allocate a stack pointer for functions that don’t need one

Images don’t show the OptimizeToGlobals optimization cause the testing function I have for that is too long
Before and after on the RGB function

Created Motion and Pen library
Added support for top level variables
Made an optimization to get rid of useless “replace with -1” blocks(image 1 is before, image 2 is after), this greatly reduces the code size(images are only one example, it was being put everywhere)

Did warning cleanup
Added a nullability “system”, I allow the ? to be put on struct fields, and you can put null as the argument on those fields, then you have to check if they are null and do !! to use them

Did a lot of refactors
Created SensingLib and CalendarLib
Made a system to allow standard libraries to be inlined into your code to reduce file size
Added support for structs,
Right now you have to manually free your structs

Created a library for casting primitives to other primitives
Expanded the math library a lot
Added duplicate function and struct detection(yes I forgot before)
Added code to prioritize function(int) over function(float) if calling with an int

Added “warp” and “export” modifiers to function declarations, the warp modifier maps to the “Run without screen refresh” field, and export is currently unused, but later it will be used to export a function so that its properly usable from normal scratch code

Did general bug fixes
Added more standard library functions
Added runtime type checking and exceptions

Did a small refactor on obfuscation
Created a type safe(kinda) kotlin DSL for making standard library functions way more readable and simpler to create
I rewrote the Memory library with the kotlin DSL
Image 1: Before
Image 2: After

I added an event system, making it so your code can actually run

on GreenFlag {
    main();
}

on KeyPressed(W) {
    looks::say("W was pressed!");
}

These events are translated into functions and the compiler generates
scratch events that call these

on KeyPressed(W) {
    compiler@eventlistener@F();
}

void compiler@eventlistener@F() {
    looks::say("W was pressed!");
}

The compiler also generates a initializer green flag block which clears
the heap and reserves an index for every other entrypoint(image 2)

I finished the pipeline from my language’s AST to scratch, it was pretty mind numbing

Stage 1: The code does reachability, from the source file(where the compilation started) and finds all functions that we can call from the source
Stage 2: Using those functions, we lower call expressions to statements and returns to heap accesses, here’s how that looks in code:
Before:

void main() {
    looks::say(fib(5));
}

After:

void main() {
    int fibReturn = -1;
    fib(5, &fibReturn);
    looks::say(fibReturn);
}

Notice how the nested call turned into a seperate statement. Why do we need this? Because scratch does not support returning stuff from functions, only arguments.

Stage 3: We re-parse all locals on the reachable functions because the last stage added more locals

Stage 4: We add a parameter to all functions named “stack”

Stage 5: We add free(stack) calls at the end or before the return statement, this frees our stack automatically.
We also add alloc before function calls so that we have a stack to pass to the function
Here’s how that looks like in code:
Before:

void main() {
    int fibReturn = -1;
    fib(5, &fibReturn);
    looks::say(fibReturn);
}

After:

void main(int stack) {
    int fibReturn = -1;
    int fibStack = -1;
    alloc(UNKNOWN, &fibStack); //allocate a stack for fib
    fib(fibStack, 5, &fibReturn);

    looks::say(fibReturn);

    free(stack, UNKNOWN); //free our stack
}

Notice how we have UNKNOWN in place for a number in the allocation and freeing, this is because we just added more locals for the allocations and we can’t know how many slots we need.
Standard library functions are already compiled to scratch statements so they don’t need a stack allocated for them

Stage 6: We re-parse local variables so we have a count.

Stage 7: We count the re-parsed locals for every function and figure out which local variables can be reached in every function

Stage 8: We convert all local variables to heap slots
We traverse all statements and expressions, replacing all variable accesses with heap[stack + variableIndex].
While in this process we also convert the unknowns into the actual sizes since we just counted the locals in stage 7
Here’s how this entire stage looks like in code:
Before:

void main(int stack) {
    int fibReturn = -1;
    int fibStack = -1;
    alloc(UNKNOWN, &fibStack);
    fib(fibStack, 5, &fibReturn);

    looks::say(fibReturn);

    free(stack, UNKNOWN);
}

After:

void main(int stack) {
    heap[stack] = -1;
    heap[stack + 1] = -1;
    alloc(3, stack + 1); //we just pass the index here since alloc will write to that index
    fib(heap[stack + 1], 5, stack);

    looks::say(heap[stack]);

    free(stack, 2); //We have 2 locals
}

By using a stack we get access to a lot of things like returning, recursion and local variables. Scratch doesn’t even support local variables, so we have to hack it with a global list.
Notice how we don’t free the allocation? That’s because all functions are compiled by the same compiler and that compiler added free() to all functions with a stack.

Stage 9: If a function doesn’t have any local variables, this is where we delete their allocations and free’s so that they don’t mess up the heap
For example:

alloc(0, index);

and the target function for this allocation will have its free(stack, 0) call removed.

Stage 10: Finally we create dummy scratch functions for our very lowered AST

And finally in stage 11 we create the code blocks for the scratch functions and write to disk

And it successfully calculates fib(9) as 34

I made a wrapper for the raw opcode classes that make valid code easier to make, then I did a ton of bug fixes and did a small refactor

Did a refactor, and intellij decided to corrupt some of the files, so I had to fix it

Implemented 90% of scratch’s blocks in the wrapper

Worked on a very basic scratch wrapper, it can generate a recursive function, gonna add more scratch blocks so that actual logic can be generated in this wrapper

Added static type checking, which makes sure the correct types are used everywhere.
It prevents the user from using a number in an if statement’s condition.
It also prevents you from assigning a string to an integer field

I added the AST parsing, which is able to parse the ANTLR parse tree into an AST (sorry the image is bad, its all I can show)

Worked on the first stage of AST parsing: Only parse top level elements (imports, structs, variables) and don’t parse the insides of functions