Spent ages trying to figure this out with coordinate system conversions and linear equations. Turns out you can just use enlargements.

CanvasOS panning + scrolling working correctly