Fixed it.
It was indeed the SpringPanel which caused all my troubles. Now while the SpringPanel is usually only enabled for a short time, it sometimes stayed enabled after I moved the DraggablePanel using MoveRelative().
Now the next time I hit my button to move the DraggablePanel, the SpringPanel would still be enabled and "counter act" my movement to a certain degree, resulting in the weird behaviour.
So for anybody else trying to implement "up/down" or "left/right" buttons for their scroll views, make sure to disable the SpringPanel before you call DraggablePanel.MoveRelative()
I wonder how nobody had this problem before, as I'm sure having scroll button like this is a rather common feature, isn't it?