Just to clarify the issue to make sure there is no confusion.
I have a panel which I dynamically populate with buttons, those buttons also have buttons dynamically added to those as well.
Example
Panel
--Group Button
---- Sub Button
---- Sub Button
--Group Button
---- Sub Button
This is very similar to your quest log example.
Now the main bug is that I've added a scroll bar for when the list gets to large, which works fine, as I attach more buttons it shows the scrollbar and works perfectly. When I delete buttons if I am at the top of the scroll bar it is fine, but if I am at the bottom of the scroll bar it wont update, remaining in the same position, leaving blank space at the bottom where the button used to be. Now if enough buttons are delete to make the scroll bar disappear, then you have a huge gap at the bottom, your top elements are all cut off from the clipping plain and there is no way for the user to scroll back up.