So, the thing is that the draggable panels aren't all the good when you have 100s of items in them. While dragging all widgets are actively drawn which makes UIPanel.LateUpdate crazy slow.
I recommend you build a system where the panel only has X containers active (where X is the number of elements + a few buffer elements). We've been experimenting at work with doing this and it is quite a bit faster to have 10 active elements that shift than 15 constantly active elements the way they are now. Another advantage of such a system is that it scales infinitely (almost), while the existing system caps out quite lower. I've seen examples where about 100 children with each 5 widgets would be un-scrollable on iPhone 4S, which should run just about anything.
I think there has been these kinds of solutions on these forums before, but I don't remember where.