We ran into the UIDragPanel slowdown issue in our game project with 30 or more items in the list.
Simon's solution sounds effective and easy to adopt into many lists at once by abstracting the "pooling" logic into a Monobehaviour. However, if you can't get it to work, we tried an alternative approach which might work well enough for your purposes.
We attached a box collider and a kinematic rigidbody to a game object which would essentially act as the "render window" of the drag panel, as well as to each of the list elements of the drag panel. This let us hook into the "OnTriggerEnter" and "OnTriggerExit" events in a script attached to the render window. We would use these events to toggle the active state of the drag panel children to true or false depending on whether or not they were visible. To prevent some "popping" from occurring as the objects move in and out of view, we made the collider on the render window larger than the area that was actually visible to the player.
The box collider and rigidbody for the child elements will need to be on parent objects to the actual contents of the elements, which are what you actually set to be inactive. If you set the parent object to be inactive or put the collider on the same object as the contents, the collider would also get disabled when the object is out of view, preventing the element from reappearing when it should be in view. Also, the children should be disabled by default, since otherwise they will only be set inactive when they leave the render window, making the list sluggish until the user drags to the bottom.
It seems to work smoothly in the Unity editor, although we're experiencing some lag with it on the iPad (admittedly in a scene filled with other crud that's slowing the game down a bit already), probably due to the overhead of having rigidbodies on so many objects. One option we're considering to work around this is to use a normal drag panel on smaller lists and then switch to the box collider method when the list becomes sufficiently large.
Sorry for the wall of text -- hopefully this is helpful to anyone running into this issue.