"I've attached the UIDraggablePanel script to the panel as well as a collider."
UIDraggablePanel goes on the panel, so the first step is correct. Collider though? Not sure how that factors in.
Object with a collider should have UIDragPanelContents on it instead.
When you try to drag one object, the entire list will move. That's correct, because you have UIDragPanelContents on it, meaning it drags the panel, which affects everything inside that panel. If you want it to be draggable by itself, use UIDragObject instead.