Right, and that's what "clip raycasts" option takes care of. With that option on, you can't click on colliders that are not visible.
Check the scrollable panel example that comes with NGUI and try to drag from outside the clipped region. You won't be able to.
