I'm guessing your collider is a child of the scroll view, so you end up moving it as you scroll.
It should be outside of your scroll view, not inside it. Look at NGUI's examples.
In the future, always check UICamera's Debug option first to see what's intercepting your events. It'll give you a solid idea of what's wrong.