The panel visibility check is needed. It's what prevents events from reaching widgets that have been clipped.
You haven't answered my last question. What is under the mouse when the events don't work?
Keep in mind that NGUI's raycasts are designed to hit widgets, sorted by depth. So although raycasts hit colliders, it's the widget depth that's used to determine what's in front. If you have a collider on a plain game object without a UIWidget on it, then NGUI will run through all the widget children and use the highest depth instead. If you want to control it, make sure that colliders are attached to widgets. (ALT+SHIFT+W to make an invisible widget).