Either that, or write a script that walks the hierarchy on Awake() and stores handles to every object before you disable anything. This means you'll want to have your panel start out of camera view (say -5000 X or something) and finish initialization with disabling any controls you want, perhaps even the parent panel, and then moving the panel via transform to the location you really want it.
There are a *lot* of different ways to approach object management. You might think about creating a new test project, or a test scene, and then doing a bunch of experimentation before you spend a lot of time modifying your "live" project or scene.