The one and only thing you should worry about is texture memory. You need to ensure that you don't keep atlases in memory that are not being used. More importantly, if you have UITextures in your scene, same thing applies -- don't keep their textures in memory if you are not using them.
Everything else is negligible, provided you don't keep too many things active. Hiding things off-screen but keeping them enabled is a bad idea. Disabling parts of the UI you don't need is a better idea.