This is forced, and intentional. All widgets belonging to a panel will assume the panel's layer. The panel is basically one whole object. You can't have it made up of different layered parts.
If you want to use different layers, use different panels.