I circled back to this on Friday, and it turns out rotating my panel's not going to work. I've just got too many components, so they end up fighting with each other or causing weird visual artifacts when rotation is involved. So a co-worker and I were looking at the shader today trying to figure out exactly what was going on with clipping and scaling, and one strange thing that we've noticed is that the input vertex (v.vertex in the vert function) changes based on whether or not the panel is uniformly scaled.
I'm not sure if this is fully generalizable, but with a very simple test case (a 512x512 sprite), the vertices passed in to the shader were always +/-256 as long as the panel was uniformly scaled, regardless of what that scale actually was. Once I started scaling X or Y independently of each other, those values were scaled based on the scale of the UIDrawCall. So if my UIRoot was scaled down to 0.01, all of a sudden the vertices given to the shader would drop to +/-2.56, along with whatever further scaling I was doing to the panel. Scaling the width of my panel to 0.9 would give me 2.304 instead of 230.4, etc.
Anyway, I was able to modify UIDrawCall to pass its local scale in as a shader property and have the vertex function, when the difference between x and y scales was greater than a very small value, pass a scaled value into TRANSFORM_TEX to get the soft clipping working the way I expected it to. It's not the most elegant solution, but maybe some of the information above can help with getting a proper fix in place.
