UIPanel with soft clipping on nested skinned mesh

[Wisteso]

I've found about a dozen threads looking for this feature on this forum, but all of them so far have resigned to using workarounds such as render-to-texture + separate cameras or 2D textures of their 3D object (kinda like screenshots).

However, I have seen Aren say multiple times that a custom shader should be able to allow soft clipping of mesh objects. For the game I am developing, this is my only option. Using the above two workarounds would not work for my game since render-to-texture and the screenshot approach would consume massive amounts of texture memory considering how many separate UIPanel children/elements I will need. Using clipped mesh is far more efficient since I can reuse objects already stored in memory by the game.

It looks like I am going to need to develop this shader myself, since no one has shared their shader code to my knowledge *or* no one has been successful. If anyone has any tips (or maybe they know of a completed shader), it would be very helpful and appreciated. I have pretty minimal experience with writing shaders.

[ArenMook]

Have a look inside NGUI's clipped shaders. Namely Unlit - Transparent Colored #, where # is the number of clipped regions that shader supports. Note how it uses _Clip arguments:

1. float4 _ClipRange0 = float4(0.0, 0.0, 1.0, 1.0);
2. float2 _ClipArgs0 = float2(1000.0, 1000.0);

Your custom shader will need to make use of those as well.

In the Vertex Shader you would calculate the clipped position of the vertex:

1. o.worldPos = v.vertex.xy * _ClipRange0.zw + _ClipRange0.xy;

In the Fragment shader you would use this to attenuate the Alpha:

1. float2 factor = (float2(1.0, 1.0) - abs(IN.worldPos)) * _ClipArgs0;
2. col.a *= clamp( min(factor.x, factor.y), 0.0, 1.0);

The key here is that in NGUI's case, it only considers XY in the vertex shader (v.vertex.xy). Will it still be XY in your case, since it's going to be a 3D model? That's up to you to decide.

[Wisteso]

Okay, that all makes sense. I took a crash course in CG/ShaderLab last night and I think with some luck I'll be able to figure this out. I'll post my solution here if I can get it working.

The only thing I'm wondering is how the right shader is chosen. I did a search for the string "Transparent Colored" in NGUI's source and couldn't find any code relevant for selecting the '1' vs '2' vs '3' shader. It also doesn't appear to be using the "fallback" functionality of ShaderLab either.

[Wisteso]

Many hours later...    I know a ton more than I did about writing shaders. However, I'm stuck sooner than I planned to be.

Before worrying about the actual clipping areas, I wanted to just have a working shader that would allow a 3D mesh to show over NGUI with a single transparency value (e.g. whole mesh is Alpha=0.2). I cannot get this to work properly though after numerous ideas. The problem is that NGUI is either covering my mesh OR my mesh is on top but also cutting a hole in NGUI and allowing stuff below it to show through. I'm using a modified version of the "Unlit Transparent 3" shader of NGUI.

I've tried using "Offset -2, -2", "Offset 1, -100", "Offset -1, 100", and various other offsets (no luck). I've also tried using different render queues (like "Transparent+1", "Transparent-1"). The approach I'd like to try next is doing "UsePass "Mobile/Diffuse/FORWARD" followed by a generic pass for clipping that works for ANY shader that was called via UsePass. Does this sound viable?

Haven't given up but this is definitely frustrating. Google searches are helping, but tips here will be appreciated too.

[mathiassoeholm]

I worked on something like this a few days ago, and got it working.
Here's my solution, the ClippedModel script sets the shader properties.

1. Shader "UI/ClippedUnlitModel"
2. {
3.         Properties
4.         {
5.                 _MainTex ("Base (RGB), Alpha (A)", 2D) = "black" {}
6.         }
7.  
8.         SubShader
9.         {
10.                 LOD 200
11.  
12.                 Tags
13.                 {
14.                         "Queue" = "Transparent"
15.                         "IgnoreProjector" = "True"
16.                         "RenderType" = "Transparent"
17.                 }
18.                
19.                 Pass
20.                 {
21.                         Cull Off
22.                         Lighting Off
23.                         Offset -1, -1
24.                         Fog { Mode Off }
25.                         ColorMask RGB
26.                         Blend SrcAlpha OneMinusSrcAlpha
27.  
28.                         CGPROGRAM
29.                         #pragma vertex vert
30.                         #pragma fragment frag
31.  
32.                         #include "UnityCG.cginc"
33.  
34.                         sampler2D _MainTex;
35.                         float4 _PanelOffsetAndSharpness;
36.                         float _PanelSizeX, _PanelSizeY;
37.  
38.                         struct appdata_t
39.                         {
40.                                 float4 vertex : POSITION;
41.                                 half4 color : COLOR;
42.                                 float2 texcoord : TEXCOORD0;
43.                         };
44.  
45.                         struct v2f
46.                         {
47.                                 float4 vertex : POSITION;
48.                                 half4 color : COLOR;
49.                                 float2 texcoord : TEXCOORD0;
50.                                 float2 posInPanel : TEXCOORD1;
51.                         };
52.  
53.                         v2f vert (appdata_t v)
54.                         {
55.                                 v2f o;
56.                                 o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
57.                                 o.color = v.color;
58.                                 o.texcoord = v.texcoord;
59.  
60.                                 float2 clipSpace =  o.vertex.xy / o.vertex.w;
61.  
62.                                 // Normalize clip space
63.                                 o.posInPanel = (clipSpace.xy + 1) * 0.5;
64.  
65.                                 // Adjust for panel offset
66.                                 o.posInPanel.x  -= _PanelOffsetAndSharpness.x;
67.                                 o.posInPanel.y  -= _PanelOffsetAndSharpness.y;
68.  
69.                                 // Adjust for panel size
70.                                 o.posInPanel.x  *= (1 / _PanelSizeX);
71.                                 o.posInPanel.y  *= (1 / _PanelSizeY);
72.  
73.                                 // Transform back to clip space
74.                                 o.posInPanel *= 2;
75.                                 o.posInPanel -= 1;
76.  
77.                                 return o;
78.                         }
79.  
80.                         half4 frag (v2f IN) : COLOR
81.                         {
82.                                 // Softness factor
83.                                 float2 factor = (float2(1.0, 1.0) - abs(IN.posInPanel)) * _PanelOffsetAndSharpness.zw;
84.                        
85.                                 // Sample the texture
86.                                 half4 col = tex2D(_MainTex, IN.texcoord) * IN.color;
87.                                 col.a *= clamp( min(factor.x, factor.y), 0.0, 1.0);
88.  
89.                                 return col;
90.                         }
91.                         ENDCG
92.                 }
93.         }
94.        
95.         SubShader
96.         {
97.                 LOD 100
98.  
99.                 Tags
100.                 {
101.                         "Queue" = "Transparent"
102.                         "IgnoreProjector" = "True"
103.                         "RenderType" = "Transparent"
104.                 }
105.                
106.                 Pass
107.                 {
108.                         Cull Off
109.                         Lighting Off
110.                         ZWrite Off
111.                         Fog { Mode Off }
112.                         ColorMask RGB
113.                         Blend SrcAlpha OneMinusSrcAlpha
114.                         ColorMaterial AmbientAndDiffuse
115.                        
116.                         SetTexture [_MainTex]
117.                         {
118.                                 Combine Texture * Primary
119.                         }
120.                 }
121.         }
122. }
123.  

1. using UnityEngine;
2.  
3. [ExecuteInEditMode]
4. public class ClippedModel : MonoBehaviour
5. {
6.         private UIPanel _panel;
7.         private Material _material;
8.  
9.         private int _panelSizeXProperty;
10.         private int _panelSizeYProperty;
11.         private int _panelOffsetAndSharpnessProperty;
12.  
13.         private float _virtualScreenWidth;
14.         private float _virtualScreenHeight;
15.  
16.         void Start()
17.         {
18.                 _panel = UIPanel.Find(transform);
19.                 _material = !Application.isPlaying ? renderer.sharedMaterial : renderer.material;
20.  
21.                 _virtualScreenWidth = UIRoot.GetPixelSizeAdjustment(gameObject) * Screen.width;
22.                 _virtualScreenHeight = UIRoot.GetPixelSizeAdjustment(gameObject) * Screen.height;
23.  
24.                 _panelSizeXProperty = Shader.PropertyToID("_PanelSizeX");
25.                 _panelSizeYProperty = Shader.PropertyToID("_PanelSizeY");
26.                 _panelOffsetAndSharpnessProperty = Shader.PropertyToID("_PanelOffsetAndSharpness");
27.  
28.                 Update();
29.         }
30.  
31.         void Update()
32.         {
33.                 if (_panel.hasClipping)
34.                 {
35.                         var soft = _panel.clipSoftness;
36.                         var sharpness = new Vector2(1000.0f, 1000.0f);
37.                         if (soft.x > 0f)
38.                         {
39.                                 sharpness.x = _panel.baseClipRegion.z / soft.x;
40.                         }
41.                         if (soft.y > 0f)
42.                         {
43.                                 sharpness.y = _panel.baseClipRegion.w / soft.y;
44.                         }
45.  
46.                         Vector4 panelOffsetAndSharpness;
47.  
48.                         // Get offset
49.                         panelOffsetAndSharpness.x = ((_virtualScreenWidth * 0.5f + _panel.baseClipRegion.x) - (_panel.baseClipRegion.z * 0.5f)) / _virtualScreenWidth;
50.                         panelOffsetAndSharpness.y = ((_virtualScreenHeight * 0.5f + _panel.baseClipRegion.y) - (_panel.baseClipRegion.w * 0.5f)) / _virtualScreenHeight;
51.  
52.                         // Get sharpness
53.                         panelOffsetAndSharpness.z = sharpness.x;
54.                         panelOffsetAndSharpness.w = sharpness.y;
55.  
56.                         // Set shader properties
57.                         _material.SetFloat(_panelSizeXProperty, _panel.baseClipRegion.z / _virtualScreenWidth);
58.                         _material.SetFloat(_panelSizeYProperty, _panel.baseClipRegion.w / _virtualScreenHeight);
59.                         _material.SetVector(_panelOffsetAndSharpnessProperty, panelOffsetAndSharpness);
60.                 }
61.         }
62. }
63.  

The shader is based on one of the NGUI shaders.
I use a seperate camera to render my model with a higher depth than the UI camera.
I hope this can be helpful. If you have any feedback or improvements, let me know.
Also, it only works with clipping from a single panel.

[ArenMook]

@mathiassoeholm: Thanks!

@Wisteso: NGUI will only be replacing shaders used to draw its own geometry. It won't be replacing shaders on your 3D model. It's up to you to specify the correct shader right away (so if you have it clipped by only 1 panel, have it use a shader that has only 1 clipping region). It's also up to you to set the material's properties the same way I set them in UIDrawCall.SetClipping. So if you only have 1 clipping region you only need to set 2 things.

[Wisteso]

@mathiassoeholm: Thank you very much for that! On behalf of myself and probably many others that will find this thread looking to achieve the same effect. Please let me know if you have a steam handle that I can send a gift to My only question is: Do you use separate camera + higher depth because NGUI renders on top of everything else otherwise?

@ArenMook: Okay that clears up a lot of questions I had. Thank you!!

[Nicki]

The separate cameras is for Orthographic vs Perspective, where the 3d model is in perspective and UI is in Orthographic.

[ArenMook]

@Wisteso: You can change the draw order of 3D models, wedging them in-between of NGUI draw calls if you like. It is explained in the Useful Stuff sticky here: http://www.tasharen.com/forum/index.php?topic=776.msg34546#msg34546 -- it works if you use Explicit render queues on your UI panels, leaving a gap where your 3D model is going to be.

[mathiassoeholm]

@Wisteso You're welcome! You really don't need to give me a gift for this, I'm just sharing something that I made for my own UI :-)

As Nicki points out, the extra camera is to give my 3D objects perspective.

[Wisteso]

I've been able to get all the trickiest stuff working correctly. The thing I was missing before was the renderQueue that Aren pointed out. After setting it to 4000 everything worked correctly (since I only use one camera in ortho mode).

mathiassoeholm's position/offset/softness/etc code doesn't seem to like my GUI (probably because I use some SD/HD/pixel-size tricks, but once I figure out the issues there everything should be good. I'll post my final changes after that's done.

[Wisteso]

I'm still trying to fix the issue with the positional math being off somewhere. The hours are really adding up, but I'm slowly getting closer to figuring it out.

I've attached a screenshot showing where the defined panel is (shown in purple) and where the panel space always ends up (shown in pink). I've blurred the rest of the screenshot out just as an OCD paranoid precaution.

Two observations. 1) The actual panel is being placed exactly in the middle of the screen, width and height wise. 2) The edge falloff is about twice as intense/sudden as it should be, though I this could be a separate issue.

My biggest hurdle is trying to kinda reverse-engineer the algorithm used in this shader. I think I can probably figure it out, but if anyone knows/sees anything obvious that might save hours of looking, please let me know.

[Wisteso]

(ArenMook I'm thinking there must be a more correct way to calculate the value of "bottomLeft" in my second example below. Any thoughts?)

Huzzah! I think I've figured out the issue. The culprit was the following code:

1.             // Get offset
2.             panelOffsetAndSharpness.x = ((_virtualScreenWidth * 0.5f + _panel.baseClipRegion.x) - (_panel.baseClipRegion.z * 0.5f)) / _virtualScreenWidth;
3.             panelOffsetAndSharpness.y = ((_virtualScreenHeight * 0.5f + _panel.baseClipRegion.y) - (_panel.baseClipRegion.w * 0.5f)) / _virtualScreenHeight;

Which seems like it would have always placed the panel clip area in the middle of the screen. The corrected code that I'm using now is...

1.             // Get offset
2.             Vector2 bottomLeft = _camera.WorldToViewportPoint(_panel.worldCorners[0]);
3.             panelOffsetAndSharpness.x = bottomLeft.x;
4.             panelOffsetAndSharpness.y = bottomLeft.y;

The fade falloff still seems slightly off, but that's a minor separate issue that I can fix later. There's still some improvements I think I can make before posting the completed code for others future reference.

[pretender]

I was banging my head with this all day and I can't figure out where what goes? Can somebody give steps how to setup this please! Thanks!
edit:
where should panels go? When I place the shader to 3d model (simple cube for testing) it dissapears

[pretender]

I would appreciate some help with this