Author Topic: UIPanel with soft clipping on nested skinned mesh  (Read 9904 times)

Wisteso

  • Full Member
  • ***
  • Thank You
  • -Given: 21
  • -Receive: 3
  • Posts: 103
    • View Profile
UIPanel with soft clipping on nested skinned mesh
« on: July 02, 2014, 06:14:41 PM »
I've found about a dozen threads looking for this feature on this forum, but all of them so far have resigned to using workarounds such as render-to-texture + separate cameras or 2D textures of their 3D object (kinda like screenshots).

However, I have seen Aren say multiple times that a custom shader should be able to allow soft clipping of mesh objects. For the game I am developing, this is my only option. Using the above two workarounds would not work for my game since render-to-texture and the screenshot approach would consume massive amounts of texture memory considering how many separate UIPanel children/elements I will need. Using clipped mesh is far more efficient since I can reuse objects already stored in memory by the game.

It looks like I am going to need to develop this shader myself, since no one has shared their shader code to my knowledge *or* no one has been successful. If anyone has any tips (or maybe they know of a completed shader), it would be very helpful and appreciated. I have pretty minimal experience with writing shaders.
« Last Edit: July 02, 2014, 06:20:11 PM by Wisteso »

ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,154
  • Toronto, Canada
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #1 on: July 03, 2014, 04:47:02 AM »
Have a look inside NGUI's clipped shaders. Namely Unlit - Transparent Colored #, where # is the number of clipped regions that shader supports. Note how it uses _Clip arguments:
  1. float4 _ClipRange0 = float4(0.0, 0.0, 1.0, 1.0);
  2. float2 _ClipArgs0 = float2(1000.0, 1000.0);
Your custom shader will need to make use of those as well.

In the Vertex Shader you would calculate the clipped position of the vertex:
  1. o.worldPos = v.vertex.xy * _ClipRange0.zw + _ClipRange0.xy;
In the Fragment shader you would use this to attenuate the Alpha:
  1. float2 factor = (float2(1.0, 1.0) - abs(IN.worldPos)) * _ClipArgs0;
  2. col.a *= clamp( min(factor.x, factor.y), 0.0, 1.0);
The key here is that in NGUI's case, it only considers XY in the vertex shader (v.vertex.xy). Will it still be XY in your case, since it's going to be a 3D model? That's up to you to decide.

Wisteso

  • Full Member
  • ***
  • Thank You
  • -Given: 21
  • -Receive: 3
  • Posts: 103
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #2 on: July 03, 2014, 03:20:56 PM »
Okay, that all makes sense. I took a crash course in CG/ShaderLab last night and I think with some luck I'll be able to figure this out. I'll post my solution here if I can get it working.

The only thing I'm wondering is how the right shader is chosen. I did a search for the string "Transparent Colored" in NGUI's source and couldn't find any code relevant for selecting the '1' vs '2' vs '3' shader. It also doesn't appear to be using the "fallback" functionality of ShaderLab either.

Wisteso

  • Full Member
  • ***
  • Thank You
  • -Given: 21
  • -Receive: 3
  • Posts: 103
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #3 on: July 04, 2014, 02:02:39 AM »
Many hours later...    I know a ton more than I did about writing shaders. However, I'm stuck sooner than I planned to be.

Before worrying about the actual clipping areas, I wanted to just have a working shader that would allow a 3D mesh to show over NGUI with a single transparency value (e.g. whole mesh is Alpha=0.2). I cannot get this to work properly though after numerous ideas. The problem is that NGUI is either covering my mesh OR my mesh is on top but also cutting a hole in NGUI and allowing stuff below it to show through. I'm using a modified version of the "Unlit Transparent 3" shader of NGUI.

I've tried using "Offset -2, -2", "Offset 1, -100", "Offset -1, 100", and various other offsets (no luck). I've also tried using different render queues (like "Transparent+1", "Transparent-1"). The approach I'd like to try next is doing "UsePass "Mobile/Diffuse/FORWARD" followed by a generic pass for clipping that works for ANY shader that was called via UsePass. Does this sound viable?

Haven't given up but this is definitely frustrating. Google searches are helping, but tips here will be appreciated too.
« Last Edit: July 04, 2014, 03:33:00 AM by Wisteso »

mathiassoeholm

  • Newbie
  • *
  • Thank You
  • -Given: 1
  • -Receive: 4
  • Posts: 35
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #4 on: July 04, 2014, 04:01:12 AM »
I worked on something like this a few days ago, and got it working.
Here's my solution, the ClippedModel script sets the shader properties.

  1. Shader "UI/ClippedUnlitModel"
  2. {
  3.         Properties
  4.         {
  5.                 _MainTex ("Base (RGB), Alpha (A)", 2D) = "black" {}
  6.         }
  7.  
  8.         SubShader
  9.         {
  10.                 LOD 200
  11.  
  12.                 Tags
  13.                 {
  14.                         "Queue" = "Transparent"
  15.                         "IgnoreProjector" = "True"
  16.                         "RenderType" = "Transparent"
  17.                 }
  18.                
  19.                 Pass
  20.                 {
  21.                         Cull Off
  22.                         Lighting Off
  23.                         Offset -1, -1
  24.                         Fog { Mode Off }
  25.                         ColorMask RGB
  26.                         Blend SrcAlpha OneMinusSrcAlpha
  27.  
  28.                         CGPROGRAM
  29.                         #pragma vertex vert
  30.                         #pragma fragment frag
  31.  
  32.                         #include "UnityCG.cginc"
  33.  
  34.                         sampler2D _MainTex;
  35.                         float4 _PanelOffsetAndSharpness;
  36.                         float _PanelSizeX, _PanelSizeY;
  37.  
  38.                         struct appdata_t
  39.                         {
  40.                                 float4 vertex : POSITION;
  41.                                 half4 color : COLOR;
  42.                                 float2 texcoord : TEXCOORD0;
  43.                         };
  44.  
  45.                         struct v2f
  46.                         {
  47.                                 float4 vertex : POSITION;
  48.                                 half4 color : COLOR;
  49.                                 float2 texcoord : TEXCOORD0;
  50.                                 float2 posInPanel : TEXCOORD1;
  51.                         };
  52.  
  53.                         v2f vert (appdata_t v)
  54.                         {
  55.                                 v2f o;
  56.                                 o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
  57.                                 o.color = v.color;
  58.                                 o.texcoord = v.texcoord;
  59.  
  60.                                 float2 clipSpace =  o.vertex.xy / o.vertex.w;
  61.  
  62.                                 // Normalize clip space
  63.                                 o.posInPanel = (clipSpace.xy + 1) * 0.5;
  64.  
  65.                                 // Adjust for panel offset
  66.                                 o.posInPanel.x  -= _PanelOffsetAndSharpness.x;
  67.                                 o.posInPanel.y  -= _PanelOffsetAndSharpness.y;
  68.  
  69.                                 // Adjust for panel size
  70.                                 o.posInPanel.x  *= (1 / _PanelSizeX);
  71.                                 o.posInPanel.y  *= (1 / _PanelSizeY);
  72.  
  73.                                 // Transform back to clip space
  74.                                 o.posInPanel *= 2;
  75.                                 o.posInPanel -= 1;
  76.  
  77.                                 return o;
  78.                         }
  79.  
  80.                         half4 frag (v2f IN) : COLOR
  81.                         {
  82.                                 // Softness factor
  83.                                 float2 factor = (float2(1.0, 1.0) - abs(IN.posInPanel)) * _PanelOffsetAndSharpness.zw;
  84.                        
  85.                                 // Sample the texture
  86.                                 half4 col = tex2D(_MainTex, IN.texcoord) * IN.color;
  87.                                 col.a *= clamp( min(factor.x, factor.y), 0.0, 1.0);
  88.  
  89.                                 return col;
  90.                         }
  91.                         ENDCG
  92.                 }
  93.         }
  94.        
  95.         SubShader
  96.         {
  97.                 LOD 100
  98.  
  99.                 Tags
  100.                 {
  101.                         "Queue" = "Transparent"
  102.                         "IgnoreProjector" = "True"
  103.                         "RenderType" = "Transparent"
  104.                 }
  105.                
  106.                 Pass
  107.                 {
  108.                         Cull Off
  109.                         Lighting Off
  110.                         ZWrite Off
  111.                         Fog { Mode Off }
  112.                         ColorMask RGB
  113.                         Blend SrcAlpha OneMinusSrcAlpha
  114.                         ColorMaterial AmbientAndDiffuse
  115.                        
  116.                         SetTexture [_MainTex]
  117.                         {
  118.                                 Combine Texture * Primary
  119.                         }
  120.                 }
  121.         }
  122. }
  123.  

  1. using UnityEngine;
  2.  
  3. [ExecuteInEditMode]
  4. public class ClippedModel : MonoBehaviour
  5. {
  6.         private UIPanel _panel;
  7.         private Material _material;
  8.  
  9.         private int _panelSizeXProperty;
  10.         private int _panelSizeYProperty;
  11.         private int _panelOffsetAndSharpnessProperty;
  12.  
  13.         private float _virtualScreenWidth;
  14.         private float _virtualScreenHeight;
  15.  
  16.         void Start()
  17.         {
  18.                 _panel = UIPanel.Find(transform);
  19.                 _material = !Application.isPlaying ? renderer.sharedMaterial : renderer.material;
  20.  
  21.                 _virtualScreenWidth = UIRoot.GetPixelSizeAdjustment(gameObject) * Screen.width;
  22.                 _virtualScreenHeight = UIRoot.GetPixelSizeAdjustment(gameObject) * Screen.height;
  23.  
  24.                 _panelSizeXProperty = Shader.PropertyToID("_PanelSizeX");
  25.                 _panelSizeYProperty = Shader.PropertyToID("_PanelSizeY");
  26.                 _panelOffsetAndSharpnessProperty = Shader.PropertyToID("_PanelOffsetAndSharpness");
  27.  
  28.                 Update();
  29.         }
  30.  
  31.         void Update()
  32.         {
  33.                 if (_panel.hasClipping)
  34.                 {
  35.                         var soft = _panel.clipSoftness;
  36.                         var sharpness = new Vector2(1000.0f, 1000.0f);
  37.                         if (soft.x > 0f)
  38.                         {
  39.                                 sharpness.x = _panel.baseClipRegion.z / soft.x;
  40.                         }
  41.                         if (soft.y > 0f)
  42.                         {
  43.                                 sharpness.y = _panel.baseClipRegion.w / soft.y;
  44.                         }
  45.  
  46.                         Vector4 panelOffsetAndSharpness;
  47.  
  48.                         // Get offset
  49.                         panelOffsetAndSharpness.x = ((_virtualScreenWidth * 0.5f + _panel.baseClipRegion.x) - (_panel.baseClipRegion.z * 0.5f)) / _virtualScreenWidth;
  50.                         panelOffsetAndSharpness.y = ((_virtualScreenHeight * 0.5f + _panel.baseClipRegion.y) - (_panel.baseClipRegion.w * 0.5f)) / _virtualScreenHeight;
  51.  
  52.                         // Get sharpness
  53.                         panelOffsetAndSharpness.z = sharpness.x;
  54.                         panelOffsetAndSharpness.w = sharpness.y;
  55.  
  56.                         // Set shader properties
  57.                         _material.SetFloat(_panelSizeXProperty, _panel.baseClipRegion.z / _virtualScreenWidth);
  58.                         _material.SetFloat(_panelSizeYProperty, _panel.baseClipRegion.w / _virtualScreenHeight);
  59.                         _material.SetVector(_panelOffsetAndSharpnessProperty, panelOffsetAndSharpness);
  60.                 }
  61.         }
  62. }
  63.  

The shader is based on one of the NGUI shaders.
I use a seperate camera to render my model with a higher depth than the UI camera.
I hope this can be helpful. If you have any feedback or improvements, let me know.
Also, it only works with clipping from a single panel.

ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,154
  • Toronto, Canada
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #5 on: July 04, 2014, 02:19:34 PM »
@mathiassoeholm: Thanks!

@Wisteso: NGUI will only be replacing shaders used to draw its own geometry. It won't be replacing shaders on your 3D model. It's up to you to specify the correct shader right away (so if you have it clipped by only 1 panel, have it use a shader that has only 1 clipping region). It's also up to you to set the material's properties the same way I set them in UIDrawCall.SetClipping. So if you only have 1 clipping region you only need to set 2 things.

Wisteso

  • Full Member
  • ***
  • Thank You
  • -Given: 21
  • -Receive: 3
  • Posts: 103
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #6 on: July 05, 2014, 12:39:32 AM »
@mathiassoeholm: Thank you very much for that! On behalf of myself and probably many others that will find this thread looking to achieve the same effect. Please let me know if you have a steam handle that I can send a gift to :D My only question is: Do you use separate camera + higher depth because NGUI renders on top of everything else otherwise?

@ArenMook: Okay that clears up a lot of questions I had. Thank you!!

Nicki

  • Global Moderator
  • Hero Member
  • *****
  • Thank You
  • -Given: 33
  • -Receive: 141
  • Posts: 1,768
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #7 on: July 05, 2014, 10:24:03 AM »
The separate cameras is for Orthographic vs Perspective, where the 3d model is in perspective and UI is in Orthographic.

ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,154
  • Toronto, Canada
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #8 on: July 05, 2014, 09:57:07 PM »
@Wisteso: You can change the draw order of 3D models, wedging them in-between of NGUI draw calls if you like. It is explained in the Useful Stuff sticky here: http://www.tasharen.com/forum/index.php?topic=776.msg34546#msg34546 -- it works if you use Explicit render queues on your UI panels, leaving a gap where your 3D model is going to be.

mathiassoeholm

  • Newbie
  • *
  • Thank You
  • -Given: 1
  • -Receive: 4
  • Posts: 35
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #9 on: July 06, 2014, 09:30:38 AM »
@Wisteso You're welcome! You really don't need to give me a gift for this, I'm just sharing something that I made for my own UI :-)

As Nicki points out, the extra camera is to give my 3D objects perspective.

Wisteso

  • Full Member
  • ***
  • Thank You
  • -Given: 21
  • -Receive: 3
  • Posts: 103
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #10 on: July 09, 2014, 02:06:38 AM »
I've been able to get all the trickiest stuff working correctly. The thing I was missing before was the renderQueue that Aren pointed out. After setting it to 4000 everything worked correctly (since I only use one camera in ortho mode).

mathiassoeholm's position/offset/softness/etc code doesn't seem to like my GUI (probably because I use some SD/HD/pixel-size tricks, but once I figure out the issues there everything should be good. I'll post my final changes after that's done.

Wisteso

  • Full Member
  • ***
  • Thank You
  • -Given: 21
  • -Receive: 3
  • Posts: 103
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #11 on: July 22, 2014, 05:21:22 PM »
I'm still trying to fix the issue with the positional math being off somewhere. The hours are really adding up, but I'm slowly getting closer to figuring it out.

I've attached a screenshot showing where the defined panel is (shown in purple) and where the panel space always ends up (shown in pink). I've blurred the rest of the screenshot out just as an OCD paranoid precaution.

Two observations. 1) The actual panel is being placed exactly in the middle of the screen, width and height wise. 2) The edge falloff is about twice as intense/sudden as it should be, though I this could be a separate issue.

My biggest hurdle is trying to kinda reverse-engineer the algorithm used in this shader. I think I can probably figure it out, but if anyone knows/sees anything obvious that might save hours of looking, please let me know.
« Last Edit: July 22, 2014, 11:44:59 PM by Wisteso »

Wisteso

  • Full Member
  • ***
  • Thank You
  • -Given: 21
  • -Receive: 3
  • Posts: 103
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #12 on: July 23, 2014, 02:36:33 AM »
(ArenMook I'm thinking there must be a more correct way to calculate the value of "bottomLeft" in my second example below. Any thoughts?)

Huzzah! I think I've figured out the issue. The culprit was the following code:

  1.             // Get offset
  2.             panelOffsetAndSharpness.x = ((_virtualScreenWidth * 0.5f + _panel.baseClipRegion.x) - (_panel.baseClipRegion.z * 0.5f)) / _virtualScreenWidth;
  3.             panelOffsetAndSharpness.y = ((_virtualScreenHeight * 0.5f + _panel.baseClipRegion.y) - (_panel.baseClipRegion.w * 0.5f)) / _virtualScreenHeight;

Which seems like it would have always placed the panel clip area in the middle of the screen. The corrected code that I'm using now is...

  1.             // Get offset
  2.             Vector2 bottomLeft = _camera.WorldToViewportPoint(_panel.worldCorners[0]);
  3.             panelOffsetAndSharpness.x = bottomLeft.x;
  4.             panelOffsetAndSharpness.y = bottomLeft.y;

The fade falloff still seems slightly off, but that's a minor separate issue that I can fix later. There's still some improvements I think I can make before posting the completed code for others future reference.
« Last Edit: July 23, 2014, 04:13:51 AM by Wisteso »

pretender

  • Full Member
  • ***
  • Thank You
  • -Given: 1
  • -Receive: 0
  • Posts: 155
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #13 on: December 25, 2014, 11:29:45 AM »
I was banging my head with this all day and I can't figure out where what goes? Can somebody give steps how to setup this please! Thanks!
edit:
where should panels go? When I place the shader to 3d model (simple cube for testing) it dissapears
« Last Edit: December 25, 2014, 11:43:04 AM by pretender »

pretender

  • Full Member
  • ***
  • Thank You
  • -Given: 1
  • -Receive: 0
  • Posts: 155
    • View Profile
Re: UIPanel with soft clipping on nested skinned mesh
« Reply #14 on: December 26, 2014, 04:00:13 AM »
I would appreciate some help with this :)