Look at how the Transparent Colored (AlphaClip) does the clipping compared to regular Transparent Colored shader.
The UIPanel chooses between shaders by name; that is "MyShaderName" is no clipping, "MyShaderName (AlphaClip)" is alpha clipping and "MyShaderName (SoftClip)" is soft clipping.
Actually making the shader do it, I'll leave up to you as that depends a lot on your own shader.
I hope that helps you.