But is there a reason then to allow me to modify those paddings? Maybe they should be read-only?
But anyway - i beg to you to check this feature deeper with those "negative paddings":
I can easily anchor an abstract sprite to a label size, and then change sprite like this:

And because of those negative paddings, sprite will always have a nice look! It is verry useful, when you have to switch different styles of atlases for same prefabs!

(just a bad sample, but still)