It's not a good idea to scale something that has anchored children, unless they are anchored to the root object that's being scaled. If you anchor to something outside of the scaled object's hierarchy, you are bound to run into issues.
Also make sure you are not scaling down to zero scale, as that's invalid. Can't divide by zero, so can't calculate the inverse transform.
