Having many objects will always be slower than having one object. Moving a widget makes it re-create its draw buffer, which is a slow process. If you have 2000 widgets, that's 2000 buffer re-creations per update. If you have 1 widget, that's only 1 buffer re-creation. So there is your difference.
In your case I wouldn't use labels at all. I'd use raw OpenGL calls (GL.Begin / GL.End, look into that).