It's loaded whenever it's referenced. So as soon as you load a scene that uses an atlas, it gets loaded.
A way around it is to use reference atlases, and set the reference atlas to point to nothing before doing a build. This means you will need to do your own Resource.Load, and set the reference atlas to point to the correct data atlas when you need to.