Referenced Atlases all loading into ram instead of just one

[FizzPow]

Ok so I have one Main Atlas set as a reference atlas.  And setting the reference to an SD, HD, 4X atlas depending on screen resolution.  It all works great, except for one problem.  For some reason, ALL 3 textures are loading into memory even though I only have 1 being pointed to by the Reference Atlas, and I even clear that before building.  I have written an editor script to iterate through all sprites in my scene to be sure none are referencing the non-reference atlases.

I am stumped, the only way I can get my memory to free up, is to physically remove the atlas's from the project, but as soon as I put them back in, memory is used up again.  This is obviously bad because the 4X atlas alone takes up 64MB of RAM, unacceptable on low end devices.  I have no idea what is referencing the other 2 atlases to cause unity to load them.  The app is definitely using the right atlas as far as what sprites are visibly shown, so the other textures are loading but not being used for anything.

Would appreciate any help on nailing down why this is happening or how I can debug this.  Thanks!

[ArenMook]

If you reference it in any way, Unity will load it. This is why you need to use Resources.Load instead, and reference it by name.

[FizzPow]

Trust me, I understand that very well.  I am using Resources.Load to set the reference at load time.  That's what I am saying, I cannot find where these atlases are being referenced! 

[ArenMook]

That seems strange. I assume you've placed them in the Resources folder?

[FizzPow]

Yes I have.  At the moment, I am deleting 1 panel at a time from my scene and doing a Build, and looking at editor.log to see if it removes the reference. Then narrowed it down to 1 panel, Now deleting one sprite at a time...

Ok so after doing this 100 times, I have found the culprit.. but it still makes no sense.  It is pointed to the Reference Atlas as it should be, the only thing about it, is the UISprite component was disabled on it, because I was temporarily hiding it that way.  As soon as I enabled the component, it stopped building the other atlas?!  SO I am guessing the un-enabled component was somehow still saving a reference to the atlas even though it is it not showing in the inspector?!  Very confused.

Edit: Just to clarify, I removed them from resources folder temporarily so I could see what was referencing them to put them in the build, by deleting items one at a time and building until it stopped building in the atlas.  At least I finally found the culprit but this seems like a bug or something since the inspector qualities showed it was referencing the right atlas.