Yeah I've seen that setting and it's on by default, but didn't seem to help. But now I've re-updated the atlas after switching it back to Unlit > Transparent Colored. And it seems good, I must have done some steps in the wrong order before or not properly updated the atlas. Maybe the textures were in "GUI" setting or something, now they're all in "Advanced", which has the option "Alpha is Transparency" on.
Thanks!