That sounds all sorts of weird. I would normally say that you're hitting a different collider than the one you're expecting to hit, but if you're seeing it hit in the debug output, I'm not sure what's going on.
When it's "some work, some don't" it's usually a depth issue where they are on the same z position as the background collider for instance, then it's undefined which you'll actually hit.
Thanks for this post. Thank you, thank you, thank you!!
After reading this I got an idea which lead me to going back and looking at all of the colliders again. There was nothing out of the ordinary, so I'm still not sure why some of the buttons ("inventory slots") were working and others weren't. The data for all the buttons was the same and all of the buttons were created with the same prefab. However, I decided to screw around with the colliders and finally got it working. Seems like the Z axis on the window collider needed to be adjusted. Now, I'm not sure if this is the correct way to deal with the problem, but by playing around with the numbers, I was able to get it working.
At some point, I may have to go back and look over the windows in more detail, but I'm satisfied with how it's working. Thanks again for the information.