As seen by the images
here color blindless doesn't mean you can't see the colors entirely as most people think. (as in say you can't see green, you wouldn't see just black). Seeing how you can see Blue, Red and black , you probably have Deuteranopia. Which is loss of the green color receptor or cone. The only colors that you shouldn't see correctly on Simon is the Yellow and Aqua boxes. However you should still be able to tell the difference because it should be just a different shade. The Yellow would be more redish bluish, and the aqua more of a darker blue, but with a different shade than the normal blue.
Numbers on the game would be a good idea, however unnecessary unless someone is totally color blind which is really rare and even then the colors would be different shades of black/white.