I think what really bothers me the most about being in my field is the fact that most people are convinced that there isn't a problem.
Even here, where I've found a welcoming environment and classes I enjoy and challenging, interesting research--even here we have at most 3 female faculty and 1 African-American. Even here, there's a lack of parity in admitted graduate students, and at every other level.
These people are SCIENTISTS. Look at the data! And don't try that "Well, [x group of minorities] just don't seem to be interested in the field." Scientist, remember? Ask why! Look for causes and correlations. Try experimental techniques (like putting some of x group on your recruiting posters, or nominating them for awards, or hiring them). Plot results. Repeat experiments. TRY.
Instead I get the "shrug", or the "yeah" and moving on, or, *shudder* worse yet, the "Well, I wish there were more of x group as well." Yeah, some of your best friends are x group too, right?