Matt, I totally agree that references are important. I find the most reliable references to be the ones that are not on the the candidates resume. I cannot see someone put a reference whom they know will bad mouth them, so one also has to take the integrity of the process into consideration. The other concern I have is people tend to always want to give someone a good reference as "we are all nice people" and others worry about karma when giving someone a bad reference, albeit true. People have a hard time dealing with the brutal truth hence when they know someone's prospects of employment are on the line, they would rather help than risk the person not getting hired.
Given the world is small, I often ask people that have worked with the candidate if they know the candidate and what their thoughts are of the person. Then at least I have different points of view which gives you insight as what you could possibly be managing. It is useful information, but only as effective as the maturity that comes with using it.
Rayanne, everything has to be taken with context. To be honest, if it happened to be your ex Husband I would have to take everything he says with a BAG of salt. I don't force the issue, it is about asking someone's point of view whom you actually rate. So I would not cold call a stranger. The world is small and what is best is people tend to tell me stuff before I even ask for it. Some news good, some news concerning. There are three sides to every story I believe and I trust my gut before I trust another person's experience of an individual.
I agree with Rayanne about the credibility of past references. I think job seekers should be advised to obtain as many positive references in advance of a job search as possible. Don't wait until a request is made for references. Provide them in advance as part of your profile. This might insure the most accurate references are checked. It is probably a good idea to encourage job seekers (especially young) to build a reference collection over the course of their career.