I have just finished reading the 99th letter of reference I've had to peruse in the past week, and I happen to have a few opinions about what I've seen. My opinions are also based on the approximately 57,832 other letters of reference I have read over the years. I am going to list various reference letter 'types', in order from what annoys me most to what annoys me least, just because that's the mood I am in right now. Reference letter 'type' probably says something profound about the referee's character and/or personality, but I am not qualified to evaluate that (not that my lack of qualifications stops me from commenting on other things).
The worst: this reflects my own interests/biases, but I hate the letters that say things like "She is among the best young female scientists in her field". I don't need to point out that one never sees similar statements for male scientists, do I?
Tie for second-most annoying: the "I'm so great that just my sending this piece of paper back to you with my signature and CV is a royal seal of approval even though I have written nothing of substance because I don't have time and I don't really care" reference; and the "He's great because his advisor is in the National Academy of Science and/or Engineering and that's all I need to say", or even, "He's great because his advisor's advisor got a Nobel Prize". Wow. It's so cool that brilliant, award-winning people's students are also all brilliant. [note: I used 'he' in the example because all the examples of that I have seen have been male]. I am perfectly willing to believe that talented people had talented advisors etc., but give me more information!
Quite annoying: the "I really can't bring myself to say anything too nice because it would imply that this person might be better than I am". These can be quite odious, in fact. I saw this recently in several letters. For example, a referee wrote something like "Because I have won This Award and That Award, I have the authority to state that candidate X [a woman in this case] is showing signs that she might one day be at the level of someone who could also win these awards." Amazing.
Semi-annoying: the self-serving "this person is great because he/she does research closely related to my own and therefore his/her research is immensely significant". Well, OK, that's nice. I could be convinced with some additional information other than a statement to that effect, but sometimes that information is lacking entirely.
OK to good letter of reference: clear statement of how well (or not) the reference writer knows the person in question, and opinion with examples regarding research quality or potential in the context of the field.
Best: The above, but with some examples or descriptions that make the person in question stand out in some way. If a committee is reading hundreds of these letters, they really can all start to look the same after a while, so the really well-composed letters stand out.
Alas for the person requesting letters of reference, there may be no way to know what kind of letter-writer your referees are. And, for early career people, you might not have much choice anyway. Fortunately, committees (in my experience) can be very forgiving about poorly written reference letters that are otherwise positive recommendations.
I am leaving out of my list the case of reference writers who are insincere and who, in some cases, lie about a person's abilities. That's another topic. The candidates whose letters I have been reading recently really are all excellent, so it's a (difficult) matter of ranking the most excellent from the merely excellent.
I recently read a letter written by someone who didn't know the nominee well but who had been impressed by his work over the years. This letter was written from the heart and was so well written that it was very moving to read. It was a truly great letter. I wish there were more letter-writers like that. I aspire to write letters like that.
So far this fall, I've been writing tenure/promotion letters for various people. Now the faculty application/grad application season is getting into high gear, so I will try my best to write the kinds of letters that I find most useful and interesting when I'm the reader rather than the writer.