Friday, January 22, 2010

Analytical Writing

The GRE contains an "Analytical Writing" section that is supposed to be a good indicator of .. something. But what? This is what I would like to know.

Is the score relevant to how well a student might do with some important aspects of graduate school? I took the GRE long before the advent of the Analytical Writing (AW) component, so I don't have any firsthand experience with it. I've read some of the available materials about what it is and how it is graded, and all of my recent graduate students have taken the AW exam, but I don't really know what the score means, if anything.

From what I've seen, the GRE in general does not predict whether a student will do well in research. This is not news to many (most?) people. The Quantitative score does tend to predict how a student will do in quantitative courses, but that may or may not correspond to whether a student can apply quantitative techniques in research. I've advised students with perfect Q scores and apparently no ability to think about science in a quantitative way or apply quantitative skills to research. I've advised students with lower Q scores who were quite talented at quantitative research applications.

Also based on personal experience advising students, I surmise that the Verbal score seems to indicate something about the complexity of an individual's vocabulary, but doesn't predict anything about reading/writing ability. Maybe that's where the AW exam comes in, but I still don't know what the score really means.

An exam involving writing and text analysis should test some skills that we want a grad student to have, but I have increasingly encountered extremely smart and creative students who write well and have no problem with reading comprehension but who have low AW scores, and students with high AW scores who struggle with writing and synthesizing essential points from what they read.

The conclusion that some smart students take tests well and some don't seems inevitable; this is of course one of the oft-proposed interpretations of student performance on standardized exams that are taken with strict time limits.

I don't think the GRE scores are totally meaningless. I would be very reluctant to admit a grad student with low scores on one or more of the GRE exam components unless the low scores were convincingly different from the rest of the academic record.

I have recently been gazing at applications with AW scores ranging from 3 to 6, and in many cases there seems to be no correlation between the AW score and the rest of the application, even for students for whom English is the first language. I am willing to believe that a 6 (out of 6) means the student is a good writer, just as I am willing to believe that a student with across-the-board good GREs is smart, but then there are all the students with good-but-not-great scores. I have seen no evidence that a student with an AW score of 5 (or even lower) is necessarily less-great at the things supposedly tested by this part of the exam.

In any case, we get these scores as part of grad applications, we stare at them, we try to figure out what they might mean, we consider them in the context of the entire application, and then we make our best guess in the admissions process based on the entire file.

70 comments:

Anonymous said...

what is the min. Q score for success in Science/Eng PhD programs? I would say at least 600.

The math is simple, just like SAT math. Math is the language of science, correct?

Anonymous said...

I was interested to read this and the comments a couple of days ago about verbal v math scores in the GRE. CPP said he considers verbal scores the most useful and this post imples they are at least kind of useful to FSP. I look at grad applications for psychology / neuroscience in the UK, and I pay way more attention to math abilities (even high school math) than verbal abilities. I guess it is because our applicants have been writing essays for 3 years so the writing is normally acceptible. But some never even did calculus, and I don't have time to start teaching that in grad school. I guess physics applicants are the opposite?

MGS said...

Some students may score poorly on the analytical writing section because they put no effort into it, knowing or believing it is inconsequential to their application.

puck said...

I would LOVE to hear your opinion on subject-based GREs, if your discipline uses them. While many science departments are willing to kick the regular GREs to the curb to a degree, the subject tests are seen as gospel; however, the ones our department uses (Physics GREs) also have zero correlation to how well a person will do as a researcher and scientist. The end result is that the admissions committee splits into the "numbers" crowd (ranking all the 4.0 GPA 90th percentile applicants first) and the "experience crowd" (highly ranking students with research, publication, and presentation experience). This has in term led to some alarming differences in the genders of the ranked lists, and a few of our professors have presented some compelling data about gender, racial, and country-specific bias in these exams (think all the perfect scores from China, and the relatively poor performance of American students as a whole). The message was that overly relying on them can have the end result of being discriminatory.

What are your thoughts?

Anonymous said...

I have two GRE stories that make me believe they are more-or-less meaningless.

First, one of my colleagues did a 'study' where he looked at GRE scores and tried to correlate them with success in grad school. I do not know how he defined success. He found, however, that only one score was correlated at all, the verbal, and that the correlation was negative. The better the score the worse the student did when they got here on average.

Second, when I took the GREs a decade ago, I returned to the lab where I was working and ran into one of the then current students. He asked what my scores were and I told him; he said he'd done better, even on the verbal, despite that his English was nearly incomprehensible, and it is my native language. He asked how long I'd studied, I'd studied for about 2-3 hours over a couple of days. He studied for 2 years.

qaz said...

The thing to remember about the "analytical writing" score (which seems to have replaced the "analytical" section which was about logic with a "writing" section which seems to be about grammar) is that the AW score is quantized. There are only 13 possible scores and given standard distributions, a very large number of students get the exact same score. (A score of 5.5 is the 92nd percentile, but a score of 5 is the 81st percentile and a score of 4.5 is 63rd percentile.)

The only thing we've been able to identify is that a really low AW score (less than or equal to 3.5) is indicative of problems with the English language and is correlated with a low TOEFL score.

This helps us how?

Anonymous said...

FSP: You say you took the GRE a long time ago, yet you are judging people that took it recently. I suggest you (and other professors that look at a lot of GRE scores/grad applications) take the exam again. This would help you see what the students are going through. You can prepare by buying a $20 review book and then take the test. Also, I would be interested in what your scores are.

In addition, I would like to know what score CPP would receive. CPP talks as if he is really smart, I challenge him to put his money where his mouth is and take the test sometime this fall.

Anonymous said...

My guess is that you're talking about higher GRE scores, like between 1200 and 1600.

We have seen differences between low scorers who were admitted to our program for various reasons (i.e., students who score 1100 and under) and higher scorers (i.e., students here score 1250 and over). The cutoffs I am making are arbitrary, but yet we are definitely seeing differences.

Anonymous said...

An MIT faculty member did a study on the analogous writing sample component of the SAT score. http://en.wikipedia.org/wiki/SAT#MIT_Study

His conclusion: the grade was correlated more with the length of the essay than anything else. Deliberately introduced errors in logic and fact did not affect the score.

Think about the reality of the grading process for those analytical writing essays. ETS hires people at a piece rate that works out to a puny per hour salary, and they stare at computer screens with these essays all day long, scoring them according to a "rubric."

Anonymous said...

"I have recently been gazing at applications with AW scores ranging from 3 to 6, and in many cases there seems to be no correlation between the AW score and the rest of the application, even for students for whom English is the first language."

It is this total lack of correlation with anything, including the quality of the writing in the student's essay, that has led me to totally discount the AW assessment. I do look at the V and Q GREs scores, with the caveats FSP noted.

Another aspect of GREs--I steer away from students with high GREs and low grades faster than I do from students with other combinations [except low grades and low GREs]. To me thigh GREs and low grades suggests smart but lazy, the last thing I want in a student.

Mark P

Anonymous said...

My interpretation (as an engineer)

the quantitative is ridiculously easy. This is worthless to differentiate really mathematically talented and those who took the classes in the engineering curriculum. I tend to ignore this number - if it is sub 650, it does hurt the candidate.

the verbal is ridiculously hard. Maybe those words are useful for an English major, but not for us. Also, I have seen international students who get an excellent score but can't write/read/speak at a the level of their lower-scoring American peers. I tend to ignore this number unless it is sub 400, which counts against them.

the analytical is meaningless. There are so few # scores that the percentiles are smashed together and it doesn't correlate to any ability (reading/writing/thinking) that I can tell. I totally ignore this number.

So, basically I think the GRE is worthless. I favor the old test's analytical format. I would love to see 2 versions of the quantitative - one for non-science graduates and one for us.

female Science Professor said...

Not long ago I took some practice tests to (re)acquaint myself with the GRE. That's part of the basis for my interpretation of what the exam scores mean.

Anonymous said...

I agree with anon @ 7:19am. I want FSP and CPP to take the test and post their scores. The GRE is a way for potential advisers to rank candidates, but there is no way to rank potential advisers (with a defined standard such as the GRE). Everyone that agrees with anon@7:19am please post to encourage FSP and CPP to take the GRE!!!!

Eugenie said...

I've had some major frustrations with the GRE. I took the exam twice, and received the same Q and V scores. I failed miserably at the analytical writing, with no idea why (I received a whopping 2.5 and 3).

I'm published (as an undergrad) and consider myself a decent (science) writer. I just hope the grad programs I applied to will take the gamble and consider me...

Anne said...

The fee to take the GRE is ridiculous. It's absurd to think that FSP and CPP should take it again.

I took the AW portion. As far as I can tell, they're looking for such a specific type of response to a very specific type of question. If you get a GRE book and look over what those are, and are just a decent writer, you'll do ok.

I'm a student member of my department's admissions committee this year. We don't pay attention to the AW score unless it's horrific (< about 2). If your Q score is below 650 some serious eyebrows are going to be raised. Verbal is less of a big deal, but if it's bad (say less than 50th percentile) that's going to hurt you.

Anonymous said...

A major problem with the computerized test is that it shunts you into different levels depending on your answers to key "crossroads" questions. This means that everyone is not taking the same test, which is unfortunate, because it's very difficult to get out of a lower bracket once you're placed in it.

I experienced this first hand. I clearly remember that test - the first half went fine, the questions were challenging. Then I got to a suspiciously easy question, and answered it completely incorrectly, which I realized about 2 seconds after submitting it. The test got noticeably easier from that point on, and I ended up with something like a 560. Even though I don't think it affected my success in applying to grad school (I aced the writing part), I'm still a little bitter about it.

m said...

As someone who took general and subject GREs recently (within last 5 years), my thoughts on the AW section are this: to get a high score, you probably need to be reasonably good at grammar/spelling, but you don't need to be a good writer at all. You need to read a few of their online examples of "good" and "bad" essays and write to the formula that they want.

Their formula is basically the "5 paragraph essay" that I was taught (and standardized-tested on in my state) in middle school. The first paragraph explains exactly what you are going to say, with one sentence summarizing each of your later paragraphs. You write 3 paragraphs explaining your 3 main points, each one with a topic sentence, 3 or so supporting sentences, and a conclusion sentence summarizing the paragraph. And then you write a concluding paragraph re-summarizing your 3 points.

This is what I recall all of the example tests doing, and this is what I did and got a 6. But it's a pretty boring and repetitive way to write, so I don't think the test has much bearing on whether you're a good writer.

I think it's not a question of whether you write well or really a question of testing well but just whether you are prepared to write to their formula. I've known one person who is a very good writer, but who went into the test the first time writing the way he would normally write an essay and scored very poorly. The second time he took the test, he had read examples and wrote the formula I describe, and got a 6.

I realize these are only a few examples, but I really think this is what's going on with this part of the test, so I considered it pretty much useless and a waste of time.

Anonymous said...

I agree with Anonymous at 7:28 - let's think about who is grading the relatively subjective AW portion. English professors? Scientists and engineers with great writing skills? ...temp workers with a high school diploma or GED?

Anonymous said...

sat/gre/etc are just scams for people to take your money and time for a test that mostly indicates you are rich (all of them correlate better with wealth than with anything else). and they are then an excuse for schools not to read applications/accept poor students that might otherwise be bright.

Genomic Repairman said...

The GRE is such crap, does it mean anything any more. You just need to get above a baseline. Your experience and course work are way more important that the manufactured number and are more likely to be indicative of your success.

Space Prof said...

As a member of our department's grad student admissions committee, we put far more weight on the TOEFL score than the GRE Verbal or Analytical Writing score. The TOEFL covers 4 aspects of language and seems to be a good indicator of English abilities for international students (based on our small sample size).

In the end, though, the numbers are just used for an initial triage, and the the real selection is by the departmental faculty picking out those good applicants who have the motivation, experience, and skill set they are looking for.

Anonymous said...

I agree with a lot of the comments stating that the AW portion is useless. All through college I did OK with writing English/ History type essays but never got great scores on them. On the other hand, I got excellent scores on writing lab reports and such for all of my science classes. Now that I'm in grad school for biological science, I feel like writing is easy, probably because I write like a scientist, not an English major. On the AW portion of the GRE I got a terrible grade, probably for the same reason...

hkukbilingualidiot said...

One thing that might interest everyone regarding students from Asia. Their focus had always been exams. Going to school had mostly been to focus on learning how to answer questions that an examiner will mark as correct and would give you the highest marks. No attention had even been put towards intuition or thinking outside the box. So, when you may encounter really good researchers with great ethics coming from China, you really are extremely lucky. That was one of the reasons that I left China, HK specifically, for the UK. If I had stayed I would probably never have gone to university or even grad school, where I am now.

Anonymous said...

The GRE is a crazy thing. I'm horrible at standardized tests. I've taken it 3 times. The first two times I took it, I studied what they say is being tested and my scores were..ok..the third time I studied "tricks" and my scores went up ALOT. Aside from the GRE my academic record is pretty good if I might say. I have a 3.8 GPA and plenty of research experience. I am applying to PhD programs and have had 2 interviews (one phone, one email...in person coming soon) and one guy said he wanted a writing sample because my AW scores were a bit low and another guy said he didn't care about the AW scores and was more concerned with my quant score, which the first guy didn't have a problem with... I agree with FSP I don't think the GRE is all that predictive of research ability...it can't measure passion :)

Anonymous said...

Our admissions committee has this discussion almost weekly and we are particularly perplexed about the AW portion. For scientifically inclined students, we expect reasonably high Q scores (as anon at 8:34 says, the math is ridiculously easy). The V scores need to be reasonable, but we don't pay too much attention as long as they're not too low (below 50th percentile). The AW is the section we have no good handle on. Most of our US applicants have scores in the range 3.0-4.5, which seems ridiculously low (I just learned that there was a 6 last week - we've never seen a score that high). We're not a top 10 school but we are top 25 and get pretty good applicants. The committee has discussed taking the exam so that we can calibrate a bit better. I may push at least some of them to do that this year. It is a bit scary, though - I know I wouldn't do well on the V section because I don't care to spend the time memorizing vocabulary, but it would be embarrassing if I flubbed the Q!

Anonymous said...

As a non-science grad student in the humanities, I agree that the math part is pretty easy--I got a 720 without studying (and I haven't taken a math class since high school). The verbal, on the other hand, is basically a vocabulary test, so I'm not sure exactly what it correlates to. I came out of the GRE devastated by what I thought was a low verbal GRE score (compared to my 800 on the verbal SAT), only to learn later that I scored in a very high percentile on the verbal.

Comrade PhysioProf said...

Also based on personal experience advising students, I surmise that the Verbal score seems to indicate something about the complexity of an individual's vocabulary, but doesn't predict anything about reading/writing ability.

This is the complete opposite of my experience. I ignore the analytical writing score, and also ignore subject test scores.

Anonymous said...

Let me tell you something, I applied for ms in engineering and I had a Publication but I was rejected due to my low gpa or gre toefl cuttoff. Btw. I am an international student. While sudying and doing some research about applying universities in US what I understood and now I am sure is your aim is not research you even dont mind it your aim is gaining popularity and publishing some crap in a reputed place your aim is not discovering something new. When I read some papers published in top venues what I discovered is authors are doing their best not to invent something they are doing their best to find a crap that can be accepted. Look at the incerasing number of conferences publications and Phd's but our accelleration in science is thousand times worser than the 50's and 60's. Academic world is spoiled. High gpa for whom for what at where ? Do you have any idea about the situation of countries 8-10 flight hours far from your country ? Do you know their grading schemes ? You are evaluating students by magic numbers GPA.

I got admit from another university but I decided not to attend it. This reject confirmed my suspicions about US university system. Its primary purpose is not innovation, it is only about creating white collars who work for capitalism. Look at the second part of our history is there any einstein, any revolution in medical science like polio vaccine, what about chemistry, computer and electrical science has also same slow advance rate, new cell phones have only higher megapixel rates and better graphics.

Most academics and scientists live in a closed box without interaction with real world, you judge the applicants with some prejudice and give them chance based on that you have no idea about the world. Do you know fourier has discovered his transform during his visit to egypt ? You are impeding the progress of science and creating a system similar to in medieval age. You dont want Galileos, only the ones who follow the rules of the church is okey for you. US left producing quality science on 70's and switched from production economy to service economy. Another utter crap is reference letters if someone is doing science why does he need reference letters his letters are his work. But today almost no student have good work rather than a stellar gpa. Rules rule your world and nothing extra ordinary is generated from this system.

b(oston)s(cholar) said...

My experience with the AW component is precisely the same as M's. I wrote the cheesiest possible five-paragraph essay, but made sure it was long and grammatically correct and got a 6. I have friend (a gifted writer) who took it a year or two later--she gave the question serious thought and wrote at her usual sophisticated level. She got a 4.5.

Genomic Repairman said...

Anon @ 12:44pm

Sounds like sour grapes to me. But Anon, could it possibly be that your publication was in a shitty international journal and maybe that couple that with your low scores and maybe a higher quality applicant pool and you still come up with the same end result: You standing on the sidelines while we do science. Possibly we do reject too many qualified candidates from outside the US. But I look around me and we have tons of folks from across the globe. And if our brand of science is so bad, then why in the hell are you trying to buy into it? This makes you at best a whiner or at worst a hypocrite.

Anonymous said...

It all seems impossible.

1) GPA is flawed because 1 - it is easier to get higher grades in some classes than others, and 2 - it is easier to get higher grades at some schools than others
2) GRE is flawed (for science) because 1 - you can study for the test and increase your performance greatly without learning anything you actually need to know and 2 - it tests math for ninth graders and and vocabulary for graduate students in English lit.
3) Lab/Research experience can tell you how hard someone has worked to make their resume look good, and that they are not so incompetent they have been kicked out of their labs. But it can't tell you anything about their ability to think scientifically because typically undergrads will do what they are told.
4) Letters of reference seem good but 1 - they are time consuming to read and 2 - a bad relationship with letter writer and/or crazy letter writer can doom the student.

So what on earth are you supposed to do?

Anonymous said...

I don't know, I found the analytical writing section extremely easy (I got a 6). My only studying was to write a single essay on a sample question and read a range of possible responses to the same sample essay (mine was easily better than the 5, and roughly similar to the 6).

It seems like what they are emphasizing is that you can think about the issue from several perspectives, choose a side and then present evidence or argumentation for that point. My topic was related to whether academics should or should not be expected to use their research to help people "in the real world" (funny considering one of FSP's last posts eh?).

This sort of essay is basically what I did for every non-science course in my SLAC for weekly assignments so it was very easy to replicate the experience on the exam.

unlikelygrad said...

Anon @ 12:44 PM:

If I were on an admissions committee, I wouldn't admit you either. I would reject you not because of your sorry writing skills, but because of your lousy attitude. I wouldn't want to work with you.

If you want change, get in there, roll up your sleeves, and work. People don't change when you tell them how lousy they are. They change when they see that you're interested in them as people even though you have your own ideas that work better. Institutions don't change because they're under attack from without: they change because people on the inside innovate in small but significant ways.

I come from a non-traditional background. I, too, want to innovate. I tell the professors I work for when I think something is wrong, but I also acknowledge that they have more experience than I do. The other day, one of my bosses mentioned that he'd hated one of the freshman experiments for 20 years...as soon as the guy in charge of that lab retired, he went in and changed it. Sometimes, that sort of waiting is what it takes.

And BTW, if you think academia is disconnected from the real world...you're going to the wrong school. Many schools are linked in with industry. Interdisciplinary programs often are tied in heavily to the "real world." (My own program requires a good bit of field work.)

And when you say people are slackers, all I can think is that you must not ever read the literature. There's hot science going on out there. You may not know about it, but that's your problem, not the scientific community's.

Jane said...

Will anyone else weigh in on the subject GREs? I bombed mine, so I don't particularly like them, but I didn't like the idea even before I took the test.

I did ridiculously well on the general because I have the esoteric vocabulary, but not well at all on my discipline-specific test (chemistry). I certainly resented feeling like I needed to re-learn everything I learned in college...I've always had trouble remembering organic reactions (although I'm good at synthesis problems when I'm allowed to look stuff up) and I spent most of my time last fall working on research, TA-ing p-chem, and doing well in my classes. Should I have spent more time studying for the chem GRE and less time doing those things? I wonder what would have been the most valuable use of my time.

(And the other students I know from my department and from a comparable department at a neighboring school did equally poorly.)

Jen said...

I got a 4 on the Analytical section. I don't usually toot my own horn, but I know that I'm an excellent writer. Since I didn't conform to their cookie cutter 5 paragraph essay style and I had a horrible prompt, I did poorly.

Whatever, I still got interviews with all of the grad schools I applied to. Appears they don't care much about the GRE either.

Plague of Crickets said...

It's ironic that those of us who think of ourselves as bedrock empiricists continue to use GRE scores to evaluate students. The evidence that GRE scores predict anything about graduate student success has been pitifully weak (independent studies tend to show little predictability, while ETS' own studies, not surprisingly, show decent predictability).

I'll point the finger at myself here. As a faculty member evaluating students, I look for anything that can help to discriminate between applicants. There's just something insidious about the numbers that makes me want to think they can help, even though I'm pretty sure they're useless as an evaluation tool.

What we want is students that can think logically, critically, and deeply about the field; students who are passionate and persistent enough to persevere in the face of the inevitable setbacks of any research program; and students who have the problem solving skills to identify and correct the experimental design flaws that cause their experiments to fail. No general test – maybe no test of any kind – can tell us what we really want to know about students.

Anonymous said...

Standard tests are just are estimators, they will always contain white noise. Its even harder with young applicants because they are still developing. This is especially true for my subject, where failure rates for 1st-yr coursework are horrific despite very low admissions rates.

Admissions is an art, not a science. Maybe that's why science professors find it annoying.

As for the long post by the anon foreign student, it's true that people these days are doing very small bits to try to get tenured, but that is because even scientists need to make a living! You need something on your CV so universities feel safe to give you a tenure position. Once tenured, you can then pursue your risky, breakthrough idea along with your very little bits. If it fails, at least you still have other little bits of publications AND A SUBSISTANCE-LEVEL PAYCHECK.

As for the US university system, I can agree that undergrads are trained to be white collar workers (which irks me, because that means the US high school system completely failed to teach them those skills!!!). But I don't get your complaint about grad schools. You claim that grad schools are capitalist-serving, and then say that many academics live in their own worlds. Academics are almost always trained through the grad school system, and I think a greedy capitalist would not want to hire someone living in their own world, so something doesn't hold.

MommyProf said...

I'm on the admissions committee at our school and we do use the GRE. It's not a perfect predictor of success, but low scores do often predict failure. In our field, I've actually found AW to be a good predictor. I guess being willing/able to find out what kind of writing scores high and doing what it takes to get that high score counts for success in basketweaing.

We also get a large number of international applications with very high math scores and very low verbal scores. For them, if they do well on AW (4 or above for international students), we are more likely to give them a chance at being able to function at the graduate level in courses in English.

John V said...

Looking at applicants, my favorites are ones with all 3 GRE scores above 80th percentile, preferably above 90th percentile, and with strong letters from people I know.

Just the scores can mean someone for whom the concept of research doesn't play out, just the letters is difficult to calibrate given the ridiculous hype and optimism that shows up in the majority of the letters.

GREs are just part of the picture, but to say they are meaningless is silly, except in the case in which we have much greater perspective from the other application materials than usual.

Many applicants from unheard-of schools would not stand a chance without either personal visits or test scores, as there is no other way to calibrate their virtues.

Minos said...

One recurring theme I see in FSP's original post and in many comments might be expressed, "GREs don't mean a damn thing, unless they're really low."

I think this is actually very sensible, and here's why:

If you think about questions, in the sense of Bloom's Taxonomy http://en.wikipedia.org/wiki/Bloom's_Taxonomy or similar categorization schemes, you find that these exams ask mostly for factual information ("remember"), some attempt at explanation ("understand"), or at rudimentary problem-solving ("apply").

Standardized exams almost never ask you to "analyze", "evaluate", or "create" (the higher-level uses of intelligence or subject knowledge), and when they do (such as in an essay), that's not what gets evaluated by the rubric.

So the GRE and similar tests basically asks, "how good are you at low-level thinking?" I'm going to go way out on a limb and say that if you are terrible at lower-level skills (dealing with F=ma and free-body diagrams, PV=nRT problems, or basic genetics), you probably aren't going to light the world on fire with your graduate work. If you can't do the basics, you can't do the hard stuff.

On the other hand, above a certain level, mastery of the basics hits a curve of diminishing returns. The best analytical chemist is usually *not* the one who can solve the most complex titration problem. At the graduate level, the skills that separate the good from the "meh", are those that require you to analyze, evaluate, and create. This skills won't help you if you can't balance a chemical equation, but they are the stuff of great grad students.

Anonymous said...

Jane -- Your story sounds very similar to my own. I aced the general with practically no work at all (this was in the days when the logic portion still existed, not the AW). I did pretty poorly on my physics subject exam because I chose to enjoy my REU experience the summer before the exam instead of study all the time. I had my books with me in Ithaca and everything, but I was a much happier, balanced person who worked on my REU, made new friends, visited new areas of the country (like NYC, Boston, and Niagara Falls), and generally had a fun summer. My grades were good and I had research experience, so I still ended up in a good graduate program (not Harvard or Caltech, but who wants to go there anyway?), finished my PhD on time while doing research that is highly cited, and am now on a 3-year fellowship. So my surmise is that you're going to do just fine, as long as you keep working and pushing yourself.

Something that no one else has mentioned is the fact that the GRE is a measure of how well you tested during a few hours out of your entire life. I was horribly sick during my subject test, coughing, sneezing, sniffling and generally unable to breathe let alone think (and probably distracting everyone sitting around me). I'm sure I could have gotten at least a few more points if I had been healthy that day, maybe increasing my score from "marginal" to "mediocre".

Unbalanced Reaction said...

So glad I took the GRE before the analytical section changed. What a colossal waste of time.

Many, many of my colleagues on science grad admissions committees completely ignore the analytical score. I wonder if the humanities or social sciences grad programs look more at the analytical scores?

John V said...

I'm bemused and appalled at the distain for "low-level skills" in the posts.

Brilliant synthesis of far-flung topics and breakthroughs with incredibly sophisticated techniques are skills we like, and untestable with GREs, but it's also very useful to rapidly and accurately perform mundane chores.

While not the whole picture, the ability to score in the 99th percentile on standardized tests indicates some very useful skills in concrete thought and efficiency.

By the way, I predict a perfect correlation between posters performance on GREs and their opinion of GREs' usefulness - an indication of mistaking ego for statistical truth.

[I'm grinding through a thick stack of difficult-to-evaluate applications this morning - very apropos topic, FSP.]

Kevin said...

In our engineering program, we look at all three GRE scores, though they don't have as much weight as good letters about previous research experience.

I looked at who we are thinking of admitting this year, and noticed that no one had quant lower than 710. Verbal went as low as 460 (we worried about that, but the writing was ok, and the letters and grades excellent). Writing spanned the range from 3.5 (worrisome) to 6.

Note: these are all US citizens or permanent residents, though not all are native speakers of English.

Anonymous said...

I'm on the admissions committees for two engineering departments at my school. In both departments, the GRE Q must be above 750. The test is just too easy. Similar to comments by other posters, to me high GRE V and AW scores are evidence of students who are willing to research what these tests are about and spend a little bit of time studying for them. While I don't feel the skills actually being tested by the GRE V and AW are indicative of success as a grad student, the fact that a student takes the initiative to find out how to do well on the tests and is motivated enough to properly study (and has the time management skills to do this), is a great indicator of future research success in our programs. We like to see scores above the 70th percentile for these portions of the application.

Lizgb said...

Hi, I just found this blog today and was surprised to see you all discussing something that's been very much on my mind lately.

I just took the GREs this past summer (applied to evo. bio and genetics grad programs- 3 interviews next month!), did reasonably well but not outstanding and found them, like the SATs, extremely annoying and mostly useless.

One thing I noticed that was not discussed much here is the economic issues that go into these tests. The test fee alone is absurdly high (plus extra money for rescheduling or sending additional score reports) and then there are all these classes you're supposed to take to study. I was thinking of taking one myself (I know some people who have, and they really did improve their scores) until I found out that they're all around $1000! How is a recent college grad in this economy with student loans to pay off supposed to pay for something like that? If a rich student can take one of these classes and improve their score by 200 points, then how can these tests possibly be taken seriously?

Secondly, I don't really understand what the GRE is supposed to be testing. I am terrible at algebra and arithmetic (always have been) but aced every calculus or statistics class I've ever taken. Why should I be tested on math skills I've barely used since 8th grade? Same goes for the AW section. The essay prompts (and the example essays they give you) seem very similar to the Grade Eight Proficiency Assesment that I took in middle school. I got a 5.0 in AW (which I guess is ok?), and I know if I wrote an essay anything like that in any college course I'd be laughed at.
The subject tests seem to make more sense (at least they have something to do with what I learned in college), but none of the graduate programs I applied to actually required them. Go figure.

Kevin said...

"then there are all these classes you're supposed to take to study."

You're not supposed to study for the GRE. You're supposed to already know that stuff. Just because marketers found a way to make $1k by playing off people's insecurity is no reason to fall for their pitches.

I agree that the GRE quant test is way too easy. Like the SAT, it basically tests 9th-grade skills.
What is amazing is how many applicants to grad school lack 9th grade skills in math and writing.

Lizgb said...

"What is amazing is how many applicants to grad school lack 9th grade skills in math and writing."

I disagree that not remembering things like all of the rules for triangle geometry is somehow a fault. I graduated with highest honors with a double major in two sciences. I'm obviously not stupid when it comes to math, yet I really struggled when it came to the GRE. Because you have so little time, the GRE is testing you on how well you have memorized all of the little math tricks, not whether you can actually solve a math problem if you have more than 30 seconds to think about it.

Tas said...

Hmm, if I recall correctly, the analytical writing section was made up of 2 very specific essays: one was supposed to test the student's ability to recognize and point out logical flaws in an argument; the other was to test the student's ability to support their own argument.

However, having no idea really on how they are graded, and knowing that in real life one is rarely expected to write *any* argument in less than 3 hours, I can't say that the score indicates any sort of logical or argumentative skill.

but paper-writing is an integral part of any academic career... and paper production has an effect on pay-scale and tenureship...

Anonymous said...

when considering how meaningfull (or not) GRE scores are, why not look to yourselves and your fellow faculty colleagues since all of you had at one time taken the GREs too? Do the professors with tenure, and the most funding, the largest research programs, the most pretigious awards, the most papers in C/N/S, are these the people who scored highest on their GREs decades ago??

Did you score higher than some of your faculty colleagues on the GRES many years ago yet they are more successful than you now? Or vice versa? helps to put it into perspective. But unfortunately it doesn't really help with the practical and present problem of what to do about current students' GREs scores.

Or..maybe all faculty should retake the GREs now and see how your scores compare to the students' that you are evaluating based on their GREs.

Anonymous said...

I did actually "study" when I took the GRE (over a decade ago). I didn't take a prep class (didn't see the need to shell out money for that), rather a friend of mine who had just taken the GRE gave me their book that had practice tests in it or tips and tricks or somethign like that. I would definitely recommend to anyone taking the GRE today to go to any bookstore and buy a GRE test-prep book.

The GRE, like the SAT, tends to favor certain types of questions and topics so it's not about how skilled you are at the subjects but more about saving time by quickly getting to the correct answer without spending more than 10 seconds on it.

So my GRE test prep was basically about learning all the tricks to get the correct answer without actually solving the math problem directly. I don't remember my exact scores now, just that it was in the 97th percentile that year. So obviously you can get high scores just from knowing and practicing GRE-specific test-taking tricks.

I remember the verbal was very hard because it was all obscure vocabulary. but even here there are tricks that you can learn and apply to make educated guesses to increase the probably of getting the questions right when you have no idea what the words actually mean.

So basically IME doing well on the GRE is all about learning tricks and test-taking strategies and spending your time practicing those.

Jackie M. said...

I suppose the analytical writing is better than the old analytical test, which as far as I can tell only tested whether or not a student had played a particular sort of logic game on long family car trips.

Anonymous said...

I haven't had to serve on a grad school admissions committee yet; I'm just a postdoc. But from what I remember of taking the general GRE not that many years ago, it was an utterly trivial test, pitched at pretty much the same level as the SAT. I think a low score on any segment of the test would make me seriously worry about a candidate (with appropriate latitude for those who are not native English speakers). The analytical writing test is not a test of whether one is a skilled writer. It does, however, test whether one can arrange ideas in logical order, write in the appropriate register of formal English, and put together a concise essay on a given topic. I recognize that these are not necessarily skills that one must have to be a good scientist -- I have a valued collaborator who is a native speaker but whose prose I have to thoroughly rework just to make it grammatical and comprehensible -- but, nonetheless, a lack of these skills indicates a shortcoming in one's basic education.

It's true that the verbal test has a few questions asking about obscure vocabulary. But, from what I recall, they were few. It's mostly a test of reading comprehension.

Hope said...

@Anon 1/22/2010 02:21:00 PM: It all seems impossible….

I hear ya – I sometimes think this way myself. However, I think that the only thing that we *can* do is to try to make the best possible decision with all of the far-from-perfect data that we have. That’s why I don’t fetishize or completely ignore anything on your list. And I have a lot of sympathy for people who get excluded from programs on the basis of some GPA and/or GRE cut-off. But then again, I don’t sit on grad school admissions committees….

@John V: By the way, I predict a perfect correlation between posters performance on GREs and their opinion of GREs' usefulness - an indication of mistaking ego for statistical truth.

“Perfect correlation”? – I don’t think so, John V. I’ve always done very well on SAT/GRE-type exams. It’s led to scholarship money, offers of employment from test prep centers, and maybe even strongly influenced my admission to college and grad school. But I’m not convinced that these exams are all that useful. I would be willing to ignore someone’s scores if, based on the rest of the data, they seemed like outliers.

John V said...

I'm having trouble believing many posts here are written by scientists.

Repeatedly posts claim that if other factors matter - whether one is sick testing day, whether one learned test-taking tricks, whether one possesses the "right" vocabulary - then the tests are worthless. Factor analysis, anyone?

Imagine all the ways letters of recommendation can be fluff, imaginary, or perverse, yet we try to wade through them, and they prove very useful.

Maybe I'm unique, but my own best work is underlain by reams of grind-it-out rudimentary data analysis. Similarly, I scan vast libraries of homework, literature, proposals, and emails, in which fast and accurate comprehension at a low level is very beneficial for my daily tasks as a senior prof.

Maybe I should just view some posts akin to my daughter's view of the utility of the end-of-semester 10th-grade exams she took last week.

p.s. - I was correctly called out by Hope for hyperbolically claiming a "perfect correlation" between respect for the GREs in these posts and good performance. I'm sure the correlation is imperfect, and I'm hypocritical as someone no doubt increasing the correlation.

Doctor Pion said...

Your question reminded me of a student who abandoned a very good dissertation project ABD after about 8 years in grad school. This being pre-FERPA, one of the professors commented that he had come in with the highest GRE subject scores they had seen. In retrospect, he apparently taught recitations as an undergrad so he knew that part cold. Finishing the job by doing the boring writing part? Not so good at that.

I recommend a long-term (or retrospective) longitudinal study. Assign it to a senior prof who is no longer doing research, preferably one who complains about "students these days". I'd start by looking for a pattern (or lack thereof) after sorting them into Completed PhD, Abandoned PhD (i.e. passed comps), Failed Comps, Quit before Comps, plus a separate group if you bring in students with a terminal MS as the goal.

As an aside, I'd wager that "completed an Eagle Scout project" might correlate well with success.

Dusan said...

It depends who scores them. Especially with the high scores. I mean, the scorer has to have top abilities in the area. Also, scorer must not be tired, bored, etc. The same goes for the exam taker. Slight indigestion might get you lower scores.
In any case, I got very high scores, did pretty well in grad school but not nearly as well as I tested. Reasons: I worked too much, I was tired, professor bored the heck out of me.
Where's the correlation between scores and how interested professors are in what they teach?

AstroProf said...

I loathe the GRE. I took it in 1991, don't remember the exact numbers for the quant and the verbal. Loved the analytic part (it was all logic puzzles and combinatorics -- I only missed two of the questions and got an 800). The physics subject test killed me because about half of it was based on material covered in the senior year of classes at the University of Chicago, and I took the exam at the start of my senior year, so I hadn't seen a lot of it. I got something like 47th percentile. But I was doing research, and I went overseas for a year to be a research assistant, and I had a 3.8 GPA, and that was all enough to get me into MIT for grad school. I did take the subject test again after I graduated, just to see if that last year made a difference, and I moved up to 63rd percentile.

I still think the whole thing was a waste. You have 3 hours to do 100 questions. You can't *think*, you can't be creative, you can't *explore*, you have to either know the trick or not. I still remember one question about geostationary orbit, that I was so excited I knew how to do, I spent *FIVE* minutes deriving the answer, got it as a function of Earth radius... and earth radius was not one of the given constants on the help sheet. I didn't have the earth radius memorized, so my derivation was useless. Correct, but useless. Man, was I furious!

Anyway, I went to MIT, passed classes and exams, got papers published, wrote my thesis, got a post-doc, got an NSF fellowship, and I'm now a professor, and will hopefully get tenure next year. I don't think getting a better GRE score would have made me a better scientist.

lost academic said...

I see a lot of anecdotal evidence for one kind of thinking or another thrown around in the comments. In the past few (3) years, I have spoken with 2 researchers who have made studies to correlate success in graduate school (and undergraduate, in one case) with test scores (SAT and GPA). They both said that there is still no better predictor of graduate school success than a GRE composite score (they might have not been considering writing), and the one that looked at undergraduate success had the same evidence for the SAT scores.

SamanthaScientist said...

@lost academic

Did your colleagues publish any findings? If so, do you have references?

Do you know anything quantitative about the correlations they found? And what measures they used of grad school success?

(I'm genuinely interested, by the way, and not trying to interrogate you.)

Minos said...

Alas, I must disappoint John V. with a further data point which fits his correlation poorly. I am a physical scientist who was fortunate enough to have very good test taking skills. Despite my skepticism of the value of the GRE for separating good from great grad students, my own GRE scores were laughably high when one considers my actual abilities.

I neither disdain "low level skills" nor do I find the GRE to be worthless or silly; I do think that the GRE becomes less strongly correlated with key graduate student skills the higher the score becomes, because of the diminishing returns that come with very high levels of mastery of those lower level skills.

It's a fair bet that someone who scores in the 17th percentile in the subject GRE will be a better (albeit poor) graduate student than
the 7th percentile. The former will be a less disastrous tech than the other.

I'm less confident that the 97th percentile student will be a better graduate student than the 87th. At that level, both students will be *great* at those important day-to-day skills, and what will separate them is creativity, the ability to put ideas together from different parts of the literature, and the ability to take two conflicting pieces of evidence and decide which carries more weight.

freps said...

Finding this discussion fascinating. I recently took the GRE with the result of 800 V, 6.0 AW and 590 Q. Thus, I'd love to make some sort of well-supported argument that only the verbal scores and AW scores are useful, but alas, I cannot. What I can say is that I studied dutifully for the Q part and not one minute for the V part, which would suggest a quantitative deficiency for me. Surprisingly, I have had three statistics classes in which I scored a B or better, and have a very strong capacity to apply statistical data in research questions. I found that the test measured fluency --I am a (much) older test taker than average, and I work in a field where math is almost never used. While I zipped through the verbal section in minutes, I agonized over the quantitative section because I am not "fluent" in mathematical mechanics. The frustration I have is a) the number of people who declare the quantitative portion "simple," and b) the huge number of science majors taking the GRE, forcing what should be a slightly-better than average score to be slightly worse than average. It represents to some reviewers an almost certain lack of capacity which simply does not exist.

Just my .02 for the discussion.

Anonymous said...

I got a 4.0, which I'm sure will keep me out of many graduate schools. However, from experience I know that I have been complimented on my writing by a number of professors and have had to completely rewrite unintelligible papers written by people who have gotten 5.5s. So I'm not sure what it means. I do know that I had trouble writing in the environment that I was given, since the font size on the screen was huge and I would not have been able to see even half of this paragraph without scrolling. For me, that is hard to deal with.

If you look at the GRE studies, they all just examine correlations between the mean score and either GPA or writing grades. Well of course there will be a correlation with any reasonable test! Outside of a slight correlation, it appears that most of their research focuses on consistency of the test.

Then there is the fact that you are graded by 1 computer and 1 human. If the human is within 1 point of the computer, the human's score stays. But this is a 2 point interval! So people who are worrying about slight differences between scores should realize that it is the decision of one human looking at something about the length of the statement of purpose. It seems silly to have their opinion carry as much weight as the admission's committee's evaluation of SOP writing.

Additionally, there is sort of a loop. The computer is trained (with machine learning) to have a high probability of responding similarly to a human grader (based on words used, essay length, etc). However, human graders are hired with the expectation that they are consistent to the grading standards. It appears that they are expected to grade similarly to a computer.

Maybe I'm just cranky because I didn't do well, but it seems like a bad test to me.

Anonymous said...

Related NYTimes article

Anonymous said...

I just retook the GRE (my old scores passed the 5 year validity mark) and did fine on the verbal and math sections. I haven't received my analytic writing score yet, but given what I've read here, I wouldn't be too surprised if my score went down. I made a 5.0 the first time and I'm fairly sure I wrote a rather dry and conventional essay. On my recent exam, I just let it flow.

I read today that the AW score is actually determined (partially) by a computer program. That baffled me. The computer grading, coupled with many of the comments on this blog, has me convinced that there may be too much variation, in what constitutes a quality essay, for the score to be taken seriously.

I think if I were running an admissions office, I would just ask for a 10,000 word statement of purpose, and I would not require the GRE. While the idea of reading 10,000 word essays (or novellas) is daunting, I do imagine that the number of applicants would drop drastically if that were a requirement. The majority of students today are lazy, and it's really a good work ethic that's needed for success in graduate school. As for intelligence and ability, these things should be fully evident after reading 10,000 words.

The motivation to put forth effort just cannot be underestimated. If the applicant can't put forth a simple 10,000 word essay, then I'm fairly certain they won't be able to write a thesis.

My two cents.

Kevin said...

"The motivation to put forth effort just cannot be underestimated. If the applicant can't put forth a simple 10,000 word essay, then I'm fairly certain they won't be able to write a thesis."

While I agree that writer's block and other inabilities to write are one of the biggest obstacles to students finishing a PhD, requiring 30-page essays that no one will read is not going to help. The crucial thing missing from most student writing is a genuine audience and awareness of that audience. Providing yet another audience-less word dump is not going to help students learn to write, nor are 30-page essays going to help admission committees decide who to select.

Craig said...

I recently took the GREs after a lengthy period of on-and-off studying. I scored in the 94% percentile on the verbal and absolutely bombed the math--since I am applying to writing-only grad programs, I didn't study at all for it. Hopefully that won't bite me in the ass.

Anyway, about the AW section (only got a 5 in that one): they are graded by a human and a computer. The human is something of a TA equivalent hired by ETS, not a certified expert. That human grades the score. Then the computer "reads" the writing and determines, god knows how, another score. If the scores agree, there's your score. If they differ, another human reader comes in and evaluates it, and the score is averaged between the two humans.

The program is called the e-rater, according to the standard ETS score sheet explanation I received in the mail.

Also, I think the Princeton Review GRE book I have said that the readers score your tests "at the speed of light." Which was funny and aggravating. And probably very true.

Seriously, though, a computer.

Anonymous said...

AW tests fluidity of writing, in my opinion. Personally, I value that. I know most people don't.

Spelling errors cause minimal changes in the eRater score, as do logical fallacies. But if your writing is fluid-which goes a long way toward being generally convincing- the AW portion is easy-peasy.

Anonymous said...

I am a TA for my undergraduate classes in clinical laboratory. I scored relatively low on my quantitative, because I did not prepare myself by learning these tricks you guys are talking about. I guess I have learned my lesson.

The guy I am teaching scored really high because he knew the "tricks". He will most likely be accepted, even though he is always late turning in assignments, misses class frequently, talks to other students in the middle of lecture, and just has an overall lack of responsibility. He constantly needs to be reminded of his excessive use of Facebook in class. Last semester, he didn't turn in writing assignments, because in his own words, "they take up to much time, and only count for 70 points total".

Please excuse this example of bad writing, as it is only two paragraphs--excluding this closing statement. I'll leave it too you to decide the point I am making about the credibility of the GRE. Your reasoning skills should be high enough as to not make me have to spell it out for you word by word--pun intended.

Anonymous said...

Anonymous on 1/23/14

I took it this year. I did not study for the verbal and scored in the 93rd percentile. I did study for the quantitative portion and ranked in the 63rd percentile (not bad considering I haven't taken a college math class for over 10 years).

Now the AW was another story... I received a 3.5, which is utterly pathetic. Pretty strange considering I have never received anything less than an A on a written paper during my entire college career (top 20 national public university).