When I was a grad student, I participated in the usual seminars and journal clubs at which a group of students +/- faculty read and discussed journal articles. Typically, the group would take each article apart in great detail: text, figures, tables, everything.
I was fascinated by this. I found it intensely interesting to scrutinize a paper in extraordinary detail and argue about it with a group of people with different points of view and personalities. It took me a while to gain enough confidence and knowledge to jump into the discussions, so in my early grad years I listened much more than I spoke, but later I got more comfortable contributing to the discussion.
These discussions were not for the sole purpose of tearing everything down and determining that everyone (else) was stupid. Of course any paper can be criticized, but in general we found something worthwhile in most papers. Some contained fatal errors, and I admit that it could be fun to find these, but most papers, despite their flaws, have something of interest. We were certainly highly critical of the articles, but (at least, for me) the main purpose of these discussions wasn't to attack and destroy.
I learned a lot from these discussions of journal articles, and so, as an assistant professor, I looked forward to teaching seminar courses or leading other such discussions with my own graduate students and postdocs. I wanted to show my students how to look closely at a paper and extract its essence, evaluating the data and ideas, and how to be critical of methods, assumptions, and interpretations.
And my students hated it. They hated that every week we criticized a paper and tore it apart. They found the experience deeply depressing.
I altered my approach a bit with later journal discussion classes and groups. I tried a more balanced approach, so that it was obvious that these articles had content of interest, otherwise we wouldn't be discussing them. Students still hated it. They didn't want to read the articles in as much detail as I was asking them to do, and they didn't want to be so critical. They thought I was being too mean.
When I moved to a different university, the culture of my new department was a bit more serious and I had more success with students who enjoyed detailed journal article discussions, but I have never again found the type of stimulating environment that prevailed at my grad school in this particular respect.
That's actually OK with me. Now I am satisfied with a much less intense and critical discussion of articles. I'm happy if we focus on the core concepts and interpretations, and if everyone learns something from the reading and discussions. You don't have to take apart every sentence and figure and table to get a lot out of this type of exercise. If an article makes us think and leads to interesting tangential discussions, that's great.
I wonder, though, if students who don't participate in the attack-dog style of journal reading are learning less about how to put a paper together, and are not as prepared to review manuscripts if they end up in an academic career after graduate school.
The intense paper deconstruction in which I participated as a graduate student was a great education for me in terms of the mechanics of what goes into a paper and how best to construct a solid paper. But maybe there are other ways to learn this skill; perhaps just by diving into writing and getting a lot of feedback is just as (or more) effective.
And as for learning how to review: Perhaps reviewing skills can be gained in part by looking at reviews that others do of one's own submitted manuscripts.
I know that journal clubs are alive and well at many institutions and I think that is a good thing. I don't believe that the culture of attack-and-destroy for these discussions is harmful and instills a culture of aggression and contempt. I think that intense experiences with criticism and discussion of published work can be extremely valuable training for intellectual development and acquisition of knowledge about how things are done (e.g., the mechanics of putting together a paper).
However, I no longer think, as I used to, that such experiences are critical to graduate or postdoctoral training. That is, I think students and postdocs do need to learn how to be critical -- to question assumptions, examine the evidence, think about other interpretations -- but there are other ways to get there.
14 years ago
26 comments:
My supervisor has a journal club type thing too. There are a few people who do criticize a lot - which I find a bit intense, but then it makes me think a lot when i write my own papers. I look for potential places that people can criticize and try to fill in the gaps - so overall it makes for a smoother review process as well.
Another good thing about these sessions is that since our group works on a few diverse topics it gives everyone a chance to learn about something that might not be in their immediate work.
In addition something that is particularly good for our group which has over 90% non-native English speakers - it gives them a good opportunity to present in front of an audience leading to more confidence in conferences etc.
I've had exactly the same experience. I participated in really excellent, detailed, 'attacking but not mean' journal club discussions during my master's program, but have yet to replicate the experience either as a PhD student or when I've tried to set it up as an assistant prof. When I've tried to do this in graduate classes I haven't been met with negativity, in fact students 'say' they enjoy the discussions, but for the most part I've been disappointed by the discussions. They need a lot of encouragement, from me. So many times discussion stalls and I have to push it back along. It's just not like I remembered from my own grad school days. Maybe it's just a rose colored glasses thing?
But I agree, although I think I benefited greatly from the attack dog sessions, I don't think these discussions are absolutely necessary. Same goes for attack dog seminars. The same institution where we had the great journal discussions was also pretty probing of speakers, and I loved it.
I wonder, though, if students who don't participate in the attack-dog style of journal reading are learning less about how to put a paper together, and are not as prepared to review manuscripts if they end up in an academic career after graduate school.
Of course they're not, nor are they as prepared to be as critical about their own work. It is a fucking disgrace that your institution has allowed the ignorant preferences of its students to weaken their graduate training in this fundamental way.
I initiated a journal club in my graduate department because we didn't have a place for that kind of critical, broad discussion of scientific literature. At our journal club, only PhD students attend, and one of us gives a presentation of a paper he or she is interested in. We have pizza, and we usually interrupt the presentation a lot to ask questions and discuss. We are a very interdisciplinary department, so each of us brings very a different background and expertise to the table, and we are usually able to exchange knowledge relevant to understanding the paper.
I sell the club to the other students as a chance to practice their presentation skills, to get deeper into some interesting literature, and to get to know each other better. So far, it seems to work pretty well.
It is probably a bit more informal and friendly than the "attack-and-destroy" atmosphere you describe, but we usually find something to criticize in the papers and in most papers, we find something useful. We always come away having learned something.
As a grad student not going into academia, I found these discussions very useless beyond 15 minutes of conversation. After reading a paper and discussing the highlights for a few minutes I am ready to move onto my own work in the lab or at my workstation.
There is nothing I hate more than the person who adds 'one last question' about the paper during an awkward pause 45 minutes into the discussion. Let's finish this!
After two years of this I started to skip group meetings and just go to the bar with other grad students with similar views and we would talk about our research (sometimes) instead of a paper I could care less about. On a related note, why anyone would schedule a group meeting at 3:30 on a Friday is beyond me.
I know as a graduate student, I have changed my attitude a lot from when I was a first year to later in my career. When I first started, I had the same attitude as your first group of students - why was everyone being so nit-picky? I thought the article was really interesting and they did cool science - why was everyone so focused on what they DIDN'T do perfectly?
But as I got older, I realized that a lot of the time the picky things really were vital to the message the paper was trying to convey. In fact, I learned that a lot of why I found the papers interesting on first glance despite their flaws was because those authors were great spin-artists, or to put it less cynically, they were great paper architects. :)
Now I find those fatal flaws, and I get the same type of pushback from junior graduate students in my reading groups that I used to dish out! :) And I just kind of chuckle to myself.
I had a similar experience as a grad student, and I love journal club for it. I learned how to think critically, and on what pitfalls to avoid when writing my own papers. Almost every paper, no matter how many holes there are is valuable. That's what science is about. A good experiment ends up with more question than it answered. And the flaws in a paper, just mean that there is always more work to be done.
When submitting a paper, scientists ought to make a good-faith effort to be thorough, but research is never done, so you can't wait forever either.
I thought that "tearing a paper apart" was instrumental in my approach to science, and I do see that students who are not exposed to that kind of scrutiny of scientific information are more lacks in there own research.
What other ways? (not saying I disagree, just wondering if you could elaborate).
I would say that most of my late undergrad / early grad training was learning how to critically read a paper. Not being able to do this is effectively the same as taking each paper at face value.
Having said that, none of us have the time to read each paper as deeply as we did when preparing for a J club/graduate course, so we never read everything as critically as we should, especially when so many critical controls and reagent validations are confined to the supplemental figures, as they are today. But still, I think "practicing" critical reading skills helps make the process more intuitive so that we have a better sense of what to take away from papers we need to read more quickly (and by "read quickly" I don't mean read the title, the last author's name, and the abstract).
The problem I have found with attack-and-destroy journal club approaches is that I tend to pay less attention to the true value of the paper. I prefer to think about how the research could have been done better, how paper could have been improved, and whether the results were valid despite the flaws. It's easy to criticize, but in many cases the results are still valid and/or there was no way around the flaws.
I can't generalize to anyone else, but those types of journal clubs were critical to my training. I first experienced them as an undergrad in a lab (in the late-90's) and then again as a grad student.
If nothing else, taking the top research in the field and finding the holes was a great way for me to understand that no one's research is sacrosanct and there were places for even new people to contribute to knowledge
I think journal club (jc) discussions are very personality driven. The members of my grad school lab were very critical of manuscripts, and it was fun and educational to be a part of this as a student. When people from other labs joined in our discussions, or members of our lab participated in other jc's, I noticed that we were much more critical than the "outsiders".
As a postdoc, I've learned to enjoy a more laid-back approach to jc's. I miss the brutality sometimes, but I think the students are getting something out of the tempered discussions. Still, some of the most exciting and educational jc's these days are the ones where a paper gets torn apart...it's nice to see the grad students get revved up for these discussions.
I didn't have journal club discussions as a student (not a part of the field I was in---nobody read much and nobody shared what they read). Journal clubs are very much a part of my current department. Almost all the lab group meetings do a mix of research presentations by the students and journal club presentations. And the grad students don't find this to be enough, so there are several journal club groups that have been started by students. Some are "fan clubs" looking for god ideas and not too critical of what they find, others "attack groups" shredding the papers they read and looking for how to have done the experiments right.
Unfortunately, in my field it is much easier to find papers that deserve shredding than ones that survive close reading. The groups that consistently produce good papers produce very few. The groups that produce terrible papers are prolific.
I agree that these journal clubs are excellent environments for fostering critical thinking. During my graduate studies, a number of us felt as you mentioned, shy to partake in the discussion, even in our more senior years of the program. Part of this may have been due to the small size of our department therefore we were afraid of saying something stupid infront of our committee members. However, some good came out of this. A number of us started our own student only journal club that would meet over the lunch hour the day before to go through the article on our own. Two outcomes came of this. One was that we all became more comfortable speaking at the departmental journal club since we would realize that our points and questions were not "stupid". The other was that we began discussing other articles outside of the ones for journal club that we had to review in preparation for our candidacy exams and thesis defenses. We even began setting up mock candidacy exams and defenses for each other.
I miss that environment as well.
These kinds of discussions are really central to work in the humanities, for the exact reasons you state: picking apart an article (even going word-by-word, what English profs call "close reading") not only helps you understand the subject matter, it helps you understand how to structure an argument, introduce counter-evidence, etc. Even if the article or monograph has a lot of flaws, I always ask them what the author did well, to help balance it out.
My students don't like to get in depth either, and I think it's partly due to the short attention spans that students have nowadays. They are accustomed to everything being fast. Fast food, fast internet, etc. The value of taking one's time is often lost on them, I think.
I forgot to mention in my previous comment that my students have tended to conflate "critical thinking" with "criticizing" authors. When I tell them to be critical when doing peer review over other students' work, I always have to point out that reading something thoughtfully and carefully, drawing connections to course material, etc. is what I want; not for them to slam their fellow classmate.
Is there something cultural going on? The standards for what constitutes 'mean' and the characteristic ways of 'making constructive comments without being mean' are very different in Boston, Minneapolis, Atlanta, and Los Angeles.
Teaching in a part of the country I have never lived in, I manage to unintentionally insult my students nearly every week. Fortunately they are culturally competent enough to recognize me as a 'foreigner'.
That is, I think students and postdocs do need to learn how to be critical -- to question assumptions, examine the evidence, think about other interpretations -- but there are other ways to get there.
Perhaps reviewing skills can be gained in part by looking at reviews that others do of one's own submitted manuscripts.
I think you bring up an important point. It is crucial to learn to critically analyze a piece of work, identify the pros as well as cons of the approach as well as the presentation (i.e., look at the science as well as the mechanics of writing -- how to structure manuscript, where the emphasis should be, etc.)
There are multiple ways to get there; journal clubs are one, but they don't work for everyone and perhaps not for every group size and subdiscipline.
I had many of these rip-them-to-shred journal club experiences as a graduate student. I remember it being a challenge for all of us students to think of something positive to say about the papers (though of course, there should have been plenty). It really came in handy when it came time to write my own stuff up. I kept imagining what a similar journal club would do with what I had written ... the horror! ... but I'm sure it resulted in better manuscripts. I imagine it would be quite a shock for someone without these experiences to receive their first reviews back!!
Anonymous at 6:29, I always schedule our lab meetings for late Friday afternoon. It's the only time that everyone always has free. And last I checked, the weekend doesn't start until Saturday.
On the sliding scale of journal club approaches, I'd say my group tends to be pretty harsh with a mostly "attack and destroy" approach. As a younger grad student I never considered that this was "mean" or in anyway taking away the value of a paper at all. In fact, I found it comforting that problems could be found with most (even high profile) papers. It simultaneously taught me how to write better papers while also taking away some of the pressure I felt to write "perfect" papers because I learned that no such thing existed. Sometimes our advisor would dial back the criticism of a paper in order to point out that there was important value in the paper, and sometimes she would basically agree that it was total crap. I have found that there is a sort-of right of passage in our group: the first time a grad student is willing to stand up to our advisor when she thinks the paper is horrible, but the grad student thinks there is important value. Learning to defend one's views in any scientific discussion is important and it is often more difficult to defend an idea or paper as good and valuable than it is to rip it to shreds.
Being critical is good, but it is possible to take it too far, in my opinion. Nothing is perfect, but if you can't find anything good to say about a paper, you're probably being far too critical.
I say this because there's an ongoing discussion in the program committees and NSF panels in my field about whether we are too critical as reviewers. My field is driven by publications in conferences with very low (~15% or less) acceptance rates, which means that any imperfection can become grounds for rejection. The argument is that this encourages reviewers to find the imperfections, ignoring the stuff that makes a paper good, and this in turn encourages "incremental" or "technical" work that's hard to mess up. And I do find that in the journal clubs and paper review meetings, students tend to focus on the negative rather than the positive.
I have no idea whether this is measurable though. Just more anecdotes.
Anonymous@2:57 sounds like a computer scientist to me, but CS might not be unique.
Speaking of writing reviews, I was on my first (CS) conference programme committee recently and found it quite useful to read other peoples' reviews of the papers that I had reviewed. It's not something that one usually gets to see, and helps calibrate expectations, similar to journal clubs (which I never really participated in while a grad student).
@Trabor, I agree with anon@6:29. Never schedule a group meeting on Fridays. Fridays are exclusivly to take care of small things at work, taking a long lunch, then going to the bar in the early to mid afternoon to maximize happy hour. I enjoy reading papers and tearing them apart just as much as the next person, but let us do it at 10am on a Tuesday! If I were in the same lab as anon@ 6:29 I would be skipping group meetings too!
Our department used to do journal club with all faculty and students meant to attend. This was educational, and with sometimes quite high-level discussion from the faculty (and gossip too, if some people knew the authors).
But the students hardly ever spoke up. Mostly out of the slight intimidation of having the many faculty there. So now we have journal club where a student presents a paper for discussion, all the students are expected to attend, and a rotating roster of two faculty members "moderate" (ie, poke the students into discussion by posing questions, etc).
This seems to work pretty well. The students are less afraid of asking "stupid" questions, so now everyone is more likely to learn. And now they are not afraid to identify weaknesses in an argument in a paper.
And like all students, I thought they were a huge annoying waste of time at first. Now I can see how they are making me think about the papers I have to read for my own project, and also my own arguments and writing.
The one attack-dog journal group I sat in on felt like a huge waste of time. There was little explanation of why we were critical of particular details. It was just a lot of jockeying by graduate students to come up with the most criticisms. After we got all the reasonable ones out of the way we'd delve into the ridiculous for the remainder.
"found it quite useful to read other peoples' reviews of the papers that I had reviewed. It's not something that one usually gets to see, "
Most of the journals I review for now send reviewers copies of all reviews at the same time they send them to the authors. I find this very helpful. If all the reviewers found the same flaws, I feel I've not been too picky. Sometimes one of the other reviewers picks up a problem I had missed (though not often). Often one reviewer is way out of line (usually gushing about a crappy paper, but sometimes being overly critical of a good one).
Also, seeing the detailed reviews of other reviewers points out to the lazy ones what they are supposed to be doing.
Distributing the reviews to all the reviewers should be standard practice for all journals.
we have a similar journal club with complete dissection of the paper of the week. i find this to be very valuable in learning how to put together a paper and how ot review one later. my only gripe is that the PIs choose the papers and 95% or more of them choose something that is a "good example of a bad paper". examples of great papers in my field would be most helpful in conjunction wiht the bad eggs.
Post a Comment