Wednesday, April 07, 2010

Playing the Game (1)

This spring, I have to refresh my ethics training, a ritual that is annoying mostly because the university makes it so extremely difficult for faculty and postdocs not involved in the biomedical sciences to find relevant ethics training opportunities. I am really not that interested in spending 2 days learning about research with human subjects, as my research involves no human subjects (other than grad students..).

And now all our graduate and undergraduate students who receive a salary or stipend from NSF must be trained in ethics.

Ethics training is important, but it is boring: don't plagiarize, don't fabricate data, don't falsify data. And the case studies are bizarre.

So let's forget about whether the postdoc should hide the data outliers that may or may not be due to a power fluctuation during extremely expensive analyses at a national lab even though the grad student involved in the research thinks that they should at least inform their PI, and consider some more interesting situations. For example, following on yesterday's citation theme:

Has your awareness of the importance of citation data affected your decisions about publishing?

That is, would you have made a different decision about anything related to publications (authorship, type/number/format of publications etc.) if your citation metrics were not being constantly tabulated for all the world to see?

When answering the first-order question, consider only whether you have ever considered the impact of a publication decision on your citation indices. Such considerations are not a priori unethical; it is possible that such considerations will result in your making a completely ethical and smart decision that helps your career. An example of that might be to send a manuscript to a journal that is indexed instead of to an edited volume being published as a not-indexed book. You do not change anything about the authorship or content of the paper; just the publication venue.

But now consider whether you have ever made a citation-fueled publication decision that might not have been entirely ethical but that could possibly be justified because "That's how the game is played." (others are doing it; if you don't play the game, you lose; etc.).

I don't think I've made any unethical decisions about research publications, but I definitely had citations in mind a couple of years ago when I decided to revise a much-cited technical document that was originally authored by someone else (now long retired) and that was in great need of an update. In that case, though, I wasn't thinking so much of my own citation index, but was instead intrigued by the idea of transferring citations from Journal 1 to Journal 2, in part because I am involved in the editing of Journal 2 and therefore have an interest in promoting the success of Journal 2. I hasten to add that I am paid nothing for my editorial work for Journal 2, so financial considerations were not involved. There were in fact valid scientific reasons for publishing the update in Journal 2, in addition to my personal affection for Journal 2. In the end, though, I published the update in Journal 1, but I admit that my original intentions were rather craven and driven in part by thoughts of citations.

That's the worst example I can think of for myself, but perhaps that is because I am a tenured mid-career professor. If I were on the tenure-track today, I am sure it would be difficult not to consider citation index issues when making publication decisions, although if it makes any of my early-career readers feel any better, I happen to know that at least some promotion & tenure committees at research universities are specifically instructed not to consider the h-index or other citation statistics when reviewing files for tenure and promotion.

to be continued..

18 comments:

Grad student said...

Dear FSP,

This is a very interesting topic to me but I don't really understand what you mean by 'unethical' here. If I break up a big project into as many Least Publishable Units as possible, is that unethical? If I do a project which I don't think is very important scientifically, but which will get me lots of citations, is that unethical? To me these are somewhat cynical moves, but I wouldn't consider them unethical. What's the most egregious example of citation-centered unethical behavior you can think of?

Kevin said...

I don't think about my citation counts or h-index when I decide to publish---but I do try to place papers where people will read them. If I'm going to go to all that work of writing a paper, I'd like someone to read it.

Once your h-index gets into the 20s and 30s, it is hard to manipulate it much---you have to get at least 20 papers to cite your new work to get an increment of one to the h-index.

One of the biggest problems with h-indexes is that they have very different meanings in different fields. Huge fields (like molecular biology) can get h-indexes for rather modest work that only a demigod in pure math could come close to.

Adam said...

At my institution we are just now having a debate about whether ISI citation reports should be required for inclusion in a candidate's dossier for promotion. The idea behind this is not that citations should be given more weight than they already are, but that letter-writers will refer to citations in any case, but may do so based on incorrect information, e.g. because they did the ISI search wrong. Having the candidate do the search themselves and annotate it with any necessary comments allows them to proactively correct any such errors and address any other real or perceived issues (e.g., pointing out publications not listed by ISI).

I find this argument somewhat compelling because I have seen the scenario happen - in a faculty meeting where we were making a hiring decision, someone claimed something (negative) about the candidate's citations which was just wrong; the claimer didn't know how to do an ISI search. It is pretty easy to do it wrong if you don't know what you're doing, and the error is always negative - fewer citations than the person really has. I was able to correct them in this case because I had just done the search right myself, but that was just lucky.


Of course others say that requiring the citation info will lead to greater emphasis on it, too much power to the h-index etc. They are probably right, but the trouble is that the information is out there anyway. We have a decision about whether to allow it in explicitly so we can try to control it, or try to suppress it in the hope that that will reduce its influence (but let it be uncontrolled and sometimes have the wrong influence).

Anonymous said...

Maybe I'm too early in my career to have encounter this (post-doc), but I don't understand why considering impacts on your citation value would be considered unethical. You're not undermining the science, just deciding where and how to display it...

It seems to me like you're saying that "marketing" yourself by being selective in how and where you publish is unethical. But, as far as I can tell, there's no lying, cheating, stealing, etc involved in this scenario, so how do ethics concerns apply?

Anonymous said...

I'm TT, and admin here cares a LOT about these stats. Very tempting to do anything you can to get them up, to the point that I once caveat-ed it in a referee report (accepting the paper would raise my h-index! go for it!). These numbers matter a LOT to me right now.

Anonymous said...

As a grad student/post-doc I worked in labs where to be an author you had to do significant work. Therefore, I have n papers, where I did most/all of the work and most/all of the writing. I have no papers where I contributed a passing comment or did one tiny experiment (perhaps this system biased me away from those contributions; I am acknowledged on some papers).

I was fine with said system and fine with my particular n, but as a junior faculty I am now being dinged on grants for a low n relative to peers. Many of these peers are in labs that follow the system of if you did anything (and sometimes not even that), you are an author. I personally don't like that system, but I have debated if I should switch my own lab to it in order for my students to be competitive...

I would be curious to get a sense of what the cultural norms are for authorship - in my biomedical training you had to 1) conceive idea and/or 2) contribute at least 1 figure of data. To be first author you had to write the paper as well (and likely have conceived many of the ideas).

John V said...

I think decisions to maximize citations generally align with actions that best present results, so I don't see much possibility for ethical issues.

Two that are related are (1) degree of shingling - multiple paper, especially with self-referencing, can boost citations, and (2) abuse of position as reviewer or editor to require authors to cite their work. Both are at most a minor effect.

Anonymous said...

I second what "Grad student" says -- I've definitely been influenced in somewhat cynical ways by citation counts. That is, I've spent time working on topics that I consider less important than other projects that I set aside, because I watch the trends in citation and see papers piling up about the less interesting topic and decide that as a postdoc I can't afford to not take advantage of the trend.

In fact, I'm kicking myself for not being faster to jump on a trend a year and a half ago, as almost every other reasonably competent postdoc of my approximate age in my field has picked up a highly-cited paper with minimal effort, while I dismissed the trend as less interesting than longer-term projects I was working on that just aren't fashionable at the moment.

I'm not sure how ethics plays into this, but I think it's nearly inevitable that young people will sacrifice science to fashion to some extent because it's almost necessary to get a job, at least in certain fields.

zb said...

I have no idea how consideration of citation indexes could result in unethical behavior, and the example you cite certainly doesn't.

Perhaps if Journal 2 was completely irrelevant to the topic of the paper, and you stuffed it in there anyway using your power as an editor, it might sneak towards being unethical. But, it'd have to be a pretty dramatic incident to slip into unethical territory.

Citing irrelevant (truly irrelevant) papers of friends, family, and yourself, might have a a hint of the unethical about them, but again, it would have to be dramatic (i.e. cite a paper on chemical transitions in a cognitive neuroscience paper, by just sneaking it in, with no argued relevancy) to be unethical. The likelihood that someone would use citations in a way that was unethical and important seems unlikely to me.

On the other hand, the scenario of filtering outliers, I find to be realistic, a potentially corrupting scientific influence, one that can easily become unethical. I can't tell whether it's 'cause you think it's so obvious that it's wrong, that you don't think it needs to be taught? I think that's actually a very realistic scenario, and training in it would give everyone involved a frame on which to hang situations they'd actually deal with.

I think it's important to separate the cynical from the ethical because otherwise we weaken the protections against the unethical. When people see many people behaving cynically, they might feel that justifies their behaving unethically, but the two are different.

female Science Professor said...

These decisions may or may not involve ethical issues. If you decide to publish a few smaller papers instead of a more comprehensive one for the sole purpose of getting your paper number count up, that's just "playing the game". If you make decisions about authorship (e.g., adding yourself as an author to a paper that could reasonably have been published without you, leaving out possible co-authors so that your name is more prominent) to boost your publication record, that's not so good. A better term than "ethics" for this general topic would be one that just refers to being a responsible researcher.

Anonymous said...

Am I the only person out there who does a reverse LPU: collate small results into a grander whole making sure to exhausts the subject so that your work becomes the definitive reference?

balanced instability said...

I think that "massaging" the author list would certainly qualify as a breach of ethics. In my field, many journals have begun to require that individual contributions are expressly acknowledged to prevent undeserved authorship.

I have not every paid attention to my h-score. Maybe this was naive (I am just starting as a tt asst. prof). Maybe it is because the field I'm in hasn't focused on that (to my knowledge). I have always been in an environment where you tried to publish in journals that people read.

With regard to using citation indexes for hiring or promotion decisions, I feel that this is just lazy. Why should some arbitrary number that is, at least in part, a popularity contest, be a substitution for an individual's ability to evaluate the body of work in a tenure packet?

Alex said...

I don't have anything to say on this particular dilemma, but I like that you're blogging about ethical questions that don't show up in standard training videos, and I hope you blog more of this. Even in the aspects of scientific ethics that don't come from human subjects medical research, training videos often reflect the culture and perspective of biomedical research rather than other branches of science. Watching these videos can give a cocky physics grad student (I speak purely hypothetically, of course, and not from experience, *cough*) the false impression that other branches of science simply don't have to worry as much about ethics. We do, but the dilemmas are often different. Some of them are even subtle and less obvious than "don't fabricate data" but the training videos don't often take us there.

tideliar said...

"And now all our graduate and undergraduate students who receive a salary or stipend from NSF must be trained in ethics.


With regards this ^^^^

there are some short cuts you might take. We arranged a one day general "Responsible Conduct of Research" session for our postdocs (www.nationalpostdoc.org has some good resources). Two Faculty speakers and a guest who a was a friend of mine from a local institute. We made sure every got a certificate at the end of it and we can now say that all our postdocs have RCR training for a year...

The Lesser Half said...

"Many of these peers are in labs that follow the system of if you did anything (and sometimes not even that), you are an author. [. . .] I have debated if I should switch my own lab to it in order for my students to be competitive"

Yes, I've wondered the same thing. I only have 1st authored papers, but I've been told that I lose out on job opportunities to people with fewer 1st authored papers, because they have a large number of trailing author papers that make their CVs look more impressive. I was just not afforded the opportunity in grad school or postdoc to collaborate, and the few times I did the collaborators didnt publish. Now I work at a U where everyone who comments on a paper is a coauthor.

Once I'm TT, I think I'll include everyone who contributed even a tiny bit (especially students), because it wont hurt me but it will help them. Is it unethical? By definition ethics are relative to societal norms, and if the academic "society" says I should do this, then I guess I will.

Anonymous said...

You should have a poll on how many people immediately checked their citation indices after reading your last posts!

FSP, I would really appreciate a few posts on the politics, ethics, etc. of being an editor. I just recently became an associate editor for a journal in my field, and am feeling overwhelmed and unprepared - I never realized how hard it was to be fair and ethical about choosing reviewers, to decide if a ms is good enough for a given journal, strategies (or counter strategies) that editors use to make sure their particular subdiscipline is well represented in a journal....

This despite many years of grousing about editors who have handled my manuscripts and how I could do it better.

If you have posted on these subjects before, I must have missed them (or didn't realize how important it was). It sounds like you and your commenters may have some good insights.

MathTT said...

I'm right now refereeing a borderline article which cites my work. So... if I suggest an acceptance, how much is that influenced by the fact that I would get a citation out of it? I'm having a hard time deciding where I would come down on this article if it didn't cite me. (Other than, "this article should cite me because it builds on my work!" but you know what I mean.)

It feels squishy, and I've been procrastinating the review all day. Hence my presence here, in fact.

Anonymous said...

Regarding uninteresting ethics classes, when I was an REU student, we were made to take an ethics class for like 2 weeks over the summer of our REU. We were all excited about the prospect of learning proper citation methods and when to cite and what sorts of things need reporting and what can be ignored and whatnot.
It turned out to be a philosophical ethics class. Eg: we discussed the morality of abortion and things like that. Nothing to do w/ science and the work we'd be spending the rest of our lives on. Granted, we had some interesting discussions, but I think we were all disappointed that the class had no practical applications.
Bah.