Self-Check

This is a poll I have done before (4 years ago), and I am interested to see if the results will be different now. I am curious about how often you check your citation statistics -- number of citations, h-index, who is citing you etc. You can sign up to get alerts about this, or you can go and check yourself at one of the citation-counting sites. How often do you get information about your citations?

Whether or not you answered the poll 4 years ago (in April! is there something about April that makes me think about citations?), do you think your citation-checking habits have changed in the last few years? 

So:

How often do you check your citation statistics?
	As often as possible
	Quite often but not routinely
	Every once in a while when I think of it
	Once or twice a year, if that
	Never ever ever
pollcode.com free polls

Happiness Index

There are many ways to measure happiness in one's life and career. For example, I recall learning, as an undergraduate, that one's happiness could in part be calculated from the distance of one's home to the take-off and landing flight paths of major airports.

Anyway, for academic persons involved in research, happiness these days may derive in part from the number of times a publication is cited. In this case, the h in h-index does not refer to happiness, but for some people it might as well.

What I am wondering is, for any individual -- at whatever career stage, in whatever academic discipline, and with whatever personality traits you may have -- what is your personal minimum number of citations for you to feel happy, or at least moderately satisfied?

That is, when you look at your citation numbers for each publication, is there a particular number of citations that make say (to yourself, if not to anyone else), "OK, I am happy with that number of citations"? (leaving open the possibility that you would be happier if it were cited even more)

Is that number = 1? 5? 20? 37? 50? 100? 300? 1000? 7326? more?

What are the most important factors in deciding your personal minimum number of citations for happiness? I expect that the culture of each field plays a role. Is there are particular number that is considered pretty good in your field, and can you say what that approximate number is? For example, I have seen letters in support of tenure/promotion cases (not in my field) in which the letter-writer has asserted that the n < 10 citations of a paper is considered a very good number in that field. In my field, that would be considered not so good.

I expect that one's happiness with a publication's citation count experiences a bit of an uptick when that publication reaches a number that causes your actual h-index to go up, but is that your minimum number, or is your personal minimum number >> your h-index?

I don't want to give many details about myself, but my absolute minimum citation happiness number is indeed the one that makes my h-index go up (so this is a (slowly) moving target), but that milestone just causes a flicker of citation-happiness. For me, the true minimum citation happiness number is quite a bit higher, and therefore more elusive.

That's not to say that I am disappointed in or depressed about publications that don't exceed my citation-happiness threshold -- in fact, some of my all-time favorite papers are among my least cited ones. It's just that there is a certain bonus satisfaction that comes from having papers with (relatively) large numbers of citations, even for tenured professors whose careers don't hinge on these numbers.

The reason I have been thinking about citations recently (again) is because the other day, a (very) senior professor told me that he was upset about another colleague who doesn't cite his (the senior professor's work) when he should. He said that he wants his citation numbers to be as high as possible by the time he dies because "that's all we have" (as a legacy). I thought that was sad and disturbing, particularly coming from someone who has a large number of papers that have been cited more times than any paper of mine will ever be. I hope I don't feel that way when I am his (near retirement) age, even if it does make me happy when my papers are cited.

Citation Conspiracy

Someone recently told me about this, and I was wondering if anyone has participated in something similar:

A group of colleagues makes a specific effort to cite each other's papers -- those paper not involving the author/s doing the citing, so no self-citation is involved -- to help each other get their citation numbers up. They don't gratuitously cite a paper that is irrelevant to the topic at hand, but they proactively seek opportunities to cite each other's papers, and, given a range of options for citation of a particular point, they will choose to cite a paper by someone from this group.

If you have not participated in something like this, does it bother you that some people do this?

I have not participated in a citation-circle like this, and the fact that others do does not bother me. These people are not inappropriately citing their friends -- the citations are all relevant -- and it is likely that most of us do something like this anyway, even without making a concerted effort. We tend to cite papers with which we are familiar, no matter how diligent we try to be in surveying the vast literature in each of our sub/fields.

Does a citation-circle have any measurable positive effect on the career of a particpant? If it is effective, involving a sufficient number of productive (in terms of publications) researchers, it can make the difference in the citation numbers (h-index and so on). Increasingly, career advancement relies on having good citation numbers, so being in a citation-circle might be quite helpful, even if it doesn't result in a dramatic jump in citations.

Does a citation-circle harm those not in it? I suppose one instance in which a citation-circle, even one conducted in an ethical way, might have an unintended negative effect on someone not in the circle would be if one of the "proactively cited" papers becomes one of those papers that is commonly cited in introductions. And then, because it is cited prominently in some papers, it gets picked up as the go-to cite for introductions in papers on similar topics. The citation numbers can then snowball, and other papers might languish in undeserved obscurity.

I think that happens to some papers anyway, with or without a few citation-circles in action. Therefore, I was intrigued by the existence of such citation-circles, but not disturbed. But that's just me, perhaps reflecting my secure position as a mid-career professor who obsesses about citation data mostly out of curiosity rather than out of necessity. I am curious if others feel otherwise, but I need to note again that comment moderation will continue to be sporadic for a few more days (sorry).

Citing Creeps

Today in Scientopia: On creeps and their effect on the work of others

Monday Pop Quiz!

Sorry to spring this on you, but I wanted to make sure that you have all been keeping up with the material. This little quiz will help me see how you're doing, and to identify any concepts that are particularly problematic for the class, so I can be sure to focus on those in my lectures in the coming weeks.

Yes, I know this quiz is not on the syllabus. It is a so-called pop quiz. I did put on the syllabus that I would give you some of these throughout the term; I just didn't say when. That's the whole point of them. The fact that this is the first one was not intended to lull you into a false sense of comfort that there wouldn't be any, but if it had that effect, I must say I'm not too ripped up about it.

And no, I don't care if you do better on quizzes if you listen to music, you can't have your ear buds in during the quiz.

The Quiz

Let's say that you happen to read a journal article that was published > 1 year ago and you see that your own (quite old) publications are cited. But: you don't like how your published work is cited in this article -- the authors didn't mis-cite you in any egregiously wrong or unethical way, but you nevertheless don't like how they did it. For example, maybe you feel that they put too much emphasis on some things that we know now that we didn't know >20 years ago when you published your cited paper(s). Or maybe you don't like how they wrote that their results might be in conflict with some results in one part of your old work. What do you do? Do you:

(a) Make an unhappy huffing sound, shrug your shoulders, and forget about it. Maybe you will mention it to the authors if you see them at a conference, but otherwise, it's not a big deal.

(b) Write a brief but polite e-mail to the primary authors, explaining your discontent and then waiting to see how they respond.

(c) Write a formal comment and send it to the authors and ask if they'd be interested in writing a reply, perhaps for publication in the journal in question.

(d) Write a formal comment and send it to the editors of the journal and let them deal with contacting the authors to see about a possible reply, perhaps for publication in the journal in question.

(e) Fire off an angry e-mail to the editor of the journal, insulting the integrity of the editor and the journal as a whole for publishing a paper that contains this unjustified attack on you and your work. Be sure to include lots of dramatic adjectives that show -- unambiguously -- just how shocked you are that this paper was published in a journal you used to respect. 

Time's up. Put your pencils down and pass your quiz forms to front.

InCited

Yesterday I wrote about a citation-related topic, and you know how hard it is for me to stop talking about citations once I get started [FSP 2006, 2007, 2008, 2009, 2010, 2011].

I will write about citations again today because I was reminded of an incident from my postdoctoral days. I had finished a draft of the first paper related to my postdoctoral research and had given it to my supervisor to read. He was rather notorious for taking a long time to read manuscripts and then not having many comments. Another postdoc had warned me not to expect to see the manuscript again for weeks (if I was lucky) and then to be underwhelmed by the input.

So I was shocked when, later that same day, my office phone rang and my postdoc supervisor said "I just read the draft and I need to see you RIGHT AWAY. There is a SERIOUS OMISSION in the paper."

Another postdoc was visiting me in my office just then. He said "This can't be good." We shook hands, and he wished me luck in my new career, whatever it might be, and said it had been nice knowing me.

With a staggering amount of trepidation, I went down the hall to see my supervisor, whom everyone called "The Big Guy". The Big Guy had the print-out of the manuscript on his desk, and he was shuffling through the pages. When he saw me, he said

"HERE! LOOK HERE! On page 7, there is a problem. You should cite my 1984 paper."

He handed me the page in question, which had the place for the missing citation noted. I just stood there, waiting for what came next. But all he said was:

"That's all. Submit it after you add the citation."

So I lived to tell the tale. And I added the citation. And I asked my postdoc-friend: "Will we be like that someday?"

Our assumption was that since we were not like that (in our 20's), it must be something that happens to you later in your career.

So now the question is: Am I like that?

I am probably not the best one to answer that question, but I would say that I developed a strong interest in seeing my work cited (appropriately and accurately, of course), BUT I don't think that it has become a singular obsession that supersedes my interest in the Science in a paper. I'd like to keep it that way, but who knows.. I'm only at mid-career and there's plenty of time for me to become a raging citation-monger.

Does anyone think there is a generational aspect to citation-obsession? It is perhaps most important for early-career people to have good citation numbers, but does that mean they are actually the ones who tend to be more obsessed, or is it we more senior people who tend to be fascinated with our citation metrics? I fear that the real answer is "All of the above".

Cite-Me

Today in Scientopia, I discuss a question about how/whether to mention in a manuscript review that the authors should cite one (or more) of your papers.

Citation Surge

At some point, in one of my old blog-posts, I asked how often authors check their citation data: every week? every month? every once in a while when the mood strikes? never? I was in the 'every once in a while' category.. until recently.

In a flash of self-realization, I determined that the reason I didn't obsessively check my citations was not because I am above doing such things, have better things to do, and/or know that it is unhealthy to fixate on citations. No, it turns out that I was not obsessively checking my citation numbers because doing so would be even less exciting than watching grass grow. From week to week, there might be a few citations clicking up a notch or two, but this wasn't thrilling enough to inspire me to check back frequently.

But now, there is one particular paper that is surging in citations! It is very exciting! And so I check my citation numbers much more often than I used to.

It would be even more exciting if the paper in question represented a major brilliant cosmic advance in Science, but, alas, it's a utilitarian piece of work. It's just something that is useful. Apparently (and lucky for me), it seems to be very useful, and I have been enjoying watching the citations go up every week.

This is sad, I know. First, there was the fleeting thrill when the citations of this paper upped my h-index, and now there is the lame entertainment of seeing that the number of citations has changed by a number >> 1 compared to the last time I checked.

Will the thrill fade with time? Or am I now addicted to checking my numbers every week (or so)? Is there any cure for this particular obsession? Should I seek help?

Read Me?

Some comments to yesterday's post reminded me of something I have been wondering:

If you are the (or a) major author on a paper submitted for review, how many of the references you cite have you read?

My adviser in graduate school told me that I should read every article that I cite. Minor co-authors can use their discretion about which ones to read/not read, but if you're the main author, you should read all cited works.

I think we can assume that 'read' means that you read more than the abstract but didn't necessarily spend hours poring over every word in every section, although you may well have done so, especially for a thesis.

So: For articles (or the moral equivalent) on which you are what could reasonably be considered a major author according to the norms of your field:

How many of the articles you cite do you read?
	100%
	75% or more but not every one
	50-75%
	somewhat less than 50%
	less than 25% but not zero
	zero or close to zero
pollcode.com free polls

Can You Rephrase That?

As I was reviewing a manuscript recently, I saw this sentence:

.." and existing models to explain this phenomenon are inadequate [3].

[3] FSP 2001"

So, is my model inadequate or did my 2001 paper state that existing models were inadequate.. or both?

In fact, I couldn't even tell from the context of the rest of the paragraph or manuscript, which was clearly an undigested chunk of thesis. I am not even sure the citation is appropriate in this case, at least not without significant rewriting of the statement and surrounding text.

This issue was easy to deal with in a short review comment.

In two other recent manuscripts under review, I saw something like this:

"It has been proposed that dolphins strongly prefer scones to croissants [FSP 1996]."

and "It is well known that cats who fall out of a tree from a height of at least 8 meters are likely to fracture their back left leg [FSP et al., 2009]."

Thanks for the citations, but I never proposed or stated either of those things, although I know at least one of them to be true.

Those problems were easy to deal with as well. I wrote that the citations were inappropriate, as I had never discussed dolphin pastry preference or cat/tree issues, at least not in print.

I am sure that I have mis-cited references before, too, especially in long papers with lots of references and lots of co-authors. We hope that such things will be caught in review, but in some cases they are not. As a reviewer, it's easy to catch mis-citations of work we know well (i.e., our own) but we aren't always familiar with every citation in every paper.

In another recent example, an author mis-cited (in my opinion) another author (not me). I hesitated to make a comment in my review, though, because the mis-cited author was a co-author of the manuscript under review. Shouldn't he be the one to remove the mis-citation? I was quite confident that the citation was inappropriate, so I made a gentle remark about this in my review, and the citation was not removed.

I like being cited, of course, but I don't like being mis-cited. It would bother me a lot if someone thought I really had proposed that dolphins prefer scones if I had, in fact, never said that. This is not just my being ethical, although, like most of my professorial readers, I have received intensive and largely irrelevant training in the responsible conduct of research. My objections mostly stem from my dislike of being misquoted or misrepresented.

Not to obsess about citation indices (too much), but despite the possible of loss of a (mis)citation, it is possible that fixing these errors in review might ultimately lead to more citations, not fewer. For example, if someone cited my work as having said something really stupid about dolphin/scone preferences, this might discourage future readers (and citations).

Does anyone cynically believe that the increased emphasis on citation indices increases the number of mis-citations, either because authors are more eager to cite their own papers (even if not entirely appropriate) or because reviewers are less likely to correct mis-citations of their own work?

Google Scholar v. Web of Science

From the comments to yesterday's post:

ISI/Web of Science (WoS) is " better than Google Scholar by an order of magnitude."

"..my citations are definitely higher on Scopus than on WoS."

"Web of Science has a few errors in my records, though not nearly as bad as Google Scholar.."

"I prefer Google Scholar.. My prediction is that WOS will decline in popularity over time unless it makes drastic changes."

OK, so let's do the numbers.

I compared citation data in Google Scholar and Web of Science for 25 of my publications. (I did not search in Scopus).

I looked at a range of publications in terms of publication date, my place in the authorship order, and type of publication. For 18 of the 25 publications, Web of Science counted more citations, so I definitely like WoS better. For these 18, Google Scholar's citation count ranged between 0-92% of the citations in WoS; the average was 62%.

For 3 of the 25, Google Scholar counted the same number as WoS, and for 4 others Google Scholar counted more citations, although typically only slightly more than WoS (84-92%). There aren't enough data for me to conclude anything systematic based on these small numbers, but I was intrigued by the fact that 2 of the publications that had a higher citation count in GS than in WoS were in topics outside my primary research field.

The publications for which Google Scholar did a significantly worse job of finding citations than WoS --i.e., finding <40% of the citations listed in WoS -- were typically in my oldest publications and in my most recent publications, although there is one paper published in 2002 in a mainstream journal for which GS found <40% of the citations listed in WoS.

These results are not surprising; it is not news that these sites are not perfect at counting citations.

These databases are very useful for doing literature searches, and should be used primarily for this purpose rather than as key data in decision-making about jobs, promotions, and awards. Nevertheless, I have been on committees in which various members exclusively used one or the other of these sites for looking up the publication records of applicants/nominees, and I have seen citation numbers listed in many CVs and in letters of recommendation (typically without reference to which citation index was used to determine those numbers).

To some extent, this is OK. A very high number of citations is impressive, whether it is 250 or 320. For some of my papers with more modest numbers of citations, though, I might as well just make up a number between 5 and 50 than rely on the count in either Google Scholar or Web of Science.

Even so, for my field (or subfield) of the physical sciences, Web of Science is definitely "better" at counting citations for most publications. For those of you who prefer GS to WoS, perhaps you could leave a comment indicating your field. Are there particular fields for which GS is better at finding citations?

Seeking Perfection

Is your citation record in Web of Science (or the moral equivalent) perfectly correct? Or are there errors?

If there are errors, are they insignificant (not worth correcting) or significant?

If there are significant errors, have you done anything about it? (Or will you?) It is possible to request a data correction using a form provided on the Web of Science website.

There is one particular paper of mine that is particularly prone to being cited in various and sundry ways. In fact, the citations for this paper are strewn about in so many different apparent titles in my citation report that, were the errors to be fixed and the citations combined, my h-index would increase (gasp). There are other errors as well, mostly because authors citing my work used an incorrect volume, page, or year, but most of these errors do not affect my h-index.

Should I try to fix the errors, or, at least, the one that would affect my h-index? Would you, if you were (or are) in this same situation?

Avalanche of Useless Science

In a post last winter, I discussed whether papers that receive few or no citations are worthwhile anyway. I came up with a few reasons why they might be worthwhile, and noted that the correlation between number of citations and the "importance" of a paper may not be so great.

In an essay in The Chronicle of Higher Education this week, several researchers argue that "We Must Stop the Avalanche of Low-Quality Research". In this case, "research" means specifically "scientific research".

How do they assess what is low-quality ("redundant, inconsequential, and outright poor") research? They use the number of citations.

Uncited papers are a problem because "the increasing number of low-cited publications only adds to the bulk of words and numbers to be reviewed."

There you go! A great reason to turn down a review request from an editor:

Dear Editor,

I am sorry, but I am going to have to decline your request to review this manuscript, which I happen to know in advance will never be cited, ever.

Sincerely,

CitedSciProf

What if a paper is read, but just doesn't happen to be cited? Is that OK? No, it would seem that that is not OK:

"Even if read, many articles that are not cited by anyone would seem to contain little useful information."

Ah, it would seem so, but what if the research that went into that uncited paper involved a graduate student or postdoc who learned things (e.g., facts, concepts, techniques, writing skills) that were valuable to them in predictable or unexpected ways? Is it OK then or is that not considered possible because uncited papers must be useless, by definition? This is not discussed, perhaps because it is impossible to quantify.

The essay authors take a swipe at professors who pass along reviewing responsibilities: "We all know busy professors who ask Ph.D. students to do their reviewing for them."

Actually, I all don't know them. I am sure it happens, but is it necessarily a problem? I know some professors who involve students in reviewing as part of mentoring, but the professor in those cases was closely involved in the review; the student did not do the professor's "reviewing for them". In fact, I've invited students to participate in reviews, not to pass off my responsibility, but to show the student what is involved in doing a review and to get their insights on topics that may be close to their research. It is easy to indicate in comments to an editor that Doctoral Candidate X was involved in a review.

Even so, the authors of the essay blame these professors, and by extension the Ph.D. students who do the reviews, for some of the low-quality research that gets published. The graduate students are not expert reviewers and therefore "Questionable work finds its way more easily through the review process and enters into the domain of knowledge." In fact, in many cases the graduate students, although inexperienced at reviewing, will likely do a very thorough job at the review. I don't think grad student reviewers contribute to the avalanche of low-quality published research.

So I thought the first part of this article was a bit short-sighted and over-dramatic ("The impact strikes at the heart of academe"), but what about the practical suggestions the authors propose for improving the overall culture of academe? These "fixes" include:

1. "..limit the number of papers to the best three, four, or five that a job or promotion candidate can submit. That would encourage more comprehensive and focused publishing."

I like the kernel of the idea -- that candidates who have published 3-5 excellent papers should not be at a disadvantage relative to those who have published buckets of less significant papers -- but I'm not exactly sure how that would work in real life. What do they mean by "submit"? The CV lists all of a candidate's publications, and the hiring or promotion committees with which I am familiar pick a few of these to read in depth. The application may or may not contain some or all of the candidate's reprints, but it's easy enough to get access to whatever papers we want to read.

I agree that the push to publish a lot is a very real and stressful phenomenon and appreciate the need to discuss solutions to this. Even so, in the searches with which I have been involved, candidates with a few great papers had a distinct advantage over those with many papers that were deemed to be least-publishable units (LPU).

I think the problem of publication quantity vs. quality might be more severe for tenure and promotion than for hiring, but even here I have seen that candidates with fewer total papers but more excellent ones are not at a disadvantage relative to those with 47 LPU.

2. "..make more use of citation and journal "impact factors," from Thomson ISI. The scores measure the citation visibility of established journals and of researchers who publish in them. By that index, Nature and Science score about 30. Most major disciplinary journals, though, score 1 to 2, the vast majority score below 1, and some are hardly visible at all. If we add those scores to a researcher's publication record, the publications on a CV might look considerably different than a mere list does."

Oh no.. not that again. The only Science worth doing will be published in Science? That places a lot of faith in the editors and reviewers of these journals and constrains the type of research that is published.

I have absolutely no problem publishing in a disciplinary journal with impact factor of 2-4. These are excellent journals, read by all active researchers in my field. It is bizarre to compare them unfavorably with Nature and Science, as if papers in a journal with an impact factor of 3 are hardly worth reading, much less writing.

3. ".. change the length of papers published in print: Limit manuscripts to five to six journal-length pages, as Nature and Science do, and put a longer version up on a journal's Web site."

I'm fine with that. It wouldn't have any major practical effect on people like me who do all journal reading online anyway, but for those individuals and institutions who still pay for print journals, this could help with costs, library resources etc.


Let's assume that these "fixes" really do "fix" some of the problems in academe -- e.g., the pressure to publish early and often -- so what then?

"..our suggested changes would allow academe to revert to its proper focus on quality research and rededicate itself to the sober pursuit of knowledge."

Maybe that's my problem: I enjoy my research too much and forgot what an entirely sober pursuit it should be. I guess the essay authors and I are just not on the same page.

On Fecundity

A letter in the 3 June 2010 issue of Nature addresses "The role of mentorship in protégé performance". That sounds sort of interesting. I'd like to know ".. the extent to which protégés mimic their mentors' career choices and acquire their mentorship skills".

Those are two very different things, though. I can see how you could determine whether protégés follow the same career path as their advisers, but you need some assumptions to go from those data to interpretations about acquisition of mentorship skills (or lack thereof).

To address these issues, the authors (Malmgren et al.) used a large database that has tracked mathematicians and their academic "genealogy" for centuries. Despite the massive database going back to 1637, the authors analyzed only the years 1900-1960 because these data were deemed "most reliable" and this range allows the tracking of a few generations.

More recent data would be interesting to consider as well, if possible, particularly to see if the culture of academic math departments has changed. Or perhaps nothing changes the culture of math departments; the authors concluded that, despite the occurrence of some world wars etc. in the 20th century, there were "no systematic historical changes" evident in the database.

The research was designed to evaluate whether protégés "acquire the mentorship skills of their mentors". This is done by studying mentorship fecundity. I am not sure that fecundity necessarily relates to "mentorship skills" or "mentorship success", but that's how it was defined.

A brief aside: This may be a seminal paper, but I wish there were a different term that could be used than fecundity to represent the number of protégés a mentor trains. I also wish there were a better term than "protégé", although I know that technically the word is used appropriately in this paper. Advisee and mentee aren't great words either, but somehow they seem more professional to me. Or, if protégé must be used, can I be called a patron rather than a mentor?

Anyway, what we all want to know is:

Can you predict the fecundity of a mathematician?

Well, it's complicated, but you can write an equation! Also, you can make an analogy with parents (= mentors) and children (= protégés), such that a protégé's graduation date is their "birth date". I'm not sure why this new terminology was introduced, as the original concepts of mentor, protégé, and graduation date are not that complicated, but so it goes.

The results, which aren't actually explained in the paper, are "three significant correlations in mentorship fecundity", which I will condense into two:

1. Protégés of mentors with low fecundity (< 3 protégés) had more protégés than "expected".

2. The protégés of early-career mentors are themselves more fecund than "expected" and are more fecund than protégés who are advised by these same mentors later in their careers; i.e., fecundity might be influenced by the adviser's career stage.

The first interpretation did not surprise me, although one has to be clear about what the "expected" fecundity of protégés is before deciding if the result is greater or less than expected. The second one is more surprising, but whether it has any meaning depends, of course, on the methods and assumptions of the study.

It's a rather strange study and we can pick away at it, but is there anything to be learned from it about the influence of mentors on the later careers of their mentees? I doubt it, although I think the motivating question of the study is an interesting one, and perhaps impossible to study in a meaningful and quantitative way. To be relevant to modern mentorship, such a study would also have to track career paths that veer from math to engineering, or to any of various other interconnected disciplines.

And, because being a successful mentor doesn't have to mean that we clone ourselves and produce academic children who mimic our careers, perhaps there would need to be a different way of measuring the quality and success of mentorship than simply counting up the numbers of academic children and grandchildren and great-grandchildren.

Playing the Game (2)

Citation index considerations aside, have you made any publication decisions specifically because you wanted to increase your number of publications, on the cynical-but-perhaps-justified assumption that more papers = better?

For example:

1. Have you taken what probably should have been one paper and split it up into two or more shorter (but nevertheless distinct) papers for the sole purpose of increasing the total number of papers on your CV?

Figuring out how many papers should be written about the results of a particular research effort is a bit of an art, and in some cases more papers is the best decision. Sometimes you know in advance whether to write results up as one or more paper(s), and sometimes you don't know until you start writing.

My question, therefore, refers to cases in which the main motivation for splitting up a project into n>1 papers is to crank up the publication numbers.

2. Have you published what probably should have been one paper but that instead were two or more related papers that were not as distinctly separated in content as in the first scenario? (i.e., shingling)

This is also a fuzzy concept because sometimes you want to publish one short and zippy paper and then another, longer one in a more specialized journal. Is that shingling? Or what if you decide to publish some different-but-related papers because each involves different groups of co-authors and it makes logistical sense to keep the publications separate? Or maybe you want to publish a review paper that summarizes information in some of your other papers. There are many legitimate ways in which similar papers are published by the same author.

But then there are other cases. I recall one time when I brought a manuscript to review on an airplane. This was back in the Days of Paper, so I had a hard copy of the manuscript, and had turned to a page with a figure on it. I was traveling with some colleagues to a conference, and a colleague in a different-but-related field was sitting next to me. He was also reviewing a manuscript, and turned the document to a page with a figure on it. It was the exact same figure. Yes, I know that manuscripts in review are confidential, but there we were, sitting next to each other with two different manuscripts submitted to two different journals at the same time, and there was at least one thing in both manuscripts that was identical. So we started comparing. The two manuscripts had the same authors, the same figures using the same data for the same topic, and only a slightly different "spin" put on different aspects of the data. Those two manuscripts clearly represented shingling of a sort that was probably not OK. We informed the editors of the journals.

3. Have you ever submitted a paper before the project had advanced as far as it probably should have before writing up part of it as a manuscript; i.e., a premature manuscript that was probably publishable but that you knew would be much better if you worked on the research more? (but you didn't feel you could afford to wait longer because you needed publications on your CV sooner rather than later)

As with the other cases, this is not an obviously bad thing to do either. Maybe you would have waited longer to publish if you already had tenure, but there are also good aspects of publishing a preliminary paper to communicate initial results rather than waiting, perhaps years, to publish one big definitive paper. You'd want to be as confident as possible that the preliminary paper was sound, but if you feel you have something to say that is of interest, I think it can be very useful to publish early and often.

I know that that is "playing the game" and perhaps contributing to the mass proliferation of academic articles so that the flow of information is overwhelming and the very act of publishing a scholarly work is devalued etc. etc., but I think that if you have something interesting to say, it's a good thing if you write it up and get it out there.

Ideally, in the course of your career, you will publish some short papers and some longer, more detailed papers and some review papers and some papers in Awesome Journal X and some other papers in Specialized Journals Y and Z, and it will all even out. It's in the early stage of an academic career when every decision about what/where/how much to publish can seem so critical.

From what I've seen in the physical sciences, the "best" route to take at all stages of an academic career is to try for a balance between publishing a reasonable number (according to the norms of your specific field) of very good but perhaps not awesome papers in respected journals, and then some (but likely fewer) rather awesome paper(s). This is preferable to having lots of narrowly focused papers or only/mostly having "big idea" papers. Together, however, these different types of publications demonstrate the breadth and depth of your research.

When deciding how to divide up a big project into papers, I try to optimize publication quality, speed, and impact, as well as consider what is most fair to the most number of co-authors, particularly students, postdocs, and/or tenure-track colleagues. This not always a straightforward decision, of course, and sometimes I wonder if I would do just as well by consulting a Magic 8-Ball for advice.

Or maybe I should go back for yet more training in Responsible Research Conduct. Surely somewhere in all those case studies and PowerPoint presentations, there is a nifty formula for sorting all of this out easily. {note use of delusional font}

Playing the Game (1)

This spring, I have to refresh my ethics training, a ritual that is annoying mostly because the university makes it so extremely difficult for faculty and postdocs not involved in the biomedical sciences to find relevant ethics training opportunities. I am really not that interested in spending 2 days learning about research with human subjects, as my research involves no human subjects (other than grad students..).

And now all our graduate and undergraduate students who receive a salary or stipend from NSF must be trained in ethics.

Ethics training is important, but it is boring: don't plagiarize, don't fabricate data, don't falsify data. And the case studies are bizarre.

So let's forget about whether the postdoc should hide the data outliers that may or may not be due to a power fluctuation during extremely expensive analyses at a national lab even though the grad student involved in the research thinks that they should at least inform their PI, and consider some more interesting situations. For example, following on yesterday's citation theme:

Has your awareness of the importance of citation data affected your decisions about publishing?

That is, would you have made a different decision about anything related to publications (authorship, type/number/format of publications etc.) if your citation metrics were not being constantly tabulated for all the world to see?

When answering the first-order question, consider only whether you have ever considered the impact of a publication decision on your citation indices. Such considerations are not a priori unethical; it is possible that such considerations will result in your making a completely ethical and smart decision that helps your career. An example of that might be to send a manuscript to a journal that is indexed instead of to an edited volume being published as a not-indexed book. You do not change anything about the authorship or content of the paper; just the publication venue.

But now consider whether you have ever made a citation-fueled publication decision that might not have been entirely ethical but that could possibly be justified because "That's how the game is played." (others are doing it; if you don't play the game, you lose; etc.).

I don't think I've made any unethical decisions about research publications, but I definitely had citations in mind a couple of years ago when I decided to revise a much-cited technical document that was originally authored by someone else (now long retired) and that was in great need of an update. In that case, though, I wasn't thinking so much of my own citation index, but was instead intrigued by the idea of transferring citations from Journal 1 to Journal 2, in part because I am involved in the editing of Journal 2 and therefore have an interest in promoting the success of Journal 2. I hasten to add that I am paid nothing for my editorial work for Journal 2, so financial considerations were not involved. There were in fact valid scientific reasons for publishing the update in Journal 2, in addition to my personal affection for Journal 2. In the end, though, I published the update in Journal 1, but I admit that my original intentions were rather craven and driven in part by thoughts of citations.

That's the worst example I can think of for myself, but perhaps that is because I am a tenured mid-career professor. If I were on the tenure-track today, I am sure it would be difficult not to consider citation index issues when making publication decisions, although if it makes any of my early-career readers feel any better, I happen to know that at least some promotion & tenure committees at research universities are specifically instructed not to consider the h-index or other citation statistics when reviewing files for tenure and promotion.

to be continued..

Just Checking

If you have published at least one paper in a journal that is included in a citation index, how often do you check your own citation statistics? Do you know what your h-index is? (I mean, do you know the number, not what the h-index is.) Do you subscribe to a citation alert service so you are notified when someone cites your paper? Do you check up on the citation data of certain colleagues, whether or not your intentions are good?

I am not so interested in opinions about whether you think citation indices are a nifty metric or the worst thing to happen to academe since faculty meetings were invented, but go ahead and rant in the comments if you must.

Mostly I am wondering how often we are checking on our own citation data, no matter what we think of the significance (or lack thereof) of these data.

How often do you check your citation statistics?
	As often as possible
	Quite regularly, but not obsessively
	Every once in a while, when I think of it
	Maybe once a year, if that
	Never
pollcode.com free polls

Do you know what your h-index is?
	Yes, I know exactly what it is
	Sort of
	No, I have no idea
pollcode.com free polls

Waste of Time?

A comment on yesterday's post got me thinking about something: If a non-recent journal article gets a low number of citations (say, 0-2), was the research that went into that paper a waste of time (and money)?

My gut reaction is to say no, of course not. Surely something was learned during the research that led to even the most forgettable or forgotten of papers? And surely the researcher didn't know in advance that the paper would never be cited, and did the research for a good reason?

And perhaps citations are not the most perfect judge of what is or is not worthwhile. It's not difficult to think of examples of highly cited papers that aren't that great, and barely cited papers that are overlooked (especially our own).

At the same time, a paper with zero citations, even after more than 10 years, might mean something..

.. such as:

- no one else in the world is or will be interested in this topic;

- others are interested, but they never publish;

- others are interested, but they only cite other papers, not yours (for various possible reasons), creating a snowball effect of subsequent citation of papers other than yours on this topic. With time, it becomes ever less likely that your paper will be cited.

Publishing something widely believed to be wrong or stupid isn't necessarily a barrier to citations, nor is publishing something obvious, so I am not including these in my list of possibilities.

Since I am in a quantitative mood this week, I tried to decide whether there is a minimum number of citations, above which we can say that the research was worthwhile, and below which we might have good reasons to doubt this.

My musings on this topic made me dive into my citation index to look at some of my low-citation papers to see if I could reasonably defend them as worthwhile in some way. My favorite example of a deservedly ignored paper in my oeuvre has surprised me by being cited in the low double-digits. Does that mean that the paper is more worthwhile than it was a few years ago because it has now received (slightly) more than 10 citations (none by me!) instead of 2 (or zero)? No, I don't think so. The difference between 14 citations and 2 citations really isn't that significant in terms of gauging the worth of a paper. And yet, although I am well aware that it was a fairly insignificant paper, I am reluctant to say it was a waste of time.

Further rummaging in my citation history shows me that some of my most highly cited papers do not represent what I consider to be my most significant work but that happened to be on topics that are of more widespread interest than the core of my research. Does that make these more-cited papers more "important" than my others? I am not objective about this, but I don't believe that citations correlate with significance, though I admit that it depends on how you define "important" and "significance".

One more personal example: A paper that has received a very modest number of citations is frequently mentioned to me as a paper that is read and discussed in graduate seminars. I am very pleased about that. The paper is being read and used (perhaps as an example of how not to write a paper..), although it is not cited very often. I consider that paper to have been worthwhile.

So, although I agree that zero citations is not a good thing for non-recent papers, and my papers have thus far avoided this fate (though in some cases not for any good reason), I have trouble casting aspersions on papers that have received a modest but non-zero number of citations.

Key Keywords

There's an interesting article in Slate about the possible differences in outcome of scientific research based on the mode of funding: longer-term, more flexible grants vs. shorter-term, project-based grants. The article describes the results of a study that compared Howard Hughes Medical Institute (HHMI) funded investigators with NIH funded investigators who could be considered peers based on publication records and receipt of prestigious awards.

According to the study, the HHMI-funded researchers generated more highly cited work, as well as a lot of publications that had never been cited, highlighting the fact that even overall successful research results in some (many) dead ends. A possible explanation for the success of the HHMI researchers is that they had more freedom to "innovate" (the topic that is the main focus of the article).

That's interesting, but I was particularly intrigued by one of the measures used to evaluate innovation: the use of keywords in a publication:

They [HHMI researchers] are .. more likely to produce research that introduces new words and phrases into their fields of research, as measured by the list of "keywords" they attach to their studies to describe their work.

There's also some tentative evidence that HHMI scholars experiment more than their NIH-funded counterparts: ... their keywords change more often across studies, ... suggesting broader experimentation.

I can see that the variety of keywords for an individual researcher might indicate breadth or interdisciplinarity, but are keywords a good measure of innovation?

I am sure that innovation is difficult to quantify. Even citation indices are not necessarily a good measure. Some very highly cited publications are not innovative but represent a necessary technical advance upon which more innovative research can be based. And there are likely many examples of very innovative work that is not necessarily highly cited (especially if it becomes "conventional wisdom" rather quickly).

If keywords measure something important about a publication, however, perhaps I need to pay more attention to my keyword selection, possibly even making up a few new words now and then. When choosing keywords for my papers, I typically pick a few descriptive terms, but I don't give them a lot of thought. For me, keywords are an afterthought, selected quickly so I can continue with the manuscript submission process. Most of the important words are likely to be in the title and abstract anyway. Clearly I have not been thinking out of the box re. keyword selection.

I suppose if you invent something (even if it's just a new term, like thermofelinics) and it turns out to be important, you might want your paper to appear in searches as the earliest one on record that describes this new thing/process/idea. And for that to happen, perhaps you need to choose the right keywords. Perhaps you need to choose a combination of prosaic keywords and hot new keywords; let's call them: lowkeywords and keykeywords (K²W?), respectively.

So: Do keywords indicate something fundamental about us as researchers? And if so, should I take more interest in keywords and their careful selection?


Keywords: keywords, lokeywords, keykeywords, thermofelinics

Help Me Not Do This

A paper published in 2009 by some people I know contains the statement that it is problematic that a certain dataset does not exist because it would be really important to have such data but alas, such data have not been obtained, so instead they must use an ancient approximation based on a highly flawed technique.

I published just such a dataset in 2003 in a major journal, as the authors of the paper well know. One of the primary authors was a colleague of mine, although we stopped collaborating a while ago, by mutual agreement.

A few years ago, this former colleague asked me to remove his name from my research webpages because he was annoyed that my pages turned up before his in a Google search on his name. I did not think this was a reasonable request, though eventually the problem solved itself when I updated my pages to reflect new research and publications. Even so, I suppose this incident was sort of a clue that he might have some Issues.

Here is my internal debate with myself about the obvious non-citation incident:

Let it go. Their paper is lame and they undermine themselves by appearing ignorant of the literature.

Send them a passive-aggressive email with the relevant reprint attached, expressing regret at not sending the paper to them sooner, i.e. 6 years ago, and expressing surprise that they don't seem to have access to Major Journal, even though they do.

Let it go. Ignore them. Don't even admit that you read their paper.

And so on. Right now I am actually in a "Let it go. Ignore them." state of mind, but every once in a while I veer back to consideration of the insincere email/reprint option. I know I would gain nothing from contacting them and I would not feel good about it. So I won't do that, I think.

Why does it bother me that they did not cite my relevant paper? I am not upset that I lost a possible citation (really). I think I would be less bothered if they had simply left my paper out of a list of possible citations, but the overtness of the lack of citation was a bit shocking. That's what is so strange. And I suppose that is exactly why I should ignore them.