The research : teaching : service ratio by which many of us are evaluated in our annual reviews has always puzzled me in terms of how it actually works. It puzzles me even though I've been on the committee that does this evaluation for my department: the committee members independently come up with evaluations without discussion with each other and then the department chair makes the ultimate decisions based in part on these recommendations. We never know exactly what he decides (or why). We each get a letter with our salary data, but no other information or evaluation information. Other departments/universities do things more strictly by a point system, though from what I've heard, that approach can be gratuitously stressful and too rigid.
Part of the mystery has to do with the fact that the research : teaching : service numbers (45 : 45 : 10, for example) don't have any meaning in terms of describing how we really spend our time (e.g., in reality, research time >> teaching time).
There are other ambiguous aspects of how faculty are evaluated: Let's say that the expected teaching load in my department is 2.5 classes per year, and that I teach 2.5 classes in a year. Does that mean that I have done well or that I am only average for teaching that year? If I am only average, what are the implications for my annual review (which I never see)? Teaching evaluations are also important in the evaluation, of course, but is it better to teach more and get average teaching evaluations or to teach less and get better evaluations? And what about grad teaching/advising and serving on grad committees? Is that teaching, research, or service? It has elements of all three; some of my colleagues think it is mostly research, some think it is mostly teaching, and some think it is mostly service (it would be interesting to plot these opinions vs. the number of grad advisees each professor has).
In my department, it's also not clear what the research expectations are except that we should do as much as possible. That's what I do anyway, so the ambiguities of the research expectations don't bother me. It's not as if I'm going to do more or less research depending on some arbitrary standard.
A friend of mine who is a department chair at a big research university told me that in his department, the Dean sets the 'bar' for research expectations and it is set at a level corresponding to the highest-performing faculty members -- the ones bringing in millions of $$ in funding and running big labs that produce a lot of papers with the PI as co-author. If the bar is set there, most faculty get lower evaluations, even if they are doing really well by almost any normal standard. My friend says this system is really bad for morale, especially for assistant professors. Hearing that made me feel grateful for my department's more mysterious but more holistic system.
The expectations for the 'service' part of the job can also be ambiguous in terms of what we have to do to meet or exceed the expectations for that component. Part of the mystery has to do with the fact that service involves such a wide range of possible activities: department/college/university service (committees), professional service (reviewing papers and proposals, serving on panels, being an editor, holding office in professional organizations), and outreach (visiting schools, judging science fairs etc.). Are all of these equal in value? What about giving invited talks at other universities? Is that service, research, both?
I think these issues are particularly important for assistant professors who need to know what the tenure criteria are, and for senior professors facing a negative post-tenure review. For example, should early career faculty agree to serve on administrative committees and teach a graduate seminar in addition to their regular teaching load and is it OK to decline to do some reviews? For early career faculty, service should of course be minimal, but professional service activities that give you visibility in your field can be important.
Everyone has to find their own balance in terms of what they can manage, but the two priorities are to be a productive researcher and to be a good teacher.
2 years ago