Metrics Subject to Manipulation

Previously, I have cited the famous HBO series The Wire focusing in particular on the careerism vs. good science dichotomy. A closely related element that I failed to mention last time was the use of a single number or metric to measure productivity or effectiveness of organizations or individuals. This problem is addressed in The Wire is many different contexts. In our field, the manifestation is in the form of the h-index, which is used for faculty hires, for department and university rankings and for assessing research grants. (A researcher has h-index h if he or she has published papers with at least h citations.)

It is well-known that measurements of this kind can lead to a corruption of sorts because people are susceptible to trying to maximize their indices. There are even ridiculous websites claiming they can help you increase your h-index. In The Wire, this is called “juking the stats”. Statistics can be “juked” is various ways. Researchers can request others to cite their work (and cite their own work heavily), undeserving co-authors may be included for minimal work, and pressure to publish can lead to sensationalized work that is “half-baked”, etc.

The detrimental side of these indices and metrics is captured well in a couple clips from The Wire linked below. The first is in the context of the police department in attempt to reduce crime rates and the second is in reference to test scores at the grade-school level.

It seems to me that there is some awareness and push-back in the physics community with respect to these metrics, which I find refreshing. I also think most of us recognize that blanket numbers like these cannot measure the subtleties associated with one’s true scientific output. Nonetheless, as long as the “higher-ups” continue to use them, the longer they will have a strong grappling-hook on some.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s