Look Beyond Citation Counts to Kickstart Scientific Innovation | Eastern North Carolina Now

Publisher's note: The James G. Martin Center for Academic Renewal is a nonprofit institute dedicated to improving higher education in North Carolina and the nation. Located in Raleigh, North Carolina, it has been an independent 501(c)(3) organization since 2003. It was known as the John W. Pope Center for Higher Education Policy until early January 2017.

The author of this post is Anthony Hennen.

    In scientific research, new ideas have become harder to find. Innovation has fallen compared to 50 years ago. Rather than a fear of "too much change," many researchers worry about stagnation.

    One argument suggests that the low-hanging fruit of scientific research has already been picked. Older scientists made the major breakthroughs, and younger scientists now focus on the minor work that comes from major revelations.

    Another argument, however, suggests that the problem isn't a lack of fruit. Instead, scientific research that focuses on old ideas is rewarded, not research on new ones. Jay Bhattacharya of Stanford University and Mikko Packalen of the University of Waterloo in Canada make that argument in a recent working paper published by the National Bureau of Economic Research.

    "Over the last five decades, citations have become the dominant way to evaluate scientific contributions and scientists," Bhattacharya and Packalen wrote. "This emphasis on citations in the measurement of scientific productivity shifted scientist rewards and behavior on the margin toward incremental science and away from exploratory projects that are more likely to fail, but which are the fuel for future breakthroughs...As attention given to new ideas decreased, science stagnated."

    "Even baseball players are evaluated on many more dimensions than scientists," they wrote. To re-align incentives and encourage scientific breakthroughs, Bhattacharya and Packalen suggest creating a model that rewards scientists based on multiple dimensions, such as measuring novelty and work based on new ideas.


    The Martin Center spoke with Mikko Packalen via email to discuss scientific incentives and how to reverse stagnation.

    A main theme of your paper is that scientific research faces an incentive problem. Are there other incentive problems in science beyond publishing?

    First, I want to say that right now scientists around the world are doing important work attempting to respond to the Sars-Cov-2 pandemic. I very much hope that they are successful in their efforts. Our views on the slowdown of scientific progress in recent decades and on how incentives in science are distorted should in no way be construed as discouraging these ongoing efforts. The unusually collaborative nature of how the scientific community has responded to the pandemic does hint that broader institutional reforms in science are possible and worth exploring.

    As for your question, beyond publishing, in the scientific community I worry about the incentives created by new communication tools such as Twitter.

    On social media, your value is mainly measured by your follower count. Academics thus today have an increased incentive to say popular things and enter areas of investigation where there are already many others. These platforms also increase the incentive to pursue research topics that are currently the most newsworthy, as news coverage exposes you to more potential followers.

    But popularity and timeliness are anathema to most scientific progress. The most successful scholars uncover secrets that are at first very unexpected and unpopular — almost nobody believes them. The scientists are only proven to be right through relentless revision, testing, and validation of these ideas. This work can take decades.

    I thus worry that social media tempts especially young scientists to pursue topics that lead to quick popularity at the expense of engaging in exploration that might lead to uncovering more important secrets.

    Do some areas of scientific research have a bigger problem than others? For instance, I imagine that some areas with the potential for profit, such as pharmaceuticals or the tech sector, may reward risk and the pursuit of new ideas.

    I think it is true that the rate of progress has not been uniform across scientific areas, and I wouldn't expect it to be. There will always be random fluctuations in progress in almost any activity. Sometimes the efforts are successful, other times not. It is because the slowdown has persisted for decades now that gives the most reason to suspect that scientist incentives have become distorted.

    Computer science has indeed had some very visible successes in recent decades, as has biomedicine. But in terms of broader societal benefits, even these advances have thus far fallen well short of the expectations that we had based on comparable past breakthroughs. The relatively strong for-profit motive thus has not inoculated these scientific areas from concerns about stagnation.

    In general, the for-profit motive works quite poorly in terms of encouraging risky exploration that may lead to breakthroughs in the very distant future. Companies have a strong incentive to fund work done during the last mile of the marathon. They have a relatively small incentive to fund runners on early parts of the marathon when we know that only a few of them will finish. Thus, to the extent that companies now increasingly set the agenda in academic computer science research, it is not clear this is entirely a good thing in the long run.

    In biomedicine the situation is similar. We have some important advances such as CRISPR, but pharmaceutical companies have little incentive to fund the kind of exploration of the unknown that made CRISPR possible decades later.

    With academia struggling to reward risk, is there a connection with the average professor getting older? With professors getting older, should we expect the institutions they control to become more risk-averse?

    I think so. In our past work, we calculated how career stage influences the willingness to try out new ideas in research. Perhaps unsurprisingly, early-stage scientists were found to be more likely to pursue novel work than their seasoned colleagues. I would certainly expect that the relative hostility to risky work on the part of aging scientists will be increasingly reflected also in how scientific institutions operate.

    This is a difficult issue to overcome but the rise of China to the scientific frontier gives me some hope. In some areas, China is now ahead of the U.S. in terms of applying new ideas. The competition with China will force the U.S. scientific institutions to implement some reforms. Thus, while the U.S. already greatly benefits from China's scientific investments through new knowledge, the competition aspect too can have great benefits.

    What about the idea of awarding grant funding by lottery? Or would more grants and prizes created for young academics be another possibility to reduce risk aversion?


    Giving grants directly to young scholars would help produce more novel work. While many funding agencies have taken steps in this direction, we have yet to see a large-scale shift in science funding. For a long time, the Nobel-winning economist Paul Romer has raised concerns about the graying of the scientific workforce and how this will impact progress. As a solution, he has advocated giving grants directly to young students and scholars instead of older professors. Students would then have more freedom to choose what to study and with whom. Unfortunately, thus far the impact of this proposal has been limited.

    I am skeptical, however, that more generous funding to young people would by itself solve the incentive problem that we identify in the paper. As long as career incentives are dominated by citation counts, also young scientists have too strong an incentive to work on well-established research areas in favor of risky exploration of new areas of investigation. Thus, I prefer that we give more funding to young scientists directly and — at the same time — start measuring and rewarding scientific novelty.

    As for giving money by lotteries, I think it is an interesting experiment and worth exploring further. However, I think that the career incentives of scientists — as reflected by their salaries, promotions, and prestige — will still be strongly driven by citations counts, even if all funding were allocated by lotteries. Thus, in my view, for lotteries to generate more true exploration of the unknown, we first need to measure and reward also the scientific novelty of contributions.

    Furthermore, I suspect that, ultimately, we will get a better return on limited research dollars if we continue to allocate them based on merit. But, of course, we should broaden how we measure merit.

    Are government entities more likely to be risk-averse than private foundations? Are any funders taking action to reward risky research?

    From biomedicine there is some evidence that private foundations, specifically the Howard Hughes Medical Institute, have succeeded in supporting riskier research than public funding agencies. But it is not clear that their approach, which is based on long-term funding guarantees, would work on a larger scale. On the government side, DARPA has long been hailed as exemplary in terms of funding innovative work.


    The problem with public funding agencies is that they inevitably face pressure from taxpayers to produce visible results fast. In theory, we would expect private foundations to be less responsive to this kind of pressure. But I suspect that their funding sources — particularly the ones with living benefactors — would also rather see successes sooner rather than later. It is thus not clear that private foundations have necessarily better incentives than public foundations in terms of funding risky exploratory work that might lead to breakthroughs decades later.

    Furthermore, even public funders such as the NIH have tried for years to find ways to encourage more innovative science. But these efforts inevitably run into two key obstacles. First, the public perception remains that its reviewers are very conservative in their funding decisions. Second, while a funding agency can influence the direction of research somewhat, research choices are still mostly determined by general career incentives.

    Accordingly, as long as scientist prestige and salaries largely depend on citations and the ability to publish in high-impact journals, the incentive to pursue risky exploratory research paths will remain small no matter how much funding agencies emphasize innovativeness and risk in their funding announcements.

    Looking toward the future, how can professors train students to pursue risky ideas instead of following academic trends? How can they attract students who are risk-takers?

    I certainly think that professors play a key role in shaping how their students approach scientific work.

    If we express openness to new ideas and appreciate above all else those scientists who persevered in the face of early skepticism about their work and later succeeded, our students are more likely to engage in research paths that may uncover important secrets.

    Similarly, if we express our appreciation for exploration that results in work that is not yet popular in any way (and may never be, or may only look like a failure for a long time), our students will be more likely to pursue high-risk high-reward research directions.