Limitations of using Altmetrics in impact analysis

The number of published papers using Altmetrics ‘attention scores’ as a data source to measure impact is rising. According to Google Scholar, there are over 28,000 papers mentioning Altmetrics and impact.

This latest analysis published in PeerJ finds a positive correlation between citation rates and the Altmetric score for papers published in ecology & conservation journals over a 10 year period (2005-2015). This implies: the more a paper gets tweeted, blogged, or talked about in online popular media, the more it will be cited.

This seems commonsense. The more exposure a paper gets online, compared to traditional exposure via journal alerts to the limited number of subscribers, the more people will be aware of it and potentially cite it. This is why we do scicomm. (Although, hopefully people read a paper first and decide on its quality and relevance before citing.)

But the increasing excitement over Altmetrics as a data source is a little concerning. Sure, it’s easily accessible open data that can be used to answer multiple questions. But no index is perfect. Each one is limited by the way it’s measured, how it’s used and the quality of data it’s based on. This applies to all ‘open’ data sources.

The limitations of Altmetrics only came to my attention recently. It’s a cool index and it gives a little insight into a paper’s social media reach. But…

Last year I co-authored this paper on the value of blogs for ecology academics – incidentally, our key argument is about how influential blogs can be for the science community, which is also one of the key findings in the new PeerJ paper.

Now, I write a blog post for nearly every paper I publish, so of course I did so for this one, as did most of my co-authors. A few weeks later, I decided to have a look at the Altmetrics index of our paper to see how it was tracking. I was surprised to see that my own blog post wasn’t listed in the ‘Blogs’ section (or any other section), even though the blog posts of most of my co-authors were.

Intriguing. My own blog post on my own paper on blogging wasn’t registering at Altmetrics. That was kind of weird.

I had a look through their FAQs, without much insight. I had linked correctly, according to their instructions, and my blog had been an established website for nearly 10 years, and fit all their criteria for indexing…so I wasn’t sure why my post hadn’t shown up. I went back and had a look at the Altmetrics of some of my older papers – my first published paper was in 2010 and I started blogging in 2009. None of the Altmetrics scores of my papers included the blog posts I had written about my own papers. For example, this paper from 2013 includes an article I wrote for Grist about the research, but not the separate post I published on my own blog.

So I wrote to Altmetrics to query why my blog posts were missing from my own papers’ scores. I received a reply with a link to a Google form to submit ‘missed mentions’. The form only allows for one specific mention per submission. So I submitted a few separate forms, not just for the blogging paper, but also for a few older ones.

A few weeks later, I checked again. Lo and behold, my blog post on my blogging paper was now listed. Not in the Blogs tab, where you would expect to find it, but under the ‘Misc.’ tab…along with my other blog posts for completely irrelevant papers that I had also filed ‘missing’ reports for.

I wrote back again to try and correct this. The reply from Altmetrics said: “Our blog tracking is not retroactive. We’ll only be collecting blog posts and putting them on the blogs tab if they are published after we have indexed it.” Fine. But they didn’t explain why my irrelevant blogs had been linked to this particular paper’s Altmetric score.

So now all three of the blog posts I submitted as ‘missing’ (both relevant and irrelevant) are still listed under the Miscellaneous tab of our RSOS blogging paper. (Yet most papers that analyse Altmetrics don’t include the Miscellaneous mentions in their analysis…)

Which brings me back to using Altmetrics as a data source.

I had a look at my own papers. Out of 22 papers published in peer-reviewed indexed journals, only 9 include an Altmetrics score on the webpage. For those 9, there is a weak negative correlation between Altmetrics & citations (r = -0.18, p = 0.63; I ran a mixed model to account for year published, and the effect became weaker). Disclaimer: My data is not representative of the broader effect, I’m just illustrating.


Altmetrics scores, just like most other freely available data sources online, have huge caveats. They don’t pick up all share sources, and they don’t measure quality of output…which raises questions about their representativeness and use-by date.

Also, not all journals use them. For example, PlumX seems to be the latest analytics measure replacing Altmetrics; some of the higher-impact journals are already using it, like The Lancet & PLOS. So how do published analyses of Altmetrics contribute to long-term knowledge?

Yes we should be sharing research papers via social media, regardless of how much ‘attention’ it gets. Let’s not wait for Altmetrics scores to inspire us to share our research! But ambiguous social media scores tell us very little about impact. Only time will tell the quality and relevance of a research paper.

© Manu Saunders 2018


Ecosystem services or Nature’s contributions to people?

IPBES has released media summaries of their reports on global land degradation and restoration, and regional biodiversity and ecosystem services assessments. The results of these reports are really important.

Anyone who has been working in this area for the last couple of decades might have noticed that the reports refer to ‘nature’s contributions to people’ (NCP). Where did this term come from and what does it mean?

In a nutshell, it’s a new term for ‘ecosystem services’.

But do we need a new term? The term ‘ecosystem services’ was only established about 20-odd years ago (the concept is centuries’ older). I’ve been working on ecosystem services research for just over 10 years, and NCP came out of the blue for me. I heard about it a few months ago (just before the IPBES reports had been finalised), when a paper was published in Science by a group of well-respected scientists in the ecosystem services field who were involved in the IPBES assessments. Some related papers were published (here and here) in another journal, Current Opinion in Environmental Sustainability. Continue reading

Flower Visitors vs. Pollinators: no evidence that honey bees are the most important pollinator worldwide

Pollination is a complex process. It’s not as easy as an insect simply visiting a flower.

This is important to remember when talking about which species are the ‘best’ or ‘most important’ pollinators. Simply observing an animal visiting a flower is not, on its own, evidence that the animal is pollinating the flower.

Many insects (and other animals) visit flowers, to feed on pollen, nectar or other flower parts. Many of these interactions result in pollination…although some visitors are better pollinators than others. Some of these visitors commit floral larceny – they are robbers or thieves (there’s a difference!) of either pollen or nectar, and they leave without pollinating. Some might even damage flower parts so much that they indirectly affect the flower’s capacity to be pollinated by other visitors.

To know for sure that an insect is pollinating a particular flower, we first need to know what kind of reproductive system that flower has. Is it male, female or bisexual (containing both male and female parts)? Can it self-pollinate, or does it need to be outcrossed to another flower or plant of the same species? Once we know this, we then need to watch the behaviour of the insect that visits that flower. Does the insect visit one flower and fly away, or many flowers in a row? Does it move between plants? Does it actually touch the reproductive parts of the flower when it visits each flower? Continue reading

Blogging for the science community

I’m very excited to present a new paper on blogging that is a direct result of me blogging! The paper is co-authored with some of the awesome ecology bloggers I have been following for years.

I’m proud to fly the flag for the southern-hemisphere blogosphere. Social media are dominated by the northern hemisphere, particularly North America. The timezone effect and geographical silos have a strong effect on how academics interact via social media, and southern hemisphere perspectives can be easily overlooked. Yet, compared to the northern hemisphere, the southern hemisphere has more countries, plenty of unique ecosystems and wildlife, and quite different higher education and academic systems! So I really hope this paper inspires more southern hemisphere ecologists to engage with blogs. Continue reading

Do scientists need Twitter verification to be credible?

A recent blog post by Andrew Kurjata asks some questions that many people have considered. Why does Twitter’s explanation of the sort of people who can be ‘verified’ not include scientists or knowledge brokers? Are politicians, singers and actors more worthy of public interest than scientists? No, of course they’re not. So why are we putting so much faith in the blue tick in the first place?

When I joined Twitter a few years ago individuals couldn’t ask to be verified. Instead, Twitter would “reach out” to eligible accounts when the time was right. I distinctly remember the words used. How snobby, I thought. The implicit assumption was that “reaching out” would occur when the account was deemed famous enough.

Now, the policy has changed, probably because Twitter employees got sick of spending all day reaching out to famous people on Twitter. Now anyone can ask for a blue tick of approval (but not everyone gets it).

Does this change the distribution of people that get verified? As with most awards and honours, women, minorities, or introverts are less likely to self-nominate for prestige, even if they righteously deserve it. Continue reading

Why repetition is good for science

‘Avoid repetition’. Most scientists had this command drummed into their heads early in their career. Science writing should be devoid of repetitive words and sentences. I had to include a preface in my PhD thesis to warn examiners of impending repetition…because my thesis chapters were published/submitted studies from the same system and with somewhat similar sampling methods.

Sure, thoughtless repetition of words and sentences does not make enjoyable reading in any discipline. But repetition of ideas and concepts is essential to storytelling and building memories. So when it comes to science communication, repetition is key. Continue reading

What the ‘venomous bees’ story tells us about science communication

Last week, most of the mainstream Australian news media reported on a University of Melbourne press release about a new study from researchers in the Department of Pharmacology. The study analysed data from Australian public hospital admissions and death records from venomous bites and stings over the period 2001-2013.

All the media stories sent the same message, launching off the popular international myth that Australia has the most venomous creatures on Earth. Finally, this study had evidence to prove that Australia’s bees and wasps were more deadly than our snakes or spiders!

Native spider-hunting wasp (Pompilidae) dragging a paralysed spider across our dirty floor.
Native Australian spider-hunting wasp (Pompilidae) dragging a paralysed spider across our dirty floor.

Technically, the media stories were accurate, and the data in the study did show such a trend (in simplified terms). But this is another great example of how using the generic term for a whole taxonomic group, e.g. ‘bees’, can be seriously misleading. Continue reading