Limitations of using Altmetrics in impact analysis

The number of published papers using Altmetrics ‘attention scores’ as a data source to measure impact is rising. According to Google Scholar, there are over 28,000 papers mentioning Altmetrics and impact.

This latest analysis published in PeerJ finds a positive correlation between citation rates and the Altmetric score for papers published in ecology & conservation journals over a 10 year period (2005-2015). This implies: the more a paper gets tweeted, blogged, or talked about in online popular media, the more it will be cited.

This seems commonsense. The more exposure a paper gets online, compared to traditional exposure via journal alerts to the limited number of subscribers, the more people will be aware of it and potentially cite it. This is why we do scicomm. (Although, hopefully people read a paper first and decide on its quality and relevance before citing.)

But the increasing excitement over Altmetrics as a data source is a little concerning. Sure, it’s easily accessible open data that can be used to answer multiple questions. But no index is perfect. Each one is limited by the way it’s measured, how it’s used and the quality of data it’s based on. This applies to all ‘open’ data sources.

The limitations of Altmetrics only came to my attention recently. It’s a cool index and it gives a little insight into a paper’s social media reach. But…

Last year I co-authored this paper on the value of blogs for ecology academics – incidentally, our key argument is about how influential blogs can be for the science community, which is also one of the key findings in the new PeerJ paper.

Now, I write a blog post for nearly every paper I publish, so of course I did so for this one, as did most of my co-authors. A few weeks later, I decided to have a look at the Altmetrics index of our paper to see how it was tracking. I was surprised to see that my own blog post wasn’t listed in the ‘Blogs’ section (or any other section), even though the blog posts of most of my co-authors were.

Intriguing. My own blog post on my own paper on blogging wasn’t registering at Altmetrics. That was kind of weird.

I had a look through their FAQs, without much insight. I had linked correctly, according to their instructions, and my blog had been an established website for nearly 10 years, and fit all their criteria for indexing…so I wasn’t sure why my post hadn’t shown up. I went back and had a look at the Altmetrics of some of my older papers – my first published paper was in 2010 and I started blogging in 2009. None of the Altmetrics scores of my papers included the blog posts I had written about my own papers. For example, this paper from 2013 includes an article I wrote for Grist about the research, but not the separate post I published on my own blog.

So I wrote to Altmetrics to query why my blog posts were missing from my own papers’ scores. I received a reply with a link to a Google form to submit ‘missed mentions’. The form only allows for one specific mention per submission. So I submitted a few separate forms, not just for the blogging paper, but also for a few older ones.

A few weeks later, I checked again. Lo and behold, my blog post on my blogging paper was now listed. Not in the Blogs tab, where you would expect to find it, but under the ‘Misc.’ tab…along with my other blog posts for completely irrelevant papers that I had also filed ‘missing’ reports for.

I wrote back again to try and correct this. The reply from Altmetrics said: “Our blog tracking is not retroactive. We’ll only be collecting blog posts and putting them on the blogs tab if they are published after we have indexed it.” Fine. But they didn’t explain why my irrelevant blogs had been linked to this particular paper’s Altmetric score.

So now all three of the blog posts I submitted as ‘missing’ (both relevant and irrelevant) are still listed under the Miscellaneous tab of our RSOS blogging paper. (Yet most papers that analyse Altmetrics don’t include the Miscellaneous mentions in their analysis…)

Which brings me back to using Altmetrics as a data source.

I had a look at my own papers. Out of 22 papers published in peer-reviewed indexed journals, only 9 include an Altmetrics score on the webpage. For those 9, there is a weak negative correlation between Altmetrics & citations (r = -0.18, p = 0.63; I ran a mixed model to account for year published, and the effect became weaker). Disclaimer: My data is not representative of the broader effect, I’m just illustrating.


Altmetrics scores, just like most other freely available data sources online, have huge caveats. They don’t pick up all share sources, and they don’t measure quality of output…which raises questions about their representativeness and use-by date.

Also, not all journals use them. For example, PlumX seems to be the latest analytics measure replacing Altmetrics; some of the higher-impact journals are already using it, like The Lancet & PLOS. So how do published analyses of Altmetrics contribute to long-term knowledge?

Yes we should be sharing research papers via social media, regardless of how much ‘attention’ it gets. Let’s not wait for Altmetrics scores to inspire us to share our research! But ambiguous social media scores tell us very little about impact. Only time will tell the quality and relevance of a research paper.

© Manu Saunders 2018


Ecosystem services or Nature’s contributions to people?

IPBES has released media summaries of their reports on global land degradation and restoration, and regional biodiversity and ecosystem services assessments. The results of these reports are really important.

Anyone who has been working in this area for the last couple of decades might have noticed that the reports refer to ‘nature’s contributions to people’ (NCP). Where did this term come from and what does it mean?

In a nutshell, it’s a new term for ‘ecosystem services’.

But do we need a new term? The term ‘ecosystem services’ was only established about 20-odd years ago (the concept is centuries’ older). I’ve been working on ecosystem services research for just over 10 years, and NCP came out of the blue for me. I heard about it a few months ago (just before the IPBES reports had been finalised), when a paper was published in Science by a group of well-respected scientists in the ecosystem services field who were involved in the IPBES assessments. Some related papers were published (here and here) in another journal, Current Opinion in Environmental Sustainability. Continue reading

A scientist by any other name: more disciplinary diversity in science communication please

What do you say when someone outside your work circle asks what you do?

I’ve tried a few different responses, depending how much time I have to explain details. I sometimes think I should say ‘I’m a scientist’…it’s more recognisable, and maybe more ‘legitimate’ to doubters (ecology is a misunderstood discipline), and it makes the point that ecology is a bona fide science. But it’s also ambiguous.

What if the person I’m talking to thinks ‘science’ is just the physical or medical sciences? It gets a bit awkward when I hear back something along the lines of ‘Oh medical research is so important, I’m so glad you’re doing something to help’. When I say I’m an ecologist, it’s equally disheartening how many blank or confused looks I get. Continue reading

Starting a citizen science project on a shoestring budget: the Australian Wild Pollinator Count

Some years ago, I had a bright idea. I’d just finished my PhD researching communities of wild pollinators and other beneficial insects in Australian orchards. During that time I’d discovered that lots of people (scientists and non-scientists) thought that European honey bees were the main, if not only, pollinator in Australia.

Most people I spoke to about my work were amazed to learn that we had 1800+ species of Australian native bees, let alone the thousands of other insect species that also pollinate flowers.

I approached my friend Karen Retra, a local bee enthusiast, with a simple plan. Why not try and raise awareness of the forgotten pollinators by getting people outside in their backyard to look for insects? With the myriad of free online tools available, I thought it would be pretty easy to run a regular insect count that anyone could get involved in, just like the UK’s famous Big Butterfly Count or the Aussie Bird Count.

So we started the Wild Pollinator Count, an Australian citizen science project focused on pollinator insects. It runs in the second full week of April and November every year. The idea of this was so that regular contributors have the opportunity to notice differences in their local pollinator communities as the seasons change. Contribution is easy: find a flowering plant during the count week, watch some flowers for 10 minutes and record what you see, enter the data via our submission form. Continue reading

My first podcast!

Thank you to James O’Hanlon for inviting me on to his awesome science podcast In Situ Science. It was my first podcast, but it was just like a good radio interview where you’re given the time to have a conversation, not produce soundbites.

We chat about the Wild Pollinator Count, the challenges of running citizen science projects (data quality, science vs. engagement etc.), how ecosystem services might be one of the most unifying but misunderstood concepts in research, media portrayal of science, James Bond, science community blogging and more!

You can listen to the podcast here (and if you’re new to the site, be sure to subscribe and check out the previous episodes!):

Ep. 41 Pollinators, Bond films, and ecosystem services with Manu Saunders




Flower Visitors vs. Pollinators: no evidence that honey bees are the most important pollinator worldwide

Pollination is a complex process. It’s not as easy as an insect simply visiting a flower.

This is important to remember when talking about which species are the ‘best’ or ‘most important’ pollinators. Simply observing an animal visiting a flower is not, on its own, evidence that the animal is pollinating the flower.

Many insects (and other animals) visit flowers, to feed on pollen, nectar or other flower parts. Many of these interactions result in pollination…although some visitors are better pollinators than others. Some of these visitors commit floral larceny – they are robbers or thieves (there’s a difference!) of either pollen or nectar, and they leave without pollinating. Some might even damage flower parts so much that they indirectly affect the flower’s capacity to be pollinated by other visitors.

To know for sure that an insect is pollinating a particular flower, we first need to know what kind of reproductive system that flower has. Is it male, female or bisexual (containing both male and female parts)? Can it self-pollinate, or does it need to be outcrossed to another flower or plant of the same species? Once we know this, we then need to watch the behaviour of the insect that visits that flower. Does the insect visit one flower and fly away, or many flowers in a row? Does it move between plants? Does it actually touch the reproductive parts of the flower when it visits each flower? Continue reading

Science needs Nature: so why keep them apart?

A few years ago, I wrote an article for Ensia about how popular media tend to separate science and nature stories as if they’re unrelated categories. Most major online news websites have separate pages for ‘Science’ stories (predominantly technology, space and medical research) and ‘Environment’ stories (mostly pieces on nature, wilderness, environmental activism, or cute wildlife, sometimes with a few pieces on climate change thrown in for good measure). Continue reading