Limitations of using Altmetrics in impact analysis

The number of published papers using Altmetrics ‘attention scores’ as a data source to measure impact is rising. According to Google Scholar, there are over 28,000 papers mentioning Altmetrics and impact.

This latest analysis published in PeerJ finds a positive correlation between citation rates and the Altmetric score for papers published in ecology & conservation journals over a 10 year period (2005-2015). This implies: the more a paper gets tweeted, blogged, or talked about in online popular media, the more it will be cited.

This seems commonsense. The more exposure a paper gets online, compared to traditional exposure via journal alerts to the limited number of subscribers, the more people will be aware of it and potentially cite it. This is why we do scicomm. (Although, hopefully people read a paper first and decide on its quality and relevance before citing.)

But the increasing excitement over Altmetrics as a data source is a little concerning. Sure, it’s easily accessible open data that can be used to answer multiple questions. But no index is perfect. Each one is limited by the way it’s measured, how it’s used and the quality of data it’s based on. This applies to all ‘open’ data sources.

The limitations of Altmetrics only came to my attention recently. It’s a cool index and it gives a little insight into a paper’s social media reach. But…

Last year I co-authored this paper on the value of blogs for ecology academics – incidentally, our key argument is about how influential blogs can be for the science community, which is also one of the key findings in the new PeerJ paper.

Now, I write a blog post for nearly every paper I publish, so of course I did so for this one, as did most of my co-authors. A few weeks later, I decided to have a look at the Altmetrics index of our paper to see how it was tracking. I was surprised to see that my own blog post wasn’t listed in the ‘Blogs’ section (or any other section), even though the blog posts of most of my co-authors were.

Intriguing. My own blog post on my own paper on blogging wasn’t registering at Altmetrics. That was kind of weird.

I had a look through their FAQs, without much insight. I had linked correctly, according to their instructions, and my blog had been an established website for nearly 10 years, and fit all their criteria for indexing…so I wasn’t sure why my post hadn’t shown up. I went back and had a look at the Altmetrics of some of my older papers – my first published paper was in 2010 and I started blogging in 2009. None of the Altmetrics scores of my papers included the blog posts I had written about my own papers. For example, this paper from 2013 includes an article I wrote for Grist about the research, but not the separate post I published on my own blog.

So I wrote to Altmetrics to query why my blog posts were missing from my own papers’ scores. I received a reply with a link to a Google form to submit ‘missed mentions’. The form only allows for one specific mention per submission. So I submitted a few separate forms, not just for the blogging paper, but also for a few older ones.

A few weeks later, I checked again. Lo and behold, my blog post on my blogging paper was now listed. Not in the Blogs tab, where you would expect to find it, but under the ‘Misc.’ tab…along with my other blog posts for completely irrelevant papers that I had also filed ‘missing’ reports for.

I wrote back again to try and correct this. The reply from Altmetrics said: “Our blog tracking is not retroactive. We’ll only be collecting blog posts and putting them on the blogs tab if they are published after we have indexed it.” Fine. But they didn’t explain why my irrelevant blogs had been linked to this particular paper’s Altmetric score.

So now all three of the blog posts I submitted as ‘missing’ (both relevant and irrelevant) are still listed under the Miscellaneous tab of our RSOS blogging paper. (Yet most papers that analyse Altmetrics don’t include the Miscellaneous mentions in their analysis…)

Which brings me back to using Altmetrics as a data source.

I had a look at my own papers. Out of 22 papers published in peer-reviewed indexed journals, only 9 include an Altmetrics score on the webpage. For those 9, there is a weak negative correlation between Altmetrics & citations (r = -0.18, p = 0.63; I ran a mixed model to account for year published, and the effect became weaker). Disclaimer: My data is not representative of the broader effect, I’m just illustrating.


Altmetrics scores, just like most other freely available data sources online, have huge caveats. They don’t pick up all share sources, and they don’t measure quality of output…which raises questions about their representativeness and use-by date.

Also, not all journals use them. For example, PlumX seems to be the latest analytics measure replacing Altmetrics; some of the higher-impact journals are already using it, like The Lancet & PLOS. So how do published analyses of Altmetrics contribute to long-term knowledge?

Yes we should be sharing research papers via social media, regardless of how much ‘attention’ it gets. Let’s not wait for Altmetrics scores to inspire us to share our research! But ambiguous social media scores tell us very little about impact. Only time will tell the quality and relevance of a research paper.

© Manu Saunders 2018


Robo bees are back, but will they last?

The robot bee story is back in the news. I covered some of the new research and associated media hype last year. The latest: a patent has been filed for building ‘pollinator drones’ and the media (both newsy and social) are in despair, as the end is clearly nigh.

But don’t worry. Here are a few challenges the pollinator drones will need to overcome before they can take over agriculture: Continue reading

Ecosystem services or Nature’s contributions to people?

IPBES has released media summaries of their reports on global land degradation and restoration, and regional biodiversity and ecosystem services assessments. The results of these reports are really important.

Anyone who has been working in this area for the last couple of decades might have noticed that the reports refer to ‘nature’s contributions to people’ (NCP). Where did this term come from and what does it mean?

In a nutshell, it’s a new term for ‘ecosystem services’.

But do we need a new term? The term ‘ecosystem services’ was only established about 20-odd years ago (the concept is centuries’ older). I’ve been working on ecosystem services research for just over 10 years, and NCP came out of the blue for me. I heard about it a few months ago (just before the IPBES reports had been finalised), when a paper was published in Science by a group of well-respected scientists in the ecosystem services field who were involved in the IPBES assessments. Some related papers were published (here and here) in another journal, Current Opinion in Environmental Sustainability. Continue reading

Workshopping invasive insects

Last week I was delighted to attend a workshop at Monash University focused on using EICAT methods to assess environmental impacts of invasive insect species. Thank you to Melodie McGeoch and her team (Dave, Chris & Rebecca), and Andrew and Carol from the Invasive Species Council, for inviting me in the first place, and for organising an excellent, productive week. We were also very lucky to have Sabrina Kumschick and Helen Roy there to share their expertise in developing and using EICAT.

It was a ‘proper’ workshop, i.e. a small group of researchers working on a project together with planned outcomes, rather than a training or upskilling ‘workshop’ (why aren’t they just called courses?!). As an early career researcher, it was so rewarding to be there. Research workshops have similar benefits to conferences, in that you have the opportunity to discuss new ideas and network outside your normal collaborative groups. But I find workshops much more fulfilling than conferences, because you have more time to develop those ideas, learn new perspectives, and really get to know people you may not otherwise cross paths with. Continue reading

A scientist by any other name: more disciplinary diversity in science communication please

What do you say when someone outside your work circle asks what you do?

I’ve tried a few different responses, depending how much time I have to explain details. I sometimes think I should say ‘I’m a scientist’…it’s more recognisable, and maybe more ‘legitimate’ to doubters (ecology is a misunderstood discipline), and it makes the point that ecology is a bona fide science. But it’s also ambiguous.

What if the person I’m talking to thinks ‘science’ is just the physical or medical sciences? It gets a bit awkward when I hear back something along the lines of ‘Oh medical research is so important, I’m so glad you’re doing something to help’. When I say I’m an ecologist, it’s equally disheartening how many blank or confused looks I get. Continue reading

Moving for academic careers is not ‘just like other jobs’

Recently, I’ve been hearing a recurring argument for why academic relocation is not a big deal compared to other jobs, usually in response to early career researchers discussing how often, and how far, they’ve had to move for work (e.g. this recent post over at Scientist Sees Squirrel).

Yes, the need to diversify one’s experience by moving locations is common to many careers. Most people will move once or twice, usually early on, to establish their future job or career path. Some careers demand continual relocation, even after establishment.

However, moving for academic careers is very different to most other mobile careers for one very important reason that is often glossed over: the mismatch between expectation and support. Continue reading

Starting a citizen science project on a shoestring budget: the Australian Wild Pollinator Count

Some years ago, I had a bright idea. I’d just finished my PhD researching communities of wild pollinators and other beneficial insects in Australian orchards. During that time I’d discovered that lots of people (scientists and non-scientists) thought that European honey bees were the main, if not only, pollinator in Australia.

Most people I spoke to about my work were amazed to learn that we had 1800+ species of Australian native bees, let alone the thousands of other insect species that also pollinate flowers.

I approached my friend Karen Retra, a local bee enthusiast, with a simple plan. Why not try and raise awareness of the forgotten pollinators by getting people outside in their backyard to look for insects? With the myriad of free online tools available, I thought it would be pretty easy to run a regular insect count that anyone could get involved in, just like the UK’s famous Big Butterfly Count or the Aussie Bird Count.

So we started the Wild Pollinator Count, an Australian citizen science project focused on pollinator insects. It runs in the second full week of April and November every year. The idea of this was so that regular contributors have the opportunity to notice differences in their local pollinator communities as the seasons change. Contribution is easy: find a flowering plant during the count week, watch some flowers for 10 minutes and record what you see, enter the data via our submission form. Continue reading