Is peer-review working?

I used to believe it does, but now I have doubts.

If you don't know what peer-review is, the super broken down version of it is as follows:

1, you write a paper and submit to a journal;

2, if the editor decides your paper is worthy of consideration for publication, then they would send it off to two (sometimes three) external experts (reviewers) for review;

3, reviewers scrutinise your paper to the best of their abilities;

4, the reviews are sent back to the editor;

5, based on the reviews the editor makes a decision;

6, the editor notifies you of the decision along with the reviewers comments.

In principal, the peer review process ensures that any scientific research that is made public is scrutinised by equally qualified experts beforehand. Or in other words that no crap gets published.

But in reality?

There are hundreds of crap that get through peer-review, it is unbelievable. The reviewers must either not have been experts, or they were friends of the authors, or were just purely bad scientists. I can only speculate that the reviewers to such papers were such people because you'd hope a fair peer review would pick off such crap. But it doesn't.

In contrast to this, more numerically and quantitatively rigorous studies (i.e. proper scientific studies) get held up in review primarily because of the numerical nature of the studies; i.e., the reviewers that get assigned to such studies tend to have a very thorough understanding of mathematics, the expectations of whom, being a biologist (and a junior one at that), you inevitably can't live up to (at least not yet).

This total lack of consistency in review therefore, forces me to conclude that peer review really depends on who your reviewers are.

I just read an example of crap that is accepted for publication. Granted, it's in press with Medical Hypotheses so it's conceivable that it failed to get published in other more relevant journals, and perhaps peer review is working to some extent but it still got ultimately accepted.

More disturbing is a paper I saw recently in a relatively prestigious journal that I was rather surprised (or even upset) by it even getting accepted. It doesn't mention any numbers or stats. I have a similar study (not exactly the same subject matter but a similar question) with much more rigorous numerical analyses that I've been pushing to get published but it's been through several rounds of reviews only to have it repeatedly rejected. I am still waiting to hear back for the third time (it's been more than 10 months since the initial submission), and here we are, a study with no numbers getting into a relatively prestigious journal. Seriously, I wonder who the reviewers were because I'd really like to name them as preferred reviewers in all my future submissions ;)

On stormtrooper pauldrons from Star Wars

I am going to veer off from my usual (but infrequent) blog posts on palaeontology and blog about another geeky subject.

Star Wars.

I've been a big fan of Star Wars ever since I was a little boy - Return of the Jedi must be the first film I remember seeing at the cinema; my dad took me to see it when I was three but all I remember was being freaked out by Jabba the Hutt and I went through the rest of the film asleep as a sort of a defensive mechanism.

When I was a teenager, I read Kevin J. Anderson's "Jedi Academy Trilogy" then went on to read Timothy Zhan's "Thrawn Trilogy", and from there, I pretty much read every single Star Wars related novel - until I got bored after the story arc reached a time period around 20-25 years since the events of Return of the Jedi (by this point I thought it became pointless to continue reading...).

Anyway, there is one thing that has been bugging me since I was a teenager: when reading through Star Wars materials, I frequently came across passages about stormtrooper commanders being distinguished from normal troopers by wearing shoulder pauldrons (see photo of sandtrooper with orange pauldron), but you never ever saw stormtrooper commanders wearing pauldrons in any of the original three films. Granted, the commanding officer of the sandtrooper unit in Star Wars: A New Hope was wearing a pauldron (see above photo), but all sandtroopers in that sequence were wearing pauldrons, every single one of them. The only distinguishing thing about the commander was that he was wearing an orange pauldron while most troopers were wearing black, but then again, you can see at least one more stormtrooper wearing orange as well. And then there's the one wearing white, which apparently is supposed to be a sergent...

So where did this come from? What is the original source? - I have no idea...

In Revenge of the Sith some clone commanders do wear what look like pauldrons, but a lot of other Clone Wars materials seem to indicate that a pauldron is more of a garment of a distinguished clone officer rather than that of rank (e.g. ARC troopers, or Captain Rex who wears a pauldron as opposed to the more senior Commander Cody who does not). So clearly, pauldrons only have a loose relationship with rank. Of course, one can argue that sandtroopers have clear rank designations by the colour of their pauldron but as I've mentioned above, there are more than one trooper with an orange pauldron and at least two with white; it's not like the colours reflect frequencies that you'd expect from rank, so I think the colouring variation is more to do with cinematography than anything.

The reason why this issue has bugged me is because there are widely held interpretations of on-screen material (such as the role of a pauldron) that was only subsequently given by someone (mostly sci-fi authors) and somehow became "canon". My issue with this is that despite these novels, comics, games, whatever, being licensed by LucasFilm or LucasArts (or by whomever the rights are held), they are only glorified fan-fiction, as far as I can see. So I don't see any difference from these widely accepted interpretations from my never-ending imaginations. I don't especially see why I have to believe something that is not even reflected on-screen; such as the pauldron as a designation of rank in non-sandtrooper stormtroopers (we don't see any evidence for this in the films). Of course, it's fun to imagine specialised stormtrooper armour, equipment, etc, but we should bare in mind that even published materials are just fan-fiction in the end.

SVP 2009 Bristol, and the Romer Prize Session

Despite the complaints that I've heard about the hills and distances from one session to another (come on, it was only a few minutes by foot!), SVP this year was pretty good, in my opinion anyway. I noticed some really good talks with some impressive analytical methods, some really interesting posters, and I also chatted with some intelligent and enthusiastic people.

Of particular interest for me was the Romer Prize Session - not only because I was presenting, but more because Romer Session talks were almost always of high quality research, self-contained and conclusive (unlike some "on-going research", a new locality, or some more scrappy fossils...). Romer talks tend to be more analytically/numerically oriented so there are some stats and numbers to support certain ideas and claims.

There were two talks in particular that I liked, one by my very good friend Tai Kubo (Evolution of limb posture in terrestrial tetrapods inferred from Permian and Triassic trackways), and the other by the Romer Prize winner, Christian Kammerer (Effects of the Permio-Triassic mass extinction on synapsids).

Kubo's talk was on his study using trace fossils to infer limb posture in tetrapods, which on its own is quite unique already because there has been really limited efforts on using trace fossils as a potential source of data. But what makes his research truly unique was his validation of his metrics using observations of limb posture in extant reptiles and their relationship with trackways. His results are already published so I shan't go into the details but his conclusion is that the shift to an erect limb posture occurred rapidly in the Early Triassic much earlier than predicted previously by body fossils.

And on to Krammerer. I really liked his talk. The methodology was robust, statistics were sound, and above all, the sample size was just pure astonishing. One of the things I found rather exciting about his talk was that his results showed that geologically deformed skulls fall within the range of morphospace occupation (albeit perhaps at the extreme margins) expected for that species. This, I think is a significant discovery; although you'd still need a sufficient enough sample size to say that this is true for your own pet group of fossils as well... In any case, I really admire his work and his talk and I think he rightfully won the Romer Prize. So a very big congratulations to Christian Krammerer, and it is a shame I had not had the opportunity to do this in person at the conference (yes, I couldn't find half the people I wanted to meet because of the disparate locations).

All in all, a successful conference (for me): I met half the people I was meant to meet; I met some new people; I learned some new things; I caught up with a bunch of friends and colleagues that I had not seen in months, some I had not seen since our last meeting at Cleveland; and perhaps the most important aspect of the meeting for me, I disseminated my work in front of a large audience. I didn't get much feed back on my talk but I assume either no one really liked it or that no one really understood what I do...

Updates ... and SVP Romer Prize

I've just noticed that it's been about four months since I posted my last blog entry...It is rather scary how time seems to fly even when you are not necessarily having fun...

Anyway, I thought I might as well advertise this. In the upcoming SVP at Bristol, I shall be giving a talk in the Romer Session:

Myology and functional morphology of biting in avian and non-avian dinosaurs.

It's mostly about non-avian theropods now but I have a couple of birds in there for comparison; I don't know now why I emphasised birds in the title, I could have just said dinosaurs... Perhaps it's because I made much of my myological observations in birds (but also a few crocs).

I shan't write too much about it here, but the work is basically a suped-up version of my Masters thesis from way back, almost six years now... I had to come up with a way to rescue the concept if not the work, after I'd realised I had some fatal flaws in the basic assumptions of the calculations in my Masters work, and it took nearly six years to perfect.

Phylogenetic constraint

My coauthors and myself recently submitted a manuscript in which we deal a little with phylogenetic constraint. In the process, I came across something interesting and I thought it would be worthwhile to share it here.

Phylogenetic constraint is a concept of evolutionary biology that has had quite a lot of discussion. Mary McKitrick (1993) has a great way of introducing the concept of phylogenetic constraint:

"in some sense, all evolutionary studies implicate phylogenetic constraint, and reviewing the topic is like trying to catch a greased pig."

How eloquently put.

It means everyone loves talking about phylogenetic constraint but it just goes all over the place and there is no real consensus on what phylogenetic constraint is. So despite all this widely held discussion, phylogenetic constraint remains one of the most difficult and least understood subjects, and possibly one that is actually ill defined as well. The problem is, when we talk about phylogenetic constraint, we are actually talking about some causal factor for a way in which some trait evolves. For instance, we may see some evolutionary changes in skull morphology that is relatively consistent in closely related taxa, thus we may conclude that some phylogenetic constraint is working on this trait preventing it from evolving more disparately from one taxon to another, e.g. in a classic adaptive radiation model. But causation is always difficult to determine if not impossible to confidently assign.

We may not be able to satisfactorily determine if phylogenetic constraint actually is a plausible mechanism in some trait evolution, but we can observe its presumed effects, as phylogenetic conservatism or phylogenetic signal in your phenotypic data. Thus, when many of us say "phylogenetic constraint" we may actually just be talking about an observed phylogenetic conservatism, i.e. some trait that is showing a pattern of evolution consistent with phylogeny. In other words, some trait evolution that shows up as phenotypic difference in two or more taxa that is consistent with the amount of time (or phylogenetic distance) separating these taxa. Phylogenetic conservatism is thus shared characteristics amongst taxa that can predominantly be explained by phylogenetic distance.

The underlying mechanism of phylogenetic conservatism may be due to several causal factors, phylogenetic constraint being one of them (constraint due to developmental, structural, etc. that is shared amongst closely related taxa). But there is another possible mechanism for phylogenetic conservatism in a character, and that would be phylogenetic inertia (as defined by McKitrick (1993)). Characters evolving under phylogenetic inertia would not particularly be under strong selection, and change would be minimal, or in other words, "if it ain't broke, why fix it?". This is a slightly different case from constraint because there isn't really an active constraint working against evolutionary change; there just ain't any selection for that character to change faster than expected. But if there were any evolutionary changes then they would be due to inertia, so more consistent with a random walk model of evolution (i.e. Brownian motion). Under such circumstances, character change is directly proportional to time and thus proportional to branch length (thus differences between phenotypic traits in two or more taxa would be consistent with phylogenetic distance); or the rate of evolutionary change per unit branch length (or time) is constant.

Funnily enough, some widely used methods (maximum likelihood and independent contrasts analysis) actually assume this a priori (i.e. that character evolution conforms to Brownian motion), so before you go on and use these methods in your characters to say something about how characters changed through phylogeny, you really ought to test to see if your character actually does show a strong phylogenetic signal. Because if it doesn't then you may be violating the assumptions of the methods, or these methods aren't suitable for your character. But that's another matter all together.

Anyway, that aside, the terms I used here are probably all used in different contexts elsewhere (e.g. phylogenetic conservatism = phylogenetic inertia) but the point is, these concepts need to be defined well and used to mean specific things.

Principal coordinate analysis and the quest for a solution to a non-existent problem

I had an interesting experience yesterday - spent a good few hours on a silly problem. You don't need to know the technicality of the analyses at all, but I'm sure you'll appreciate the humour in this.

I am frequently running principal coordinate (PCo) analyses recently. This is because I am using an interesting application of multiple regressions and PCoA on phylogeny vs phenotypic variables called the phylogenetic eigenvector regression (PVR; Diniz-Filho et al., 1998; Desdevises et al., 2003). In short, you take a phylogenetic tree of a given group of animals (or plants, or whatever your favourite group of organism), reduce the complex topology into manageable columns of numbers (by PCoA), and test these columns with some phenotypic/ecological variable of your choice for any correlations using multiple regression. Sounds pretty easy, and it is, in practice at least. You can code R to do this very efficiently, if you know the R language already.

Anyway, yesterday, I reread the protocol that I had been following for the last few weeks and realised that, PCoA on phylogenetic trees could sometimes result in negative eigenvalues - which is kind of annoying if you think of eigenvalues as representing the "amount of variation in the data explained by that axis" (Hammer and Harper 2006); a negative value indicates a negative contribution???. Supposedly, this is because of the nature of the phylogenetic distance matrix not necessarily being Euclidean distances. So I had a look at my PCoA results and realised to my horror that a lot of my values were negative. Holy shit! Do I have to go back and reanalyse?

But first, I did the sensible thing and checked if my distances were Euclidean or not (using the is.euclid() in R). Surprisingly, or unsurprisingly, my distances were Euclidean. Strange. But the values are negative...

I sat there scratching my head for a while.

I read further and noted that in cases where you get negative eigenvalues, you may need to transform your original distance matrix following some standard procedures.

I searched for the relevant references and there were several suggested transformation procedures. All of them seemed pretty straight forward. So I tried all of them in turn.

None of them worked. The negative values are still there....

I'm really stuck now. I don't know what the cause of this problem is. Is there something inherently wrong with my data? Is there some other transformation that I could still use? Is there another command in R that could potentially solve this problem - it's really common in R for you to miss a basic command - ? or, is there something fundamentally screwed up with the PCoA command in R, and I've discovered some serious programming failure?

But at this point, it's time for my coffee break. I went out for coffee with my girlfriend, complained to her about it, of course with no solution other than stress relief (which of course I am extremely grateful for her to provide me). I went back to my office, sat down in front of my computer again for more head scratching - by this time, it's more like head-banging-on-desk.

But then, as I was reviewing the R commands for PCoA, it all hit me. How could I be so stupid?

There's this thing in R that returns what's called "points" and "eig", the former being the coordinate points of each specimen along each PCo axis within the multidimensional space, and the latter being the eigenvalues associated with each axis. And "points" are returned by default. I had been looking at the "points" all this time. Of course, the points are going to include negative values because the whole ordination is done so that the points are scattered around the origin.

I turned the "eig" feature on, and R returned the eigenvalues; all positive.

I never thought I could be extremely happy with myself at the same time as being incredibly furious for making such a stupid mistake.

The moral of this story is: you learn from your mistakes.

Chimpanzee plans stone attack

I just read an interesting BBC News article. Apparently, a chimpanzee at the Furuvik Zoo in Sweden had been storing hundreds of stones in anticipation of throwing them later at the zoo visitors. Planning ahead is a cognitive behaviour that has not been traditionally associated with nonhuman animals. This behaviour was observed over the last decade and reported in the journal Current Biology.

Previous reports of planning for future states in animals were all experimentally induced and as such one can be skeptic about these behaviours as being potential lab artefacts. However, this zoo chimpanzee showed spontaneous planning that provides support that previous observations made in the lab may not necessarily be artefacts of experiments; at least in great apes. Primary evidence for this is: that the chimpanzee had collected stones or made concrete discs (see below) early in the morning before the zoo was opened to the public but never when the zoo visitors were present; that the chimpanzee was in a calm state of mind during caching of stones and not at all agitated as he is during his dominance displays towards zoo visitors; that there is a delay between the collecting and throwing of the stones of a few hours; that the chimpanzee does not collect stones for subsequent throwing during the zoo's off-season; and that 'the caches were always located at the shoreline facing the visitors’ area' (Osvath 2009, p. R191).

So this chimpanzee was clearly preparing to throw projectiles at zoo visitors later during the day.

Another striking behaviour is the preparation of concrete discs or missiles from the concrete structures at the centre of the enclosure. Because the zoo is in the subarctic, concrete structures undergo extreme conditions, and are vulnerable to freeze-thaw fracturing (from expansion of freezing water in microcracks and the gap left behind in subsequent thawing) and the surface layer gets partially detached. This is visibly unrecognisable but can be detected by the hollow sound it makes when the damaged area is knocked on. The chimpanzee knows this and has been seen knocking on concrete surfaces from time to time and occasionally hitting harder to knock off concrete fragments. This process of making and using concrete discs is suspected of being a discovery or an invention by the chimpanzee as no one showed him this behaviour. Further, the regcognition that a hollow sound is indicative of damaged concrete and the subsequent link to making projectile weapons out of this, shows just how chimpanzees are actually capable of 'sequentially ordered advanced cognitive operations'.

On a side note, considering the amount of rude mannerisms I've observed and heard in zoo visiters, it's no surprise the chimpanzee did this. If I were in a zoo being stared at by noisy visitors all day long, I'd soon start throwing stuff at them too. My mother told me about a bunch of noisy 7-8 year-olds from a local football club shouting and screaming at all the monkeys at the Ueno Zoo (and believe it or not, this behaviour was led by the team coach!). But because they were so loud (and systematically shouting at all the monkey enclosures), by the time they arrived at the Japanese Macaque enclosure, the alpha male had already rounded up all the troops and retreated them into their den, and the kids (and the coach) were left to wonder where they'd gone...I sometimes wonder who is the smarter...

Reference:
Osvath, M. 2009. Spontaneous planning for future stone throwing by a male chimpanzee. Current Biology, 19: R190-R191. doi.org/10.1016/j.cub.2009.01.010