virtual astronomer

on open science and anonymous peer review

2011-09-14T00:42:00.001-04:00

I recently participated in a Facebook discussion about the Scientific American article by Mary Carmichael:

All Together Now: Scientists Take Peer Review Public

The discussion took place behind the doors of a closed FB group and was posed next to the question: "So is this Open Science?"

Is public peer review open science? I found myself drawn to the question of who constitutes the public audience and whether or not the "public" (whoever they might be) passively observed or actively participated in the publication review. I was somewhat surprised that the gist of the subsequent comments focused on the role that anonymity plays in peer review; the question of who is this public was addressed only to express concern that an active audience would provide something other than the civil, "evidence based" review provided by anonymous peers moderated by activist editors.

I've come to realize that its probably a bit hard to think about the ramifications of open public peer review because one giant issue consumes everyones thinking. The core argument for anonymity is pretty simple: anonymity insulates the scientific culture from the friction (aka 'static') produced in open debate. The argument moves serially to assert that anonymity is necessary to protect junior scientists from retribution by senior authors, whether that retribution be by exclusion or through subsequently negative peer review behind the closed doors of job evaluations or well, behind the anonymity of peer review. This reasoning has certainly been repeated elsewhere, probably like the passing of an oral history, and most recently observed by me in the comments (e.g.) on Peter Coles post / poll about peer review.

Holy cow. Look, I have no idea if the stupid mistakes I've made as an open peer reviewer are viewed that much worse by the offended authors than the stupid mistakes I've made in my papers. What I do know is that the intellectual contributions I've made to papers as its anonymous reviewer have yielded exactly ZERO career benefit compared to the papers I've written.

Anyway if you've been able to listen to me past the whine then you've come to the point of my post, which is that I wanted to reproduce some of my FB comments here. I'm asserting that the issue is not about absolutist (or even pragmatic) approaches to the current culture (where apparently retribution is an assumed behavior) but about changing the culture...

This argument is about culture and cultural norms. Cultures rarely change overnight so we can't very well ask that we drop anonymous peer review in favor of something else for some of the very reasons that you mention -- the current culture does not support it.

Open science is about acting to change current cultural norms and create new ones. These people [in the SA article] are acting this way for this reason; they aren't asking you to do so. They are acting to change the current culture where a senior scientist can destroy [behind closed doors] another scientist over valid disagreements, where overloaded editors are incapable of scientifically brokering referee/author debates, where reviewers get zero attribution for hours or days of effort put into peer reviewing of a paper, and where arbitrary actions by gatekeeper journals sculpt science in an unaccountable way. There are other things that they are trying to change that we haven't even mentioned: a culture that does not value or recognize public engagement through education and public outreach, a culture where data are wasted due to negative or unclear results..

And create new ones: time and time again "communities" have shown themselves capable of crafting expected norms of behavior and this is exactly what happens in science blog comments and forums. The "new articles comments" get winnowed away by a community that does not accept them.

Open science isn't trying to recreate the journal system either although they each believe in improving science through peer action. The journal system is what it is albeit one that exists entirely on the backs of unpaid, uncredited reviewers. Open science is proposing to create new systems (on the backs of unpaid, uncredited actors) that may also improve scientific communication and collaboration. The fact that the "rest of us" can see into the process is another important but also distinct goal.

2011-09-13T23:22:00.002-04:00

I think I'll restart my posting. For a little while.

wiki how?

2010-12-08T22:50:00.001-05:00

tried to write a wikiHow article for an astronomy software tutorial. check back to see if that can work.

also spent a decent fraction of my time looking into archiving wiki content through mirroring, XML source dumps (damn you Trac), etc.

finally, even though a wiki produces pure web content with simple editing tools, complete source packaging, detailed revision control mechanisms (throw in the social web connections if you like that sort of thing) it gonna be a haul to convince...

meta: CMS

2010-12-07T17:01:00.000-05:00

spent most of the day adding facets (but little depth) to my knowledge of software like drupal and wordpress. all attempts to move from the meta to the prototype were unsuccessful. #vmwarefail

cross-domain following

2010-09-30T23:50:00.000-04:00

followed the VAO team meeting with twitter and twiki. Tobin (Michigan) gave me a run down on his work studying the structure and kinematics of proto-stellar envelopes.

thanks to the conveniences of the CfA webserver and under some redirection from Fay (MSR) i spent some time learning about cross-domain hosting policies for images that appear in flash or silverlight apps. Fay moved my data to MSR for now as there is no work around to the way my home institution is exposing our URLs. i added a todo -- uploading my WWT images to an S3 bucket as that appears to be a common way to serve images granting the necessary cross-domain access to SL/Flash.

literate programming cache

2010-09-28T00:29:00.002-04:00

a quick post to capture a few new literate programming threads that I will now uncache.

following a dexy thread lead me to to a vimeo. apparently the code word is artifact.

yes, googling gets me somewhere on stackoverflow.

six months ago I couldn't find anyone weaving with python.

here is some code. here is some more. here is a list. here is a wiki.

where are some users?

must look up why we all love ReST and how long that will last.

more python and peer review

2010-09-27T21:18:00.003-04:00

they go together like bread and honey these days.

after a few hours of *bucks and a traffic ticket, a 2nd referee report, which repeats verbatim much of what was in the initial review, is off.

I am really starting to worry that if I had to follow Hogg's research blog rules then I would have nothing to write about.

The highlight of my day was submitting a bit to our astropython website and doing some brainstorming on ways to increase astropython reader involvement within the confines of the current blog structure. also: what is the best way to accept user contributed code, store it, tag it, share it?

python and peer review

2010-09-24T01:45:00.000-04:00

devoted a questionable amount of time issuing posts to or editing invisible parts of astropython.org. yes, the time average postings to astropython are up due to two of us; still there are few commentors and no new contributors.

then i went back to kindergarten -- cutting figures out of (printed) papers to compare with other figures all as part of a being a journal referee. besides making me print something out, this effort typifies for me the actual barrier in the current publishing paradigm -- papers (sans data) as slim advertisements of research -- to meaningful refereeing: if the closest thing to data i have are cut out paper figures then how exactly am I suppose to provide the high quality / blue ribbon value that journals claim peer-review provides for arbitrating scientific progress?

To oppose "actual" I imply the "imaginary" problem of a referee having too much data to review.

Nevertheless, the review is undone, an editor is likely peeved, and my papers, hypothetically much more tightly written, conceptually woven than the one I'm engaged in review, languish.

all along the data-literature vector did our tangents tangle

2010-09-23T00:19:00.001-04:00

today was begun with a catchup up meeting for the Seamless astronomy group that ballooned far beyond its scheduled hour. the breadth of the discussion was intentional for the benefit of Alberto Pepe, a new postdoc arriving to work with Goodman on some or all Seamless facets. we will see where he plans to dive in, although I'm hoping for a collaborator on my sharing/citation vector. my discussion hijacks included more hypothesizing on scholarly attribution/credit sourced in readings like this.

i am less then certain that we achieved much beyond demonstrating at least 2 or 3 of the 9 proven tips which make [sure your meetings] fail.

then I decided to solicit a bit of curry on the VAO mailing lists regarding who is the intended audience for the forthcoming user forum.

ps: I'm not Australian.

Q/A

2010-09-21T17:12:00.000-04:00

yet again, the meta stomped the data. I can't see when I'll get back to doing some actual research...

the productive part of the day was spent breaking the concept of a user oriented Q/A forum for astronomy into "task" elements for the powers that be to track. the unproductive part of the day was having to change a password, which entails retraining my digits through repetition.

Working?

2010-09-21T00:21:00.000-04:00

Lets try making this blog into a working research diary. It is possible that it will now be better loved.

Sorry, Evernote but at the least I will have some hope that my text formatting will work.

Browsing RDF

2010-06-23T10:19:00.002-04:00

A tim berners-lee tweet about Semantic Web browsers nudged me to revisit my little project comparing different in browser options.

I am pretty dismayed at the range. I also have to disagree with the master, as I find it hard to conclude the Tabulator UI is anything except an poor example. Admittedly, I do not know the extent to which the Tabulator "framework," being a long standing project created by the original krew, is the source that powers these other browsers.

Here is a web album of some screen grabs of my browsing my default RDF test case using different in browser options.

RDF Browsing examples

#astropython

2009-10-05T10:15:00.001-04:00

huh, #astropython wasn't used yet.

Tried Twitter -- Learn to spell!

2009-09-28T11:43:00.004-04:00

While in search of a social means to talk about astronomy, I tried twitter.

Now I just need to learn to spell.

Pb me onward!

2009-04-13T16:48:00.013-04:00

XMM-OM Serendipitous Ultra-violet Source Survey

I was unsuccessfully looking for a VO interface to the Chandra Source Archive in the various VO registries when I found this catalog with the VO Desktop software. I was curious: 6 band photometry; the three UV bands lie within the GALEX NUV. So a little simplisitic messing around to make the plot above is captured as a recipe for UV goodness with the SUSS:

Downloaded entire FITS table (750k sources) from within VO Desktop.
Loaded into TOPCAT. Created color (U-B, etc) columns.
Plot is the UV color (189nm - 268nm) as a function of B-V (nearly Johnson; from the XMM-OM). Nice pair of probably luminosity class sequences.
Plot symbols are rendered using the magnitude errors in the UV bands as auxilliary axis. The symbols become more transparent the larger their errors. Sources with magnitude errors > 0.25 mag are effectively rendered invisible.

Masking 2MASS

2007-12-05T23:35:00.001-05:00

In TopCat one can easy apply a weight to the pixel values in a scatter plot. In this example, I have a database of 3.5 million 2MASS sources seen toward the Pipe Nebula and I have made sure I retained all the database columns provided by the project, including quality flags, observational details and information regarding cross-matches to optical source catalogs. In the figure above, I have made a simple equatorial plot of all 3.5 million sources; in TopCat one can change the point size and transparency, which I have done to make a surface density plot of the region. The general trend in increasing surface density from the NW corner to the SE corner is in the general direction of decreasing Galactic lattitude. A couple of other features are worth noting: a series of white patches of lower surface density stretching from about (RA,DEC) = 258.0,-27.5 to 265.0,-25.5 correspond to where a dark molecular cloud blots out background stars; other smaller white spots are where sources near very bright stars were excluded from the 2MASS catalog; small dark spots are high spatial concentrations of sources, which in this case are Globular Clusters (I can quickly spot 5 of them). I color coded each pixel by the date of its observation by the 2MASS telescopes (see vertical color bar). What does visualizing this tell us? There are a range of observation dates over a 2-3 year period, the data appear tiled in strips that are narrow in RA and very long in DEC, and there is some kind seam in the data at about DEC=-23.6.

Here I use the positional uncertainty quoted in the database to color the pixels. While the quoted positional accuracy (catalog wide) is of order 0.1" at size scales smaller than this there are some interesting (to me!) stepwise variations in the quoted accuracy.

Finally, there is an interesting column that lists the number of optical sources within 5 arcsec of the 2MASS source. Huh. I can only guess we are seeing somekind of footprint of variations in the sensitivity of the optical data, which appear block wise corresponding to the individual photographic plates. I can understand why the optical-infrared associations change near the edges of the dark cloud. But otherwise the grid like pattern is intriguing. To me, of course

What epoch are we in?

2007-10-29T11:01:00.000-04:00

I am doing some workflow experiments using various virtual observatory tools. In one example I downloaded the USNO-B and 2MASS catalogs for a 1 degree region around the young star cluster in NGC 2264.

With TopCat I did a cross-match, and extracted a few interesting quantities and plots. The first plot is the total proper motion vector from USNO-B vs great circle distance between the USNO-B and 2MASS positions on the sky. The color weighting corresponds to the angular separation divided by the total proper motion vector or the net difference in epoch between the two datasets.

Now I just plot the net positional difference versus proper motion in right ascension...

... and declination. Err, both are quoted as epoch 2000, so what is going on here? Naively it seems the epoch of the two datasets are separated by 6-7 years. I know the 2MASS are 1999.9 but what about those USNO-B positions...