…the Data Policy states the ‘minimal dataset’ consists “of the dataset used to reach the conclusions drawn in the manuscript with related metadata and methods, and any additional data required to replicate the reported study findings in their entirety. This does not mean that authors must submit all data collected as part of the research, but that they must provide the data that are relevant to the specific analysis presented in the paper.
I have been trying to parse this for the last couple of days. I cannot see how “the dataset used to reach the conclusions” could somehow be something different from “all data collected as part of the research.” Trivially you could mean “experiments the results of which are not in the paper.” But that’s dumb, who would even consider including that?
So what IS the distinction PLOS is trying to make? Are the videos of monkeys that Marc Hauser used the “minimal dataset” or are they in the vague subsection of “all data collected as part of the research” category you don’t need to make accessible? If you had 30 DVDs of videos, how would you make them accessible with metadata and DOIs if you wanted to? I can’t see how those videos could be anything other than “the dataset used to reach the conclusions.” But the impression I’ve gotten from others (and from the lengthy PLOS clarifications and FAQs), however, is that the output file from the manual coding of these videos would be the expected “available” data under the PLOS policy.
This illustrates my point, and what PLOS seems unwilling to address. Making the video accessible in the way they demand is insanely burdensome. Making the data file that is the result of coding the video accessible is easy but pointless. Everything important about the analysis occurred between the video and the coded file.
This isn’t just true of behavior videos…it’s nearly anything where you perform measurements that attempt to extract “important” features from complex, continuous phenomena. In other words, a LOT of experimental science. When you have large video or physiology datasets, there is usually not an agreed upon standard for how you convert that into a manageable set of measurements that are amenable to quantitative analysis, or necessarily even agreement on what features are important to measure. For the data I collect, for example, some of are analyzed purely by code. This is great for resting easy about bias, but often makes terrible, systematic mistakes that have to be manually checked, because code is so stupid. For example, often there is no “correct” threshold or segmentation criterion that is going to work in every case, so you either have to have exclusion criteria or a partial human judgment step or some other way of filtering your data. At the other end of the spectrum are totally subjectively scored events (like Hauser’s monkeys). Because humans are humans, you do your best to make the experimenter blinded to whatever independent variables you are interested in. That’s not always possible, so I try to avoid experiments that rely solely on “eyeballing” something.
Most behavioral or physiological analysis is somewhere between “pure code” analysis and “eyeball” analysis. It happens over several stages of acquisition, segmenting, filtering, thresholding, transforming, and converting into a final numeric representation that is amenable to statistical testing or clear representation. Some of these steps are moving numbers from column A to row C and dividing by x. Others require judgment. It’s just like that. There is no right answer or ideal measurement, just the “best” (at the moment, with available methods) way to usefully reduce something intractable to something tractable.
It is interrogating or revisiting this judgment that is the only purpose I can see to “open data” for these kinds of experiments, so that means you need every TB of raw data to make this useful. So, again, is the PLOS policy to make this happen? If so, how? If not, it is pointless. That’s why I think for these kind of data and in the absence of a massive investment (on the scale of Genbank) repository for enormous video files and raw time series data (in what formats? annotated how? paid for by whom?), the PLOS policy is either impossibly burdensome or pointless. Even then, given the large set of uncontrollable and unknown variables that affect the collection of experimental data like this, pooling them or comparing them would be very bad science indeed.
I’m trying to think of more ways to frame this. here’s another try:
There is no answer to the question “what is rat behavior in response to X?” in the sense that there is an answer to “what is the structure of rat myoglobin?” There is only what some particular group of rats did when some version of X (as understood and applied by a given experimenter under lab conditions that they can only partially control) happened to them. The “data” are everything those rats did that the experimenter chose to observe/measure after they did X to them. (You’ve already made important choices in deciding what to measure.) Many information-reducing and analytical steps later, there is a summary of what you decided was important and quantifiable about what the rats did.
This approximate, mediated, interpreted, judgment-based, tentative kind of conclusion is what neuroscience (and many sciences) lives with. We never get discrete right/wrong answers about sequences, or phosphorylation sites, or the number of planets around a particular star, or the mass of a subatomic particle. We aren’t measuring facts of nature, we are asking “what kinds of things usually happen when…?”
If you think that’s frustrating to the “standardize your formats so all your data can be pooled/analyzed by others” set, imagine how frustrating it is to those of use who are doing the experiments. I wish there were a sequence of numbers I could pull out of a behavioral dataset that is the True Result of That Experiment, let alone The True Facts of That Behavior. There is not and never will be, there is only the raw video/trace of some stuff that happened one time.
The gene jockeys keep saying “we have these standards/repositories because the community demanded them.” It might be worth reflecting on reasons why other communities haven’t demanded them beyond being bad scientists, old fashioned, selfish, and uncollaborative.
I will also note that an ongoing issue at PLOS ONE is the extent to which a given editor buys into the PLOS ONE mission. This is as clearly an articulated policy as one could hope for (and one I deeply believe in), and the majority of editors understand it and follow it even when reviewers don’t. A significant portion of the editors, however, don’t get the mission and don’t follow it, happily rejecting papers based on reviewers that say they aren’t “novel” or “exciting” enough. A smaller proportion don’t even seem to know that P1 has a mission that differentiates if from other journals and go on about how P1 should be working to increase its impact factor. Given this editorial variance, I can’t wait to see the wildly differing experiences people have navigating this ball of confusion about what a “minimal dataset” is.
The publisher of the largest scientific journal in the world, PLOS, recently announced that all data relevant to every paper must be accessible in a stable repository, with a DOI and everything. Some discussion of this is going on over at Drugmonkey, and this is a comment that got out of hand, so I posted it here instead.
What is the purpose of this policy? I don’t see how anyone could be fooled into thinking this could somehow help eliminate fraud. Fraud is about intent to deceive, and one can deceive with a selective dataset as easily (or, actually, much more easily) than with Photoshop.
What else? Well, you could comb through the data of that pesky competitor or some other closely related work, looking for mistakes or things they missed that you could take advantage of. Frankly, I can’t imagine bothering. I mean, how could you not have something better to do? Like collecting data. Maybe you’re a stats scold who wants to check the assumptions behind every ANOVA. I can sorta see this… I wish data were presented better in a lot of papers. But I don’t really lose sleep over this for reasons described below.
Then, there is the data utility argument: Everyone should have access to datasets so they can run their own analyses and make new discoveries with the same data. Efficiency, right? Leverage the “utility” we get out of the Johnson’s payroll taxes. This, to me, is lunacy, outside a small subset of disciplines (genomics, crystallography apparently) that generate data that is inherently amenable to curated storage and is to a large extent independent of the details of how it was acquired (a sequence is a sequence, a series of electrical events in a neuron is just a bunch of stuff that happened one time). Two thoughts occur to me:
1. Would I put my name on an analysis of someone else’s data? I would not. I want to be intimately familiar with how the data were generated. I want to talk to the person. I want to have seen the experiment done at least once. Everything else is GIGO.
2. For real, can you get grants and tenure for publishing new analyses of other people’s data? If yes, sign me the fuck up for that shit.
3. OK, three thoughts. Isn’t this the mother of all repeated measures fallacies [edit: Ian points out below that what I mean here is multiple testing, not repeated measures]? Don’t we all know better than to keep re-analyzing the same data until something comes up p<0.05? I mean, y’all remember the recent p-value hysteria about how science was crumbling under its rotten foundation of bad stats? Right. Well, who in the hell believes something is A True Fact because of one p-value? Or one dataset? Or one paper? Scientific knowledge is not data. It is consistent results from a wide range of experimental approaches and negative results from attempts to falsify a hypothesis. FROM DIFFERENT EXPERIMENTS. ACROSS TIME AND SPACE. Science is the motorboat… data are the wake behind it…shit we’ve already churned through. I know there are special cases of clinical trials or whole genomes or whatever that have special utility as resources for data mining and meta-analysis, or where data collection is prohibitively expensive. Most science, I argue, is not like that. Want to know more? Get more data.
4. Someone publishes using analysis A that the whizzle is critical for regulation of yumping. If someone takes that same dataset, does analysis B, and then says “Yes, the whizzle does regulate yumping,” do you believe it more? You should not. What if analysis B says the whizzle is not required for yumping. Do you believe the first paper less? Not particularly, because based solely on one dataset, the whizzle-yumping connection is provisional at best anyway. What to do? Someone has to go get more data, using their science brain to design a better experiment.
4. This is the second point #4. Who pays for this? If I pay out the nose to dump a TB of videos and physiological recordings on Figshare or whatever, who decides if I’ve annotated it sufficiently? Am I obliged to spend time answering every query/complaint/conspiracy theory someone emails the PLOS editorial staff?
5 or 6. But, you say, the clarification of the policy seems to say as long as you just put in “spreadsheets of measurements” you are ok. OK, so remind me again what the point of this was? If you’re showing me the spreadsheet, you’ve already done the important part of the analysis. Now we’re just back to the stats scolds and whizzle-doubters
Sharing data is good. You should do it, it’s part of being a good science citizen. A sweeping, ambiguous, unenforceable mandate like this is nuts.
Maybe I am thinking too narrowly about the kind of data in my subfield. Every video? Every confocal stack? HDs full of EM data? Every trace? Every recording? Who is going to host and curate this?
I dunno….convince me, wackaloons.
New to neuroscience? I’m here to help*. Here are the meetings you might consider going to:
Society for Neuroscience
This is the big one! Go once, you’ll love it or hate it
Pros: It is all skippable without guilt. Everyone is there.
Cons: Vendor swag always disappoints. Food is not food.
What to bring: BodyGlide (ok to publicly reapply in Poster Areas NN-OO)
Gordon Conference: Molecular and Cellular Neurobiology
Does the postsynaptic density contain 147 proteins, or 741 proteins? Competing slides of colored shapes and arrows hold the answer.
Pros: Plenary speakers arrive/leave immediately before/after their talks, so less gasbaggery at chow time.
Cons: Postsynaptic density.
What to bring: Slides from the 90s, apparently.
Cold Spring Harbor: Neural Circuits
Highlights from the Nature Methods table of contents are read to you out loud.
Cons: One camper gets hurt water skiing every summer and ruins it for everyone.
What to bring: News of the outside world.
Janelia Farm: Any conference
Want to know what the combined smell of money and fear is? Me neither.
Cons: Not really a farm.
What to bring: Perspective.
These are not real meetings, just a way to spend grant money on ski trips.
Pros: See above.
What to bring: Skis I guess?
Your Model System / Part of the Brain Meeting
<long sigh> Yeah I guess you should go
Pros: Shared delusion that your shit matters
Cons: Shared delusion that your shit matters
What to bring: Schmoozing A-game. These people review your papers/grants.
* Tenure file, under “outreach” and “mentoring.”
Let’s get something out of the way quick here:
So, when someone notes that I only hire people whose last names start with a letter from the first half of the alphabet, that is in and of itself not evidence of bias. Maybe I’ve received far more applications from that group, maybe the population is not evenly distributed between the two halves, maybe I’ve only hired 3 people, so sampling bias is likely.
Maybe I have an aversion (conscious or not) to all the latter letters because of a humiliating pants-wetting event in kindergarten when I was scolded, laughed at, or beaten by a nun because I thought ellemenoppy was one letter.
One of these is bias on my part (you can figure out the answer at home). In science, studies have shown strong evidence for the following:
- There is bias against under-represented minorities at the NIH in awarding grants. That does not mean simply that they are less likely to win grants. It means that ALL THINGS BEING EQUAL with regard to the applications, being a minority (and in particular being black) gets you judged more harshly by NIH reviewers. (Ginther et al.)
- There is bias against females in scientific assessment. The most recent example of this I can think of is the finding that a CV with a male name on it is judged more favorably by scientists (male or female) than the same CV with a female name on it. (Moss-Racusin et al.)
So it is somewhat astounding that almost every way the NIH can think of to address bias against URM applicants is to try to tell them how to write better applications. The jaw-dropping arrogance and cluelessness and lack of accountability in this position blows my mind. Details and discussion at http://scientopia.org/blogs/drugmonkey/2014/01/16/nih-blames-the-victim/
Then, there is this absurd Lukas Koube guy. Who is he? My best Googling guess is that he is an accountant with no scientific or academic credentials*. He is of the opinion that we should “only focus on quality,” and continues in the idiotic vein of pretending bias doesn’t exist. He sent this opinion to Nature, who…. wait for it… published it. So here is some random non-scientist donut huffer from Texas who has no idea what bias even is chiming in on how scientific journals should use “quality” as their publishing criterion.
I am fairly confident Nature regularly gets “Opinion” letters from Creationists about evolution or nano-brain implants from people living in shacks in Montana. Why aren’t these published? Because they are the same thing: someone who just doesn’t get reality and whos opinion is worthless. It’s like we’re having a discussion about national unemployment levels and someone chimes in with “I saw a hiring sign at Starbucks today.”
I have a theory: someone at Nature isn’t really sure what bias is either, and therefore thought this MRA idiocy occupied a legitimate position within this discussion. That is a total embarrassment for a journal that is clearly striving to at least appear responsive to their own problems with gender skew.
* I don’t think you need these credentials to have an opinion, but we are talking about an opinion piece about scientific publication in a scientific journal.
There was a discussion on Twitter the other day about PhD overproduction that took a few unexpected turns, and it all started with this:
OK, fine. “Life is what you make it” is one of those things that is both trivially true but can also be deeply oblivious if you are, say, talking to someone with a terrible illness or the victim of an accident or a crime. Not every situation we find ourselves in life is “what we make it.” It is also oblivious in the context of, say, Boomers who lived through the most ginormous, unrepeatable economic expansion in US history saying it to GenX or Millenials. Bootstraps, kids! (If I never get smug condescension pretending to be “advice” from a Boomer again in my life, it will be too soon.) But this is not Hope’s context. In the context of graduate education, yes and no. Your PhD “experience” is in large part what you make it…and I hope this is what she meant. Your career prospects are only under your control in a weak and limited fashion… they are in large part what you find various job markets to be 10 years after you started graduate school, not anything you made or are able to make.
“PhD is a fishing license” seems to be what set Drug Monkey off.
As you can imagine, this then became what Twitter does best: #Angry. First, the fishing license analogy became overextended. Then the usual “academic lab work exploits trainees’ desire for unattainable positions” vs. “no one has promised them anything, many reasons to do a PhD.” These positions are natural enemies, but are also non-mutually exclusive. There are many labor markets that are both inherently exploitative and also the best/only opportunity available. College athletics is both a route to education funding and often highly exploitative, for example.
Furthermore, no malice or scheming is required on behalf of individual managers to contribute to exploitation, which will arise naturally from an excess of labor and an absence of regulation. Where most of the blame lies is with the failure of regulators: when things like the Tilghman Report (PDF) are commissioned and written and greeted with denial and open resistance, or when the NIH refuses to even acknowledge the existence of the vast majority of postdocs (those not paid by NRSAs) in their analysis of the trainee pipeline, or when SfN’s solution is to beg industry to keep hiring our discarded postdocs. Oh, and maybe teaching? Or be a science writer! Policy something? All the #alt-bullshit.
However, whether you agree with Hope’s sentiment or not (which can be read in a more sympathetic light not as aspirational or inspirational but as a warning about the likelihood of getting what you want), it bears almost no relation to the tone of PhD program recruiting materials, and it does not change the fact that most people who are both highly qualified and have the desire to run academic research programs will not get that opportunity, and that is a shameful waste of talent and training. Even if you argue that academia is meritocratic*, a meritocracy that leaks more talent than it keeps is still fucked up. The ideal meritocracy has objective criteria for merit, and it rewards all those who meet those criteria. We have a meritocracy that takes a small, essentially random subsample from a huge pool of essentially equivalent meritorious aspirants. At best, the meritocracy we have is the one from Glengarry Glenn Ross: “Third prize is you’re fired.” Even if you came in third out of a thousand.
Finally, at least from the point of view of most public funding agencies, training working scientists is explicitly the purpose of funding their PhDs**. So the idea that you should be 1) preparing them for a wide range of careers, or 2) not making an implicit commitment to producing research scientists is at odds with the intentions of the people who are giving you the money (at least for most publicly-funded scientist). I see no reason for the public to pay for job training for the consulting, pharmaceutical, and patent law industries, which is where I have seen most people go outside of academia. Perhaps private corporations who hire these PhDs should be required to pay back the NIH? Ha ha.
Free markets, including labor markets, work best when regulated*** and when information is maximized. When information is withheld or avoided (How many students and postdocs are paid for by the NIH – RPGs included? What are your PhD graduates doing 10 years later?), that’s dishonest and helps create the conditions for exploitation.
So what do I do? I’m a new PI. In my funding ecosystem, I will explicitly be judged on whether or not I am training people, i.e. have graduate students doing the work. When I met a group of senior grad students here, one out of about 7 or 8 said they were planning to do a postdoc. When I asked what recent grads actually do, they said, “They all do postdocs.” These students have learned what their prospects are like, but they are not being given the tools or opportunity to change course. They aren’t sure what their options are, but most of them probably know a hell of a lot more than the faculty, most of whom have never set foot outside of academia.
* It’s not particularly.
** By it’s own stated logic (“help ensure that a diverse pool of highly trained scientists”) the NRSA program shouldn’t even exist. RPGs are paying for producing far, far more highly trained scientists than any F-award.
*** “Free” and “regulated” also not mutually exclusive, libertarian dipshits.
A lot hand wringing on the tenure track (and the job hunt) is about publication number and venue. I don’t think I have much more to say on venue (other than I do get the sense that perceptions might be starting to shift), but number is interesting. My operating assumption here is, with apologies to Dobzhansky, Nothing in academic careerism makes sense except in light of local tribal norms. This is why all attempts to have standard metrics (alt* or otherwise) are doomed and ridiculous. The relevant tribes here are your field, department, and university. Look at these tribes, because Twitter and blogs don’t have these answers, and you will go crazy trying to parse and apply the various norms of other tribes in your own context.
I was thinking of this because recently Potnia Theron (who is on a tear of incredibly useful blogging) blogged about someone who was denied tenure in her department. It sounds like “Maria” had deeper issues, but Potnia also points out that she “didn’t publish,” which apparently meant her lab produced 6 papers (4 + 2 soon after submitting her tenure dossier). This sent the familiar surge of icy panic through my veins that wakes me up each morning, until I remembered to look at MY tribes, and that I don’t know Potnia’s field, department, when they go up for tenure, what counts as a paper (I mean, anyone with enough friends can get their name on 6 papers if they really want to), etc.
I work at a well-regarded research university, but I can’t find anyone in my department who went up for tenure with anything close to 6 corresponding author papers. My postdoc supervisor is probably going to sail through at an ILAF with exactly 6. And this is 2 years later than tenure submission time in my current department, where labs tend to be small….size varies from ~3-5 trainees for Assistant Profs once they are up and running (at least a year or two, for peculiar local reasons), usually without a postdoc until substantial external operating grants are obtained, maybe ~3 years in, and often not until after tenure. Sometimes never. So that’s tribe #1, and the most important for my own tenure considerations.
My next tribe is my field and subfield. People close to my field will be on my tenure committee and, I assume, people from my field will be writing letters. I looked at a range of people from the most BSD/glam PIs to the most recently tenured PIs labs I could think of and looked at the number of publications.
This is raw pub # by year. Only final author counts here (if co-first is bullshit, then co-corresponding is bullshit times bullshit to the power of bullshit). In my field, a middle author PI probably mailed someone a plasmid or sent them some of their postdoc’s Matlab code…so that also doesn’t count. Reviews don’t count, obviously. And, perhaps controversially, methods papers don’t count**.
PI “a” is the most famous person I could think of in my field, and then we go down by career stage. PIs “f”-“h” are in the “recent tenure” category. All of these PIs are at R1/ILAF type places. I then looked at current trainee number. BSD “c” has over twenty postdocs and graduate students. Senior PI “b”—a department chair—has four. Assuming approximate lab size stability over the period of study, we can calculate papers per trainee-year… that is, the annual productivity per trainee for each PI. I gave them a major break by NOT counting technicians or the occasional masters student.
Tailing off on the junior side might be expected, as the time frame in question includes a lot of their pre-tenure “starting up” time and periods when their labs were likely smaller. Also of note: counter to my own assumption about BSD postdoc factories, PI “c” has both the largest lab and the highest per-trainee productivity.
Putting this in my personal context… my first year I will have a trainee for half a year (0.5 trainee-years). I plan to add 2 in my second year, and one per year after that. This would be ambitious growth for my department. So, when I submit my tenure file (after year 5), I’ll have had 17.5 trainee-years of labor in total (but much less money than any of the people in my analysis). That gives a range of 1.2 papers to 4.6 papers according to the productivity rates above, with anything above ~3.5 papers being at the level of the top labs in my field over a similar time period.
So, what is most important is to know what standards will be applied to YOU rather than to worry about the details of the standards you hear from others. This has nothing to do with being a snowflake or your field being harder or whatever. Undoubtedly, there IS variation in fields about what is considered sufficient for a paper, what “counts” in authorship or venue, etc, but that’s not really the point here either. The point is finding out the specific situation you are in, analyzing it carefully and, when appropriate, quantitatively, and making sure you are judging (and berating) yourself by the right criteria. Of course, none of this is as important as regularly talking to your chair and teh oldz about expectations.
Finally, I know this is beggars-n-choosers for people on the job market, but even at ILAFs there are supportive departments where they expect and want all hires to get tenure (and—surprise!—almost all do), and there are departments where they use tenure as a carrot to grind you out, because only half or two-thirds of people will get tenure. My personal opinion is that the latter places are inherently toxic and fucked, and I wouldn’t want to work in one any more than I’d sign up for a redo of the Stanford Prison Experiment.
* I won’t even link to it, but anyone else see the 2013 Altmetrics Top 100 thing? Yeah, so altmetrics apparently equals clickbaitishness. Woopdeefuckendoo.
** I only say methods/tools papers don’t count because they are often a route to getting an “easy” pub when you can’t get the data. I know, I’ve done it. Also: fuck BRAINI and “optogenetics also works in this brain area/organism/zip code.” Anyway, no labs I’m looking at here are principally methodology labs.
I am not a bioethicist, and I have never conducted human research or, for that matter, animal research requiring much regulation. So while I have some awareness of the legacies of Tuskegee and the many other evils large and small carried out in the name of research, and think these are incredibly important and not taken seriously enough, I have not grappled first-hand with the concept of Informed Consent or real-life IRBs. So my disclaimer here is that my views come from a place of extreme ignorance, but hey, this is the internet, right?
My first poorly-informed impression: I would guess that the default, baseline level of being “informed” (not in any paperwork sense, but just level of understanding) of a 23andme customer seeking out the company and purchasing their own SNP data from them for $100—without any disclaimers or literature from the company—is much higher than the “informed consent” that is routinely obtained in situations and countries where perceptions of authority and individual choice in medical care are very different from in the US. Yet the US model for informed consent is used globally, both by US researchers abroad and by local researchers around the world. So…if 23andMe is conducting research, is it a problem if they don’t use IRBs? Yes. Given the number of people involved, the kind of research/data, and the relative wealth/security of the participants, is it a big problem relative in the context of medical ethics and how informed consents and IRBs are used, under the FDA’s mandate or not? I don’t think so.
It feels like an important distinction to me that customers seek out 23andMe, not the other way around. They are not recruited to a study. 23andMe is selling information: your own SNP data along with a lot of published associations on a slick web site. That fact that consumers might do something misinformed/stupid/harmful with this information is a given for every service/product for all time. Not trying to sound libertarian—I am somewhere between Canadian and Vulcan on the “collective good” vs. “individual rights” spectrum.
Of course, what services like 23andMe do potentially do is create an industry of snake oil, quack remedies, fear, and self-diagnosis. OK: so right now that’s already about 20% of the internet (“Edit your genome with this one weird trick”). I’d say personal genomics is going to increase the general level of health stupidity and exploitation of fear for profit in this country by ~3-5 farts in the current shitstorm of anti-vax, homeopath, faith healing garbage.
Finally… I don’t know the history of 23andMe’s IRB issues, but here’s what it says in the one paper I’ve read (on cilantro tasting like shit to some poor souls):
Participants were drawn from the customer base of 23andMe, Inc., a consumer genetics company. This cohort has been described in detail previously [14, 28]. Participants provided informed consent and participated in the research online, under a protocol approved by the external AAHRPP-accredited IRB, Ethical and Independent Review Services (E&I Review).
So, I don’t know. I could be easily convinced 23andMe has been sloppy about this stuff, but I can’t get worked up that they are a risk to consumers to any alarming degree, especially compared to the shit FDA is doing on behalf of big pharma, like approving drugs that are purposefully designed and marketed to be abused. 23andMe is not sloppy in their service… I checked a bunch their SNP data for me against my whole genome sequence and the error rate was 0, so that’s cool.