Comments on: The Problem of Publishing in the Social Sciences http://alephblog.com/2008/02/27/the-problem-of-publishing-in-the-social-sciences/ Helping Institutions and Ordinary People Invest Better by Focusing on Risk Control Fri, 25 May 2012 17:02:54 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: David Merkelhttp://alephblog.com/2008/02/27/the-problem-of-publishing-in-the-social-sciences/comment-page-1/#comment-17103 David Merkel Sun, 02 Mar 2008 04:24:10 +0000 http://alephblog.com/2008/02/27/the-problem-of-publishing-in-the-social-sciences/#comment-17103 Unknown, good points. I cut my teeth on the 1969 stock splits article by Fama, Fisher, Jensen and Roll. Distinguished men all. Roll’s critique of the CAPM is still a classic. And, yes, Fama has demonstrated intellectual flexibility that many others have not. I like it when the data forces me to abandon old notions; then again, the markets are fluid, and good money management requires a balance between fixed principles and recognizing changed conditions. Unknown, good points. I cut my teeth on the 1969 stock splits article by Fama, Fisher, Jensen and Roll. Distinguished men all. Roll’s critique of the CAPM is still a classic. And, yes, Fama has demonstrated intellectual flexibility that many others have not. I like it when the data forces me to abandon old notions; then again, the markets are fluid, and good money management requires a balance between fixed principles and recognizing changed conditions.

]]> By: The Unknown Professorhttp://alephblog.com/2008/02/27/the-problem-of-publishing-in-the-social-sciences/comment-page-1/#comment-16990 The Unknown Professor Wed, 27 Feb 2008 13:08:17 +0000 http://alephblog.com/2008/02/27/the-problem-of-publishing-in-the-social-sciences/#comment-16990 Excellent piece. A few comments: - A couple of years back, a social scientist published a paper that looked at the distribution of p-values in published studies. Not surprisingly, there was a significant "overage" of studies with p-vales that were "barely" significant (i.e. at the 4.99% level) and very few that just missed being the cutoff under the null, it should be approximately uniformly distributed. Having done this research gig a while, it looks to my jaundiced eye like many of the researchers "trained" their data (you call it specification searching. It's an old joke - if your first pass give you results with p-values of (e.g.) 6% or 7%, you can usually find a tweak (additional regressors, transformation, eliminate a few "bad" observations that'll get yo over the hump. - We studied a paper in grad school that looked at the distribution of the R-Squared. It looked at the problem that arises because we have such large data sets (# obs) and so many potential regressors (think of all the data in Compustat - bsically the universe of financial statement variables on disk). It turns out that an R-2 of 6-7% comes up quite often with a large enouygh data set even if regressors are chosen by chance. Given that so many studies get R-squared in that range, it makes you think. - While the #3 effect is common, it just increases my admiration for folks like Gene Fama. He made his early career off market efficiency and the CAPM, and then 20 years later got into his size/book-market stuff that cast doubt on his earlier stuff. By the way - love the blog. Your stuff is highly relevent to the classes I teach (securities analysis and a student-run investment portfolio class). I require it as reading for my investment students. It's good to see someone playing out their faith in the marketplace and doing both well and good. Excellent piece. A few comments:

– A couple of years back, a social scientist published a paper that looked at the distribution of p-values in published studies. Not surprisingly, there was a significant “overage” of studies with p-vales that were “barely” significant (i.e. at the 4.99% level) and very few that just missed being the cutoff under the null, it should be approximately uniformly distributed. Having done this research gig a while, it looks to my jaundiced eye like many of the researchers “trained” their data (you call it specification searching. It’s an old joke – if your first pass give you results with p-values of (e.g.) 6% or 7%, you can usually find a tweak (additional regressors, transformation, eliminate a few “bad” observations that’ll get yo over the hump.

- We studied a paper in grad school that looked at the distribution of the R-Squared. It looked at the problem that arises because we have such large data sets (# obs) and so many potential regressors (think of all the data in Compustat – bsically the universe of financial statement variables on disk). It turns out that an R-2 of 6-7% comes up quite often with a large enouygh data set even if regressors are chosen by chance. Given that so many studies get R-squared in that range, it makes you think.

- While the #3 effect is common, it just increases my admiration for folks like Gene Fama. He made his early career off market efficiency and the CAPM, and then 20 years later got into his size/book-market stuff that cast doubt on his earlier stuff.

By the way – love the blog. Your stuff is highly relevent to the classes I teach (securities analysis and a student-run investment portfolio class). I require it as reading for my investment students. It’s good to see someone playing out their faith in the marketplace and doing both well and good.

]]>