In this insightful discussion, Spencer Greenberg delves into the replication crisis plaguing academic psychology research. He discusses projects aiming to improve reliability through replications and details warning signs like questionable statistics. Spencer advocates raising scientific standards to restore public trust. He also champions "renegade science" by independent researchers and highlights tools enabling robust studies outside academia. Overall, Spencer makes a thoughtful case for multiplying skill with truth-seeking to unlock discoveries that benefit society.

Links:

https://www.positly.com/

https://www.guidedtrack.com/

Simone: [00:00:00] Okay, here we go. Hi everyone. We have a very, very special guest today who we have known actually almost as long as we've known each other. We met Spencer Greenberg back in like around 2015 when he was first working on some of his projects that are now pervasively used.

Simone: Which is really, really cool. He is someone that we've profoundly respected for many years. He has been running Clearer Thinking for a ton of time, but more recently he launched the Clearer Thinking podcast, which is a series of interviews with incredible people that we really enjoy. I'm addicted to it personally.

Simone: So please check it out.

Malcolm: I'll just summarize the important point is he's probably one of, if not the most respected social figure in the E. A. And rationalist movement in the New York area, which is a very big thing because it's one of the major hubs of the

Simone: movement. Yeah. And Spencer, could you tell us what your top projects are right now?

Spencer: Yeah. Well, thanks for having me on. So [00:01:00] one of the projects I run is called clear thinking, which you mentioned at clear thing dot org. And what we do is we take interesting ideas from psychology, economics, math, and so on, that people might learn in passing, maybe they'll learn in blog posts or reading books, but they don't generally apply them to their lives.

Spencer: And so our goal is to make it really easy to apply these ideas to your life to try to achieve the things that you want to achieve. So we have these interactive modules, we have over 70 of them right now, and you can use them all for free. And I also do the Clear Thinking Podcast, as you mentioned. In addition to that, we have a bunch of other projects for accelerating social science.

Spencer: So our goal is to try to help. Psychological research go faster, be more robust, be more reliable and help unlock important ideas about human nature that can be of benefit to society. Speaking of that,

Malcolm: what we wanted to focus on this podcast is you recently did some research into the replication crisis, how bad it's gotten, and I think you have some theories on how it could be fixed.

Malcolm: So I'd love you to just dive into that first, explaining what the replication crisis is, its scope and your [00:02:00] research, and then going through potential

Spencer: solutions. Sure, yeah, it's a topic I think about a lot. So basically, there are many really interesting findings in psychology that have unfortunately failed to replicate, which means that basically when people try to redo the same study, collect a new sample of study participants, they just don't get the original answer.

Spencer: And that's been very disturbing. A bunch of findings that were in textbooks and that are really famously known just don't work, it seems. So some examples of this would be from the social priming literature where they do things like have someone hold a warm cup of coffee. And then people would find that there would be rated as more warm, or they'd rate things as more warm because we make these psychological metaphors.

Spencer: Well, it's a really cool sounding idea, but doesn't necessarily replicate. Or another example, when you prime people with words that are related to being older, the people then walk slower. Well, again, a really cool concept, but it didn't replicate when people tried it again. And so the question is, why are so many findings not replicating?

Spencer: And how pervasive a problem is [00:03:00] this. And so looking at the many different replication studies that have occurred, my best guess is that from top journals and very top journals, probably about 40% of the results don't replicate. Cool. Enormous. Wow. Yeah, it's pretty, it's pretty it's pretty shocking. Some people, their response is, Well, science is hard, human nature is complicated.

Spencer: What do we really expect? But my view is that, no, 40% not replicating is way, way too high. It should be something on the order of 5 to 10%. I think that might be reasonable. Yes. But the problem with 40% is it's almost a coin flip. If you read a paper, will this, will this result hold up, right?

Malcolm: Yeah.

Malcolm: So do you think that this is one, one thing that might be fun to go into for the audience is how does this system work? Like the, the scientific system that gets things in these journals and where do you believe it's failing?

Spencer: Yeah, so it's an interesting question why this is happening, because you might think isn't this what peer review is designed to prevent, right?

Spencer: People are submitting to these [00:04:00] top journals, experts in their fields are reviewing the papers. But I think the fundamental problem there, it's not that peer reviewers want to let in garbage papers. They don't really have an... Reason to let in garbage papers. They don't get paid more if they let in bad papers.

Spencer: They don't get more prestige if they let in bad papers. No, the reviewers are reading the papers and trying to let in the problem is that, that whether something replicates or not is not generally visible from reading the paper. Oh,

Simone: because I was going to ask are there warning signs?

Spencer: Yeah. Well, we've, in recent years, there's been a lot of interesting things learned about the warning signs, but people didn't really know what the warning signs were necessarily.

Spencer: In the past, and it's very difficult for people to tell what is a legitimate paper and what's not. I can tell you about some of the warning signs. Please. One is about p values, which are a rather technical subject and something that is confusing people. But the basic gist of a p value is when you're testing a hypothesis using statistics, let's say you say, well, if I give people this psychological treatment, then they'll have less anxiety [00:05:00] at the end of the study on average than if I don't give them the psychological treatment, right?

Spencer: And then you want to know, well, did the result that I got exceed what you'd expect by chance, right? So let's say you find that people who, who got the psychological treatment had An average anxiety rating of 10, whereas those that didn't get it had an average anxiety rating of 7 out of 10. So that looks good.

Spencer: It looks like the psychological treatment worked. But the question is, well, it's 5 compared to 7. Is that really good enough that we can conclude the treatment works? And so what you do is you compute what's known as the p value. And the p value is basically saying, what's the chance I would get a result that's this different?

Spencer: So, a difference this large or larger, if in fact there was no effect. And so if the p value is really small, let's say it's 0. 01, that means there's only a 1% chance you get a result this extreme or more extreme if there was no effect. And so there, so that probably is not due to chance.

Spencer: Whereas if you get a higher p value like, 0. 3, that means there's a 30% chance you'd get a result this extreme or more extreme if there was no effect. And so maybe it's, it might be due to chance. So the idea of the p values is the smaller they are, [00:06:00] the less likely your result is to be due to chance.

Spencer: Now one of the really interesting things that has been found empirically And this is, we've actually looked at this, we looked at over, I think it was over 200 different replication studies looking at what traits actually predict what replicates and what doesn't. And smaller p values in the original paper are associated with better replication.

Spencer: So, if the p value was, let's say, 0. 04, that's still considered statistically significant, because by the definition of statistically significant, anything below 0. 05 is considered statistically significant. And so it may well have been published as a true finding, but it's less likely to replicate than if the original p value had been 0.

Spencer: 001. And so, Part of the reason this is believed to be the case is that people do kind of fishy statistics to get the p value down to be just below 0. 05 so that they can go publish the paper.

Malcolm: So my question is, I mean, do we ever see with really low p values, they're not being replicable as well?

Malcolm: Because, I mean, my understanding is that you just should almost never see that. I mean, that seems like such an obvious thing. If the p value is [00:07:00] low, it's more likely to replicate. But are we still seeing instances in which the p value is low and it's

Spencer: failing to replicate? That's a really good question, because let's say you had a really low p value there's only a one in a hundred thousand chance you get a result this extreme if in fact there was no effect, right?

Spencer: Well, that's, well, that's really strange, does that mean there's definitely an effect? Well, it turns out there can be other reasons besides just a statistical fluke, false positive, why something wouldn't replicate. For example, there could have been a mistake in the original study.

Spencer: Either a mistake in the analysis, right? Maybe they just screwed up the Cisco analysis, and it wasn't really a low p value. Or maybe there was a mistake in the design, where people weren't allocated their groups properly, or other things like this. Of course, there's also the possibility of fraud. That could be another reason.

Spencer: They could have just made up the data, right? So there could be various reasons why, even with a small p value, it won't replicate. But it's certainly more likely to replicate.

Malcolm: So what was your study in this space? Where, where you recently did a study in this space.

Spencer: Elaborate on that. So we launched a project called Transparent Replications.

Spencer: If you want the details, you can find it on our website, [00:08:00] clearthinking. org. And the idea of our project is we want to replicate new psychology papers coming out in the top general science journals. So the Journal of Science and the Journal of Nature, which are just incredibly famous journals that lots of people want to publish in.

Spencer: And our goal is that we want to get to the point where we're doing so many of these replications that if you are a psychologist going to publish in these journals, that there's more than a 50% chance we're going to replicate you. And if we can achieve that goal, then it means that people submitting to these journals are suddenly going to have to grapple with the fact that they're, there's a good chance they're going to be replicated and therefore they're gonna have a different incentive to do their research in a way that makes sure it replicates because if it doesn't replicate, everyone's going to find out.

Malcolm: Now, this is really interesting. So, one question I have is, have you guys worked with, I mean, if there are still really no repercussions. For being unreplicable, what are the extent of the repercussions you expect with a project like this? And how do you expect the academic field to react more broadly?

Malcolm: I mean, if you call out high [00:09:00] status people within the academic field, the academic field's going to begin to frame you like a villain, or negatively, because you are a threat to them, like you're this new exogenous threat. How have you seen them react to it, and do you think that they will apply? heavy penalties to the people who publish unreplicable

Spencer: findings.

Spencer: Yeah, those are really good questions. A lot of interesting things to dig into there. First of all, we've seen a really positive response, largely from the academic community, about our project, which was really nice to see. I think there's a lot of acknowledgement and an increasing acknowledgement that there's a real problem and that standards have to change.

Spencer: The reality is, while it might be good for one individual researcher to push through crap, it's really, really bad for the field. If you're a psychologist, you're, the value of you being a psychologist has declined and declined and declined. General public has become aware of all these issues to the point where some people are just not trusting the papers anymore.

Spencer: That's terrible for the field. So it's actually, although it is kind of a collective action problem, it is really good for the field to raise its standards and it will actually benefit academic [00:10:00] psychologists as a group. So I think that many of them realize, wait a minute, if we raise our standards and we can show the public that our standards are raised, it will actually raise our prestige and raise our credibility.

Spencer: That being said, it's never fun to be told that your paper doesn't replicate, right? Nobody, nobody likes to hear that. Even, even truth seekers, it's upsetting to hear that. But we really try to be fair to researchers and we try to make it clear that we're being fair. So what we do is we contact them, we tell them that we're running a replication.

Spencer: That their paper was selected through this systematic process. We use. We're not singling them out. We use this process to select them. That's kind of clear, clearly defined. And then we say, here's our exact. We built copy of your study. Please look at it and tell us if in any way it deviates from your original research.

Spencer: Because we want to be 100% fair to your research and then, and we want to make sure that if it doesn't replicate, it's because the original paper didn't replicate, not because we screwed something up. We also give them a chance to respond on our final report if they want to give any comments and if they find any mistakes in our work, they can of course tell us and we'll correct those mistakes.[00:11:00]

Malcolm: And so how is all this

Spencer: funded? So we thankfully we have a grant. We're really grateful to the, we've actually got two grants. We're really grateful to those that gave us grants to help us do this.

Simone: What's the end goal in terms of How you hope academia is going to shift going forward, at least like social science research.

Simone: Is it like, are people going to have different methodologies? Are you also trying to, I don't know, make alternate processes available or show people better ways of doing things? What's

Spencer: your goal? So my personal goal is to make psychology into a science that is better at figuring out important truths so that those truths can come out and better.

Spencer: Society and better human lives, right? So that's really what I want to happen. And academia is really the only game in town, pretty much. There are some companies that do some psychological research, but a lot of it is locked away. It doesn't ever get out there in the world. And so if without academia producing important truths about humans, like we're just [00:12:00] Not getting a lot of them, right?

Spencer: So, my hope is that with a project like this, we can help work in the right direction to get scientists to produce more robust findings that then can benefit society. Now I will say also with something like this, it really does hinge on a change in incentives, right? So in order for our project to work, we need it to be the case that people do their research differently.

Spencer: And I think that while, while people are often resistant to hearing that their own work didn't replicate, I do think that it, that other researchers, when they see that and they say, ah, this paper didn't replicate, it does really greatly diminish their belief in that paper. And I think they're much less likely to cite it if they know it hasn't replicated, they're like, ah, that didn't replicate, so, so my hope is that it really does act as a significant incentive to doing better research.

Simone: I really admire the work you're doing in reforming. This kind of academic research, but I also kind of want to touch on what we might call like renegade research or like the resurgence of gentlemen scientist research specifically because you're kind of one [00:13:00] of our heroes on that front.

Simone: You've done a ton of research through clear thinking. You also. You created GuidedTrack and Positly, which have really made it possible for many people to do research on their own without

being

Spencer: Quick side note,

Malcolm: so these softwares, if any of you like to do your own research or you want to go out there and inexpensively run a study, they make it possible to get participants at reasonable costs and they make it possible to run and design a study reasonably.

Simone: Yeah, so GuidedTrack specifically enables you to create These surveys, it even includes, and this is you don't need to learn how to code. It's so easy to use. And this is what Spencer was working on when we first met him. People like Ayla use it for her surveys. And then positively enables you to recruit audiences to fill out those surveys.

Simone: So if you're not famous like Ayla and you can get a large sample size, in fact, she, I think still even uses positively for sample sizes as well, like for participants. So yeah, like these two things together are really making it possible. For gentlemen or [00:14:00] gentlewomen scientists to do these things like, what are your thoughts on the future of renegade science?

Simone: And are you thinking about additional tools or processes to give to people outside of academia? I mean, clearly you're trying to reform academia. I think that's beautiful. But what can non academics learn about doing good social science and other research using them? your tools and other other tools to do these things.

Spencer: Yeah, I'm glad you mentioned our tools because that is part of our mission is to get answers to these important questions, not just through academia, but through all means. And that also means to independent researchers who might be doing stuff outside of academia. So yeah, we created positively and got a track to make it easier.

Spencer: We have a new tool we're going to call it hypothesize to help with the data analysis as well. So if you think about it, If you think about doing research, you've got to build your study, which is what Guided Track is for, you've got to recruit your sample, which is what Positively is for, you've got to analyze your data, and that's what Hypothesis is for.

Spencer: So we're trying to create the trifecta there. Yeah, I think there's just a ton of potential to do interesting research. For example, you mentioned AILA. I mean, AILA has done really interesting research on sexuality. [00:15:00] That is just really different than what academics have done, as far as I can tell, and I think it really adds something to knowledge of the topic.

Spencer: So, I'd love to see more of this, and we do a lot of our own research at Clear Thinking. For example, we ran a randomized control trial on habit formation, where we implemented a habit formation intervention. We... Tested where we track people over, I think it was about six weeks to see if they stuck with their habit using our tool versus a control group that didn't have access to our tool.

Spencer: And we were able to show that way that our tool actually improve people's habit formations. You can actually use it for free on our website. It's called the Daily Ritual Tool clearthing. org. So that's the kind of study we like to run, but we have a bunch of studies running most of the time testing a whole bunch of hypotheses.

Malcolm: You know what I'd really like to see emerge is a gentleman scientist network of the various people like Ayla, like you, who are running studies outside of academia and then disseminate them through social media because I would really, I mean, the core reason I was thinking it'd be really great if there was a network like this is one that makes it easier for people to find these and potentially even [00:16:00] create like a collated journal that's just specifically these types of people.

Malcolm: But I would really love if your service at one day Could run to see if these studies replicate at a higher or lower rate than the studies coming out of academia and traditional journals. Because I would suspect that for example, a list studies would replicate at a higher rate due to her large sample sizes.

Malcolm: Despite the fact that there's a perception that it's lower quality research than, than what's coming out of academia.

Spencer: It's interesting. So when someone is self taught, I mean, I think there are advantages and disadvantages. Disadvantage is that there may be methods they don't understand very well or best practices that they just blow past because they haven't been taught about them.

Spencer: And I think that is a real concern. And I think, I bet if you ask Ayla, I bet she'll say that her earlier studies are much worse than her ones today because she's picking up on some best practices and learning some of the things that maybe she didn't know going into it. On the other hand, a really nice advantage of is you don't have Certain pressures on you to write certain kinds of papers.

Spencer: [00:17:00] You don't have pressures to constantly publish. And so you can kind of take your time or for example, in one of our lines of research, we ran something like 15 different studies. Before we put anything out as a result, because we just really wanted to figure out what was true about the topic, and that's how long it took us to feel confident we knew the answer.

Spencer: We just didn't feel the pressure to just put something out immediately, just, because, oh, wow, we need three papers right now, in order to be on the tender track, right? So I think that's a big advantage. When I think about really good research, I think of it as a few different things coming together, really high level skill, which, which might involve training, but also might be self taught skill.

Spencer: And really high levels of truth seeking and those things kind of multiplying together. So if you have someone who has no skill whatsoever in doing research, I think it's fair to say they're going to have no output. So, zero times anything is zero. On the other hand, if you have someone who's absolutely no truth seeking they're completely indifferent, well, they're essentially gonna just make up information, right?

Spencer: And so, again, you get a zero. And so it's like these, this truth seekingness times a skill, they come together to create [00:18:00] good research. And I think One thing that independent researchers often have going for them is they tend to be really truth seeking because it tends to be why they're motivated in the first place, right?

Spencer: They don't have the career pressure. They just really want to know this topic.

Simone: Paul Graham recently released an essay on doing great work. And like the TLDR of it is get into fields where you have a good aptitude and where you're super self motivated and curious, which there's the truth seeking, there's the, the skill and aptitude and then specifically look for the gaps in current knowledge where there seems to be Not good explanations or just not a lot of research or attention, and I think what's really telling there is, is this really points toward or in favor of the renegade scientist camp because in academia, it's hard sometimes to get funding to get academic support, to get someone to work with or to get funding when it is one of those gaps because that that may not be where the money is.

Simone: That may not be where the institutional support is or the attention or the prestige. So it gets us really excited.

Malcolm: The one [00:19:00] question I wanted to ask you is of the studies that you have ran, which was the most surprising result to you or the result that changed your world perspective most?

Spencer: Oh, that's a great question.

Spencer: I'm going to think about it for a moment though.

Spencer: I keep having all these different ones go through my head and I'm like, nah, that one, that wasn't that surprising or that one didn't change my view that much.

Simone: I'm worried that every social science thing boils down to use it or lose it. People are lazy. It's really hard to change. Like I can't agree on that.

Spencer: Actually, there is one result that comes to mind that we haven't released it yet because we're still analyzing it. So I'm a little reluctant to talk about it, but I will tell, I'll talk about it preliminarily, but like with caveat, we're still analyzing. So what [00:20:00] we actually end up concluding will be seen, but we ran a study on decision making that actually completely shocked me.

Spencer: We, we put people through a decision making protocol where they kind of really. Like how to go through every single like kind of pro and con related decision. And again, they actually had worse. They were less happy with their decisions as fall as followed up like months later when they actually knew the outcome of the decision.

Spencer: And so I'm still processing what exactly that means and why that came out. And we still a lot of work to do to understand that result. But I think if that ends up holding up, I think that will be the most surprising one to me that. That's getting some things. Yeah, that is fascinating.

Malcolm: And something I would tell our audience.

Malcolm: So the way I really want to wrap up this episode for our audience is one of the biggest studies that we've ever done in terms of changing our world perspective was the study that we did, that we ran using data that you had collected for a completely different study. So you had collected data to try to find out.

Malcolm: What correlated with the way that you voted in the last [00:21:00] presidential election cycle, but one of the things that you asked people was how many kids they had, and it was that data set that allowed us to look at and find out what was really correlating with high fertility. And so there's a few things I would, I would impress upon our audience, which is 1.

Malcolm: If you do want to do a study, you can go out there and do it yourself. We're going to put links below here to all of these products that allow you to go out and run these studies yourself. But 2. The great thing about independent researchers, and you can do this to some extent, even with professional researchers, is if you have an idea, that doesn't mean you necessarily need to collect the data yourself.

Malcolm: You can reach out to someone like Spencer or someone like Ayla, or someone like us. And if we've run a similar study in the past, and we still have the data in like a shareable format we can share it with you, which can allow you to do deeper, more interesting digs. on even subjects that might be really tangential.

Malcolm: So that's a really fun way that you can approach things. And if any of our listeners do like really interesting studies, we'd love to have you

Spencer: on. And [00:22:00] this is, I will say to that, there's also this norm that's been changing where people have been getting better at publishing their data sets. We try to do publish our data sets most of the time and a lot more psychologists are publishing their data sets.

Spencer: So you can just find more and more data out there to test hypotheses that you already have. And Some data might be able to help answer a question in just 10 minutes of analyzing it. That's

Malcolm: really exciting. Well, be sure to check out his podcast, the Spencer Greenberg Clearer Thinking Podcast. Clearer

Spencer: Thinking with Spencer

Simone: Greenberg.

Simone: Yeah.

Spencer: Oh, sorry.

Malcolm: You can just Google it or find it in podcasty locations.

Simone: Yeah. And give it a good review because it's so good. More people need to listen

Malcolm: to it. Give it a good review. Yes. Skip subscribing to this one today. Just give his podcast a good review.

Simone: You're doing God's work friends. Good. Well, I'm looking forward to our next conversation already, Spencer.

Simone: And we'll

Spencer: see you soon. Thanks for having me on.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit basedcamppodcast.substack.com/subscribe