Why people do a PhD

a rambling exploration

What brings me to the question

I myself made a mistake of doing a PhD in an irrelevant field (math). After a number of years of reflection, therapy, building a career and seeing more of the world, I have a decent idea of why and how that mistake was made, and how it could’ve been avoided. So now I’m curious to understand to what extent some of the mechanisms and issues I’ve discerned apply to others as well.

Given helping people out of PhD misery is something that I care about, I’ve been career advising multiple late-stage PhD students, often through r/quant, and thus saw up close quite a few folks who’d wake up and be like “wtf, actually I don’t want to do academia at all, what do I do now” anywhere between the last year and last month of their PhD.. Admittedly an extremely biased sample, and yet the experience, on top of various conversations with my past PhD colleagues and other PhD students, makes me wonder to what extent the choice to pursue (and stay in) a PhD is conscious and well thought-out for all the people making it.

Even more interesting are community differences in attitudes towards a PhD. In my current NYC rationalist-adjacent circles, skepticism towards a PhD (and academia generally) is widespread, despite most community members being very obviously “PhD material”, having both the intellectual firepower as well as deep interest in ideas. There is a good empirical and theoretical foundation for this skepticism that I’ll touch on later - for now though I’ll just note I certainly haven’t had access to this circle of ideas back in my home country, and would guess many students from abroad, or from not-very-top US schools, wouldn’t have had it either. Which makes one wonder, how many PhDs could’ve been avoided if more people were socialized into higher-end US intellectual culture earlier? Anecdotally, I’ve certainly seen exposure to the modern US ideas about problems with today’s academia, as well as understanding of the US employment opportunity set more broadly, dampen the interest in pursuing the academic career in quite a few internationals (ever wondered why most hard science PhDs are internationals? the answer is simple: a lot of strong US students just don’t view it as a very good option - and maybe they know their system better than you do looking from afar). Which, again, raises the question of whether the choice regarding a PhD would’ve been the same for some people if they had access to a wider information set.

Why PhD is the only choice: caricatures

I’m a smart gal and the only thing I’ve ever been doing is studying.. guess I’ll do a PhD and become a professor maybe this way I’ll be able to continue doing what I’ve been doing my whole life.

I’m a smart gal sitting in this esoteric subject class that I kinda like and other folks here that I kinda get along with seem to be thinking of continuing to a PhD guess I’ll do that as well, guess that’s what smart people like me are supposed to be doing.

I’m a smart but quite maladjusted guy so it’s hard for me to fit in/feels too intimidating to even try that hackathon/that trading competition but I enjoy taking classes so guess I’ll stick to the ones that don’t have any of that social activity that have many maladjusted folks just like me and that seem to only lead to a PhD as a next step.

I want to contribute to the progress of humanity. Clearly Science is the most important thing for the progress of humanity, most professions are anti-scientific and business is evil. Clearly Science can only be done in academia. So that’s what I’m gonna do. (Confidently says me who never held a job or done anything besides course work.)

The only thing I’ve ever done is taking classes, so I’m no good at anything else. Clearly 20 years is enough experience to judge, I’m likely not ever gonna be good at anything else, so I’d better stick to what fits me and forever stay in academia.

My professors asked me what I liked, and the choice seemed to be only between various school subjects, so I picked the one I liked most, guess that’s my life passion that I’d better pursue, right.

Education is great, and PhD is the most advanced education one can get, surely anyone who is capable of doing a PhD is gonna do it, and the other folks are just not smart enough. (or repeated later: academia is great, clearly the smartest people in the world are in academia, and the ones who leave are just not smart enough)

I’m a smart gal and want to use my brains at work, the only way to do that is research, and the only place to do research is academia.

I’ve been studying this subject for N years, surely I should do a PhD next. Or: I’ve been studying this subject for N years, I don’t know anything else, guess my best choice is to continue.

My current model

Modern academia is largely dysfunctional, in need of a reform, does limited amount of meaningful work contributing to human progress (see Progress Studies and adjacent literatures on this, explainers of why Silicon Valley is experimenting with alternative models for doing science, etc). I’m no expert in the topic and only aware of the general picture, but one possible mechanism could simply be: limited mechanisms for reallocation of resources, growth and resourcing driven largely by internal profession politics, previously important but now less relevant fields still having staffing and clout to continue claiming resources. Of course there is variation, with some PhDs, like ML or Econ or some biomed areas, providing incomparably better non-academic options as well as opportunities for real impact within academia (in my mind these are quite correlated, and related to the field being alive rather than dead), than others (math, physics, a lot of more traditional engineering).

Academia is the right fit for some, but, as hinted in the previous section, currently it tends to suck in many more people than it objectively is the best fit for. Big part of the reason seem to be underexploring: for people who love and enjoy studying, continuing on the academic track is often the path of least resistance. Among people who bother to try out say a tech internship, or better to work a bit after college before committing to a PhD, many abandon the idea of a PhD. It’s more complicated for people who, like me, have personality issues/lack social skills while in college, making it challenging to successfully explore alternatives, even if those alternatives will eventually prove preferable, once the social skills are improved.

You can summarize the above as overconfidence: students being too comfortable in their existing information set (extremely biased, gained mostly from school and professors), underestimating their own capacity to grow and change, underestimating value of checking out other options and exploring other things. People end up using a “greedy algorithm”, making a reasonable decision given their current limitations and limited knowledge. Lack of mentorship is important here: we all have access to professors while in college, but how many of us have had good access to people who were just like us in college but then went non-academic paths?..

Psychology shows people are really bad at judging how they’d feel in a situation that they haven’t experienced. And it seems to me people are really bad at internalizing how much they can grow and change given the right circumstances and experiences. And of course it’s quite impossible for people to imagine the new knowledge and perspectives they can acquire and how that would change their conclusions. What this means is that exploration is super important unless one really knows themselves. Unfortunately though, many people who’d be considering a PhD are by nature “divers” rather than “dabblers”: it’s not natural for them (and I’m one of them) to try random things and explore, and much more natural to pick a spot and dig deep - this might be a fine or even great quality once one found the right career for them.. but it’s a terrible quality for finding it. In more enlightened communities people would be aware and guard against these pitfalls.. but many of us wouldn’t be so lucky as to be part of one of those in our school or college years.

So, the point is, limited self-knowledge informed by life lived largely focusing on schoolwork, combined with understandable pro-academia bias from within academic institutions, combined with personality limitations of the kinds of people who tend to end up on the PhD tracks, result in heavy pro-PhD bias in those people decision-making, leading to many mistakes. I was one of the relatively lucky ones who realized relatively early and with high conviction that academic path is not for me. I look with sadness at some people I know who keep on going that path, from PhD to postdoc to assistant professor to beyond, visibly dissatisfied, yet unable to master the faith to step into the unknown, give up the comfort of certainty, take risk and find a better life for themselves.

Don’t get me wrong, I’m not saying nobody should do a PhD. I actually quite admire the people who “do it right” (in my view): graduate college, try a few plausible careers, earn some money, learn about themselves and the world, understand perfectly well how terrible academic career is in certain respects (pay, location, academic politics, <50% time on research etc), yet still decide that on balance PhD is right for them and that is what they want to do. What I am saying is that a lot of people who go for a PhD end up their due to the bias in their pathway, not due to it being really the best choice for them.

Why people stay in academia

It would only be half bad if too many people went for the PhD, quickly realized their mistake, dropped out/left with masters and moved on to other things. However, while quite a lot of people do get more or less disillusioned about academia few years into the PhD, many fewer actually drop out, and many of even the relatively disillusioned ones still continue in academia beyond PhD. Why is that?

One big reason is simply the persistence of the narrow mindedness/not having explored/willing to properly explore other options theme we discussed earlier. This issue actually might become worse for some as they spend more time in the subject/actually commit to a PhD, as the weight of commitment as well as the sunk cost fallacy make it somehow unseemly to even think of alternatives. Anecdotally, in many of the cases I’ve seen, once a person is willing to properly think of other options they are well on their way to escape - it’s the folks who are both dissatisfied yet dismissive of alternatives that are harder to reach.

Another, somewhat related, is pessimism/scarcity mentality/perceived lack of opportunities. Reality is, modern America is a uniquely advantageous place to be a smart person in. Most STEM and even non-STEM PhDs are perfectly capable of quickly finding a pretty good well-paying job, and, longer term, of building a fitting and satisfying career. True, they might have to discard a couple limiting beliefs and work on their social skills a bit. And they might need to start lower than their PhD snobbery might want them to. But these are all peanuts in the grand scheme of things, not done simply due to the lack of will rather than any kind of intrinsic complexity.

But this is often not how many PhDs view themselves. Because academia is a “superstar career”¹ , many PhDs internalize the scarcity mentality. Because it's so hard to succeed there, many PhDs have low confidence and think they'd have a similarly hard time elsewhere. Because academia requires so much preliminary education they think other fields are similarly hard to enter.

1. Oversubscribed field with too many entrants relative to opportunities, with few opportunities perceived to be so valuable that they justify a lot of misery only to have a shot at them. Typical, albeit more extreme, examples of superstar careers are acting or singing. While academia is not as extreme, it’s important to understand it’s very much on that end of the spectrum, compared to regular careers like law or medicine or software engineering, where typical entrant who was able to get the relevant credentials can pragmatically expect little trouble getting a reasonably satisfying role.