Why Is Reading Research So Hard?

First thing I ever learned about the world outside the ivory tower: investing is hard.

Indeed, right after my PhD contract expired, I lost all my money in some speculative schemes as I was gullible and a bit delusional (I summarized some of these painful lessons here).

Then I traveled, I befriended many professional investors. In fact, I soon figured out that reading academic research and seeking real innovation beneath it — even as an educated person — is as hard as investing…

…especially when you don’t have the experience of working full-time in academia.

Why? Well… Let me explain the reality of academic careers.

Notably, I spent almost a decade as an active academic researcher in neuroscience.

When you open a scientific paper as an unsuspecting reader, it’s like you were  a guest of a fine-dining restaurant. A waiter nonchalantly serves you a tasty-looking, delicious-smelling dish.

When you work as a PhD candidate for real, it’s like working in the restaurant. Instead, you walk behind the curtain and you see the dirty kitchen.

Indeed, sweaty people run around, yelling at each other, with rats in the corner. And it stops looking tasty.

And so, I’ve put together an essay summarizing the flawed academic system and how it (systemically) produces confusing results and oversells findings.

Looking for the Secret Sauce.

In the startup world, competing on the size of the project is an (almost) lost cause. Overall, for the past twenty years, the IT market has matured into an oligarchy of a few leading players. Think tech giants such as Google, Meta, Apple, Microsoft, or NVIDIA.

When you look deeper into subsidiaries, synergy in products and services, and where the money flows. In fact, it soon becomes obvious that these market monsters don’t even compete anymore. Instead, it is a symbiosis where everyone on top of the food chain wins, and bigger players either buy out or squash the smaller ones.

No wonder that founders and investors keep on looking for the „secret sauce.” This means novelty that patents can protect, or that trade secrets can simply hide.

Something verified as a working solution and hidden from the public. This way, a small player can get afloat and peacefully grow in the market without constant danger of getting eaten alive.

Meanwhile, competitors can simply copy a business model. Innovation in science and technology, though, puts a much higher barrier to entry for all competitors.

Why Does Academia Look Like the Perfect Hunting Ground?

At first glance, it makes perfect sense to look for innovation by digging into academic research…

…especially given that, in this chaotic world, academia seems to have a relatively clear hierarchy with good metrics. It has an explicit „PR point system” — almost as well quantified as Olympic sports:

The best students study at the most prestigious universities on the Shanghai list.

Professorship positions go to the smartest and most accomplished.

Journals with the highest impact factor publish the most impressive discoveries and innovations.

The most acknowledged published pieces get the most citations.

The most accomplished researchers achieve the highest H-index that jointly measures a researcher’s productivity and the overall impact of their research work.

Plus, the most influential figures with the largest long-term impact receive a Nobel prize. As a result, that prize comes with a cheque that’s supposed to offset the trauma of working and making it in academia. It’s sort of an Oscar award, just for brains.

What can go wrong, right?

What Is the Reality of Academic Careers?

Every work environment is like a different video game — each with its own mechanics, reward system, and laws of physics. In other words, it’s almost as if the gravity itself worked differently. I covered some of the insights about the specific mechanics of academia in my book „What Is out There for Me?”

Consequently, academia is particularly fascinating in this respect: it offers a high level of social recognition, which makes it a high-stakes, high-reward environment.

And the numbers are cruel. Despite little experience, PhD candidates (sort of private soldiers in the academic army) enjoy the most deep-work research time during the week.

As a result, they produce the majority of fresh manuscripts. Plus, they are the cheapest labor.

No wonder that universities hire as many new PhD candidates as possible, to increase the number of publications per head. This also boosts the university rankings.

Higher rankings — higher tuition, more undergrad students, and more money. Simple.

Yes, it’s a full-fledged capitalism.

As a result, the odds of becoming a tenure-track researcher as a PhD graduate are slimmer than ever. Overall, current estimates put the odds in STEM at around 10%.

In some subfields, though, it’s 5% or less. According to a peer-reviewed analysis in PNAS Nexus, the average likelihood of landing a tenure-track engineering faculty position between 2006 and 2021 was just 12.4%, with comparable or steeper odds across other STEM fields.

Notably, one of the most competitive fields is cognitive neuroscience. This is perhaps due to the rise of Andrew Huberman and other popculture phenomenons.

Indeed, they made it to the spotlight and turned neuroscience into pop culture. And with great power comes great responsibility… which is, unfortunately, often abused in search for individual career success.

The 9 Circles of Scientific Hell.

In 2012, an anonymous yet highly popular neuroscientist and Twitter/X creator at the time, Neuroskeptic. He published a seminal opinion piece entitled „The 9 Circles of Scientific Hell.”

Ultimately, I find it eye-opening. It was the first systematic classification of all the little sins and acts of misinformation. This kind of misinterpretation of data is unfortunately common in the high-pressure academic community.

You see, scientists don’t just lie or „cheat.” Instead, they „torture the data until it surrenders” — a practice often referred to as p-hacking.

In short, it’s the act of knowledge marketing put to the extreme. According to Neuroskeptic (2012) in Perspectives on Psychological Science, this data-torturing spans a full spectrum of methodological sins. The essay organized these into what it called the “Nine Circles of Scientific Hell,” from post-hoc storytelling all the way down to outright data fabrication.

As a result, those who passively tolerate such practices or oversell their results populate the upper levels of this „Dante’s inferno” in science. Deeper circles, meanwhile, represent increasingly serious forms of statistical manipulation.

What Does p-Hacking Actually Look Like?

Several circles directly capture the logic of p-hacking.

“Post-hoc storytelling” criticizes the practice of presenting unexpected findings as if researchers had predicted them in advance.

One form of preventing this practice is a procedure known as „preregistration.” According to Nosek et al. (2018), preregistration separates prediction from postdiction and makes research findings more credible.

It’s a scheme in which a researcher sets the hypotheses before the experiment. Then, the researcher plans the whole experimental procedure in collaboration with a journal.

In fact, that journal promises to publish the outcome, no matter what the result might be. This encourages researchers to conduct experiments according to the plan. They then report the results regardless of their convergence or divergence from the original hypothesis.

Looks good, but in practice, not every research journal acceptsreregistered studies. Plus, such studies often end up with low impact as they — obviously — rarely report groundbreaking results.

Next, “p-value fishing” depicts researchers trying multiple analyses until achieving statistical significance, without correcting for multiple comparisons. Some instead use an overly liberal cutoff, building an elaborate, arbitrary argument for why that cutoff is beneficial. If you don’t know how dreadful this practice can be, visit Tyler Vigen’s Spurious Correlations, where you will find tons of non-existent associations coming from randomly comparing time series representing unrelated variables.

What Other Red Flags Should Investors Watch For?

Another technique, a „creative use of outliers,” reflects the selective removal of inconvenient data points to obtain desired results. Ugly one—yet, very common.

Likewise, other circles highlight related systemic issues that bias the scientific record. These include nonpublication of negative results, selective reporting of outcomes, and the pressure to produce striking, fundable conclusions.

The lowest level of the inferno—data fabrication—represents the most extreme violation, contrasting with the more common but still harmful everyday practices above it.

Overall, p-hacking is not an isolated technical mistake but part of a broader research culture that incentives, publication bias, and career pressures shape. Even behaviors that academia normalizes can collectively undermine scientific integrity.

What Can a Tech Investor Take From This?

First of all, p-hacking is objectively hard to detect. You look at a research paper and think to yourself: „What did the artist have in mind?”

Still, backengineering the research methodology and finding flaws in the data analysis is hard even for independent reviewers actively working in the same field. It’s even harder for someone who tries to assess the quality of innovation from the sidelines.

But at least you can look for certain red flags that should limit the odds of flawed research scamming you.

1. Limitations.

To be fair, every research paper in any discipline of STEM should have a long „Limitations” section.

You see, every new model or method can only work under certain circumstances. That list of circumstances should be pretty long (possibly, longer than the whole rest of the research paper!)

Realistically, every novelty can only improve ONE parameter among many dimensions of any given problem. That improvement often comes at a cost of some of the other parameters (simple example that’s easy to understand: Vitalik Buterin’s trilemma).

Suppose the invention promises a long list of groundbreaking results without an equally long list of limitations. In that case, it’s probably too good to be true.

2. Look at the Jargon.

It’s not only about red-flag words such as „trend” or „pattern.” Instead, most of the time, authors serve fuzzy rhetoric more softly and nicely.

What Do These Red Flags Actually Sound Like?

If the discussion section is full of round statements such as:

1. Taken together, these findings provide converging evidence for…
2. The present results offer novel insights into…
3. This work advances our understanding of…
4. The observed pattern points to a potentially important role for…
5. These data reveal a previously underappreciated relationship between…
6. Our findings highlight the relevance of…
7. This study uncovers a subtle but meaningful effect of…
8. The results suggest a promising direction for…
9. This challenges existing accounts of…
10. The findings call for a reconsideration of current models of…
11. This work provides a new framework for understanding…
12. The present data bridge a critical gap in the literature.
13. These results redefine how we think about…
14. This offers a unifying perspective on…
15. These findings may have far-reaching implications for…
16. The present work opens new avenues for translational applications in…
17. This approach holds considerable promise for real-world settings.
18. The results may inform the development of next-generation interventions.
19. Our findings have potential relevance for policy and practice.
20. This line of research may ultimately contribute to improving outcomes in…
21. This pattern is consistent with the notion that…
22. One intriguing possibility is that…
23. These effects may reflect an underlying mechanism involving…
24. The data point toward a dynamic interaction between…
25. These findings provide a foundation for future large-scale investigations.
26. The present study represents an important first step toward…
27. Future work building on this approach may yield transformative insights.
28. This proof-of-concept demonstrates the feasibility of…
29. Taken together, this work marks a significant step forward in…
30. Our findings pave the way for a new line of inquiry into…

…just run. If the innovation really works, the numbers should back it up, and this para-marketing jargon should not be necessary to cover its impact.

3. Browse for Innovation in Other, Tangent Disciplines of Science.

Are you sold on a „groundbreaking discovery”? After all, the lab next door may have already invented whatever someone pitches to you as the biggest invention since the wheel. It may sit in a lab full of unsuspecting researchers in another STEM discipline, under another name.

Here’s a simple example. One of the most popular, internationally acclaimed (deemed to be a future Nobel Prize laureate), and at the same time, controversial researchers in neuroscience, Karl Friston of UCL London, is a tourist and a borrower.

He rose to fame in computational and cognitive neuroscience for introducing a whole set of „new” methods to clinical research practice:

• Statistical Parametric Mapping (which simply translates hypothesis testing into the language of human brain mapping),
• Dynamical Causal Modeling (which is nothing but fitting systems of Ordinary Differential Equations, smoothing them with the hemodynamic response function against fMRI data, and comparing simple models using Bayesian Inference),
• Free Energy Principle (which is nothing but adapting the laws of thermodynamics to the language of neural networks in computational neuroscience)

My many, his word is a word of God. He won all the possible accolades in science, except for the aforementioned Nobel Prize.

Yet none of the concepts he promoted were fundamentally new. Still, he just kept on reframing and adapting research frameworks he sampled from Physics, Mathematics, and Computer Science.

And I’m pretty sure there are dozens of Karl Fristons out there, in any discipline.

The Tower of Babel.

Plus, of course, additional chaos comes from the fact that many researchers and tech/science hobbyists out there reinvent the wheel. They simply aren’t aware that another discipline already made similar discoveries or inventions.

Funny enough, you can decompose ANY innovation into a sequence of simple logic steps. Each step falls into one of about 40 categories. Russian researcher Gienrich Altszuller first described the ontology of innovation in 1946, coining the term TRIZ (as in Teoria Reszenija Izobretatielskich Zadacz).

That said, unsuspecting researchers in adjacent disciplines can follow these steps in parallel. The scientific diaspora is just too large to follow — and language-wise, it’s the Tower of Babel.

Conclusion.

Nevertheless, just read research papers with a healthy degree of skepticism. Especially with the blowup of new research literature in times of AI and open preprint servers, finding signal above the noise is hard. This holds regardless of how many PR points one scores (the university’s reputation, the researcher’s H index, citations, and so on and so forth).

Always check back and confront the „innovation” with insiders in the field. Which, in many ways, makes the game of looking for the next big thing even more rewarding and beautiful!

And I’m far from blaming any individual for the systemic problems in academia. The same as investors are under constant pressure of allocating LP’s funds even without good opportunities.

People can ease that pressure but never fully control it. Similarly, researchers are under constant pressure of publishing even in absence of real discovery or inspiration (which, too, no one can fully control).

So, What Should Investors Actually Do?

The key is to accept the world as it is, appreciate other people’s knowledge and still develop a healthy degree of skepticism. This way, we don’t buy into a FOMO and don’t bend under authority. Also, that authority often comes from another world that’s incomprehensibly complex and highly reputable.

I also believe that in the grand scheme of things, algorithms or AI will NEVER replace human intuition. Even with the most shiny numbers on the table, you’ll still need to take a look at the founder’s face. Furthermore, you still need your intuition to tell if the founder is fully truthful or they tuned up the numbers to the extreme.

In times of AI, deep fakes, fabricated data and overall informational noise, intuition only gained in relevance as it’s easier than ever to twist reality and put lipstick on a pig.