Tech fantasies and impostor syndrome

Posted to: DEV; Hacker News

I’d be curious how old-school engineers feel, but to me it seems like software engineering has unique epistemological challenges.

This is because software is immediately complex, and there are really no theoretical limits on its potential complexity. Computer code is all about instructing computer chips to run what amounts to many thousands or millions of instructions. That’s the point of computers. And it’s incredibly easy to throw in an extra loop and add another ten thousand operations.

That, coupled with the fact that no part of software is empirically visible: all instructions are sent to an opaque and unseen microchip.

Of course there are touch points where code is shared, reviewed, can be reasoned about together in real terms. But for the most part, any given line of code has only been looked at by a handful of people, or often a single person. And a whole codebase is vanishingly unlikely to be understood in its entirety by even a single person. Let alone a whole technology stack.

The implications of this visibility problem are huge, this is why computer science is a thing and observability is such a hot topic. Since it’s incredibly hard to know exactly how any given program was written, we have to instead observe and study the behaviour of our programmes as if they were an unknowable black box.

I believe this lends anything to do with computers - software development, machine learning, high-frequency trading, hacking - an immediate mystique. And this mystique carries with it a very large number of negative consequences for the industry and the world.

In my experience, there’s usually a level of pretence in those who work in tech. They have to exhibit competence over and confidence in domains that are almost entirely intangible and theoretical. Often, the person who appears the most technically competent is the person most willing to speak confidently about the reality of domains that we can all only picture in our heads.

And I believe this in turn makes impostor syndrome rampant in the software industry. Since it is, in practical terms, impossible to fully understand the domains within which we’re all working, it would be a very unusual person who genuinely has 100% confidence in their ability. We’re surrounded by fantasists desperately trying to convince everyone around them that they know exactly what they’re doing, but we couldn’t possibly actually think that of ourselves, and so we must be impostors.

People respond to impostor syndrome in different ways. Personally, my impostor syndrome used to manifest as an attempt to bluff my way through it, to appear superior to others so they wouldn’t dare call me out. I think this is probably quite a common response, especially in men. Other people respond by withdrawing and becoming more timid, nervous to take any action.

This can very easily lead to a toxic mix. People lying about how well they understand their products, all trying to keep their head above water by trying to prove they know the unknowable more than the next person. People over-promise, desperate to exhibit their competence. They get themselves into messes of complexity and then when they can’t deliver they shift the blame around.

And for more senior people in organisations this problem is exacerbated from both ends: They both have more responsibility over unknowably complex tech, and they are further from the actual implementation. So they make unfounded promises based on even less knowledge, based on high level assumptions about what should be possible, and then have no choice but to pass the pressure to deliver down to the engineers below them.

There are of course remedies to this. These are good practices in work in general, in human interaction in general, but because of the unique epistemological challenges of the tech industry, I believe they’re absolutely essential in tech:

These are just some ideas, but overall the important thing is to encourage transparency, honesty, humility and understanding. Realise that the team is going on a journey of discovery together, and they need to support each other to have any hope of keeping a handle on the complexity.

By @nottrobin