Chapter fourteen

The observer effect, with appropriate hedging.

I owe the reader a confession before this chapter begins, and I want to get it out of the way before we go any further.

The phrase observer effect has, in the popular imagination, become attached to the most ambitious and most regularly abused metaphor in popular science writing, which is the idea that quantum mechanics tells us that consciousness creates reality, that observation produces the universe, that the watcher and the watched are one, and so on, in a steadily expanding cloud of half-truth and false comfort that has, over forty years, done more damage to the public understanding of physics than any single phenomenon I can think of.

I want you to know, before this chapter starts, that I am not going to do that. I have considered, and rejected, the version of this chapter that would have done it. I want to give you a piece of mathematics that is, on inspection, useful for thinking about anxious self-observation, without trading on metaphors that, in their proper context, mean something quite specific and have very little to do with the human nervous system.

So we are going to spend the first half of this chapter being scrupulous about what the observer effect actually is, in physics, and what it is not. Then we are going to apply the version that does cleanly map to the anxious mind, which is the boring, mundane, classical version, and we are going to apply it carefully.

If you came here for quantum mysticism, I apologize. There are many other books that will sell it to you. This one will not.

There are, in physics, at least three distinct things people commonly call the observer effect. They are related but they are not the same, and the confusion between them is, I think, the root of most of the bad writing on this topic.

The first is the quantum measurement problem. This is a genuinely strange feature of quantum mechanics in which, before a measurement is made, a quantum system can be in what is called a superposition of states, and the act of measurement appears to collapse the superposition into one definite outcome. This is real physics. It is also, importantly, not about consciousness. A photon scattering off an electron is a measurement in the relevant sense. A speck of dust drifting through a beam is a measurement. The measurement does not require a person, or a mind, or even an awareness. The measurement requires only an interaction between the quantum system and any sufficiently large environment that records information about it. The physicists who actually study this for a living do not, as a rule, think consciousness is involved. The physicists who get quoted in popular books sometimes do, but they are usually doing philosophy or theatre.

The second is the Heisenberg uncertainty principle. This is a separate result, often confused with the measurement problem, which says that certain pairs of quantities, like the position and momentum of a particle, cannot be simultaneously known to arbitrary precision. The more precisely you measure one, the less precisely the other becomes determinable, by an amount given by Planck's constant. This is also real physics. It is also not about consciousness. The uncertainty is a property of the quantities themselves, not a fact about who is measuring them.

The third is the classical observer effect, and this is the one that is actually useful for our purposes. The classical observer effect is the entirely mundane fact that, in many systems, the act of measuring something disturbs the thing you are measuring, simply because the measuring instrument has to interact with the system in order to get a reading. A tire pressure gauge has to let a little air out of the tire to measure the pressure. A thermometer has to absorb a tiny amount of heat from the substance it is in. A pollster who calls a thousand people about an election ends up, in the act of asking, slightly changing the way some of those people think about the election. None of these is quantum. All of them are real. The disturbance is, in each case, small enough that for many practical purposes we can ignore it. In some systems, however, the disturbance is not small at all, and in those systems the measurement becomes a dominant influence on what is being measured.

Now I want to write down the principle, in plain language, so that we can apply it.

The classical observer effect says: some systems are sensitive to being measured. The measurement, in those systems, is not a neutral act. It is an intervention. The system you observe, after observation, is no longer quite the system you were observing.

This is, in the natural sciences, a perfectly ordinary statement. The interesting question is which systems are sensitive in this way and which are not. A boulder, for the most part, does not care that you are looking at it. The boulder is essentially the same boulder whether you watch it or do not watch it. A pendulum is similar. A cup of tea, mostly, is similar. These are systems with high inertia compared to the perturbations the measurement applies. They absorb the measurement without significant change.

Other systems are not like this. A spider in the corner of a room is sensitive to being watched. The watching constitutes an event in the spider's environment that the spider must, in some small way, account for. A small bird at a window is sensitive. A child being asked whether they are tired is sensitive. A friend being asked if everything is okay between you is sensitive. In all of these cases, the asking is not neutral. The asking is a small, sometimes large, intervention in the system being asked about.

The list of systems that are sensitive to observation includes, in technical terms, almost every biological system, almost every social system, and almost every cognitive system. Living things, in general, respond to being observed because they are, in their bones, adaptive. They are constantly recalibrating to their environment, and the act of measuring them changes their environment by adding the measurement itself.

The human mind, particularly the anxious one, is among the most sensitive observable systems in the known universe. The mind, asked to report on itself, is changed by the asking. The mind, watching itself for signs of failure, is altered by the watching. The watching does not produce neutral data. The watching produces a different mind from the one that would have been there had the watching not occurred.

I want to return now to the founder, briefly, because the example is the cleanest one we have available.

I described, in Chapter 12, how I ran a greedy algorithm on the management of a company I had founded, and how my insistence on reviewing every piece of work my designers and programmer produced eventually choked the company at the level only I could operate at. That was the optimization-theoretic reading. I want to read the same scene now through the lens of the observer effect, because the two readings are about the same events but they illuminate different aspects of why what happened, happened.

The optimization reading said: I was making the locally optimal move at every step, and the locally optimal moves added up to a globally catastrophic outcome. The observer-effect reading says something complementary and somewhat more uncomfortable, which is that the act of my watching the designers was, in itself, an intervention in their behavior. The designer, watched, was a different designer from the unwatched designer. The unwatched designer made decisions on her own authority, took small risks, sometimes shipped a flawed first version that she planned to refine, and learned, over months, how to do this well. The watched designer did none of these things. The watched designer made decisions she expected me to override. She did not take small risks because the risks were not hers to take. She did not ship flawed first versions because the version that reached me had to be the version that passed my review. She did not learn, over months, how to do anything well, because the function I was forcing her to optimize for was not the design but the design as I would receive it, which is a different and much less interesting function.

The designer I observed every day was not the designer who would have existed if I had not been observing her. The observation was the perturbation. The perturbation was the cost.

I want to be clear about something. The designer was not, in any meaningful sense, less competent because of my watching. The designer was extremely competent. The watching did not make her worse at design. The watching changed what she was optimizing for. The system, faced with a measurement, did what systems faced with measurements do, which is to recalibrate to satisfy the measurement. The designer's behavior, in other words, was not data about her abilities. The designer's behavior was data about her abilities conditional on being watched by me, and that was a different, smaller, more anxious version of her abilities than the unconditional one.

This is, in technical terms, the same problem as the placebo effect in clinical trials. The patient who knows they are being measured behaves differently from the patient who does not. The clinical researcher, knowing this, designs the study to be double-blind, so that neither the patient nor the doctor can perturb the system with their expectations. The good clinician understands, in their bones, that observation is intervention. Most managers do not understand this. Most parents do not understand this. Most anxious people do not understand it about themselves.

Now the harder version of the chapter, because I have not yet applied this where it most needs to be applied, which is to the anxious mind watching itself.

The anxious mind, particularly one trained by a difficult childhood, has a near-constant background process running, which is the process of self-observation. Am I doing well? Am I happy? Am I being too much? Am I being too little? Did that thing I just said come out right? Is the person I am with enjoying my company? Have I been a good partner today? Have I been a good friend? Am I, in some way I cannot quite specify but that I will know when I see it, failing?

This process feels, from the inside, like a quality control mechanism. The anxious mind thinks it is collecting data about how it is doing, the way a manager might collect data about how a process is running. The data, the mind believes, is neutral. The data is just there to be observed. The observation, the mind believes, is benign.

The observer effect, applied here, says something extremely uncomfortable. The act of watching yourself for signs of happiness is one of the most reliable known ways of not being happy. The act of monitoring yourself for signs of competence is one of the most reliable known ways of being briefly incompetent. The act of asking yourself, every fifteen minutes, whether you are okay is, in the strict sense of the chapter, an intervention that significantly reduces the probability that the answer is yes.

The system you are monitoring is your own mood, your own performance, your own relationship with whoever you are with. All of these are extraordinarily sensitive systems. They respond to being watched. They respond, specifically, by becoming less of what they would have been had you not watched.

The cheap version of this argument, which I want to dismantle before going further, is the cliché that thinking about it makes it worse, which is sometimes true and sometimes not, and which is too imprecise to be useful. The precise version is this. Some systems in your life are sensitive to observation in a specific way: the more often you measure them, the more the measurement dominates the variable being measured. Mood is one such system. Romantic connection is another. Creative flow is a third. Sleep is a fourth, and I will note that watching yourself for signs of falling asleep is, in technical terms, the most reliable method of staying awake yet discovered. These are not all the same kind of system, and the mechanism by which observation perturbs them is different in each case. But the structural property is the same. Watching the system, in each case, is not free. The watching has a cost. The cost is paid out of the thing being watched.

Here is a small simulation that demonstrates the principle on a toy model. The model is artificial. The point is the shape of the effect.

import random

def simulate_mood(
    duration=100,
    base_mood=6.0,
    observation_frequency=0.0,
    observation_cost=0.4,
    seed=0,
):
    """
    Simulate a mood over time. At each step, the mood drifts
    randomly. With some frequency, the simulated person checks
    in on their mood. Each check imposes a small cost on the
    mood being measured, because checking is an intervention.

    Inputs:
        duration:              number of time steps
        base_mood:             starting mood, on a scale of 0 to 10
        observation_frequency: probability of checking at each step
        observation_cost:      how much each check perturbs the mood
        seed:                  random seed
    """
    rng = random.Random(seed)
    mood = base_mood
    history = []

    for step in range(duration):
        # mood drifts a little
        mood += rng.gauss(0, 0.3)
        mood = max(0, min(10, mood))

        # the person might check on their mood
        if rng.random() < observation_frequency:
            mood -= observation_cost

        history.append(mood)

    return sum(history) / len(history)


# A person who never checks on their mood:
avg_unwatched = simulate_mood(observation_frequency=0.0, seed=42)
# Result: ~5.9

# A person who checks on their mood about every five steps:
avg_moderate = simulate_mood(observation_frequency=0.2, seed=42)
# Result: ~5.0

# A person who checks on their mood at every available moment:
avg_constant = simulate_mood(observation_frequency=1.0, seed=42)
# Result: ~3.1

print(f"Unwatched:      {avg_unwatched:.1f}")
print(f"Moderate watch: {avg_moderate:.1f}")
print(f"Constant watch: {avg_constant:.1f}")

The simulation is, I want to be honest, simplified to the point of cartoon. Real mood is not a number drifting between zero and ten. Real observation does not subtract a fixed amount from the thing observed. The simulation is a toy. But the toy is doing one specific job, which is to show, in numbers, that a system in which observation has any cost at all will be reliably lower-valued under more frequent observation. The cost may be small. Aggregated across hundreds of checks per day for years, the small cost becomes the dominant feature of the trajectory.

I want to close with a piece of practical advice, because the chapter is incomplete without it.

The advice is not stop watching yourself. The advice cannot be that, because the watching is, in many cases, a survival adaptation, and turning it off completely is neither possible nor desirable. The watching is useful in small amounts. The watching becomes destructive only when its frequency exceeds the system's capacity to absorb it without distortion.

The practical advice is to ask, periodically, the diagnostic question: is this measurement giving me useful information, or is this measurement, by being taken, dominating the variable it was supposed to assess? If you are checking your mood every fifteen minutes, the measurement is not giving you useful information. The measurement is, by its frequency, becoming a dominant cause of low mood. The right move is not to deny the mood exists. The right move is to reduce the measurement frequency to one that the system can absorb, which, for most adults and most systems, is something like once or twice a day, with the rest of the day spent doing things that produce mood as a byproduct rather than examining it as a primary variable.

This is, on inspection, the same advice as Chapter 6, restated in a different vocabulary. Chapter 6 said happiness is a byproduct, not a goal. Chapter 14 says the same thing in observation-effect terms: watching for happiness is a measurement, and the measurement, taken too often, reduces the variable. The two chapters are about the same mistake from two angles. Both arrive, by different routes, at the same operational conclusion: do the activity, do not monitor the activity, and let the byproducts assemble themselves while you are looking elsewhere.

Chapter 15 takes a turn into classical mechanics, and into the question of why human relationships involving more than two people are, in a strict mathematical sense, harder than relationships involving exactly two. The chapter is named for the two body problem, which is solvable, and is really about the three body problem, which is not. The mathematics is precise. The implications for family dynamics, work teams, and complicated friendships are, in my experience, fairly devastating.

For now, the page closes here. The system you are watching is not a neutral system. The watching has a cost. The cost is paid out of the very thing the watching was supposed to measure. The fix is not to stop measuring. The fix is to measure less often, with more care, and to spend the saved attention on doing the activity that the measurement was originally supposed to be in service of.