Does deliberation limit prediction?

There is a longstanding debate about the claim that "deliberation crowds out prediction" (DCP). The question at the center of this debate is whether I can treat an action as a live option for me and at the same time assign a probability to whether I will do it. Spohn and Levi argue that we cannot assign such probabilities, for example, while Joyce and Hájek argue that we can.

A claim related to DCP that I've been thinking about is as follows:

Deliberation limits prediction (DLP): If an agent is free to choose between her options, it will not always be possible to predict what action an agent will perform in a given state even if (i) we have full information about the state and the agent, and (ii) the agent does not use a stochastic decision procedure.

DLP is weaker than DCP in at least one respect: it doesn't say that agents can never make accurate predictions about things they are deliberating about, just that they can't always do so. DLP also stronger than DCP in at least one respect: it extends to the predictions that others make about the actions of agents and not just to the predictions that agents make about themselves.

Here is a case that I think we can use to support a claim like DLP:

The Prediction Machine

Researchers have created a machine that can predict what someone will do next with 99% accuracy. One of the new test subjects, Bob, is a bit of a rebel. If someone predicts he'll do something with probability ≥50%, he'll choose not to do it. And if someone predicts he'll do something with probability <50%, he'll choose to do it. The prediction machine is 99% accurate at predicting what Bob will do when Bob hasn't seen its prediction. The researchers decide to ask the machine what Bob will do next if Bob is shown its prediction.

We know that no matter what the machine predicts, Bob will try to act in a way that makes its prediction inaccurate. So it seems that either the prediction machine won't accurately predict what Bob will do, or Bob won't rebel against the machine's prediction. The first possibility is in tension with the claim that we can always accurately predict what an agent will do if we have access to enough information, while the second possibility is in tension with the claim that Bob is free to choose what to do.

(Note that we could turn this into a problem involving self-prediction by supposing that Bob is both the prediction machine and the rebellious agent: i.e. that Bob is very good at predicting his own actions and is also inclined to do the opposite of what he ultimately predicts. But since self-prediction is more complex and DLP isn't limited to self-predication, it's helpful to illustrate it with a case in which Bob and the prediction machine are distinct.)

The structure of the prediction machine problem is similar to that of many problems of self-reference (e.g. the grandfather paradox, the barber paradox; the halting problem). It's built on the following general assumptions:

Prediction: there is a process f that, for any process, always produces an accurate prediction about the outcome of that process

Rebellion: there is a process g that, when fed a prediction about its behavior, always outputs a behavior different than the predicted behavior

Co-implementation: the process g(f) is successfully implemented

In this case, f is whatever process the prediction machine uses to predict Bob's actions, g is the (deterministic) process that Bob uses when deciding between actions, and g(f) is implemented whenever Bob uses f as a subroutine of g.
We can see that if process g(f) is implemented then either f does not produce an accurate prediction (contrary to Prediction) or g does not output a behavior different than the predicted one (contrary to Rebellion). Therefore it cannot be the case that there exists a process f and there exists a process g and the process g(f) is implemented, contrary to Co-implementation. So if agents are free to act and to use a deterministic decision procedure like Bob's to pick their actions, it will not be possible to predict what they will do in all states (e.g. those described in the prediction machine example) even if we have full information about the state and the agent, as DLP states.

Joyce (p. 79–80) responds to a similar style of argument against our ability to assign subjective probabilities assigned to actions. The argument is that "Allowing act probabilities might make it permissible for agents to use the fact that they are likely (or unlikely) to perform an act as a reason for performing it." Joyce's response to this argument is as follows:

I entirely agree that it is absurd for an agent's views about the advisability of performing any act to depend on how likely she takes that act to be. Reasoning of the form "I am likely (unlikely) to A, so I should A" is always fallacious. While one might be tempted to forestall it by banishing act probabilities altogether, this is unnecessary. We run no risk of sanctioning fallacious reasoning as long A's probability does not figure into the calculation of its own expected utility, or that or any other act. No decision theory based on the General Equation will allow this. While GE requires that each act A be associated with a probability P(• || A), the values of this function do not depend on A's unconditional probability (or those of other acts). Since act probabilities "wash out" in the calculation of expected utilities in both CDT and EDT, neither allows agents to use their beliefs about what they are likely to do as reasons for action.

The General Equation Joyce states that the expected value of an action is the probability of a state given that an action is performed (e.g. the state of getting measles given that you received a measles vaccine), multiplied by the utility of the outcome of performing that act in that state (e.g. the utility of the outcome "received vaccine and got measles"), where we sum over all possible states. This is expressed as Exp(A) = Σ P(S || A) u(o[A, S])).

But suppose that Bob derives some pleasure from acting in a way that is contrary to his or others' predictions about how he will act. If this is the case, it certainly does not seem fallacious for his beliefs about others' predictions of his actions to play a role in his deliberations (Joyce's comments don't bear on this question). Moreover, it does not seem fallacious for his own prior beliefs about how he will act to play a role in his decision about how to act, even if such reasoning would result in a situation in which either he either fails to accurately predict his own actions or fails to act in accordance with his own preferences. (Similar issues are also discussed in Liu & Price, p. 19–20)

When confronted with self-reference problems like this, we generally deny either Prediction or Rebellion. The halting problem is an argument against its variant of Prediction, for example. It shows that there is no program that can detect whether any program will halt on any input. (If the halting program is computable then the program that uses it as a subroutine is also computable, meaning that we can't drop Rebellion and retain Prediction in response to it). The grandfather paradox, on the other hand, is generally taken to be an argument against its variant of Rebellion: there's an important sense in which you can't travel back in time and kill your own grandfather.

Denying Co-implementation is less common. This is because there is often no independent reason for thinking that g and f can never be co-implemented. And the argument shows that there is no instance in which f and g could ever be co-implemented, which remains true even if no one ever actually attempts to do so. Most of us would conclude from this that the processes cannot be co-implemented. (One could, in the spirit of compatiblism, argue that all we have shown is that f and g are never co-implemented and not that they cannot be co-implemented, but I assume most would reject this view.)

In the case of the prediction machine, we can deny that it's possible for Bob to act in a way that's contrary to the predictions that are made about him. This might be defensible in the case of self-prediction: if Bob cannot prevent himself from forming an accurate prediction about what he will do between the time that he forms the intention to act and the time that he acts, then he will never be able to rebel against his own predictions. But it is much less plausible in cases where Bob is responding to the predictions of others.

Alternatively, we could try to argue that Bob and the prediction machine will simply never communicate: perhaps every time the researchers try to run this experiment the machine will break down or spit out nonsense, for example. But this response is unsatisfactory for the reasons outlined above.

Finally, we could simply embrace DLP and concede that we cannot always produce accurate predictions about what agents like Bob will do, even if we have access to all of the relevant information about Bob and the state he is in. Embracing DLP might seem like a bad option, but the states we've identified in which we can't make accurate predictions about agents are states in which our predictions causally affect the very thing that we are attempting to predict. It might not be surprising if it's often impossible to make accurate predictions in cases where our predictions play this kind of causal role.

Conclusion: It seems like DLP could be true but, if it is, it might not be something that should concern us too much.