Predictability modulates human brain response to reward (2001)

RESULTS

After the scans, subjects were queried about their preference for the two stimuli. Eighteen of 25 subjects (72%) preferred juice, and the remainder preferred water. Most subjects had a distinct preference for one or the other, although we did not ask them to quantify this. Although there was significant head motion during the scans, all of the translations and rotations around each stimulus were generally small and were not significantly different between any of the conditions. For example, the mean ± SD translation associated with each stimulus was 0.041 ± 0.069 mm in the predictable condition and 0.044 ± 0.069 mm in the unpredictable condition (paired t test;p = 0.853).

The brain response to the preferred fluid displayed surprisingly little differential activity relative to the nonpreferred fluid (Table1). We did not observe any significant activity difference in classical reward regions such as the nucleus accumbens, hippocampus, or medial prefrontal cortex. The primary activity change for preferred > nonpreferred occurred in the somatosensory cortex in an area near the mouth and tongue region (t = 4.19, MNI coordinates, −60, −12, 16).

The main effect of predictability was substantially greater than the main effect of preference (Fig. 3). For the unpredictable run relative to the predictable run, bilateral activation was observed in a large expanse of medial orbitofrontal cortex that included the nucleus accumbens (Table 1). Additional areas of activation included a large area of parietal cortex bilaterally and paracentrally and small focal activations in both the left mediodorsal nucleus of the thalamus and right cerebellum. Because none of these regions overlapped with the main effect of preference, they were maximally activated by unpredictable stimuli, regardless of preference. For the predictable run relative to the unpredictable run, an area of the right superior temporal gyrus was activated, as well as focal activations in the left precentral gyrus and right lateral orbitofrontal cortex.

  

View larger version:  

• In this page
• In a new window

• Download as PowerPoint Slide

Fig. 3.  

The main effect of predictability showed that reward-related regions had a greater BOLD response to the unpredictable stimuli. A, Planes centered at (0, 4, −4) show that bilateral nucleus accumbens/ventral striatum (NAC) and bilateral superior parietal cortex were more active in the predictable condition. B, A small region in the right superior temporal gyrus was relatively more activated by the predictable stimuli. Significance was thresholded atp < 0.001 and an extent >10 contiguous voxels.

The interaction between preference and predictability identified areas in which one effect modulated the other independently of both main effects. The right insula, left posterior cingulate, and right cerebellum displayed a significant interaction for the contrast (preferred–nonpreferred) × (predictable–unpredictable). The opposite contrast, (preferred–nonpreferred) × (unpredictable–predictable), did not reveal any activations significant at the p < 0.001 level; however, a small region in the left superior temporal gyrus (MNI coordinates, −48, −4, −16) was significant at the p < 0.01 level (t = 3.15).

The computer simulation suggested that unpredictable rewards should evoke more dopamine release than predictable ones (Fig.2 B). When the rewards are predictable, each stimulus perfectly predicts the subsequent one, and the error signal, which is presumed to be mediated by dopamine, gradually decreases. When the rewards are unpredictable, there is no opportunity for the system to learn, and the response to each stimulus is greater.

DISCUSSION

Our results demonstrated an interesting separation in the brain response to predictability and to subjective reports of preference. The brain response to preference was exclusively cortical, but the response to predictability showed specific activation of reward systems also known to be the target of midbrain dopaminergic neurons. If we presume that activation of these reward areas is pleasurable to humans, then this finding suggests that the subjective report of preference may be dissociated from neural circuits known to be powerful determinants of conditioned behaviors.

Both the water and the fruit juice caused significant activations throughout the brain, and although some of this response was attributable to the motoric aspects of the task, specific subsets of these regions were decomposed into dimensions of preference and predictability. The effect of preference was restricted to cortical regions associated with sensory processing, and the preferred stimulus resulted in greater activation in these regions. These regions lie near sensorimotor cortex known to be activated during tongue movements (Corfield et al., 1999) and swallowing (Hamdy et al., 1999). In previous work on the brain response to tongue movement, there was substantial activation of the cerebellum, a finding notably absent in the main effect of preference. The differential brain response, i.e., preferred–nonpreferred, removes common regions of activation; therefore, the absence of cerebellar activation suggests that differential tongue movements were unlikely to be the cause of the pattern of cortical activation for subjective preference. The fact that a somatosensory region was correlated with stated preference was suggestive that some differential neural processing occurred for the two stimuli. It was surprising that this was manifest in a primary sensory processing area and not in classical reward areas. Although subjects were forced to designate one substance over the other as their preference, both fluids were chosen purposely to be pleasurable, in contradistinction to one being aversive. Because both fluids were generally pleasurable, the effect of preference might not have been strong enough to result in a significant activity difference in reward regions. This would be consistent with findings that midbrain dopamine neurons are preferentially activated by appetitive rather than aversive stimuli (Mirenowicz and Schultz, 1996). Nevertheless, our findings suggest a system differentiation of subjective preference from simple reward, which supports previous hypotheses that “wanting” is not the same as “liking” (Robinson and Berridge, 1993).

Unlike the effect of preference, unpredictability correlated as a significant main effect with activity in the nucleus accumbens, thalamus, and medial orbitofrontal cortex, whereas predictability was correlated predominately with activity in the right superior temporal gyrus. The former regions correspond closely with known dopamine projection sites (Koob, 1992; Cooper et al., 1996). It was somewhat surprising that unpredictability, and not preference, was correlated with activity in these reward areas. If increased activity in these regions was associated with pleasure, then one might conclude that unpredictable rewards were more pleasurable than predictable ones. However, most of the subjects did not discern any difference between the predictable and unpredictable conditions. If the unpredictable rewards were more pleasurable than predictable ones, or vice versa, then this must be occurring at a subconscious level. An alternative explanation presumes that dopamine is released in increased amounts to unexpected rewards (Montague et al., 1996; Schultz et al., 1997;Schultz, 1998). Dopamine can decrease neuronal excitability (Cooper et al., 1996) and may also directly constrict the microvasculature (Krimer et al., 1998), but increased accumbens activity has also been associated with the subjective pleasure of cocaine (Breiter et al., 1997). These findings suggest that our observed increase in activation with unpredictability could be related to increased dopamine release, either because the accumbens projects to the VTA or because it receives a projection from the VTA, either of which would be consistent with the model results. This interpretation should be tempered by two important facts: (1) the mechanisms that would couple increased dopaminergic transmission to changes in the BOLD signal are unknown, and (2) we have no independent measure of dopaminergic transmission, only changes in the BOLD response. The possibility that we are observing indirectly changes in dopaminergic activity is exciting but cannot be decided unequivocally in an fMRI experiment. It is, however, consistent with previous findings using positron emission tomography that dopamine is released into the ventral striatum under conditions of monetary incentive (Koepp et al., 1998). Coupled with the amplifying effect of unpredictability, it is also consistent with hypothesized effects of dopamine on neuronal “gain” (Cohen and Servan-Schreiber, 1992), with the end result that some regions will increase and others will decrease.

The specific regions activated relatively by unpredictability corresponded to brain regions associated with appetitive functions. In addition to the nucleus accumbens, the medial orbitofrontal cortex showed a main effect for unpredictability. This region has been shown in primates to integrate both the rewarding and neutral aspects of taste sensations and is thought to reflect primarily the motivational values of these stimuli (Rolls, 2000). This region also contains neurons that discriminate relative preference for reward (Tremblay and Schultz, 1999). The orbitofrontal cortex is typically difficult to image with fMRI because of the susceptibility artifact from the nasal sinuses (Ojemann et al., 1997). However, the region that we identified is generally superior and caudal to the usual artifact location. This region has previously been found responsive to pleasant tastes (Francis et al., 1999). A second region, in the superior parietal lobe, was probably not related to the rewarding aspects of the task but rather the result of changes in attention. This region has previously been implicated in visuospatial attention, especially during expectation violations (Nobre et al., 1999). Another region, in the left temporal cortex, showed a borderline significant modulation by unpredictability. In recent fMRI experiments, the left temporal lobe has been associated with processing the predictability of sequential stimuli (Bischoff-Grethe et al., 2000). Here, we extend these previous findings from neutral stimuli to pleasurable stimuli, suggesting that this region may perform a generic monitoring of predictability independently of stimulus valence.

The brain regions that we identified as responding to unpredictability in either a direct or modulatory manner have been implicated in a number of experiments on financial reward. Money can be rewarding to humans, but it is reinforcing only because it has acquired these properties through complex conditioning. Similar to the finding that cocaine acts on different neurons than natural reinforcers (Carelli et al., 2000), it is possible that conditioned reinforcers, such as money, act on different neural systems than natural reinforcers such as food and water. Activity in both the ventral striatum and midbrain have been correlated with absolute levels of financial reward (Thut et al., 1997;Delgado et al., 2000; Elliott et al., 2000; Knutson et al., 2000), a finding notably absent in our results. As noted previously, both the juice and water were mildly pleasurable, and so there may not have been a substantial difference in absolute reward, although we assumed a slight difference in the theoretical model. Also, we did not use any aversive stimuli or anything that could be construed as a negative reward, which may also account for this difference. Interestingly, the regions we identified as being either directly affected or amplified by unpredictability corresponded to the regions found previously to be sensitive to the context dependence of the financial reward (Rogers et al., 1999; Elliott et al., 2000). In particular, both the subgenual cingulate and medial thalamus were correlated with unpredictability in our study and were found to be context-dependent by Elliott et al. (2000).

Because predictability modulated the effect of preference, it is important to distinguish the potential sources of prediction. In a classical conditioning experiment, a neutral stimulus precedes the reward. After training, the previously neutral stimulus becomes the predictor, or conditioned stimulus. Because there are comparatively few data on the use of oral stimuli in fMRI, we chose to simplify the experiment and control for the motoric aspects of the task by using two different oral stimuli, water and fruit juice. Thus, the source of prediction in our experiment necessarily came from the sequence of stimuli themselves. In some ways, this is simpler than introducing another stimulus modality, such as a visual cue, but because both stimuli were rewarding, we cannot make any conclusions regarding the process of conditioning. Both the theoretical model (Schultz et al., 1997) and neurophysiological data (Schultz et al., 1992, 1993) suggest that reward predictions are computed during the interval preceding reward delivery. Because we do not know the time scale over which such predictions are computed, we chose to analyze the experiment as simply two conditions, predictable and unpredictable. By maintaining a psychologically reasonable interval between stimuli, 10 sec, there was insufficient time to resolve differences in interstimulus processing. Presumably such processing does occur, and this could be resolved with a differently designed experiment.

In summary, activity in human reward regions can be modulated by the temporal predictability of primary rewards such as water and juice. These results provide important support for computational models that postulate that errors in reward prediction can drive synaptic modification and extend these conclusions from nonhuman primates to humans. The regional specificity of this modulation also suggests that information, as embodied by the relative predictability of a stimulus stream, may be a form of neural currency that can be detected with fMRI.