Biological substrates of reward and aversion: a nucleus accumbens activity hypothesis (2009)

A. Pharmacological evidence

It is well established that drugs of abuse (Di Chiara and Imperato, 1988) and natural rewards (Fibiger et al., 1992; Pfaus, 1999; Kelley, 2004) have the common action of elevating extracellular concentrations of dopamine in the NAc. Moreover, lesions of the NAc reduce the rewarding effects of stimulants and opiates (Roberts et al., 1980; Kelsey et al., 1989). Pharmacology studies in rats (e.g., Caine et al., 1999) and monkeys (e.g., Caine et al., 2000) suggest that D2-like receptor function plays a critical role in reward. However, it has been studies involving the direct microinfusion of drugs into this area that have provided the strongest evidence for its role in rewarding states. For example, rats will self-administer the dopamine releasing agent amphetamine directly into the NAc (Hoebel et al., 1983), demonstrating the reinforcing effects elevating extracellular dopamine in this region. Rats will also self-administer the dopamine reuptake inhibitor cocaine into the NAc, although this effect is surprisingly weak in comparison to that reported with amphetamine (Carlezon et al., 1995). This observation has led to speculation that the rewarding effects of cocaine are mediated outside the NAc, in areas including the olfactory tubercle (Ikemoto, 2003). However, rats will avidly self-administer the dopamine reuptake inhibitor nomifensine into the NAc (Carlezon et al., 1995), suggesting that the local anesthetic properties of cocaine complicate studies in which the drug is applied directly to neurons. Co-infusion of the dopamine D2-selective antagonist sulpiride attenuates intracranial self-administration of nomifensine, demonstrating a key role for D2-like receptors in the rewarding effects intra-NAc microinfusions of this drug. When considered together with evidence from a variety of other studies (for review, see Rompré and Wise, 1989), these studies are entirely consistent with theories prevailing in the 1980’s that dopamine actions in the NAc play a necessary and sufficient role in reward and motivation.

While there is little controversy that dopamine actions in the NAc is sufficient for reward, other work began to challenge the notion that they are necessary. For example, rats will self-administer morphine directly into the NAc (Olds, 1982), away from the trigger zone (the VTA) in which the drug acts to elevate extracellular dopamine in the NAc (Leone et al., 1991; Johnson and North, 1992). Considering that μ- and δ-opioid receptors are located directly on NAc MSNs (Mansour et al., 1995), these data were the first to suggest that reward can be triggered by events occurring in parallel with (or downstream of) those triggered by dopamine. Rats will also self-administer phencyclidine (PCP), a complex drug that is a dopamine reuptake inhibitor and a non-competitive NMDA antagonist, directly into the NAc (Carlezon and Wise, 1996). Two lines of evidence suggest that this effect is not dopamine-dependent. First, intracranial self-administration of PCP is not affected by co-infusion of the dopamine D2-selective antagonist sulpiride; and second, rats will self-administer other non-competitive (MK-801) or competitive (CPP) NMDA antagonists with no direct effects on dopamine systems directly into the NAc (Carlezon and Wise, 1996). These data provided early evidence that blockade of NMDA receptors in the NAc is sufficient for reward and, by extension, reward can be dopamine-independent. Blockade of NMDA receptors would be expected to produce an overall reduction in the excitability of NAc MSNs without affecting baseline excitatory input mediated by AMPA receptors (Uchimura et al., 1989; Pennartz et al. 1990). Importantly, rats also self-administered NMDA antagonists into deep layers of the PFC (Carlezon and Wise, 1996), which project directly to the NAc (see Kelley, 2004) and have been conceptualized as a part of a inhibitory (“STOP!”) motivational circuit (Childress, 2006). When considered together, these studies provided two critical pieces of evidence that have played a prominent role in the formulation of our current working hypothesis: first, that dopamine-dependent reward is attenuated by blockade of D2-like receptors, which are inhibitory receptors expressed predominately in the NAc on the MSNs of the indirect pathway; and second, that events that would be expected to reduce the overall excitability of the NAc (e.g., stimulation of G_i-coupled opioid receptors, reduced stimulation of excitatory NMDA receptors, reduced excitatory input) are sufficient for reward. This interpretation led to the development of a model of reward in which the critical event is reduced activation of MSNs in the NAc (Carlezon and Wise, 1996).

Other pharmacological evidence supports this theory, and implicates calcium (Ca2+) and its second messenger functions. Activated NMDA receptors gate Ca2+, an intracellular signaling molecule that can affect membrane depolarization, neurotransmitter release, signal transduction, and gene regulation (see Carlezon and Nestler, 2002; Carlezon et al., 2005). Microinjection of the L-type Ca2+ antagonist diltiazem directly into the NAc increases the rewarding effects of cocaine (Chartoff et al., 2006). The mechanisms by which diltiazem-induced alterations in Ca2+ influx affect reward are unknown. One possibility is that blockade of Ca2+ influx through voltage-operated L-type channels reduces the firing rate of neurons within the ventral NAc (Cooper and White, 2000). It is important to note, however, that diltiazem alone was not rewarding, at least at the doses tested in these studies. This might indicate that baseline levels of Ca2+ influx via L-type channels within the NAc are normally low, and difficult to reduce further. A related possibility is that microinjection of diltiazem reduces aversive actions of cocaine that are mediated within the NAc, unmasking reward. For example, activity of the transcription factor cAMP response element binding protein (CREB) within the NAc is associated with aversive states and reductions in cocaine reward (Pliakas et al., 2001; Nestler and Carlezon, 2006). The activation of CREB depends on phosphorylation, which can occur via activation of L-type Ca2+ channels (Rajadhyaksha et al., 1999). Phosphorylated CREB can induce expression of dynorphin, a neuropeptide that might contribute to aversive states via activation of κ-opioid receptors in the NAc (for review, see Carlezon et al., 2005). The potential role of intra-NAc Ca2+ in regulating rewarding and aversive states is a common theme in our work that will be explained in greater detail below.

B. Molecular evidence

Mice lacking dopamine D2-like receptors have reduced sensitivity to the rewarding effects of cocaine (Welter et al., 2007). Ablation of D2-like receptors also reduces the rewarding effects of morphine (Maldonado et al., 1997)—presumably by reducing the ability of the drug to stimulate dopamine via VTA mechanisms: Leone et al., 1991; Johnson and North, 1992)—and lateral hypothalamic brain stimulation (Elmer et al., 2005). One interpretation of these findings is that loss of D2-like receptors in the NAc reduces the ability of dopamine to inhibit the indirect pathway, a putative mechanism of reward. These findings, when combined with evidence that human addicts have reduced dopamine D2-like receptor binding in the NAc, suggest that this receptor plays an essential role in encoding reward (Volkow et al., 2007).

Other advances in molecular biology have enabled the detection of neuroadaptative responses to drugs of abuse and the ability to mimic such changes in discrete brain areas to examine their significance. One such change is in the expression of AMPA-type glutamate receptors, which are expressed ubiquitously in the brain and composed of various combinations of the receptor subunits GluR1-4 (Hollmann et al., 1991; Malinow and Malenka, 2002). Drugs of abuse can alter GluR expression in the NAc. For example, repeated intermittent exposure to cocaine elevates GluR1 expression in the NAc (Churchill et al., 1999). Furthermore, GluR2 expression is elevated in the NAc of mice engineered to express ΔFosB, a neuroadaptation linked with increased sensitivity to drugs of abuse (Kelz et al., 1999). Studies in which viral vectors were used to elevate GluR1 selectively in the NAc indicate that this neuroadaptation tends to make cocaine aversive in place conditioning tests, whereas elevated GluR2 in the NAc increases cocaine reward (Kelz et al., 1999). Potential explanations for this pattern of findings likely involve Ca2+ and its effect on neuronal activity and intracellular signaling. Increased GluR1 expression favors formation of GluR1-homomeric (or GluR1-GluR3 heteromeric) AMPARs, which are Ca2+-permeable (Hollman et al., 1991; Malinow and Malenka, 2002). In contrast, GluR2 contains a motif that prevents Ca2+ influx; thus increased expression of GluR2 would favor formation of GluR2-containing Ca2+-impermeable AMPARs (and theoretically decrease the number of Ca2+-permeable AMPARs). Thus GluR2-containing AMPARs have physiological properties that render them functionally distinct from those lacking this subunit, particularly with respect to their interactions with Ca2+ (Fig. 1).

Fig. 1

Schematic illustrating the subunit composition of AMPA (glutamate) receptors. For simplicity, the receptors are depicted with 2 subunits. GluR2 contains a motif that blocks Ca2+ flux through the receptor, and thus heteromeric receptors that contain at …

These early studies involved place conditioning studies, which generally require repeated exposure to drugs of abuse and presumably involve cycles of reward and aversion (withdrawal). More recent studies examined how alterations in GluR expression modeling those acquired through repeated drug exposure affect intracranial self-stimulation (ICSS), an operant task in which the magnitude of the reinforcer (brain stimulation reward) is precisely controlled (Wise, 1996). Elevated expression of GluR1 in NAc shell increases ICSS thresholds, whereas elevated GluR2 decreases them (Todtenkopf et al., 2006). The effect of GluR2 on ICSS is qualitatively similar to that caused by drugs of abuse (Wise, 1996), suggesting that it reflects increases in the rewarding impact of the stimulation. In contrast, the effect of GluR1 is qualitatively similar to that caused by prodepressive treatments including drug withdrawal (Markou et al., 1992) and κ-opioid receptor agonists (Pfeiffer et al., 1986; Wadenberg, 2003; Todtenkopf et al., 2004; Carlezon et al., 2006), suggesting that it reflects decreases in the rewarding impact of the stimulation. These findings indicate that elevated expression of GluR1 and GluR2 in NAc shell have markedly different consequences on motivated behavior. Moreover, they confirm previous observations that elevated GluR1 and GluR2 expression in NAc shell have opposite effects in cocaine place conditioning studies (Kelz et al., 1999), and extend the generalizability of these effects to behaviors that are not motivated by drugs of abuse. Perhaps most importantly, they provide more evidence to implicate Ca2+ flux within the NAc in reduced reward or elevated aversion. Because Ca2+ plays a role in both neuronal depolarization and gene regulation, alterations in GluR expression and AMPAR subunit composition in NAc shell likely initiate physiological and molecular responses, which presumably interact to alter motivation. Again, the mechanisms by which Ca2+ signal transduction might trigger genes involved in aversive states are described in detail below.

A. Pharmacological evidence

Some of the earliest evidence that NAc plays a role in aversive states came from studies involving opioid receptor antagonists. Microinjections of a wide-spectrum opioid receptor antagonist (methylnaloxonium) into the NAc of opiate-dependent rats establishes conditioned place aversions (Stinus et al., 1990). In opiate-dependent rats, precipitated withdrawal can induce immediate-early genes and transcription factors in the NAc (Gracy et al., 2001; Chartoff et al., 2006), suggesting activation of MSNs. Selective κ-opioid agonists, which mimic the effects of the endogenous κ-opioid ligand dynorphin, also produce aversive states. Microinjections of a κ-opioid agonist into the NAc cause conditioned place aversions (Bals-Kubik et al., 1993) and elevate ICSS thresholds (Chen et al., 2008). Inhibitory (G_i-coupled) κ-opioid receptors are localized on the terminals of VTA dopamine inputs to the NAc (Svingos et al., 1999), where they regulate local dopamine release. As such, they are often in apposition to μ- and δ-opioid receptors (Mansour et al., 1995), and stimulation produces the opposite effects of agonists at these othr receptors in behavioral tests. Indeed, extracellular concentrations of dopamine are reduced in the NAc by systemic (DiChiara and Imperato, 1988; Carlezon et al., 2006) or local microinfusions of κ-opioid agonist (Donzati et al., 1992; Spanagel et al., 1992). Decreased function of midbrain dopamine systems has been associated with depressive states including anhedonia in rodents (Wise, 1982) and dysphoria in humans (Mizrahi et al., 2007). Thus one path to aversion appears to be reduced dopamine input to the NAc, which would reduced the stimulation of inhibitory dopamine D2-like receptors that seem critical for reward (Carlezon and Wise, 1996).

Other studies appear to confirm an important role of dopamine D2-like receptors in suppressing aversive responses. Microinjections of a dopamine D2-like antagonist into the NAc of opiate-dependent rats precipitates signs of somatic opiate withdrawal (Harris and Aston-Jones, 1994). Although the motivational effects were not measured in this study, treatments that precipitate opiate withdrawal often cause aversive states more potently than they cause somatic signs of withdrawal (Gracy et al., 2001; Chartoff et al., 2006). Interestingly, however, microinjections of a dopamine D1-like agonist into the NAc also produce somatic signs of withdrawal in opiate–dependent rats. The data demonstrate that another path to aversion is increased stimulation of excitatory dopamine D1-like receptors in rats with opiate-dependence induced neuroadaptations in the NAc. Perhaps not surprisingly, one consequence of D1-like receptor stimulation in opiate dependent rats is phosphorylation of GluR1 (Chartoff et al., 2006), which would lead to increased surface expression of AMPA receptors on the MSNs of the direct pathway.

B. Molecular evidence

Exposure to drugs of abuse (Turgeon et al., 1997) and stress (Pliakas et al., 2001) activate the transcription factor CREB in the NAc. Viral vector-induced elevation of CREB function in the NAc reduces the rewarding effects of drugs (Carlezon et al., 1998) and hypothalamic brain stimulation (Parsegian et al., 2006), indicating anhedonia-like effects. It also makes low doses of cocaine aversive (a putative sign of dysphoria), and increases immobility behavior in the forced swim test (a putative sign of “behavioral despair”) (Pliakas et al., 2001). Many of these effects can be attributed to CREB-regulated increases in dynorphin function (Carlezon et al., 1998). Indeed, κ-opioid receptor-selective agonists have effects that are qualitatively similar to those produced by elevated CREB function in the NAc, producing signs of anhedonia and dysphoria in reward models and increased immobility in the forced swim test (Bals-Kubik et al., 1993; Carlezon et al., 1998; Pliakas et al., 2001; Mague et al., 2003; Carlezon et al., 2006). In contrast, κ-selective antagonists produce an antidepressant-like phenotype resembling that seen in animals with disrupted CREB function in the NAc (Pliakas et al., 2001; Newton et al., 2002; Mague et al., 2003). These findings suggest that one biologically important consequence of drug- or stress-induced activation of CREB within the NAc is increased transcription of dynorphin, which triggers key signs of depression. Dynorphin effects are likely mediated via stimulation of κ-opioid receptors that act to inhibit neurotransmitter release from mesolimbic dopamine neurons, thereby reducing the activity VTA neurons, as explained above. This path to aversion appears to be reduced dopamine input to the NAc, which would produce reductions in the stimulation of inhibitory dopamine D2-like receptors that seem critical for reward (Carlezon and Wise, 1996). As explained below, there is also evidence that elevated expression of CREB in the NAc directly increases the excitability of MSNs (Dong et al., 2006) in addition to the loss of D2-regluated inhibition, raising the possibility that multiple effects contribute to the aversive responses.

Repeated exposure to drugs of abuse can elevate GluR1 expression in the NAc (Churchill et al., 1999). Viral vector-induced elevation of elevated GluR1 in the NAc increases drug aversion in place conditioning studies, an “atypical” type of drug sensitization (i.e., heightened sensitivity to the aversive rather than the rewarding aspects of cocaine). This treatment also increases ICSS thresholds (Todtenkopf et al., 2006), indicating anhedonia-like and dysphoria-like effects. Interestingly, these motivational effects are virtually identical to those caused by elevated CREB function in the NAc. These similarities raise the possibility that both effects are part of the same larger process. In one possible scenario, drug exposure might trigger changes in the expression of GluR1 in the NAc, which would lead to local increases in the surface expression of Ca2+-permeable AMPA receptors, which would increase Ca2+ influx and activate CREB, leading to alterations in sodium channel expression that affect baseline and stimulated excitability of MSNs in the NAc (Carlezon and Nestler, 2002; Carlezon et al., 2005; Dong et al., 2006). Alternatively, early changes in CREB function might precede alterations in GluR1 expression. These relationships are currently under intensive study in several NIDA-funded laboratories, including our own.

A. Testing the hypothesis with electrophysiology

One caveat to the inhibition/reward hypothesis is that widespread and prolonged inhibition of NAc firing, as in inactivation or lesion studies, does not appear to produce rewarding effects (e.g. Yun et al., 2004b). This raises the possibility that it is not inhibition of the NAc, per se, that encodes reward but rather the transitions from normal basal firing rates to lower rates that occur when rewarding stimuli are present. Prolonged inhibition may degrade the dynamic information normally encoded in the transient depressions of NAc firing.

Electrophysiology-based tests of the predictions of this hypothesis fall into two basic categories. The first category involves manipulating an animal’s behavioral state to produce sustained changes in responsivity to rewarding stimuli followed by testing for electrophysiological correlates of this altered reward state. For example, the early withdrawal state from chronic exposure to psychostimulants is characterized by anhedonia and lack of responsiveness to natural rewarding stimuli. What would the inhibition/reward hypothesis predict about the electrophysiological status of NAc neurons during this state? The major prediction is that NAc neurons would exhibit decreases in the activity suppression normally produced by exposure to a rewarding stimulus (e.g. sucrose). To our knowledge, this has not yet been investigated. Possible mechanisms for such a decrease in inhibition, should it occur, might include overall increases in neuronal excitability produced by any combination of changes in intrinsic excitability (e.g. increased Na+ or Ca2+ currents, decreased K+ currents) or synaptic transmission (e.g. decreases in glutamatergic or increases in GABAergic transmission). On the other hand, the available data on NAc MSN excitability during early psychostimulant withdrawal suggest that it is actually decreased during this phase (Zhang et al., 1998; Hu et al., 2004; Dong et al., 2006; Kourrich et al., 2007). As noted above, it is possible that a prolonged depression in excitability may degrade reward-related information contained in transient firing inhibitions, perhaps by creating a “floor” effect and reducing the magnitude of these inhibitions. This possibility remains to be tested.

Considering the apparent link between NAc and ventral pallidum in reward encoding (see above), we would predict that any excitability changes produced by sustained modulation of an animal’s reward state might be particularly evident in striatopallidal/D2 neurons. Although studying the detailed physiological properties of these neurons has been difficult in the past, the recent development of a line of BAC transgenic mice that expresses GFP in these neurons (Gong et al., 2003; Lobo et al., 2006) has made it possible to visualize them in in vitro slice preparations, greatly facilitating the potential for physiological characterization of D2 cells.

The second category of electrophysiology-based tests involves using genetic engineering (see below) to alter the functional expression of key components of the cellular machinery for excitability or excitability modulation in NAc neurons. In theory, this could enable modulation of the inhibitions or excitations associated with reward or aversion, respectively, in NAc neurons. With this in mind, perhaps the most useful target molecules would be those that participate in activity-dependent modulation of neuronal excitability, rather than in maintaining basal firing rates. These targets would likely provide a better opportunity to modulating stimuli responsiveness than more general targets (e.g. Na+ channel subunits), thus enabling the evaluation of the inhibition/reward hypothesis. For example, the firing frequency of active neurons can be controlled by various ionic conductances that produce spike after-hyperpolarizations (AHPs). By targeting NAc neurons with genetic (or possibly even pharmacologic) manipulation aimed at the channels that produce AHPs, it may be possible to decrease the magnitude of aversion-related excitatory responses in these neurons and thus to test whether this physiological change correlates with reduced behavioral indices of aversion.

B. Testing the hypothesis with behavioral pharmacology

One of the most obvious pharmacological tests would to determine if rats self-administer dopamine D2-like agonists directly into the NAc. Interestingly, previous work indicates that while rats self-administer combinations of D1-like and D2-like agonists into the NAc, they do not self-administer either drug component alone, at least at the doses tested (Ikemoto et al., 1997). While on the surface this finding might appear to invalidate our working hypothesis, electrophysiological evidence suggests that co-activation of D1 and D2 receptors on NAc neurons can, under some conditions, cause a reduction in their membrane excitability that is not seen in response to either agonist alone (O’Donnell and Grace, 1996). In addition, more work is needed to study the behavioral effects of intra-NAc microinfusions of GABA agonists; historically, this work has been hindered by poor solubility of benzodiazepines—which are known to be addictive (Griffiths and Ator, 1980) despite their tendency to decrease dopamine function in the NAc (Wood, 1982; Finlay et al., 1992: Murai et al., 1994)—and the relatively small number of researchers who use brain microinjection procedures together with models of reward. Still other ways of testing our hypothesis would be to study the effects of manipulations in brain areas downstream of D2 receptor-containing MSNs. Again, early evidence suggests reward is encoded by activation of the ventral pallidum, a presumed consequence of inhibition of the MSNs of the indirect pathway (Tindell et al., 2006).

C. Testing the hypothesis with genetic engineering

The development of genetic engineering techniques that enable the direction of inducible or conditional mutations to specific brain areas will be an important tool with which to test our hypotheses. Mice with constitutive deletion of GluRA (an alternative nomenclature for GluR1) show many alterations in sensitivity to drugs of abuse (Vekovischeva et al., 2001; Dong et al., 2004; Mead et al., 2005, 2007), some of which are consistent with our working hypothesis and some of which are not. The loss of GluR1 early in development could dramatically alter responsiveness to numerous types of stimuli, including drugs of abuse. In addition, these GluR1-mutant mice lack the protein throughout the brain, whereas the research reviewed here focuses on mechanisms that occur within NAc. These points are especially important because loss of GluR1 in other brain regions would be expected to have dramatic, and sometimes very different, effects on drug abuse-related behaviors. As just one example, we have shown that modulation of GluR1 function in the VTA exerts the opposite effect on drug responses compared to modulation of GluR1 in the NAbc (Carlezon et al., 1997; Kelz et al., 1999). The findings in GluR1-deficient mice are not inconsistent with the combined findings from the NAc and the VTA: constitutive GluR1 mutant mice are more sensitive to the stimulant effects of morphine (an effect that could be explained by the loss of GluR1 in the NAc), but they do not develop progressive increases in responsivity to morphine (an effect that could be explained by the loss of GluR1 in the VTA) testing occurs under conditions that promote sensitization and involve additional brain regions. Accordingly, one must be cautious in assigning spatial and temporal interpretations to data from constitutive knockout mice: the literature is becoming replete with examples of proteins that have dramatically different (and sometimes opposite) effects on behavior depending upon the brain regions under study (see Carlezon et al., 2005).

Preliminary studies from mice with inducible expression of a dominant-negative form of CREB—a manipulation which reduces the excitability of NAc MSNs—are hypersensitive to the rewarding effects of cocaine while being insensitive to the aversive effects of a κ-opioid agonist (DiNieri et al., 2006). Although these findings are consistent with our working hypothesis, further studies (e.g., electrophysiology) might help to characterize the physiological basis of these effects. Regardless, an increased capacity to spatially and temporally control the expression of genes that regulate the excitability of NAc MSNs will enable progressively more sophisticated tests of our working hypothesis.

Funded by the National Institute on Drug Abuse (NIDA) grants DA012736 (to WAC) and DA019666 (to MJT) and a McKnight-Land Grant professorship (to MJT). We thank M.J. Kaufman, B. deB. Fredrick, and S.S. Negus for permission to cite unpublished data from their brain imaging studies in monkeys.