The clinical relevance of neuroplasticity in corticostriatal networks during operant learning (2013)

Go to:

Abstract

Operant learning is one of the most elementary forms of behavioral adaptation (Rescorla, 1994). Through interchange with its environment, an animal is able to learn about the consequences of its actions, and thereby modify the current environment through new behaviors to produce more favorable conditions (Skinner, 1953). The resultant change in behavior is dramatic and long-lasting. Some scholars have argued that operant learning is the basis of “knowledge” (Schnaitter, 1987), may underlie “creativity” (Pryor et al., 1969), is the basis of decision-making, and contributes to the intractable nature of drug addiction. As the behavior of an organism is altered by response-outcome contingencies, physiological mechanisms are activated which ensure that these alterations become nearly permanent; they are “stamped in,” as Thorndike hypothesized (Thorndike, 1911). Even Skinner intimated that response-outcome contingences change us: “Men act upon the world, and change it, and are changed in turn by the consequences of their action.” (Skinner, 1957, p. 1).

In light of the ubiquity of operant behavioral relations in our psychological lives, the neurobiology of operant learning (i.e., the initial acquisition of an operant response) has received surprisingly little attention when compared to other basic learning processes such as spatial learning (e.g., Morris Water Maze) or Pavlovian fear conditioning. Yet, operant relations are thought to be at work nearly every moment of our lives and in many prominent neuropsychiatric conditions: drug abuse, autism, and other severe problem behaviors. In this review, we highlight the last two decades of Ann Kelley’s research career, when she pursued a greater understanding of the neurobiology of operant learning with the hope that the molecular, cellular, and genomic constituents of operant learning, instantiated in distributed networks, would inform better treatment alternatives.

Costly behavioral-health problems and Operant behavior

Drug abuse is one of the most damaging, recalcitrant and costly behavioral-health problems in the U.S., and indeed, the world. Abuse of drugs in this country alone costs an estimated $484 billion annually in health-related problems, accidents, lost work, and insurance premiums (Policy, 2001). It is also estimated that 540,000 people die each year from drug-related illnesses. These estimates do not include the non-monetary or indirect psychosocial costs paid by parents¹, spouses, siblings, friends, and our community in general. It is quite likely that every citizen in this nation has been adversely affected by drug abuse and addiction in some way (e.g., as the victim of criminal behavior, an automobile accident, or through the actions of a family member). Drug addiction is being increasingly viewed in terms of fundamental changes in cognitions and behaviors, with emphasis on relating the compulsive nature of addiction to pathological changes in decision- and emotion-coding networks (Everitt et al., 2001). Thus, a better understanding of operant learning systems may enhance our understanding of the neural causation of addiction.

According to the Centers for Disease Control (CDC), 1 in 88 children have been identified as having autism (Control, 2012). Autism spectrum disorders (ASDs) affect individuals from all ethnic backgrounds and socioeconomic levels. ASDs can prove profoundly debilitating and likely require life-long care at great expense to the community (>$3,000,000 per individual) (Ganz, 2007). More recently, applied behavior analysis (ABA) and certain derivatives (e.g., Denver Start Model), which emphasize dynamic and flexible academic, social, and communicative behavior, have demonstrated that incredible gains are possible with early, intensive therapy (Sallows and Graupner, 2005, Dawson et al., 2010, Warren et al., 2011). These models have been so successful that many children diagnosed with ASDs are later termed “indistinguishable” from their peers. Some estimate that 40-50% of children diagnosed with autism are fully remediable (McEachin et al., 1993). In addition, the overwhelming success of ABA therapy in the treatment of autism has lead to the general idea that it is synonymous with autism therapy (Dillenburger and Keenan, 2009), much to the displeasure of practitioners, to name a few, of organizational behavior management (OBM), clinical behavior analysis, and animal training; professions that use behavior analysis applied to situations not involving autism. Of interest here is the fact that most ABA principles are based on contemporary operant theory and the experimental analysis of behavior: evaluating possible establishing operations, identifying the consequential functions of inappropriate behavior, reinforcing good behavior, punishing unwanted behavior, and assessing these relations in a greater socio-economic context (e.g., behavioral economics). In their seminal piece on ABA, Baer, Wolf and Risley (1968) lay out a clear relationship between operant theory and the “conceptual systems” dimension of ABA, although a full review of that paper is beyond the purview of this current review. Thus, because the etiology of ASDs are largely viewed as neuro-genetic, and in light of the prominent role operant behavior plays in learning and therapy vis-à-vis ASDs, a greater understanding of the neurobiology of operant behavior might help our considerations of ASDs.

The term “severe problem behavior” encompasses a wide range of issues from school bullying to extreme self-injury. Severe problem behaviors can be displayed by typically-developing children, but are more prevalent in children with developmental and/or intellectual disabilities. Severe problem behaviors create substantial social and educational obstacles for individuals due to their intensity and seeming unpredictability. Treatment may involve suspensions from school, placement in special environments, engaging the criminal justice system, incarceration or institutionalization. Rather than considering these patterns as “maladaptive” or “inappropriate,” psychologists and educators are now viewing many of these problem behaviors as functional. In other words, when considered as operant behavior, the reinforcing contingencies promoting these severe behavior problems can be determined, assessed, and changed. Due to the dangerous nature of these problems and the intrusion of likely neurophysiological issues, however, many individuals spiral into difficult or untenable living conditions or circumstances with a lack of treatment. The possibility that these serious problems emerge through a combination of genetic-environment interactions is only now being seriously considered. A better understanding of the neurobiology of operant behavior would improve treatment alternatives.

Mechanisms of neural plasticity in long-lasting behavioral change

It is now well accepted that long-lasting behavioral modifications via operant contingencies are the result of significant changes in the brain: the strengthening of synaptic connections, re-configuring of neural ensembles, synthesis of new proteins, upregulation of gene expression, and epigenetic modifications. Long-term potentiation (LTP) has served as one of the most frequently interrogated plasticity-related systems and data strongly implicate NMDAR activation as a key initiating event. That is, high frequency patterns of synaptic stimulation activate NMDAR resulting in an influx of Ca²⁺, in turn activating multiple signaling mechanisms, several of which converge on ERK (Extracellular Receptor signaling Kinase). ERK is thought to regulate a variety of transcription factors that coordinate the formation and stabilization of long-term memories (Levenson et al., 2004). There exists substantial data confirming the role of the NMDAR-Ca²⁺-ERK cascade in long-lasting behavioral change and memory formation in fear conditioning and Morris Water Maze learning (Atkins et al., 1998, Blum et al., 1999, Schafe et al., 2000); a more recent report implicates this cascade in food-rewarded conditioning, as well, although in an invertebrate model (Ribeiro et al., 2005). NMDAR-induced neural plasticity, through transcriptional regulations via the ERK pathway, therefore, provides a neural representation of operant conditioning and an elegant model for studying long-lasting behavioral change.

In a direct extension of this model, Kelley and colleagues (Kelley et al., 1997) first explored the role of NMDAR activation in operant learning within the nucleus accumbens, a site hypothesized to play a major role in the complex integration of sensory, reward and motor information. Following habituation to standard operant conditioning chambers and magazine training, injections of the NMDAR antagonist (+/-)-2-amino-5-phosphonopentanoic acid (AP-5) were made directly into the nucleus accumbens core (NAc) of food-restricted rats immediately prior to the first four, 15-minute long, operant conditioning sessions. With a lever now inserted into the chamber, presses were reinforced with sucrose pellets². Across the first 4 training sessions, rats treated with AP-5 made very few lever-presses, in contrast to vehicle-treated rats. All rats were left untreated for the next 5 sessions and both groups quickly reached asymptotic levels of lever-pressing. Importantly, a microinjection of AP-5 into the NAc prior to a 10^th session did not have any discernible effects. Separate experiments found no effect of AP-5 on spontaneous, unconditional eating and motor behavior in identically-treated (e.g., surgery, deprivation, etc.) rats. Therefore, when compared to saline-infusions, AP-5 infusions/NMDAR blockade in the NAC impaired initial operant learning, but had no effect on subsequent performance, nor did NMDAR blockade affect motivation for sucrose or spontaneous motor behavior. Thus, these data appear consistent with the general consensus that NMDAR activation is crucial for learning via its role in neural plasticity.

These studies, conducted in Ann Kelley’s laboratory, are the first demonstrating a role for NMDA receptors in operant learning within a key node of a cortico-limbic-striatal network. Hernandez et al (Hernandez et al., 2005) directly replicated this effect, and, notably, demonstrated a time-limited contextual role for NMDAR activation in operant learning for post-session AP-5 infusions had no effect on learning. In other words, NMDAR activation during exposure to the chamber and operant contingencies was required for learning to occur but not necessary after the session. This finding contrasts with post-session drug effects on other behavioral preparations, such as fear conditioning (Castellano et al., 1993). Kelley et al. (Kelley et al., 1997) also showed that infusions of AP-5 into the nucleus accumbens shell (NAS) had very little effect on operant learning, suggesting that operant conditioning entails plastic changes in a discrete network rather than ubiquitous neural action of NMDARs. A more precise characterization of this network could benefit countless neuropsychiatric conditions that involve learning or plasticity-related deficiencies by helping neurobiologists identify discrete nuclei that are critical for carrying out behavior while simultaneously identify specific receptor mediation of said behavior.

To expand on these results, Baldwin et al. (2000) found that AP-5 infusions in the basolateral amygdala (BLA) and the medial prefrontal cortex (mPFC) also impaired operant learning, but AP-5 had no effect on operant learning when infused in the dorsal (dSUB) or ventral (vSUB) subiculum. Further, these effects were again limited to the initial conditioning phase as NMDAR blockade had no effect on subsequent operant performance, spontaneous motor behavior or spontaneous feeding. McKee et al. (McKee et al., 2010) extended the role of NMDAR activation in operant learning to the dorsal medial striatum (DMS) and anterior cingulate cortex (ACC), but found no role for the orbito-frontal cortex (OFC) in operant learning. Control studies found no evidence for motivational or motor deficits. Andrzejewski et al. (Andrzejewski et al., 2004) also explored the role of NMDARs in the central nucleus of the amygdala (CeA) and 2 other striatal subnuclei. While learning deficits were observed after AP-5 infusions into the CeA and posterior lateral striatum (PLS), but not the dorso lateral striatum (DLS), there were also profound effects on spontaneous motor and feeding behavior with AP-5 infusions in the CeA and PLS. These results suggest that operant learning depends upon NMDAR activation within a distributed network, each possibly contributing distinct sensory, motivational, motor, and learning processing. Certainly, future studies are needed to evaluate the limits of the “operant” network.

Together, these initial studies indicate that the NAC, BLA, mPFC, DMS and ACC are critical areas in a cortico-limbic striatal network controlling operant learning that is not needed for later performance. Although further work may clarify this network and perhaps more specific roles of each region, such a network appears to underlie the learning of addictive or maladaptive behaviors that may be more striatally regulated once established.

Dopamine involvement in reward processing and plasticity

Reinforcement-based processing also depends heavily on mesocorticolimbic DA systems, comprising DA neurons in the ventral tegmental area (VTA) and their projections to nucleus accumbens (NAc), amygdala, prefrontal cortex (PFC), and other forebrain regions, but the exact nature of the role of DA in reward processing is still a source of contention. One early theory suggested that DA-mediated the pleasures of reward because many natural and drug rewards activate mesocorticolimbic systems and their blockade impairs the behavioral effectiveness of most reinforcers (Wise and Bozarth, 1985). A second hypothesis contends that mesocorticolimbic DA neurons learn and predict reward deliveries, because they fire to appetitively-conditioned stimuli, but not to the unconditional stimuli (or to the rewards themselves) (Schultz, 1998, 2002). A third, very influential hypothesis, asserts that mesocorticolimbic DA systems encode incentive properties attributed to the neural representations of stimuli and rewards. Indeed, DA does not mediate the hedonic influence of sweet rewards, but is required for behavior directed towards the same rewards (Berridge and Robinson, 1998). Fourth, some have argued that mesocorticolimbic DA systems subserve effort-related functions that impact reinforced-behavior due to the fact that DA depletions have little impact on operant responding when reinforced on an “easy” schedule (an FR-5, for example), but have dramatic effects on more effortful schedules (Salamone et al., 1994, Salamone et al., 2001). Nevertheless, while DA’s role in operant behavior is unequivocal, the exact nature and details of its role likely remain a function of the preparation used and the theoretical orientation of the experimenter.

We tested the role of DA on operant learning via D1R activity in many of the same structures noted above. Baldwin et al. (Baldwin et al., 2002b) showed that D1R blockade in the PFC impaired operant learning but had no effect on performance. D1R blockade in the BLA and CeA also impaired operant learning (Andrzejewski et al., 2005), in a dose dependent fashion. However, the role of D1R in other structures has been difficult to dissociate from other D1R-mediated drug effects. For example, Hernandez et al (Hernandez et al., 2005) demonstrated a profound effect on operant behavior following pre-session D1R blockade in the NAc; however, nose-poking into the food tray (often considered a Pavlovian appetitively conditioned response) was also substantially reduced. Andrzejewski et al (Andrzejewski et al., 2006) found that D1R blockade in vSUB, but not the dSUB, impaired operant learning, but again, motivational deficits were discovered. While it appears likely that DA D1R activation is a crucial for directing the plasticity associated with operant learning, the precise role remains somewhat elusive. Emerging evidence, however, led us to postulate a critical interactive role of NMDAR and D1R in operant learning.

Intracellular convergence of NMDAR and DA D1R activation: coincidence detectors

From this evidence, we began to theorize that NMDARs in conjunction with DA D1Rs, and in particular the coincident detection of incoming signals, play a critical role in shaping synaptic configurations, and likely predominant neural ensembles, that underlie operant learning (Jay et al., 2004). NDMARs and DA D1Rs interact in dynamic ways. For example, NMDA-dependent LTP in striatal slices is blocked by D1 but not D2 antagonists (Weiss et al., 2000). In vivo evidence for NMDA-D1 interaction in plasticity-related phenomena suggests that LTP takes place in multiple circuitries and structures. For example, LTP in hippocampal-prefrontal cortex synapses is dependent on co-activation of NMDA and D1 receptors, as well as intracellular cascades involving PKA (Jay et al., 2004). In both striatum and prefrontal cortex, D1 activation potentiates NMDA-receptor-mediated responses (Cepeda et al., 1993, Seamans et al., 2001, Wang and O’Donnell, 2001). The potentiation of hippocampal-evoked spiking activity of accumbens neurons requires cooperative action of both D1 and NMDA receptors, while a similar synergism is observed for the amygdalo-accumbens pathway (Floresco et al., 2001b, a). Molecular studies complement these findings, showing NMDA-receptor dependence of D1-mediated phosphorylation of CREB (cAMP response element binding protein) (Das et al., 1997, Carlezon and Konradi, 2004), a transcription factor thought to be an evolutionarily conserved modulator of memory processes and key protein in cellular pathways affected by addictive drugs (Silva et al., 1998, Nestler, 2001). Strong support for the contention of coincident activation comes from the demonstration of long-term enhancement of synaptic strength when corticostriatal excitation and dopaminergic activation are temporally coordinated (Wickens et al., 1996). Other data suggest that glutamate and dopamine signals, via NMDA and D1 activation, converge to induce ERK activation in the hippocampus and striatum, thereby reconfiguring networks involved in learning and drug use (Valjent et al., 2005, Kaphzan et al., 2006). Thus, given the requirements necessary for learning, it is intriguing to speculate that the coordinated arrival of dopaminergic and glutamatergic signals, and its neuromolecular consequences, serve as the coincidence detector that initiates transcriptional changes leading to enduring synaptic alterations. It is important to note that these very cascades are the ones proposed to be modified in the addictive process (Hyman and Malenka, 2001).

In a direct test of this hypothesis, Baldwin et al. (Baldwin et al., 2002b) found doses of AP-5 and R(+)-7-chloro-8-hydroxy-3-methyl-1-phenyl-2,3,4,5-tetrahydro-1H-3-benzazepine hydrochloride (SCH-23390) (a D1R antagonist) in the PFC that had no discernible effect on operant learning. However, when combined and infused into the PFC of naïve rats, operant learning was significantly impaired, suggesting strong synergy between the two receptors. That is, plasticity associated with operant behavior is possible with a small amount of NMDAR or D1R blockade, but not both. Although we have seen some dose-dependent effects, we wondered if operant learning was an “all or nothing” phenomena, like concept learning (Osler and Trautman, 1961). In our experience, it appeared, that our rats first spent their time in the chamber exploring, nose-poking, sniffing, grooming, rearing, etc., while only occasionally lever-pressing. After a couple of sessions, control rats “got it” and proceeded to lever press much more frequently, and reared, explored, sniffed, groomed, etc., less (e.g., responses for which there were no programmed consequences), just as Staddon and Simmelhag demonstrated in their seminal experiment on superstitious behavior (Staddon and Simmelhag, 1971). Therefore, initial operant learning may engage a “tipping point” or threshold-like process, in contrast to a more gradual and smoothly changing one. Figure 1 shows the cumulative responses of two rats with cannulae targeting the NAc. One was infused with vehicle prior to the first five sessions while the second was infused with AP-5. The similarity in functions is striking and seems to conform to our notion: there is a very gradual and slow increase in responding, transitioning, relatively quickly, to a high, and steady, rate of responding. Note that the AP-5-treated rat is delayed in this transition, suggesting that this “tipping point” is delayed by NMDAR blockade.

 

Figure 1

Cumulative lever presses across sessions. The behavior of two representative rats, one vehicle-treated and one AP-5-treated, following infusions into the nucleus accumbens core (NAc) prior to the first 5, 15-minute long sessions. Infusions ceased after …

While these behavioral data and other observations may present a convincing argument regarding this “tipping point” hypothesis, it would be of great import if the neurobiology followed suit, for this would imply a “critical period” for operant learning and suggest targets for intervention in a time-dependent fashion. At the very least, it appears that operant learning is highly contextualized vis-a-vis temporal, environmental and neurophysiological relations.

An intracellular signaling model of operant learning

The intracellular molecular constituents of learning (in general, not necessarily operant learning), as noted earlier, have received a great amount of interest. Our own findings regarding the role of NMDAR activation were thoroughly informed by these findings regarding LTP. However, the intracellular signaling cascades responsible for LTP are now well-elucidated. Are they the same cascades responsible for reconfiguring the synaptic pathways during operant learning? Baldwin et al (Baldwin et al., 2002a) inhibited protein kinase activity, crucial constituents of intracellular signaling necessary for LTP, in the NAc of rats prior to operant learning sessions with the compound 1-(5-isoquinolinesulfonyl)-2-methylpiperazine dihydrochloride (H-7). In a separate group of rats, cAMP-dependent protein kinase (PKA) activity was inhibited by the drug Rp-adenosine 3′,5′-cyclic monophosphothioate triethlyamine (Rp-cAMPS) immediately prior to operant learning sessions. In both cases, learning was impaired suggesting that protein kinase signaling generally, and PKA activity specifically, were necessary for operant learning. Thus, several key intracellular components of the neural plasticity associated with operant learning have been identified.

PKA, PKC and other protein kinase activities converge intracellularly, according to several prominent models, at ERK (Valjent et al., 2005, Kaphzan et al., 2006). Phosphorylated ERK (pERK) translocates to the nucleus of neurons, where it is modulates the activity of CREB, widely held as an evolutionarily conserved mediator of long-term neural plasticity. Surprisingly, we have found little role for ERK in operant learning. First, U0126 (a pERK inhibitor) infused into the NAc prior to operant learning sessions produced no observable effect (Figure 2, panel A). We used the identical paradigms and preparations as with previous reports, however, given our lack of experience with this drug, it is possible that this negative effect was the result of an unknown technical problem. Second, we explored ERK phosphorylation after operant learning using standard Western blots and commercially available antibodies. Two groups of 6 rats were run: 1) standard operant training (FR-1/VR-2) and 2) yoked control (received the same number of reinforcers but did not have to lever-press to produce them). Brains were collected within five minutes of the 5^th session and processed by Western blot. No differences in ERK, pERK or the pERK/ERK ratio were noted in any of 12 areas studied, including the NAc (Figure 2, panel B). There was a slight, but statistically significant, effect in pERK in the vSUB and PFC, constituting roughly a 20% increase relative to yoked controls. Although the effect was statistically significant, it was very modest and quite possibly a Type 1 error given the number of comparisons we conducted. Third, we attempted to visualize, and hopefully, semi-quantify pERK throughout the brain after operant learning by using standard immunohistochemical methods on free-floating brain sections. These rats were treated identically to the Western blot experiments, however following brain collection, whole brains were sliced and pERK antibodies were used to localize pERK.

 

Figure 2

Role of ERK in operant learning. Panel A shows that U0126 infused into the NAc prior to learning sessions has no effect when compared to vehicle-infused controls. Panel B shows neither ERK-1 nor ERK-2 phosphorylation is increased in rats learning an operant …

Once again, while there was significant pERK staining in the PFC and vSUB, there was very little in the NAc (Figure 2, panel C). These data conform closely with the results of Westerns and suggest a limited role for ERK in operant learning, in contrast to the myriad of studies demonstrating a crucial role for this kinase in other forms of learning (Levenson et al., 2004, Chwang et al., 2006, Kaphzan et al., 2006). However, coincident NMDAR/D1R activation can recruit ERK-independent signaling routes to the nucleus.

CREB’s role in neural plasticity

pERK’s modulation of pCREB is critical during learning because CREB is a transcription factor increasing or silencing the expression of certain genes. These genes are thought to be the regulators of the synthesis of particular proteins that form the building blocks of receptors, membranes, and other structures crucial to neural plasticity. Indeed, we have demonstrated that protein synthesis in the NAc is critical during operant learning (Hernandez et al., 2002). Using the protein synthesis inhibitor, anisomycin, we showed that immediate post-session infusions into the NAc blocked subsequent operant learning, implicating transcription factors and de novo protein synthesis. Interestingly, infusions 2 or 4 hours after the session had no effect; anisomycin also had no effect during a performance test or a feeding test. Once again, it appears that we have uncovered key features of a tightly controlled, temporally and contextually, learning system involving multiple structures, receptors, signaling mechanisms, and now, protein synthesis.

The finding of protein synthesis dependency of operant learning was arguably one of the more important in our laboratory, yet it posed a large open-ended question regarding the specificity of this protein synthesis. We therefore conducted several experiments to identify which genes may be synthesized/upregulated during operant learning. Using standard in situ hybridization methods with rats treated much like the ones used for the pERK Western studies, we found that the immediate early genes (IEGs) Homer1a and egr1 (zif-268) were upregulated, compared to control rats, immediately after the 3^rd operant training session within discrete cortico-limbic-striatal nodes. Gene expression was elevated widely throughout the cortex and striatum, and in some cases, the hippocampus, but surprisingly, not in the ventral striatum (i.e., NAc). In contrast to the “early learning group”, a second group of rats experienced 23 operant learning sessions. Yet Homer1a and egr1 expression was now reduced compared to the early learning group, in nearly all nuclei studied, suggesting that these genes are involved in plasticity-related functions during early exposure, but not later exposure, to operant contingencies. The single exception was the ventrolateral striatum (VLS), which appears to remain, genetically speaking, “on line” even during extended operant exposure. Even though many scholars have termed long operant training as “habit formation” these responses remain adaptable and flexible (consider the “temporary” effect of reinforcement or the reduction one would see when operant contingencies are eliminated or extinguished): it is interesting to speculate that the VLS may subserve this monitoring function.

Other glutamate receptors also assist in plasticity associated with operant learning

Homer1a is thought to regulate and traffic group 1 metabotrophic glutamate receptors (mGluR1 and mGluR5). mGluR5s potentiate the activity of NMDARs by altering their permeability to Ca²⁺ (Pisani et al., 2001), raising the interesting possibility that one mechanism of NMDAR-induced plasticity may depend heavily on mGluR5 activity. Recently, we directly tested the role of mGluR5 activity on operant learning by blocking their activity with the drug 3-((2-Methyl-4-thiazolyl)ethynyl)pyridine (MTEP). Our preliminary results suggest that blockade of mGluR5 activity in the DMS impairs operant learning, although follow up experiments on this finding are ongoing.

AMPA receptor activation and operant learning has also been explored in our laboratory. Hernandez et al. (2002) demonstrated a time-limited role for AMPAR activation in the NAc during operant learning. The effect, however, endured for many sessions and may have been the result of some down-regulation or long-term internalization of glutamate receptors. While this contention needs additional empirical support, we found it very surprising that pre-session blockade of AMPAR would produce such a long-term effect relative to post-session blockade, which produced no change in operant learning.

Epigenetic changes during operant learning

In addition to activating transcription factors, NMDAR and D1R activity also induces modifications, such as histone acetylation, to chromatin, the protein that organizes and condenses genomic DNA. These modifications provide recruitment signals involved in gene transcription/silencing and influence access to DNA by the transcriptional machinery. NMDAR activation and associated intracellular signaling cascades, including histone 3 (H3) acetylation, govern long-lasting behavioral change, Pavlovian fear conditioning and instrumental Morris Water Maze learning (Atkins et al., 1998, Blum et al., 1999, Schafe et al., 2000). We recently began to explore whether operant learning modifies chromatin. Indeed, Histone H3 acetylation expression increased in certain structures during performance of an operant behavior, versus sucrose feed controls. In this experiment, rats lever pressing on an RI-30” schedule were sacrificed 30 minutes after a session. Brains were collected, processed and incubated with anti-acetyl-Histone H3 (Lysine 14) using standard protocols.

Interestingly, relative to yoked controls, we saw elevated histone H3 acetylation in the DMS, a structure widely considered as a key contributor to operant learning. These are some of the first data that we know of showing histone modifications during operant learning. However, increases in the global level of histone H3 acetylation could be a result of modifications at promoters of genes other than IEGs and, further, the rats used in this experiment had extensive training. Thus, additional information on the locus of that acetylation during operant learning is necessary. Nevertheless, these data, in conjunction with many other reports, strongly suggest that epigenetic processes are engaged during operant learning. Long-lasting modifications, like histone acetylation, may help us understand the enduring nature of operant behavior, its resistance to change, and the recalcitrance of certain disorders to treatment.

Epigenetic processes also appear to be modified during drug administration and learning. During cocaine self-administration, a D1R-dependent instrumental paradigm, chromatin modifications are induced in certain regions of the striatum at the promoters of many plasticity-related genes, such as Cbp, NR2B, Psd95, and GluR2. Cbp is critical for stimulation-induced activation of CREB and has intrinsic histone acetyltranferase (HAT) activity (Shaywitz and Greenberg, 1999). Transgenic mice expressing a truncated form of Cbp have several learning deficits (Wood et al., 2005). NR2B, a subunit of the NMDAR complex, contains the glutamate binding site and is essential for LTP, while the subunit NR2A is not (Foster et al., Foster et al., 2010). The NR2B subunit is phosphorylated by CaMKII, dephosphorylated by PP1, and mediates NMDAR internalization (Roche et al., 2001). Psd-95 inhibits NR2B-mediated internalization of NMDAR (Roche et al., 2001) and governs synaptic localization and stabilization of NMDARs (Li et al., 2003). GluR2 is a subunit of the AMPAR and contains a crucial phosphorylation site also modulated by intracellular protein kinase and protein phosphatase activity. Phosphorylation of GluR2 partially governs AMPARs permeability to calcium and other cations. Interestingly, mGluR5 stimulation in the rat dorsal striatum induces GluR2 phosphorylation, an effect blocked by NMDAR antagonism (Ahn and Choe, 2009).

An Intra-cellular convergence model of operant learning

Against this backdrop of dynamic and interesting work, we created a model of NMDAR-DA D1R convergence that may promote of greater understanding of the neural plasticity involved in operant learning. Figure 4 illustrates the prevailing hypothesis that glutamate-coded sensory/information processing signals activate NMDAR, and AMPAR, leading to Ca²⁺ influx into the cell. DA activation of D1Rs activates adenyl cyclase (AC, designated with a black arrow), and in turn, cAMP. The two signaling pathways interact in several places, for example, as CaM, induced by NMDAR activation, influences AC (although this is a somewhat oversimplified representation). PKA activates MEK, but also inhibits Ras/Raf (designated with a bar-headed line), suggesting that not only do the pathways converge, but also may compete for signal dominance.

 

Figure 4

An intracellular signaling model of operant leatning. The functional and structural changes involved in neural plasticity implicates coordinated NMDAR and DA D1R activation throughout cortical-striatal-limbic networks. This figure summarizes the prevailing …

Several points of possible convergence are demonstrated, most notably the activation of CREB, MEK and ERK. Critical plasticity-related effects are also demonstrated, like the CREB-dependent transcription of IEGs Arc, Homer1a, and egr1. Homer1a trafficks mGluR5 receptors (represented by a gray arrow), which subsequently potentiate Ca²⁺ influx via Gαq-protein coupled phospholipase C (PLC) activity (this potentiation is represented with a yellow arrow and lightening bolts); mGluR5 activity also potentiates DA D1R activation. Arc is transported to recently-activated synapses, likely performing a sort of “tagging” role. Recently, emerging data suggest an important role for Arc and ERK in AMPAR-subunit insertion and regulation of L-type voltage gated calcium channels. DARPP-32, activated by PKA activity, accumulates in the nucleus, inhibiting protein phosphatase 1 (PP1) activity, which is directly involved in chromatin modifications via intrinsic dephosphorylation activity (symbolized by a half-circle-headed arrow “grasping” a phosphate group). Histone deactylease (HDACs) actions are represented with an inverted-arrow headed line “grasping” acetyl groups from Histone 3 (H3). These histone modifications relax or compact chromatin thereby enabling or suppressing gene transcription (the particular modifications denoted in the figure do not necessarily represent the actual modifications required at the promoters of the IEGs for transcription) (Figure 4 is based on (Sweatt, 2001, Kelley and Berridge, 2002, Haberny and Carr, 2005, Ostlund and Balleine, 2005, Valjent et al., 2005). Therefore, the neuromolecular convergence of information from cortico-striatal-limbic NMDAR and DA D1R provides a possible substrate for plasticity in reward-based learning. The specific brain nuclei and neurons represented in this model are only now coming into focus, but likely involve key striatal, limbic, and cortical sites. Our strong suspicion is that medium spiny neurons, in the striatum especially, may be well-suited for plasticity-related functions due to their unusually high density of voltage-dependent ion channels that produce exceptional state-transitions (Houk and Wise, 1995) in combination with the convergence of widespread, glutamate-coded cortical, limbic, and thalamic afferents, as well as monoaminergic inputs from midbrain.

Kelley and colleagues (Kelley et al., 1997) initially pronounced a crucial role for the NAc in neural plasticity and operant learning. Indeed, our laboratory has explored the role of nucleus accumbens in a variety of behavioral paradigms using an expertly-arranged multi-disciplinary approach (e.g., the experimental analysis of behavior, behavioral neuroscience, molecular and cellular neuroscience, etc.). Dr. Kelley was one of the experts on the structure, physiology, connectivity and function of nucleus accumbens. However, several of our own experiments appear to contradict Dr. Kelley’s initial pronouncement. The convincing lack of MEK/ERK involvement in the NAc during operant learning and the lack of gene expression serve as two bold exceptions to the contention that plasticity in the NAc is crucial for operant learning. First, it may be that the MEK/ERK is not be involved in operant learning anywhere in the brain. Our studies of 12 other sites yielded very little difference between operant learning and yoked controls. Perhaps, the MEK/ERK pathway is involved during the “critical period” or “tipping point” when rats seem to “get it” and our studies did not have the temporal resolution to detect this effect, particularly as ERK activation is a dynamic and relatively rapid event. Perhaps our doses of U0126 were too low to inhibit ERK activation. However, an equally likely hypothesis is that CREB-mediated transcription of genes involved in neural plasticity are activated directly by other signaling pathways, such as PKAc or CAM (see Figure 4), bypassing the MEK/ERK pathway. And perhaps, we have not identified the critical plasticity-related genes or the myriad of possible epigenetic modifications to NAc neurons that enable and instantiate operant behavior. We hope to engage these questions with the same rigor and enthusiasm that Ann did.

Clinical implications

The prevailing hypothesis of this review is that the model presented in Figure 4 can inform treatment of many clinical problems. Of obvious relevance is drug addiction, for drug abuse profoundly affects many of the same molecular processes engaged by operant learning. In recent years, some of the most remarkable findings in research on addiction are those demonstrating significant overlap of the mechanisms mediating drug addiction and normal reward-related learning (Hyman and Malenka, 2001, Nestler, 2001, Wang et al., 2009). We are certain that many of the reviews in this special edition have elegantly highlighted the relationship between drug addiction and normal reward-related learning. Admittedly, this relationship has proved to be crucial in our understanding of addiction, however, we would like to cite some important new links between Dr. Kelley’s work on operant learning with emerging data and findings on other clinical problems. Those implications fall into two general themes: 1) clinical problems with associated learning impairments that could be served by a better understanding of how operant learning proceeds via neuromolecular mechanisms of plasticity and 2) clinical problems associated with ongoing, already learned, and possibly very resistant, operant behavior and its neuromolecular constituents. This latter case subsumes the problem of addiction, we think, as it is properly viewed as ongoing operant behavior with very damaging and long-lasting side effects.

As noted in the introduction, autism spectrum disorders are now thought to affect 1 in 88 children. Communication deficits, social interaction problems and stereotypic behavior patterns characterize autism, although communication skills can be typical in children with Asperger’s. Early intensive behavior therapy (EIBT), based on operant principles, forms the backbone of comprehensive treatment regimens that are yielding incredible results. This early therapy, which is highly individualized and contextualized, typically involves at least 40 hours of one-on-one therapy per week, often for many years. Data indicate that the earlier the intervention begins, the better the success rate. In many of these cases (some estimates are between 40-50%), complete mainstreaming into regular classrooms is possible with minimal or no additional supports (Lovaas, 1987, Sallows and Graupner, 2005, LeBlanc and Fagiolini, 2011). These findings intimate neural plasticity as a driving component in the success of EIBT. Researchers in the autism treatment community are widely speculating about “critical periods” of development which coincides with heightened neural plasticity (LeBlanc and Fagiolini, 2011). Thus, our research on operant learning may have two possible implications: 1) it is possible that the autistic “brain” may have reduced plastic potential, and only through intensive practice and therapy are these reductions overcome and 2) it may be possible, with a more complete understanding of operant learning, to induce periods of plasticity so older children could benefit from therapy.

While it is a highly speculative contention that operant learning, EIBT, and neural plasticity share underlie ASDs, there are several sources of converging supportive evidence. To begin, the leading heritable cause of ASDs is Fragile X syndrome (FXS), a single gene trinucleotide repeat problem with the FMR1 gene. FXS is associated with learning impairments, social behavioral deficits as well as some physical (primarily facial) abnormalities. The FMR1 gene encodes the Fragile X mental retardation protein (FMRP), which is required for normal neural development (Crawford et al., 2001, Antar et al., 2004). In addition, FMRP strongly modulates group 1 mGluR activity, and lack of FMRP activity dysregulates NMDAR LTP (Antar et al., 2004). Our recent work with the mGluR5 inhibitor MTEP suggests a role in operant learning for this receptor under “normal” conditions. Pharmacotherapies based on modulating mGluR5 activity are now being investigated for use in humans with FXS (Hagerman et al., 2012).

Another form of autism, referred to as “regressive autism” because children with this form develop typically for a period and then lose “normal” communication and social skills, has recently been linked to decreased activity of PKA and the catalytic subunit of PKA, namely the c-isoform. When compared post-mortem to non-regressive autistic controls, regressive autism frontal cortices showed decreased activity and expression of PKA (Ji et al., 2011). No differences were noted in other cortical regions, nor was there a difference between non-regressive autism and non-autistic controls. Thus, regressive autism may be linked to PKA-mediated phosphorylation of proteins and anomalous intracellular signaling. Once again, our work has demonstrated a crucial role for PKA in operant learning, converging nicely with this recent work on regressive autism.

Rubenstein-Taybi syndrome (RTS) is an autosomal dominant disorder caused by mutations of the CREB binding protein (CREBBP) gene. Short stature, broad thumbs, distinctive facial features, and moderate to severe learning difficulties characterize RTS (Bartsch et al., 2010). Of critical import here is the obvious link between operant learning, CREB function, and RTS. Perhaps children with RTS could benefit from EIBT or some pharmacological therapy that enables, supplements, or supplants CREB modulation of gene transcription. CREB phosphorylation appears to control IEG function and the synthesis of new proteins, and likely regulates neural plasticity associated with operant learning.

Lastly, our data and intracellular model implicate epigenetic processes as responsible for the enduring nature of operant behavior. Our common consideration of operant behavior as “habit formation”, repeated demonstrations of spontaneous recovery, and the seemingly unlimited recall period associated with operant repertoires contribute strongly to this idea. Indeed, many severe problem behaviors have proven exceedingly recalcitrant to treatment, thus leading to restricted social opportunities, chemical restraint, hospitalization and institutionalization. However, a broad class of diagnostic tools, often referred to as the “functional analysis of problem behavior” or “functional behavior assessment (FBA)”, have been developed to identify the controlling relations for these severe behaviors. Generally, these behavior classes are viewed as operant, reinforced by attention, access to preferred items/activities, or escape/avoidance of unwanted circumstances (Lerman and Iwata, 1993). With this information in hand, therapy can be directed in such a way as to provide alternative sources of reinforcement or alternative appropriate operants that produce those wanted circumstances, potentially even long after the original operant learning of the inappropriate behavior. Is it possible that a greater understanding of operant learning could provide pharmacotherapeutic targets, like histone acetylation, that enhance operant extinction and/or promote new operant learning?

While many of these notions are highly speculative, the work of Dr. Ann Kelley and colleagues in the area of operant learning is likely to inform, at the very least, the nature and course of drug addiction. We would also like to extend our theory and findings to help understand the learning deficits associated with ASDs, FXS and RTS, as well as the difficultly associated with the strength of certain severe problematic operant repertoires.

Footnotes

References