The Classical Conditioning Model with
Spiking Neuron and Its Application to the
Inverted Pendulum Control System1
Beibei Yang Xiaogang Ruan
Institute of Information and Control Institute of Information and Control
Beijing University of Technology Beijing University of Technology
Beijing, P. R. China 100022 Beijing, P. R. China 100022
Abstract
A spiking-neuron based cognitive model with classical conditioning behaviors is
proposed. With a reflex arc structure and a reinforcement learning method based on
the Hebb rule, the cognitive model possesses the property of ‘stimulate-response-
reinforcement’ and can simulate the learning process of classical conditioning. An
experiment on the inverted pendulum validated that this model can learn the balance
control strategy by classical conditioning.
Keywords: Cognitive model; Spiking neuron; Hebb Learning; Inverted Pendulum
1 Introduction
Ivan Pavlov, a Russian scientist in biology, presented classical conditioning in 1927, which is the basic
form of learning. In his well known experiment, a dog would repeatedly hear a tone which was always
followed by the presentation of food. When the food was presented, the dog started to salivate. After a
number of such presentations, the dog will learn that food follows shortly after the tone and would start
to salivate already when it heard the sound. In the experiment, food is the unconditioned stimulus (US),
and tone is the conditioned stimulus (CS). The pairing of a CS with a motivationally significant US
defines classical conditioning as an experimental procedure. While the unconditioned stimulus
produces an unconditioned response (UR), the response produced by the conditioned stimulus is called
the conditioned response (CR) and is used as an index of the learning process. Initially, no CR is
produced, but as learning progresses, the intensity or probability of the CR increases. It has attracted
researchers for many years and several models were developed.
Classical conditioning is the basic learning form and it has attracted many researchers to devote to this
field and proposed several learning models. Roscorla and Wagner proposed the first computational
model of classical conditioning in 1972 [1]. Sutton and Barto [2] proposed both the Time-Derivative
model and the Temporal-Difference model in 1981 and 1987 respectively. The former is an early
reinforcement model in conditioning, as an extension of the former, the goal of conditioning in the
latter model is to predict the temporally discounted value of all future rewards, and the model can
successfully model the inter-stimulus interval (ISI) dependency. Klopf introduced the Drive
1 Support by NFS Grant # 60375017
Reinforcement (DR) model, which separates inhibitory and excitatory learning and can simulate the
secondary conditioning and the reacquisition effect, in 1982 [4]. Schmajuk and DiCarlo introduced a
model in 1992[5], this model was able to model a number of classical conditioning phenomena and
especially to model the effect of various types of configurational stimuli and the effects of hippocampal
lesion on conditioning. Balkenius presented a model based on a neural interpretation of the
conditioning mechanism in 1995[6], this model can handle secondary conditioning and blocking very
well, but it can neither model reacquisition effects nor distinguish between delay and trace conditioning.
Christopher and Anders presented a classical conditioning model that is composed of some
interconnected Bayesian confidence propagating neural networks (BCPNNs) in 2002[7], by
implementing Hebbian learning, this model is able to make a closer tie between the output and the
underlying neural activity.
This paper aims at simulating the spiking transfer and process mechanism of information in biological
nervous system, and introduces a spiking-neuron based cognitive model with classical conditioning
behaviors. With a reflex arc structure and the implement of Hebbian reinforcement learning, the
cognitive model possesses the property of ‘stimulate-response- reinforcement’ and is able to simulate
the learning process of classical conditioning. An experiment on the inverted pendulum validated that
this model can learn the balance control strategy by classical conditioning.
2 Architecture of the Classical Condit ioning Model
The Architecture of the Classical Conditioning Model (CCM) is showed in Figure 1, where
− Reflex Agent (RA) is constructed according to the reflex arc of the biological nervous system and is
used to realize the reflex function, ., generate the response for the stimulus.
− Learning Agent (LA) implements the Hebbian learning and modifies the connections weights of RA.
− Evaluate Agent (EA) evaluates the learning results and instructs the modifications in LA.
Learning Agent
Reflex Agent
Evaluate Agnet
Stimulus Response
Figure 1: Architecture of the Classical Conditioning Model
Therefore, the Classical Conditioning Model can be described as a triple:
CCM=<RA,EA,LA> (1)
where
− RA : R→S is the mapping from stimulus (S) to responses (R). 1 2{ , , , , }nS US CS CS CS= " is the set of
stimulus,
− EA:{ , }R v→S is the mapping from the 2-tuple {S,R} to the evaluation v.
− LA: →W W will modify the weight matrix W to achieve learning.
Classical conditioning is a cognitive process. CCM can only respond to US in the beginning, but it can
gradually exhibit the cognitive behavior and set up the classical conditioning during the iteration
process of stimulus- response-reinforcement.
3 Reflex Agent and Spiking Neuron
3 .1 Ref lex Age nt
The structure of the reflex agent is showed in Figure 2, where
− Stimulus Neuronal Group (SNG) act as the receptor and the afferent nerve of the biological reflex arc,
they receive the outside stimulus and transform these stimulus to the nerve impulses.
− Central Processing Neuronal Group (CPNG) act as nerve center, they process the received nerval
information.
− Response Unit (RU) act as efferent nerve and effectors, they transform the nerval information to
responses.
No inner connections exist in each group, and connections exist only between the neuronal group of
SNG and CPNG and between the neuronal group of CPNG and RU.
SNG
...
...
S
0P
US
1CS
nCS
S
0W
S
1W
S
nW
CPNG
CP
RW
...
SW
Response
Unit
S
1P
S
nP
Response
Figure 2: Structure of RA
Also, the reflex agent can be described as a 4-tuple:
RA=<SNG,CPNG,RU,W> (2)
where
− { }SNG P | 0 1Sk k , , ,n= = " , in which
• (0)0 0P {v | 1 2 }SS i i , , ,n= = " is the unconditioned stimulus neuronal group (USNG), and (0)vSi
denotes the i-th neuron of 0PS .
• ( )P {v | 1 2 }( 1 2 )S kSk ki i , , ,n k , , ,n= = =" " is the conditioned stimulus neuronal group (CSNG), and
( )vS ki denotes the i-th neuron of PSk
− { }CPCPP = , in which
• CP {v | 1, 2, , }Ci Ci n= = " is the central process neuron set, and Cn is the number of neurons
− RU {v }R= is a response neuron
− { , }S R=W W W is the weight matrix, in which
• { | 0 1 }SS k k , , ,n= =W W " is the weight matrix between PSk and CP , ( )( ) C KS kSk n nijw ×=W .
• ( ) CRRR ij n nw ×=W is the weight matrix between CP and RU
• ( 1, 2, , )Sk k n=W " can be modified in the progress of learning, 0SW and RW are constant
matrixes.
3 .2 Spik ing Neurons
All neurons in this module adopt the Spiking Response Model developed by Gerstner[8], which can
simulate the electrical pulse of biological nerval information and transform information in the form of
spiking.
Firing occurs whenever ui reaches the threshold from below, where ui is the membrane potential of
neuron i. The evolution of ui is decided by three parts, the potential caused by the last spike, the
potential caused by other neurons and the potential caused by external inputs.
In the Spike Response Model (SRM), the state of a spiking neuron is described by the membrane
potential iu of neuron i . iu will remain at the resting potential restu ( 0restu = in this paper) if there is
neither a spike arrives nor an input. Once there is an incoming excitatory or a spike emitted by a
pre-synaptic neuron making iu exceed the firing threshold θ , the neuron will emits a spike.
( ) d ( ) ( ) and 0
d
if
ii
u tt t u t
t
θ= ⇔ = > (3)
Where, ( )fit is the spike time train of neuron i.
According to SRM
∑ ∑ ∫∞ −−+−−+−=
j f
ext
i
f
jiijijii dsstIsttttttwtttu 0
)( )(),ˆ(),ˆ()ˆ()( κεη
(4)
where
− iˆt is the last spiking time of neuron i
− )ˆexp()ˆ()ˆ( 0
rec
i
ii
tttttt τηδη
−−−−=− , in which
• ( )sδ is the pulse function
• 0 0η > is a coefficient
• recτ is the refractory period or the recharging period
− ijw is the weight between neuron j and i
−
0
1( , ) exp( ) ( )s
m
ts t t t dt
C
ε ατ
′ ′ ′= − −∫ is the effect of an incoming spike on the neuron membrane, in
which
• mτ is a membrane potential time constant
• ( ) exp( ) ( )
s s
q ss sα τ τ= − Θ is the response function of postsynaptic current, in which
sτ is a membrane current time constant
( )sΘ is a step function
− 1( , ) exp( ) ( ) ( )
m
ts t s t t
C
κ τ= − Θ − Θ is the effect of an input on the neuron membrane potential
In the real biological nerve system, the electrical properties of neurons are different even though they
belong to the same kind of neuron. SRM simulates this dynamic property of neurons by introducing
some noises. This paper introduce noises to both the time constant and the threshold, where the time
constant are decided in a certain range according to the uniform distribution, and the threshold are
decided according to the following probability function.
1 exp{ exp[ ( )]}P uβ θ= − − − (5)
Where, β is a firing coefficient, u is the membrane potential, and θ is the threshold.
The response of spiking neurons are shown in figure 3, where 1u is the membrane potential of
neuron 1v , 2u is the membrane potential of neuron 2v , 1u only receive the input, and 2v receive both the
input and the spike emitted by 1v . In figure 3, real line denotes the membrane potential, dashed line
denotes the input, and dash-dotted line denotes the threshold. Once the membrane potential exceeds the
threshold, the neuron will emit a spike with 3 as its membrane potential.
20 40 60 80 100 120 140
0
1
2
3
t (ms)
u
20 40 60 80 100 120 140
0
1
2
3
t (ms)
u
1
2
Figure 3: Response Curve of Spiking Neurons
4 Reinforcement Learning Mechanism
4 .1 Evaluate Agent
Menzel and Giurfa[9] studied the complexity of cognitive functions in the honeybee brain, they found
a particularly striking neuron in the bee brain named VUMmx1 (ventral unpaired median neuron of the
maxillary neuromere 1), which serves the function of a value system. Referring to the different
response to the unconditioned and conditioned stimulus before and after the setup of conditioning, this
paper introduces an evaluate agent that is shown in figure 4, where
Evaluation
v(t)
NV
Evaluate Agent
S
0P
1e
0e
ne
Re
...
RU
S
1P
S
nP
Figure 4: Evaluate Agent
− NV: Evaluate neuron;
− 1( ) ( 0,1, , )kk k ne k n×= =e " : the linkage weight matrix between PSk and NV;
− Re : the linkage weight between RP and NV. They are used to estimate the setup and forgetting of
conditioning.
− NV adopts the Spiking Response Model, its membrane potential is the evaluation value.
∑ ∑∑
= =
−−+−= n
k
n
j f
fkS
j
vv
jk
vv
k
ttttetttv
0 1
))(( ),ˆ()ˆ()( εη
(6)
Where, ke vary as the setup and forgetting of conditioning. Only 0e and 1e are used in the experiment.
0 0
0 0 0
1 case 1
( ) case 2
( 1) (1 ( 1)) otherwise
e t a
e t c e t
⎧⎪= ⎨⎪ − + − −⎩
(7)
1 1
1 1
0 case 1
( ) case 2
( 1) otherwise
e t a
c e t
⎧⎪= ⎨⎪ −⎩
(8)
In which,
• 01 0a− < < and 10 1a< < are the value of 0e and 1e at the time when conditioning set up
respectively;
• 0c and 1c are coefficients, and 0 1, [0,1]c c ∈ ;
• Case 1 means that no conditioning has been set up at time t or the conditioning acquired has been
forgotten at time t.
When no conditioning has been set up, the value of 0e is big and thus NV has a strong response onUS ,
1e is zero and NV has not response to 1CS . When a conditioning has been set up, ., when 1CS can
generate a CR , 0e will be a negative and US will inhibit NV, and 1e will increase so that 1CS can
respond to NV. Once a conditioning been forgotten, 0e and 1e will resume their initial state.
4 .2 Learning Algor i thm
Bi and Poo [10] studied the synaptic modifications in cultured hippocampal neurons, they found that the
synaptic change depended greatly on relative timing of pre- and postsynaptic activity. Referring to their
results, a reinforcement learning method based on the Hebbian learning with a decay item is introduced. This
algorithm is characterized with the property of ‘stimulate-response-reinforcement’ and it is detailed as
follows:
nknjni
dsstSsWtSstSsWtStvwtw
kC
t kS
j
C
i
C
i
kS
j
kS
ij
kS
ij
,,2,1 ,,,2,1 ,,2,1
)()()()()()()()(
0
)()()()(
""" ===
−−+−+−=∆ ∫γ (9)
where
− The first term is decay term that prevent the weights high up to the maximum value, the decay rate 0γ > ;
− The second term is the Hebbian learning term that modifies the weights according to the timing of pre- and
postsynaptic spiking, and v(t) is the evaluation. In which,
• ( ) ( )S kjS t and ( )CiS t is the spike trains of pre-synaptic neuron j and postsynaptic neuron
i respectively.
• ( )W s is the critical window and it is defined as follows
(1 ) exp( / ) s 0
( )
( 1)exp( / ) s 0
ij
ij
w s
W s
w s
α τ
α τ
+ +
− −
− <⎧= ⎨ − ≥⎩
(10)
In which, 0α+ > , 0α− < , are the amplitude of the critical window; 0τ+ > , 0τ− > , are the time
constant of the critical window; ( ) ( )f fj is t t= − is the interval between the time of pre- and
postsynaptic spiking.
5 Experiment on Inverted Pendulum Control System
The proposed spiking neuron based cognitive model is applied to the inverted pendulum system to learn
the balance control technique. The state of the system at time t is specified by four variables, .,
( , , , )x xθ θ� � , where θ is the angle between the pendulum and vertical, θ� is the angular velocity of the
pendulum, x is the horizontal position and x� is the velocity of the cart. In this experiment, the inverted
pendulum is the environment and the states of the inverted pendulum are outside stimulus, the proposed
CCM is expected to set up conditioning and learn the balance control technique of the pendulum in the
process of stimulate-response-reinforcement. The experiment system is shown in figure 5.
Learning Agent
Reflex Agent
Evaluate Agnet
Stimulus Response Inverted
Pendulum
Figure 5: Structure of the inverted pendulum control experiment system
In the experiment, Once θ increase and large than ±20º, a failure will occur and a new trial will start.
Once θ exceeds ±10º, US, as an input signal to USNG, will produce an unconditioned response to
prevent the pendulum from falling down. CSNG in RA consists of four groups and each group receives
a state variable as CS.
The experiment results are shown in figure 6. The left plot shows the angle curve in the learning process.
From it we can learn that the learning process begins with an initial angle of 10º. In the beginning of the
learning process, no conditioning is set up, thus RA does not respond to CS and has output only if US
presented; after conditioning has been set up, RA can make proper responses before the angle exceeds
±10º, ., even if there is no US, CS will produce conditioned response to balance the pendulum. The
right plot of figure 6 shows the curve of the control process with initial degree at -15º, it proves that
after conditioning, the cognitive model can balance the pendulum in a large range of initial angle.
0 1 2 3 4
-10
-5
0
5
10
time (second)
an
gl
e
(d
eg
re
e)
0 1 2 3 4 5 6
-15
-10
-5
0
5
time (second)
an
gl
e
(d
eg
re
e)
Figure 6: Results of the Inverted Pendulum Experiment
6 Discussion
A spiking-neuron based cognitive model with classical conditioning behaviors is introduced. With a
reflex arc structure and the Hebb learning, the cognitive model possesses the property of
‘stimulate-response- reinforcement’ and can simulate the learning process of classical conditioning. An
experiment on the inverted pendulum validated that this model can learn the balance control strategy by
classical conditioning.
In the real-world situation, the model of the environment is always imprecise even unknown. A
controller based on the proposed cognitive model of classical conditioning can learn control strategy
from the interaction with imprecise even unknown environment without prior knowledge. After
conditioning been set up, the cognitive model based controller can even successfully balance the
pendulum in a larger range than it learnt. This capacity of the spiking-neuron based cognitive model
proposed in this paper may suggest its great potential in many applications.
References
[1] Rescorla, R. A. & Wagner, A. R. (1972) A theory of Pavlovian conditioning: variations in the effectiveness of
reinforcement and non-reinforcement, Classical Conditioning II: current research and theory, pp 64-99. New
York: Appleton-Century-Crofts.
[2] Sutton, R. S. & Barto, A. G. (1981) Toward a modern theory of adaptive networks: expectation and prediction.
Psychological Review. 88:135-170.
[3] Sutton, R. S. & Barto, A. G. (1990) Time-derivative models of pavlovian reinforcement, Learning and
computational neuroscience: foundations of adaptive networks. Cambridge, MA: MIT Press.
[4] Klopf, A. H. (1980) The hedonistic neuron: a theory of memory, learning and intelligence. Washington, D. C.:
Hemisphere.
[5] Schmajuk, N. A.& DiCarlo, J. J. (1992) Stimulus configuration, classical conditioning, and hippocampal
function. Psychological Review. 99:268-305.
[6] Balkenius, C. Natural intelligence in artificial creatures. Lund University Cognitive Studies 37.
[7] Christopher, J. & Anders, L. (2002) An associative neural network model of classical conditioning.
TRITA-NA-P0217.
[8] Gerstner, W. & Van Hemmen, J. L. (1992) Associative memory in a network of ‘spiking’ neurons. Network.
3:139-164.
[9] Randolf, M. & Martin, G. (2001) Cognitive architecture of a mini-brain: the honeybee. Trends in Cognitive
Sciences. 2:62-71.
[10] Bi, G. & Poo, M. (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike
timing, synaptic strength, and postsynaptic cell type. J. Neurosciences. 18:10464-10472.
