cognitive psychology short essay

12/2/2019

1

Learning and Predictive Processing

What is predictive processing? • “Grand unified theory” of the brain

• Emphasizes role of learning to predict next state of the world

• The better your internal models of predictions are, the more adaptive and successful you are at navigating the world • In PP models, “knowing” something

or “understanding” something means being able to predict it correctly

• Representationalist • Large focus on schemata • But still a role for embodiment

(especially in learning embodied associations)

• Mostly associated with constitution hypothesis

Predictive Power • Point is – things stay the same more often than they

change (stable world hypothesis). • We can save on storage and processing power by only coding

DEVIATIONS from the normal situation • If we only code “differences” between frame 1 and 2, we

don’t code all the redundancy – compression algorithms

• Need to have some way of predicting value at time/space X1 given value at time/space X0

12/2/2019

2

Compression in Perception

• These same principles of redundancy hold true in the world as well!

• If we only code for “differences” between our predicted model of the world (based on assumed redundancy) and the current state of the world (based on incoming sensation), we massively save processing power.

• If we have an accurate predictive model of the world…

• We only have to focus on the times our predictions are wrong (prediction error)!

Bayesian Statistics • Mathematical way of conceptualizing predictive

processing models

• Different way of thinking about statistics than what you might have been taught

• Basic idea • You have some sort of new finding • Instead of looking at p-value in isolation (frequentist

stats)… • …you consider likelihood of this finding, given all other

scientific or worldly knowledge you have (Bayesian stats)

• “Extraordinary claims require extraordinary evidence”

12/2/2019

3

The Bayesian Brain • Does the brain do something like Bayesian statistical

evaluation and updating?

• Yes! • Schemata – mental models about the state of the world

• More likely to believe information if it fits into a schema (confirmation bias)

• More skeptical of information that is schema-inconsistent

• Perception – biased towards perceiving things that are likely • Hollow mask illusion

• Actually…every single “top-down” effect we’ve discussed

• And just in general – things seem more likely to happen again when they have happened before

Beyond Bayes • Bayesian methods evaluate current likelihood based on

prior probability

• Predictive processing models take this a step further • Predict what’s going to happen based on prior probabilities,

before event even occurs • Predictive network can be updated when it is frequently wrong • Predictive models can GENERATE explanations for incoming

sensory information • Generative modeling • Based on most likely cause (out in the world) of these neural spike trains

(in the brain)

• Generative models can generate these explanations for sensory information that hasn’t even been processed yet! And then check accuracy against incoming information • Forward modeling

Imagination and Generation

• A generative model of perception comes with another benefit

• Can generate a perception without sensory input – imagery and thought

• Once a generative model is fully developed, can be used for imagination

• A better generative model implies more accurate imagination and embodied simulation (basically thought)

• Analog representation works because we have a generative model of what we are representing

12/2/2019

4

Precision in PP • When do we care about predictions being right or wrong?

• Sometimes, wrong predictions don’t really matter. • Sometimes, prediction is more useful than attending to incoming

sensory information. • Sometimes, we need to be sure our predictions and the state of the

world line up exactly.

• Driving a car down highway 5 • Well-practiced task • If it’s clear without a lot of cars – let prediction take over to reduce

cognitive load • If it’s foggy and crowded – weight prediction less, and give more

cognitive resources to processing incoming info

• How do we do this…? • Attention! • If you need to be precise with your predictive error (very important

to minimize error), or things are changing frequently (making prediction difficult) you’ll need to pay attention.

Prediction and Schizophrenia

• Schizophrenia might involve a weakening of prior knowledge/expectation on prediction

• Self-generated experiences appear to be externally caused • People with schizophrenia can “self-tickle” more

successfully

• Thoughts are attributed as coming from outside sources – hearing voices?

• Also lines up with learning and EF deficits

Prediction and Autism • Sensory overstimulation and overload in particular

• Too much prediction error signaling? • Explains attentional issues – constantly hyperaroused trying

to explain away prediction error • Explains social issues – don’t feel like you can confidently

predict other’s behavior (which is uncomfortable) • Possible driving force of repetitive behavior and restricted

interest (i.e., it’s more predictable)

• Also – people with autism are less subject to illusions based on top-down knowledge work! (hollow mask, illusory contours, McGurk) • Failure of mapping priors and predictions with incoming

sensory information – also results in the increased error signaling above

12/2/2019

5

Learning • Associate stimulus with response; input with output;

concept with concept…learning to predict!

• What does it mean to understand something? • Relation to embodied process?

• Associate abstract concept with concrete (embodied) concept?

• From enactivist point of view – understanding is only important in so far as it guides action and behavior

Correlative Learning Algorithms

• Hebbian learning • Neurons that fire together wire together

• Contiguous learning – any association of firing implies concepts are connected

• Thing A is associated with thing B

• Rescorla-Wagner (R-W) associative learning • Contiguous AND contingent (weaker form, but similar to,

necessary and sufficient conditions)

• Contingent learning – firing of B is contingent on firing of A

• Thing A predicts thing B

Comparing Learning Algorithms

• What associates with ice cream? • Hebbian learning

• Things associated with ice cream • Ice cream truck jingle • Hot day • Afternoon • Summer • Sun

• Rescorla-Wagner (R-W) associative learning • Things that are associated with ice cream AND are a very

good predictor of ice cream • Ice cream truck jingle

• Thus, simple Hebbian learning does not discriminate between “meaningless” correlations and “causal” correlations

12/2/2019

6

R-W Learning and Prediction

• Key point of R-W learning – not just association, but prediction

• R-W learning calculates predictive error of thing A leading to thing B and works to minimize error • Associations can have low or high predictive error

• Afternoon -> ice cream – high predictive error (many afternoons aren’t associated with ice cream, even though some are)

• Ice cream truck jingle -> ice cream – low predictive error

• Strengthen associations with low predictive error, weaken associations with high predictive error

Behaviorist Learning • Organism receives some sort of reward (US or previously

learned CS)

• Goal of learning – why did I get the reward, how to get reward again

• Requires surprise – if not surprised by presentation of reward, then you’ve already learned how to get the reward

• Work backward – what priors were associated with receiving reward? • Form hypothesis – A, B, and/or C led to reward • R-W learning – which (or what combination of) factors best predicts

reward

• After learning – reward is predictable given particular stimuli or particular actions, and is no longer surprising

• Note that this also works for punishment if we reverse stuff (how do I learn to AVOID)

12/2/2019

7

Prediction Error

• Signed prediction error (R-W learning) • US more than expected – increase association

• US less than expected – decrease association

• US the same as expected – no signed prediction error

• Unsigned prediction error (Pearce-Hall learning) • Degree of “surprise” or “novelty”, regardless of valence

• When unsigned prediction error is large, attention is required to identify and resolve error

• Goal of learning? Minimize all prediction error.

Learning is fun! • Learning is extremely important for adaptive life

• Need to learn to approach/avoid biologically relevant stimuli (unconditioned stimuli; US)

• Learn what is associated with the US (learn the conditioned stimulus; CS); what behaviors lead to reward

• The process of learning, therefore, needs to be something explicitly sought out • Needs to be intrinsically engaging…needs to be FUN!

Prediction and Play

• A quick side note about play!

• Play (games, pretend play, imaginative play) is a natural behavior seen in human children and many other young animals – play is fun!

• What is the significance of play? • Learning!

12/2/2019

8

Prediction and Play

• Play allows us to refine predictive models in a safe and (largely) consequence-free environment

• Early play usually involves motor control – refining perceptual abilities and motor skills

• Later play usually involves simulating life or competition in order to develop higher level strategy or decision making abilities

• All play involves making predictions and evaluating predictive error, in a safe way

Learning in Humans

• For adult humans: • Learning typically involves a “teacher” or some sort of

desired outcome that can easily be checked against • Supervised learning

• Biologically relevant learning involves reinforcement • Do something clearly biologically relevant (e.g., eat food, have

sex), get a reinforcing effect in the brain (reward circuit)

• What if the explicit correct answer is not provided? What if we are talking about perceptual learning?

• Before language is learned, babies don’t know if their predictions are “correct” or not

Learning to play Mario • Goal

• Get to the end of the level • Avoid dying

• Movement • Forward-backward with d-pad • Jump with A

• Obstacles • Don’t run into enemies • Don’t fall in pits • Can’t scroll screen backward

• Learned behaviors • Time jumps • Avoid/kill enemies • Get coins (= more lives/more chances)

12/2/2019

9

Learning to play Mario

https://www.youtube.com/watch?v=bg1uj9EYTTk

Eventually…

https://www.youtube.com/watch?v=Gum4GI2Jr0s

12/2/2019

10

Neural basis of learning

• How is learning implemented in the brain?

• Need to alter connections between neurons in a way that changes how information is processed

• What can we learn about learning from behaviorism, drugs, and addiction?

Learning and psychopharmacology • Many “drugs of abuse” primarily stimulate DA (dopamine)

system and sometimes also 5-HT (serotonin) system • DA long conceptualized as “reward” chemical necessary for

reinforcing behavior (reward circuit - VTA) • Drugs that primarily increase synaptic DA in VTA often addictive

• Cocaine, amphetamines

• Drugs that primarily increase synaptic 5-HT (and to a lesser extent DA) also subjectively pleasurable, but less likely to lead to addiction • MDMA

• Relative ratio of DA to 5-HT concentration • As 5-HT to DA ratio increases – NEGATIVE correlation with self-

administration and addictive behaviors • Implies some sort of “balance” between 5-HT and reinforcing

effects of DA, but why?

• What about a selective 5-HT drug (no DA)? • mCPP (a piperazine) – not particularly pleasurable nor reinforcing

Fischer & Ullsperger, 2017

A theory of DA, 5-HT, and Learning • Dopamine

• Signals predictive error in response to US • Update association based on signed prediction error

• Increase levels of DA when US is greater than expected • Once something is learned – DA release minimal (no signed

predictive error) – low subjective reward • Addiction likely as feedback control is bypassed

• DA released due to drug, not due to predictive error • Use of drug directly leads to high subjective reward

• Serotonin • Signals “unsigned” predictive error • Some sort of predicted outcome was different, for some reason

(surprising/novel) – this thing is significant • Cues attention – attend to something to determine why predictive

error was large

Fischer & Ullsperger, 2017

12/2/2019

11

A theory of DA, 5-HT, and Learning • DA also strongly related to motor control

• Block DA receptors using DA antagonists (antipsychotics) • Extrapyramidal effects; tardive dyskinesia occur after upregulation of DA

receptors • Not enough DA being produced / degeneration of DA system

• Parkinson’s disease • Tic disorders (e.g., Tourette’s)

• DA antagonists often prescribed • DA releasers (such as cocaine or methamphetamine) can cause tic-like

behaviors

• Why motor control? • Function of learning – associate behavioral program (aka motor

output) with desired outcome • So with too much DA – movements become spastic, difficulty inhibiting –

random movements being reinforced • And with not enough DA – movements become labored, difficulty controlling

– inability to reinforce appropriate movements

• Last note – atypical antipsychotics also antagonize 5-HT2 receptors, and the risk of TD and EPS is much lower

A theory of DA, 5-HT, and Learning • A novel US appears!

• 5-HT released due to unsigned prediction error • Surprise! This is an opportunity for learning! Pay attention! • This helps to cue attention and direct behavior to determine

associates of US – exploratory learning, play, investigation, etc. • DA released contingent on signed prediction error

• For given associate – how well does this behavior predict outcome? • If it predicts it well – increase association between behavior and US –

increase levels of DA • If it doesn’t predict it – decrease association between behavior and

reward – reduce levels of DA

• Why do we want both? • 5-HT only – surprise!! But…no way of resolving surprise. No

opportunity for learning. • DA only – still reinforces learning – but no control for overall

novelty of stimulus – when do you stop learning?

Last bit of Psychopharmacology

• So now we can go back to the questions raised from psychopharmacology…

• Why does increasing 5-HT ratio show a negative correlation with addiction? • High 5-HT indicates high unsigned prediction error

indicates surprise meaning… • Explore! Do different stuff! Something is not predicted

properly and you don’t know why so figure it out!

• Why are selective 5-HT releasers not commonly recreational? • High unsigned predictive error, but no learning = not fun

12/2/2019

12

Depression and Learning • Learned helplessness model

• Learned helplessness occurs when animal receives punishment no matter what they do

• They’ve learned to be helpless

• Thus, the process of learning is inhibited (learning ain’t free, and what’s the point of spending energy on it if it’s useless?)

• What do antidepressant drugs tell us? • Increasing synaptic 5-HT spurs neurogenesis and

plasticity • Aha! Neurogenesis is necessary for learning! Is this the function

of the 5-HT system and unsigned prediction error? Initiate neurogenesis such that DA signed learning can occur?

Learning and Embodiment • So if learning is all about mapping perception

and/or action to a desired outcome

• Then it seems clear why learning new concepts needs to tie in to embodied processes or learned sensorimotor knowledge • All learning is grounded in embodied experience

(somewhere down the hierarchy)

• What sort of things have we talked about in reference to learning and embodiment…?

Learning and Mirror Neurons

• Why the mixed data and controversy about role of mirror neuron system?

• From R-W predictive error learning account – visuomotor associations are no longer important once association has been sufficiently learned (no longer surprising)

• Under this view, learning is embodied, but the persisting knowledge is more symbolic and representational

Cooper et al., 2013; Burgess et al., 2013

12/2/2019

13

Learning and Connectionism

• So we need a way to code US, R-W associations, and both signed and unsigned prediction error in order to link perception/action/concepts together

• How do we model this? Neural networks! • Purpose of neural network – predict likely output given

input and previously learned associations

• Multiple ways to implement in neural nets • Reinforcement learning

• Genetic algorithms

• Recurrent networks and backpropagation of error

Learning in Machines • How do neural networks

learn? • Provide input and desired

output (training data set) • Use a learning algorithm-

Hebbian, R-W (back- propagation) to change connection weights to get the right answer

• Provide input without desired output and evaluate accuracy of the model (testing data set)

• Reinforcement learning – “right answer” is vaguely specified

Stimulus

Response

Cognition?

A robot learning how to move

https://www.youtube.com/watch?v=99DOwLcbKl8

12/2/2019

14

MarI/O

https://www.youtube.com/watch?v=qv6UVOQ0F44

How would this change if the game was unpredictable?

Prediction error, precision weighting, attention