psych 421

deefer

Chapter7.3_Neuroeconomics_Bayesinthebrain-createdfor3rdedition.ppt

Home >Psychology homework help >psych 421

Chapter 7.3:
Neuroeconomics: Bayes in the brain

Neuroeconomics

An interdisciplinary area within cognitive science, located at the interface between neuroscience and the following fields:

Microeconomics. The branch of economics that studies how individual consumers allocate resources.

The psychology of reasoning. The experimental study of how people reason and make decisions.

Behavioral finance. The study that focuses on how individuals make investments, and what that can tell us about the financial markets.

Decision theory. The mathematical theory of rational choice and decision-making.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Two types of questions

Theoretical reasoning: What should I believe?

Measure the support evidence provides for a hypothesis

Use probability theory and Bayes’s rule

Practical reasoning: What should I do?

Measure the utility of different courses of actions

Use probability theory, Bayes’s rule, and expected utility theory

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Expected monetary value

Suppose you are risk-neutral, what is the most that you would pay for the opportunity to play a game in which you win $10 if a fair coin is tossed and lands heads, but get nothing if it comes up tails?

Two possible outcomes: lands heads: $10; lands tails: $0

On average: $5 -> expected monetary value of the game

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Limitation of the concept of expected value: St. Petersburg game

A fair coin is tossed. If the coin lands heads, you receive $2 and the coin is tossed again. If it lands tails, the game ends. But if it lands heads, then you will receive $4 and the coin is tossed again. The game will continue as long as the coin lands heads. At each round the pay-off (for heads) will be twice what it was in the previous round.

The expected value of the game: (0.5  $2) + (0.5  $4) + (0.5  $8) + . . ., which is infinite.

Problem: it is absurd that in this game the value a person is willing to pay to play the game is infinite.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Solution: utility rather than value

Utility is an index of the strength of your preference

More utility assigned to X than to Y is to say that one prefers X to Y and, given the choice, will choose X over Y

Utility is not linear with money

Money has diminishing marginal utility.

E.g. The $10 that takes your net worth from $100 to $110 will mean much more to you than the $10 that takes you from $1,000,000 to $1,000,010.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Calculating expected utility

Identify the different possible outcomes of an action.

Assign a utility to each outcome.

Assign a probability to each outcome (how likely it is)

For each outcome, multiply its utility by its probability.

Adding together the results for each outcome.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Bayesian rational decision

Bayesians think that rational decision-makers will always act in a way that maximizes expected utility.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Risk-averse vs. risk-loving

Risk-averse: assign a $1 lottery ticket a lesser utility than you assign to having $1 in cash.

Risk-loving: assign a $1 lottery ticket a greater utility than having $1 in cash.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Case study: Neurons that code for expected utility

Paul Glimcher and Michael Platt found intriguing evidence of neurons that have significant Bayesian characteristics.

Method:
Tiny electrodes are inserted into the monkey brain to make single cell recording while the animal is awake (there are no pain or touch receptors in the brain).
The electrodes are small and sensitive enough to detect the firing rates of individual neurons.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Background knowledge of the experiments

Saccadic eye movements move the line of sight very quickly from one place to another in the visual environment. This allows the perceiver to scan the environment (to detect a predator, for example).

Smooth pursuit eye movements allow the eyes to track objects moving continuously in a single direction.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Train the monkeys to perform saccadic eye movements

Monkeys are placed in front of a screen, fixating on a colored light directly in front of them.

The experimenters flash a red spot on the right side of the screen. When the monkey made a saccadic eye movement towards the red spot, it is rewarded with a portion of fruit juice.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Three questions in the process

How does the monkey detect the red spot?

How are saccades actually generated?

What happen in between detecting the red spot and generating saccades?

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

How does the monkey detect the red spot?

We know how the monkey detects the red spot because the primate visual system has been comprehensively mapped out.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

How are saccades actually generated?

The superior colliculus, which is located in the mid-brain, and the frontal eye field, which is in the frontal cortex, both play an important role in controlling saccades.

These two brain areas are organized like a map, which individual neurons responsible for specific locations to which a saccade might be directed.

Just before the monkey makes a saccade to a specific location, the neuron corresponding to that location fires in the superior colliculus and/or frontal eye field.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

What happen in between detecting the red spot and generating saccades?

LIP (the lateral intraparietal area) projects to both the superior colliculus and to the frontal eye field.

Platt and Glimcher experimentally show that LIP is essentially carrying out Bayesian calculations.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

The route of saccade production

From Glimcher, P. W. (2003). Decisions, uncertainty, and the brain: The science of neuroeconomics (Vol. 375). Cambridge, MA: MIT press, p230.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Basic set-ups

There is a particular action: make a saccade.

There is a particular reward: delivery of fruit juice.

Platt and Glimcher’s breakthrough idea was to vary both the size of the reward (utility) and how likely it is to be delivered (probability).

This design allowed them to explore whether neurons are sensitive to those variations in the reward.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Probability detecting neurons

Aim: Explore whether neurons in LIP are sensitive to probability.

Design: Platt and Glimcher set up a saccade experiment where the probability that the saccade would be rewarded varied.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Method

The monkeys fixated on a light at the center of the screen.

Afterwards, two other lights--targets for the saccades--appeared on the screen, one on the left of the fixation point and one on the right.

Then the fixation light changed color, turning either red (80% of the time) or green (20% of the time) (or reverse). The monkeys had been trained that when the fixation light turned red, they would be rewarded if they made a saccade to the left, and a saccade to the right would be rewarded if the light was green.

Researchers recorded throughout each trial from representative neurons in LIP.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

From Glimcher, P. W. (2003). Decisions, uncertainty, and the brain: The science of neuroeconomics (Vol. 375). Cambridge, MA: MIT press, p257.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Results

For each neuron, researchers compared the trials where the stimulus and the movement were the same.

For example, they compared all the trials in a given block where the monkey made a saccade to the left in response to a red light.

Doing that made it possible to compare blocks where the probability of a red light (and hence a reward for making a saccade to the left) was low with blocks where red had a high probability.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Neurons sensitive to probability

Prior to the fixation light changing color, in blocks where the probability of being rewarded with a left saccade was high, a typical LIP neuron had a much higher firing rate than in blocks where the probability of reward was low.

The neurons are encoding the prior probabilities.

After the fixation light changed color, the firing rate for the saccade that was definitely going to be rewarded was maximal, even if the probability that the saccade would be rewarded had been low.

The neurons are updating the priors to posteriors when the fixation light changes color.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

From Glimcher, P. W. (2003). Decisions, uncertainty, and the brain: The science of neuroeconomics (Vol. 375). Cambridge, MA: MIT press, p260.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Utility detecting neurons

Aim: Explore whether neurons in LIP are sensitive to utility.

Design: Platt and Glimcher set up a saccade experiment where the quantity that the saccade would be rewarded varied.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Method

The monkeys fixated on a light at the center of the screen.

Afterwards, two other lights--targets for the saccades--appeared on the screen, one on the left of the fixation point and one on the right.

Then the fixation light changed color. The probability of red vs. green was held constant, with each color coming up on exactly half the trials.

The quantity of the reward varied. In one block, for example, the reward for looking left might be double that for looking right.

Researchers recorded throughout each trial from representative neurons in LIP.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Neurons sensitive to utility*

Before the light changes color, the neuron fires more strongly on average in the high-reward block than in the low reward block.

After the light changes color (and so when the reward is revealed), the average firing rate increases significantly in both conditions. But the difference across the two conditions remains constant. The neuron responds more vigorously to the larger reward.

*The experiments only show that neurons are sensitive to reward, not utility. Still, they are highly suggestive.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

From Glimcher, P. W. (2003). Decisions, uncertainty, and the brain: The science of neuroeconomics (Vol. 375). Cambridge, MA: MIT press.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Combining probability and utility: “expected utility” detecting neurons

Aim: Explore whether neurons in LIP are sensitive to expected utility.

Design: Platt and Glimcher set up a free choice experiment where the expected utility of looking at a stimuli varied.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Method

The monkeys fixated on a light at the center of the screen.

Afterwards, the monkey could choose to look at a stimulus on the left or a stimulus on the right. Everything was held constant within each block of 100 trials, but the quantity of the (fruit juice) reward varied across blocks.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

A Bayesian choice

From a strict Bayesian perspective, the optimal way of responding in this type of experiment is to maximize expected utility over the long run.

i.e., to sample the two alternatives until one figures out which yields the highest reward and then stick with that option until the end of the trial.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Actual results

The monkeys did not adopt the optimal strategy.

They displayed a form of matching behavior.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Matching behavior

The monkeys split their choices between the two alternatives in a way that matched the distribution of total reward across the two alternatives.

In other world, if looking right yielded twice the reward of looking left, then they looked right twice as often as they looked left.

The monkeys seemed to be engaged in a form of maximizing behavior: maximizing something else that was not that far off from expected utility.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Melioration theory

Value is fixed by local rates of reward.

An animal behaving according to melioration theory will choose the option that seems most attractive at that time.

Attractiveness at a time is fixed by the rewards that that option has yielded in recent trials.

So, if looking left has had a good track record in yielding the reward on the past few trials, then the monkey will continue to look left. But if looking left starts to look unsuccessful, the monkey will switch and start to look right.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Maximizing according to melioration theory

They monkeys were behaving consistently with melioration theory, i.e., the monkeys tended to choose the option that had been most highly rewarded over the previous 10 trials.

Expected value as calculated by melioration theory is a local quantity that looks back to the history of recent rewards, whereas expected utility is a global measure of overall rewards.

Expected value in the melioration theory sense is a good approximator of expected utility in many contexts.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

From Glimcher, P. W. (2003). Decisions, uncertainty, and the brain: The science of neuroeconomics (Vol. 375). Cambridge, MA: MIT press.

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020

Conclusion

The experiments do not show that neurons in LIP are calculators of expected utility. But they show that neurons are sensitive to the varying benefits to the animal of different courses of action.

“In our free-choice task, both monkeys and posterior parietal neurons behaved as if they had knowledge of the gains associated with different actions. These findings support the hypothesis that the variables that have been identified by economists, psychologists and ecologists as important in decision-making are represented in the nervous system.” (Platt and Glimcher 1999)

Cognitive Science  José Luis Bermúdez / Cambridge University Press 2020