Concepts / Goal Gradients and Delayed Reinforcement

Goal Gradients and Delayed Reinforcement

The important question is not simply whether a delay exists. It is whether something during that delay can provide a more immediate reinforcing influence.

  • Programming

Why Delay Creates a Learning Problem

In instrumental conditioning, an action and its reward do not always occur together. When the primary reward follows the action only after an interval, the reward has a longer distance to bridge before it can influence learning. Hull's account addresses this problem by proposing that the primary reward does not have to be the only influence operating during the delay.

Key point

The important question is not simply whether a delay exists. It is whether something during that delay can provide a more immediate reinforcing influence.

Tracing the Action-to-Reward Interval

Start with the sequence proposed in Hull's account. An action occurs first. A delay follows. The primary reward arrives at the end of that delay. If stimuli regularly appear during the interval, those stimuli can favor the development of secondary reinforcement. The learner is therefore not influenced only at the final point; an additional reinforcing influence can operate during the interval.

Actioninstrumental responseDelayintervalRegular stimuliduring the delaySecondaryreinforcementmore immediate influencePrimary rewardat the endfollowed bycontainscan favor development ofends with
What happens between an action and a later primary reward when regularly occurring stimuli are present?

Secondary Reinforcement as a Bridge

Definition

In the context of delayed reinforcement, secondary reinforcement is the reinforcing influence that develops from stimuli regularly occurring during the delay. It supplements the delayed primary reward by providing a more immediate influence during the interval.

The source presents regularly occurring stimuli as conditions that can favor secondary reinforcement. The key feature is their position in the sequence: they occur after the action but before the primary reward. This gives them a possible role in connecting the earlier action with the later outcome.

Actionearlier responseDelay stimulusregularly occurringSecondaryreinforcementimmediate influencePrimary rewardlater outcomeLearningsupported across delayis followed bycan favorsupportsalso influencesis separated from by delay
How is a stimulus during the delay related to the response and the later primary reward?

Extending the Goal Gradient

The source describes a goal gradient as something that can span time. Molar stimulus traces, understood in Hull's theory as the lingering presence of a stimulus, are part of the explanation. Hull's proposal is that secondary reinforcement works together with these traces, allowing the resulting gradient to extend beyond the period that stimulus traces alone would cover.

In practical terms, extending the goal gradient means that the influence associated with the goal can reach farther through the delay. Without secondary reinforcement, the primary reward is separated from the action by the entire interval. With secondary reinforcement, a more immediate influence appears within that interval, so learning need not rely only on the reward at the end.

Actioninitial responseActioninitial responseDelayno intermediate influenceDelay stimulusintermediate influencePrimary rewardfinal influencePrimary rewardfinal influencefollowed byends withfollowed during delay byprecedes
How does secondary reinforcement change the pattern of influence across the delay compared with waiting only for the final reward?
Actionbeginning of intervalDelayprimary reward stilldistantDelay stimulussecondary influencePrimary rewardend of intervalPrimary rewardlater outcomedelayeventually reachesdelay includescontinues toward
How can secondary reinforcement alter the goal gradient as the learner moves through the delay toward the primary reward?

Supported and Obstructed Delays

Worked example

Comparing Two Delays

Compare two situations in which an action is followed by a delayed primary reward. In one situation, regularly occurring stimuli during the delay support secondary reinforcement. In the other, conditions obstruct that secondary reinforcement.

  1. 1
    Situation A

    The delay includes regularly occurring stimuli that can favor secondary reinforcement. A more immediate reinforcing influence is therefore available during the interval.

  2. 2
    Situation B

    The conditions obstruct secondary reinforcement. The learner must rely more heavily on the primary reward at the end of the delay.

  3. 3
    Prediction from Hull's account

    Learning should decrease with increased delay less in Situation A than in Situation B, because the two delays provide different opportunities for secondary reinforcement.

Result

The relevant comparison is not simply delay versus no delay. It is a delay with conditions favoring secondary reinforcement versus a delay in which secondary reinforcement is obstructed.

Actionfollowed by delayActionfollowed by delayRegular stimulisecondary reinforcementsupportedObstructionsecondary reinforcementblockedPrimary rewardlater outcomePrimary rewardlater outcomefollowed during delay byprecedesfollowed during delay byprecedes
What differs when intermediate reinforcing stimuli are available during the delay versus blocked or absent?

Mistakes in Comparing Delays

  • Common mistake

    Treating the delayed primary reward as the only influence during the interval.

    What it looks like

    Assuming that nothing relevant happens between the action and the final reward.

    Why it fails

    Hull's account proposes that regularly occurring stimuli during the delay can favor secondary reinforcement.

    Do this instead

    Inspect the delay for stimuli that may provide a more immediate reinforcing influence.

  • Common mistake

    Treating every stimulus during the delay as automatically reinforcing.

    What it looks like

    Concluding that the mere presence of an intermediate stimulus proves that secondary reinforcement has developed.

    Why it fails

    The source says that regularly occurring stimuli can favor secondary reinforcement; it does not say that every stimulus necessarily becomes reinforcing.

    Do this instead

    Describe the stimulus as a condition that can support or favor secondary reinforcement.

  • Common mistake

    Comparing a delay only with no delay.

    What it looks like

    Asking whether learning is worse with a delay, without asking what occurs during that delay.

    Why it fails

    The important comparison is between delays with different opportunities for secondary reinforcement.

    Do this instead

    Compare a delay that supports secondary reinforcement with a delay in which secondary reinforcement is obstructed.

  • Common mistake

    Confusing secondary reinforcement with the primary reward.

    What it looks like

    Calling an intermediate stimulus the final reward simply because it occurs before the primary reward.

    Why it fails

    Secondary reinforcement is the more immediate influence during the interval, whereas the primary reward remains the later outcome.

    Do this instead

    Keep the intermediate reinforcing influence and the delayed primary reward conceptually separate.

  • Common mistake

    Ignoring the role of stimulus traces.

    What it looks like

    Explaining the entire extension of the goal gradient using secondary reinforcement alone.

    Why it fails

    The source presents secondary reinforcement as working in conjunction with molar stimulus traces.

    Do this instead

    Include both the lingering stimulus traces and the secondary reinforcement proposed to extend the gradient.

Apply the Distinction

MEDIUM

A response is followed by a delay and then by a primary reward. During the delay, regularly occurring stimuli are present in one condition but secondary reinforcement is obstructed in another. Explain which condition should show the smaller decrease in learning as the delay increases, and identify the mechanism that accounts for the difference.

Hints
  • Look for what occurs during the delay, not only for what occurs at its end.
  • Identify whether the conditions favor or obstruct secondary reinforcement.
  • Use the comparison between the two delays rather than comparing either delay with no delay.
Worked example

Practice Answer

Which condition should show the smaller decrease in learning as the delay increases?

  1. 1
    Identify the supported condition

    The condition with regularly occurring stimuli provides an opportunity for secondary reinforcement during the delay.

  2. 2
    Identify the obstructed condition

    The other condition lacks that opportunity because secondary reinforcement is obstructed.

  3. 3
    Compare learning across longer delays

    Learning should decrease less in the supported condition than in the obstructed condition, because the supported condition includes a more immediate reinforcing influence during the interval.

Result

The condition favoring secondary reinforcement should show the smaller decrease in learning as delay increases.

Key Takeaways

  1. 1
    A delayed primary reward does not have to be the only influence operating during a delay.
  2. 2
    Regularly occurring stimuli during the delay can favor the development of secondary reinforcement.
  3. 3
    Secondary reinforcement provides a more immediate reinforcing influence and can extend a goal gradient across time.
  4. 4
    Hull's proposal connects secondary reinforcement with molar stimulus traces in explaining how a goal gradient can span time.
  5. 5
    Learning decreases with increased delay less when conditions favor secondary reinforcement than when those conditions obstruct it.

At a glance

  • Secondary reinforcement can operate during the interval between an action and a delayed primary reward.
  • Regularly occurring stimuli can support this secondary reinforcement and provide a more immediate influence during the delay.
  • Together with molar stimulus traces, secondary reinforcement can extend a goal gradient beyond the period that traces alone would cover.
  • The effect of delay depends partly on whether conditions support or obstruct secondary reinforcement.
  • The key comparison is between delays with different opportunities for secondary reinforcement, not merely between delay and no delay.

Related Topics

Related concepts

More concepts starting with G