Concepts / Weighted Sums in Neural Network Neurons

Weighted Sums in Neural Network Neurons

Activation functions determine how a neuron converts its scalar input into an output.

  • Programming
Interactive lab

Try it: Neural Network Forward Pass

How a neural network turns inputs into an output: each neuron computes a weighted sum plus bias, applies an activation function, and passes the result on.

How it works

  1. Each hidden neuron computes z = w1·x1 + w2·x2 + b.
  2. It applies an activation: ReLU keeps positive values and zeroes negatives; sigmoid squashes z into 0–1.
  3. The output neuron does the same with the hidden activations as its inputs.
  4. Changing any input, weight or bias changes every value downstream of it.

Default run (7 steps): Inputs x1 = 1, x2 = 0.5. Activation: ReLU. … Prediction y = ReLU(1.25) = max(0, 1.25) = 1.25.

Simplified: Tiny 2-2-1 network with hand-set weights, not a trained model. No training happens here. Activations offered: ReLU, sigmoid and sign (texts differ on the value of sign at exactly 0; here it is 0).

Educational simulation

Loading the simulation…

From Predecessors to Output

A neuron does not send the combined influence of its incoming signals directly to its output. First, the neuron receives a scalar weighted sum from the connected predecessor neurons. It then applies an activation function to that single scalar input. The activation function is the rule that converts the neuron's combined input into its output.

Predecessor outputsconnected neuronsWeighted sumscalar input aActivation functionσ(a)Neuron outputweighted contributionssingle scalarconverted value
How does the scalar weighted sum flow through the activation function to produce the neuron's output?

The Neuron's Internal Sequence

  1. 1
    Outputs arrive from neurons connected to the receiving neuron.
  2. 2
    The receiving neuron combines those connected outputs into a weighted sum.
  3. 3
    That weighted sum is treated as one scalar input, represented here by a.
  4. 4
    The selected activation function receives a as its input.
  5. 5
    The activation function produces the neuron's output.
Predecessor 1outputWeighted sumaActivation functionσ(a)OutputPredecessor 2outputPredecessor Noutputweighted contributionweighted contributionweighted contributionone scalar inputtransformed value
Does the activation function act on each incoming value separately, or on the combined weighted sum after the inputs are aggregated?

Tracing a Scalar Input

Worked example

Following a Weighted Sum Through a Sign Function

Suppose connected predecessor neurons provide outputs that, after their weighted contributions are combined, give the receiving neuron the scalar input a. Determine what operation happens next when the chosen activation function is the sign function.

  1. 1
    Aggregate

    The predecessor outputs and their weights first produce the single scalar weighted sum a.

  2. 2
    Apply the function

    The sign activation function is applied to that scalar, using the rule σ(a) = sign(a).

  3. 3
    Interpret the result

    The output is determined by whether a is negative, positive, or at the sign boundary. It is not determined by applying the sign function independently to each predecessor output.

Result

The receiving neuron follows the sequence weighted predecessor outputs, scalar input a, sign transformation, neuron output.

Formula
Negative anegative signSign outputZero aboundary casePositive apositive signnegative sign-based resultconvention requiredpositive sign-based result
What output does the sign function produce when the scalar weighted-sum input is below, at, or above the sign boundary?

Three Activation Behaviors

FunctionBehavior described in the sourceWhat happens around a cutoff or sign boundary
SignBases the output on the sign of the scalar inputMakes a sign-based transformation rather than a gradual one
ThresholdMakes a cutoff-based transformationChanges behavior at a threshold
SigmoidUses a smooth formula that approximates threshold behaviorChanges gradually rather than reproducing an abrupt cutoff exactly

The source distinguishes these functions by how they transform the single scalar weighted-sum input.

The sign and threshold functions are both decisive transformations, but they use different ideas. The sign function looks at the sign of its scalar input. The threshold function uses a cutoff. The sigmoid function is different in shape: it uses a smooth formula whose behavior approximates the threshold function rather than making the same abrupt cutoff.

Signsign-basedScalar input aweighted sumSign-based outputThresholdcutoff-basedCutoff-based outputSigmoidsmooth approximationSmooth outputinputinputinputsign transformationcutoff transformationsmooth transformation
How do the sign, threshold, and sigmoid functions differ in their input-output behavior across the same range of scalar inputs?

Why Sigmoid Is Smooth

A threshold function is described as cutoff-based: its behavior changes at a threshold. The sigmoid function is described as a smooth approximation because it captures threshold-like behavior without making that change as an abrupt jump. As the scalar input moves through the transition region, the sigmoid changes gradually.

Thresholdabrupt cutoffScalar input aCutoff outputSigmoidsmooth transitionScalar input aGradual outputsame scalar inputabrupt changesame scalar inputgradual change
How does the sigmoid's gradual transition around the threshold compare with the threshold function's abrupt jump?

Boundary Cases and Conventions

  • Common mistake

    Applying the activation function to each predecessor output separately

    What it looks like

    Treating every incoming value as though it independently becomes a neuron output before the incoming values are combined

    Why it fails

    The neuron first receives a weighted sum from its connected predecessor neurons and then applies the activation function to that single scalar input

    Do this instead

    Trace the weighted sum first, then apply the selected activation function to the resulting scalar

  • Common mistake

    Treating the weighted sum as the neuron's final output

    What it looks like

    Stopping the trace immediately after obtaining a

    Why it fails

    The activation function still has to convert the scalar weighted sum into the neuron's output

    Do this instead

    Continue from a to σ(a)

  • Common mistake

    Calling the sign function gradual

    What it looks like
    Describing sign(a) as a smooth transition
    Why it fails

    The source describes the sign function as a direct sign-based transformation rather than a gradual one

    Do this instead

    Associate gradual behavior with the sigmoid description, not with the sign function

  • Common mistake

    Assuming that every threshold convention is specified automatically

    What it looks like

    Assigning a particular output to an input exactly at the threshold without knowing the definition being used

    Why it fails

    The source describes threshold behavior as cutoff-based but does not specify the exact output convention at the boundary

    Do this instead

    Use the threshold definition supplied for the problem; if none is supplied, identify the boundary case instead of inventing its exact output

Edge case

For the sign function, a negative scalar input has a negative sign and a positive scalar input has a positive sign. The source does not specify the convention for an input of exactly zero, so that boundary output should not be guessed. Similarly, a threshold function requires its threshold and boundary convention before an exact numerical output can be determined.

Trace It Yourself

EASY

A receiving neuron gets outputs from connected predecessor neurons. Their weighted contributions combine to form a scalar input a. Trace the neuron using the sign function. What should you inspect first, and what transformation is applied after the scalar has been formed?

Hints
  • Begin with the combined scalar weighted sum, not with separate activation outputs for each predecessor.
  • Use σ(a) = sign(a) after the weighted sum has been formed.
  • If the value of a is not given, describe the result in terms of whether a is negative, zero, or positive.
MEDIUM

Compare two descriptions: one function makes a cutoff-based transformation, while another uses a smooth formula that approximates that behavior. Identify the threshold and sigmoid descriptions, then explain the difference in the transition around the cutoff.

Hints
  • Threshold is associated with cutoff-based behavior.
  • Sigmoid is associated with a smooth approximation.
  • The key contrast is abrupt change versus gradual change.

Key Takeaways

  1. 1
    A neuron first receives a scalar weighted sum from connected predecessor outputs.
  2. 2
    The activation function is applied to that one scalar weighted sum, not separately to each incoming value.
  3. 3
    The sign function performs a direct sign-based transformation.
  4. 4
    The threshold function performs a cutoff-based transformation.
  5. 5
    The sigmoid function uses a smooth formula that approximates threshold behavior through a gradual transition.

At a glance

  • Incoming predecessor outputs and their weights form a single scalar weighted sum.
  • The activation function converts that scalar into the neuron's output.
  • Sign behavior depends on whether the scalar input is negative, zero, or positive, with the zero convention requiring a definition.
  • Threshold behavior is cutoff-based, while sigmoid behavior is a smooth approximation to that cutoff behavior.
  • To trace a neuron correctly, aggregate first and activate second.

Related Topics

Related concepts

More concepts starting with W