Loss Functions in Deep Learning

A lower combined loss is meaningful only if the two component comparisons are correctly assigned.

Programming

Two Images, One Result

Neural style transfer begins with two different roles. A content image supplies the subject and arrangement that should remain recognizable. A style image supplies the visual character that the generated image should adopt. The goal is a third image that combines both roles: it preserves the content of the original image while taking on the style of the reference image.

How do the content image and style image flow through the network and combine to produce the generated image?

Key point

The central question is not whether the generated image resembles one input exactly. It is whether the generated image is close to the content image in the content representation and close to the style image in the style representation.

Following the Two Comparisons

The loss function gives the system a way to judge the generated image. One comparison checks generated content against the original content image. A second comparison checks generated style against the reference style image. These comparisons are then combined into one objective. The generated image is adjusted so that both distances are driven down together.

How do content loss and style loss each influence the generated image, and how are they combined into one optimization objective?

Definition

Content loss measures how unlike the generated image is from the original content image when both are viewed through the content representation.

Definition

Style loss measures how unlike the generated image is from the reference style image when both are viewed through the style representation.

Representations from a CNN

The comparisons are possible because a deep convolutional neural network produces activations at its layers. Instead of judging the images only as raw visual inputs, style transfer uses these activations as representations. Different layers provide different kinds of information, allowing the content and style comparisons to focus on the properties they are intended to preserve or adopt.

What representations are produced at different CNN layers, and which layers are used to measure content versus style?

Why Upper Layers Preserve Content

Upper convolutional layers are used as a content representation because their activations capture more global and abstract information. This makes them useful for preserving what the image is about: its subject and arrangement. Fine visual details may change as the generated image adopts the reference style, while the higher-level information needed to recognize the original content remains the target of the content comparison.

How does the image representation change from lower to upper convolutional layers, and why do upper layers retain objects and structure despite losing fine details?

Key point

Upper-layer activations do not need to preserve every fine detail to preserve content. Their value is that they retain more global and abstract information, which is closer to the subject and arrangement that should remain recognizable.

Representing Artistic Style

Style loss does not ask whether the generated image copies the style image as a complete scene. It uses relationships among CNN feature activations to represent the reference image's visual character. The corresponding relationships in the generated image are compared with those from the style image. A smaller difference means that the generated image is closer to the reference style according to this representation.

How are feature activations transformed into a representation of style, and how does that representation compare the style and generated images?

Best practice

When explaining or designing the objective, keep the roles separate: compare generated content with the original content image, and compare generated style with the reference style image. Mixing these pairings changes what the loss asks the generated image to preserve.

A Complete Style-Transfer Trace

Worked example

Separating Subject from Visual Character

Suppose the content image shows a recognizable subject with a particular arrangement, while the style image supplies a distinct visual character. What should the loss comparisons require from a generated image?

1
Assign the content role
Treat the original image as the source of the subject and arrangement that should remain recognizable.
2
Assign the style role
Treat the reference image as the source of the visual character that the generated image should adopt.
3
Build the content comparison
Use a CNN-based content representation and compare the generated image with the original content image.
4
Build the style comparison
Use relationships among feature activations to represent style, then compare the generated image's representation with the style image's representation.
5
Combine the comparisons
Place content loss and style loss in one objective so the generated image is guided toward both goals at the same time.

Result

A successful result remains recognizable as the content image while adopting the visual character of the style image.

The important trace is the chain of responsibilities. The content image is not compared with the style representation, and the style image is not used as the target for content preservation. Each image is assigned a role, each role receives a representation, and the generated image is compared with the appropriate reference before the losses are combined.

Optimization as Repeated Correction

The generated image is optimized through repeated updates. After an update, the two comparisons can be checked again: does the generated image better match the original content in the content representation, and does it better match the reference style in the style representation? The objective is successful only when both distances are driven down together rather than improving one while ignoring the other.

What changes in the generated image after each optimization step as content and style losses decrease?

Debugging the Objective

A useful debugging strategy is to find the first place where the intended result diverges. Start with the desired roles, verify that each role has a corresponding representation, confirm that the generated image is compared with the correct input for each part of the loss, and then verify that the content and style distances are combined into one objective.

Common mistake
Treating the content image and style image as if they supplied the same information.
What it looks like
Using the reference style image as the target for preserving the original subject.
Why it fails
The two inputs have different assigned roles: one supplies content and the other supplies style.
Do this instead
Keep content comparisons tied to the original content image and style comparisons tied to the reference style image.
Common mistake
Using the wrong representation for content preservation.
What it looks like
Ignoring the role of upper-layer activations when trying to preserve the subject and arrangement.
Why it fails
Upper-layer activations capture more global and abstract information, making them useful as a content representation.
Do this instead
Use the upper-layer representation when the goal is to preserve higher-level image content.
Common mistake
Checking only one part of the objective.
What it looks like
Driving style loss down while failing to check whether the original subject remains recognizable.
Why it fails
The objective must drive both content and style distances down together.
Do this instead
Inspect both comparisons and confirm that they are combined into one objective.
Common mistake
Assuming that a correct network representation alone guarantees the intended result.
What it looks like
Selecting representations correctly but comparing the generated image with the wrong reference.
Why it fails
The generated image must be compared with the correct input for each part of the loss.
Do this instead
Trace each comparison from its generated representation back to the appropriate original or reference image.

Practice the Role Assignment

MEDIUM

A generated image keeps the subject from the original image but does not adopt the visual character of the reference image. Identify which role, representation, or comparison you would inspect first, and explain why.

Hints

Start by checking whether the style image has a corresponding style representation.
Then verify that generated style is compared with reference style.
Finally check that style loss is included in the combined objective.

Your attempt (not saved)

EASY

Explain why upper convolutional layers are useful for content preservation even though style transfer is intended to change the image's visual details.