Texture generation

The reference image supplies visual qualities such as textures, colors, and patterns.

Programming

Two Images, One Generated Result

Neural style transfer is a two-source image-modification task. It begins with a target image and a reference image, then produces a generated image intended to combine information from both. The target image supplies the subject and larger arrangement that should remain recognizable. The reference image supplies visual qualities such as textures, colors, and patterns.

How do the target and reference images contribute different information to the generated image?

Tracing Content and Style

Worked example

Separating the Two Contributions

Suppose a generated image should keep the recognizable subject and broad arrangement of a target image while taking its visual character from a reference image.

1
Identify content
Look for the higher-level macrostructure: the broad arrangement and recognizable subject that should remain identifiable.
2
Identify style
Look for textures, colors, and visual patterns. Style can include patterns that appear at different spatial scales.
3
Combine the contributions
The intended output keeps the target's content while reproducing the reference's style.

Result

The target answers what the image is broadly about; the reference answers how that image should visually appear.

Which visual structures come from the target image, and which visual qualities come from the reference image?

Turning Visual Goals into Losses

The algorithm needs a measurable way to express what the generated image should achieve. A loss function provides that measurement. Neural style transfer uses two kinds of distance: style distance and content distance.

Definition

Content loss measures how closely the generated image's content representation matches the target image's content representation.

Definition

Style loss measures how closely the generated image's style representation matches the reference image's style representation.

How do content loss and style loss each measure whether the generated image matches its intended goal?

The representations are important because the algorithm is not defining content and style only by looking at the ordinary image description. Deep convolutional neural networks provide representations that allow these two ideas to be defined mathematically. Once those representations exist, the generated image can be compared with the target for content and with the reference for style.

Following the Optimization Goal

Neural style transfer follows a general deep-learning pattern: describe the desired result with a loss function, then minimize that loss. In this case, minimizing the combined loss is intended to make the generated image's content representation closer to the target's content representation and its style representation closer to the reference's style representation.

What changes in the generated image as optimization reduces content loss and style loss together?

Worked example

Reading the Combined Objective

Explain what the algorithm is trying to improve when it evaluates a generated image.

1
Check content
Compare the generated image's content representation with the target image's content representation. This comparison expresses whether the target's subject and broad arrangement are being retained.
2
Check style
Compare the generated image's style representation with the reference image's style representation. This comparison expresses whether the reference's textures, colors, and patterns are being reproduced.
3
Combine the checks
Use both distances as the loss-based description of the desired result rather than treating content or style as the only goal.
4
Minimize the combined loss
Adjust the generated result toward the two intended matches: target content and reference style.

Result

The optimization goal is a generated image that is close to the target in content representation and close to the reference in style representation.

Preservation and Transformation

Content preservation and style transformation describe two sides of the same task. If the result is judged mainly by its content match, the target's recognizable subject and broad arrangement are the central concern. If it is judged mainly by its style match, the reference's textures, colors, and visual patterns are the central concern. Neural style transfer expresses both aims together through content distance and style distance.

What changes when the intended result emphasizes preserving target content versus applying reference style?

Edge case

Do not interpret a strong style match as a replacement for content matching. The intended generated image combines the two contributions: target content and reference style. Considering only one side leaves out part of the stated objective.

Mistakes About Style Transfer

Common mistake
Treating the reference image as the source of the subject
What it looks like
Assuming that the generated image should copy the reference image's recognizable subject.
Why it fails
The target supplies the subject and larger arrangement. The reference supplies style.
Do this instead
Ask which image supplies content and which supplies visual qualities. Target means content; reference means style.
Common mistake
Treating style as only color
What it looks like
Describing style transfer as a process that changes colors but not textures or patterns.
Why it fails
Style includes textures, colors, and visual patterns, including patterns at different spatial scales.
Do this instead
Include all three categories when explaining style.
Common mistake
Describing style and content without representations
What it looks like
Saying that the algorithm compares vague visual impressions without explaining how the comparison becomes measurable.
Why it fails
Deep convolutional neural networks provide representations that allow style and content to be defined mathematically.
Do this instead
Explain that the generated image is compared with the target and reference through content and style representations.
Common mistake
Using only one loss
What it looks like
Evaluating a generated image only by whether it resembles the target.
Why it fails
The stated goal has two parts: matching the target's content representation and the reference's style representation.
Do this instead
Describe the combined loss as containing both content distance and style distance.

Check Your Understanding

EASY

A generated image keeps the recognizable subject and broad arrangement of one image but adopts the textures, colors, and patterns of another. Identify which image is the target, which is the reference, what content loss measures, and what style loss measures.

Hints

The image supplying the recognizable subject and broad arrangement is the target.
The image supplying textures, colors, and patterns is the reference.
Content loss compares content representations.
Style loss compares style representations.

Your attempt (not saved)

What do you think happens?

Before checking the answer, predict what minimizing the combined loss is intended to do.

Make the generated image match only the target image
Make the generated image match only the reference image
Move the generated image toward the target's content and the reference's style
Measure the images without changing the generated result

Reveal answer

Answer:

Move the generated image toward the target's content and the reference's style

The combined loss includes content distance and style distance. Minimizing it is intended to make the generated image's content representation closer to the target's and its style representation closer to the reference's.

Key Takeaways

1
Neural style transfer uses two sources: the target supplies content, and the reference supplies style.
2
Content is the higher-level macrostructure, including the recognizable subject and broad arrangement.
3
Style consists of textures, colors, and visual patterns, including patterns at different spatial scales.
4
Content loss compares the generated image's content representation with the target's; style loss compares its style representation with the reference's.
5
Minimizing the combined loss is intended to produce an image that preserves the target's content while reproducing the reference's style.

At a glance

Neural style transfer combines target content with reference style.
Content describes higher-level structure, while style describes textures, colors, and visual patterns.
Content loss and style loss make the two goals measurable through image representations.
Deep convolutional neural networks provide representations for defining content and style mathematically.
Minimizing the combined loss is intended to create a generated image that matches the target in content and the reference in style.

Related concepts

More concepts starting with T