Sequence Data

A sequence is more than a collection of values. It is a collection whose order carries information. Numerical timeseries data consists of measurements arranged in sequence at regular time intervals. A weather station, for example, records measurements, waits for a fixed interval, records them again, and continues this process over time. The position of each measurement in the sequence tells us when it occurred relative to the others.

The weather dataset comes from a station at the Max Planck Institute for Biogeochemistry in Jena, Germany. It contains 14 weather-related quantities recorded every 10 minutes. The example uses data from 2009 through 2016, although the original data reaches back to 2003.

At each time point, the dataset can be viewed as a group of aligned measurements: the weather quantities recorded at that moment. These quantities include air temperature, atmospheric pressure, humidity, wind direction, and other weather measurements. The timestamp places that group on the time axis, while the measured quantities describe the weather at that point.

The temperature-forecasting task gives the model a few days of recent weather data and asks it to predict air temperature 24 hours in the future. This creates two different roles. The recent measurements are the historical input supplied as evidence. The later air-temperature measurement is the future target that the model must estimate.

Separating the input history from the forecast target makes the task precise. It is not an undefined question about weather. It is a sequence-to-future-value problem: use an ordered window of recent measurements to estimate a specified later value.

A single weather record contains measurements taken at one moment. The sequence view adds the relationship between that record and the records before and after it. Because the station records measurements repeatedly at a fixed interval, neighboring records are connected by their positions in time. A temperature value can therefore be interpreted as one point in an ordered history rather than only as an isolated number.

This definition uses the same sequence idea in a different domain. In weather data, positions correspond to measurement times. In language, positions correspond to earlier and later tokens. The task is not necessarily to memorize one fixed continuation. The model represents statistical structure and uses the preceding sequence to determine which next tokens are more or less probable.

A trained language model can generate a sequence without writing the entire sequence in one step. It begins with a short piece of text called conditioning data. The model uses that initial sequence to predict a next token. The predicted token is appended to the sequence, and the enlarged sequence is supplied for another prediction.

This repeated feedback is the central mechanism. The first prediction has the conditioning data as context. The next prediction has the conditioning data plus the first generated token as context. Repeating the cycle can produce a sequence of arbitrary length.

Language modeling is the general idea of predicting a next token from previous tokens. Character-level language modeling chooses individual characters as the prediction unit. In the LSTM character model described here, the model receives strings containing N characters and is trained to predict character N + 1.

The output of this LSTM character model is a softmax over all possible characters. That output forms a probability distribution for the next character. The model can then use the selected character as part of the enlarged sequence for the next prediction.