In the TRBM ( Fig. 1D; see also Fig. 4.1) the temporal dependence is modelled by a set of weights connecting the hidden layer activations at previous steps in the sequence to the current hidden layer representation. The TRBM and CRBM have proven to be useful in the modelling of temporal

data, but each again has its drawbacks. The CRBM does not separate the representations of form and motion. Here we refer to form as the RF of a hidden unit in one sample of the dataset and motion as the evolution of this feature over multiple sequential samples. This drawback makes it difficult to interpret the features learnt by the CRBM over time as the two modalities are mixed. The TRBM explicitly separates representations of form and motion by having dedicated weights for the visible to hidden layer connections (form) and for the temporal evolution of these features (motion). Despite these benefits, the TRBM has proven AP24534 quite difficult to train due to the intractability of its probability distribution (see Fig. 4). In this work we develop a new approach to training Temporal Restricted Boltzmann Machines that we call Temporal Autoencoding (we refer to the resulting TRBM as an autoencoded TRBM or aTRBM) and investigate how it can be applied to modelling

natural image sequences. The aTRBM adds an additional step to the standard TRBM training, leveraging a denoising Autoencoder to help constrain the temporal weights in the model. Table 1 provides an outline BIBW2992 of the training procedure whilst more details can be found in Section 4.1.3. In the following sections we compare the filters learnt by the aTRBM and CRBM models on natural image sequences and show that the aTRBM is able to learn spatially and temporally sparse filters having response properties clonidine in line with those found in neurophysiological experiments. We have trained a CRBM and an aTRBM on natural image sequence data taken from the Hollywood2 dataset introduced in Marszalek et al. (2009), consisting of a large number of snippets from various Hollywood films. From the dataset, 20×20 pixel patches are extracted in sequences 30 frames long. Each patch

is contrast normalized (by subtracting the mean and dividing by the standard deviation) and ZCA whitened (Bell and Sejnowski, 1997) to provide a training set of approximately 350,000 samples. The aTRBM and CRBM models, each with 400 hidden units and a temporal dependency of 3 frames, are trained initially for 100 epochs on static frames of the data to initialize the static weights WW and then until convergence on the full temporal sequences. Full details of the models’ architecture and training approaches are given in the Experimental procedures section. The static filters learned by the aTRBM through the initial contrastive divergence training can be seen in Fig. 2 (note that the static filters are pre-trained in the same way for the CRBM and aTRBM, therefore the filters are equivalent).