The vanishing gradient problem
Hochreiter showed formally why deep and recurrent networks do not train: the error signal decays exponentially with each layer backwards.
Why it matters
The main reason networks failed was explained, fifteen years before it was worked around.
On the backward pass the error is multiplied by the product of an activation derivative and a weight; when that factor is smaller than one in absolute value the signal decays exponentially with the number of steps, and when it is larger the signal explodes. The problem is therefore neither missing compute nor missing data but the arithmetic of the method itself. Written in German as a diploma thesis, the work went largely unnoticed by the English-language field; Hochreiter retold its analysis in the 1997 LSTM paper, which calls Bengio's 1994 analysis "very similar, more recent". The word "vanishing" does not occur in the thesis itself: the name is later.