Error message

Notice: unserialize(): Error at offset 0 of 14 bytes in variable_initialize() (line 919 of /home/datasc6/public_html/includes/bootstrap.inc).

Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations

Behnam Neyshabur, Yuhuai Wu, Ruslan Salakhutdinov, Nathan Srebro
May, 2016
Abstract:
We investigate the parameter-space geometry of recurrent neural networks (RNNs), and develop an adaptation of path-SGD optimization method, attuned to this geometry, that can learn plain RNNs with ReLU activations. On several datasets that require capturing long-term dependency structure, we show that path-SGD can significantly improve trainability of ReLU RNNs compared to RNNs trained with SGD, even with various recently suggested initialization schemes.

Resource Type: