Orthogonal transformations and gradient updates

We show that if the contour lines of a function are symmetric with respect to some rotation or reflection, then so is the evolution of gradient descent when minimising that function. Rotation of the space on which the function is evaluated effects a corresponding rotation of each of the points visited under gradient descent (similarly, for reflections).

This ultimately comes down to showing the following: if $f:{\mathbb{R}}^N\to \mathbb{R}$ is the differentiable function being minimised and $g$ is a rotation or reflection that preserves the contours of $f$, then

$$\mathrm{\nabla }{|}_{g(u)}f=g(\mathrm{\nabla }{|}_{u}f)$$

for all points $u\in {\mathbb{R}}^N$.

Examples

We consider below three one-dimensional examples that demonstrate that, even if the function $f$ is symmetric with respect to all orthogonal transformations, it is necessary that the transformation $g$ be orthogonal in order for the property ($\text{???}$) above to hold.