We show that if the contour lines of a function are symmetric with respect to some rotation or reflection, then so is the evolution of gradient descent when minimising that function. Rotation of the space on which the function is evaluated effects a corresponding rotation of each of the points visited under gradient descent (similarly, for reflections).
This ultimately comes down to showing the following: if is the differentiable function being minimised and is a rotation or reflection that preserves the contours of , then
for all points .
We consider below three one-dimensional examples that demonstrate that, even if the function is symmetric with respect to all orthogonal transformations, it is necessary that the transformation be orthogonal in order for the property (1) above to hold.