So why did the camera break when setting the same position twice?
Because the vector calculations that control the camera orbit take two motion endpoints and a target point, calculates the surface normal and corner angle of the resulting triangle, and uses that as an axis of rotation for a vector that pivots around the target in an arc passing between the two other corners, and you can't take a surface normal for a triangle with an angle of zero, because that's not a triangle.