It likely has nothing to do with the FOV. There has been a long standing bug where the camera zoom was not being initialized properly causing a sudden shift in some cases. That should be fixed in tomorrow's hot fix.It regularly messes up the camera distance and then forces me to readjust it by zooming further in or out (the bug can cause both - the camera suddenly zooming further in or out).
I didn't have these problems as long as I left the FOV slider at the default settings - the camera distance always stayed the same.