Citizenship

Hi everyone. Florida resident here. I did my fingerprints on May 1st and on May 8th it said my case is in line for interview. I'm planning to move to California later this year do I have to let them know when I move? I planned on just flying back for both the interview and the oath ceremony since I will still have family living at the address they have on file they would let me know when my interview letter arrives as well as the oath ceremony letter. When I called to ask this same question the man I spoke with said uscis wouldn't like that very much if I moved and didn't let them know. But it seemed like an odd response and almost like his own opinion and not fact?? Also if I do update my address when I move and they transfer my file to California will that delay my process? That's honestly the main reason I don't want to.