For me this starts happening when I have a lot ship/planet/whatever, and at the same time the ship that is moving changes the FOW.
When I make a ship with a huge sensor range, and move ships while they stay in that sensor range everything is smooth. "Luckily" you can make a huge-scale ship with only sensors that can single handedly remove the FOW from an entire insane sized map.