Numbers of planets aren't much of an issue, its mainly the fact that when every planet has its tactical/logistics slots filled and there are trade ships all over the place with multiple race that it gets super bogged down.
Adding more and more planets (when in a late game scenario) just amplifies the problems. So really, you can make it as large as you want but will it take forever to finish a game? YES, will it eventually become super laggy at the end of a game? YES.