It becomes a valid point because Vince keeps forcing the issue. His failed forays into football, boxing PPV promotion, and bodybuilding (the key failures from "my" era of watching the WWF expand) make it easy to argue the "well, he's great at wrestling promoting but he sucks at the other stuff he wants to get into" point.
I would guess he's gotten "better" at his outside-wrestling-endeavours (movies, for example) since he has the means to surround himself with people that actually know how to make forays into the field(s), but I can see how the idea of Vince's "Genius" is hampered by those excursions outside of the WWE's base product for some people.
At the end of the day, for me, he cleaned out all his competition (some by his hand, some by the competition's own idiocy)...his genius is that he made himself the only real game in town, which allows him to pursue and succeed/fail in his other projects almost at his leisure. He has to get his due.