I've seen the word kayfabe creeping into non-wrestling usage recently, especially among young left-wing people on social media actually. I think to a degree some of those terms have just crossed over into popular culture, or certainly online culture at least. It's harder than ever to tell what a casual fan is. Because of the proliferation of insider language on social media etc. there are people that we definitely wouldn't consider to be 'knowledgeable fans' who are appropriating this language. For example, I've seen tweets from people who love Dean Ambrose and think he's the best ever talking about 'four star matches' etc. without any sense of this meaning anything beyond being a term of praise for wrestling matches. As wrestling becomes dragged further and further into the realms of internet culture and nerd fandom, it becomes less and less possible (and potentially relevant) to tell what a casual fan is.
I think the term casual fan in the way it's employed is in a sense virtue signalling. It's used by the sort who think John Cena and Roman Reigns are the worst wrestlers in history and such nonsense and thus 'casual fan' is used as a distinguishing term (along with 'women and kids' because they are homogeneous blocks) for those who love Rollins, Owens, Balor etc. to express their sense of superiority.