"Just seemed disturbing he couldn't find someone his own age."

I don't think it was ever established that he couldn't. For all we know, he might have had quite an active romantic life going on behind the scenes during 1963-1975. I don't seem to recall that possibility being explicitly ruled out. In fact I'm inclined to think it's likely to be the case, considering how easily he was able to keep his hands off Jean; he probably had other avenues to exercise himself romantically and was able to leave Jean for the boys.

Besides which, I think your premise is flawed anyway. If you happen to fall in love with someone twenty years younger than yourself, or if you happen to fall in love with someone twenty years older than yourself, that's got nothing to do with "couldn't find someone his own age", unless one is asserting you're falling back to the older/younger person as a sort of a "Plan B" after you've been rejected by everyone your own age. And I never got that kind of a vibe from Xavier, that he was resorting to settling for Jean because everybody his age had rejected him.

It's important to remember that falling in love isn't a choice. Xavier didn't choose to fall in love with Jean. These things just happen. The choice is in what we do about it. And Xavier's choice was to keep his hands off -- the ethical choice.