Not counting the Silver Surfer, who once teamed with Subby and Hulk AGAINST the Avengers...
Regarding the original 3: Dr. Strange (as well as Luke Cage) was a Defender first and not an official Avenger until this century... in a group a lot of longtime fans complained was too much like the Defenders (though Doc did party with Roy Thomas' Avengers after Jan's wedding). Sub-Mariner, the second founding father of the Non-Team, was an adversary of the Avengers in the 60s (Av #3, 40). The third guy, the Hulk, was an original Avenger, but also fighting them by the third ish.