Just from what I can remember, Lavin turned down NC State before he took the job at SJU. Just he figured it would be an even harder job recruiting against Duke/NC each year.
He turned down the job because he wanted to be in a city like New York, Chicago or LA.
Mostly New York or LA for his wife who was an inspiring actress at the time.