This all depends upon what metrics you want to use to rate the pitchers. The Yanks have never been known for their pitching. Heck, probably the best pitcher they ever had, was moved to Rightfield.. some nobody by the Name of Babe Ruth.
If you Rank by wins, you have Whitey Ford, Red Ruffing and Andy Pettitte.
If you are one of those new fangaled statheads, you might use WHIP. Then your list of top 3 looks like Mariano Rivera, Earnest Bonham and Jack Chesbro.
You start looking at ERA, logically this should favor the old time players and seems to... except at the top of the list is Mariano Rivera, with Russell Ford and Chesbro following behind. (Note: R. Ford and Chesbro pitched for the Highlanders in the early 1900's... just shows how dominate Mo really is.)
So, how do you want to rank them? The Yanks have never really had that one pitcher who dominated to that degree except for the Babe, and they moved him away from the mound.