It all depends how old you are when you are drafted. The kids normally take 5+ years, but a lot of college players only take 3 or 4 since they are ready to play full season baseball right away. 5 years probably isn't a bad average though overall.
That 5 year average is why it is so hard to take over a team and build it up from scratch in baseball. You can't just have 1 or 2 good drafts and be good like in other sports. Even doing it in 5 years is rushed. Look at all the major trades AA needed after taking over for Ricciardi. He didn't even wait for the prospects. He traded most of them to expedite things and it still took 5 years.