How Much is a Prospect Worth? A study

by Galt on Feb 6, 2008 11:50 AM EST Comment 19 comments

Really cool (though analytical) paper about the success, and therefore "worth" of prospects by Victor Wang.

It's from August, but I hadn't seen or heard about it before. While this is part of a subscription-only newsletter, I'm not concerned about this being copywrited, because in this Hardball Times Article (which is cool in and of itself): http://www.hardballtimes.com/main/article/the-bright-side-of-losing-santana/ the author links the original paper, here: http://www.philbirnbaum.com/btn2007-08.pdf

Basically, Wang looked at the prospects in the top 100 of BAs lists through every year in the 90s. And then looked at their performance in the first six years of their careers.

Looking only at top 10 prospects: For hitters, the likelihood of them becoming stars (based on WARP) was 17%, bust 21%. For pitchers it was 4% stars and over 50% busts. For prospects 11-25, the bust rate increased for both subgroups.

0 recs | Comment 19 comments

Story-email Email Printer Print

Comments

cool

that's a neat study. i hadn't seen that before.

of course, the fact that the study is based on the 1990s top prospect list goes a long way towards explaining the huge gap between hitter and pitcher prospect values. that's the decade that spawned the infamous TINSTAPP slogan due to the incredible rate of pitcher flameouts. i suspect that a more inclusive study would show a considerably smaller (though still significant) disparity.

by jpahk on Feb 6, 2008 12:06 PM EST reply actions 0 recs

no more TINSTAPP?

wait -- why would there be TINSTAPP any more? i can't tell which of these you're saying:

TINSTAPP never really existed; the 1990s had an anomalous amount of pitcher flameouts that didn't reflect inherent risk in pitchers
after the 1990s, we became aware of TINSTAPP, and our prospect lists have now properly adjusted pitchers downward
we have better methods now than we did a decade ago for determining which pitching prospect will succeed and which will fail
something else?

by bleedjaxblue on Feb 6, 2008 12:40 PM EST up reply actions 0 recs

okay

i obviously need to clarify what i was talking about. it's sort of a combination of 1 and 3. i think the 90's did have an anomalously high rate of pitcher flameouts. i also think we're better at identifying top pitching prospects than we were 15 years ago. i never really did believe in TINSTAAPP anyway, but i do think that many people still underestimate the risk associated with pitching prospects (relative to hitting prospects).

anyway, i don't know which is the anomaly--i obviously haven't done a historical study--but anecdotally, it seems to me that the "famous" pitching prospects of the 90's were by and large flameouts. brien taylor, ben mcdonald, steve avery, generation K, todd van poppel, etc... of course there are success stories too but they are somehow less memorable. meanwhile, in recent years we have seen an absolutely incredible collection of pitching talent hit the ground running in the show, the most spectacular example being the AL rookie class of '06 (liriano, verlander, weaver, papelbon). like i said, i don't know if this signals a sea change or if it's just a blip. obviously it's anomalous in that we aren't going to see another class like that for a while. but i think overall, things have gotten better since the 90's.

by jpahk on Feb 6, 2008 12:50 PM EST up reply actions 0 recs

thanks

wasn't sure which you meant. and those both seem possible to me, though i probably am prone to believe in TINSTAAPP a little more than you. after all, even of those four young pitchers you mentioned, they all have had durability questions -- if not major injuries -- this early in their career. i would hardly bet the house on any of them.

by bleedjaxblue on Feb 6, 2008 1:45 PM EST up reply actions 0 recs

True

As successful as those four guys have been, it is still best to keep in mind that Steve Avery was a two-time 18 game winner at the age of 23, and he was done by the age of 25.

by aCone419 on Feb 6, 2008 2:38 PM EST up reply actions 0 recs

another possibility

4. after being burned mercilessly throughout the 1990s, teams are now taking much better care of their young pitchers.

by wily mo on Feb 6, 2008 12:55 PM EST up reply actions 0 recs

by the way

in preparing for my fantasy league's minor league draft last month, i did a workup of the last five years' drafts (starting in 2001 or 2002, don't remember) broken down by HS hitters / HS pitchers / college hitters / college pitchers. it wasn't science or anything: i just made four spreadsheets, one for each player category. made each draft year one column. listed the players from that sheet's category (say, all the 2003 HS hitters) in the order they were drafted. then bolded the ones who'd done anything in the majors. then just looked at the spreadsheets and considered which ones looked better.

the subjective results:

HS hitters
HS pitchers

(gap)
3. college hitters

(large gap)

4. college pitchers

by wily mo on Feb 6, 2008 12:59 PM EST up reply actions 0 recs

of note

the strength of each category, of course, was highly variable by year. for example, there were one or two years where nearly all of the top HS pitchers have panned out (at least so far) - ironically, 2002 was one of these. then another year - whenever volstad was the top guy - there's been literally nobody who you would particularly care about. volstad is still the #1 guy from that draft, and he's just volstad.

by wily mo on Feb 6, 2008 1:04 PM EST up reply actions 0 recs

question

were the college players more predictable than the high school players (i.e., was the order in which college players were drafted closer to their ultimate value than the high school players)?

by bleedjaxblue on Feb 6, 2008 1:54 PM EST up reply actions 0 recs

yes

for hitters, i think so.

the #1 college hitter drafted is almost always the best or one of the best in the class (teixeira, gordon, etc. - i don't have the stuff in front of me, gordon isn't the perfect example since right now braun and maybe tulo look better. but he's definitly more or less as advertised.)

every once in a while there's that high school superstar (a-rod, griffey, the uptons), but generally HS hitters tend to produce studs at sort of a flat rate up and down the first round. the first half of the first round probably does a bit better than the second half of the first round, but then you have guys coming out of the sandwich (example - salty), second round (ex. - mccann) or even later (ex. - kemp) who are just as good as the guys at the top of the first.

also of note - the moneyball movement seemed to want to debunk the notion that high school players have higher ceilings than college players, but to my eye it's actually quite true that they do.

by wily mo on Feb 6, 2008 2:17 PM EST up reply actions 0 recs

sweet

yeah -- that was definitely my intuition as well.

as for the Moneyball-debunking, i'm in agreement as well. even if it weren't my intuition, i've never understood, logically, how it could be true. you'd figure the draft takes out, what, 75% of the "best" players graduating high school (totally making number up -- i'd actually be interested in knowing what it actually is, if anyone knows off-hand)? obviously, that leaves some players to reach a high ceiling. plus someone can always develop out of nowhere. but it always seemed to me there really SHOULD be more high-ceiling guys from high school, since those kids generally try to cash in soon, assuming anyway sees their ceiling as a possibility at all.

by bleedjaxblue on Feb 7, 2008 4:09 AM EST up reply actions 0 recs

good one

hadn't thought of that.

i guess it's hard to figure out what percentage of "failed" pitchers did so because of injury, since injuries aren't always that discrete, but i'd be curious to know how much the TINSTAAPP effect went away if you just looked at pitchers who "stayed healthy."

of course, part of TINSTAAPP for me, too, has been that there are more star pitchers that "come out of nowhere" than there are star hitters. i guess none of these explanations really offer anything toward why that might be true -- if it is true. (no real idea if it is -- i've tried looking at it a few times, and never really got a clear enough answer, since it's hard figuring out how to rank people NOT on a prospect list at all, or comparing how many years they were on the prospect list versus, for example, perennially being LOW on a prospect list.)

by bleedjaxblue on Feb 6, 2008 1:51 PM EST up reply actions 0 recs

+1

I think this is the biggest reason the gap has shrunk. It's still a much better bet to go with a hitter, but not nearly as much as ten years ago.

by rwperu34 on Feb 6, 2008 3:48 PM EST up reply actions 0 recs

Study I'd like to see...

I like the 3 categories of results: star/"okay"/bust.

I'd like to see a cross-chart of John's rankings and level of minor-league ball reached (say 50 IP/150 AB to avoid the 1-game guys), with the results being displayed as %'s as this author did... it would look like:

Rating MLB AAA AA A+ A A- Rk
A 50/45/5 40/35/15 30/35/25
A- 45/40/15 35/25/25 etc.
B+
B
B-
C+
C

IMO, the trickiest part of doing this would be treatment of players who are rated for multiple years, but I think that however they are handled, such a "report" would be very interesting.

by BobbyMac on Feb 6, 2008 12:25 PM EST reply actions 0 recs

The problem with a study like this

A red flag always comes up when a study uses historical BA top 100 rankings as an indicator of prospect success or failure

Different prospecting sites prefer or avoid different profiles of prospects, and Baseball America specifically loves to take risks on toolsy player which a lot to prove versus guys who are more of a sure thing. This naturally leads to a very different risk profile in their prospect lists than in John's list or BP's list. The classic example would be Dustin Pedroia, who was ranked pretty low by BA but much higher by BP (and I believe john). Someone who was looking for a sure thing would take rank Pedroia higher. Someone looking to say "see, we ranked that superstar higher than everyone else" would naturally push Pedroia down in favor of Triunfel or Beltre or similar styled prospects.

Thus, it's not really fair to use this study as "How much is a prospect worth", but more of "How often has Baseball America been right and what kind of risk profile does their list have?" Calling this study a prospect valuation study begins with the assumption that BA's prospect rankings accurately capture the values placed on prospects by the entire baseball community, which to me is demonstrably false.

-1 and only member of the Jed Lowrie fan club!

by Jgaztambide on Feb 6, 2008 12:36 PM EST reply actions 0 recs

+1

1941 .406

by FrozenTed9 on Feb 6, 2008 12:40 PM EST up reply actions 0 recs

just to be clear

BP didn't rank Pedroia highly, BP's Pecota (and Nate SIlver) did. Kevin Goldstein was even more down on Pedroia than BA was (ranked him as an 'average' prospect) and I don't think even had him in his top 100, if memory serves.

John ranked Pedroia as a 'B'. I wouldn't say he nailed it, but closer than BA and BP/Goldstein -- but it's BP/Pecota that can take a bow on that one.

by scooter on Feb 7, 2008 2:06 AM EST up reply actions 0 recs

How much...

...would a prospect be worth if he had to declare for the draft like they do in the NBA and NFL?

by HumboltThunderbolt on Feb 6, 2008 12:57 PM EST reply actions 0 recs

in the NFL

this would be very interesting because of the megacontracts in the top of the first round. there may even be losses with the top picks, and especially those that don't pan out (ryan leaf and tim couch stand up)

when your ERA has letters and not numbers, it's a bad thing- kevin goldstein

by WakeboardJock on Feb 7, 2008 10:48 AM EST up reply actions 0 recs