BP take on prospect rankings

by SLK on Feb 8, 2006 3:35 PM EST Comment 33 comments

I found Nate Silver's piece on prospect rankings very interesting and thought others may wish to take a look. This is an open article, and does not require a subscription...

http://www.baseballprospectus.com/article.php?articleid=4750

I think I have always had the view expounded here regarding prospect rankings.

Of course this is also why I like John's approach. He has a general system he tries to follow. So players in A ball are viewed a little more skeptically, often because their projections are similar to what older players in AAA are currently doing. I appreciate the objectivity. (Although I generally adjust John's grades downward some for when he ranks AL Central prospects - I've felt Detroit, Minnesota and KC tend to be the Lucy to John's Charlie Brown on many occasions...=)

0 recs | Comment 33 comments

Story-email Email Printer Print

Comments

John's biases

(Although I generally adjust John's grades downward some for when he ranks AL Central prospects - I've felt Detroit, Minnesota and KC tend to be the Lucy to John's Charlie Brown on many occasions...=)

I think John works really hard to filter out his biases on prospects. I remember regularly e-mailing him when he worked for ESPN.com complaining about the dearth of writing related to Twins prospects.

His response was, [paraphrase] "I probably overcompensate for the fact that I'm a Twins fan and, thus, neglect them more than any other system."[/paraphrase]

cmathewson

by cmathewson on Feb 8, 2006 4:01 PM EST reply actions 0 recs

I kid because I care

Oh I know John tries to account for this and generally does a good job. He hasn't been the only one to fall for the prospects of Middle America. Being a long time reader of John's as well as former employees of Bill James, it's just hard not to see the Charles Schultz-esque humor when it becomes time to rank the prospects of the Twins, Tigers and Royals. (Not to say those teams haven't produced good or even great prospects - the Twins in particular - but some of their flops have been of dramatic proportions)

by SLK on Feb 8, 2006 4:23 PM EST up reply actions 0 recs

Pecota

Pecota may have some highly sophisticated uses that are lost on those of us not drowning in statistical analysis. But the projections/predictions they have put out are basically useless.

I'm not sure how you would compare them to BBA or John. But his basic point, that a AAA player is more valuable than a AA player ignores the reality that BBA is not ranking players, its ranking prospects.

The odd thing is that the further a prospect is from the big leagues the more uncertainty there is about his ability. When a player gets to AAA and is 25 years old the projections of upside and downside start to converge. If the point they converge at is fourth outfielder, they probably aren't a very high prospect. And at AAA the downside is still not making the major leagues at all.

On the other hand a 23 year old at AA with the same projection still has a much higher upside. But his downside if he doesn't make the major leagues his downside is exactly the same as the AAA player who doesn't make the major leagues.

by TT on Feb 8, 2006 5:01 PM EST reply actions 0 recs

agreed

I think with some adjustment PECOTA could get into the prospect ranking game, but not in this form. Like you said, it penalizes heavily for uncertainty. With prospects, uncertainty has to be accepted to some degree. If Silver puts together a top 100 using this method, I'll bet it's 80+% hitters just because there's more risk with pitchers.

Erick Aybar is not a better prospect than Francisco Liriano. He's just not.

by jeck on Feb 8, 2006 5:52 PM EST up reply actions 0 recs

I don't think...

...That Nate Silver would say that Aybar is better than Liriano. If you pay attention to how BP does their prospect rankings, they take into account how PECOTA projects players, but that is not the end all be all. In fact, it is more just to help differenciate similar players.

As far as what PECOTA is good for, it is simply stating how the player would do IF given that amount of playing time. It is not intended, IMO, to be a playing time projection and thus a guide saying player Y will hit X number of home runs next year.

BP's usefullness is in evaluting players based on their performance...it is just another way of looking at players.

by dbimberg on Feb 8, 2006 8:48 PM EST up reply actions 0 recs

re:

My response was to his latest article, the one linked above. The method he uses in it does put Aybar well ahead of Liriano. It puts Shin-Soo Choo ahead of Chad Billingsley. Ian Kinsler ahead of Nick Markakis. Jose Lopez ahead of Daric Barton, Ian Stewart, Lastings Milledge, etc.

What you wrote describes how PECOTA has been used in the past and how it should be used. How it was used in the article today is a waste of time until Silver takes into account the bias in favor of players who have reached higher levels (which essentially credits players just for getting older) and against pitchers.

by jeck on Feb 8, 2006 9:31 PM EST up reply actions 0 recs

perhaps you are correct...

...I haven't fully digested what this latest article means... I am surprised about the Aybar vs Liriano rankings...

by dbimberg on Feb 8, 2006 9:46 PM EST up reply actions 0 recs

How do you know?

Are the lists published somewhere that I missed?

I'd frankly be shocked if Aybar's ahead of Liriano. Look at Liriano's new PECOTA card. It has the guy averaging 4.0 WARP a year as his baseline for the next 5 years. It gives him a 10% chance to be a Superstar (EqERA < 3.00) by 2007! His UPSIDE (per the definition) is going to be through the roof. It thinks he yas a 70-80% chance to be at least a rotation regular for the forseeable future.

PECOTA likes Aybar quite a bit too, but it thinks he's Jimmy Rollins the Sequel. It has him averaging a shade under 4 WARP/year for his baseline, and it doesn't give him much of a "superstar" shot until 2009. He has some impressive names on his comparables list, but there are also quite a few guys like Joe Thurston, Macier Izturis 2003, Jason Bourgeois, etc. He's "only" got a 60-70% shot to be at least a regular. He'll be up there on the list, but I highly doubt it'll be in front of Liriano.

I think UPSIDE also has a built-in adjustment for pitchers... I thought it's implied in the definition. If it's not and the numbers are much lower than those for hitters, he may come up with a bit of a "fudge factor" to even things up. Though then again, if he's only going position-by-position there may be no need.

by delomir on Feb 9, 2006 1:34 AM EST up reply actions 0 recs

excel

It's (Peak WARP*10) + Peak Upside. You just need a spreadsheet and a BP subscription.

by jeck on Feb 9, 2006 7:03 AM EST up reply actions 0 recs

look closer

well, thats only because PECOTA has Liriano only pitching 150-160 innings a year, which is either a failing of the system or an acknowledgement that young pitchers are injury prone, depending on how you look at it. Also, there's the inherent difference in how you value pitching versus hitting. Either way, if you look at the rate stats, the performance prediction is in Liriano's favor, as he posts ERAs around 3.90 for the next five years, while Aybar only gets his EqA above league average in 2009.

by amol on Feb 9, 2006 11:54 AM EST up reply actions 0 recs

isn't that the point?

I'm not saying PECOTA thinks Aybar is a better prospect than Liriano. I'm saying Nate Silver's method for translating PECOTA into prospect rankings does. Therefore it's probably not a very good method.

by jeck on Feb 9, 2006 12:03 PM EST up reply actions 0 recs

context

well, I think its just a method that has its limitations, as every method does. it certainly seems to be a solid way of comparing prospects within a position, and thats just with the combined score thing, which I don't think is presented as the final word in evaluation. At the very least, I think you should reserve judgement until he finishes the series.

by amol on Feb 9, 2006 12:16 PM EST up reply actions 0 recs

the rest

the rest of your problems with the rankings have the similar problem of not accounting for position. Both Lopez and Kinsler are guys who posted strong years at triple A while playing second base, while the rest put up better numbers at lower levels while playing less valuable defensive positions. Obviously you have a different opinion of the relative value of various positions, as well as the difference in performing at higher levels, but the fact that you don't agree with Silver on those points doesn't make his system worthless.

by amol on Feb 9, 2006 12:08 PM EST up reply actions 0 recs

Alright

I misunderstood what he meant by "Peak WARP."

I think the key is hidden on Liriano's PECOTA card. His "Upside" scores for the 5 year forcast add up to 119, while they have posted 118.9. Ordinarily this wouldn't be a problem, their numbers are probably more exact than the published ones, but in this case it indicates that they didn't do an extrapolation for Liriano's age 27 and 28 seasons, while they did for Aybar. Thus Aybar's Upside score is way higher.

If you just add Aybar's 5 year Upsides you get 123.4 (hehe, cool), basically a wash with Liriano's projection. That include's PECOTA's anticipation of a big breakout for him in 2009. Also, it's projecting more than half of his mean value to be his defense, to 2.5-3 wins/season. Maybe if at his peak he'll hit like Paul Molitor and field like Orlando Hudson, he IS a better prospect than Liriano.

I haven't gone through all the pitchers, but I did a random sampling of some good ones and I didn't think the general pitcher WARP trend was all that pronounced downwards. Slightly so, moreso than for batters, but it looked to me like in Liriano's case it sees him having an early peak more than just the general attrition for pitching. Which could well be the case, and if it's true then Aybar might end up being more valuable overall. We'll know in 5 years I guess.

If the pitcher decline does cause a notable discrepancy between hitting and pitching Combined Rankings, Silver's almost certainly going to come up with a factor to even things up a bit if he does do an overall top prospects list. Which may not necessarily occur.

by delomir on Feb 9, 2006 5:47 PM EST up reply actions 0 recs

Very interesting, but...

Very interesting article. I look forward to his next articles re: the other positions. However, I think is useful to keep in mind the statement that Kevin Goldstein made in the recent BA Chat on the A's prospects: 'I think one of the things some might be seeing are the statistics only, and that's a very dangerous way to rank prospects.'

Anyway, I wonder how much the the PECOTA rankings will factor in to BP's prospects rankings this year and in future?

by rhd on Feb 8, 2006 8:30 PM EST reply actions 0 recs

PECOTA

"As far as what PECOTA is good for, it is simply stating how the player would do IF given that amount of playing time."

More accurately, how well they will have to do to win that much playing time. And there is obviously no way of knowing whether its accurate or not for players that don't.

by TT on Feb 8, 2006 9:10 PM EST reply actions 0 recs

that's silly

Silver makes it clear in his past few pieces that if a player doesn't make the majors in 06, you can use his minor league equivalents (converting A+ or AA stats to ML ones) to compare to pecota's suppositions. It's a computer; it's got limitations; but it too has upside. :-)

by Azteca on Feb 8, 2006 10:48 PM EST up reply actions 0 recs

Missing the point (I think)

The main thrust of the article is that often times there is agreement on the projection of a prospect, even what that prospects upside is.

What is irrational is that two players with the same projection and upside are often miles apart in their "prospect status".

Now if there is disagreement in the projections, then all bets are off.

by SLK on Feb 8, 2006 9:52 PM EST reply actions 0 recs

So

is Silver right? Does Victorino still qualify for BA's list? As I read, I was thinking Shane's service-time, in being twice selected for the Rule 5 draft, precluded his ROY eligibility. Botts made Texas's list; maybe it was just an oversight.

by Azteca on Feb 8, 2006 10:54 PM EST up reply actions 0 recs

57 games

He cannot be a rookie any longer, as he's appeared in 57 games. I don't know how frequently BA ranks a non-rookie though.

by BobbyMac on Feb 9, 2006 2:25 AM EST up reply actions 0 recs

BA does not look at service time

BA only considers IP and AB cutoffs when determining if they are still prospect eligible.

Service time affects MLB rookie status, but does not factor in for BA.

by SLK on Feb 9, 2006 8:31 AM EST up reply actions 0 recs

hated the article

Quote:
There's been relatively little attempt at serious, econometrically oriented valuation of prospects, and the overall tenor of the debate resembles an investment bank picking its clients by reading tea leaves and two paragraph business descriptions.

And before this says you can move a guy up or down on BA's list 10 spots and it wouldn't matter.

To me this is saying that BA does not analysis on how they rank their players. I hardly think this is the case.

He then goes on to say about using Pecota valuation for your ranking.

To me this was a very unprofessional article and seemed bitter. If you believe your method is better suggest it is better and say why. But to use examples and imply that BA could just as well use tea leaves to set their ranking is terrible.

ps: on the players he mentioned. Maybe BA was thinking that a 25 year old with low upside will probably never get the plate appearances that PECOTA projects (as this is important in pecota) and will be a bench player - while the younger player has time to make improvements. I'm sure they didn't use a dart board.

And if you didn't realize it he is picking on all sites when he says this. Just BA is the one he directly targets.

by pedrophile on Feb 8, 2006 11:06 PM EST reply actions 0 recs

Please re-read

"And before this says you can move a guy up or down on BA's list 10 spots and it wouldn't matter."

This is a misreading. What he says is a lot of times what prospect "analysts" do is take BA's list, move people up or down a few spots, and then call it a day. He actually is speaking out against this.

by SLK on Feb 9, 2006 8:20 AM EST up reply actions 0 recs

What's a Prospect?

What is irrational is that two players with the same projection and upside are often miles apart in their "prospect status".

There isn't agreement about who the best basebal players were even after the fact, so why would you expect there to be agreement on what makes the best prospect?
The downside for every prospect is that they don't make it to the major leagues. Whether they fail in rookie ball or get to AAA doesn't matter. As players progress the range of their potential upside decreases, but their potential downside stays the same.

If a guy is projected as a fourth outfielder when he gets to AAA, he is not much of a prospect. The chances he will exceed that projection are minimal. And he still has the downside of not having a career at all or a very limited one.

That same projection for a younger player at AA has a lot less certainty. The downside is the same. But there is the possibility that the player will develop into a starting center fielder. That makes the younger player a lot better prospect.

by TT on Feb 8, 2006 11:17 PM EST reply actions 0 recs

you're incorrigible :-)

by Azteca on Feb 9, 2006 12:27 AM EST up reply actions 0 recs

Analysts

Often make upside projections. Yes there is a "possibility". But most times when projecting prospects the upside is what reasonable people think is the best case scenario.

I understand your critique, but think it's overstated.

I agree that there is still a lot of work yet to be done when comparing hitters to pitchers,etc. The definition of a replacement player is probably not set in stone. But for comparing within a particular position, I think there is a level of irrationality with rankings. (The Navarro - Russell Martin example being one of them.)

by SLK on Feb 9, 2006 8:25 AM EST up reply actions 0 recs

Upsides

But most times when projecting prospects the upside is what reasonable people think is the best case scenario.

The upside projections are not ceilings. If you follow prospect evaluations you will notice that "the best case scenario" changes from year to year. Sometimes it goes up, more often it goes down, but it has variation in both directions. That is not true of the "worst case scenario".

I think there is a level of irrationality with rankings.

If by irrational you mean subjective, yes. But there is nothing irrational about putting more weight on the possibility a player will be a major league regular than the likelihood they will find a spot on a major league roster.

There are a lot more players with fourth outfielder skills out there than there are with regular centerfield skills. And the guy who is a regular is going to contribute a lot more than a guy who projects as a defensive backup at an offensive position.

by TT on Feb 9, 2006 10:18 AM EST reply actions 0 recs

I think this is open to dispute.

Someone at BP recently commented that one of the better CFers in baseball last year, Aaron Rowand, came up through the minors playing the corners. Personally, I don't know Rowand's development path very well (prospect retro, John?), but it seems there are lots of guys who might look like 'tweeners but actually become proficient in CF, which, in turn, makes their bats more palatable. Rowand's maybe unique, but I don't think so.

I liked Silver's piece, but maybe I'm in the minority around here. BA, as Nate says, wields a lot of power, and it's not a bad thing to carefully analyze how that pub comes about its rankings.

by Azteca on Feb 9, 2006 10:31 AM EST up reply actions 0 recs

Rowand

Someone at BP recently commented that one of the better CFers in baseball last year, Aaron Rowand, came up through the minors playing the corners.

If true, its a good example of how projecting a player's upside is not only off high of the mark.

it seems there are lots of guys who might look like 'tweeners but actually become proficient in CF, which, in turn, makes their bats more palatable.

I think the question here is if that is more likely for a 23 year old player at AA than a 25 year old at AAA. Rowand, for instance, was 23 his first year in the big leagues.

by TT on Feb 9, 2006 11:15 AM EST reply actions 0 recs

by the way

that particular defensive career trajectory not unique, or anything like unique. just among current good defensive CFs, kotsay was a RF for the marlins before moving to CF with the padres. johnny damon played mostly LF with kansas city (admittedly, they did have carlos beltran) and the A's stuck him in center. darin erstad played 1B/LF for a long time before the halos finally realized he was a world-class CF.

by jpahk on Feb 9, 2006 12:30 PM EST up reply actions 0 recs

very poor examples

Kotsay, Damon and Erstad were all CF in the minors. They only played corner initially in MLB b/c teams are hesitant to shift incumbent star players off CF for rookies. The Rowand situation is a player groomed as a corner in the minors who becomes the CF in the majors.

In Billy We Trust

by I Love Oakland As on Feb 9, 2006 5:33 PM EST up reply actions 0 recs

Winners

that particular defensive career trajectory not unique, or anything like unique

I am not sure what the relevance of that is to this dicussion. No career trajectory is unique and no projection is 100% certain.

Put this in a different context by analogy. You have two runners in races and you project them to finish with the same time. One is a 100 yards from the finish and five yards behind the leader. The other is a half mile from the finish and five yards behind the leader. I think we can agree that the runner closer to the finish is more likely to finish close to the projected time. But the guy a quarter mile back has a better chance to win the race.

In the prospect race a guy who is likely to place has some value, but the real objective is to find the winners.

by TT on Feb 9, 2006 12:59 PM EST reply actions 0 recs

shrug

the relevance is merely that azteca, who originally pointed out that rowand started out in a corner, said "Rowand's maybe unique but I don't think so." i was merely providing evidence to back up his opinion.

by jpahk on Feb 9, 2006 2:01 PM EST up reply actions 0 recs