Crystal Ball Review Finale

by John Sickels on Oct 9, 2006 4:18 PM EDT in News Comment 23 comments

Crystal Ball Summary

OK, let's wrap this up. The other CBs I did during the season have too much "in-season knowledge" about 2006, so I don't think it proper to evaluate them. So we'll work with the ones we've reviewed over the last week.

First of all, when I started doing the CBs last year, it was as a lark, a way to have fun, and I didn't try to take it particularly seriously. I did put a lot of effort into them. . .they are not just numbers on a page. It can take 2-3 hours to do a good CB. Basically I look at the player in question, find guys who were comparable to him at that point in their careers, see how those other guys turned out, then make some educated guesses about what the future might hold. But I was curious about how such an approach would work compared to more formal, formula-based systems such as PECOTA, so I decided to keep track and evaluate at the end of 2006 how the CB did.

The whole hit/miss category thing is an awkward way to analyze this, and perhaps I'm overthinking it in general. Ideally I'd like to set up some sort of metric and compare the CB directly to other systems, but I don't have the time right now. So anyhow, here are the results. You might disagree with the way I've rated a particular CB, and that's fine, let me know either way.

I'm going to separate hitters and pitchers.

Hitters
Type 1 Hit (dead-on or very close to dead-on)
Jeremy Hermida, Khalil Greene, Matt Murton, Jeff Francoeur, Jason Kubel, Prince Fielder, Josh Willingham, David Wright
Type 2 Hit (generally within the realm of expectation and variance)
Carl Crawford, Rickie Weeks
Type 1 Miss (The projected season was too good compared to actual)
Mark Teixeira
Type 2 Miss (The projected season wasn't good enough compared to actual)
Ryan Howard, Joe Mauer, Grady Sizemore, Lastings Milledge, Wily Mo Pena, Delmon Young

Pitchers
Type 1 Hit
None
Type 2 Hit
Huston Street
Type 1 Miss
Felix Hernandez (borderline case)
Type 2 Miss
Francisco Liriano, Scott Kazmir, Brandon McCarthy, Jon Papelbon

Overall the CB concept seems to work better for hitters than pitchers, however part of that may be because I didn't project as many components for pitchers compared to hitters, making true ERA comparisons more difficult. I will change that in the future.

Here is the bottom line question for you guys: should I continue doing these? Should I try to refine the CBs and try and make it a true projection method? Or should I drop the idea entirely?

0 recs | Comment 23 comments

Story-email Email Printer Print

Comments

Dont drop it!

You did really really well with it overall. I think a lot of people really enjoy them too. Its a great feature as long as you enjoy writing them half as much as we enjoy reading them

It would be interesting to compare them to some of the projection methods and see how it all goes. You did very well overall though.

Thanks a bunch for all the hard work and take care.

by grozzy on Oct 9, 2006 4:18 PM EDT reply actions 0 recs

Keep it up!

It's a good exercise and a reality check about what a good career looks like.

by Doug on Oct 9, 2006 4:21 PM EDT reply actions 0 recs

Keep on trucking

You did a great job! I think your right when you say that you'll need to change up the pitching system, but overall you did spectacularly!

Mop Up Duty

3 writers & daily updates on a wide range of baseball topics, including prospects.

by TheK on Oct 9, 2006 4:44 PM EDT reply actions 0 recs

How did PECOTA do?

Did I miss PECOTA's scorecard for the same players?

by jbm on Oct 9, 2006 4:50 PM EDT reply actions 0 recs

Keep 'em coming

by Boxkutter on Oct 9, 2006 6:03 PM EDT reply actions 0 recs

crystal balls are my FAVORITE feature!

by yanksfan6129 on Oct 9, 2006 6:04 PM EDT reply actions 0 recs

Felix

Felix was a hit, as you originally said, and as the poll now shows the majority of this site thinks.

Good job on the overall concept. It's fun and you do a good job.

by FunWithHeadlines on Oct 9, 2006 7:42 PM EDT reply actions 0 recs

Re: Felix

Felix was a total miss. Felix had to have a late season rally to even get within a run of the ERA John projected. Sure he nailed the Ks, but who in the heck didn't think Felix could K >= 8/9IP. Pretty obvious as well that Felix wasn't going to be allowed to pitch 220 innings. So basically John got the low hanging fruit on Felix but missed the main course. That makes him a miss.

Weeks was a huge miss too. Off by SEVENTY points of SLG!

I think overall John did a very good job. But let's call a spade a spade on the ones he was off on.

by im not new on Oct 10, 2006 8:27 AM EDT up reply actions 0 recs

Nope

He got the Starts, Wins, Innings, Ks and BBs basically correct, regardless of the reasons why he got them correct. The only thing he got wrong was ERA. The majority of the voters on this site say he was a hit, and I agree. I mean if you get 5 out of 6 right and you count that as a miss, it means you think ERA is more important than any other measurement. I don't.

by FunWithHeadlines on Oct 10, 2006 9:35 AM EDT up reply actions 0 recs

re: Majority

Simple majority of voters is a pretty weak bar to hold yourself to.

"When I read about the evils of drinking, I gave up reading." - Henny Youngman

by TINSTAAPP on Oct 10, 2006 9:56 AM EDT up reply actions 0 recs

Which is why...

...that was only one part of my argument, the rest of which you ignored.

by FunWithHeadlines on Oct 10, 2006 10:29 AM EDT up reply actions 0 recs

Re: ERA

If Sickels missed by 0.30 on the ERA I might agree with you. But a full run? Yeah, that ERA gap trumps the other stuff easily.

Bobby Crosby - a poor man's Adam Everett.

by natsfan2005 on Oct 10, 2006 9:59 AM EDT up reply actions 0 recs

Re: ERA

The primary reason why ERA was John's seemingly sole big miss on Felix is he didn't project very many components. If he had projected more variables (i.e. hits and homers) the additional items likely would have been fairly consistent with the ERA spread and thus made his misgivings there more universal.

by im not new on Oct 10, 2006 10:05 AM EDT up reply actions 0 recs

Other components

I don't think many folks would have projected the >1 H/IP and 23 HR allowed, so it's likely that the projected ERA that John would have come up with would have still be substantially lower than the 4.52 that Felix ended up with.

Usually, pitchers with the combination of K/9, BB/9, and GB/FB (2.39) that Felix had would have projected to the ERA John gave him, but he just turned out to be more hittable this year than most prognosticators would have thought. Personally, I'd call it a slight miss, but most likely a blip on the radar for Felix in terms of being so hittable.

by jc3 on Oct 10, 2006 10:17 AM EDT up reply actions 0 recs

Agreed...

you can't have every periph right and say it was a miss.

He missed by ERA, a few lucky breaks here and there and that ERA might have been what John predicted.

Definite hit IMO.

by SenorGato88 on Oct 10, 2006 11:47 AM EDT up reply actions 0 recs

Disagree

You're calling Wins, Innings and Starts peripherals. They're not. What they are is primarily playing time. Being off on ERA IS more significant as that's the performance piece. Performance is what matters most.

Bobby Crosby - a poor man's Adam Everett.

by natsfan2005 on Oct 10, 2006 12:08 PM EDT up reply actions 0 recs

success

there is a great deal of success with the CBs which makes readers want to read them because it isn't just taking numbers out of your ass. It is more then that and it can't hurt to have them as it further cements your knowledge on the game.

Go Jays!!!!

by achengy on Oct 10, 2006 12:53 AM EDT reply actions 0 recs

Luv CB's

Keep it coming...

by louief1 on Oct 10, 2006 9:20 AM EDT reply actions 0 recs

Don't worry about the pitchers

Projecting young pitchers is inherently more difficult than hitters. It is completely normal for the hit/miss rate to be much lower.

Including home runs allowed for pitchers in the projections will allow for a fairer comparison.

by Mike Green on Oct 10, 2006 9:51 AM EDT reply actions 0 recs

Missed one

Didn't you do a David Wright crystal ball last fall?

by jjr on Oct 10, 2006 10:39 AM EDT reply actions 0 recs

Are these...

just CBs you did prior (or real early on in Papelbon's case) to the 2006 season?

I only ask because I thought you did a Nick Markakis one as well... but that was when he was hot. So I guess you cant really count that as a 2006 prediction if you did it in August.

Well, I just answered myself. Nevermind...

by Boxkutter on Oct 10, 2006 2:58 PM EDT reply actions 0 recs

Wasn't McCarthy

The Type 1 miss....

his strikeout rate was better than your projected but his walk rate was worse and, more importantly his ERA was much worse...

by Dfarth on Oct 10, 2006 11:07 PM EDT reply actions 0 recs

Evaluating hits and misses

With minimal effort, you could compare the actual season to the Crystal Ball season, just as you compare Oliver Perez to Sandy Koufax or Pete Falcone, using Bill James's similarity score. This gives some number less than 1000 -- the higher, the more similar. Than choose some reasonably high cutoff (since they are, in fact, imaginary and real versions of the same player). Say, 900 or so. Or better yet, you can probably find a good cutoff just by looking at the distinctions you've already made between hits and misses. But this way, it will all be quantified using a method that is already widely in use (i.e. on Baseball-Reference.com)

http://www.baseball-reference.com/about/similarity.shtml

by DaleCoop14 on Oct 11, 2006 2:09 AM EDT reply actions 0 recs