I've been thinking about this topic lately, and I think it's especially interesting with the Futures Game later today . . .seeing a prospect live is a lot of fun, but how much do you need to see said prospect before you start really valuing your observations, particularly compared to qualitative analysis?
I'm not arguing that there is NO value in a small sample size of observations . . .even three at bats for a position player or a couple of innings of work out of a pitcher leaves you with more information about a player than you would have had otherwise. Yet there is certainly the potential to be misled by those observations, or to place a greater value on those observations than would be prudent.
How much accurate information do you think you can derive from a prospect in a single observation? An expert observer could naturally tell more than a novice, but even a novice can tell a lot. There is no real great secret to scouting other than showing up to games and (ideally, but not necessarily) seeing a lot of games. Sure you've got your guys who can note that a particular player is having an issue because of a mechanical flaw, and much credit to them, but to me much of the time that ends up being the stuff in that 90-95th percentile of observation that, while nice to know, doesn't necessarily tell the "need-to-know" story behind the player.
Once you have that qualitative information, how heavily would or do you weigh it relative to your take on the player's performance record? If a particular player "doesn't impress you" (to take a phrase I've seen used in a number of reports lately) on a given day, how much of a truly bad thing is that? It's obviously not the player's job to impress you, and even good players have awful days, or days where their bat speed isn't quite there or their focus not where it needs to be.
I know I've thrown a lot of ideas out here, but there's a lot of food for thought here. One game is better than no games. Ten is better than one. And the potential to get sucked into what you see rather than what a player is doing is always there . . .it clearly happens to scouts all the time, as well it should, because scouts aren't there to make judgments on a player's current level of performance (otherwise, we wouldn't need scouts). So where do you find the balance, and how do you set that curve?