Evaluating managers -- Miley vs. Narron

Ryan Van Bibber over at Beyond the Box Score has undertaken a process that is probably one of the holy grails of stat-based baseball evaluation: evaluating the impact of a manager. He uses the case study of the Reds' 2005 season, in which Narron took over for Miley mid-season. An excerpt:

Cincinnati, however, had exactly what I was looking for as the ax fell on Dave Miley on June 20, 2005, after 70 games, and Jerry Narron got the call for the 93. It's not a dead-even split, but it's close enough not to be skewed by minor league replacement players. The tumultuous 2005 season endured by the Reds became my starting point.

Van Bibber looks at batting and pitching lines to see how the managers impacted team performance. There's no question that the Reds performed better under Narron (46-46) that year than Miley (27-46), most notably thanks to a tremendous improvement in the pitching during the second half. But it's really difficult to attribute those changes to a manager, especially in the way he's trying to do it here.

For one thing, there's no attempt at a control. One approach that might be useful would be to example typical variation between first half and second half performance in teams, and see whether the Reds diverge significantly from the range of variation that is typical. Teams regularly play differently in the two halves of the season (look at Houston the last few years), and often do so without notable changes in personnel or even strategy. I'm not sure that the Reds are out of bounds in this case.

For another, there's are additional confounding factors. Player composition changes (this was acknowledged), resulting in underperformers getting benched or disabled and giving other players a shot. Also, in the case of the Reds, staff changes also occurred. Most notably, Don Gullett was replaced by the late Vern Ruhle as pitching coach at the same time that Miley was fired, and, if anything, Ruhle should probably be credited with any coaching that improved the pitching in the second half.

To be fair, Van Bibber will continue this work in a subsequent post that will look more at the specific tendencies of the managers and try to quantify how those might affect team performance. I'll be interested to see what he does.

I may try to do some managerial evaluation at some point. I think it is possible to do, but I think the effects of managers & coaches are probably fairly small and thus require a large sample size (multiple seasons) to identify differences. Furthermore, I think it's very important to try to first understand typical variation in team & player performances, and then try to quantify how managers and coaches might cause a shift. Not an easy thing to do, as we have a hard enough time understanding player performance! Nevertheless, it's possible to do if you make some assumptions and use appropriate methods (regression could help here). I'll put it on my idea list of things to pursue in the future.