Saturday, January 12, 2008

A dirty way of predicting reliever leverage when pLI is not available

Note: while I'm posting this separately so that it is visible, it's really just meant to be an update to my piece on reliever leverage in the player value series. I'm appending it to that article as well.

As discussed earlier, when thinking about reliever value, it's insufficient to strictly consider the rate at which they give up runs because some runs are more valuable than others. Closers, in particular, tend to pitch in high leverage situations, and therefore should get more "credit" for their ability to pitch above reliever replacement level than a pitcher who only pitches in games that have a lopsided score.

For players since 2002, we can get actual pLI data from FanGraphs, and I discussed how to employ those data to adjust reliever run value estimates previously. However, what if you want to look at reliever value among players who played prior to 2002, like in my proposed series on past winning Reds teams? In that situation, you'd need some way of inferring reliever usage from other statistics.

One way to try to do this is by looking at performance--better pitchers should be used in higher-leverage situations. However, when attempting this approach, I've found that there's just very little predictive power (i.e. huge amount of scatter), even though there is a significant relationship between ERA (or FIP) and pLI. Whether that's due to within-team competition, inconsistent reliever performance, or poor decisions by managers, performance is just not a very good way to predict pLI.

On the other hand, as Darren implied, even in historical databases like Lahman's, we have at least one statistic that tells us something about usage: saves. Saves are well documented to be a rather poor indicator of reliever quality. Nevertheless, they do tell you who was pitching in the 9th inning of a team's games, which tends to be the inning with the highest leverages. So we should be able to use saves to infer something about reliever usage. Here's what I did:

Methods

I pulled stats, including both traditional pitching statistics and pLI, from fangraphs on all pitchers, 2002-2007, who threw at least 25 innings in relief in a season. There is some selection bias in such a sample, because it will tend to exclude a lot of bad pitchers who weren't given the opportunity to throw 25 IP. But it still does include pitchers that span much of the range in terms of performance, and gets around the issue of dealing with stats on pitchers with extremely small samples (not that 25 IP is a big sample...).

Next, I calculated saves per inning (Srate) as an indication of the proportion of a pitcher's innings that were associated with saves:

Srate = Saves/IP

It's important to use a rate because you want to know something about a player's opportunities. If someone gets 20 saves in 20 innings, they're probably pitching in much higher leverage situations, on average, than someone who gets 20 saves in 70 innings. Ideally, I'd also use blown saves--and maybe holds--but those stats are not available in the Lahman database or on baseball-reference's team pages, so I'm going to ignore them for now.

I also converted to pLI to a "rate" statistic using the approach suggested by Tom Tango:

rateLI = pLI/(pLI+1)

Such that:
pLI = 2 ---- rateLI = 0.667
pLI = 1 ---- rateLI = 0.500
pLI = 0.5 ---- rateLI - 0.333

This was important because as a pure ratio, pLI changes at a faster rate above 1.0 than it does below 1.0, which makes it hard to model using a regression-based approach.

Anyway, here's a plot of Srate vs. rateLI:
Obviously, that's a pretty ugly-looking relationship down in the zero/low-saves groups. But as you can see, there's a pretty nice relationship among pitchers who actually have a modest number of saves and their pLI. In other words, once someone starts to get saves, you can reasonably predict that he'll have an above-average pLI, and the player's pLI should steadily increase from there.

I decided to run with this and, in what I completely admit is a really terrible abuse of regression math (I've violated just about every assumption one can violate), I fitted a line to this relationship. I found that a second-order polynomial seemed to fit the data well. Furthermore, I forced the y-intercept to come in at a rateLI=0.5 (pLI=1.0), such that the average pitcher without saves is expected to pitch in average leverage (otherwise, the equation tended to predict that the vast majority of pitchers would have a pLI=0.8, and that's not reasonable). Here's the equation:
rateLI = -0.3764*(Srate^2) + 0.5034*Srate + 0.5

which we can convert back to pLI by:

pLI = rateLI/(1-rateLI)


Now, this rather shaky regression equation isn't something that I'd try to publish in the SABR newsletter, much less an academic journal. It's not built upon rigorous math. But it actually works pretty darn well. For demonstration, here's a table showing a hypothetical pitcher who has thrown 70 innings, and how his predicted pLI changes as the number of saves (and thus his Srate) increases:
Saves (70 IP)
Srate rateLI pLI
0 0.00 0.50 1.0
5 0.07 0.53 1.1
10 0.14 0.56 1.3
15 0.21 0.59 1.4
20 0.29 0.61 1.6
25 0.36 0.63 1.7
30 0.43 0.65 1.8
35 0.50 0.66 1.9
40 0.57 0.66 2.0
45 0.64 0.67 2.0
50 0.71 0.67 2.0
As you can see, the numbers seem to plateau at around a pLI of 2.0, which is about where MLB closers tend to plateau. David Weathers, for example, who had a Srate = 0.42 last season, had an actual pLI=1.95, which isn't far from his predicted pLI using this method. Pitchers with a smaller number of saves per IP--setup men, mostly--are assumed to have above-average but still relatively moderate leverage. Finally, guys without saves are assumed to have average leverage.

Anyway, I think that this is a pretty reasonable way to adjust for historical reliever leverage, at least among closers. Obviously, we're going to undervalue some relievers that aren't yet in the setup role but pitch in lots of big-time leverage situations in the 7th or 8th innings. But I think this approach will capture a lot of what we're trying to do with a reliever leverage adjustment.

On a moderately related note...last night, I spent some time setting up spreadsheets and my database to start on the Winning Reds historical series. Should be pretty efficient at this point, which should make it easier to get through the teams at a good clip as long as I keep the writing under control. I'm excited to get started on the series, but I think I'll do a dry run first in wrapping up the 2007 Reds' season. Look for that shortly.

12 comments:

  1. Justin, can you make the generic case in why we should look at leverage in assigning value? It seems to me that for offense, we try to neutralize away opportunity. Isn't leverage akin to OPS w/ RISP or some other such measure. There seems to be an inconsistency in this approach. If we're going to include the "when" issue for relievers, shouldn't we be doing it for everybody?

    ReplyDelete
  2. For starting pitchers and hitters, seasonal deviations from average leverage are more or less random events, and few hitters or pitchers with a full season of data come in with a pLI that's particularly different from 1.0. Therefore, a hitter's runs above replacement, for example, translates closely to his wins above replacement. And wins really is what I'm after, ultimately, because that's the currency of success in baseball.

    The same is not true for relievers. Managers have considerable power in determining what sorts of situations relievers are used. Therefore, because closers, in particular, tend to be used more in high-leverage situations, their runs allowed influence wins more than other players. This is predictable and repeatable, which is why I think it's important to consider it in these estimates.

    Including leverage w/ releivers is not the same as including OPS with RISP in one's RC estimate. That's an adjustment that looks at how a hitter did in a fraction of his plate appearances, and then adjusts the overall production by his performance in that fraction of appearances. Those clutch performances, as I'm sure you're aware, have very little repeatability, which is why I tend to not be interested in bothering to adjust for them.

    But with relievers, we're adjusting their entire performance based on the leverage of situations in which they pitched. So if a closer pitches in situations that are, on average, twice as important as normal situations (pLI=2), this approach essentially sets any RAR he provides to have double the value...no matter how well he does overall.
    -j

    ReplyDelete
  3. If I can summarize, the purpose of using a leverage adjustment is to account for the fact that opportunities are not distributed equally in regards to their potential effect on game outcome. Thus, to estimate the effect of a given player's performance on game outcome, we must adjust for this usage difference.

    Is this fair/accurate?

    Without challenging the accuracy of your claims, do you have any data back your claim in your first paragraph?

    It would seem to me that the manager does have great affect over the leverage of batters. How many men were on base for Brandon Phillips versus Adam Dunn by virtue of the place in the lineup? Ryan Howard had lots of opportunities to drive in runs, certainly a statistically significant difference of greater opportunities than were they randomly distributed. All else equal, were he asked to bat 8th, rather than cleanup, surely his performance would have had a significantly different effect on wins.

    That managers choose to bat hitters such to maximize the effect of their performance seems akin to reliever use. Now, unlike pitchers, managers cannot control the specfic scenario usage with hitters. They can however, try to maximize the interaction effect of having ones best hitters grouped together. That Ryan Howard batted 4th and not 8th did have some multiplicative effect on the win-value of his performance. Perhaps, as you suggest, it's merely an order of scale, but it seems to deserve more than a paragraph's worth of treatment. Maybe you can direct me to a thread elsewhere where this was discussed?

    ReplyDelete
  4. What?! You want me to support my claims with data?! :)

    I don't know of a comprehensive study on this, but the effect is pretty dramatic. I'd just recommend going to fangraphs and looking at the 2007 leaderboards for hitters, starters, and relievers.

    Qualified hitters had pLI's ranging from 0.92 to 1.17.

    Qualified starters had pLI's ranging from 0.87 to 1.09.

    There's no "qualified" button for relievers, of course, but among those with >50 IP, pLI's ranged from 0.48 (Chris Bootcheck) to 2.35 (Trevor Hoffman).

    I'll grant that some of that may be sample size, but Hoffman (to take one example has had pLI's of 2.02, 2.12, 2.08, and 2.35 from '04-'07. That means that we effectively have to DOUBLE his RAR to get a reasonable picture of his impact on wins.

    You'll see minor deviations from 1.0 for hitters and starters, but rarely are they more than 10% or so (compared to 100% for closers). Looking through the numbers, it certainly does seem to be the case that middle-of-the-order hitters tend to have slightly higher pLI than other hitters, but we're talking about pLI's of 1.00-1.10 instead of 0.95-1.05. Nothing like the doubling of pLI that we see in closers.

    ReplyDelete
  5. Fair enough, I didn't realize fangraphs had it for hitters and the figures back up your point. Good stuff. Still, while the effect might not be as large on a per PA basis, because the volume of performance is so much larger, the effect on overall value may still be important for hitters. A 10% boost or decrease is nothing to sneeze at.

    Interestingly, the two highest pLI for positional players are both Giants -- two guys who tended to bat very near the highest OBP guy in baseball.

    ReplyDelete
  6. I think I found you, Berliad... :)

    Just thought you might enjoy this:

    http://www.israelbaseballleague.com/baseballinisrael/biblical/

    ReplyDelete
  7. @Rick, I agree that it's something that might be worth thinking about moving forward. ... But I'm ok with leaving it as it is for now. :)

    @Alex, you got me. :)

    The link is pretty funny though. Word 'round the campfire is that the Israeli Baseball League is unlikely to continue much longer, which is unfortunate.
    -j

    ReplyDelete
  8. Justin, I'm very glad you wrote this piece, as I was thinking about a portion of this just a few days ago. I didn't feel like e-mailing David Appelman (he never responds, it's very annoying), so I'll ask you... How did you get the pLI numbers from Fangraphs in bulk? I can't imagine you went player by player writing it down, that would have taken days. If you can just write a little step by step of how to extract data from fangraphs that would be much appreciated.

    Dan

    ReplyDelete
  9. Hi Dan,

    I just went to the 2007 leader boards at fangraphs (linked above) and did the 'ole copy and paste. The only irritating thing about it is that they only show 50 or so players at a time IIRC. That's part of the reason that I limited my sample to 25 IP or more in a season. :)

    If you're interested in my spreadsheet, I'm happy to send it your way. Just fire me an e-mail: jinaz.reds@gmail.com
    -j

    ReplyDelete
  10. Justin, post #1 here shows how to do historical LI.

    ReplyDelete
  11. Tango,

    Wow, thanks for that, and especially for your step-by-step explanation in post #5 in the linked thread. I may give that a go and see how well I can automate it.

    I sure wish that Sagarin's files included lahman or retro ID's, though I can usually construct a lookup string.
    -j

    ReplyDelete
  12. Wow, awesome post.

    I run a Rays' blog over at RaysBaseball.Blogspot.Com

    Would you like to trade links? If so, just post a comment on my blog and we've ourselves a deal.

    ReplyDelete