New statistic aims to answer question: How good is that goalie?

Dwayne Roloson of the Tampa Bay Lightning has embodied the role of the prototypical Stanley Cup playoff hot goaltender, leading his team to a 3-0 lead in the Eastern Conference semifinals against the top seeded Washington Capitals. The 41-year-old had a losing record in the NHL's regular season, but has stopped over 94 percent of shots he's faced in the playoffs -- but he's probably not actually this good.

Summing up the performance of an NHL goalie with a single statistic is a difficult task. New research that makes adjustments to traditional measures attempts to do just that. This effort endeavors to remove the influence of team defense and team offense from goalie statistics, which raises questions about the ultimate meaning of sports statistics.

The Limitations Of Stats

For any sport it is difficult to develop a single formula whose solution explains everything about a player. Batting average in baseball, quarterback rating in football, points per game in basketball: each of those numbers explains something about a player, but also has limitations. For evaluating goalie performance, counting the portion of shots-on-goal that he corrals, called save percentage, is one popular statistic. It makes no distinction between goals that are primarily the netminder's fault and those due to other factors, such as poor defense or great offensive play.

Furthermore, each goalie faces a different collection of shots. Slap, snap, and wrist shots from anywhere on the ice are all equally counted. The data might show that one goalie performed better than his peers against a certain type of shot from a given area of the ice. This partially explains why save percentage provides a limited picture of goalie performance.

"Anybody who knows anything about hockey knows that not every shot has the exact same chance of going in," said Tom Awad, a writer for the website Hockey Prospectus. "So obviously if you could find some way to quantify how dangerous each of the shots is you could improve the metrics of goaltending that we already have."

Two average goalies might stop 1,800 out of 2,000 shots in a season and have identical save percentages despite facing different collections of shots. A 1 percent difference in save percentage would mean a difference of about 20 goals allowed in a season, which could mean the difference of several points in the season standings -- enough to decide home-ice advantage in the playoffs.

Comparing Goalies

So how do you compare goalies to one another? Michael Schuckers, a statistician at St. Lawrence University in Canton, N.Y. tried to remove the effect of other players on a goalie's performance, a new variation of something others have attempted previously. He presented the research at a March conference at the Massachusetts Institute of Technology in Cambridge.

Schuckers constructed maps of goalie performance using data collected at each arena. The NHL tracks the type of shot, where it was taken on the ice and when either team has a power play. These maps allow Schuckers to take a goalie's actual performance and simulate his performance against any distribution of shots.

"For last season we basically modeled what the save probability would be for every goalie for all 74,000 shots taken in the league," Schuckers said. He calls the end result a defense independent goalie rating.

Goalies that face a more difficult set of shots than the league average have higher ratings than their save percentage, while the rating drops for those who faced an easier set of shots. If the rating were a more accurate gauge of performance, it could demonstrate differences over a long season, even though the gap between one season's save percentage and the rating is usually less than 1 percent.

"The places we see big differences are in teams with either very, very good defenses or teams with bad defenses," Schuckers said. "If a goalie is on a team with a [poor] defense, they're more likely to face shots that are in close."

Gabriel Desjardins, a statistical analyst and writer specializing in hockey said that he thought Schuckers' work was interesting. "I think he needs to take a larger sample and see how this holds over."

Desjardins contends that good defenses do not in fact impact the distribution of shots on the ice in a significant way. "[In] the NHL, the talent distribution is so small that you don't see this kind of effect except in the short run only," he said.

"[Desjardins] thinks that the difference is so small that you shouldn't bother with it and I think the difference is small but large enough that you should bother with it," Awad said. "Most people tend to be in one of those two camps."

That good defenses would force worse shots makes intuitive sense, but many things that make intuitive sense don't work out the obvious way when it comes to elite athletes.

Desjardins claimed that defenses can be best evaluated by looking at how many shots and scoring chances they allow.

Schuckers admitted that the issue of whether shots are randomly distributed or impacted by the defense is not settled.

"I think that many folks don't appreciate the difference and I don't think we as a hockey stats community have a strong enough handle on these processes to know which is the valid perspective," Schuckers said.

Icy Data

There is another problem in figuring this out: the quality of the data on which it is based. Awad said that some arenas are known to record incorrect shot distance information, while others consistently record suspiciously high or low shot totals.

And this is only part of the issue. Both Awad and Desjardins said that the variation in skill of starting goaltenders is very small.

"Most people agree that it takes a good 3 years of data for a goaltender to get a decent idea of how good he really is," said Awad.

With so many factors at play, deciding on a single goalie statistic might be impossible. What does it mean to ask who was the best goalie, home run hitter, or 3-point shooter? Is the question who performed the best or who should be the best going forward? Recording the most likely outcome is different than recording what actually happened.

"I guess that I am not interested in a goalie's skill level, but in how they perform. One year's worth of data is just that: one year's worth," Schuckers said. "Would I make a personnel decision on one year's worth of data? Probably not."

Proclaiming the best goalie of the playoffs based on 10 games is an even more questionable decision. Yet, one more solid performance from Roloson could carry his Lightning halfway through the Stanley Cup playoffs and into the Eastern Conference finals.