Improving the GM Hiring Process

There is an important distinction between descriptive and predictive stats. If you’re looking to predict future performance, like an NFL team’s future draft performance, you want to have predictive stats. 

What you don’t want are descriptive measures of previous performance that have no bearing on future performance. This is part of a larger conversation I had with Alex Stern on football analytics.

Here’s an example from recent history. The Chargers 2004-2006 seasons, Marty Schottenheimer’s last three as head coach, were their three best seasons (according to Pro Football Reference’s Simple Rating System) since 1979. However, the Chargers fired Schottenheimer after a 14 win season in 2006. 

Due to alleged animosity between the head coach and general manager, ownership had a decision to make. Keep the head coach on the heels of a 14 win season, or keep the general manager on the heels of several great drafts. 

They chose to retain General Manager A.J. Smith.

Adam Harstad published a wonderful podcast episode explaining why the Chargers were mistaken in assessing which aspects of each role were descriptive vs predictive. Here’s something I wrote recently on NFL draft performance: 

Nobody can consistently ‘beat’ the NFL Draft over a long period of time. There are numerous mental biases that are simply too strong to overcome in order to consistently make ‘optimal’ decisions. The best paper I’ve ever read about the subject is by Cade Massey and Richard Thaler. Yet, GMs are constantly hired, named GM of the Year, and then fired based on the results of draft classes that are more random than we think. Consider that, in 2019, 7 of the past 10 executives of the year were fired. If the advantage isn’t in choosing players, where is it? What separates good from great teams is their focus on second-order thinking. This is something that Lombardi often talks about, and something that Andrew Brandt recently wrote about.

In short, we shouldn’t be paying a premium to bet on a coin flip.

What can we do?

According to the work of Dan Hatman, there are plenty of improvements to be made. Dan Hatman, former college coach and NFL scout, is focused “on the objective analysis of subjectively collected data, the personal and professional development of the next generation of evaluators and is constantly looking to Help Build Champions.”

He writes, “if Owners want to see their football operations staff execute better, they should invest more in their football operations staff.” I mentioned this while writing about market inefficiencies. This is a perfect example, spending on coaches and operations. This spending doesn’t count against the salary cap. As Michael Lombardi often says, Sean McVay should be the $100m dollar man on the Rams. 

Since there is uncapped spending on front office positions, and executives aren’t subject to a draft, there is increased potential for favorable outcomes. 

If the opportunity is there, how should teams go about making the right hiring decisions? This is what Hatman is after. Time after time, teams make the same mistakes. For example, selecting general managers from the narrowest of pools. “Of the 71 GM searches studied since 2007, only 19 led to a new decision maker that did not have clear ties to the Owner/Search Committee/Head Coach. So turning that into a percentage, only 27% of GM searches resulted in Ownership actually changing course and installing new systems in their personnel department.”

What’s the point of paying millions of dollars to search firms if you’re going to disregard their findings anyway? Or if they keep recommending the same few names?

Here’s another one: focusing on a plan instead of a toolbox. As Mike Tyson eloquently puts it, “everyone has a plan until they get punched in the mouth.” It’s the equivalent of solely relying on printed out MapQuest directions. Fortunately for me, Google Maps has been around since I started driving, but it wasn’t always that way. If you got thrown off course and only had your printed directions, good luck. The best general managers would still be able to find their way to a championship.

Here’s why this is important. Hatman references Pat Kirwan’s description of what an NFL GM faces:

Running an NFL franchise isn’t easy. Can you evaluate personnel, manage a salary cap, negotiate contracts, select a head coach, handle the media, make tough decisions, cooperate with an owner, deal with the league office and, most importantly, carry out a vision for a winning franchise? If so, then you qualify for the job.

Instead of focusing on the plan itself, Hatman recommends focusing on the toolbox of skills at the candidate’s disposal. “With the hypothesis being, the larger the tool box they possess, the better positioned they may be in regards to solving problems. We should be looking for multi-track minds.”

Additionally, the general manager hiring process is a winner-take-all scenario. Hatman writes, “Those who interview for GM jobs get on the radar of every other team and therefore are more likely to be asked to interview down the road.” 

Instead of focusing solely on the established pool of candidates, teams should be working to find referrals for under-the-radar candidates. I loved this post by Erik Torenberg on Frameworks for Hiring. Hatman is doing exactly that. He runs a survey of NFL staffers where he asks “Can you name 1 or 2 people in football that you’ve met or worked with that you think have a unique background/skill set/thought process about team building? Someone, who even if they are not ready for GM today, shows signs of being the type of person you would follow moving forward.”

If you were asking one person, you’d find out the best reference they worked with. However, that does not guarantee that the candidate is “the best,” just the best that person has been exposed to.

Last year, Planet Money published a podcast episode that perfectly demonstrates this process. Here’s the description:

In 2005, Franklin Leonard was a junior executive at Leonardo DiCaprio's production company. A big part of his job was to find great scripts. The only thing — most of the 50,000-some scripts registered with the Writers Guild of America every year aren't that great. Franklin was drowning in bad scripts... so to help find the handful that will become the movies that change our lives, he needed a better way forward.

Today on the show — how a math-loving, movie nerd used a spreadsheet and an anonymous Hotmail address to solve one of Hollywood's most fundamental problems: picking winners from a sea of garbage. And, along the way, he may just have reinvented Hollywood's power structure.

This is what we need to see more of in the NFL. The closest thing we currently have is Dan Hatman’s yearly General Manager Candidate Study.

More than anything, the NFL is a people business. People are the most important resources for the league and every team. The sooner that teams can get the right people on board and working in tandem, the better. Before they can do that, though, they need to know how to go about it the right way. This includes knowing who to talk to, how to properly frame past performance, and how to value diverse skill sets.