The new challenges of measuring and evaluating people performance – a few non-personal lessons from prison


 Photo by aussiegal

A short story about prison

I don’t remember where exactly I read it. I think it was in one of Marcus Buckingham’s books. Anyway, the writer described an interview with a manager of the prison authority in England. That manager told the interviewer about the ways in which that organization became much more effective. Now, when you think of a prison, you would probably think about things in the lines of tightening security. But the most important activity that was described had to do with the way the prison authority measured its effectiveness. Instead of measuring how many people got out or escaped, which was the traditional way to measure the effectiveness of prisons, the manager changed the way that organization measured it success. They started measuring how many people who got out of prison legitimately, returned to prison. The manager said that he realized that the objective of a prison is to make sure prisoners who return to society don’t go back to the life of crime. In how many other places in life do we still measure the wrong thing because of habit or because of the available data?

From prison – to the basketball court

I was reminded of that story this week when I read this amazing article by Michael Lewis from the New York Times called: “The non-stat All-star”. In a nut shell, Lewis describes the story of Shane Battier, the NBA basketball player of the Huston Rockets. Battier, is the kind of player that his contribution to team does not show on the “regular” statistics usually measured during a basketball game. But if you look closely, you see that the effect he has on his team is amazing.

Now, I know what you are thinking. Because my E-book tries to draw conclusions about life from the basketball court to real life, and Lewis writes in the article that: “In its spirit of inquiry, this subculture inside professional basketball is no different from the subculture inside baseball or football or darts. The difference in basketball is that it happens to be the sport that is most like life”, I am discussing this article in this blog. But that is not the case. I bam discussing this article because of these quote:

There is a tension, peculiar to basketball, between the interests of the team and the interests of the individual. The game continually tempts the people who play it to do things that are not in the interest of the group… When I ask Morey if he can think of any basketball statistic that can’t benefit a player at the expense of his team, he has to think hard. “Offensive rebounding,” he says, then reverses himself. “But even that can be counterproductive to the team if your job is to get back on defense.” It turns out there is no statistic that a basketball player accumulates that cannot be amassed selfishly. “We think about this deeply whenever we’re talking about contractual incentives,” he says.

Isn’t this just like life in organizations?

Sounds familiar? Similar to basketball, where the interest of the individual player is to do things that benefit him, but hurt the team, the business world is a world where the individual has all the incentives to act for his own benefits even if it is not beneficial for the organization. In other words, the interest of the individual and organization are not in alignment. This is what is usually called, the agency problem or the principal-agent problem.   

After reading Lewis’s article Tom Davenport wrote in the Harvard Business blog that:

In business, we’ve all known managers whose units or companies perform better when they’re in charge. Unlike professional basketball, however, most companies haven’t yet begun to evaluate managers or employees systematically based on their individual and team contributions. No plus/minus statistics have been developed for a business context. The emerging field of human resource analytics has a lot to learn from the Houston Rockets

It is the measurement, stupid!

I think the story Lewis describes, although about basketball, actually represents some of the most urgent problems the world is facing today:

  • 1. In order to create comparative advantage, new ways to make people more effective are in high demand. If we create a way to the get people to do the right thing for the organization where other employees of other organizations don’t, we create a comparative advantage.
  • 2. Many of the incentives we have in the world today, measure the wrong things. Just read some of the explanations for the Global Financial Crisis.
  • 3. Today it is much easier than it used to be to measure things. Basketball, off course is just an example. Think about companies which accumulate enormous amounts of data in ERP systems. The data is immense, and it is almost free to collect it. but still, although we have more data and statistics than we ever had, we still succussed at measuring the wrong thing. It is more than that. The more data we have, the tendency to measure the wrong thing only increases.  

What can we do? – let’s return to prison

Two things you can ask yourself:

  1. Am I measuring the right thing? Think about the prison example and about the way they used to measure basketball players. The fact that we can measure or that we have a certain stat does not mean it is the right one to use.
  2. What can that data I have tell me about the non obvious things. In Rudy Giuliani book “Leadership” he describes some of the process he implemented in order to improve New-York city. Most of them revolve around using statistics they had to measure different things. One story I remember vividly is again, about prison. They found out, that when sales in the prison cantina went up, it meant there are going to be prisoner riots. The prisoners were gathering food for the hard time after the riots. 

Vital signs

A few months ago, after reading a manifesto tilted: “Redeeming Sisyphus – Get Out of Control! Get More Done!” by I. Barry Goldberg of Entelechy Partners I wrote:  

Everybody knows saying by Peter Drucker that what isn’t measured does not get managed. But modern economics and behavioral economics, also shows us that if you decide on the wrong measures (or in Goldberg name: “vital signs”) you can create negative incentives. Books like “The Goal” by Eli Goldratt have been saying this for years. So, I believe the challenge of managers in the next few years, especially in the more subtle fields that are hard to measure will be to create the right vital signs

After reading Lewis’s article about Battier I think this is an understatement of the challenge. The challenge of managers today is to find the needle of right measures in the haystack of statistical data. The challenge is to re-think the way we measure our success. The challenge is to re-invent the way we interpret the actions of people.



Understanding the customer

A few days ago, Guy Kawasaki of “How to Change the World” blog, interviewed, Dave Wanetick, the managing director of IncreMental Advantage. The headline of the interview was “The Art of the Customer Surveys“. In a nutshell, Wanetick claims that most costumer surveys are actually useless because physiological and practical reasons influence the outcome of the surveys and make them obsolete. Actually Wanetick argues that the most efficient survey should consist of one question:

According to Dave, some of the most revealing customer surveys can be quite simple. Dave cites Fred Reichheld’s idea that one can distill customer satisfaction surveys down to one question:

“Would you recommend our service to your friends and colleagues?”

This is a powerful question because it gauges whether or not customers like your product enough to put their own reputations on the line with their friends and colleagues.

I must admit that a lot of what I read in the interview made sense to me. It is true, some people don’t like filling out surveys. There is ample importance to the framing effect in surveys. Hassling customers might irritate them. But do all these facts lead to the conclusion the there surveys are ineffective and should not be used?

Every novice business man knows that adapting the product or service to the client is very important. Some writers like Peter Drucker and Craig Stull, Phil Myers, David Meerman Scott, talk about the importance of understanding the “non-clients” and thier needs. So how are business supposed to do that without surveys? I was not convinced that you can understand everything you need about the client (or non-client) by just one question.

I know that I get a lot of surveys. Some I disregard. But some I fill out. When I do, I try to be as honest as possible. If I fill that the survey might be of relevance to me, I really try to give them my honest opinion. Maybe they will improve because of me – would that be great for me? I know that one time I filled a survey for one of the MBA Schools and actually won an I-Pod for doing it. I answered that survey honestly and it made me happy to be rewarded for it.

How many surveys did you fill out? Did you take it seriously? I guess that most of you will answer that they did. I think most people will.

So, what is the conclusion? I think surveys are a tools. When using a survey, you should know its limitations. You should know that it might help you reach a decision and that it isn’t a tool that makes the decision for you. You should be aware of the technique’s problems and be true with yourself by using it so it will produce accurate results and not the results you want it to produce. You should honor your customers, especially the ones that comply and fill it out.

Most of the tools we use to evaluate process of humans are flawed in some way. Tests like SAT or GMAT, are not always accurate. When I try to evaluate an instructor giving a class, my evaluation will be a little different from my colleague’s evaluation, even if we took the same training. It does not mean we have to stop using it or that the evaluation does not hold insights. It just means we should not fall in love with the tool and use it wisely.