If You Can't Give Me All 6's

Whether you purchase a new car, stay at a hotel, dine in a restaurant or even shop on eBay, chances are you'll be surveyed to find out how satisfied you are with the experience. Customer satisfaction surveys are so popular because they provide valuable insights for companies looking for ways to improve customer relations.

Some companies, however, put so much emphasis on these surveys that customers feel coerced into providing a false high rating. It's not uncommon for an employee to tell the buyer, "If there's any reason you can't give us all 6's, give me a call before filling it out so that we can make things better." In addition, some salespeople are so bold as to mention how a bonus or other incentive depends on receiving only the highest marks.

The reality is that consumers are being bombarded by employee attempts to make them aware of the almighty survey and to sway the results. We call it "survey talk" and it is spreading from industry to industry. Builders have recently joined the ranks.

The April issue of Professional Builder included an article by my colleague and friend Scott Sedam, who describes the "survey talk" he observed with one builder. He accurately maintains that integrity was being sacrificed in order to get good survey results. I, too, have seen builders engage in aggressive campaigns to increase scores, and for the past several years our team has been actively studying the effects of this activity on consumers.

Perhaps it's a necessary practice that we self-promote for high scores with our customers and in some cases even campaign for better ratings. Politicians often go door to door to get out the vote, so what's wrong with businesses doing the same thing? This sounds reasonable. But a client experience gave me reason to further examine this viewpoint.

The president of a major home building company called me up one day and said, "Paul, I think you have a major problem in your survey. Our scores in every category are near the best in the country. We have no major low spots in our data. And, yet, our 'recommend to friend' scores are stuck at the 89 percent level."

I examined the data and confirmed what he was saying. In fact, this builder was scoring extremely well in lots of areas. It is rare to have a builder do this well on the survey and not score high on the "recommend to friend" question. I was confused.

Naturally, I was quite motivated to resolve this issue and headed down to spend a few days with him and his staff. I conducted various employee brainstorming sessions, and interviewed buyers to see what the cause was of this apparent "glass ceiling" in their recommend levels. I found nothing. Buyers expressed clear satisfaction with all areas, with the exception of a couple of minor issues — nothing severe enough to explain their lukewarm willingness to recommend the builder to a friend. Clearly this builder was doing something that was damaging the loyalty of his customers.

It eluded me until the last day of my visit. I had lunch with the client and explained how I really couldn't explain the source of the problem and that I would need to go back and reanalyze the situation. The builder was frustrated and vented, "Paul, we do everything for these customers. We break our backs to make them happy, and your survey isn't picking it up. We even go to every home right before your survey goes out and tell them how to fill the darn thing out." I asked him to explain, and he went on about how he instructs his superintendents to knock on every door and tell the buyers that it is very important that they fill out their survey with all 6's and check "definitely yes" by the "recommend to friend" question. He even rationalized his tactic by pointing out that this is what his car dealer did when he brought his car in for service. Eureka! I found it. It was clear that he did a good job with his customers, so why was he badgering customers door to door? Was this causing his customers to be offended and not respond as positively? I told him that perhaps we are putting too much pressure on buyers and that we should back off a bit and see what happens. He did and the glass ceiling lifted. That's when the concept of "survey talk" was born.

I decided that an in depth analysis was needed to better understand the impact it had on consumers. Is it good to do? Well — as an infamous President once said — it all depends on what your definition of "it" is. Not every builder undertakes survey talk with the same vigor as the client described above, and that makes it more complex to research.

Since 2002, NRS has been asking homebuyers, "To what degree did your builder attempt to influence your responses to your survey?" Buyers responded on a 10-point scale, from a high degree of influence to no influence at all. The latest survey of 63,651 homeowners from the 2004 NRS Award presented by Professional Builder, helps to shed light on this subject (see www.housingzone.com [1] and search NRS for more information on last year's program details).

When we examine the two highest levels of influence from builders, the results support the theory that overly aggressive "survey talk" results in lower ratings. Specifically, there was a statistically significant drop of 4.08 points (p<.001) in the "recommend" levels from the second highest to the highest levels of influence. The bottom line: If you campaign too hard, you will reach a point of diminishing returns.

However, we also analyzed the data to see if there is any positive effect with the other levels of influence measured by this question, and the results were surprising. This comparison revealed that there is a statistically significant increase between those companies that do engage in survey talk and those that do not. Specifically, there was a statistically significant increase of 2.59 points (p<.001) in the "recommend" levels for those companies that did some campaigning vs. those that did not.

This result supports the idea that some survey talk has a positive effect. But how much is too much? Imagine a graph of your company that charts the intensity of survey talk with buyers' favorable ratings. As survey talk increases, ratings begin to improve. But at some point, additional survey talk causes ratings to fall. The area between these two points is what we call the Zone of Awareness, and this is where every builder wants to be. To be outside this zone means either you're not communicating enough with buyers about the survey or you're going overboard. Either way, you're doing more damage than good.

We often hear that most things in moderation won't hurt us and may, in fact, be beneficial. Take that same advice in regards to survey talk. For companies that are doing well, I recommend holding back on aggressive campaigns.

More needs to be done to fully understand what is going on here with customers, but clearly there is a negative effect if your employees go too far. Go with controlled messages like letters, signs, and placards. It's like those studies that suggest that a glass or two of wine each day is more beneficial than not drinking any alcohol at all. Yet, we all know that too much alcohol can lead to a host of medical problems. The rule of moderation applies to "survey talk", and your satisfaction results depend on it.

For 2005 we have modified the NRS survey to further investigate this relationship with buyers. In the new study we will be applying new questions that delve deeper into the level of campaigning that a builder may undertake and the effects on buyer loyalty.

We anxiously await the response to these new questions, which will shed more light on this elusive topic and provide the data that builders need to improve their customer satisfaction scores.

Stay tuned as we release these exciting results at the 2005 Benchmark Conference in September and the NRS Conference in November.