The Danger in User Surveys

One of the dangers, as pointed out by the BABOK is that surveys are not good at collecting actual behavioral information about our users. It seems to me that a large number of executives at Wal-Mart should have done a better job at understanding exactly when and why to use a survey.

If you’re not familiar with Wal-Mart or their recent attempts to de-clutter their stores, its well worth looking into. Back when this was announced a couple years ago, I remember thinking to myself, why would they want to decrease the number of products they carry in each of their stores? There are many things I dislike about Wal-Mart, poor product quality, never enough cashiers, the other shoppers in the store with me, but the one thing I never complained about was how many products they offered.

Sure, it wasn’t always the easiest store to find what you needed, but with the help of a sales associate or two, you almost always could find it. I can think of only a few times I ever shopped a Wal-Mart and was completely unable to find exactly what I was looking for or a close enough substitute that I left satisfied. Given that the number of options available was a good thing for me, I was astounded to find that management had decided to shrink their offerings.

Don’t get me wrong; I was all in favor of wider isles and better organization, but not at the cost of fewer choices. I don’t live at Wal-Mart, I just shop there. I can deal with clutter if I have to because at any time, I can turn around and leave the store.

This is a good lesson for those of us who work in projects to remember, that just because we think we know an answer and especially when we think we receive validation that our answer is correct, it doesn’t mean anyone was really asking that question. Customers said they would like less clutter in the stores, but did anyone ask in the survey if the customer preferred clutter or choices? Obviously, given the outcome, customers choose clutter and options.

Have you ever used a survey in a project? Was the answer you received as faulty as the one Wal-Mart found?