I love Woot.com. I visit that site everyday to see what new product and shirt they have and to read the content.
It's funny because when you walk around the office you see people on their favorite sites weather it is Facebook, Amazon, Twitter, or whatever it is you normally don't see them on the site for the company they work for. Now this isn't huge problem for a lot of people but when you work on the web team you would think you would know your site inside and out.
I was recently helping someone with their site. They were doing an A/B test and need help doing the analysis. No problem, glad to help. After a couple of weeks I gave them the results and the alarms instantly started going off.
"Is this right?"
"Did you look at the right stuff?"
Let me double check, yes
"I need to see your code, this can't be right."
At this point it was obvious, it wasn't that they expected a certain result it's that they didn't understand something fundamental about the test, so I asked them "What is it that you were expecting to happen and why?
"Well the total conversions from the test don't equal the total site conversions."
Yeah, I know...so?
Well, because, the two test pages are not the only places on the site to convert from.
"But we are testing two floating pages that the customer can't get past unless they convert. They have to convert from there."
That's when I sat down with the guy and showed him his site. The page leading to the test had at least 7 links on it that went down a different conversion path. It wasn't immediate but it ended up there.
This was a guy that had been working with that particular site for at least a couple of years and was supposed to be the expert on it. It's not that he was stupid or that he didn't care it's just that he spent so much time in the weeds doing stuff that he never took the time to surf the site he was working on.
Over the years this has been a reoccurring phenomenon that I have noticed and yes, I'm ashamed to say, I've even fallen prey to it before. We get so busy checking page tags, running reports, analyzing A/B tests, and looking for low hanging fruit that will bump conversion half a percent that we sometimes forget to use the product we're trying to improve and not just use it from an analysts point of view but try using it from a customers point of view.
Take an hour a week while your watching 24 or something and surf your site. Do searches on Google, Bing, Ask, etc... to see how your site ranks, clink on the links to see what pages it takes you to. Do searches on your own site, does it return results? A couple of times a year actually buy something from your site and compare it to your favorite sites, how does it compare? Obviously there's a ton of stuff you can do but you get the point. Do it and it will provide just what the doctor ordered, insight!