Isn’t Sherlock Holmes a great, iconic name for a character? Author Robin Sloan thinks so. To name the main character in his next novel and to find a name that resonated with people, he decided to use Google AdWords to do his market research. Now there’s a cool idea.
Here were his steps, as outlined on his website:
- Created a campaign using a bunch search terms including mystery, detective story, sherlock holmes, noir, and more.
- Settled on test character names.
- Created an ad for each name, all with the same body text but each with a different name swapped in for the headline.
- Launched the AdWords campaign with a limited ($40) budget.
About 100,000 impressions later, the author had results, and definite statistical significance on the effectiveness of the ads.
Using This Fun Experiment To Discuss Market Research Experiment Parameters
I love this idea. It’s creative, interesting, and fun at heart. The analyst nerd in me looks at how the lighthearted market research experiment was put together and thinks about quality score, and audience, and the most important metric to judge success. So, humor me, and let’s use this market research test as a launching point for a discussion on these topics.
Keyword Selection Is Market Audience Selection
The selection of test keywords affects the test group and therefore the test’s usefulness and relevance. The context that the results will be used in isn’t an online business, it’s a book. In an online business, select marketing survey traffic based on the words you plan to advertise on or that you believe are at the dead center of your message. In this fun exercise, though, people searching for the terms will never be the exact audience mimicking the readership of Robin Sloan’s books, but given that, the search terms used (and their match type) will determine how close you are. If the author is prominent enough, an exact match on his name or on the titles of his past works are probably the best way to select the test subject for market research. Although it’d be an interesting idea to use the keywords “sherlock holmes” or “sir arthur conan doyle” to select test subjects if you want the opinion of fans of that work instead. Also, exclusively using exact match is likely your best bet for selecting the purest audience.
Landing Page Quality Factors In When Surveying Your Public
Practically speaking, landing page quality score would likely get you in trouble with a test like this, eventually. That’s my best guess. Google has a fun-loving streak, but I’ve seen tests like this get a landing page penalty in the past. You’d likely have to build out some content about the book and the characters related to the keywords selected and rather than send it to an author homepage, or the test would get expensive as Quality Score sunk to the 1 to 3 range. If the keywords that represented the most appropriate market audience weren’t high traffic and the test was going to run for quite some time, it’d be even more likely Google would find landing page quality issues if there was inconsistencies between the messaging in the ad and the landing page or a lack of original, relevant content.
Is CTR The Best Metric To Judge Success?
Lastly, click through rate (CTR) might not be the best metric to judge the iconic status of a name, just as CTR isn’t always the best way to judge an ad’s effectiveness at selling a service or product. Many times I’ve seen ads with higher click through rates have lower conversion rates. For instance, at an extreme a character named “Free Apple Laptop” wouldn’t be particularly iconic, but put those words in where the character name is in the test ad, and the split test would likely result in “Free Apple Laptop”. In online businesses, profit per impression is the best method for judging the best ad in a split test.
None of this analysis detracts from the playful genius of the experiment Sloan undertook, though, and I must say, seeing his post really brightened my day.
By Rob Sieracki
Director of Paid Search
