Extracting Data from Forums: 3 Sources to Discover What Your Market Really Thinks

Robert Collier, the great ad man of the early 20th century, once summarized the secret of all effective marketing as entering “the conversation already taking place in the customer’s mind.”

That’s powerful advice … and difficult.

Why?

Because most of the sources we normally turn to for market research are woefully incomplete.

For example, surveys and questionnaires that aim at gathering qualitative data are notoriously ineffective. Generating an accurate picture of your audience, demands at least an 80% response rate. Unfortunately, Survey Gizmo estimates that external surveys produce an average of 10-15%.

On the other hand, online analytics that promise to deliver quantitative insights also possess limitations that make them unreliable when it comes to making important market-research, product-development marketing decisions. Most notably, while analytics help you understand the number, they lag when it comes to capturing emotions.

So, where can you turn to?

The answer is forum mining.

Forum mining is essentially collecting and structuring user-generated and user-posted comments from across the web. The insights from forum mining are powerful because they come directly from the actual people in your target market: their minds in their words.

Not only can you “stream” this type of data by live monitoring the platforms your customers or potential customers post on, you can also extract a wealth of knowledge by scrapping them for historical data. By digging into the last two to three years of user-generated and user-posted comments, you can unlock a treasure trove of otherwise unattainable wisdom.
To do this effectively at a large scale, you’ll need an unstructured-data gathering tool like Webz.io.

But before that, you’ll also need to identify exactly where you plan to do your audience research.
Here are three major sources — along with a host of specific and very practical suggestions under each — for using forums to discover what your audience really thinks.

1. Review Sites

Did you know that consumer reviews are nearly a 12 times more trusted than product descriptions? This means that the data you collect from review sites not only comes directly from users but is also far more likely to affect what prospective customers think. If you are creating a new product, looking to improve an existing product, or even trying to unearth effective marketing tactics, review sites are a gold-mine.

Where should you look?

Amazon Customer Reviews
Since 1995, Amazon has allowed customers to post product reviews.
This has made it one of the go-to places for people who want to buy a product or get a service before making the final decision. Besides the customer reviews, Amazon includes a rating scale that helps customers make more informed and faster decisions.

For your purposes, categorizing and mining the actual reviews of other people’s products will guide you into making far more customer-driven decisions.

Product Hunt
Product Hunt is a relatively new and particularly unique product-listing service.
It has four main tabs: Popular, Today, Chats and Profile. The Popular tab shows the “best-of” in every category while the Today tab shows the newest products in each of the many sub-categories. Product Hunt describes itself as “a curation of the best new products, every day.” However, probably the most valuable thing thing about it is the community of experts, makers, and consumers who not only post products, but their reviews.

Yelp
The cumulative number of Yelp reviews is 90 million and its average monthly unique visitors is 142 million. Wow! Yelp allows you, as a user or business, to set up a profile. As a user, you can review products and as a business you can have your products reviewed. You are also allowed to respond to reviews. For free.
Yelp is especially helpful if you — just like the next source — if you offer a service to your customers or if you have both an ecommerce and physical business.

Angie’s List
Unlike most, with Angie’s List you will have to pay to get certified and unbiased reviews of businesses in the US. The upside is that the reviews you get are more reliable than what you get on most of the free review services.

It also means the reviews are not anonymous. This cuts down on fake reviews along with biased reviews posted by companies themselves, both of which lead to misrepresentation and bad data.

ConsumerReports
ConsumerReports is an independent nonprofit organization that focuses on testing, rating and recommending products.
The reviews provided are legitimate and reliable. For each, editors and authors clearly explain the criteria used, a big-picture overview, a buying criteria, price scale, and sharing (social) buttons. It is a great site to go to if you are looking to create a new product or improve an existing one.
They even have a fantastic article that’d be great supplemental reading after you’ve finished this one: “A user’s guide to user reviews: How to avoid the hype and get the most from Yelp, the Better Business Bureau, and other online review sites.”

Epinions
Epinions.com belongs to the Shopping.com family. It’s a site that solicits reviews on (almost) anything you could ever buy online.

2. User Comments

Whether it’s on Facebook, industry or company blogs, or YouTube, readers and viewers like to comment. They like to be heard and most importantly, they like to see action on what they comment about. And that should be your point of focus – acting on the comments.
If only you knew exactly where to find all the users screaming to be heard.
Below are the top sources of user comments you can use to your advantage in not only improving existing products and services but also creating newer and better ones.

Industry Blogs
Yes, this is about the competition.
Why? Because you have to know exactly what their customers want … and then provide it before the competition does.
Of course, one of the realities about getting comments from blogs is that not as many people that read actually comment. However, when someone does take the time to respond, you know they really have something to say.

And this is what you’re after.

Suppose, for instance, a post is about a new service (similar to yours) another company is offering. By isolating and reviewing negative comments, you’re able to side step your competitions pitfalls.

Still, you’re not just looking for what your competition is doing wrong. You also want to know what they are doing right. Some users will also comment about the good they see in the service. That’s your chance to use take their own nuggets and use them to your advantage. And no, this is not to encourage you to simply copy what those users say. It’s to help you understand what users want.
Also, it is important to follow industry-specific blogs.
Mining the comments in relevant blogs is a huge step in improving your own products and services. For instance, if you are in social media marketing you should definitely follow blogs such as RazorSocial, Buffer Social, Convince & Convert, and more. Are you in the tech industry? Are you following Mashable, Gizmodo, GigaOM, ZDNet or TechCrunch and observing how their audience interacts with them?

It’s time you do.

Reddit
Reddit implements ratings so well that you are able to read the comments that matter the most and ignore the rest. This is because the best comments are shown first while the bad ones are buried deep. This gives you a chance to quickly scan, collect, and then utilize what’s most relevant.

YouTube
When someone watches something on YouTube and are affected, they’re naturally inclined to leave a comment. This is true both of what they love … and what they hate. By communicating what they do or don’t like about a video (which could be of a product or service) they are directly telling you which features to include and which ones to avoid in your own.

Relevant Facebook Pages
Just like company blogs, relevant Facebook pages provide a detailed portrait of what users like or dislike about a company, a product, or a service. Moreover, it’s an incredibly effective way to spy on your competition to see what they are doing right, where they get it all wrong, and use insights from both to improve your business.

3. Question and Answer Sites

Below are some of the most reliable sites on the web when it comes to asking questions, getting answers, and getting people’s opinions on popular subjects.

GitHub
Besides being the go-to resource for software development help, GitHub also functions as a social networking site for software developers themselves. As a GitHub user, you get to have a profile which showcases your previous work and previous contributions to various projects through pull requests. Project revisions are discussed publicly. This gives industry experts a chance to contribute and collaborate towards collective advancements.

It also give you an invaluable resources for your own SaaS product or service.

Quora
Just like its tagline says, Quora is “the best answer to any question.” Established just six years ago, Quora has grown to be one of the most reliable Q&A websites. Quora allows users to ask and answer questions on any subject under the sun. To improve the effectiveness of the answers, the site also allows users to edit other users’ answers.
The great thing about Quora is you can use both as a way to interact directly with your audience, by posting your own questions, or “listen in” on what other people are saying by mining their boards through keywords.

Answerbag
Similar to Quora, Answerbag allows users to browse through various categories to post and find questions and answers. However, before you can dig in you have to register. While sometimes a hindrance (especially if you’re trying to automate the process of forum mining), registration helps ensure a higher quality of comments.

Blurt it
Unlike most of the other sites listed here, Blurt it allows you to ask and answer questions without having to register. There are various categories of questions, answers and opinions through which you can browse and search.

Askville
This is an Amazon-powered website. When you ask a question, other related questions are displayed to help you find an instant answer. In case the question is unique, you are allowed to post it on the site so people can help answer it directly.
Once it’s answered, you receive a notification via email so you can easily browse through the answers.

Answer Bank
Answer Bank provides answers on various topics including law, music, family, seasonal, and more. Most of the questions are topical and far more personal than some of the sites listed above. They revolve around the current news, the entertainment, and sports world, which makes it a great place to mine if your product or service overlaps with popular culture.

What Does All this Mean?

Is collecting user-generated data from forums easy?
No.
But if you do it right — especially if you supercharge your process by automating it through a service like Webz.io — gathering consumer-generated data needn’t be impossible or ineffective.
Webz.io specializes in collecting and structure massive amounts of unstructured data. This means using forums as data mining resources will normally take place behind the scenes.However, becoming an active and valuable member of the platforms and sites that matter most to your customers is also important. Just keep the answers to these three questions in mind:

“Where do I look?”
You should know the kind of data you want to collect first and then where to look for it. Research from the most relevant sources you can find to avoid wasting time. Also, consider your ideal customer and where she is most likely to be. That’s where you need to be.

“What are my intentions?”
Sounds like a stupid question, right? Maybe not.
The biggest reason why most people don’t consider going to forums for information is because they know most of the members are there to promote their own services and not add value. This means …

“Can I offer something in return?”
If you are going to answer other members’ questions, focus on providing value in exchange for learning.
If your main intention is to randomly post links leading to your site then you, all your efforts will be suspect until your priorities are set straight.
Instead, go behind the scenes with Webz.io and start “entering the conversation already going on in your customer’s mind.”