A common problem seen in mid to large websites is the custom or CMS generated URL parameters, which can increase the number of duplicate pages indexed in Google. As a solution, Google Webmasters Tool introduced an advanced feature – URL Parameters. The parameter settings are recommendations for Googlebot. If you want to make sure that BOTS follow crawl and index settings, use NOINDEX directive and robots.txt. They are more powerful.
Warning: Before you start updating the URL parameters, be warned that incorrect changes can remove your pages from the index. Update the settings only under the supervision of an expert.
The URL parameters address a common use case.
Let us say that your e-commerce site shows 100 mobile phones under a category – “Samsung”. The customer is interested in sorting the page according to price and filters the page based on price.
You start with a url
www.example.com/samsung-mobile
After the customer has sorted by price the URL looks like
www.example.com/samsung-mobile?sort=asc
The content of the page remains the same but the order has changed.
Should Google index both the pages? No! Remember, each URL should be mapped to a page with unique content. The order does not matter.
So what kind of URLs are eligible for exclusion?
1) Where to Start: As a rule of thumb, URLs with key-value pairs should be first point of attention.
www.example.com/samsuing-mobile?price=asc&sort=latest
Key: price Value: asc
Key: sort Value: latest
Your category pages does not qualify for this feature
Ex: www.example.com/category/mobile-phones
www.example.com/category+mobile+phones
2) Not All Key-Value Pairs: URLs with key-value pairs that sort or displays a subset of the original page qualifies for URL settings but not all key-value pairs qualify.
For Example: If a websites by default shows 10 pages and the customer has an option to show all, then the page with the show all parameter is the page that webmasters should target for indexing.
By default, example.com shows 10 best-selling Samsung Mobile Phones.
www.example.com/samsung-mobile?show=10
At the bottom of the list, there is a button to show all. When the customer clicks the button, the page shows all Samsung Mobile phones.
www.example.com/samsung-mobile?show=all
Webmaster should not include ‘show ‘ as a URL parameter in webmasters tool. Let Google bot decide.
3) Look for Filters
Most e-commerce sites will have standard and custom filters to show a subset of the entire results.
In most e-commerce, there will be standard filters like
1) Size
2) Color
3) Price
Or custom filers like
1) Weekly Best Selling
2) Monthly Best Selling
You can exclude any filters that generate the subset of the entire result. But before you include parameters that represent the subset, check your analytics and see the traffic to the subsets. Don’t exclude subset pages that are more useful for the customer.
4) Dynamic Pages
Many e-commerce sites pass parameters to the URL to display a product.
Ex: www.example.com/shop.php?prod-id=123
Although they are not easily readable URLS, still they are unique and eligible for crawling.
Before including any parameters for exclusion, always use one guideline - “Don’t exclude any parameters that generates unique, useful or all pages?”