BlackHat360

Yesterday we discussed the HOW portion of detecting duplicate content. Today I want to get into the actual process itself.

A wide spread Theory in the SEO world states that duplicate content not only carries a heavy penalty, but in fact can and will lead to a domain being banned or deindexed. Today I am going to discuss why I believe that this is not only unfounded, but perhaps completely untrue.

Lets start with some facts and figures. I’ve had the pleasure of reading dozens of research papers from msn, yahoo, google, and other leading members of the academic and professional search arena. From these papers it’s easy to determine that duplicate content detection is entirely possible in theory and at least partly in practice, but I believe the “practice” portion is where almost everyone may be wrong.

So what would it take for the big G to pull off duplicate content testing in the real world? Well, lets start by looking at the numbers. Lets assume it’s still 2004 and google still has “only” 8 billion pages in their index. Estimates show that they have several PETABYTES of data across their datacenters. So i’m joe webmaster and I put up a page about sprinklers. Does anyone here really believe that Google or anyone else on this planet actually has enough computer processing power to take my single page about sprinklers, shingle it and compare it to their other 7,999,999,999 pages of content each of which needs to be shingled as well? Shingling as we discussed yesterday, is the process by which search engines determine unique content from duplicate content. Of course, you do have the problem of it being a very intensive calculation because you’re not comparing A->B you’re comparing every document against all other documents.  I think they call this a O(n²) problem.  and it happens to be a very expensive process cpu time wise. Unless a page is flagged to begin with, it would be cost and time prohibitive to carry out such an expensive calculation on every page in their data set.

So if this is the case, what is duplicate content used for? What is the scope of the data google is looking for? I believe they check for duplicate content on a PER DOMAIN BASIS, meaning they take a single domain, check the content and run comparisons to give the overall domain a content quality or duplicate content quality score. Lets see why that makes sense on several levels. First, it’s within the ability of their crawler to do such a thing from a cpu processing power perspective, it also makes sense that they would factor this into the overall quality score for a domain.

Now the evidence:

1) A year ago I put up a 100 percent clone of wikipedia. I used the wikipedia template, I copied the data from their database, etc. This new domain was 100 percent identical to that of wikipedia.com.

The result? I rank well for thousands of terms, the domain has almost 1 million pages indexed in google, and it receives 3-5K uniques per day. So much for a duplicate content penalty. Of course the content is highly unique from page to page on the domain, but it isn’t unique when the scope is expanded to include the entire internet.

2) PublicBlend.com - By definition all social media sites contain 100 percent duplicate content that would never pass a shingling algorithm. All of our stories come directly from other web pages. In fact they are direct copies of articles from all over the internet.

The result? PublicBlend.com has been steadily growing in search engine traffic every month and now receives over 3,000 uniques a day from google alone. (we recently changed the domain name, so the indexing has started over)

3) News sites, not just social media, but regular news media as well. Reuters is the source for 90 percent of the news on the net. Everyone duplicates their stories word for word yet they all rank well for the resulting stories.

I hope the above sparks some debate and discussion on the topic of duplicate content. It may also raise some other interesting questions:

From a white hat perspective, what happens when 50 spam sites scrape your feed?  Will your content get penalized or will the spam sites get penalized? How would a search engine determine who wrote the article first? Would they simply rely on domain trust? If so that opens the door to all sorts of gaming options using old trusted domains.