BlackHat360

Today, Microsoft publicly unveiled its soon-to-launch search engine Bing. It will become available over the next few days, and be fully launched by June 3. On the surface, Bing has a distinct gloss. The home page features a rotation of stunning photography, for instance, which can be clicked on to produce related image search results. But the most significant changes are under the covers. “We have taken the algorithmic programming up an order of magnitude,” says Microsoft senior vice president Yusuf Mehdi. Each search result page is customized according to what type of search you do (health, travel, shopping, news, sports). The algorithms determine not only the order of results on the page, but the layout of the page itself, concluding what sections appear. These sections can include anything from guided refinements and a list of related searches in the left-hand pane to images, videos, and local results.

I’ve been playing around with a preview version of Bing for about a week. It is designed to be “more of a decision engine,” says Mehdi. Bing helps people make decisions through guided search and a focus on task completion. In a time when a new Website is created every 4.5 seconds, information overload is becoming a real problem. ” People are getting hundreds of thousands of links but not getting what they want,” says Mehdi. Bing tries to alleviate problem by offering up different experiences depending on the search.

The internal codename for Bing is Kumo (which is what you see in the screenshots), and the current release is called Kiev. Rather than a spare, blank screen, Bing’s homepage surrounds the search box with a single beautiful image, such as the one of the tribesmen above or a kinkajou. You can hover over parts of the image to get factoids about the image or click through to an image search result page to explore more. The left-hand pane offers the option to narrow your search on images, videos, shopping, news, maps, or travel. Each of these has a different look and feel. A travel search will turn up a page based on Microsoft’s Farecast technology asking you where you want to go, with flights, hotels, and destination information. A news search offers up headlines, photos, videos, and local news in a column on the right. A shopping search will bring up products and is tied into Microsoft’s Cashback program.

Every search also generates a guide on the left to help you refine your search. A search for “kinkajou,” for example, lets you refine by images, facts, sale, breeders, care, diseases, and videos. A search for “Samsung LCD TVs” brings up an entirely different set of guided results: shopping, review, manual, repair, buy, stand, images, and videos. If you search for images of “butterflies,” it lets you sift to show just Monarch, Swallowtail, Viceroy, Owl, and other types of butterflies. All of this categorization and concept-matching is Microsoft’s early attempt to bring in some basic semantoc search technologies into a mainstream search engine. Each guided option is dynamically generated, just like the different sections of the search results page. “Google, tried to preempt this,” says Mehdi, referring to Google’s new search refinement options it launched last week, which is also in the left pane. Those Google options, which include the ability to search across different time periods or for related keywords, are “completely static,” criticizes Mehdi. “There is nothing new about it. It is a very minor rev, not as sophisticated as what we are doing. For us ever query is special.”

Bing also takes advantage of Microsoft’s acquisition of Powerset to provide better previews and snippets of text when you hover over a result. Also, whenever a search brings up a “reference” tab in the guided exploration pane, clicking on that will bring up an enhanced Wikipedia article with semantic tags.

Onstage at the D7 conference, Steve Ballmer acknowledges: “There is no way to change the whole game in one step.” But search “deserves a good feature war.” And Bing will be rolling out new features as it goes forward. But is it enough to get people to switch? Bing is certainly not a game-changer, but it does cut out a lot of the back and forth that happens with so many searches today. If Bing can help people find what they are looking for faster, it will put pressure on Google to keep advancing the ball as well.

Within the next few days, Microsoft is expected to unveil its latest attempt at trying to be a player in the world of web search. After it has failed to get live.com any traction against Google, it will apparently launch a new engine called “Bing” — the project formerly known by its working title “Kumo.” This should be unveiled at the D conference which starts today in Carlsbad, CA — but it looks like Microsoft may be giving us a peak at the logo a tad early.

While it appears that Microsoft may have already taken it down, I visited bing.com in my browser about 10 minutes ago and sure enough saw the favicon you see above. It’s a lowercase “b” with a yellow/orange dot in the middle. It would appear that this will be at least a part of the Bing logo. The light blue and yellow/orange color combination matches that of Kumo. I find that combination to be quite ugly — sort of like the Cleveland Cavaliers basketball uniforms (below) from the 1990s — but hey, that’s just personal taste. All that really matters is now the search engine actually performs.

This favicon, which again, may only be a part of the logo, also looks a lot like the logo for Blinkx, the video search engine. That features a red lowercase “b” with an eye in the middle.

Microsoft is spending some $80 to $100 million on a marketing campaign for Bing, according to Ad Age. That’s huge by any standard, but especially when you consider that Google only spent $25 million on all of its marketing last year. I don’t know what Microsoft plans to spend all that money on, but I get the sneaking suspicion that Bing Crosby will be involved in some way or another.

I’ve been quite busy for the past week or so, but that hasn’t stopped me from tracking results. There are 10 main key phrases that I consider vital for my traffic. As of this morning I am on the first page of Google for all 10. One of the more popular phrases has 54 million results and is quite competitive, so I have to say i’m quite pleased about that. Traffic is up, subscribers are up, and i’m ranking for all of the phrases I need at this point. Now that i’ve seen the results, all I need to do is target a few dozen other phrases and take things to the next level. Stay tuned.

Google has just launched a new “search options” feature on its main search page. When you click on “Search options” you can filter your search by different types of results (videos, forums, and reviews), by time (recent, past 24 hours, past week, past year), as well as seeing related searches, a “wonder wheel” view, or a timeline view.

At Google’s Searchology event, which is going on right now, Marissa Mayer listed the following as the hardest unsolved problems in search:

- Finding the most recent information
- Expressing that you want just one type of result
- Assessing which results are best
- Knowing what you’re looking for
- Expressing your searches in keywords

Notice that real-time search is the No. 1 problem. (Twitter and a bunch of startups from OneRiot to Tweetmeme are also working on it, with the latter two launching their own real-time search efforts today). And it certainly is a problem for Google, even with the new recent results option. Try searching for any of teh top trending results on Twitter right now like Miss California (vs. Twitter search results) or Star Trek (vs. Twitter results), and you don’t even get any Twitter results on Google.

While real-time search is still a big problem, it is not the only problem. Some of the new options address the difficulty of searching back through time. The recent results get as real-time as Google can get, but you can also expand the timeframe. And you can look at an actual timeline of results, which looks for dates within results and then places them chronologically (this is sort of hit or miss—just because a date is mentioned in a text does not mean the entire result is about or from that period of time). Google now also lets you see related searches as an option. And the Wonder Wheel is more of a visual aid to see how different related topics are clustered together. When you click on any spoke of the wheel, it then causes that search term to be at the center. We’ve seen many of these techniques in the past, but Google is giving them a higher profile by putting them in its main search page..

One of the next frontiers of search is taking all of the unstructured data spread helter-skelter across the Web and treat it like it is sitting in a nice, structured database. It is easier to get answers out of a database where everything is neatly labeled, stamped, and categorized. As the sheer volume of stuff on the Web keeps growing, keyword search keeps getting closer to its breaking point. Adding structure to the Web is one way to make sense of all that data, and Google is starting the tackle the problem with a Google Labs project called Google Squared, which Marissa Mayer mentioned earlier today at the company’s Searchology briefing.

Google Squared extracts data from Web pages and presents them in search results as squares in an online spreadsheet. Michael was at the event and got a personal demo (see video below). From Michael’s Searchology notes:

Google Squared is launching later this month in labs. Google Squared returns search results in a spreadsheet format. It structures the unstructured data on web pages. So a search for Small Dogs returns results with names, description, size, weight, origin, etc., in columns and rows.

Google is looking for data structures on the web that imply facts, and then grabbing it for Squared results. “It takes an incredible amount of compute power to create one of those squares,” she says.

This type of technology has obvious applications for many types of targeted searches, including product search, health search, scientific searches, you name it. There are dozens of semantic search startups trying to impose structure on the Web to perform similar tricks. Another high-profile search startup which is launching on Monday, Wolfram Alpha, takes a slightly different approach in that it simply ingests massive amounts of information into its own databases where it can query it to its heart’s delight. Already there is a bit of a rivalry between Google and Wolfram because getting back structured results is a major new direction for search.

Wolfram does a pretty good job parsing the information in its own databases, but those databases will never match what is available on the Web. Wolfram’s databases currently store only 10 terabytes of information, a tiny fraction of what is on the Web. (I will be posting my impressions of Wolfram’s search engine soon). Google Squared is an early attempt to take the messy data which exists on the Web and place it into simple tables. It is still very experimental and isn’t always on target, but you can see where this is going. Turning the Web into a giant database will crush any attempt to segregate the “best” information into a separate database so that it can be processed and searched more deeply.

In the video demo below, a search for “camera” sorts the results in different columns by images, description, and manufacturer, resolution, etc.. You can refine results by clicking on a particular column such as manufacturer. A search for “rollercoasters” sorts results by name, image, description, height, length, and number of inversions. But sometimes it gets confused. A search for “spaceships” turns up a Corvette and a missile carrier. It is going to be a while before this makes it out of Google Labs

In case you missed it, this is a multi part series about Link Exchange. You can get caught up to speed on the details and setup here: Link Exchange

For the rest of you, lets dive into the numbers. This is now the 4th day of use. I am still running without a link throttle and averaging 16-18,000 link views per hour. Let’s take a look at the screenshot:

 I have of course removed the link ur’s and the keywords i’m targeting. We can see a tremendous number of google link views which really seems to be translating into increased 

indexing and higher rankings. The domain is now pretty much 100 percent indexed, and as the following image will show, the traffic has really increased. Rankings for nearly all of my target keywords are now showing up on the first page of Google with several in the number 1 position. Now, we all know Google is important, but hardly the only  game in town. Lets take a look at Yahoo and MSN/Live. Yahoo at the start of this was showing 0 pages indexed. We simply didn’t show up. As of today, we have 71 pages indexed and  show 506 incoming links. Quite impressive, and even more impressive is the fact that we now rank in Yahoo for two of our most important target keyword phrases. In fact, today we received our first human traffic from yahoo for our main target phrase.  I expected Yahoo to take several weeks, but the sheer linking power of LE seems to be reducing that expected lag. As for msn, the same starting scenario with us having 0 pages indexed just last week. Today, we see 29 indexed pages. Definitely an impressive start for only being 4 days in. 

Next we will take a look at the most important data. The analytics:

As we can see from the graph, it’s quite the increase over our starting point. Recall that we were averaging 0-2 searches per day. Yesterday we had 98 incoming hits from search engines, and in fact we had several new members register as a result. 

That’s about all for today. Expect another update on Monday as we follow the progress

The order that pages appear in the results of a search at a search engine may be influenced by the number of pages that link to that page, and by rankings of the pages that link to that page.

I’m starting a new series here with some case studies related to software I use on a regular basis. While the majority will be Black Hat software, i’m starting off with one that can be used by both Black Hat site builders and White Hat site owners. The software in question is Link Exchange which is part of the Simplified Search Engine Suite. As of today I am told that there are about 200 servers participating in the Link Exchange and close to 100 million links.

With that out of the way lets get started with a little bit of background and the setup.

I’m going to be using the software on this blog. I figure this is a fantastic starting place because the site is fairly new, and I have done absolutley nothing for link building or SEO. So, to get going I did some basic keyword research, checked out the competition for my main phrases and ended up with a list of 70 or so keywords as a starting place. Now, before we jump into the rest, lets take a look at the search engine traffic for the month of April for this blog. I’m going to focus only on search engine traffic rather than overall traffic so referring sites, subscribers, and direct traffic don’t taint the stats and throw the graphs off. 

  As you can clearly see, we’re starting off with a clean slate. The search engine traffic for this place is basically zero. I checked the serps for all 72 of my targeted keywords/key phrases and we didn’t rank in the top 100 for any of them. I’m also keeping tabs on the total number of indexed pages for the domain which as of the start date was 316 pages.

So, lets fire up Link Exchange and see what we can do. The software is web based, so you do need your own dedicated server to run it. I happen to have several including the one that hosts this blog, so I installed it here. The system is points based, so you need to display links somewhere in order to receive points which in turn can be spent on incoming links. The cool part is that the sites that display the links and the ones you receive links with are completely independent. In my case i’m using some forums I run to gain link points, then spending those points on this blog. Make sense? The interface is very easy to use as you will be able to see in my screen captures. I started by logging in and clicking the import links button. As you can see in the picture, you are presented with an easy to use form where you simply insert your anchor text and url. There is also an option to flag the link as a white hat link or a black hat link. Even though the blog is a black hat blog, the blog itself is white hat in nature, so I selected that as the option. The developers run through the links every few hours to make sure no one is trying to sneak a black hat link into the white hat links and so forth. You also have the option of uploading a csv with all of your links. This comes in handy when you have a large number of links to submit at one time. I didn’t, so the quick import works fine.

Now that we have some links in the system, we need to configure our sharing options. You can turn linking on and off at any time through the interface, but the real power comes from the advanced options. Here you chose which search engines are actually allowed to view the links. Did I mention that these links are cloaked and only viewable by search engines? Very handy for keeping people from being able to simply use the system to find other users of teh system. So, in my case I want all search engines to view the links, so I leave the boxes unchecked. Next up are the link throttle options. This allows you to determine how often your pages or domains receive links. This can be especially handy for black hat sites where slower link building may be more desirable. In my case i’m going to go with unlimited links for now to see what this system can actually deliver volume wise. The last set of options are all related to the anchor text. This one is interesting. You can actually have the system vary your anchor text based on several criteria so your incoming links are more varied and natural looking. Say you have a 5 word phrase you are targeting for example. This allows you to randomly drop words from the beginning of the phrase, the end of the phrase or both. You can also manually set how often this should take place based on a percentage of the overall link views. 

I took care of all of this Monday morning. Now, nearly 48 hours later, how has it done? I must say I didn’t expect results this quickly. 48 hours in, my indexed pages have risen from 316 to 415 as of this writing. I also noticed that all of my pages have been recrawled and recached which is nice because I made some changes to the page titles a couple weeks ago for better SEO. So, that right there accounts for a 24 percent increase in the number of indexed pages. More important than that however is the results of the keyword ranking. I am now in the top 100 for nearly all of my keywords with the majority o nthe first or second page of Google. My main key phrase that i’m targeting is now actually sitting at number 1. It was number 5 yesterday, so it moved up another 4 spots. Now, due to an error on my part, I don’t have full analytics traffic data for yesterday, so the following image accounts for yesterday evening and part of today. Remember that just a couple days ago, this site was hovering around 0-2 search engine hits a day.

Not bad, 15 hits in less than 24 hours.  That’s quite the increase over the past days. It’s exciting to see results so quickly. The coming days/weeks should be quite interesting. 

Earlier I talked about the link throttle a bit. I wanted to see what sort of volume the system could deliver. Lets take a look at that graph in the built in Link Exchange Analytics system:

I blanked out the keywords i’m targeting. I have to keep some info to myself ;). As we can see, google saw my links 166,000 times in a 24 hour period. That’s very impressive and shows that the system can clearly deliver on volume if the link throttle is turned off. The graph shows the number of link views per hour, we can see in the keyword table which anchor text was shown and how often, we can also see for which domains the majority of the links were shown. 

That’s about it for today. Follow me again tomorrow for a followup as I continue to use the system.

Microsoft Explores an Algorithm to Increase PageRank for Pages Linked to by Blogs.

In the new patent document, they ask if the rankings of web pages in search results would be improved by a providing a slight increase in the PageRank of pages linked to by blogs. They tell us that:

This idea is based on the assumption (or hope) that blogs are still mostly human-authored, and that links from blogs generally represent sincere endorsements on the part of the authors.

The December post explored how a search engine might be able to identify blog pages and distinquish them from non blog pages, and told us that:

Search engines are increasingly implementing features that restrict the results for queries to be from blog pages.

But limiting the number of blogs that show up in search results doesn’t necessarily mean that a search engine doesn’t like blogs. It may mean that search engines would prefer to show a diversified set of search results, including blog pages and other results.

Ranking Algorithms

Search engines often look a couple of different kinds of ranking factors when determining the order that search results are shown to searchers.

Query-Independent and Query-Dependent

One way to classify ranking algorithms is query-dependent (or dynamic) or query-independent (or static).

Query-dependent ranking algorithms rely upon the query terms someone uses to rank pages, while query-independent look at other factors such as how important they may believe a page to be based upon things such as whether or not important pages link to that page (an example of a query-independent ranking algorithm would be PageRank).

Query-independent ranking algorithms assign a quality score to each document on the web, and can be run ahead of time. Query-dependent ranking algorithms depend upon the query used, and have to be run when a user submits a query.

Content, Usage, and Link Based Ranking Algorithms

It’s also possible to classify ranking algorithms as content-based, usage-based, and link-based.

Content-based ranking algorithms - use the words in a document to rank the document among other documents. For instance, a higher score might be assigned to a document that contains the query terms at the beginning of a document, in a prominent font, or in a certain kind of HTML element.

Usage-based ranking algorithms - may assign a score based on estimages of how often documents are viewed from looking at web proxy logs or looking at click-throughs on search engine results pages.

Link-based ranking algorithms - look at the hyperlinks between web pages to rank those pages, assigning a score to pages based upon links pointing to pages. endorsement of the page.

PageRank - an example of a query-independent link-based ranking algorithm.

The PageRank formula is often explained as follows. Consider a web surfer who is performing a random walk on the web. At every step along the walk, the surfer moves from one web page to another, using the following algorithm.

With some probability d, the surfer selects a web page uniformly at random and jumps to it; otherwise, the surfer selects one of the outgoing hyperlinks in the current page uniformly at random and follows it. Because of this metaphor, the number d is sometimes called the “jump probability,” namely the probability that the surfer will jump to a completely random page.

If the web surfer jumps with probability d and there are |V| web pages, the probability of jumping to a particular page is d/|V|. Since any page can be reached by jumping, every page is guaranteed a score of at least d/|V|. The PageRank of a particular web page is then the fraction of time that the random surfer will spend at that page.

But what if that surfer started favoring pages that were linked to by blogs a little more?

Splitting PageRank

One of the problems behind using PageRank is that some commercial web sites try to inflate PageRank by creating links that point to a page solely for the purpose of endorsing that page, artificially increasing the value of the page.

This patent filing describes in some detail how a portion of PageRank from a page might be split (or distributed) equally amongst the links found on the pages of a site, and how the distribution of PageRank could be slightly altered to favor (or show a bias towards) pages that are linked to by blogs.

If blogs are, as the authors note in the patent, “still mostly human authored, and generally represent sincere endorsements of their authors,” then this bias might help counteract the artifical inflation of PageRank scores by people who would create links pointing to pages solely for the purpose of artifically increasing the PageRank of pages.

The patent filing is:

Ranking Method using Hyperlinks in Blogs
Inventors: Steve Chien and Dennis Fetterly
Assigned to Microsoft
US Patent Application 20080243812
Published October 2, 2008
Filed March 30, 2007

Abstract

A method for static ranking of web documents is disclosed. Search engines are typically configured such that search results having a higher PageRank.RTM. score are listed first. A modified scoring technique is provided whereby the score includes a reset vector that is biased toward web pages linked to blogs. This requires identifying web pages as either blogs or non-blogs.

Identifying Blogs

Some of the kinds of things that a search engine crawling program might look at when deciding whether a page is from a blog might include:

1. Whether a page is hosted in a known blog hosting DNS domain such as blogspot or wordpress.com
2. What features are containted in the non-HTML markup words and phrases contained in the page
3. What the targets of outgoing links might be in the page, and
4. Whether the string “blog” occurs in the URL

Experimenting with a Bias Towards Pages Linked to by Blogs

The authors of this patent performed experiments where they downloaded over 472 million pages, and found links to an additional 6 Billion pages within those pages.

They reranked the PageRank of these pages using a bias towards pages that they identified were linked to by blogs, with a preference towards using blog pages that had higher PageRanks, which they tell us tend to be “frequently updated, more informational rather than personal, and free of spam.”

They also tell us that some other characteristics of blogs may prove useful in refining this technique, such as looking at the number of subscribers to a particular blog, and associating a higher endorsement value to blogs with greater numbers of subscribers.

Conclusion

Can sending more PageRank to pages that are linked to by blogs something that will increase the relevance and importance of pages that show up in search results? Are links to pages from blogs still actual endorsements from the authors of those blogs?

Do search engines love blogs?