Searchlight

Several weeks ago at SMX West I had the pleasure of meeting and having lunch with Brian White from Google. White works on Matt Cutts' Web spam team, tirelessly working to make Google's search results the best they can be, ensuring the best user experience. Quite a hefty task indeed.

You'd think that someone who spends his days fighting the never-ending battle that is Web spam might be a bit negative or jaded. If that is the case, he does an amazing job hiding it. Instead, he was upbeat and you could feel the excitement in his voice as he spoke. Here's a guy who loves what he's doing and truly wants to not only improve the searchers' experience on Google, but wants to make the Web a better place. You can't help but like a guy who's fighting the good fight.

... Read more

Google's Webmaster Central has become a very important resource for anyone who has a Web site, works on a Web site, or, like SEO practitioners, helps others with their Web sites.

Google continues to roll out more features and better functionality to existing features, and now they just did a little bit of both with the addition of their Generate robots.txt function.

Google had previously added a robots.txt analyzer, which at this point is still the more useful of the two tools. For those who aren't aware, the robots exclusion protocol helps with instructing search engines how to interact with a Web site. There are a number of directives available, but the main purpose of the robots.txt file is to instruct the search engines about content that a site owner doesn't want the robots to crawl.

Why in the world would you not want search engines to crawl any of your content? You may have content that, for whatever reason, you don't want others to find through search results. Note, however, that this is not the same as secure information that requires authentication through a log-in.

Your site may have its own search function that creates "search results" for your site. Search engines generally do not want to include search results within search results, so this content may not be returned for searches on the engines anyway, so you might want to focus the crawlers elsewhere for greater crawler efficiency.

Or you may have duplicate content issues that you could use robots.txt to filter out. This is especially common with a content management system (CMS) that creates a separate printer-friendly page.

Regardless of your specific needs, having a robots.txt file can be important to a site. Rarely is there a site that can't benefit from disallowing at least some content. Even if you have nothing to disallow, you may want to take advantage of the auto-discovery feature for your XML sitemap. Finally, depending on your server log system or analytics package, not having a robots.txt file can be problematic if it inflates your "404 File Not Found" error reporting, which can happen because search engine spiders will request the robots.txt file automatically when they come to your site.

Right now, the robots.txt generator is rather basic and I hope that Google will add more features to it going forward. Currently, site owners have to paste in URLs and URL patterns to build the file. It would be great if it would provide a list of URLs or patterns extracted from a site to help automate the procedure for anyone not familiar with the protocol.

There is more information about the protocol, though a bit more on the technical side, at the robotstxt.org site and you can find more engine specific information on crawling and robots.txt from Google, Yahoo, MSN, and Ask.com.

One important tip is that the following directive tells all spiders they are allowed to go anywhere:

User-agent: *
Disallow:

And, more importantly, the following directive, which I sometimes see when I think people really wanted the above:

User-agent: *
Disallow: /

The latter tells the spiders to stay out of the entire site--clearly two very different results, so be sure you understand which does what.

Google's new teleportation, its search-within-search function, is getting mixed responses, at least from some site owners, who may be remembering occasions when teleportation in the Star Trek transporter went wrong. Earlier in the month, Google introduced the teleportation functionality as a way to better help searchers find information within a site by providing a search box below the snippet of the top listing, which performs a "site:" search on the domain of that listing using the additional search terms the searcher added in.

The "site:" advanced query is quite familiar to those within the search industry, but much less so to the average searcher. So bringing this functionality front and center for the searcher should be a well-received addition.

When I first saw this, I thought it was interesting--once I was able to get it to show up. It doesn't come up for every site, mainly big-name sites, nor does it come up for every search. One that it did come up for was searching for Amazon.com. After playing around with the teleportation search, I also began wondering how these big-name retailers would react and thought that some might not care for this new functionality. Why would they object?

Let me show you--except I can't use Amazon to do it anymore. According to the New York Times, Amazon is one such retailer that has already objected and asked Google to turn off this functionality for its site. It seems that most of the talk so far, like that happening at Search Engine Land (here and here), has been more about acknowledgment than anything else, but Rishi Lakhani's post at SEO Smarty shows that others have had similar thoughts as I.

Now, before we go much further, understand that I'm not suggesting ulterior motives here on Google's part or that this is even a good or a bad thing. For regular users, I think this will be well received, and Google pays a lot of attention to delivering the best user experience it can--but that isn't to say that there isn't going to be a potential upside for the PPC program as well.

So let's take a look at some examples of how this may impact results and get a feel for why some site owners may be less than thrilled with this functionality. Let's use national retailer Target as an example while we still can since its site is powered by Amazon. We'll try this on searches for plasma TVs.

Below we see the results that someone might see doing a search in Google just for "plasma tv" which includes eight paid search ads.

Below we see the results that someone might see doing a search in Google for "target plasma tv." Notice how there are no paid search results showing up, and not surprising, Target shows up in the top organic listing.

Then let's see what happens if someone searches just on "target." No surprise that Target.com shows up No. 1 again in organic results and still no paid search ads. What is different is the appearance of the teleportation, search-within-search, box showing up below the sitelinks in the Target result, labeled as "Search target.com."

Then when we do a teleportation search for "plasma tv," we get the following search results. Notice that this creates the advanced search query "plasma tv site:target.com." Now the searcher gets Target.com specific search results in the organic area, hopefully relevant to the search, but also eight paid listings that Target is now competing with.

This isn't all as cut-and-dried as this example may seem. The appearance of ads can vary widely from none to many. But for now it does serve as an example of at least one scenario that site owners need to be aware of.

So what does teleportation mean for the various players? Well hopefully, for the searchers, it does get them to what they are looking for faster and easier, but this can really vary as well and may or may not be more helpful than getting directly to the site.

For Google, it means that searchers will have performed at least one more search on Google, instead of clicking through to Target.com immediately. And it may mean that it has gained an opportunity to serve up more targeted (no pun intended) search ads that otherwise may not have been served up (as we can see from the other Target focused searches which yielded no ads). Even more subtle here is the fact that many advertisers may not have bid against a big brand name to begin with. Currently, advertisers can use a trademarked brand as a trigger word as long as they don't use it in the ad itself. As much of the legislation in this area continues to be formed and reformed, who knows whether this will always be the case--but it would seem that teleportation search may provide an additional means to serve up ads around another brand without even needing the advertiser to use that brand as a trigger word.

But how might Target feel about this? Well, if it does help get searchers to their destination, then it might be happy with this. But it also might mean that its natural results are competing against paid-listings that it may not have been competing against under the other Target related searches. It also means that it may not be able to cull additional search information from its own site-search. While the quality of on-site search may vary from excellent to completely worthless, some sites invest heavily in their on-site search to not only deliver good results, but also to serve as insight into what their visitors are looking for. Being able to follow the search path, which they may be losing because of teleportation, may help improve the site experience.

Needless to say, Target might prefer to get people directly to its site and have people search on-site, which at least in this example allows it to serve up a richer experience.

Good, bad or otherwise, what this means to site owners is that SEO may be more important than ever. Now, getting to the top listing may not be enough. Defending your brand may not be enough. Securing multiple listings through blended search may not be enough. What happens to the site that has excellent search, but terrible indexation in Google? Now more than ever, site owners need to focus on creating the most search-friendly site as they can to make sure that Google and other search engines can spider and index the site as completely as possible. For some sites, this is a huge challenge, trying to overcome legacy CMS and e-commerce systems. Fortunately, there are solutions like Netconcepts' own GravityStream proxy optimization that can help many sites overcome these obstacles, but GravityStream isn't for everyone.

One thing this clearly means is that site optimization is more important than ever. Optimization will help to make sure that the teleportation results for your site are highly relevant and speak to the searcher, hopefully gaining the click-through from the searcher. If you are like Target and experience millions of searches a year just on your brand name, then you don't want to leave your optimization to chance when it comes to teleportation.

The blogosphere and Twitter have been abuzz with talk about this article by Shari Thurow, published Thursday on Search Engine Land. The article warns of supposed dangers against the SEO tactic of "PageRank sculpting." Readers are coming away feeling reticent to employ the tactic, fearing retribution from the engines in the form of penalties. The article paints PageRank sculpting as poor usability and black hat. I can't be any more adamant about this: neither is the case.

No disrespect intended to the article's author, but this article is classic FUD (Fear, Uncertainty, and Doubt). "Nobody ever lost their job by choosing IBM," the classic FUD saying goes. "Nobody ever lost their ranking by refusing to implement PageRank sculpting" is the FUD peddled in this article.

PageRank sculpting is a powerful tactic that is worthy of inclusion in any advanced SEO practitioner's toolkit. I have written and blogged and spoken in favor of the tactic. I stand by the tactic, as do many other top-notch SEOs, and most importantly, so does Matt Cutts.

Matt Cutts, the head of Google's Webspam team, has publicly condoned the use of PageRank sculpting on repeated occasions. For example, in this SEOMoz interview:

Does Google recommend the use of nofollow internally as a positive method for controlling the flow of internal link love?

A) Yes--Webmasters can feel free to use nofollow internally to help tell Googlebot which pages they want to receive link juice from other pages

(Matt's precise words were: The nofollow attribute is just a mechanism that gives Webmasters the ability to modify PageRank flow at link-level granularity. Plenty of other mechanisms would also work (e.g., a link through a page that is robot.txt'ed out), but nofollow on individual links is simpler for some folks to use. There's no stigma to using nofollow, even on your own internal links; for Google, nofollow'ed links are dropped out of our link graph; we don't even use such links for discovery. By the way, the nofollow meta tag does that same thing, but at a page level.)

Google has even used the technique on its own properties.

So my question to Ms. Thurow is: "Have you ever conducted any testing of the PageRank sculpting technique?" We at Netconcepts have, and it works.

According to our tests, there are plenty of occasions where it can be a valuable tool, if used wisely. For example, if you have an e-commerce site and the category pages contain three links to every single product page--the product name as a text link, the product image thumbnail as an image link, and the words "View Product" as a text link--you could nofollow the image and "View Product" links and funnel more PageRank through the much more contextually relevant product name-based text links.

If SEO is going to be respected as an experimental science instead of black magic, it needs to be implemented with an experimental approach and all tactics tested for effectiveness (within the bounds of what is acceptable according to the engines). With SEO, you don't just "set it and forget" using the purported "best practices" as defined by the SEO bloggers and speakers (and sure, feel free to include me in that set).

So if Ms. Thurow wrote this piece without any testing, it's just unsubstantiated opinion--and I wholeheartedly disagree with it. :)

And I'm not the only one either. Just take a look at the other prominent SEOs who have countered via blog posts (like "Why There's Nothing Wrong With Sculpting Your Pagerank" and Sphinn comments and desphinns (click on the "Who Desphunn This" tab to see the "desphinns").

As much as most high-quality SEOers (those who practice search engine optimization "honorably") like to think of their pursuit as pure and intellectual, the fact of the matter is that search engine optimization is a form of marketing, no matter how you approach it.

Perhaps short of nonprofit organizations and the occasional Internet artist, people interested in getting their Web pages in top search results are interested in making money from the traffic that will come from those search engine links.

A recent Hitwise article revealed some interesting demographic numbers that the company has measured through its sources. In comparing American Google and Yahoo search users, Hitwise found that Google has a higher percentage of users in older age brackets than Yahoo.

This negates some of existing beliefs in the market about the composition of those two user groups; a commonly held belief has been that Yahoo was for the aged, Google for the youth. Maybe not:

• In the category of users 55 and older, Google has 1.5 percent more of the market share.
• In the category of 45- to 54-year-old users, Google has 2 percent more of the market share.
• In the category of 35- to 44-year-old users, the market share was more or less equal.
• In the category of 25- to 34-year-old users, Yahoo has almost 3 percent more of the market share.
• In the category of 18-to 24-year-old users, Yahoo has 1.5 percent more of the market share.

It makes for a very nice, even curve! One implication that can be drawn from this is that, since older Americans tend to have more money than their younger counterparts, and Google users in general also tend to be more affluent, targeting the Google search engine over other engines makes much more marketing sense. Of course, this is in addition to the fact that Google's market share in the search world hovers somewhere around 60 percent, while Yahoo's hovers around 20 percent.

Google's search dominance isn't any sort of secret, but if these demographic numbers are accurate, the point is further magnified. Other search engines shouldn't be ignored by those investing in search optimization, but it's no wonder that SEOers are so quick to focus on Google; it offers the best bang for the SEO buck.

Are you looking for that edge online? Something that your competitors don't have? Forget wasting all your energy on a great design and developing superior content--it's not what you know, but who you work with.

Sooner or later, most of us in the industry get an e-mail, either passed on by someone we know, through our own e-mail, or possibly through one of our own sites that offers to help us achieve success online. Most of these are fairly nondescript and rather generic.

Then the other day, I had one passed on to me that was more than just a couple of lines of loose promises. This one proclaimed: "I specialize in getting sites listed at the top of Google in organic listings."

And it went on, even specifying that, "After the first month, it is only $300 month." And apparently, "That's all there is to it."

OK, so this one actually perked my curiosity. I couldn't help but go check out the site.

I looked around for a bit and was almost ready to just mosey on by when, midway down on one page, it caught my eye, the claim of all claims, the one that I had heard rumors about, but until this day, had never actually come across in the wild: special relationships.

Could my eyes have deceived me? No, there it was, in the fine HTML print: "We have special relationships with the search engines."

OK, by now you must be on edge as to the power that these special relationships could possibly muster. And there, like a beacon in the night, "...indexed within 2 weeks..."

Wow. So 1990s.

For those of us in the industry, this brings both a chuckle and a sad realization that there are still people out there who will play into the desires and fears of Web site owners, offering some magical potion that will be their ticket to success.

OK, this one was particularly sad and amusing because of the two-week window. I mean, talk about working some miracles! Let's face it--it's 2008--in this day, you would almost have to work to not have a site (or even a fair amount of a site) be indexed within two weeks.

Sadly, this is one of those things that gives search engine optimization a bad name. Just because someone claims to perform SEO, doesn't mean that they can, nor should that then be a negative reflection on the industry.

Certainly there are those who, after working so long in the industry, have established connections with those at the search engines, as peers do in nearly every industry. This does not equate to special powers to get the engines to overlook offenses or to serve up rankings based on these relationships. And if it did, I have to imagine that it would cost a bit more than $300 a month.

For anyone who doubts this, Google itself has stated its view on the matter.

As for the provider of the services that I found so amusing, well, with several pages with identical title tags, I have to question their expertise in SEO. As for their relationship with Google, it appears that only their home page has any PageRank, and seeing that it is a PR1, they might want to renegotiate their special relationship as they are clearly getting the short end of the stick.

The term and concept of Microhoo can be dated back a couple of years now, but it looks to be much more of a reality today than it ever was. Thursday night, Microsoft sent a letter to Yahoo offering $31 per share in cash and stocks. This is more than 50 percent over the worth of the company relative to its Nasdaq trading price this week. The bid by Microsoft to buy Yahoo (it adds up to $44.6 billion) is surely a way for both companies to best do battle against Google, and such a move is ripe with many serious ramifications both inside and outside the world of SEO.

Speculation is rife on the Internet about the possibilities of the deal coming to fruition and the ramifications if it does. Stock prices are swelling and wavering, and investors are scrambling to properly position themselves. So, what's the most important issue in my view? Well, I just don't think they're going to call it Microhoo. I think there are better options.

Yicrosoft, for example, has a better ring to it and seems less questioning than "micro-who?" The name Mihoo looses the identity of both companies, and Yacrosoft sounds downright regurgitated. Mahoo, I am told, has predefined connotations (Google it), and Yicrohoo, if it doesn't have predefined connotations, really should. And what about Microsoft's Live search portal? It would seem to be the logical match to Yahoo. Could they go with Livehoo? It's certainly better than Yalive, which looks slick but could easily be mispronounced as "olive."

The winner in my book is Yasoft. It just sounds cool. Oh sure, there are a couple of small entities out there that may lay claim to the name, but no one that couldn't be bought out for five figures. The "Ya" says "yes" while the "soft" says "safe," as if it could just as easily be a high-end brand of bathroom tissue...which is exactly what Yahoo stockholders are going to need to dab at their tears if this deal doesn't go through. Come to think of it, they should grab the name either way.

Like many companies, Netconcepts recently celebrated the holidays with a little staff get together and mystery gift exchange. There were the usual, but always popular gifts of coffee, tea, chocolates, and wine, as well as a very popular Atari console ... and as we did the "you can open one or steal one" gift rule, it certainly was the most fought over gift.

Unfortunately due to timing, Stephan wasn't able to make it as he was out visiting our friends in Seattle and participating in a great SEOmoz Whiteboard Friday otherwise he might have been torn between the Atari and the unique gift that follows and shows that even search geeks have a sense of humor (twisted as it may be).

Every search marketer has their bag of tricks and tools, but why settle for a bag when you can have a kit ... or more accurately, a box with built in carrying handle? And at $15 or less, the cost limit for our gift exchange, within range of even the budding search marketer.

What comes in a Search Marketer's Kit? I thought you'd never ask. Luckily, each item in the kit comes with its own identification tag.

... Read more

Over the past year, there has been a lot of talk about the best way to handle Flash on your site. I previously covered quite a few aspects about this heavily-debated topic in Flash Alternatives Blessed by Google and in Progressive Enhancement is Good for SEO. In my previous interview with Maile Ohye, Google's support engineer I had asked her about Google's view on Flash. Maile confirmed that Google looks at the content within "noscript" tags, but she advised to be careful to mirror accurately the Flash-based content you include within the noscript tags or it will look like cloaking to Googlebot.

In my recent interview with Matt Cutts, Google engineer and head of their webspam team, I questioned as to the status of Google reading textual content within Flash .swf files. Here's what Matt had to say:

"It is a good question. I think that we do a pretty good job of reading textual content. Now, stuff within Flash is binary and you can define it in terms of characters and strokes - so you can have things that look like normal text - but that are completely weird and are not really normal text. So it can be difficult to pull the text out a Flash file. I think we do pretty well. It used to be the case that we had our own, home-brew code to pull the text out of Flash, but I think that we have moved to the Search Engine SDK tool that Adobe/Macromedia offers. So, my hunch is that most of the search engines will standardize on using that Search Engine SDK tool to pull out the text. The easiest way to know whether you have textual content that can be read in a Flash file, is that you could always use that tool yourself and verify as well."

Not only did Matt suggest that Flash users take advantage of the search engine SDK tool, he also confirmed that Google is hoping to standardize it and work with Adobe to continue updating it.

So there you have it. If you use Flash on your website, you owe it to yourself to use the Search Engine SDK tool to gain insight into how Google "sees" your Flash content. If the Search Engine SDK tool is used by Google, why shouldn't you?

For more great advice courtesy of Matt Cutts, I invite you to either read the transcript of my interview with Google's Matt Cutts at Pubcon or you can listen to the Matt Cutts at Pubcon interview podcast (31 minutes, 3.8 MB).