Google's Webmaster Central has become a very important resource for anyone who has a Web site, works on a Web site, or, like SEO practitioners, helps others with their Web sites.
Google continues to roll out more features and better functionality to existing features, and now they just did a little bit of both with the addition of their Generate robots.txt function.
Google had previously added a robots.txt analyzer, which at this point is still the more useful of the two tools. For those who aren't aware, the robots exclusion protocol helps with instructing search engines how to interact with a Web site. There are a number of directives available, but the main purpose of the robots.txt file is to instruct the search engines about content that a site owner doesn't want the robots to crawl.
Why in the world would you not want search engines to crawl any of your content? You may have content that, for whatever reason, you don't want others to find through search results. Note, however, that this is not the same as secure information that requires authentication through a log-in.
Your site may have its own search function that creates "search results" for your site. Search engines generally do not want to include search results within search results, so this content may not be returned for searches on the engines anyway, so you might want to focus the crawlers elsewhere for greater crawler efficiency.
Or you may have duplicate content issues that you could use robots.txt to filter out. This is especially common with a content management system (CMS) that creates a separate printer-friendly page.
Regardless of your specific needs, having a robots.txt file can be important to a site. Rarely is there a site that can't benefit from disallowing at least some content. Even if you have nothing to disallow, you may want to take advantage of the auto-discovery feature for your XML sitemap. Finally, depending on your server log system or analytics package, not having a robots.txt file can be problematic if it inflates your "404 File Not Found" error reporting, which can happen because search engine spiders will request the robots.txt file automatically when they come to your site.
Right now, the robots.txt generator is rather basic and I hope that Google will add more features to it going forward. Currently, site owners have to paste in URLs and URL patterns to build the file. It would be great if it would provide a list of URLs or patterns extracted from a site to help automate the procedure for anyone not familiar with the protocol.
There is more information about the protocol, though a bit more on the technical side, at the robotstxt.org site and you can find more engine specific information on crawling and robots.txt from Google, Yahoo, MSN, and Ask.com.
One important tip is that the following directive tells all spiders they are allowed to go anywhere:
User-agent: *
Disallow:
And, more importantly, the following directive, which I sometimes see when I think people really wanted the above:
User-agent: *
Disallow: /
The latter tells the spiders to stay out of the entire site--clearly two very different results, so be sure you understand which does what.
Like many companies, Netconcepts recently celebrated the holidays with a little staff get together and mystery gift exchange. There were the usual, but always popular gifts of coffee, tea, chocolates, and wine, as well as a very popular Atari console ... and as we did the "you can open one or steal one" gift rule, it certainly was the most fought over gift.
Unfortunately due to timing, Stephan wasn't able to make it as he was out visiting our friends in Seattle and participating in a great SEOmoz Whiteboard Friday otherwise he might have been torn between the Atari and the unique gift that follows and shows that even search geeks have a sense of humor (twisted as it may be).
Every search marketer has their bag of tricks and tools, but why settle for a bag when you can have a kit ... or more accurately, a box with built in carrying handle? And at $15 or less, the cost limit for our gift exchange, within range of even the budding search marketer.
What comes in a Search Marketer's Kit? I thought you'd never ask. Luckily, each item in the kit comes with its own identification tag.
Official Super Deluxe Search Marketer's Kit
MSN, please say it isn't so.
As earlier reported, the MSN Live Search team had reinstated the link queries, though with a twist... now requiring a "+" in front of either the link: or linkdomain: query operators. But now it appears that the query is no longer working again.
But right now, it appears there is no love to be found at MSN for link queries:
Live Search link query results.
Nor linkdomain queries:
Live Search linkdomain query results.
Hopefully, this is just a glitch, but if not, then hopefully the Live Search team will make up their minds once and for all and either reinstate the special queries or kill them altogether.
The problem is that search marketers and Web site owners alike use these queries to measure and benchmark their visibility on the Web, as well as the success of their link building. These queries were originally decommissioned due to extreme load from automated queries.
Now that MSN has launched its new Webmaster Center, there is a thought that it will build this into its interface. While this may help alleviate the automated queries, it will probably be limited to the site that is validated within Webmaster Center. Unfortunately, this means that there won't be any way to measure against similar or competitor sites.
Games always seem to capture a lot of attention this time of year, in part because of kids and toys for Christmas or adults and getting together for the holidays. It would seem that search marketers enjoy games, too. While this often revolves around contests or challenges to get a site to rank No. 1 in the SERPs, at PubCon this year, SEOmoz even got some search marketers to unplug long enough to play an SEO version of Werewolf.
With only a week before Christmas, I couldn't help but wonder what games might be dancing around in search marketer's minds. While these classic renditions probably haven't made it to the shelves yet, maybe this will plant the seed for next season.
When it comes to games, it's hard to beat the classics, of course...so here are 10 searchized-games for search marketers:
Connect Four Reciprocal Links — (based on the classic Connect Four game), This game is based on the outdated concept of reciprocal linking. The winner is the one who can create the biggest reciprocal linking circle. Of course, this game is purely for fun.
Battleship Text Link Buys — (based on the classic game of Battleship) This text-link-buy smack-down game puts each opponent in the role of seeking out the other's text link buys and reporting them to Google. The one who sinks the other's PageRank first is the winner.
Keyword Scrabble — (based on the classic Scrabble game) Keywords are a foundation of the search world. Players take turns building on keyword phrases, earning points based on the smallest keyword phrase that achieves the highest rank in Google, Yahoo, or MSN.
Twister Search Career — (based on the zany classic Twister) Like any industry, the search industry often features moving from one company to another. In this game, instead of colored circles, the playing mat features company logos. Just like the Web industry, each player must reach out and touch a number of different company logos, constantly moving from one company to another without falling down.
SERPs Memory — (based on the age-old game of Memory) Test your visual and memory skills by viewing a listing of top-10 SERPs from Google, Yahoo, and MSN before they disappear, and then matching up the listings across the SERPs. The one who gets the most matching results across the engines wins.
Spammer Mouse Trap — (based on the fun game of Mouse Trap) Ever wonder what a day in the life of Matt Cutts would be like? Find out with this exciting game where you get to build a trap to catch search spammers.
Search Operation — (based on the nerve-racking game of Operation) Search marketers have to have nerves of steal when every move they make can result in a site plummeting in the SERPs. Test your skill by removing bad bits and pieces from a Web page; but be careful--alert the spiders and you may get a shock.
Stratego Search Position — (based on the battle strategy game of Stratego) Search marketing, like most forms of marketing, is often a matter of strategy. Position and move your Web pages to overthrow your competition, gaining stronger search positioning, winning more site visitors, and achieving the highest ROI.
The Game of Life Search — (based on The Game of Life) One of my favorites growing up, this version has you go through the life of a search marketer, mastering your trade, building your skills, moving from company to company and site to site to achieve your search life goals.
And quite possible the grandest game of all...
Search Monopoly — (based on the classic game of Monopoly) This game can provide hours of fun and is one of the most involved of search games. Chose your playing piece, from exciting favorites like white hat, black hat, Googlebot, Slurp, or MSNbot, and work your way around the game board, accumulating Web properties. Beware--landing on a Made-For-AdSense site or SPLOG can cost you, as can landing in the Google sandbox. Pull a Chance card and find that "Matt Cutts caught you buying Text Links. Lose two PageRank points and wait a turn to file for re-inclusion." But it isn't all bad...pull a card from the Community Chest and you may find that you "Win a one year SEOmoz Premium Membership," or "A free pass to SMX." The winner, of course, is the one who accumulates the most Web properties and finishes with the largest amount in VC funding.
As far as I know, none of these games exist yet, but a lot can happen between now and next year.
*All of these game concepts are based on related games that are the property of Hasbro. Until these games exist, you'll just have to buy the classics and pretend or modify to make your own.
MSN's Live Search team announced back on August 22 that they would be launching a set of tools for Webmasters. At that time, this was strictly a private, by invitation beta. Even then, Webmasters and SEO practitioners alike were excited and hopeful as one of the much awaited features was the ability to pull up backlink information. MSN had previously turned off the special "link" and "linkdomain" query operators that provided a count of links pointing to a page or entire site, respectively.
The Live Search team is really trying to give everyone something to be thankful for. Karen Blakeman reported in October that Microsoft had restored the link and linkdomain queries, though with the slight modification of leading them off with a "+" sign, like:
+linkdomain:www.cnet.com
With apparently no official announcement from Microsoft, news of this seems to have just now picked up notice after Barry Schwartz reported it on Search Engine Land.
And now the Live Search team has ... Read more
Inbound links control PageRank, which in turn has a deep impact on Google rankings. Other search engines work similarly, rating the quality and number of links a given website receives as if they were votes for that site. So, how does one get a good take on inbound links for a site? There are many advanced linking softwares out there, like Advanced Link Manager, that can perform complex and deep analyses. But many webmasters or site owners may not need that much power, or would prefer to invest software dollars elsewhere. In most cases, simple link queries can offer a good overview of a site's inbound linking status.
What exactly is a link query? It is essentially a command line, followed by a web address, typed into a search engine. Most commonly, it looks like this:
link:www.yoursitehere.com
This line works for both Google and Yahoo. Each will return results listing websites that link to that address, but they will also list total numbers of inbound links as each engine counts them (note that currently Yahoo is a bit quirky in that it returns different numbers depending upon whether or not you?re logged into Yahoo). These numbers are rarely going to match between search engines as they use different methods to crawl and define inbound links, but taken together they draw a good picture of the linking popularity of a given site.
At the beginning of the year, MSN Live Search stopped offering the link query command. Though there was a rather cryptic work-around, it s no longer needed as MSN recently re-introduced the option with a slight variation: a plus sign (+) must be added to the beginning of the command:
+link:www.yoursitehere.com
Check it out for yourself here. Now anyone can get free reports on inbound links from the big three search engines. Use the information to gauge and build a link building strategy. Check out Stephan Spencer's earlier post on link building for more information.
Web site owners and SEOs alike often feel at odds with the search engines, but times are changing. This was often the case in the past when the engines made updates and changes to their algorithms that seemed to send Web sites into a SERP tailspin, leaving everyone scrambling to regain their precious page-one positions. The engines were also a lot less forthcoming with information and guidance, perhaps taking the view that giving this information gave too much power to the spammers and phishers.
While this view was understandable on the surface, it didn't float all that well in reality. In the real world, those who are out to game and manipulate the engines may have as many or more resources to keep up with the engines than "the rest of us." So over the last few years, the search engines have continued to be more open with what they consider important as well as what abuses may get sites into trouble, perhaps realizing that there are also a lot of sites that may not have been purposely trying to mislead the engines, but were just victims of bad advice. And of course the algorithms have become far more powerful and fine-tuned than they once were.
By openly helping everyone, they are really just helping to raise the bar of quality for all sites, and maybe even making it even harder for bad sites to game the engines. Along with providing more detailed information and answering more and more questions publicly, the greatest advancement they have made has been in creating tools to actually give site owners (who have validated their sites) more information about their sites than they've ever experienced before.
Webmaster Central and Webmaster Tools
Google introduced Webmaster Central, which continues to add more and more features for site owners. Not surprising, Webmaster Central is leading the pack in delivering great information and tools to Webmasters. At the center, literally, is Webmaster Tools, which provides site owners with fairly detailed information on site crawling, queries, considerably more backlink information than can be queried outside of Tools, and much more. The query information in particular provides an unprecedented view of the search phrases that a site is showing up for, including those terms that aren't actually delivering traffic to the site.
Site Explorer
Yahoo's Site Explorer is still lacking in a few areas compared with Google's Webmaster Tools, but they almost make up for that with their powerful link information. Through simple drop-down menus, it is quick and easy to tailor results based on links to a specific page or the entire site, to include all links or to exclude links from the site and focus on external links only. Yahoo added a new feature that may give even more control to site owners. The Dynamic URLs tab gives site owners the ability to inform Yahoo of their site's dynamic URL patterns to help eliminate duplicate content issues, better handling of multiparameter URLs, addressing session IDs, and even presenting "cleaner" URLs in search results. Ideally, it would be best to address as many of these issues through rewrites and the robots.txt file on the server, but this is a great addition as a backup or for when that isn't possible.
Webmaster Portal
Trying not to be outdone, the Live Search team at MSN recently announced their entry into the mix with the Webmaster Portal, currently in beta and by request only. Little detailed information is available, but their tool also is claiming to help troubleshoot crawl issues, assist with sitemaps, and provide site statistics, including a replacement to the "link:" operator query that was decommissioned back in March. The portal is slated to be fully available to the public by late fall, but it may be worthwhile to request an invite to participate in the beta now.
The advancement in all of these tools is great news to Webmasters and SEOs alike. They continue to put more information and control into our hands. Not wanting to be outdone by the others, hopefully each of the engines will add each other's additions to their own toolsets. As each of these is free, there is no reason for site owners not to take a few minutes to validate their sites and start spending a little time each month putting these tools to work for them. This is one invitation from the spiders you don't want to turn down.
Search (or more precisely the search results) is undergoing some of the greatest changes we've ever seen with the influence of local search, mobile search, personalization, and universal search. Search marketers and site owners alike are bouncing between wild speculation and frantic contemplation of how search will change and what impact those changes will have.
Who better to turn to on the topic of search results than Gord Hotchkiss and his team at Enquiro. Now famous for their eye tracking studies revealing the infamous "Golden Triangle" and F-shaped patterns of user scanning of SERPs, Enquiro takes a bold step to try to track user interaction with Google universal and personalized results, and an even bolder leap of predicting the result pages we may see in 2010.
Enquiro just made available a must-read whitepaper, Search Engine Results: 2010, for anyone dealing with search in any way. Along with the traditional eye tracking studies done for this report, they assembled a dream-team of industry experts to discuss thoughts and ideas on the future of search results.
Merely a sampling of highlights, what follows are some of the great nuggets to explore from this report, starting off with a quote that truly encapsulates the report's essence:
There is, without a doubt, great passion about the future of search and we strongly believe that the next three years will represent the most exciting era yet in the short history of web search.
Intent
Usually when we talk about websites, search engines, and intent, we refer to the intent on the part of the SEO or web designer... white-hat versus black-hat, delivering relevant content versus gaming the engines. This however is a whole new intent... the intent of the user.
Much of the future for search will come down to the ability for search engines to identify searcher intent. A great illustration of this comes from Justin Osmer, product manager for Microsoft Live search,
"An area that we're focusing on over here at Live search is thinking more about the mode in which people are in when they're using search."
Are searchers looking to buy, to research, or just to be entertained? Each of these modes may dictate very different results for the same search.
Google personalization and universal are trying to tap into that intent as well, based on previous search history as well as by serving up a mix of content types, including maps, blog posts, videos, and the traditional textual results. Danny Sullivan, editor-in-chief of Search Engine Land and chief content officer for Third Door Media, added to the discussion on the importance of relevancy in how the information is presented, such as providing maps for appropriate location searches or the ability to list blog results based on recency as well as relevancy. It's not just about presenting the results, but presenting them in the format that matches the searcher's intent.
Interaction
One area that will see great exploration will be in how users interact with search engines. As RSS adoption continues to grow and the sheer amount of information in its many formats expands, users will continue to look to search engines to be more than just a search destination, but a source of information aggregation; the search engine as portal, pulling and updating news and other content based on the user's preferences.
A particularly interesting comment was made by Marissa Mayer, Google's VP of Search User Experience and Interface Design, that furthers the sense that search engines will continue their evolution beyond search:
"I think that people will be annotating search results pages and web pages a lot. They're going to be rating them, they're going to be reviewing them. They're going to be marking them up..."
The separate mention of "web pages" may be another reason why the development of a Google Browser would be so important. Tapping into the web browser might lead to that ability to annotate and rate those pages and further help Google identify what content interests the user.
While Chris Sherman, executive editor of Search Engine Land, feels that advancement within search personalization is still fairly limited, he offered up an interesting interactive approach that the search engines might pursue,
"...submitting a page of content and analyzing the full text of that page and then tying that in conjunction with our past behavior..."
as a way to allow users to interact with search engines and help bring about better results.
New Search Patterns
Google personalized and universal results are still slowly trickling into everyday searches. Enquiro created sample pages representing a futurized version based on where these new results may be headed to see the potential impact on user interaction through eye tracking.
While this was very limited and exploratory research, it may hint at the important impact these changes may have. The eye tracking revealed how the introduction of images, as long as they are relevant, may begin to alter the search pattern into more of an "E" pattern than an "F" pattern, with the image forming the middle connection. Searchers may then expand their attention to the title and description accompanying that image, and then may move above or below the invisible line drawn by that image, depending on which cluster of results appears more relevant based on personalization.
To wrap up, search marketing is making a natural shift, somewhat downgrading those things that have been synonymous with SEO, like rankings, keywords, and optimization, to a much greater focus on the users, tying into their intent and interests at the time of search. Personalization will make site stickiness ever more important. Securing a position in users history, becoming an authoritative go-to source for information, will be more critical than ever. Winning in the SERPs will require much more than just position.
This new whitepaper from Enquiro is filled with much more detailed and interesting discussion than could ever be covered here, so be sure to download your own copy today. The time to prepare for all the exciting changes in search is now, after all, 2010 is only three years away.
As part of this week's Search Engine Strategies - San Jose convention, I was invited to a private party with fifty or so of top folks in the SEO world. The event was called the "Vintage Tub and Bath Event" and took place at the beautiful Japanese gardens and restaurant in Saratoga, Hakone. Attendees included representatives of Yahoo, Live Search (MSN), and, of course, Google.
SES San Jose 2007
One of the top entertainments of the evening was slot-car racing. There were many races, and they were open to anyone at the party. But then the smackdown came: a race between the search engines to see who is truly fastest.
It was an exciting and tense race, with each of the participants balancing control and speed just as real race car drivers do. And, like real races, the cars were painted up to represent each company's logos. In the end, Yahoo took the checkered flag, followed by MSN Search in second, and (oh my goodness) Google losing the race! (Ask was a no-show).
The Search Engine Slot Cars
When asked what his winning strategy was, Yahoo jockey Tim Mayer answered, "I just stayed on the track."
Emcee Allan Dick quickly relayed that information. "Did you hear that folks?" he asked. "Yahoo says to stay on track!"
Rest well tonight, People of Search. It's all been solved. Yahoo is the fastest.
- prev
- 1
- next





