Searchlight

Google's Webmaster Central has become a very important resource for anyone who has a Web site, works on a Web site, or, like SEO practitioners, helps others with their Web sites.

Google continues to roll out more features and better functionality to existing features, and now they just did a little bit of both with the addition of their Generate robots.txt function.

Google had previously added a robots.txt analyzer, which at this point is still the more useful of the two tools. For those who aren't aware, the robots exclusion protocol helps with instructing search engines how to interact with a Web site. There are a number of directives available, but the main purpose of the robots.txt file is to instruct the search engines about content that a site owner doesn't want the robots to crawl.

Why in the world would you not want search engines to crawl any of your content? You may have content that, for whatever reason, you don't want others to find through search results. Note, however, that this is not the same as secure information that requires authentication through a log-in.

Your site may have its own search function that creates "search results" for your site. Search engines generally do not want to include search results within search results, so this content may not be returned for searches on the engines anyway, so you might want to focus the crawlers elsewhere for greater crawler efficiency.

Or you may have duplicate content issues that you could use robots.txt to filter out. This is especially common with a content management system (CMS) that creates a separate printer-friendly page.

Regardless of your specific needs, having a robots.txt file can be important to a site. Rarely is there a site that can't benefit from disallowing at least some content. Even if you have nothing to disallow, you may want to take advantage of the auto-discovery feature for your XML sitemap. Finally, depending on your server log system or analytics package, not having a robots.txt file can be problematic if it inflates your "404 File Not Found" error reporting, which can happen because search engine spiders will request the robots.txt file automatically when they come to your site.

Right now, the robots.txt generator is rather basic and I hope that Google will add more features to it going forward. Currently, site owners have to paste in URLs and URL patterns to build the file. It would be great if it would provide a list of URLs or patterns extracted from a site to help automate the procedure for anyone not familiar with the protocol.

There is more information about the protocol, though a bit more on the technical side, at the robotstxt.org site and you can find more engine specific information on crawling and robots.txt from Google, Yahoo, MSN, and Ask.com.

One important tip is that the following directive tells all spiders they are allowed to go anywhere:

User-agent: *
Disallow:

And, more importantly, the following directive, which I sometimes see when I think people really wanted the above:

User-agent: *
Disallow: /

The latter tells the spiders to stay out of the entire site--clearly two very different results, so be sure you understand which does what.

As much as most high-quality SEOers (those who practice search engine optimization "honorably") like to think of their pursuit as pure and intellectual, the fact of the matter is that search engine optimization is a form of marketing, no matter how you approach it.

Perhaps short of nonprofit organizations and the occasional Internet artist, people interested in getting their Web pages in top search results are interested in making money from the traffic that will come from those search engine links.

A recent Hitwise article revealed some interesting demographic numbers that the company has measured through its sources. In comparing American Google and Yahoo search users, Hitwise found that Google has a higher percentage of users in older age brackets than Yahoo.

This negates some of existing beliefs in the market about the composition of those two user groups; a commonly held belief has been that Yahoo was for the aged, Google for the youth. Maybe not:

• In the category of users 55 and older, Google has 1.5 percent more of the market share.
• In the category of 45- to 54-year-old users, Google has 2 percent more of the market share.
• In the category of 35- to 44-year-old users, the market share was more or less equal.
• In the category of 25- to 34-year-old users, Yahoo has almost 3 percent more of the market share.
• In the category of 18-to 24-year-old users, Yahoo has 1.5 percent more of the market share.

It makes for a very nice, even curve! One implication that can be drawn from this is that, since older Americans tend to have more money than their younger counterparts, and Google users in general also tend to be more affluent, targeting the Google search engine over other engines makes much more marketing sense. Of course, this is in addition to the fact that Google's market share in the search world hovers somewhere around 60 percent, while Yahoo's hovers around 20 percent.

Google's search dominance isn't any sort of secret, but if these demographic numbers are accurate, the point is further magnified. Other search engines shouldn't be ignored by those investing in search optimization, but it's no wonder that SEOers are so quick to focus on Google; it offers the best bang for the SEO buck.

The term and concept of Microhoo can be dated back a couple of years now, but it looks to be much more of a reality today than it ever was. Thursday night, Microsoft sent a letter to Yahoo offering $31 per share in cash and stocks. This is more than 50 percent over the worth of the company relative to its Nasdaq trading price this week. The bid by Microsoft to buy Yahoo (it adds up to $44.6 billion) is surely a way for both companies to best do battle against Google, and such a move is ripe with many serious ramifications both inside and outside the world of SEO.

Speculation is rife on the Internet about the possibilities of the deal coming to fruition and the ramifications if it does. Stock prices are swelling and wavering, and investors are scrambling to properly position themselves. So, what's the most important issue in my view? Well, I just don't think they're going to call it Microhoo. I think there are better options.

Yicrosoft, for example, has a better ring to it and seems less questioning than "micro-who?" The name Mihoo looses the identity of both companies, and Yacrosoft sounds downright regurgitated. Mahoo, I am told, has predefined connotations (Google it), and Yicrohoo, if it doesn't have predefined connotations, really should. And what about Microsoft's Live search portal? It would seem to be the logical match to Yahoo. Could they go with Livehoo? It's certainly better than Yalive, which looks slick but could easily be mispronounced as "olive."

The winner in my book is Yasoft. It just sounds cool. Oh sure, there are a couple of small entities out there that may lay claim to the name, but no one that couldn't be bought out for five figures. The "Ya" says "yes" while the "soft" says "safe," as if it could just as easily be a high-end brand of bathroom tissue...which is exactly what Yahoo stockholders are going to need to dab at their tears if this deal doesn't go through. Come to think of it, they should grab the name either way.

Social media has certainly come into its own over the last year. Whether you log more hours online than you do sleeping or are the type that only turns your computer on every couple of days — gasp — there is a good chance that you have some familiarity with social media, even if you didn't know that's what it was.

Maybe you get a chuckle every now and then from a video on YouTube, upload photos to Flickr for friends and family to see, ask and answer questions on Yahoo Answers, do a little networking and reconnecting with old classmates and work colleagues on LinkedIn, keep track of popular news on Reddit, jot notes on your mates' walls in Facebook, or have carved out your own little corner of the web with a MySpace page. If you have a username and login at even half of those sites, then you know all to well the impact of social media. And for many, that list is but a mere sampling of their social circle.

But what happens when social media collides with daily life? Good, bad, or otherwise, most of us live extremely busy lives. We've found that all of the great tools that have been developed to simplify our lives and allow us more time, often just allow us to do more with the time we have. Is that a bad thing? After all, it is our life and if we are doing the things we want to do and that are important to us, wasn't that the ultimate goal to begin with? If you are expecting answers to those questions, sorry, they aren't coming.

The problem, or at least the challenge, with social media for individuals is simply one of mass. There is only so much time during the day and night that any one of us can put toward the social media venues. At some point, most of us will probably gravitate to a few key sites. Will this gravitation be a slow fade, or cold turkey? Will it be individually, or a revolt of mass proportions? Only time will tell of course.

But hold that thought. The story for businesses is a little different. Social media provides a powerful channel to reach out and interact with the community. While things in business rarely come without a cost, social media can be an extremely cost effective extension of nearly everything a company does. Even as social venues shift in popularity or come and go, businesses can introduce themselves and interact with those who share the same interests. As marketing channels go, the ability to connect with highly targeted, highly interested individuals at a personal level, is pretty hard to beat.

So while the future and popularity of individual social media venues may as yet be undetermined, and individual users' levels of interaction may vary, social media is here to stay. There's so much more that could be said, but I have to go check my Facebook page.

Like many companies, Netconcepts recently celebrated the holidays with a little staff get together and mystery gift exchange. There were the usual, but always popular gifts of coffee, tea, chocolates, and wine, as well as a very popular Atari console ... and as we did the "you can open one or steal one" gift rule, it certainly was the most fought over gift.

Unfortunately due to timing, Stephan wasn't able to make it as he was out visiting our friends in Seattle and participating in a great SEOmoz Whiteboard Friday otherwise he might have been torn between the Atari and the unique gift that follows and shows that even search geeks have a sense of humor (twisted as it may be).

Every search marketer has their bag of tricks and tools, but why settle for a bag when you can have a kit ... or more accurately, a box with built in carrying handle? And at $15 or less, the cost limit for our gift exchange, within range of even the budding search marketer.

What comes in a Search Marketer's Kit? I thought you'd never ask. Luckily, each item in the kit comes with its own identification tag.

... Read more

Games always seem to capture a lot of attention this time of year, in part because of kids and toys for Christmas or adults and getting together for the holidays. It would seem that search marketers enjoy games, too. While this often revolves around contests or challenges to get a site to rank No. 1 in the SERPs, at PubCon this year, SEOmoz even got some search marketers to unplug long enough to play an SEO version of Werewolf.

With only a week before Christmas, I couldn't help but wonder what games might be dancing around in search marketer's minds. While these classic renditions probably haven't made it to the shelves yet, maybe this will plant the seed for next season.

When it comes to games, it's hard to beat the classics, of course...so here are 10 searchized-games for search marketers:

Connect Four Reciprocal Links — (based on the classic Connect Four game), This game is based on the outdated concept of reciprocal linking. The winner is the one who can create the biggest reciprocal linking circle. Of course, this game is purely for fun.

Battleship Text Link Buys — (based on the classic game of Battleship) This text-link-buy smack-down game puts each opponent in the role of seeking out the other's text link buys and reporting them to Google. The one who sinks the other's PageRank first is the winner.

Keyword Scrabble — (based on the classic Scrabble game) Keywords are a foundation of the search world. Players take turns building on keyword phrases, earning points based on the smallest keyword phrase that achieves the highest rank in Google, Yahoo, or MSN.

Twister Search Career — (based on the zany classic Twister) Like any industry, the search industry often features moving from one company to another. In this game, instead of colored circles, the playing mat features company logos. Just like the Web industry, each player must reach out and touch a number of different company logos, constantly moving from one company to another without falling down.

SERPs Memory — (based on the age-old game of Memory) Test your visual and memory skills by viewing a listing of top-10 SERPs from Google, Yahoo, and MSN before they disappear, and then matching up the listings across the SERPs. The one who gets the most matching results across the engines wins.

Spammer Mouse Trap — (based on the fun game of Mouse Trap) Ever wonder what a day in the life of Matt Cutts would be like? Find out with this exciting game where you get to build a trap to catch search spammers.

Search Operation — (based on the nerve-racking game of Operation) Search marketers have to have nerves of steal when every move they make can result in a site plummeting in the SERPs. Test your skill by removing bad bits and pieces from a Web page; but be careful--alert the spiders and you may get a shock.

Stratego Search Position — (based on the battle strategy game of Stratego) Search marketing, like most forms of marketing, is often a matter of strategy. Position and move your Web pages to overthrow your competition, gaining stronger search positioning, winning more site visitors, and achieving the highest ROI.

The Game of Life Search — (based on The Game of Life) One of my favorites growing up, this version has you go through the life of a search marketer, mastering your trade, building your skills, moving from company to company and site to site to achieve your search life goals.

And quite possible the grandest game of all...

Search Monopoly — (based on the classic game of Monopoly) This game can provide hours of fun and is one of the most involved of search games. Chose your playing piece, from exciting favorites like white hat, black hat, Googlebot, Slurp, or MSNbot, and work your way around the game board, accumulating Web properties. Beware--landing on a Made-For-AdSense site or SPLOG can cost you, as can landing in the Google sandbox. Pull a Chance card and find that "Matt Cutts caught you buying Text Links. Lose two PageRank points and wait a turn to file for re-inclusion." But it isn't all bad...pull a card from the Community Chest and you may find that you "Win a one year SEOmoz Premium Membership," or "A free pass to SMX." The winner, of course, is the one who accumulates the most Web properties and finishes with the largest amount in VC funding.

As far as I know, none of these games exist yet, but a lot can happen between now and next year.

*All of these game concepts are based on related games that are the property of Hasbro. Until these games exist, you'll just have to buy the classics and pretend or modify to make your own.

These days, there are several ways to get a customized or internal search engine free of charge. Google's Custom Search Engine (CSE) and Yahoo's Search Builder lead the way, and their products are fairly easy to implement. After the Big Two, there are several services trying to make their mark, but one that that stands out is the Swicki from Eurekster.

A Swicki is a combination search portal and widget that can be customized on any topic or topics either within a Web site, group of Web sites, or the Web at large. The end product is a custom search experience returning relevant, targeted results as well as revenue opportunity (yes, you can monetize your visitors!) for blog and Web site publishers.

A Swicki is created in a fairly simple, four-step process. And, they can be shared, so if creating one seems like too much of a hassle, there are literally thousands to simply grab for free from Eurekster.com. These neat little customizable search engines look nice on a Web site or blog and they pull information from a combination of sources, including Blinkx's video feed.

But there's more to a Swicki than that, especially from an SEO perspective. First of all, the company has its own search engine, spidering the Web and producing its own results--and, it works very well. Secondly, the social search widget included in the Swicki features a customizable tag cloud that adds popular search terms automatically to the Web page upon which it is embedded, making it easy for the Swicki user to reflect popular keywords or build their brand.

But what's unique is that the Swicki search results ranking is reflective of every previous keyword search, click, vote and user behavior within each customized Swicki, lending a publisher-guided and community-powered slant to the results. While this might not be ideal for general search, it can really improve upon the search experience when used within the context of a niche or topic of interest.

Search is a constantly evolving and changing entity, and this year has certainly seen more than its fair share of change. Possibly highest on the list has been the move to blended or universal results. Much of the focus and discussion has revolved around Google Universal, but they aren't the only game in town and all the majors have now entered into blended results to some extent or another.

This presents interesting opportunities and challenges. Those who focus on developing a fuller and broader Web presence, adding video, news, blogs, images, local content and social media to the mix, greatly increase their opportunities to gain rank positions across all the engines. The challenge however will be that the results across the engines may become even more varied, as each engine puts its own spin on the blending.

It's still very early in the game as the engines continue to ramp up and tweak their algos as they evolve and introduce searchers to blended results. It will be interesting to watch the evolution of blended search across the different engines. Today, results vary within the engines. Some searches show little to no signs of blended results, and others, show considerably more.

For an interesting example, let's take a look at a single search across the engines and see how they compare. Since we are talking about how blended search is evolving, what better search term than "cavemen" to use as a test. It is an interesting term for our test because it has such a wide frame of reference and actually poses a rather interesting challenge for the engines. Depending on the searcher, it may carry very different connotations:

• Prehistoric humans
• The definition of the word itself
• The ad campaign for the Geico commercials
• The newly launched sitcom on ABC based on the Geico commercials

Here are screenshots of that search in Google, Yahoo, MSN (Live Search) and Ask. It is interesting to see the variety of the impact of blended results across the engines, from very little to considerable impact. As you experiment with different searches though, you may see the engines' position on this scale shift up or down.

Along with calling out some of the specific blended results, I've also noted some of the refinement options, such as related searches, narrowed or expanded searches, and Yahoo's new "Search Assist."

While the survival of the new Cavemen series may be in question, the continued evolution and survival of blended search is without doubt.

Whether you have multiple locations or just one, local is fast becoming an important market online. Traditionally, the local marketing battlefield consisted of phone directory listings, billboards, newspapers, TV, radio, and anywhere else that made sense to get your name seen. The web and search engines have opened up a new local battlefield, and smart businesses are moving quickly to gain a foothold.

Surprisingly though, many businesses have failed to tap into one of the more powerful tools available for reaching the local market. If that's surprising, then what makes this amazing is that this tool is free and will become even more important as local search information continues to be blended into regular search results.

The major search engines saw the growth and importance in local search years ago and started putting tools in place to not only help searchers, but also businesses. What the search engines may have discovered was that many searchers weren't aware of these local search tools, like local.google.com and local.yahoo.com. Instead, people just searched for local things where they searched for everything else, which further helps to explain things like universal search.

What was really exciting though was that the search engines also provided businesses with a way to get listed. Yet many businesses have yet to tap into this opportunity. If this sounds like you, then there is no time to waste, and better yet, getting listed takes no time at all. The process varies by engine, but is often as simple as having a postcard sent to your business address, receiving an automated phone call at your business location, or filling out a form or email.

Google

Go to local.google.com and click on the "Add or Edit your business: Learn more" link along the left side. If you don't have a Google Account yet, you'll need to create one. Once you've done that, you'll get access to the Local Business Center, where you can add or edit your business listing.

Yahoo

At listings.local.yahoo.com, you'll see that Yahoo provides different listing levels, including a basic free listing. The Basic listing is done in 4 easy steps, while the paid versions have an extra step or two.

MSN Live Search

Not as direct, you'll need to click on the Help link in the upper right corner at local.live.com, and then select the first FAQ item, "How do I add my business listing to Live Search Maps?" From there you'll get a link to an infoUSA.com form for adding or updating your information as need be.

Ask

Even Ask wants to list your local business. Even less intuitive than the others, you'll need to follow the Help link in the upper right corner at city.ask.com, then toward the bottom of the FAQs is one labeled, "I'm not finding my business in AskCity. How can I get it added?" With Ask, you'll need to send off an email with some information to Ask's Customer Service.