June 22, 2005 6:25 PM PDT

Microsoft pushes spam-filtering technology

Related Stories

Should Microsoft own antispam?

November 9, 2004

Microsoft-backed antispam spec gets filtered out

September 23, 2004
If your e-mail does not have a Sender ID, Microsoft wants to junk your message.

Sometime around November, Hotmail and MSN will flag as potential spam those messages that do not have the tag to verify the sender, Craig Spiezle, a director in the technology care and safety group at the software maker said Wednesday. The move is meant to spur adoption of Sender ID, he said.

Sender ID is a specification for verifying the authenticity of e-mail by ensuring the validity of the server from which the e-mail came. While the purpose of curbing junk mail may be laudable, the debate on how to stop the tide of junk mail is still ongoing. According to Microsoft, up to 90 percent of e-mail is spam.

Critics say Sender ID, which includes technology developed by Microsoft, is not an accepted standard and has many shortcomings. Also, there are technologies that compete with Sender ID, such as Yahoo's DomainKeys.

"We think Microsoft is trying to strong-arm the industry into the adoption of an incomplete and not accepted standard," said Dave Rand, chief technologist for Internet content security at security software company Trend Micro.

Microsoft's move increases pressure on e-mail senders to adopt Sender ID. The technology requires Internet service providers, companies and other Internet domain holders to publish so-called SPF (Sender Policy Framework) records to identify their mail servers.

About 1 million domains currently publish SPF records, Microsoft said. That's far from the 71.4 million registered domains worldwide at the end of last year. Still, because some large e-mail senders such as AOL support Sender ID, about 30 percent of e-mail today carries Sender ID information, according to e-mail filtering company MessageLabs.

Criticism for the technology
Sender ID has not been a success because it is not very highly regarded, said Ray Everett-Church, co-founder of the Coalition Against Unsolicited Commercial E-mail and co-author of the book "Fighting Spam for Dummies."

"Microsoft has been trying to shove Sender ID down the throats of the Internet community for several years now, to little effect," he said.

Microsoft's unilateral move may hurt Internet users, he said. "Sender ID isn't widely deployed, meaning that average users are now at risk for having their legitimate e-mail tagged as spam when they send messages to Hotmail users."

Experts say one of the problems with Sender ID is that it doesn't work with e-mail forwarding services. The basic premise of Sender ID is to check if an e-mail that claims to be coming from a certain Internet domain is really being sent from the e-mail servers associated with that domain.

"If you receive mail forwarded through, for example, a university alumni account, the Sender ID check fails," said Matt Sergeant, a senior antispam technologist at MessageLabs.

The Internet Engineering Task Force, a standard-setting body, dissolved a working group on Sender ID in September. Still, Microsoft is plowing ahead with Sender ID, perhaps in a last-ditch effort to make good on a promise by Chairman and Chief Software Architect Bill Gates to can spam by 2006.

"All domain holders and e-mail senders should be publishing SPF records and planning to do that now if they want to improve the legitimacy of their mail, plus protect their domain and consumers. It is the responsible thing to do," Microsoft's Spiezle said.

Turning on the filters at Hotmail and MSN will give e-mail senders a reason to adopt Sender ID, Spiezle said. Without an incentive, many have said that they won't publish SPF records, he said. "We're in a catch-22," he said. "What we're trying to do is to do the right thing by giving everyone advance notice."

However, this Microsoft effort to push adoption of Sender ID is likely to fail, certainly with such a short deadline, said Jonathan Penn, an analyst at Forrester Research. "Hotmail is in no position to dictate that organizations adopt Sender ID," he said.

Adopting Sender ID or any other technology requires time and money, Penn said. "Company budgets are on a yearly cycle, and most of them have no money for such a project this year," he said.

Microsoft argues that publishing SPF records is simple. It usually does not require new hardware or software and the most arduous part is doing an inventory of mail servers and the subsequent maintenance of the record, Spiezle said.

46 comments

Join the conversation!
Add your comment
There are other problems with this as well.
As much as I support the idea and concept of Sender ID and SPF, I've found too many caveats with them which even the developers of the technologies cannot explain -- such as the technology not working properly with multi-homed hosts, nor does it play well with machines running an SMTP server bound to an IP alias where procmail is in use.

It's quite sad, really; such oversights are what made SMTP into what it is today...
Posted by katamari (310 comments )
Reply Link Flag
I have much better email system
I have come up with much better idea to ELIMINATE spam completly,its much cheaper and eaisier to create and implemet then all systems so far,actually it is bit simillar to MS but it actually can stop entire spam industry in its tracks.
My questions is ,not being programmer,where is best place to turn to develop or present idea.

If everyone used my system it will completly prevent spam,spaming would be worthless business and regardless or where you use my system it would work.
Again I wonder what is best place to start with idea,I would prefer bigger company to develop rather then using small one.
Posted by (32 comments )
Reply Link Flag
re: I have much better email system
Oh yeah? Well, I also have unsubstantiated claims of a better e-mail system, and mine actually beats the pants off of yours in simulations performed by unbiased third parties.
Posted by Bob/Paul (2 comments )
Link Flag
no big company!!!
It sounds like a great idea ... if you are looking for someone to make it real I (also) have a great idea : why donĀ“t you try to contact an OPEN SOURCE project team. It would be a FREE system for spam and all the people would use (and improve) it for FREE!!!
Posted by (5 comments )
Link Flag
Email system outlines
Here is what you can do with my system.
Use exiting email address,you can actually GIVE your email address to anyone you want,evan call your favorite spammer and asked him to spam you,its actually very CHEAP to create,very easy to use,hackable but so complex to hack into that it would take decades,actually it can handle multiuse emails with one single email address and yes it actually has tracking system that is so simple but yet very effective that it will ELIMINATE spam completly.
If anyone knows place to presnt this email me bostech.fl@netzero.com
Posted by (32 comments )
Reply Link Flag
Starting in november, MSN will stop offering email services
Starting in november, MSN will stop offering email services. What they plan to do is exactly that. Their users will not be able to receive all their email anymore. Sender-ID is a proprietary and patented method of MicroSoft, and was not adopted because it is not compatible with the licences of the servers used by most email providers. So M$ plans to cut up the email world into pieces, much like the instant messaging world. MSN/Hotmail users should prepare and get an alternative email account before November. Either that or start using their "Junk mail" folder as an Inbox starting November. And of course the new Inbox would receive all the spam.

Neither SPF nor Sender-ID are technologies that can control spam. They can to a very limited extent help avoiding phishing, by identifying the forgery of addresses like INFO@MICROSOFT.COM. But they would not stop mail "from" addresses in all of INFO@MICR0S0FT.COM INFO@MlCROSOFT.COM INFO@M1CROS0FT.COM or INFO@M1CR0SOFT.COM.
Posted by hadaso (468 comments )
Reply Link Flag
What "tag to verify the sender"?
Can it be that the writer of this article doesn't know what he is writing about? The phrase "flag as potential spam those messages that do not have the tag to verify the sender" from the first paragraph suggests that. There is no "tag" in messages conforming to either Sender-ID or SPF. The only thing required is a specially formated TXT record published in the DNS for the domain. And then of course that all email sent "from" the domain be sent from a server in the range designated by this record. This is no real problem if recipients would follow the standard, since the standard includes the option to designate "any host" as a possible server. But you can count on MSN not to honor the standard and filter mail if the designated range is "too wide". And of course if it's Sender-ID(TM) then there's also the burden on the senders to follow a quite complex set of rules on formatting email headers, (not all of these rules are 100% compiant with RFC2822 - the email message formatting standard) and to first get permision from M$ to use their patented method.
Posted by hadaso (468 comments )
Reply Link Flag
Message has been deleted.
Posted by stenar (29 comments )
Reply Link Flag
what about...
...those of us who use our own computers as SMTP servers? i use sendmail on linux to send my e-mail and i do not believe i would be able to obtain a SenderID as i do not use an e-mail provider. how do M$ intend that i send e-mails to sheep on MSN? i don't want to use an e-mail provider because my previous one, for unknown reasons, is not allowing me to access my outgoing server to send them, thus the reason i use sendmail. it seems that M$ would happily try and cut off 70% of consumers off from its service. they forget that they do not own e-mail like they own the desktop and the internet so they cannot push their weight around.
Posted by Scott W (419 comments )
Reply Link Flag
Or..
Those of us who use postfix relay to send email from a domain via an ISP?

Also does anyone know how it works with mailing lists where the domain of the email server certainly doesn't match the domain of the sender? or are MS basically saying that Hotmail users cannot get mail from lists like Yahoo groups
Posted by Steve_a (22 comments )
Link Flag
It doesn't matter what email server you use...
Just put a SPF record (TXT record) in your DNS to say that the IP address of your SMTP server is authorized to send mail for that domain. Takes just few minutes, and your messages are fine.

If you want to use SPF/SenderID to authenticate inbound messages, that is a different story. But authenticating your domain messages is easy.
Posted by mbharr (6 comments )
Link Flag
once more.....
... MS is late to the party, and forgot to get dressed. If it takes a
hacker more than 10 minutes to defeat this Sender ID crap, the
hacker should be demoted to newbie. The key to controlling SPAM
is to eliminate the few sources which provide most of the SPAM.
There are adequate laws on the book for that now, what we need is
someone with the hiorsepower to apply them, not another useless
MS 'innovation' .
Posted by Earl Benser (4310 comments )
Reply Link Flag
SPF vs. Sender ID
The quotes from the Microsoft guy specifically refer to SPF. Nowhere is he quoted as mentioning Sender ID. Is it possible that the author of this story doesn't understand the difference between the two?

The fact that a Microsoft guy is again talking about SPF is probably the big news here.
Posted by roger.d.miller (41 comments )
Reply Link Flag
Sender ID = Caller ID + SPF
Actually, Microsoft talks about SPF as part of Sender ID and I don't think that is incorrect. Sender ID came about when Microsoft and Meng Wong of Pobox.com agreed to merge their technologies, Caller ID for E-mail (Microsoft) and SPF (Meng Wong.)
Posted by JorisEvers (48 comments )
Link Flag
Should expect no less...
...from MS. Like always, Microsoft wants to own everything. If they could buy the Internet, they would've done so already. Even after the Internet Engineering Task Force refused Sender ID as the standard, MS is still trying to push it. The IETF should somehow penalize Microsoft for such a move.

What Microsoft is doing is not only harmful to consumers, but is also disrespectful to the Internet standards bodies, like the W3C and the IETF. Their policy seems to be "Our standard was refused, but it will be the standard because we say it is." This is a subversive attitude, not wanting to play by the rules.

Now, I do applaud Microsoft for the Hotmail Spam Filters. They have done miracles for me, to the point where I only recieve about three or four Spam messages a week, and those are delivered to the Junk Mail folder. It makes me wonder why MS seems to think that their Sender ID is necessary, when they already have good filters.

The real solution to spam is in the consumer. Basically anyone can avoid being spammed: all you have to do is not subscribe to any newsletters, avoid entering porn sites and don't click on any "Free Screensaver" or "1,000,000th visitor" ad you see. This does not guarantee you will never recieve spam, but it will be minimal.
Posted by Sentinel (199 comments )
Reply Link Flag
HotMail market share
Enforcing Sender ID will probably have quite an impact on HotMail marketshare. Downwards. I noticed that GMail passed HotMail last month, and I'd expect many more to migrate if they find much legitimate email junked. I had that problem myself, and moved to GMail, which has a much better spam filter. User interface & size, too, now we're at it.

I expect that this would backfire monumentally. But probably they'll abandon the idea before it launches when they notice that ISP's are not following their intentions.
Posted by Frodo42 (6 comments )
Reply Link Flag
but...
... the general public has a good record of doing nothing. MS hardcore of customers (sheep) simply do not know of any other way. i expect that all hotmail users will get an e-mail detailing this "wonderful" feature to lock everyone in and think that MS are innovating yet again.
Posted by Scott W (419 comments )
Link Flag
Let's see if the Antitrust overseeers balk at THIS!
I doubt it because I believe the "remedy" was largely a sham.

Again, Microsoft shouldn't be able to dictate and create defacto
"standards" by leveraging its Monopoly in the Windows Distribution
Channel.
Posted by technewsjunkie (1265 comments )
Reply Link Flag
Sounds like a reason to avoid MSN and Hotmail.
Not that I've used or recommended those services in the first place.
Posted by rcsteiner (48 comments )
Reply Link Flag
Yahoo! Mail offers 1GB
And its spam filter is very efficient :)
Posted by 201293546946733175101343322673 (722 comments )
Link Flag
People are leaving already
According to Netcraft, GMail has surpassed HotMail significantly. I believe the most active HotMail users are leaving in droves - many of my friends did. This announcement, even if MS doesn't ever follow through, will increase the exodus, and leave HotMail as an insignificant player in the market.
Posted by Frodo420024 (13 comments )
Link Flag
easy
If all it takes to comply (would 'obey' be better?) is to a SPF/TXT record to my DNS zone, I'll go ahead and add one.
Posted by (1 comment )
Reply Link Flag
You're missing the point here!!!
The problem is that a single company can NOT CREATE an standard. They can propuse an idea!!! Because tomorrow will come another company with another "standard" for spam and before you know it you would be adding thousands of records to your DNS server and not to mention you would be fighting with the incompatibility between them!!!
Posted by (5 comments )
Link Flag
Microsoft can sit and spin!
If MS think they can cram this down my throat, just because some of my users may get their emails flagged at spam if they send emails to hotmail.com users, then they can think again. I am definately not adopting Sender ID, because of their approach!
Posted by (1 comment )
Reply Link Flag
Just Use Gmail
Its junk mail filter is one of the best :)
Posted by 201293546946733175101343322673 (722 comments )
Reply Link Flag
Filtering spam isn't the only problem...
Several people have responded with something like, "Just use product XYZ because it's spam filtering roolz...". This ignores one of the main problems with spam, which is the congestion it creates on the internet. Plenty of studies peg spam as accounting for 50-90% of all internet traffic. This means that in the end, you the subsciber, are paying for spam because it forces ISPs and backbone providers to have hardware and bandwidth they otherwise wouldn't need. They then pass this cost right along to you.
Posted by dilireus (4 comments )
Link Flag
Microsoft pushes spam-filtering technology
Just what we all need, Microsoft pushing an ID strategy that
won't allow forwarding of email! Brilliant! Genius! Incredible!

And completely worthless.

When are the people in Redmond going to get a clue about how
email actually works - that people forward messages all the
time, especially people using academic accounts and those who
use anonymous posting systems for good reasons - like those
who want to STAY anonymous because they have problems with
cyberstalkers and REAL stalkers, whacked-out ex's, nasty
divorce issues and even problematic issues like spammers who
have beaten every anti-spamming technology to come down the
pike to date?

When will they understand that some people actually VALUE their
privacy? When will they understand that Microsoft has NO right
to know WHO they are?

Yet they want everyone to identify themselves or they won't be
allowed to use their Hotmail service. If that's the case, then my
suggestion to the public is to use something else and teach
Redmond a lesson - that they are NOT the only players on the
Web!

Lee Darrow, Chicago, IL
Posted by mstrhypno (49 comments )
Reply Link Flag
Get your own house in order
two thirds of the spam I receive comes from either Hotmail or Yahoo with the sender indicated by random letter string. Maybe Uncle Bill should spend less time playing Big Brother and more time ferreting out zombies on his network
Posted by (1 comment )
Reply Link Flag
Hurts consumers with their own domain
I have a big problem with it, although I don't believe it will ever be widely accepted anyway. I have a family domain name for personal use through myfamilyaddress.com. I already have to send email from my ISP's mail server instead of my domain's mail server, because my ISP blocks port 25 (another attempt to curb spam). So they want to block you from sending email via your domain's mail server, then not accept your email unless it comes from your domain's mail server. You can't do both otherwise people would be locked into only using their ISP provided email. Hmmmm, that smells like a monopoly. I suppose we'll all have to use MSN as our ISP in order to use hotmail.

Same goes for thousands of people who work from home and send mail from their corporate addresses using their ISP's SMTP server due to port 25 blocking. They'd all be bocked by Sender ID.

I say who cares about Hotmail, there are plenty of free email providers that provide better service than hotmail anyway. I personally have a rule setup to block all incoming mail from hotmail because so much junk mail is coming from hotmail addresses (albeit usually spoofed addresses).
Posted by (5 comments )
Reply Link Flag
Not really
SPF (I'm not up on the details of exactly what Caller ID adds to the system) is no problem as long as you control the domain of the "from" address used in the email. You can set mypersonaldomain.com's SPF record to include mail.myisp.net, or even include all of myisp.net's SPF record. As long as you control the DNS for mypersonaldomain.com, you have full control over SPF for emails from ___@mypersonaldomain.com.

The problem is when someone tries to use their own server (mail.mypersonaldomain.com) to send mail with a "from" address at another domain (user@myisp.net). myisp.net's SPF doesn't include mail.mypersonaldomain.com, and you can't add it because you don't control myisp.net's DNS. But as long as you send ___@myisp.net through mail.myisp.net, it's not a problem.

Regarding port 25 blocking, get around that. Many ISPs have been driven into blocking port 25 to block all the zombie PCs, and I thank them for that. Some will open up port 25 if you give them a valid reason. If you're one of the zombies they're trying to block, you most likely won't have a valid reason, or even know that you're being blocked. Many hosting companies now offer an alternate port (often 2525) simply because so many ISPs are blocking 25. Also, standard SMTP is sent in plaintext, even your password. SMTP over SSL defaults to port 465, so it gets around port 25 blocking too.

Get an ISP that lets you do what you want with your connection (at least reasonably speaking) and/or a host that supports alternative methods for commonly blocked stuff, as well as some basic email security. I realize that many people don't have many/any options for ISPs, so it's important to get a decent host. Then again, it's very likely that they already offer these features, they just aren't very well-known.
Posted by (3 comments )
Link Flag
The bottom line.... Avoid MSN and Hotmail like the plague!!!
This is just another reason to avoid the pariahs known as MSN and Hotmail

there has been bad talk about the pair for years and very little good.

the message before me (title Hurts consumers with their own domain name) has implemented a drastic but good idea... he has his filters trash any hotmail sender due to high spam.. spam that wouldn't be stopped by the "ID" in question since hotmail obviously includes it in their eMail

in all my time on the internet I've only encountered two, maybe three, actual people who use hotmail and the two I can think of are both overseas

might I suggest an extension to my predecessors extreme?

go ahead and automatically trash hotmail senders... BUT FIRST evaluate the subject line for a code word (a suggestion COLDMAIL) that will send it to inbox if found.... have friends add the codeword to all eMails and they won't be trashed
Posted by qazwiz (208 comments )
Reply Link Flag
This is hilarious...
...I have no idea why anyone would use hot mail or any other web mail service as their primary email. I use a local provider for fast, reliable email. I also have a hotmail account that I use as a trash receptacle. I only use it for things that are likely to generate spam and every time I log on to the hotmail account I delete nearly all of the messages because they're ALL SPAM! Why would I trust M$ to control spam?
Posted by Michael Grogan (308 comments )
Reply Link Flag
Simple? Maybe for one domain you MS idiots
>Microsoft argues that publishing SPF records is >simple. It usually does not require new hardware or >software and the most arduous part is doing an >inventory of mail servers and the subsequent >maintenance of the record, Spiezle said.

We maintain 40+ servers with a total of 10,000 domains hosted. It ISN'T simple, it ISN'T easy AND it screws up the forwards our dear customers value so much. I hate it when someone who doesn't know a THING about implementation of this sort of operation on a large scale states it's "simple". Who shall I send the invoice to? Spiezle or Microsoft Inc?
Posted by (1 comment )
Reply Link Flag
Is it really as difficult as you make it out to be?
It sounds like you have a hosting company. Depending on the setup, the users may be allowed to alter their own DNS, in which case SPF wouldn't even be your problem.

Assuming it is your problem, and nobody has anything too crazy hosted, one simple line in the default DNS template would cover the vast majority's SPF with absolutely no configuration: "v=spf1 mx -all"

That will designate the domain's MX(es) as the only valid sender(s). For most domains, especially hosted ones, the MX is used to send and receive mail, so simply using the MX record will be just fine in most cases. I know that it isn't going to cover every single case, but it will work for many people, especially the ones I'm guessing you're referring to.

Another option is to have all client records include your company's SPF record, then maintain all the servers there. Assuming you have different servers hosting different accounts, you would essentially be setting up an SPF record that stated any of your hosted domains could use any of your hosting mail servers. While it wouldn't technically be 100% correct, it would only be a problem if an account on server1.hostco.com was being spoofed by a mail from server2.hostco.com. If you've got spammers using any of your servers, it's a problem, regardless of whether the spoofed domains belong to your other customers or to external sites.

Yes, bigger clients who are using multiple mail servers, where SMTP servers are separate from the MX, would need more configuration. But I'm pretty sure it's not a case of having to enter 40 server names for each of the 10,000 domains either.
Posted by (3 comments )
Link Flag
...and then MS implements RSS: a brand new SPAM Superhighway!!!
What good will it do to implement half-thought-out authentication schemes for email when MS is proposing to implement a brand new scheme (RSS) for allowing unauthenticated visitors (or visitors who have been authenticated but are unwelcome right now) to take over all of your bandwidth and all of your machine cycles?

This is like forcing a Sunday newspaper into someone's face while they are trying to drive!!!

Doesn't anybody at MS understand the value of "speak only when spoken to" operation? Wasn't enhanced PRODUCTIVITY supposed to be the primary rationale for the desktop computer? We're already wasting WAY too much time on machine cycles on things which beep, sing, send useless reports to third parties, repeat things which don't need to be repeated, and log things which never should be logged. Do we now have to have each and every machine saddled with something which ties up our machine with communication just for the sake of communication? And talk about potential "buffer overflow" problems...!!!
Posted by landlines (54 comments )
Reply Link Flag
 

Join the conversation

Add your comment

The posting of advertisements, profanity, or personal attacks is prohibited. Click here to review our Terms of Use.

What's Hot

Discussions

Shared

RSS Feeds

Add headlines from CNET News to your homepage or feedreader.