Here’s a fascinating look back at the spam wars by former Gmail spamfighter Mike Hearn.
I was involved for most of the previous decade as (among other things) the email admin for a small ISP. We used a mix of public blacklists, a private blacklist, virus filtering, SpamAssassin with both shared rules and local custom rules, and various other tools all tied together, some at the Sendmail level and the rest through MIMEDefang. It worked tolerably well, though of course it wasn’t perfect. I find it amusing that Gmail declared victory on spam in 2010, the same year that I changed jobs to a position that was more software developer and less sysadmin.
Privacy is a growing concern these days, so he also talks about the impact that widespread end-to-end email encryption would have on spam fighting. If you’re the mail handler, you can’t filter on, say, links found in the message, or characteristics of the writing or formatting, or anything else in the content. You can’t even run statistical analysis on all known spam and non-spam to see which the new message fits better. All you can do is look at where it came from and where it’s going.
Moving the spam filter to the client lets you do content filtering on your own mail, but you can’t take advantage of the larger volume of data that an ISP can, which means your filtering isn’t going to be as effective. And if your main email client is your phone, that’s really going to slow it down — and chew up battery.
Encrypting more of our communication is probably the way to go, but we’ll have to come up with new approaches to some previously-solved problems like this.
It got me thinking: Most of us not only accept that our email providers will look inside our mail to filter spam and viruses, we expect it. That’s weird. The idea of the post office looking inside our letters is so abhorrent that even tracking programs raise concerns. The idea of an actual person reading our email in transit creeps us out. Many people have problems with the idea of automated systems (like Gmail) reading our email for purposes of targeted advertising. But spam filtering? We get upset if it’s not happening!
That says something interesting about our priorities, and about how big an impact unfiltered spam has on our email.