Obligatory “I’m still alive” post, and new blog announcement

March 18th, 2011

Yes, I realize I haven’t posted here in a while, though I intend for that to change. Currently, I’m relatively busy with my new blog, Winterdrake, which is about several “geeky” subjects: video games, retro gaming, comic books, fantasy, science fiction, RPGs and so on.

I have been thinking about consolidating my blogs a bit more, and focusing my efforts on a maximum of three or four. Therefore, Winterdrake has “absorbed” my old video games blog, The Games of My Life (not linked here since it’ll eventually vanish), although I have been moving some of the better posts there (while expanding them — some of them are now more than 50% new content). That won’t happen to this blog, though; I think “geeky” and “hard tech” don’t necessarily mix, and I don’t want to annoy readers interested in one subject with the other. People who, like me, are interested in both, should read (or subscribe to) both blogs, of course. :) There have been a couple of more technical posts on Winterdrake, too, but I don’t intend for it to become my “real” tech blog; that’s for this blog here. I may cross-link both blogs when a post makes sense in both, though.

So… enjoy Winterdrake. Meanwhile, I have been spending some time optimizing my anti-spam gateway, and may post some tutorials about that here, in the near future. Stay tuned…

 

What if everyone used SPF?

November 27th, 2009

To end my SPF series, I’m going to consider the following question: what if everyone used SPF? Would it “end” spam?

The answer is, of course, “no, but…”. But, first, let’s understand the question itself.

In “everyone used”, what does “used” mean? If you followed my previous posts on this subject, you’ll know that there are two distinct parts: having an SPF record for your domain, and configuring your SMTP server to reject email purporting to be from an address with a valid SPF record, when it doesn’t come from an authorized server.

Let’s assume that the question implies both.

So, if every legit organization had an SPF record and enforced SPF in their incoming email servers, what would it mean?

It still depends on what was meant by “enforced”. For instance, as it is now, it makes sense to use an SPF record (and reject mail coming from an unauthorized server, as I mentioned before), but not to require an SPF record, as most of the world is still not using it.

If that changed, though… it would certainly make things a lot easier for the “good guys”. Think about it: what does SPF prevent? The faking of sender addresses. Who ever does that? Spammers. Therefore, who has ever a reason not to use an SPF record? Spammers. In a world where every legitimate organization used SPF, having such a record wouldn’t mean that the sender wasn’t a spammer, but not having one would certainly mean that he was. Ergo, reject any mail from a domain without an SPF record, even before verifying whether the origin server is authorized for that domain.

Of course, spammers would adapt, and have SPF records for their own domains. But never again would they be able to fake a sender address. They would never again be able to efficiently pretend to be your internet provider, or your bank, or Facebook, or anything like that. They would have to use their own domains in the sender address… and they don’t exactly tend to look “nice”; besides, they’re not what your ISP or your bank would use.

A world with SPF would mean a world where you could actually trust the “From:” field. Can you imagine such a thing?

FreeBSD 8.0 is out today

November 27th, 2009

See the release notes, if you’re curious.

I had been using FreeBSD 6.x and later 7.0 until I bought my current home server, about a year and a half ago, which had a SATA controller that FreeBSD didn’t detect out of the box. I used that as an excuse to try out Linux as a server again, and switched to Ubuntu, which I’m still using right now. But FreeBSD is a very nice OS, and I miss some parts of it (such as pf, ported from OpenBSD, which I also used before, and which is infinitely better and more sane than the almost Lovecraftian horror of iptables.)

If I ever get a second home server, maybe I’ll try one of the BSDs again, just for fun. :)

Postfix’s log file when using SPF: what it looks like

November 26th, 2009

In the previous post, I wrote, at the end of the recipe:

restart Postfix. Check your logs to see if everything is working properly.

But what to look for? Here are a few real life examples, from my own Postfix log file:

Nov 26 07:07:27 sh postfix/policy-spf[10685]: : SPF none (No applicable sender policy available): Envelope-from: bounce-1492514-406278644@lyris.diskeepermail.com
Nov 26 07:07:27 sh postfix/policy-spf[10685]: handler sender_policy_framework: is decisive.
Nov 26 07:07:27 sh postfix/policy-spf[10685]: : Policy action=PREPEND Received-SPF: none (lyris.diskeepermail.com: No applicable sender policy available) receiver=sh.dehumanizer.com; identity=mailfrom; envelope-from="bounce-1492514-406278644@lyris.diskeepermail.com"; helo=lyris.diskeepermail.com; client-ip=68.177.217.241

This (note the “SPF none”) is what will happen for most emails: no SPF record exists. Unfortunately, most of the world is still not using it. The mail passes this check, since it’s not a good idea to enforce SPF yet (but may fail one of the remaining ones, such as an RBL list or SpamAssassin.)

Nov 24 16:19:56 sh postfix/policy-spf[32619]: : SPF pass (Mechanism 'ip4:209.128.72.240/28' matched): Envelope-from: promotions@iambic.com
Nov 24 16:19:56 sh postfix/policy-spf[32619]: handler sender_policy_framework: is decisive.
Nov 24 16:19:56 sh postfix/policy-spf[32619]: : Policy action=PREPEND Received-SPF: pass (iambic.com: 209.128.72.242 is authorized to use 'promotions@iambic.com' in 'mfrom' identity (mechanism 'ip4:209.128.72.240/28' matched)) receiver=sh.dehumanizer.com; identity=mailfrom; envelope-from="promotions@iambic.com"; helo=ns1.iambic.com; client-ip=209.128.72.242

Above is an example of a legitimate mail from a sender with a properly configured SPF record. The record says “here are the server(s) that send email from this domain”, and the origin server is indeed one of those. The email passes (note the “SPF pass”), and, if you’re using SpamAssassin, it takes that into account, meaning that the mail is less likely to be considered spam. (See? That’s a great reason to have an SPF record for your domain!)

Finally, the following:

Nov 24 19:57:13 sh postfix/policy-spf[1210]: : SPF fail (Mechanism '-all' matched): Envelope-from: pedro@dehumanizer.com
Nov 24 19:57:13 sh postfix/policy-spf[1210]: handler sender_policy_framework: is decisive.
Nov 24 19:57:13 sh postfix/policy-spf[1210]: : Policy action=550 Please see http://www.openspf.org/Why?s=mfrom;id=pedro%40dehumanizer.com;ip=204.212.122.254;r=sh.dehumanizer.com

is a good demonstration of the usefulness of SPF. You’ll notice that the mail pretends to come from my own domain, but since I have configured the SPF record for it, Postfix is able to see that the origin server is not authorized to send mail from that domain. And, since my record ends with “-all“, it means that the record is to be “taken seriously”, so the mail is refused then and there (note the “SPF fail”), even before checking RBL lists and the like.

SPF, part 3: configuring Postfix to check SPF records when receiving mail

November 26th, 2009

(for extra fun, read parts 1 and 2 first.)

Now that you know how to configure an SPF record for your domain(s), the natural next step, if you administer an email server, is to start checking SPF records for mails you receive.

Now, spammers are infamous for not respecting rules, not “playing nice”, so, you might ask, what makes me think they’ll set up SPF records for their domains, which would be kind of self-defeating? The obvious answer is that SPF doesn’t depend on the spammers’ collaboration. Since legitimate email senders use SPF to tell the world which servers they use to send email, it prevents others — such as the aforementioned spammers — from faking sender addresses to pretend they’re from those senders / domains.

To put it simply: if there’s an SPF record for the “gmail.com” domain, then you can — and should — reject mail purporting to be from something@gmail.com that doesn’t come from the servers listed in that record. In other words, anyone who fakes a @gmail.com address can’t fool your server, assuming you have it configured to use SPF.

Now, how to do that? There are many ways, of course, depending on the email server you use. The simple recipe below uses postfix-policyd-spf-perl to make Postfix reject mail from domains with properly configured SPF records, when the mail comes from an unauthorized server (that is, one not listed on the record.) This assumes you already have Postfix up and running.

  • install postfix-policyd-spf-perl. In Ubuntu, just do an apt-get install postfix-policyd-spf-perl .
  • add this line to /etc/postfix/main.cf:
    spf-policyd_time_limit = 3600s
  • add the following to /etc/postfix/master.cf:
    policy-spf  unix  -       n       n       -       -       spawn
         user=nobody argv=/usr/bin/policyd-spf

    (change the path of policyd-spf if it’s installed somewhere else; that’s where the Ubuntu package puts it.)

  • in /etc/postfix/main.cf, find the smtpd_recipient_restrictions section, and, immediately after permit_mynetworks (and permit_sasl_authenticated, if you’re using that), add:
    check_policy_service unix:private/policy-spf
  • restart Postfix. Check your logs to see if everything is working properly.

Now, remember when in part 2 I mentioned a lesser-known reason for configuring the SPF record for your domain? It’s this: it’ll stop a lot of incoming spam. These days, a lot of spam email pretends to be from you (that is, it uses your email address as both the “From:” and “To:”), or at least from your domain (e.g. administrator@yourdomain, manager@yourdomain, and so on.) I don’t know why spammers do that, but apparently it works, or else they wouldn’t do it (maybe victims get confused and think the email comes from someone in their company, or maybe webmail services tend — or at least did so once — to trust the user’s own address, I don’t know.) Well, if you have SPF for your domain and your email server checks SPF records for incoming mail, then spam messages such as these will be always rejected, even when other methods of stopping spam (RBL lists, SpamAssassin, etc.) fail. Never again will you receive emails pretending to be from your own address.

What’s this site running?

November 25th, 2009

As an intermission (there’s more to come in the SPF series), here’s what’s changed on my server since, oh, about a year and a half ago:

  • The OS is now Ubuntu Karmic Koala (9.10), and all the server’s running software comes from the standard Ubuntu packages, which means that whatever version is in Karmic1, that’s what I’m running here.
  • I’ve switched, definitely, from Apache to nginx. It’s faster, more efficient, and ridiculously easier to configure (to put it in perspective, in terms of ease of configuration, it’s like OpenBSD’s amazing pf to the unholy abomination that is Linux’s iptables.) nginx works perfectly with the two pieces of software I use the most on my server, WordPress and MyBB, even while using a SEO plugin with the latter, which requires some non-trivial redirect rules.
  • As I don’t trust any ISP’s email server to distribute the mail my server sends (mostly confirmation emails from my forums), and since most ISPs and companies these days block mail sent from dynamic IP addresses, I keep a 256 MB Slicehost slice, with a static IP address and, most importantly, reverse DNS, which I use as a smart host for my home server. The slice doesn’t have a lot of power in terms of CPU (and it’s not meant to), but, as bandwidth is much cheaper in the US than in my poor country, the “small” limit the cheapest slice includes is a lot; I use it for serving static files, mostly for my forums (all images and Javascript files are served from there), and I still have bandwidth to spare.
  • I’ve stopped using a Squid proxy in my home network, and nowadays access the web directly… except for when I indulge in one of my newest weird habits: reading webcomics like this one or this one while having lunch or dinner. At such a time, moving instantly from comic to comic is a must… so I simply re-enable Squid (with more aggressive caching than I’d use for normal browsing; after all, existing comics typically aren’t going to change, are they?), do a nice little wget in my server to download and cache the entire comic, and then enjoy reading the whole of it (in as many meals as it takes) as if it was stored locally…
  • A few changes to my email server’s configuration, mostly related to spam filtering… but I’ve been writing about that, haven’t I? And there’s still more to come. :)
  1. with updates, of course — people who are afraid of installing updates (“but… it might break something!”) are nothing more than mewling weaklings who are utterly incompetent as sysadmins; they should never be allowed within a mile of any server. Even one running Windows. []

SPF, part 2: how to configure SPF for a domain

November 25th, 2009

(this is part 2 of a series. You should read part 1 first, and after this post you should read part 3.)

Suppose you have a domain and send email from it (in fact, even if you don’t, this is still a good idea; more about that later). How to configure SPF for it?

Easy: you simply add a specially formatted TXT record to the domain. (Where? If you have your own public DNS servers, you’ll have to edit the zone file1; else, if your domain (note: the domain, not the web server) is maintained by a registrar or ISP, it should provide for you an administration interface where you are able to add and edit records.)

OpenSPF.org provides a “wizard” to build an SPF TXT record for your domain, and I suggest you try it out. However, I think I can give you an example that is mostly self-explanatory, which is the SPF TXT record for my own dehumanizer.com domain:

v=spf1 a a:mail.dehumanizer.com a:sh.dehumanizer.com a:sh2.dehumanizer.com mx include:netcabo.pt -all

(In fact, that record is kind of overkill; I could remove about half of it and everything would still work. But it’s useful as an example.)

v=spf1 means it’s an SPF record.

a means that “dehumanizer.com” (the host that that name resolves to) is authorized for that domain. I could remove that one, as the following one takes care of it (it’s the same host).

a:mail.dehumanizer.com means that mail.dehumanizer.com is authorized. The same for the following two (sh.dehumanizer.com and sh2.dehumanizer.com; the latter doesn’t exist anymore, so I could remove it.)

mx means that the mx records for the domain are also authorized. They’re currently mail.dehumanizer.com and sh.dehumanizer.com, which were previously taken care of, so I could remove this one as well.

include:netcabo.pt means that whatever is authorized for the netcabo.pt domain is also authorized here. This is from when I used my ISP as a smart host, some time ago; as I no longer do that, I could remove this one too.

Finally, -all means, in effect, that, yes, I’m serious about this, all of the authorized servers for this domain are listed, which means that any other hosts are unauthorized, and emails coming from them with a “From:” of this domain should be refused. You should configure this part only after successfully testing your configuration. Until then, there’s a “soft fail” option, ~all, which means the same, but adds “but don’t take it seriously.”

If you’ve been paying attention, you’ve probably already noticed that I could replace that record with the much simpler:

v=spf1 a:mail.dehumanizer.com a:sh.dehumanizer.com -all

for the exact same results (and I’ll probably do that a bit later today.)

What’s the benefit of this? Well, as you are telling the world “mails from my domain come only from my servers”, the rest of the world will more easily be able to kill mails pretending to be from your domain… while also “respecting” legit mail more. In effect, you tell other servers “here’s how you can tell which mails from my domain are genuine”. Many ISPs and filtering systems (such as SpamAssassin) take advantage of a properly configured SPF record to raise the “trustworthiness” of emails coming from the correct servers (while lowering it, or even simply refusing to accept the mail, for messages coming from a server not listed in the SPF record.)

There’s also another (lesser-known) advantage of using SPF for your domain, when you also receive email to it. But that’s for part 3, where I’ll explain how to configure Postfix to look at SPF records when receiving mail…

  1. note: put the entire record contents between quotes; e.g. for Bind, use dehumanizer.com. IN TXT "v=spf1 a a:mail.dehumanizer.com a:sh.dehumanizer.com a:sh2.dehumanizer.com mx include:netcabo.pt -all" (that’s a single line) []

SPF, part 1: what is SPF, and how to fight spam with it

November 25th, 2009

(Note: this is the first of a series of posts related to email servers and spam. This one is more of a theoretical intro; future posts will delve into the gory details.)

(Later note: here are parts 2 and 3.)

You may have heard of SPF (Sender Policy Framework), but what is it? To put it simply, it’s a way for domain owners to say to the world: “these servers are the only ones that send email from this domain.”

What is the point? Well, if you have been using email for any amount of time, you’ve probably noticed that a lot of spam fakes its sender address (the “From:” field). In fact, you may have found yourself that it’s incredibly easy to do; most SMTP servers simply accept any sender, as long as either 1) the destination address is theirs, or 2) your IP address is on their client list. In other words, a company’s email server accepts mail to the company’s employees, and also allows those employees to use it to send mail to the rest of the world.

SPF, when correctly configured by everyone involved, prevents that.

“Everyone involved” is, of course, an utopic scenario. Still, there is no reason for you not to do your part, whether you are a domain owner who sends email from that domain, the administrator of an email server that receives mail to its users, or both.

More to come…

Back in business

November 25th, 2009

Yes, I haven’t blogged here for a while (a year and a half, in fact). Stuff happens. :) However, I have recently been playing with my home server’s Postfix anti-spam configuration (due to a conversation in a mailing list), and I’ve been feeling the itch to write a couple of technical articles about that… perhaps too technical for my personal blog. Which gave me the excuse to resurrect this one.

So, a few dozen spam comments deleted, a new theme, several software upgrades, and The Tlog (yeah, I know, not the best name ever) is back.

More updates: Ubuntu, WordPress

May 2nd, 2008

Just updated this server to Ubuntu 8.04 (why do they always have these odd version numbers?). As always, it was incredibly difficult: I had to type this entire command:

do-release-upgrade

and even remember to press Enter afterwards! This Linux stuff is really too tough for us mere users…

Also, WordPress 2.5.1 on this blog and a couple others. Incidentally, if you’ve just upgraded to 2.5.1 and suddenly your RSS feed is empty, you were probably hit by this bug. That link includes info for fixing it, too (basically, you have to replace two files with newer development versions).


Creative Commons Attribution-NonCommercial-NoDerivs 2.5 Portugal
This work by Pedro Timóteo is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 2.5 Portugal.