- Published at
- Tuesday 22nd November, 2016
DKIM is “DomainKeys Identified Mail”, one of a
number of technologies designed to let your email flow freely to its
intended recipients, even while the world is full of spam. It’s part
of something called DMARC, which is an overall
statement of how you’re using various email protection technologies.
DMARC is getting increasing amounts of attention these days,
within governments and in the private sector. If you implement it
correctly, your email is more likely to be delivered smoothly to
GMail, Hotmail and AOL among others. This year the UK government
moved to using strong DMARC policies,
and it’s gradually becoming an accepted part of best practice.
This brief article should give an up to date view of how to set things
up if you’re using exim4 on Debian (exim 4.84 on Debian 8 at time of
writing). It assumes that you’re comfortable administrating a
Debian machine, and in particular that you have at least a basic
understanding of exim.
Some background on DKIM
DKIM works in the following way:
The server that sends the original email (maybe a GMail server, or
maybe your company’s mail machine) adds a signature to the
outbound email, using a private cryptographic key. The signature
also specifies a selector, which identifies the key being used.
(This allows you to have several selectors, and hence several keys,
for different servers.)
Any other server that receives that email can verify the
signature. It does this by looking in your domain’s DNS records
for the public part of the cryptographic key. If you send emails
from me@mydomain.com, then a server receiving your email can look
up the signature’s selector within _domainkey.mydomain.com to
find the public key it needs to verify the signature.
Webmail systems like GMail take care of DKIM for you, providing you
either use their interface directly, or send your mail out through
them. But if you run your own email server, you need to set things up
yourself.
DKIM signing for your outgoing email
This is based on an article by Steve Kemp on
the Debian Administration
website. I’ve simplified the instructions slightly, and corrected what
appears to be a single error.
Install the right version of exim4
Debian has two versions of exim: “light” and “heavy”. You’re probably
using light, which is the default, but you need heavy for DKIM signing
support. You should be able to do the following without any
interruptions to email service, because exim configuration is separate
from the light and heavy packages:
apt-get install exim4-daemon-heavy
/etc/init.d/exim4 restart
apt-get purge exim4-daemon-light
Generating the keys
We’ll store our cryptographic keys in /etc/exim4/dkim. Except for
the first two lines, you can just copy the following straight into a
root shell (or make a script out of them if you’re going to be doing
this a lot).
DOMAIN you just want to set to the domain part of your email
address. SELECTOR will identify your key in the DNS. You’ll want to
be able to change keys in future, and one way of making this easy is
to use dates.
export DOMAIN=mydomain.com # change this to your own domain!
export SELECTOR=20161122 # a date is good for rotating keys later
cd /etc/exim4
# Ensure the dkim directory exists
test -d dkim || (umask 0027; mkdir dkim; chgrp Debian-exim dkim)
# Generate the private key
OLD_UMASK=`umask`; umask 0037
openssl genrsa -out $DOMAIN-private.pem 1024 -outform PEM
# Extract the public key
openssl rsa -in $DOMAIN-private.pem -out $DOMAIN.pem -pubout -outform PEM
umask $OLD_UMASK; unset OLD_UMASK
chgrp Debian-exim $DOMAIN{-private,}.pem
# Format the public key for DNS use
echo -n 'v=DKIM1; t=y; k=rsa; p=';\
sed '$ { x; s/\n//g; p; }; 1 d; { H; d; }' $DOMAIN.pem
The very last line reads the public key file and converts it into the
right format to use as a DNS record. Apart from some stuff at the
start (including t=y, which is “testing mode”, which you’ll want to
remove once you’ve got everything working properly), this involves
stripping the comment lines at the start and end, and then squashing
everything else onto one line, which we do using sed(1).
Note at this point we haven’t placed the key files into
/etc/exim4/dkim – they’re still in /etc/exim4. This is because we
want to set up the DNS record (so others can verify our signatures)
before we start signing using our new private key.
A note about key length: here, we’ve created a 1024 bit private
key. You can create a longer one, say 2048 bits, but some DNS hosts
have difficulty with the length of record generated
(although
Google have documented a way round that). I
hope that 1024 will continue for some time to be sufficient to avoid
compromise, but don’t take my word for it: check for the latest advice
you can find, and if in doubt be more cautious. If you need to split a
key across multiple DNS records, ensure you test this carefully.
Setting up the DNS record
Now you need to create a DNS record to contain the public key. The
last line of the above commands printed out the value of that DNS
record, which will start “v=DKIM1; t=y; ...”. The name will be SELECTOR._domainkey.mydomain.com. The
type will be TXT (which these days is mostly a general-purpose type
for DNS uses that don’t have their own type).
In a lot of DNS control panels, and in most DNS server zone files, you
don’t need to include your domain itself, so you’ll just need to add
something like:
SELECTOR._domainkey TXT v=DKIM1; t=y; ... from above
Configuring exim
Exim on Debian already has suitable configuration to create DKIM
signatures; it just needs configuring appropriately. We can do this by
adding lines to (or creating) the /etc/exim4/exim4.conf.localmacros
file, which is read in immediately before the main configuration
file. We want the following:
# DNS selector for our key.
DKIM_SELECTOR = 20161122
# Get the domain from the outgoing mail.
DKIM_DOMAIN = ${lc:${domain:$h_from:}}
# The file is based on the outgoing domain-name in the from-header.
DKIM_FILE = /etc/exim4/dkim/DKIM_DOMAIN-private.pem
# If key exists then use it, if not don't.
DKIM_PRIVATE_KEY = ${if exists{DKIM_FILE}{DKIM_FILE}{0}}
Once you’ve added these lines, you should restart exim (service exim4
restart or similar).
There are three important macros here (the fourth, DKIM_FILE, is
just to avoid duplication and potential errors in configuration):
DKIM_SELECTOR is the selector we chose earlier.
DKIM_DOMAIN sets the domain we’re signing for (which we extract
from the email’s From: header, and lowercase it).
DKIM_PRIVATE_KEY is either 0 (don’t sign) or points to a
private key file. (It can also contain the key as a literal string,
but we’re not going to use that.)
So this configuration says, simply: if there’s a file
/etc/exim4/dkim/<DOMAIN>-private.pem for the domain I’m sending
from, then DKIM sign that email using the key in that file and the
given selector.
Note that this can be different for every email. If you know that
you’ll only ever send email from one domain for this server, then you
can simplify this configuration even further. However, in practice I’ve
always run mail servers for multiple domains, and it’s easy enough to
set this up from the beginning.
All the domains you send from that support DKIM will use the same
selector, but you could replace that with a lookup to be able to vary
things if you wanted. (Similarly, you could include the
DKIM_SELECTOR in the DKIM_FILE, which could make rotating keys
easier in future.)
And enabling DKIM signatures
All the configuration is in place, but the key files aren’t in
/etc/exim4/dkim, so let’s move them into place.
mv /etc/exim4/$DOMAIN*.pem /etc/exim4/dkim
Checking this works
mail-tester.com provides a simple way
to test things like DKIM setup. Go to the site, and it’ll give you an
email address. Send an email to it from your server, and it’ll show
you a useful report on what it saw. DKIM is in the “authenticated”
section, along with SPF, DMARC and some other details.
If this didn’t work, then you need to go digging. The first thing to
look for is in the exim mainlog (/var/log/exim4/mainlog). If you
have a line like this:
2016-11-21 18:21:27 1c8tDb-0003IV-NR DKIM: signing failed (RC -101)
then you have some work to do to figure out what went wrong. (-101
is PDKIM_ERR_RSA_PRIVKEY, which you can chase through the code to
find that it means exim_rsa_signing_int() returned an error, but
that’s not helpful on its own.)
What you want to do is to raise exim’s logging verbosity. You can do
this with another line in the local macros file:
MAIN_LOG_SELECTOR = +all
This macro is used as the default for log_selector,
which
Exim documents in a detailed if largely unhelpful fashion. +all
should throw a lot more detail into your mainlog that will help
you figure out what’s going on.
Rotating your keys
You are advised to “rotate” your DKIM keys (ie generate new ones, get
them into the DNS and then start using them for signing) regularly; I
suggest either every three or six months. (Three months conveniently
matches the refresh period for Let’s Encrypt
certificates.)
In order to do that, you basically do the same thing as above: first
you create the new keys, and add the public key as a new DNS record
using a new selector. Then you want to wait long enough for that DNS
record to appear (which depends on how your DNS hosting works, but
usually a few minutes is enough for a completely new record). Then you
can move the new key files into place and flip the selector in your
exim configuration (don’t forget to restart exim).
This is the point where putting the selector into the filename can
help; you can move the files into place first, then flip the selector
and restart exim to start using the new keys.
What next?
You’ll want to drop the t=y part of your DNS record to move out of
testing mode. It’s unclear what different providers actually do in
testing mode; some accounts suggest that they do full DKIM processing,
but others suggest that they verify the DKIM signature but then won’t
act on it. Removing the testing flag once you’re happy everything is
working smoothly is in any case a good idea.
Once you have DKIM set up, you probably want to add in SPF. Once
that’s there, you can tie them together into a public policy using
DMARC. Google has a support article on adding a DMARC record, including advice on gradual rollouts.
If the steps in this article don’t work for you, please get in touch
so I can update it!
- Published at
- Sunday 5th June, 2016
I’m a mentor for Xapian within Google Summer
of Code again this
year, and since we’re a git-backed project that means introducing
people to the concepts of git, something that is difficult partly
because git is complex, partly because a lot of the documentation out
there is dense and can be confusing, and partly because people insist
on using git in ways I consider unwise and then telling other people
to do the same.
Two of the key concepts in git are branches and remotes. There are
many descriptions of them online; this is mine.
remotes are repositories other than the one you’re using, and to
which you have access (via a URL and probably some authentication)
remotes have names; you probably have one called “origin”, which
will be wherever you cloned your repository from – throughout this
article I assume that collaborators all push their changes back to
this remote, and fetch others’ changes from there
branches are, somewhat confusingly, pointers to particular commits
in your repository; they are either local branches or remote
(tracking) branches
local branches are ones that you can work on
remote tracking branches are branches in your repository which
mirror local branches in remotes you’re working with
remote tracking branches don’t automatically update when someone
changes the remote; you have to tell them to update, and you do so
using git fetch <remote>, which pulls down the commits you don’t
have, and adjusts the remote tracking branches to point to the
right place
at that point your repository has commits from other people, but
they aren’t yet incorporated into the code you see on your local
branch
you can see which commits are behind which branches using git
show-branch
you can then incorporate commits from remote tracking branches into
your local branch using a range of options; here I’ll talk about
git merge and git rebase, because they both play well in
collaborative environments
For the rest of this article I’m only going to consider the common
case of multiple people collaborating to make a single stream of
releases (whether that’s open source software tagged and packaged, or
perhaps commercial software that’s deployed to a company’s
infrastructure, like a webapp). I also won’t consider what happens
when merges or rebases fail and need manual assistance, as that’s a
more complex topic.
Getting work from others
One of the key things you need to be able to do in collaborative
development is to accept in changes that other people have made while
you were working on your own changes. In git terms, this means that
there’s a remote that contains some commits that you don’t have yet,
and a local branch (in the repository you’re working with) that
probably contains commits that the remote doesn’t have yet either.
First you need to get those commits into your repository:
$ git fetch origin
remote: Counting objects: 48, done.
remote: Total 48 (delta 30), reused 30 (delta 30), pack-reused 18
Unpacking objects: 100% (48/48), done.
From git://github.com/xapian/xapian
9d2c1f7..91aac9f master -> origin/master
The details don’t matter so much as that if there are no new commits
for you from the remote, there won’t be any output at all.
Note that some git guides suggest using git pull here. When working
with a lot of other people, that is risky, because it doesn’t give you
a chance to review what they’ve been working on before accepting it
into your local branch.
Say you have a situation that looks a little like this:
[1] -- [2] -- [3] -- [4] <--- HEAD of master
\
\-- [5] -- [6] -- [7] <--- HEAD of origin/master
(The numbers are just so I can talk about individual commits
clearly. They actually all have hashes to identify them.)
What the above would mean is that you’ve add two commits on your local
branch master, and origin/master (ie the master branch on the origin
remote) has three commits that aren’t in your local branch.
You can see what state you’re actually in using git show-branch. The
output is vertical instead of horizontal, but contains the same
information as above:
$ git show-branch origin/master master
! [origin/master] 7 message
* [master] 4 message
--
* [master] 4 message
* [master^] 3 message
+ [origin/master] 7 message
+ [origin/master^] 6 message
+ [origin/master~2] 5 message
+* [origin/master~3] 2 message
Each column on the left represents one of the branches you give to the
command, in order. The top bit, above the line of hyphens, gives a
summary of which commit each branch is at, and the bit below shows you
the relationship between the commits behind the various branches. The
things inside [] tell you how to address the commits if you need to;
after them come the commit messages. (The stars * show you which
branch you currently have checked out.)
From this it’s fairly easy to see that your local branch master has
two commits that aren’t in origin/master, and origin/master has three
commits that aren’t in your local branch.
Incorporating work from others
So now you have commits from other people, and additionally you know
that your master branch and the remote tracking branch origin/master
have diverged from a common past.
There are two ways of incorporating that other work into your branch:
merging and rebasing. Which to use depends partly on the conventions
of the project you’re working on (some like to have a “linear”
history, which means using rebase; some prefer to preserve the
branching and merging patterns, which means using merge). We’ll look
at merge first, even though a common thing to be asked to do to a pull
request on github is to “rebase on top of master” or similar.
Merging to incorporate others’ work
Merging leaves two different chains of commits intact, and creates a
merge commit to bind the changes together. If you merge the changes
from origin/master in the above example into your local master branch,
you’ll end up with something that looks like this:
[1] -- [2] -- [3] -- [4] --------- [8] <--- HEAD of master
\ /
\-- [5] -- [6] -- [7] --/
You do it using git merge:
$ git merge origin/master
Updating 9d2c1f7..91aac9f
Fast-forward
.travis.yml | 26 ++++++++++++++++++++++++++
bootstrap | 10 ++++++++--
2 files changed, 34 insertions(+), 2 deletions(-)
create mode 100644 .travis.yml
It will list all the changes in the remote tracking branch which were
incorporated into your branch.
Rebasing to incorporate others’ work
What we’re doing here is to take your changes since your local
branch and remote tracking branch diverged and move them onto the
current position of the remote tracking branch. For the example above
you’d end up with something that looks like this:
[1] -- [2] -- [5] -- [6] -- [7] -- [3'] -- [4'] <--- new HEAD of master
Note that commits [3] and [4] have become [3'] and [4'] –
they’re actually recreated (meaning their hash will change), which is
important as we’ll see in a minute.
You do this as follows:
$ git rebase origin/master
First, rewinding head to replay your work on top of it...
Applying: 3 message
Applying: 4 message.
Some caution around rebasing
Rebasing is incredibly powerful, and some people get trigger happy and
use it perhaps more often than they should. The problem is that, as
noted above, the commits you rebase are recreated; this means that if
anyone had your commits already and you rebase those commits, you’ll
cause difficulties for those other people. In particular this can
happen while using pull requests on github.
A good rule of thumb is:
you can rebase at any time up until the point when you submit code
for review (either at the point you open the pull request, or the
point where you ask people to look at it)
from then on, you shouldn’t rebase until everyone has finished
reviewing the code, you have made changes based on those comments,
and they have checked those changes to ensure their concerns have
been addressed; if someone suggests a change which you then make,
but you rebase in the process, it can be difficult for them to see
what’s happened
when making changes based on pull request comments, you can use
git commit --fixup <earlier commit> to quickly make a commit with
a message that will be easy to flatten into the earlier commit just
before merging the pull request
at the end of review, before a pull request is merged, you can do a
final rebase (a lot of projects have a process where someone will
explicitly prompt that this is the time to do so); that allows you
both to ensure you’re properly integrated with the latest upstream
code and to collapse “fixup” commits into the right place
Rebasing during pull requests is discussed in this Thoughtbot
article.
In summary
Most of the time, your cycle of work is going to look like this:
git add -p to add changes to the git stage
git commit -v to create commits out of those changes
git fetch to get others’ recent changes
git show-branch to see what those changes are
git merge or git rebase to incorporate those changes
Following that you can use git push and pull requests, or whatever
other approach you need to do to start the review process ahead of
getting your changes applied.
