rpms/dovecot
5
0
Fork 0
Code Issues Pull Requests Releases Activity

remove unused scripts from the cvs, dsync and doveadm should be enough

Browse Source
This commit is contained in:
Michal Hlavinka 2010-05-17 16:06:56 +00:00
parent c8da1b8e2e
commit 0a8ff62794
4 changed files with 0 additions and 734 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

315
maildir-migration.txt
View File

@ -1,315 +0,0 @@
Replacing UW-IMAP with Dovecot on Red Hat Enterprise Linux 3.
G.R.Keech <rkeech@redhat.com>
2004-11-09
Version 1.0
Summary
The IMAP server in Red Hat Enterprise Linux version 3 (RHEL3) is the
University of Washington (UW) implemenation. UW IMAP uses the mbox
mail storage format which has been found to have significant
scalability and performance issues. This paper describes how to
replace UW IMAP with the Dovecot IMAP server which uses the maildir
mail storage format and, consequently, is significantly faster.
Maildir vs Mbox
Format issues. The mbox format holds email with one file corresponding
to an entire mail folder. Maildir, on the other hand, stores messages
one directory per folder, with each message being a single file. The
mbox format doesn't scale well when many users have large mail folders
which are polled frequently. When the sum of sizes of all the
frequently polled mail greatly exceed the system's RAM, then mail
boxes cannot be effectively cached in RAM. This leads to a situation
where mbox files are being continually re-read with each poll by each
mail client, leading to excessive IO activity and poor performance.
Example. Here is a real example of poor performance using UW IMAP.
The site characteristics were as follows:
* about 116 active mail accounts with users using a mix of both
Outlook express and Outlook;
* Mail clients poll for new mail every five minutes;
* 1GB system RAM in mail server;
* Mail server using Postfix on RHEL3 (kernel 2.4.21-20.EL);
* System has single 3.0GHz P4 processor and SATA disk;
* 112 inboxes totalling 3.3GB (average about 30MB each);
* 8.5GB of filed email in imap folders.
Performance with mbox. When in use the load average on the mail server
was normally between four and six with spikes above 15 not uncommon
and spikes above 20 occasionally. Most of this load was associated
with instances of uw-imap in disk wait state. The user experience in
checking mail was very poor, despite this being a reasonably small
site. When users checked for new mail they frequently encountered
server timeouts. For users with large inboxes, it was not uncommon
for it to take between 60 and 90 seconds just to check for new mail.
Performance with maildir. Once the imap service was changed to
Dovecot, with all the other variables remaining unchanged, the
situation improved dramatically. Load average is very rarely above
0.5 and is usually under 0.2 during working hours. The user
experience was improved significantly. Users no longer
experience timeouts, and the time taken to check for new mail
is usually only two or three seconds and always less than 10 seconds.
Setting up Dovecot
Dovecot. The Dovecot IMAP server will replace UW IMAP in RHEL4. The
Dovecot project is at http://dovecot.procontrol.fi/. It was built for
RHEL3 based on the package from the Beta of RHEL4. The RHEL3 version
used in this migration can be found at
http://people.redhat.com/rkeech/#dovecot.
Operation. Dovecot, unlike uw-imap, runs as a conventional
daemon-based service, ie it is not run from xinetd. When running
A typical process hierarchy when Dovecot is running looks as follows:
dovecot --+--- dovecot-auth
+--- imap
+--- imap-login
One instance of the "imap" process will exist for each mail client
connection. These imap processes execute with the user context of
whichever mail account user is connecting. The main "dovecot" process
runs as root, as does "dovecot-auth". The imap-login processes run
as user "dovecot".
Configuration. Dovecot is configured with /etc/dovecot.conf. The relevant
configuration setting that had to be changed in this case were as follows:
protocols = imap
default_mail_env = maildir:/data/mail2/%u/
User home. In this particular migration, mail account users did not
need shell access and did not need home directories for anything but
mail folder storage. Accordingly it was possible to change the users'
home directories to be the same as the mail spool area. This can be
achieved with (for this example)
usermod -d /data/mail2/<user> <user>
User shell. Mail accounts can function without a user shell. However
if a mail auto-responder (ie "vacation") is used then a shell is necessary.
Under RHEL3 a restricted shell is possible and reduces the security
concerns associated with providing shell access to mail users. For
a restricted version of bash make a sybolic link called "/bin/rbash"
pointing to "/bin/bash", and make /bin/rbash the users' shell.
Postfix delivery. The location of mail delivery by Postfix is set
according to the "mail_spool_directory" parameter. When using uw-imap
this was set to "/var/spool/mail". A trailing slash on the
mail_spool_directory value directs Postfix to use maildir format. The
change to Postfix configuration in this case was done with:
postconf -e "mail_spool_directory = /data/mail2/"
Directory structure. For mail delivery to work with maildir a number
of sub-directories must exist in the user's personal mail spool
directory. If user "fred" has mail delivered to /data/mail2/fred/
then that directory should have subdirectories "cur" "new" and "tmp"
in it. The "new" directory is where incoming mail messages are put.
Other folders structured in subdirectories, eg an imap folder called
"Sent" for user "fred" would be /data/mail2/fred/.Sent, and there would
be a line "Sent" in the file /data/mail2/fred/.subscriptions.
Account creation. In the case where the mail spool area is also the
home directory, then for new accounts to accept maildir mail delivery,
the following directories should be created:
/etc/skel/cur
/etc/skel/tmp
/etc/skel/new
thus when an account is created, the necessary subdirectories are
automatically created in the new user's home for a maildir-style
inbox.
Migrating Mbox folders to Maildirs
Conversion. The tools to convert existing mbox-style mailboxes
and folders to maildir format are not provided with Dovecot or RHEL3.
A tool called perfect_maildir was found on the Net which performs
the conversion for a single folder only. A script was written to
provide for bulk conversion of folders for all users.
Perfect_maildir. The tool lives at http://perfectmaildir.home-dn.net.
Versions prior to 0.3 should not be used. Perfect_maildir is a Perl
script.
Migration plan. The migration to maildir was performed within the
following bounds:
* existing inboxes were in /var/spool/mail
* existing mail folders were under /home/<user>/
* new inboxes and mailfolders were combined under /data/mail2/<user>/
Migration scripts. The scripts prepared to handle the bulk conversion are
"migration-users" and "migrate-folders". migrate-users creates the
necessary new user mail directories under the new base directory, which
in this case was /data/mail2/. In a situation where mail accounts
needed to be setup, without the need to migrate folders then migrate-users
would be sufficient, ie the second script would not be used.
migrate-folders does the main work of the migration by calling perfect_maildir
once for each mbox mailfolder including the users' inbox. migrate-users
must be run before migrate-folders.
Pre-migration. Before migrating, some things to check are:
Will the file system have enough inodes now that the number of files
is much larger? "df -i" will help. Run tune2fs as required to
increase the allowance for inodes.
Will there be a very large number of mail accounts? The maximum
number of subdirectories per directory is 32k. If the number of
accounts does or might exceed this, then change the Dovecot
default_mail_env to suit. See the Dovecot documentation for more
information.
Where do you want dovecot to send its log messages? By default it
uses syslog and sends to the mail facility. This can be changed to
log to a its own log file if it is not convenient to log to maillog.
Migration steps. The actual mail file migration was performed as below
using the scripts from http://people.rehdat.com/rkeech/#maildirmigration
Note, the migration steps described here leave the original mbox inboxes
and mail folders intact. Should the migration fail, then the capacity
to continue using mbox format is retained.
1. Ensure that no new accounts are created until the migration is
complete. Normal email operation can proceed until step X. Do not
change Postfix's mail_spool_directory until step Y.
2. Create a working directory /root/migrate/.
3. Create a list of users whose mailboxes are to be migrated
/root/migrate/userlist-master. The file has one user per line.
4. Create a list of mbox folders (excluding inboxes) to be migrated
/root/migrate/folderlist-master The file has one mbox per line and
excludes the path to the user's home, eg
+-----------------
|fred/Drafts
|fred/private
|fred/Sent
|fred/office
|fred/projectx
|mary/Sent
|mary/private
Check that the folderlist does not contain any folders with characters
likely to be problematic when handled by scripts, ie folders whose names
include quotation marks, comas, pound signs, "&" symbols etc. Any such
folders should have their names changed. The name change in the
folderlist file should reflect the name change on disk. Folder names
with spaces, hyphens, percent signs, and periods are known to be OK.
5. Proceed with a test migration with a small subset of users by
creating /root/migrate/userlist and /root/migrate/folderlist based on
subsets of userlist-master and folderlist-master. The test migration
should be done with accounts that have representative mail in both
inbox and imap folders.
6. Ensure that the new mail base directory is clear. In this example
it means that /data/mail2/ is emtpy.
7. Run migrate-users which should read /root/migrate/userlist and
create the new user mail directories under /data/mail2/. Check that
a directory for each user is created under /data/mail2/.
8. Start dovecot on an alternative port so as not to intefere with the
normal uw-imap. This is done by editing /etc/dovecot.conf with
imap_listen = 1.2.3.4:1043
Obviously change address 1.2.3.4 to correspond to the local server.
Run
service dovecot start
Dovecot should start cleanly.
It is assumed that the "protocols" and "default_mail_env" settings
are already set in /etc/dovecot.conf as described in "Configuration"
above.
9. Run migrate-folders which should reate /root/migrate/folderlist and
populate all the necessary maildirs under each user directory.
This will invoke perfect_maildir.pl as required for each mbox file.
If the mail folders are large the this step could take some time.
10. Test that the files created under /data/mail2/ are visible as
folders via Dovecot by configuring an imap mail client.to point
to port 1043 on the mail server using on of the test migrated accounts.
The imap mail client program should correctly sign on to the
new imap server and the migrated folders should be visible to
the user. The mail client should test that:
a. new folders can be created and deleted.
b. that existing mail from migrated folders is visible and that
it sorts correctly by date
c. test that existing mail can be moved between folders.
d. the mail client should see both an inbox and all other folders
which were in the folderlist.
If step 10 passes (ie folders are visible as expected) then proceed with
full migration per the following steps. Do not proceed further unless
proper and expected mail folder operation is seen in the imap test of
step 10.
11. Prepare new folderlist and userlist files from the respective
master files in the mitrate directory.
12. Remove the test mail folders prepared steps 7 and 9 from the
new base directory (/data/mail2/* in this example).
13. Declare mail as unavailable.
14. Stop incoming mail (service postfix stop) and prevent further
use of imap (chkconfig imap off).
15. Run migrate-users and migrate-folders again.
16. Check a test user on the alternate port like step 10.
If all is OK, then
17. Re-configure dovecot to operate on the normal imap port
by commenting out the imap_listen directive in dovecot.conf.
Re-start dovecot.
18. Check that mail can be read on the normal imap port.
19. Change email delivery into the new folders:
postconf -e "mail_spool_directory = /data/mail2/"
20 Re-start mail
service postfix start
21. Check and double check.
22. Go home.

195
migrate-folders
View File

@ -1,195 +0,0 @@
#!/bin/bash
# author: G.R.Keech <rkeech@redhat.com>
# name: migrate-folders
# date: 2004-10-20
# This script assists in the conversion of mail boxes
# in mbox format to maildir format.
# See also migrate-users.
# Applicability.
#
# This script is intended for the common case on Red Hat systems
# where mail users have mail folders in their home directories
# under /home, and have inboxes in /var/spool/mail/
#==================================================================
# Change the value of the elements in this section as required.
#This is a list of folders, one per line. This does not
#include the inboxes. This might be prepared starting with
#the output of "find /home -type d".
FOLDERLIST=/root/migrate/folderlist
# folder is the existing mbox folder being migrated.
# It is a path under /home.
# eg if oldfolder is fred/personal, then folder is personal,
# user is fred.
#Specify the location of the new location for mail folders.
#This cannot be the same as the old location because it will
#create directory names that contend with existing file names.
NEWBASE=/var/spool/mail2
#The script to convert invidual mail folders to maildir format.
#http://perfectmaildir.home-dn.net/
FOLDERCONVERT=/usr/local/bin/perfect_maildir.pl
#This is a list of users to have their mail folders created.
#One user per line.
#Suggest create with cut -d: -f1 /etc/passwd > ~/migrate/u1
#then remove inappropriate entries by hand.
USERLIST=/root/migrate/userlist
# Detailed migration information is sent to this file
MIGRATELOG=/tmp/foldermigrationlog-$(date -I)
#=================================================================
echo
echo "Have you created the users' mail directories yet? (y/n)"
echo
read ans
if [ "$ans" != "y" ]
then
echo Good Bye.
echo use the migrate-users script first.
exit 0
fi
echo
echo This will copy existing mbox-style mail folders listed
echo in the file $FOLDERLIST. Maildir-style folders will
echo be created under $NEWBASE
echo
echo "Do you want to continue? (y/n)"
read ans
if [ "$ans" != "y" ]
then
echo Good Bye.
exit 0
fi
echo
echo Note: Detailed folder migration information will be sent to $MIGRATELOG
echo
echo Press enter to start
read ans
if [ ! -x ${FOLDERCONVERT} ]
then
echo Error: file ${FOLDERCONVERT} is not available to execute.
exit 1
fi
if [ ! -d ${NEWBASE} ]
then
echo Error: directory $NEWBASE does not exist
exit 1
fi
if [ ! -f "${USERLIST}" ]
then
echo Error: user list file \"$USERLIST\" does not exist.
exit 1
fi
#-----------------------------------------------------------------
echo
echo Testing that the base of the folderlist entries corresponds to usernames
while read oldfolder
do
user="$(dirname "$oldfolder")"
if grep ^${user}: /etc/passwd &> /dev/null
then
echo -n .
else
echo User \"$user\": is bogus.
echo The string \"$user\" from the file \"$FOLDERLIST\" needs to
echo correspond exactly to a username. Edit the file accordingly.
exit 1
fi
done < $FOLDERLIST
echo
echo PASS
echo
nusers=$(wc -l $USERLIST | awk '{ print $1 }' )
n=1
#-----------------------------------------------------------------
# Iterate through user list and migrate folders.
while read user
do
#-----------------------------------------------------------------
# Step 1: Check stuff
if grep ^${user}: /etc/passwd &> /dev/null
then
echo -n "$n / $nusers : User \"$user\" is OK: "
n=$(( $n + 1 ))
echo "User \"$user\"" >> $MIGRATELOG
inbox=/var/spool/mail/${user}
if [ \( ! -f "${inbox}" \) -o \( ! -s "${inbox}" \) ]
then
echo User \"${user}\" has no inbox to convert.
else
#-----------------------------------------------------------------
# Step 2: Migrate user inboxes from /var/spool/mail/.
newdir="${NEWBASE}/${user}/"
$FOLDERCONVERT "$newdir" < "${inbox}" >> $MIGRATELOG 2>&1
chown -R ${user}:mail "${newdir}"
find "$newdir" -type f -exec chmod 600 {} \;
echo -n " inbox "
fi
#-----------------------------------------------------------------
# Step 3: Migrate other mail folders from user home directories.
while read oldfolder
do
folder=$(basename "${oldfolder}")
fuser="$(dirname "$oldfolder")"
if [ "$user" = "$fuser" ]
then
if [ ! -f "/home/${oldfolder}" ]
then
echo Error folder \"${folder}\" does not exist.
break
fi
if [ ! -d ${NEWBASE}/${fuser} ]
then
echo Error ${NEWBASE}/${fuser} does not exist.
break
fi
newdir="${NEWBASE}/${fuser}/.$folder"
mkdir -p "$newdir"/cur
mkdir -p "$newdir"/new
mkdir -p "$newdir"/tmp
chmod -R 770 "${newdir}"
$FOLDERCONVERT "$newdir" < "/home/$oldfolder" >> $MIGRATELOG 2>&1
chown -R ${user}:mail "${newdir}"
#chmod 600 "$newdir/cur/*"
find "$newdir" -type f -exec chmod 600 {} \;
echo "$folder" >> ${NEWBASE}/${fuser}/.subscriptions
chmod 600 ${NEWBASE}/${fuser}/.subscriptions
chown ${fuser}:mail ${NEWBASE}/${fuser}/.subscriptions
echo -n .
fi
done < $FOLDERLIST
echo
else
echo User "$user: is bogus."
fi
done < $USERLIST
echo
echo
echo To make the new base mail directory active, change the
echo mail_spool_directory setting for postfix using
echo postconf -e \"mail_spool_directory = ${NEWBASE}/\"
echo and change Dovecots default_mail_env setting in
echo /etc/dovecot.conf to
echo default_mail_env = maildir:${NEWBASE}/%u
echo

77
migrate-users
View File

@ -1,77 +0,0 @@
#!/bin/bash
# file: migrate-users
# author: Richard Keech <rkeech@redhat.com>
# This script assists in the conversion of mail boxes
# in mbox format to maildir format.
# See also migrage-folders.
#This is a list of users to have their mail folders created.
#One user per line.
#Suggest create with cut -d: -f1 /etc/passwd > ~/migrate/u1
#then remove inappropriate entries by hand.
USERLIST=/root/migrate/userlist
#Specify the location of the new location for mail folders.
#This cannot be the same as the old location because it will
#create directory names that contend with existing file names.
NEWBASE=/var/spool/mail2/
echo this will create user mail folders in $NEWBASE from
echo the list of users in $USERLIST.
echo
echo "Do you want to continue? (y/n)"
read ans
if [ "$ans" != "y" ]
then
echo Good Bye.
exit 0
fi
if [ ! -f "$USERLIST" ]
then
echo Error: user list file \"$USERLIST\" does not exist.
exit 1
fi
if [ ! -d "$NEWBASE" ]
then
echo Error: new base directory \"$NEWBASE\" does not exist.
exit 1
fi
while read user
do
if grep ^${user}: /etc/passwd &> /dev/null
then
echo User \"$user\" is OK.
else
echo User \"$user\": is bogus.
exit 1
fi
mkdir ${NEWBASE}/$user
newdir="${NEWBASE}/${user}/"
mkdir -p "$newdir"/cur
mkdir -p "$newdir"/new
mkdir -p "$newdir"/tmp
chmod -R 770 "${newdir}"
chown -R ${user}:mail "$newdir"
done < $USERLIST
echo
echo New mail directories have been created under $NEWBASE
echo
echo If required, prepare a list of existing mbox folders
echo as /root/migrate/folderlist and run migrate-folders.
echo
echo To make the new base mail directory active, change the
echo mail_spool_directory setting for postfix using
echo postconf -e \"mail_spool_directory = ${NEWBASE}/\"
echo and change Dovecots default_mail_env setting in
echo /etc/dovecot.conf to
echo default_mail_env = maildir:${NEWBASE}/%u
echo
echo If you want to migrate existing mail folders then defer
echo the dovecot and postfix changes until the folder migration
echo is complete.

147
perfect_maildir.pl
View File

@ -1,147 +0,0 @@
#!/usr/bin/perl -w
# "Simple but Perfect" mbox to Maildir converter v0.3
# Copyright (C) 2001-2003 Philip Mak <pmak@aaanime.net>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
use strict;
use Date::Parse qw( str2time );
#### Settings
# This will be used to set the file time (needed for courier-imap and some others)
# $datestyle = "date": extract date from the "Date: " header
# $datestyle = "from": extract date from the "From " mbox header
my $datestyle = "from";
# Use maildir++ format (append the message size to the filename)
my $maildirplus = 0;
####
# Get the hostname
my $hostname = `hostname`;
chomp ($hostname);
# check for valid arguments
my ($maildir) = @ARGV;
if (!$maildir) {
print STDERR "Usage: perfect_maildir ~/Maildir < mbox\n";
exit 1;
} elsif (! -d $maildir) {
print STDERR "Cannot open $maildir\n";
exit 1;
}
# check for writable maildir
unless (-w "$maildir/cur") {
print STDERR "Cannot write to $maildir/cur\n";
exit 1;
}
unless (-w "$maildir/new") {
print STDERR "Cannot write to $maildir/new\n";
exit 1;
}
my $num = 0;
my $time = time;
my $date;
my $delivered_time;
repeat:
# read header
my $headers = '';
my $flags = '';
my $subject = '';
while (my $line = <STDIN>) {
# detect end of headers
last if $line eq "\n";
if ($datestyle eq "from") {
# Get date from the "From " line (this should appears here for the first message only)
$date = $1 if $line =~ /^From [^ ^\t]+[ \t]+(.{24})/;
} elsif ($datestyle eq "date") {
# Get date from the "Date: " header
$date = $1 if $line =~ /^Date: (.*)$/;
}
# strip "From" line from header
$headers .= $line unless $line =~ /^From ./;
# detect flags
$flags .= $1 if $line =~ /^Status: ([A-Z]+)/;
$flags .= $1 if $line =~ /^X-Status: ([A-Z]+)/;
$subject = $1 if $line =~ /^Subject: (.*)$/;
}
$num++;
if ($datestyle =~ /(from|date)/) {
$delivered_time = str2time("$date");
} else {
$delivered_time = $time;
}
# open output file
my $file;
if ($flags =~ /O/) {
$file = sprintf( "%s%05d%s", "$maildir/cur/$delivered_time.", $num, ".$hostname" );
my $extra = '';
$extra .= 'F' if $flags =~ /F/; # flagged
$extra .= 'R' if $flags =~ /A/; # replied
$extra .= 'S' if (($flags =~ /R/) || ($flags =~ /O/)); # seen
$extra .= 'T' if $flags =~ /D/; # trashed
$file .= ":2,$extra" if $extra;
} else {
$file = sprintf( "%s%05d%s", "$maildir/new/$delivered_time.", $num, ".$hostname" );
}
# filter out the "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA" message or the message doesn't exists
if (($num == 1 and $subject eq "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA") || (!$headers)) {
$file = '/dev/null';
$num--;
}
open(FILE, ">$file");
print FILE "$headers\n";
while (my $line = <STDIN>) {
if ($datestyle eq "from") {
# Get date from the "From " line (this should appears here for the first message only)
$date = $1 if $line =~ /^From [^ ^\t]+[ \t]+(.{24})/;
}
# End of current message
last if ($line =~ /^From ./);
# unescape "From"
$line =~ s/^>From (.)/From $1/;
print FILE $line;
}
close(FILE);
utime( $time, $delivered_time, $file ) if ($datestyle =~ /(from|date)/);
if ($maildirplus) {
my $size = -s $file;
my $mdplusfile = $file;
$mdplusfile =~ s/\.$hostname/.$hostname,S=$size/;
rename $file,$mdplusfile;
}
goto repeat unless eof(STDIN);
my $elapsed = time - $time;
print "Inserted $num messages into maildir $maildir in $elapsed seconds\n";