How to Set Up an SMTP Server: A Step-by-Step Guide

How to Set Up an SMTP Server: A Step-by-Step Guide

smtpdiy
Published · Updated

Prepare server

  • sudo hostnamectl set-hostname example.com
  • sudo apt install mailutils postfix
    • Choose "internet site", and type your domain (example.com)

Test postfix

  • mail your-test@yopmail.com -s "Subject"
  • You should receive a mail from debian@example.com

TLS

Generating certificates

  • sudo apt install certbot
  • sudo certbot certonly --standalone --rsa-key-size 4096 --agree-tos --preferred-challenges http -d example.com
    • You might need to kill your running webserver (port 80) to complete the challenge in standalone mode

Giving rights to postfix

  • sudo chown -R root:postfix /etc/letsencrypt/live/example.com

Postfix config

/etc/postfix/main.cf

# TLS parameters
smtpd_tls_cert_file=/etc/letsencrypt/live/example.com/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/example.com/privkey.pem
smtp_use_tls=yes

SPF and DMARC

  • SPF: v=spf1 ip4:<your_ipv4> ~all
  • DMARC: v=DMARC1;p=none;pct=100;rua=yourmail@example.com;sp=none;aspf=r;

DKIM

SMTPD

Enable submission

  • Uncomment submission's lines in /etc/postfix/master.cf
  • change smtpd_sasl_type=dovecot by smtpd_sasl_type=cyrus
  • add -o smtpd_sasl_security_options=noanonymous

Install Cyrus SASL

  • sudo apt install sasl2-bin
  • sudo usermod -aG sasl postfix

Configure SASL

  • /etc/default/saslauthd

    • START=yes
    • MECHANISMS="sasldb"
    • OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"
  • sudo systemctl restart saslauthd

  • systemctl status saslauthd

    • Make sure it is running with the /var/spool arguments

Add a user

  • sudo saslpasswd2 -c -u example.com user

    • The user does not have to be the user you will send mail with, it is just credentials
  • sudo testsaslauthd -u user -p password -f /var/spool/postfix/var/run/saslauthd/mux

    • Add a space before the command to not keep this in your history
    • Note the custom socket path, it is required

Link postfix with Cyrus

/etc/postfix/sasl/smtpd.conf

pwcheck_method: saslauthd
mech_list: PLAIN LOGIN

sudo postfix reload

Final test

  • Use https://www.mail-tester.com/ to make sure everything is working properly
  • Example code (TS) below, can be ran with npx ts-node test.ts
  • Use real data to test your setup or SpamAssassin will not be happy
import nodemailer from 'nodemailer';

const smtpEndpoint = 'example.com';
const port = 587;
const senderAddress = 'My name <my-address@example.com>';
const smtpUsername = 'user';
const smtpPassword = 'pass';

const transport = nodemailer.createTransport({
	host: smtpEndpoint,
	port: port,
	secure: false,
	auth: { user: smtpUsername, pass: smtpPassword }
});

transport.sendMail({
	from: senderAddress,
	to: 'your-mail-tester-addr',
	subject: 'A real subject',
	text: 'A real body'
});

Additional steps for Outlook/Live

Configure virtual aliases

/etc/postfix/main.cf

virtual_alias_maps = hash:/etc/postfix/virtual

/etc/postfix/virtual

@example.com debian

sudo postmap /etc/postfix/virtual

Sign up to sendersupport

Useful to debug

  • tail -f /var/log/mail.info
  • -v in submission parameters (/etc/postfix/master.cf)

Additional resources

Before You Run Your Own SMTP Server

Running an SMTP server is possible, but it is not the easiest path to reliable inbox placement. Mailbox providers judge your domain, IP address, authentication, sending history, complaint rate, bounce rate, and message quality together. A technically working Postfix server can still land in spam if reputation is weak.

Use your own SMTP server when you need control and understand the operational work. For ordinary newsletters, transactional mail, or cold outreach, a reputable email service provider is usually easier to maintain.

If you do self-host, start with these assumptions:

  • You need a dedicated IP with clean reputation.
  • You need reverse DNS that matches your sending hostname.
  • You need SPF, DKIM, and DMARC configured before sending volume.
  • You need TLS certificates and submission authentication.
  • You need bounce handling and suppression.
  • You need logs, monitoring, and rate limits.
  • You need a warm-up plan before sending real volume.

Skipping any of these can make the server appear suspicious even when your mail is legitimate.

DNS Records You Should Confirm

DNS is where most self-hosted SMTP setups fail. At minimum, confirm these records before testing campaigns.

A Record

Your sending hostname should resolve to the server IP.

mail.example.com -> 203.0.113.10

PTR / Reverse DNS

The IP address should resolve back to your hostname. This is configured with your hosting provider, not inside your domain DNS panel.

203.0.113.10 -> mail.example.com

MX Record

If the same server receives mail, the domain needs an MX record. If it only sends mail, an MX record is still useful for abuse, bounce, and postmaster addresses.

example.com MX 10 mail.example.com

SPF

SPF authorizes your server to send for the domain.

v=spf1 ip4:203.0.113.10 -all

Use ~all while testing if needed, but move toward -all once every legitimate sending source is accounted for.

DKIM

DKIM signs outgoing mail. Without DKIM, forwarded mail and modern filtering can become harder to pass. Use a current key length and rotate keys periodically.

DMARC

Start with monitoring, review reports, then move toward enforcement.

v=DMARC1; p=none; rua=mailto:dmarc@example.com

When you understand all legitimate sources, move gradually to p=quarantine or p=reject.

Deliverability Checks After Setup

Once Postfix can send, do not jump straight into campaigns. Test the server like a production system.

  1. Send to a mailbox you control at Gmail, Outlook, Yahoo, and your own domain.
  2. Inspect message headers to confirm SPF, DKIM, and DMARC pass.
  3. Confirm TLS is used.
  4. Check that the From, Return-Path, and DKIM signing domain align.
  5. Send a realistic plain-text message and a realistic HTML message.
  6. Watch /var/log/mail.info for deferrals, rejects, and authentication errors.
  7. Add hard bounces to a suppression list immediately.

You should also set up role addresses such as postmaster@example.com and abuse@example.com. Some providers expect them to exist, and they help with reputation and incident handling.

Warm-Up and Sending Limits

A new SMTP server has no sending reputation. Even if authentication is perfect, high volume from a new IP or domain can trigger filtering.

Start small:

  • Day 1-3: send only a small number of messages to people who expect them.
  • Week 1: monitor bounces, complaints, and replies.
  • Week 2: increase gradually only if results are clean.
  • Ongoing: keep bounce rate low and avoid sudden spikes.

Do not warm up with unverified or purchased lists. Use email verification before any campaign so hard bounces are removed before they hit your reputation.

Security Basics

An SMTP server is a target. If it becomes an open relay or a weakly protected submission server, it can be abused quickly.

Use these safeguards:

  • Require authentication on submission port 587.
  • Disable open relay behavior.
  • Use strong passwords or application-specific credentials.
  • Keep Postfix, SASL, and system packages updated.
  • Restrict admin access with SSH keys.
  • Monitor queue size for unusual spikes.
  • Rate-limit accounts that send unexpectedly.
  • Keep backups of configuration files before major changes.

If your queue suddenly fills with mail you did not send, stop Postfix, investigate credentials, rotate passwords, and check logs before bringing it back online.

When to Use a Sending Provider Instead

Self-hosted SMTP is educational and powerful, but it creates maintenance work. Use a provider when you need:

  • Shared or managed IP reputation
  • Webhooks for bounces and complaints
  • Suppression-list automation
  • Analytics and event tracking
  • Easier DKIM setup
  • Support during deliverability incidents

Use your own server when control matters more than convenience. Either way, the fundamentals stay the same: authenticate the domain, verify the list, send wanted mail, monitor results, and remove bad addresses quickly.

Common Troubleshooting Scenarios

If mail sends but does not arrive, start with logs. Postfix usually tells you whether a message was accepted, deferred, rejected, or stuck in the queue.

Useful commands:

mailq
sudo postqueue -p
sudo tail -f /var/log/mail.info
sudo postfix check

If messages are deferred, read the remote server response. A DNS problem, blocked port, missing PTR record, or poor IP reputation will usually appear in the log message.

If authentication fails, inspect the final message headers in a mailbox you control. Look for spf=pass, dkim=pass, and dmarc=pass. If DKIM fails, check the selector, DNS TXT record, private key path, and whether the signing service is running.

If Gmail or Outlook accepts mail but places it in spam, review reputation and content. Authentication passing is only the baseline. You still need consistent sending volume, low bounces, clean links, and recipients who actually want the message.

If bounces increase after importing a list, stop and verify the list before sending again. SMTP configuration cannot compensate for stale or low-quality data.