Backend Interview: How Would You Diagnose SMTP TLS Failures with TLS-RPT?

Question and scenario

You operate a multi-MX receiving domain across several providers. After enabling MTA-STS, some senders report TLS handshake failures. Explain how you would use SMTP TLS Reporting (TLS-RPT) to observe, diagnose, and safely roll out the fix.

What the interviewer is testing

• Distinguishing telemetry (TLS-RPT) from enforcement (MTA-STS and DANE).
• Connecting DNS, certificates, STARTTLS, HTTPS policy files, and backup MX paths.
• Using staged rollout, metrics, alerts, and rollback to protect deliverability.

Clarifying questions before answering

Confirm the domain's MX set, whether MTA-STS or DANE is enabled, the TLS-RPT destination, the report window, and whether failures are global or concentrated in one sender or backup MX. Clarify whether the goal is to detect plaintext downgrade, explain delivery failures, or validate an enforcement rollout.

30-second answer framework

TLS-RPT, defined by RFC 8460, is telemetry; it does not enforce encryption. I would publish an smtp.tls TXT record, collect aggregate reports, and group failures by policy, sender, MX, and reason. Then I would validate STARTTLS, certificate identity and chain, DNS, and the MTA-STS HTTPS file for every MX. After fixing issues, I would observe in testing mode before enforcement and keep a tested rollback path.

Step-by-step deep answer

1. Publish a record such as v=TLSRPTv1; rua=mailto:tls-reports@example.org; authorize the destination and verify parsing, retention, and redaction.
2. Parse RFC 8460 JSON aggregates for policy, sending MTA, MX, successes, and failure reasons. Cluster by time, provider, and backup MX instead of relying on one failure-rate number.
3. For every MX, verify DNS, port 25, STARTTLS, certificate chain, and name. Compare sender observations with mail logs and an explicit SMTP TLS probe.
4. For MTA-STS, check _mta-sts, the HTTPS /.well-known/mta-sts.txt file, its mode, MX patterns, and caching. For DANE, check DNSSEC validation and TLSA alignment with the certificate.
5. Test every MX, stale record, and failover path in testing mode before enforce. Alert on a rising failure rate or a critical provider/path.
6. Re-test with the same matrix after fixing certificates, policy files, DNSSEC, or provider configuration. If delivery breaks, lower the MTA-STS mode or remove the bad policy entry while keeping TLS-RPT collection.

dig TXT _smtp._tls.example.org
dig TXT _mta-sts.example.org
curl https://mta-sts.example.org/.well-known/mta-sts.txt
openssl s_client -starttls smtp -connect mx1.example.org:25 -servername mx1.example.org

High-quality sample answer

I treat TLS-RPT as telemetry, not as the encryption switch. First I publish smtp.tls and send reports to a controlled destination, then build a baseline by sender, target MX, policy, and failure reason. For each reported path I check STARTTLS, certificate chain and name, MTA-STS TXT and HTTPS policy, or DANE DNSSEC/TLSA; I include backup MXs and stale TTLs. After remediation I stay in testing mode through a complete report window and failover test before enforce. If delivery fails, I roll back the policy mode while retaining reports, rather than silently accepting plaintext. This separates observation, repair, and enforcement into reversible stages.

Common errors

• Treating TLS-RPT as TLS enforcement instead of reporting.
• Checking only the primary MX and missing backup, stale-DNS, or provider-specific paths.
• Checking certificate expiry but not SAN, chain, TLSA, or the policy file's mx patterns.
• Switching to enforce without a full report window, thresholds, and rollback.
• Interpreting no reports as no failures without confirming sender coverage and endpoint authorization.

Follow-up questions and responses

What is the boundary between TLS-RPT and MTA-STS?

TLS-RPT aggregates evidence about negotiation outcomes. MTA-STS uses an HTTPS policy to require senders to validate TLS. They work together: reporting validates an enforcement rollout.

A report says certificate-mismatch. What do you check first?

Reproduce the handshake against the reported MX, inspect SAN, SNI, chain, and DNS, then check whether a load balancer or backup node still serves an old certificate.

Why does DANE require special attention to DNSSEC?

TLSA trust depends on DNSSEC validation. Certificate rotation must update TLSA atomically; otherwise DANE-capable senders can reject the connection and report the failure.

How do you roll back after enforcement?

Keep versioned DNS and policy files, lower MTA-STS from enforce to testing or remove the faulty entry, confirm delivery and reports recover, then fix the root cause while TLS-RPT continues collecting.