MTA-STS
MTA-STS lets a domain declare that mail sent to it must use TLS, closing a gap that opportunistic encryption leaves open.
The problem
SMTP encryption is opportunistic by default: a sending server offers STARTTLS, and falls back to plaintext if it is not available. An attacker who can strip the STARTTLS offer can force the connection into the clear.
MTA-STS, specified in RFC 8461, lets a receiving domain publish a policy saying that TLS is required, so a sender that supports MTA-STS refuses to fall back.
Two pieces: a record and a file
The DNS half is a TXT record at _mta-sts.example.com carrying a policy version and an id. Changing the id is how a sender learns the policy has been updated.
The policy itself is not in DNS. It is served over HTTPS from mta-sts.example.com at a fixed path, and lists the permitted MX hostnames and a mode of enforce, testing or none.
Because the policy is fetched over HTTPS, its authenticity rests on the web certificate rather than on DNSSEC. That is the design choice that makes MTA-STS deployable without a signed zone.
MTA-STS and DANE
DANE solves the same problem by publishing certificate information in DNS, which requires DNSSEC to be trustworthy. MTA-STS leans on the web PKI instead.
They are not mutually exclusive, and a domain may publish both. Which one a sender uses depends on what that sender implements.