Is a Response in Challenge-Response anti-spam systems necessary?

[David Segall]

There have been a few posts recently which have advocated a
"Challenge-Response" system for minimising spam. The idea is to do
some preliminary filtering and, if you still cannot decide whether an
email is acceptable, you send an email requesting the user to respond.
If there is no response, you classify the email as spam.

You could save the sender some trouble by just replying with a "Thank
you for your email I will attend to it ASAP" message instead of
demanding a response. You would treat the original message as valid if
your reply did not bounce. This would require a little extra work to
identify bounce replies as opposed to, for example, "I'm on holidays"
replies.

The result would be more spam than a system requiring a response but,
given the preliminary filtering, would that be significant? It would
have the advantage, particularly for users who depend on incoming
email, that it would not discourage genuine email correspondence.

[Malcolm Dew-Jones]

David Segall (***@nowhere.net) wrote:
: There have been a few posts recently which have advocated a
: "Challenge-Response" system for minimising spam. The idea is to do
: some preliminary filtering and, if you still cannot decide whether an
: email is acceptable, you send an email requesting the user to respond.
: If there is no response, you classify the email as spam.

: You could save the sender some trouble by just replying with a "Thank
: you for your email I will attend to it ASAP" message instead of
: demanding a response. You would treat the original message as valid if
: your reply did not bounce. This would require a little extra work to
: identify bounce replies as opposed to, for example, "I'm on holidays"
: replies.

: The result would be more spam than a system requiring a response but,
: given the preliminary filtering, would that be significant? It would
: have the advantage, particularly for users who depend on incoming
: email, that it would not discourage genuine email correspondence.

After you've filtered the mail, then why respond to any of it at all
unless you want to communicate with the person?

Why do you need to confirm anything?

If you think you wish to communicate with them then you're going to have
to send them a message one way or another eventually, so why send anything
other than a real reply. That will quickly reveal if they are who they
say they are, and if it's a spammer, will give away nothing more about
yourself than a challenge would anyway, and if it's a fake address then
what else can you do? At least you're only doing this when you had a
genuine desire to talk to the purported sender.

And if you don't wish to communicate then why should you care whether they
are real. Simply ignore them.

I don't understand how I am going to be more interested in "hot deals on
hardware" just because the person promoting them is a real person?
Either I'm interested enough to reply at all, or I ain't interested at
all.

$0.02

[David Segall]

Post by Malcolm Dew-Jones
: There have been a few posts recently which have advocated a
: "Challenge-Response" system for minimising spam. The idea is to do
: some preliminary filtering and, if you still cannot decide whether an
: email is acceptable, you send an email requesting the user to respond.
: If there is no response, you classify the email as spam.
: You could save the sender some trouble by just replying with a "Thank
: you for your email I will attend to it ASAP" message instead of
: demanding a response. You would treat the original message as valid if
: your reply did not bounce. This would require a little extra work to
: identify bounce replies as opposed to, for example, "I'm on holidays"
: replies.
: The result would be more spam than a system requiring a response but,
: given the preliminary filtering, would that be significant? It would
: have the advantage, particularly for users who depend on incoming
: email, that it would not discourage genuine email correspondence.
After you've filtered the mail, then why respond to any of it at all
unless you want to communicate with the person?

Because my current filtering is not good enough to exclude all spam
without excluding people I want to hear from. I want to live in a
virtual world where all the messages I see are welcome and all the
messages I reject were unwanted.

Post by Malcolm Dew-Jones
Why do you need to confirm anything?
If you think you wish to communicate with them then you're going to have
to send them a message one way or another eventually, so why send anything
other than a real reply. That will quickly reveal if they are who they
say they are, and if it's a spammer, will give away nothing more about
yourself than a challenge would anyway, and if it's a fake address then
what else can you do? At least you're only doing this when you had a
genuine desire to talk to the purported sender.
And if you don't wish to communicate then why should you care whether they
are real. Simply ignore them.
I don't understand how I am going to be more interested in "hot deals on
hardware" just because the person promoting them is a real person?
Either I'm interested enough to reply at all, or I ain't interested at
all.
$0.02

[Alan Clifford]

On Sun, 14 Dec 2003, David Segall wrote:

DS>
DS> You could save the sender some trouble by just replying with a "Thank
DS> you for your email I will attend to it ASAP" message instead of
DS> demanding a response. You would treat the original message as valid if
DS> your reply did not bounce. This would require a little extra work to
DS> identify bounce replies as opposed to, for example, "I'm on holidays"
DS> replies.
DS>

If you were sent spam with a forged "from:" being the address I use in
mailing lists, your response would not bounce, it would just disappear.
You would have no bounce message to analyse and would conclude the
original message was legitimate.

Alan

( If replying by mail, please note that all "sardines" are canned.
There is also a password autoresponder but, unless this a very
old message, a "tuna" will swim right through. )

[David Segall]

Post by Alan Clifford
DS>
DS> You could save the sender some trouble by just replying with a "Thank
DS> you for your email I will attend to it ASAP" message instead of
DS> demanding a response. You would treat the original message as valid if
DS> your reply did not bounce. This would require a little extra work to
DS> identify bounce replies as opposed to, for example, "I'm on holidays"
DS> replies.
DS>
If you were sent spam with a forged "from:" being the address I use in
mailing lists, your response would not bounce, it would just disappear.
You would have no bounce message to analyse and would conclude the
original message was legitimate.

Is there a better way to check your from address? I assume a simple
check for an MX record would indicate that the address is valid. How
would your server respond to a VRFY command?

I'm prepared to accept an increase in spam compared to a C-R system in
order to avoid irritating those that are making genuine enquiries. My
post was an attempt to quantify the increase.

Post by Alan Clifford
Alan
( If replying by mail, please note that all "sardines" are canned.
There is also a password autoresponder but, unless this a very
old message, a "tuna" will swim right through. )

[wayne]

Post by David Segall
You could save the sender some trouble by just replying with a "Thank
you for your email I will attend to it ASAP" message instead of
demanding a response. You would treat the original message as valid if
your reply did not bounce.

Even better would be to use something like Exim's "call back" system
where this is checked during the SMTP session. On my system, any
email that comes with a envelope from that is invalid gets rejected.
The domain must have an MX record, there must be a server on the IP
address, and the sending domain's MTA must be willing to accept email
from the envelope from.

While this is almost as expensive a check as generating a challenge
message or your "thank you" message, it has the advantage that no
actual message gets sent. Combined with all the advantages of
rejecting email during the SMTP session, I consider this to be well
worth the expense.


I'm pretty sure that other MTAs besides Exim have similar features.


-wayne


-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----

[David F. Skoll]

Post by wayne

Post by David Segall
You could save the sender some trouble by just replying with a "Thank
you for your email I will attend to it ASAP" message instead of
demanding a response. You would treat the original message as valid if
your reply did not bounce.

*Any* kind of automatic reply (whether it's a "challenge" or an
"I'll get back to you") has potential to be abuse. I treat many such
messages as spam.

Post by wayne
Even better would be to use something like Exim's "call back" system
where this is checked during the SMTP session.

No, that's a bad idea for many reasons: The caller could be using a
"call back" system that might result in a loop. And it wastes network
resources, and can even be used as a tool to facilitate a DoS attack.
(I'll leave it to the reader to figure out how.)

Furthermore, many systems (M$ Exchange, for one) do not validate
recipients at the RCPT command, so Exim's callback is useless for
those systems.

Finally, once a sufficient number of systems use callbacks, spammers
will simply fake their mail from valid e-mail addresses or from "<>".

--

[Jem Berkes]

Post by David F. Skoll
Finally, once a sufficient number of systems use callbacks, spammers
will simply fake their mail from valid e-mail addresses or from "<>".

The <> loophole was the great flaw in my own attempt at a C-R system... now
I use statistical filters and blocklists for superior results.

[Alan Connor]

Post by Jem Berkes

Post by David F. Skoll
Finally, once a sufficient number of systems use callbacks, spammers
will simply fake their mail from valid e-mail addresses or from "<>".

The <> loophole was the great flaw in my own attempt at a C-R system... now
I use statistical filters and blocklists for superior results.
--
Jem Berkes
http://www.sysdesign.ca/

Not following you here, Jem. I don't have any problem with the from <>
return addresses. Any mail with those gets sent straight to the bit bucket
if it isn't from an individual or org or business on my passlist.

I checked my mail logs yesterday and discovered that every since these threads
have begun some clown or clowns have been doing everything possible to
bust through my filters.

I didn't even know they were trying.

And I've given them the URL to the source code....

Getting closer to my new release of elrav1. It's going to be much faster and
use much less bandwidth.



AC

[Sam]

Post by Alan Connor
Getting closer to my new release of elrav1. It's going to be much faster and
use much less bandwidth.

The emperor still has no clothes.

[Jem Berkes]

Post by Alan Connor

Post by Jem Berkes
The <> loophole was the great flaw in my own attempt at a C-R
system... now I use statistical filters and blocklists for superior
results.

Not following you here, Jem. I don't have any problem with the from <>
return addresses. Any mail with those gets sent straight to the bit
bucket if it isn't from an individual or org or business on my
passlist.

I'm not saying the C-R could not be made to work -- certainly, mail that
claims to be a mailer bounce <> can be subjected to some strict checks and
adequatedly controlled, and may not be a problem.

But I'm saying that instead of bothering to configure all that, I decided
that I'm happy with spamprobe that is giving me > 99.5% accuracy and no
false positives.

[Alan Connor]

Post by Jem Berkes

Post by Alan Connor

Post by Jem Berkes
The <> loophole was the great flaw in my own attempt at a C-R
system... now I use statistical filters and blocklists for superior
results.

Not following you here, Jem. I don't have any problem with the from <>
return addresses. Any mail with those gets sent straight to the bit
bucket if it isn't from an individual or org or business on my
passlist.

I'm not saying the C-R could not be made to work -- certainly, mail that
claims to be a mailer bounce <> can be subjected to some strict checks and
adequatedly controlled, and may not be a problem.
But I'm saying that instead of bothering to configure all that, I decided
that I'm happy with spamprobe that is giving me > 99.5% accuracy and no
false positives.
--
Jem Berkes
http://www.sysdesign.ca/

We have very different systems. It is no bother for *me* to deal with the
<> mail.

The From: header in bounced mail is that of the original sender, as is
the Reply-To: header, if the mail is legitimate. No need to even bother
with the Resent* headers.

AC

[wayne]

Post by David F. Skoll
*Any* kind of automatic reply (whether it's a "challenge" or an
"I'll get back to you") has potential to be abuse. I treat many such
messages as spam.

Agreed, and that includes bounces being sent to forged from
addresses. Granted, these bounces can not always be eliminated, but
well run systems should try hard to avoid creating them.

Post by David F. Skoll

Post by wayne
Even better would be to use something like Exim's "call back" system
where this is checked during the SMTP session.

No, that's a bad idea for many reasons: The caller could be using a
"call back" system that might result in a loop.

Such call-back systems *must* use MAIL FROM:<> in order to avoid
loops, which exim does.

Post by David F. Skoll
And it wastes network resources,

Calling back to the claimed sender does use resources, however so does
a lot of other checks, such as making sure that the sender's domain
exists, using DNSBLs, using bulk email detectors (DCC/Razor/Pyzor), etc.

Post by David F. Skoll
and can even be used as a tool to facilitate a DoS attack.

Using a call-back system does not amplify the bandwidth needed to do a
DoS attack. If someone wants to do a DoS attack, they can just as
easily make a direct attack.

Post by David F. Skoll
Furthermore, many systems (M$ Exchange, for one) do not validate
recipients at the RCPT command, so Exim's callback is useless for
those systems.

Well, such systems are not particularly well designed then. Either
the rejection of invalid RCPT TO addresses has to come after that DATA
command completes or by not rejecting at all during the SMTP session
and sending bounces to the claimed MAIL FROM address. If the MTA
rejects after the DATA command, it can't tell the sender which
addresses are invalid. If it sends bounces, then forged from
addresses get bombed.

Post by David F. Skoll
Finally, once a sufficient number of systems use callbacks, spammers
will simply fake their mail from valid e-mail addresses or from "<>".

This is a *good* thing. In particular, it means that there would be a
reduction in the number of bogus bounces being sent. Secondly, since
the null path is only supposed to be used for delivery status
notifications, additional anti-spam checks can be performed.



-wayne


-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----

[David F. Skoll]

wayne wrote:

[...]

Post by wayne
Calling back to the claimed sender does use resources, however so does
a lot of other checks, such as making sure that the sender's domain
exists, using DNSBLs, using bulk email detectors (DCC/Razor/Pyzor), etc.

DNSBLs, DCC, et al. are optimized for frequent lookups. SMTP servers
aren't particularly.

Post by wayne
Using a call-back system does not amplify the bandwidth needed to do a
DoS attack. If someone wants to do a DoS attack, they can just as
easily make a direct attack.

That's not what I had in mind.

Post by wayne

Post by David F. Skoll
Furthermore, many systems (M$ Exchange, for one) do not validate
recipients at the RCPT command, so Exim's callback is useless for
those systems.

Well, such systems are not particularly well designed then.

Agreed, but they are very common.

Post by wayne
Either
the rejection of invalid RCPT TO addresses has to come after that DATA
command completes or by not rejecting at all during the SMTP session
and sending bounces to the claimed MAIL FROM address. If the MTA
rejects after the DATA command, it can't tell the sender which
addresses are invalid. If it sends bounces, then forged from
addresses get bombed.

From my experimentation, yahoo.com rejects after DATA if there was a single
RCPT. If there was more than one RCPT, it accepts and then generates
bounces.

I believe this is to deter dictionary harvesting.

Post by wayne

Post by David F. Skoll
Finally, once a sufficient number of systems use callbacks, spammers
will simply fake their mail from valid e-mail addresses or from "<>".

This is a *good* thing.

No, it's not.

Post by wayne
In particular, it means that there would be a
reduction in the number of bogus bounces being sent.

No, the bounces will go to unrelated third parties. A spammer with a
CD of a million e-mail addresses will simply choose one as the "sender",
another as the recipient, and blast them out. Repeat for the remaining
addresses.

Post by wayne
Secondly, since
the null path is only supposed to be used for delivery status
notifications, additional anti-spam checks can be performed.

If the spammer uses "<>", yes. But if the spammer uses random-but-valid
e-mail addresses, the sender check will be useless.

Regards,

[wayne]

Post by David F. Skoll

Post by wayne
Calling back to the claimed sender does use resources, however so does
a lot of other checks, [...]

DNSBLs, DCC, et al. are optimized for frequent lookups. SMTP servers
aren't particularly.

SMTP servers are generally designed and configured to handle the loads
proportional to how much email the send and receive. If everyone used
simplistic, non-caching call backs, such as exim uses, then the number
of connections to SMTP servers would indeed double. The actual load
would be less because no DATA command would be given.

Post by David F. Skoll

Post by wayne
Using a call-back system does not amplify the bandwidth needed to do a
DoS attack. If someone wants to do a DoS attack, they can just as
easily make a direct attack.

That's not what I had in mind.

Well, I can guess several other things you might have had in mind,
such as the number of open TCP connections and such, but that doesn't
change my point that these same attacks can be just as easily done
directly.

Post by David F. Skoll

Post by wayne

[...] systems (M$ Exchange, for one) do not validate recipients

Well, such systems are not particularly well designed then.

Agreed, but they are very common.

They are "common" in the sense that they are not hard to find, but
they do not represent a large percentage of the servers that my system
runs across. YMMV.

Post by David F. Skoll
From my experimentation, yahoo.com rejects after DATA if there was a single
RCPT. If there was more than one RCPT, it accepts and then generates
bounces.

Which, as you agree, is not particularly well designed. It causes
bogus bounces to be sent to third parties.

Post by David F. Skoll

Post by wayne

Finally, once a sufficient number of systems use callbacks, spammers
will simply fake their mail from valid e-mail addresses or from "<>".

In particular, it means that there would be a
reduction in the number of bogus bounces being sent.

No, the bounces will go to unrelated third parties.

No bounces will be generated in the case of spammers using <> as the
sender. In the case of spammers using only valid email addresses,
then fewer bounces will be generated, even by MTAs that don't reject
at the RCPT TO time.

Post by David F. Skoll
[...] But if the spammer uses random-but-valid
e-mail addresses, the sender check will be useless.

All test that check the validity of data are "useless" when passed
only valid data. At this time, a large percentage of spam has invalid
data.


In the long run, I think that proposals such as DRIP and SPF/DMP/RMX
offer better solutions than sender call-backs. In the short run,
sender call-backs are more effective and will automatically be
no-op'ed if both systems support these proposals. (The DRIP proposal
answers the question "is this IP address authorized to run a relay?".
The SPF/DMP/RMX proposals answers the question ("is this IP address
authorized to send email claim to be from a given domain?")


-wayne



-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----

[D. Stussy]

Post by David F. Skoll
...
Finally, once a sufficient number of systems use callbacks, spammers
will simply fake their mail from valid e-mail addresses or from "<>".

Do you mean that they aren't already doing so? :-)

[Louis]

Allow me to provide some input on this:

CBV (Callback Verifiers) work! It is the best technical method
available to eliminate the majority of spoofing spammers at the
current SMTP level.

All other proposals have a deployment problem (namely, the DNS lookup
proposals) and have an emperical track record to introduce a major
performance problem.

Yes, CBV have two problems:

- Overhead
- Redundancy

But these can be solved just like any other overhead, redundancy
issues is solved; caching, learning, white/black listing, etc. More
Below.

We have implemented a CBV system and by far, we have rejected 90-94%
of spoofing spammers. This system is more of a "SMTP Compliancy
Tester" which includes a test for an Open Relay to check for "zombie"
systems. Our stats are showing atleast 15% of the rejection are open
relay sites.

To further add "intelligence" in addressing the overhead/redundancy
issue, we discovered what is ultimately the basis for most of the
DNS-lookup systems; identify what I call the DIP attribute (sender
domain:client IP pair).

The DMP proposal offers a good logic using TXT record to identify the
allowable sender machine and non-allowable sending machine. The spec
logic may require multiple lookups. This has a HIGH DNS lookup
redundancy problem adding a significant delay problem to failed
lookups.

The SPF proposal simplies the logic by having a single TXT record. A
less delay problem, due to less lookup potential. But still a problem
for failed DNS lookups.

Same issue with all other DNS based solutions: Failed lookups, which
can be viewed as a deployment issue, but it still doesn't solve the
other TRUST issues.

The TRUST of these proposals is only gain when:

- checking against your own local domain spoofing,
- checking against remote domain with resultant "DENY" result.

Any other result can not be trusted in any of the proposals.

We explore the idea of using the local domain trust to minimized CBV
overhread. This works.

So to make a long story short, CBV works with additional machine
learning features. To augment DNS-based lookup solutions, it is best
only to check against your own local domain spoofing. Using it for
other remote domain lookups drastically degrades the performance of
the system, and until the deployment is wide-spread, it should not be
used, if at all.

Issues such as NULL-ADDRESS, DoS, are non-issues. There is a reason
why spammers hardly use the NULL-ADDRESS - automatic imposed system
restrictions are then applied, namely, final destination mail only.

The only issue really issue is how some systems delay the CBV RCPT TO:
validation process.

FACT: Whether or not the RCPT TO: validation is performed dynamically
or delayed, Return-Path validity is a RFC requirement at some point
of the process. Otherwise the system doesn't work.

Systems that delay the process perpetuate the spoofing problem, hence
why most of the anonymous access come from YAHOO.COM and similar
domains. YAHOO needs to recognize the delay contributes to the
problem and if they did instant validation, SPOOFERS will be reduces.
Hey, look at it this way, AOL does not delay the validation. AOL is
just as big or bigger than YAHOO. AOL helps solve the problem. YAHOO
does not.

SUMMARY:

CBV works! The proof of concept is that the return path can be
validated to eliminate 90-94% of the spoofers which by far, is the
majority of the "spam problem."

The recently signed CAN-SPAM Act will now mandate validate return
address. Eventually, the CBV will became a audit/tracking system if
we are too assume that most "spammers" will become CAN-SPAM compliant.

The actual method used by CBV today is only because it offers current
SMTP operations. Future research should be done to VALIDATE the
return address at a more optimal level.

In the mean time, to be CAN-SPAM ready, your SMTP system must offer
two things:

- Validation of Return Path

- Optional Support for LABEL filtering which can be handled at the
DATA stage, however, I recommend an ESMTP (Extended SMTP) support
design which offers a "SUBJECT" command concept at the SMTP protocol
level.

These are two basic technical compliancy issues SPAMMERS must adhere
too in regards to CAN-SPAM.

This our experience and input on the matter.

If you wish to see some stats:
http://www.winserver.com/public/security

Hector Santos, CTO
Santronics Software, Inc.

Post by David F. Skoll

Post by wayne

Post by David Segall
You could save the sender some trouble by just replying with a "Thank
you for your email I will attend to it ASAP" message instead of
demanding a response. You would treat the original message as valid if
your reply did not bounce.

*Any* kind of automatic reply (whether it's a "challenge" or an
"I'll get back to you") has potential to be abuse. I treat many such
messages as spam.

Post by wayne
Even better would be to use something like Exim's "call back" system
where this is checked during the SMTP session.

No, that's a bad idea for many reasons: The caller could be using a
"call back" system that might result in a loop. And it wastes network
resources, and can even be used as a tool to facilitate a DoS attack.
(I'll leave it to the reader to figure out how.)
Furthermore, many systems (M$ Exchange, for one) do not validate
recipients at the RCPT command, so Exim's callback is useless for
those systems.
Finally, once a sufficient number of systems use callbacks, spammers
will simply fake their mail from valid e-mail addresses or from "<>".
--

[Alan Connor]

Post by Louis
CBV (Callback Verifiers) work! It is the best technical method
available to eliminate the majority of spoofing spammers at the
current SMTP level.

Sure sounds un-necessarily complex and not very thorough to me.

By using a combination of pass-list/block-list/challenge-response,

I have eliminated 100% of all spam with no muss or fuss, and no one

that I want to hear from has any problems getting through to me.


Why mess with perfection?


<snip an interesting article>


AC

[Louis]

Post by Alan Connor

Post by Louis
CBV (Callback Verifiers) work! It is the best technical method
available to eliminate the majority of spoofing spammers at the
current SMTP level.

Sure sounds un-necessarily complex and not very thorough to me.

Actually, its not complex at all. Most of work is in solving the
overhead and redundancy issues. It is 100% thorough, eliminating
invalid return addresses and enforcing compliance.

Post by Alan Connor
By using a combination of pass-list/block-list/challenge-response,
I have eliminated 100% of all spam with no muss or fuss, and no one
that I want to hear from has any problems getting through to me.

Oh, we also reach 100% when combining all the various methods always.
:-)

However, I personally do not like the C/R method (bad term for it IMO)
because it involves the user and has "intrusive" attributes that is
not desirable in the business world. Not saying it is not useful, I
prefer to concentrate my efforts at the protocol level which I believe
is going to be the ultimate method anyway.

Post by Alan Connor
Why mess with perfection?

I respect that. In my view, it isn't perfect, using C/R is after the
fact, pass-list/block-list requires learning, which is ok, but my
goals as a mail system designer (which includes all parts of a
complete mail system, multiple UI software, senders, receivers,
gateways, frontends, hosting, etc), is to do what is possible at the
protocol level because after all, that is what the spammers (or
anonymous access clients) are exploiting. Regardless of what the mail
content is, the problem today is uncontrollable anonymous access, a
problem that needs to be solved regardless of whether mail is spam or
not.

Here is an average breakdown:

~4% drop after receiving the greeting,

of what remains (making it to the HELO/EHLO state point):

~15% fail with non-compliant syntax/format host information. The
majority fail with unbracketed domain literals or attempt to use your
IP which does not match the connection peer IP.

of what remains (making it to the MAIL FROM: state point):

~85% is refused due to unverifiable return paths.

of what remains (making it to the RCPT TO: state point)

~75% is refused due to unknown recipient.

of what remains (making it to the DATA state point)

~11% are refused due to invalid state point. These are usually
streamers who are ignoring all refusal response codes.

of what remains, the data is received and it is pass to a MAIL
FILTERING system. This is 3rd party software where we offer the DATA
EVENT hook. We are not in the "mail interpretation business." That is
left to the admin and users.

~11% is filtered in our specific system with some basic filtering
rules, black list, etc. In our case, we have an exclusive support
intranet system, and most users are restricted in using our system for
email.

Now, with all our work, I've average maybe real spam 3-4 messages per
week and these are first seen as headers. FROM: TO: SUBJECT and I can
quickly delete them and also possible add a new rule. More
importantly, the majority of these, if not 100%, are from YAHOO.COM
because of their lame delay validation system which only perpetuates
the spam problem. Surely, if a C/R was implemented it would more than
likely catch these.

But guess what?

C/R does not solve the problem at the protocol level. The SPAMMER has
learn that the mail was accepted. They really don't care if you
respond to the challenge or read the mail or not. They are in the
business of selling an "email list" to potential clients:

"Our research has shown that 90% of our email addresses
are accepted by various systems. We don't guarantee that
the user will read them, but it will be accepted by the
system."

So the most important asset is the email list with addresses being
accepted.

C/R really doesn't help the real technical goals are with new
anti-spam (or rather anonymous access) research have to accomplish. I
prefer to solve the problem at the protocol level because it solves
the problem (or rather addresses it) at the most fundamental level.

So in my view, systems that do not validate the recipient address are
only perpetuating the problem. Of course, there are many legitimate
large systems who will post validate the address due to scalability
issues or because they are using legacy software (like our own 1996
version which didn't offer SMTP realtime local domain final
destination checking) or maybe are still using a UUCP/UUCICO gateways
in which case has a major with routed return paths anyway.

As AOL has shown, that's only an excuse for not improving the system.
They have shown it is possible to provide instant final destination
checking for large systems. Instant final destination will eliminate
obfuscated addresses, "Joe-Jobbing" and once spammers know this is the
standard practice across the board, their efforts will change to
obtaining "valid email addresses" and also providing "valid return
paths" in full compliance with CAN-SPAM.

Anyway, that's my view. :-)

Thanks for your comments.

Hector Santos/CTO
Santronics Software, Inc.

[Alan Connor]

Post by Louis

Post by Alan Connor

Post by Louis
CBV (Callback Verifiers) work! It is the best technical method
available to eliminate the majority of spoofing spammers at the
current SMTP level.

Sure sounds un-necessarily complex and not very thorough to me.

Actually, its not complex at all. Most of work is in solving the
overhead and redundancy issues. It is 100% thorough, eliminating
invalid return addresses and enforcing compliance.

Post by Alan Connor
By using a combination of pass-list/block-list/challenge-response,
I have eliminated 100% of all spam with no muss or fuss, and no one
that I want to hear from has any problems getting through to me.

Oh, we also reach 100% when combining all the various methods always.
:-)

Still seems like the long way around the barn to me :-)

Post by Louis
However, I personally do not like the C/R method (bad term for it IMO)
because it involves the user and has "intrusive" attributes that is
not desirable in the business world. Not saying it is not useful, I
prefer to concentrate my efforts at the protocol level which I believe
is going to be the ultimate method anyway.

Post by Alan Connor
Why mess with perfection?

I respect that. In my view, it isn't perfect, using C/R is after the
fact, pass-list/block-list requires learning, which is ok, but my
goals as a mail system designer (which includes all parts of a
complete mail system, multiple UI software, senders, receivers,
gateways, frontends, hosting, etc), is to do what is possible at the
protocol level because after all, that is what the spammers (or
anonymous access clients) are exploiting. Regardless of what the mail
content is, the problem today is uncontrollable anonymous access, a
problem that needs to be solved regardless of whether mail is spam or
not.

Couldn't agree with you more about anonymous access. That sure is the problem.
I don't accept it myself, which is the whole point of the "CR", which
I prefer to call an RAV (Request-for-Address-Validation).

They are sort of like Caller-ID for my computer....

Post by Louis
~4% drop after receiving the greeting,
~15% fail with non-compliant syntax/format host information. The
majority fail with unbracketed domain literals or attempt to use your
IP which does not match the connection peer IP.
~85% is refused due to unverifiable return paths.
of what remains (making it to the RCPT TO: state point)
~75% is refused due to unknown recipient.
of what remains (making it to the DATA state point)
~11% are refused due to invalid state point. These are usually
streamers who are ignoring all refusal response codes.
of what remains, the data is received and it is pass to a MAIL
FILTERING system. This is 3rd party software where we offer the DATA
EVENT hook. We are not in the "mail interpretation business." That is
left to the admin and users.
~11% is filtered in our specific system with some basic filtering
rules, black list, etc. In our case, we have an exclusive support
intranet system, and most users are restricted in using our system for
email.
Now, with all our work, I've average maybe real spam 3-4 messages per
week and these are first seen as headers. FROM: TO: SUBJECT and I can
quickly delete them and also possible add a new rule. More
importantly, the majority of these, if not 100%, are from YAHOO.COM
because of their lame delay validation system which only perpetuates
the spam problem. Surely, if a C/R was implemented it would more than
likely catch these.
But guess what?
C/R does not solve the problem at the protocol level. The SPAMMER has
learn that the mail was accepted. They really don't care if you
respond to the challenge or read the mail or not. They are in the
"Our research has shown that 90% of our email addresses
are accepted by various systems. We don't guarantee that
the user will read them, but it will be accepted by the
system."

No. I send very, very few C/Rs. Very little spam makes it that far
"down" into my mailfilter. And if they fail to return a C/R twice,
they are block-listed.

Your view of mail programs that use C/Rs is simplistic.
Please see my YNM post currently on the "board".

Post by Louis
So the most important asset is the email list with addresses being
accepted.
C/R really doesn't help the real technical goals are with new
anti-spam (or rather anonymous access) research have to accomplish. I
prefer to solve the problem at the protocol level because it solves
the problem (or rather addresses it) at the most fundamental level.
So in my view, systems that do not validate the recipient address are
only perpetuating the problem. Of course, there are many legitimate
large systems who will post validate the address due to scalability
issues or because they are using legacy software (like our own 1996
version which didn't offer SMTP realtime local domain final
destination checking) or maybe are still using a UUCP/UUCICO gateways
in which case has a major with routed return paths anyway.
As AOL has shown, that's only an excuse for not improving the system.
They have shown it is possible to provide instant final destination
checking for large systems. Instant final destination will eliminate
obfuscated addresses, "Joe-Jobbing" and once spammers know this is the
standard practice across the board, their efforts will change to
obtaining "valid email addresses" and also providing "valid return
paths" in full compliance with CAN-SPAM.
Anyway, that's my view. :-)
Thanks for your comments.

You are missing something here. No one I know uses RAVs/CRs *alone*.
I do intensive filtering before the remainder is passed to the RAV stage
of the filter.

I very much like your idea, but I think we are actually addressing the
same problem in different ways.

Anonymous mail shouldn't be allowed at all. So I don't accept it.
If the majority of people refused to accept it, however they chose to do
it, then spamming would just stop.

I don't care whether spammers use valid return addresses or not, nor
whether they are mailing to me specifically.

I still wouldn't accept it, and if you forced them to do that, then I
will have to modify my program to eliminate that too.

Now that I think about it, what you are up to is going to make spam
even HARDER to deal with.

But maybe not. I don't think they will be able to afford to have enough
staff to respond to RAVs, and it can't be done with a computer. Got that
covered.

If I have to, I can refuse to accept any mail from strangers over 5k, that is
in anything but plain text. Delete it on the server.

---------------------------


Very interesting.

AC

[Jonathan de Boyne Pollard]

L> Regardless of what the mail content is, the problem today is
L> uncontrollable anonymous access, [...]

False. Anonymity is not the problem. People who dislike anonymity are living
with the false beliefs that they already know everyone with whom they will
ever communicate (This is patently false for the simple, and obvious, reason
that people are born and die.) and that names are anything more than an agreed
convention.

The actual problem is that it is very cheap in the world of electronic mail
(when compared to other systems) for a sender to make and to transmit multiple
copies of something, and that because of the current mail architecture this
puts recipients at a massive disadvantage with respect to senders. The best
solution to this, of course, is to create an architecture that actually _takes
advantage_ of the fact that copying is cheap, instead of one that is
confounded by it.

[Louis]

Post by Jonathan de Boyne Pollard
L> Regardless of what the mail content is, the problem today is
L> uncontrollable anonymous access, [...]
False. Anonymity is not the problem.

Oh, anonymous abuse of systems is not a problem?

Post by Jonathan de Boyne Pollard
People who dislike anonymity are living
with the false beliefs that they already know everyone with whom they will
ever communicate (This is patently false for the simple, and obvious, reason
that people are born and die.) and that names are anything more than an agreed
convention.

It really has nothing to do with disliking ANONYMITY. It has to do
with the ABUSE it brings and has brought to the industry when it comes
to spam and the high cost it imposes on the mail system.

Note, when I say anonymous, I MEAN anonymous. I am not talking about
trackable anonymous access.

Post by Jonathan de Boyne Pollard
The actual problem is that it is very cheap in the world of electronic mail
(when compared to other systems) for a sender to make and to transmit multiple
copies of something, and that because of the current mail architecture this
puts recipients at a massive disadvantage with respect to senders. The best
solution to this, of course, is to create an architecture that actually _takes
advantage_ of the fact that copying is cheap, instead of one that is
confounded by it.

Well, I agree with your basic thought. You view it as COST. Good.
Well COST inherently implies access control, recording and
accountability.

Sure, we can design a new system. If fact, you don't have to go very
far as the original mail/hosting/transport systems were all based on
authenication, access control, as well as COST of using the system.
Placing a COST on a message posting or file upload/download for
example, isn't a novel idea. We have a high end commercial BBS system
and it still stands strong. It has complete accountability.

But thats not the reality - expecting a complete revamp of the system.
We need to work with the current SMTP system and use whatever is
feasible, makes sense and also make sure it is consistent with
CAN-SPAM. Hopefully, one day the specs will change, but I believe
most people doubt it will change significantly.

Thats my take.

Hector Santos, CTO
Santronics Software, Inc.

[Jonathan de Boyne Pollard]

L> Regardless of what the mail content is, the problem today is
L> uncontrollable anonymous access, [...]

JdeBP> False. Anonymity is not the problem.

L> Oh, anonymous abuse of systems is not a problem?

What I actually wrote is right there. Read it.

L> It really has nothing to do with disliking ANONYMITY. It has to
L> do with the ABUSE it brings and has brought to the industry when
L> it comes to spam and the high cost it imposes on the mail system.

The abuse is not the result of anonymity. Anonymity does not bring about the
abuse. I'll say it again: Anonymity is not the problem.

JdeBP> The actual problem is that it is very cheap in the world of
JdeBP> electronic mail (when compared to other systems) for a sender
JdeBP> to make and to transmit multiple copies of something, and that
JdeBP> because of the current mail architecture this puts recipients
JdeBP> at a massive disadvantage with respect to senders. The best
JdeBP> solution to this, of course, is to create an architecture that
JdeBP> actually _takes advantage_ of the fact that copying is cheap,
JdeBP> instead of one that is confounded by it.

L> Well, I agree with your basic thought. You view it as COST.
L> Good. Well COST inherently implies access control, recording
L> and accountability.

All of which are merely tangential to the actual problem, which is that the
cost is incurred in the "wrong" place. You can perform accounting for costs
until you turn blue in the face, but that doesn't address the actual problem,
which is that there is very little actual cost to making and to transmitting
multiple copies of something and that this works to the disadvantage of
recipients.

L> Placing a COST on a message posting or file upload/download for
L> example, isn't a novel idea.

It's also the wrong idea. An "electronic postage stamp" is not the
answer. Nor are simple variations on the scheme such as end-to-end
authorization and accounting. As I said, the best solution is a system where
the fact that making copies of messages is cheap to do actually works in the
system's, and the recipients', favour.

L> We have a high end commercial BBS system and it still
L> stands strong.

And in its heyday, Fidonet and BBS networks suffered from the very problem
that senders were at an advantage compared to recipients, because of the
relatively low costs of creating multiple copies of messages, and almost (were
it not for the fact that dial-up charges tended to militate against it by
imposing costs on the actual transmission of those multiple copies by the
original sender) came to the point of having to address this same problem.

L> But thats not the reality - expecting a complete revamp
L> of the system.

Actually, it is. Even a cursory study of the history of computing reveals
that complete revamps occur quite frequently.

[Alan Connor]

On Tue, 06 Jan 2004 20:32:48 +0000, Jonathan de Boyne Pollard <***@Tesco.NET> wrote:
You are wrong. The only way to stop spam is to refuse anonymous mail.
Works great for me and hundreds of others that I know of.
It's not a fact of life in OUR worlds.

Refusing bulk mail that hasn't been passlisted is the other key.


AC

[Jonathan de Boyne Pollard]

AC> The only way to stop spam is to refuse anonymous mail.

False. This is an example of the error that has been made time and
again in combating UBM. Anonymity isn't the problem. (And, as I
have said before, anonymity is a fundamental and unavoidable fact of
life.) Unsolicited bulk mail is the problem. "Anonymous" is not
synonymous with "unsolicited bulk", and therefore refusing mail with
the former quality does not refuse mail with the latter.

AC> Works great for me and hundreds of others that I know of.

<URL:http://homepages.tesco.net./~J.deBoynePollard/FGA/smtp-anti-ubm-dont-work.html>

AC> It's not a fact of life in OUR worlds.

Anonymity exists for you as it does for the rest of us. You've merely
chosen to attempt to steer clear of it. But of course by the very act
of attempting that, you acknowledge the existence of anonymity as a
fact of life. And, of course, your attempt is futile, since anonymity
is unavoidable.

[Andrea Griffini]

On Tue, 06 Jan 2004 20:32:48 +0000, Jonathan de Boyne Pollard

Post by Jonathan de Boyne Pollard
The abuse is not the result of anonymity. Anonymity does not bring about the
abuse. I'll say it again: Anonymity is not the problem.

I think however that it would make perfect sense that
someone *decide* to drop directly in the wastebasket all
untraceable mail (and this could be done directly at
the receiving server if that user has that flag enabled).

Indeed when I answer the phone it's way better you say
who you are first, as otherwise the connection may drop
without me even listening to what you want to say.

Sure I think anonymous mail should travel regularly to
whoever decides to accept it. It's not a decision that
should be on the ISP.


Andrea

[Louis]

Post by Jonathan de Boyne Pollard
L> It really has nothing to do with disliking ANONYMITY. It has to
L> do with the ABUSE it brings and has brought to the industry when
L> it comes to spam and the high cost it imposes on the mail system.
The abuse is not the result of anonymity. Anonymity does not bring about the
abuse. I'll say it again: Anonymity is not the problem.

I guess you will just have to accept the idea I don't agree with you.
I have no problem with authenticated, subscribed, or known entities.
It is those who are spoofing the system that is causing the majority
of the problem. That is the fact. If you are referring to a
"cause," well, in my view, that is a technical issue that is currently
related to the SMTP protocol.

Post by Jonathan de Boyne Pollard
L> Well, I agree with your basic thought. You view it as COST.
L> Good. Well COST inherently implies access control, recording
L> and accountability.
All of which are merely tangential to the actual problem, which is that the
cost is incurred in the "wrong" place. You can perform accounting for costs
until you turn blue in the face, but that doesn't address the actual problem,
which is that there is very little actual cost to making and to transmitting
multiple copies of something and that this works to the disadvantage of
recipients.

Hmmm, correct me if I am wrong, you are saying the real problem is
that it is too easy to create spam (bulk or otherwise)?

That is a technical problem with the protocol which is inherently an
weak "anonymous access" control issue. It is this part of the
specification that allows spammers the ease to duplicate.

Post by Jonathan de Boyne Pollard
As I said, the best solution is a system where
the fact that making copies of messages is cheap to do actually works in the
system's, and the recipients', favour.

So the problem is the EASE of "making" or "sending?" I am not sure I
follow you if you are saying it is too easy to "make copies." SMTP
allows you to send 1 message as X number of copies. It is the
protocol that makes the "copies."

The realistic industry abuse is where there is MULTIPLE ANONYMOUS
systems sending thousands of messages, usually 1 PER SECTION creating
a major scalability issues. I can understand the idea to put a cost
on the sender. But that isn't going to solve the problem TODAY.
TODAY, you have to control the anonymous access.

Post by Jonathan de Boyne Pollard
And in its heyday, Fidonet and BBS networks suffered from the very problem
that senders were at an advantage compared to recipients, because of the
relatively low costs of creating multiple copies of messages, and almost
(were it not for the fact that dial-up charges tended to militate against
it by imposing costs on the actual transmission of those multiple copies
by the original sender) came to the point of having to address this same
problem.

Having made my career on BBS and FIDONET and the Internet for the last
25+ years, on all aspects, I think I consider myself a knowledgable
voice on this subject.

Anonymous ACCESS was by far, the HALLMARK for most of the problematic
issues beginning in electronic communications, including email and in
PUBLIC communications such as THIS forum.

The point is that the SENDERS were always managed. It was anonymous
entry concepts that began to cause all kinds of issues at all levels.
If the SENDER was controlled by some means, it usually wasn't a
problem and that including imposing a cost on the sender, if need be.

In any case, the PROBLEM started with the internet integrated into it.
I understand what you mean that now the SPAMMERS were able to easily
spam you, with little to no cost, hence in your VIEW is the "problem."
However, they could ONLY do so because the real problem was TECHNICAL
at the protocol level. In other words, the flexible anonymous access
architecture of the SMTP SYSTEM allowed them to do so. Fix this and
the SPAMMER is gone (or controlled).

Post by Jonathan de Boyne Pollard
L> But thats not the reality - expecting a complete revamp
L> of the system.
Actually, it is. Even a cursory study of the history of computing reveals
that complete revamps occur quite frequently.

Sure, in general. But not in this area.

In the old days, you had:

X.400, MHS in the commercial world and Fidonet (as a standard) in the
public world. You also had QWK, OPX and BW as the basic three
standards for offline mail format.

SMTP/POP3 was still its infancy and mostly in government and academia.

Once the internet was made the "global network" for the world,
SMTP/POP3 became the standard.

So there was really only one change in the annals of electronic mail
history.

I sincerely doubt we will see a TOTAL revamp at the level we seen
before. We might see changes with continued extensions to the
protocol, but there is FAR too much invested today to completely
revamp it.

However, I have no doubt in my mind as an experience engineer in the
field, that uncontrolled anonymous access is at the beginning of its
end. I tend to believe most developers in the field are under the gun
and will agree with me that "things will be changing" in this area.
It already has. When Santronics releases its new upgrade this month,
hundreds of thousands of systems will have built-in "CAN-SPAM Ready"
anonymous access SMTP control features.

Thanks

Hector Santos
http://www.santronics.com

[Jonathan de Boyne Pollard]

L> All other proposals have a deployment problem
L> (namely, the DNS lookup proposals) [...]

They actually have a more fundamental problem than that. They attempt to
correlate (the domain portions of) envelope sender mailbox names with the IP
addresses of SMTP Relay clients. But there is simply no such correlation.
The legitimate owner of a mailbox should be able to send mail with that
mailbox as the envelope sender from anywhere in the world that he or she
likes. Similarly, one should be able to send mail via one's connection to
Internet using as the envelope sender whatever mailbox one might legitimately
own.

This is not their only fundamental problem, either.

L> AOL helps solve the problem.

... and resurrects another problem as a consequence of doing so, taking the
world back to the state that it was in several years ago. There is a
fundamental conflict. "Call back" schemes that verify envelope sender
mailboxes require, in order to be most effective, that envelope recipient
mailbox local parts be verified by the SMTP Relay server. But doing so
enables RCPT Harvesting (aka "Rumpelstilzchen attacks"). Implementing the
measures that people have implemented to prevent RCPT Harvesting over the past
few years greatly reduces the utility of "call back" schemes.

L> CBV works!

No scheme to combat unsolicited bulk mail via SMTP works. They either cause
an arms race (biased in favour of the UBM senders; as is the case with
Bayesian filters, for example) that in the long run does little more than
cause SMTP to become less usable and patchy (as all of the various blocks and
the collateral damage from their false positives accumulate), or (as here)
they simply highlight one of the fundamental conflicts in SMTP-based
electronic mail.

[Louis]

Hi Johanthan,

Post by Jonathan de Boyne Pollard
L> All other proposals have a deployment problem
L> (namely, the DNS lookup proposals) [...]
They actually have a more fundamental problem than that. They attempt to
correlate (the domain portions of) envelope sender mailbox names with the IP
addresses of SMTP Relay clients. But there is simply no such correlation.

Sure there is. More below...

Post by Jonathan de Boyne Pollard
The legitimate owner of a mailbox should be able to send mail with that
mailbox as the envelope sender from anywhere in the world that he or she
likes. Similarly, one should be able to send mail via one's connection to
Internet using as the envelope sender whatever mailbox one might legitimately
own.

Only if he is authenticated and authorized.

First lets not forget a few things.

Fundamentally, this is a "Anonymous Access" issue. As long as the
user is authorized to send mail, the user can connect from anywhere
and send to local or remote mail. That is not the issue. SMTP offers
the traditional IP relay tables, the Extended SMTP AUTH method and
possible the POP BEFORE SMTP method. The latter two was "invented" to
solve the roaming user problem (their machine IP changing for whatever
reason). So the concept of an Unknown Machine is a non-issue as long
as the user is authorized.

Second, the basic rule of thumb with SMTP security is:

anonymous access - can only send mail to local final destination
authorized access - can send local or remote (relay)

If the user authenticates, then there is no problem.

So the problem to be solved is how to control the high cost and REAL
ABUSE of anonymous access clients on the industry at all levels using
the current SMTP specification.

So lets focus on "Anonymous access" because a "legitimate user" is one
that is authorized do what what you describe.

So what is "anonymous access?"

Basically, in terms of "thee problem," it is one that is untrackable
and untracable. Basically, "anonymous" in the pure sense of the word.

The "problem" is the abuse that is costing the industry billions of
dollars.

The efforts in my view to address "the problem" is to add back
tracability and accounability. This is what CAN-SPAM Act attempts to
do. CAN-SPAM does not eliminate spam, in fact, it makes it worst, by
telling spammers:

"Hey, as long as you follow the rules, go ahead and spam!"

What are the basic CAN-SPAM rules?

I summarized them as two things:

- Valid return path, and
- Labeling.

That is all CAN-SPAM is telling spammers to do:

"Don't hide, spell it out, give the users a change to
opt-out and you can spam. If you don't follow the
new rules, then we will come and get you!"

and btw, it says one more thing,

"Follow the specifications set forth by IETF."

and if you listen to Dave Crocker, the "Father of SMTP", he says:

"Lets solve the problem using the current SMTP specification.
Lets not break it!"

With that mind set, industry developers really don't have must to work
with to solve the problem. In fact, many developers have basically
throw up their hands and said "$%#K'em" You now have developer and
vendors going on their own to do things that is basically NON-RFC
compliant concepts.

AOL and RoadRunners and a few others have already SHUT out the dynamic
IP user - technically a violation of the RFC specification as it
currently written because as I said above, if you got local
destination mail to send, anonymous access should be allowed.

But the abuse was too BIG with dynamic IP machines. It was time to put
restrictions. I agree with the idea, but I don't agree with how they
did it.

Now you have the "Alliance", Microsoft, AOL, EarthLink, YAHOO and
VERISIGN who are coming up with their own "Domain Key" specification
concept. Whats funny is that YAHOO says they will rollout this new
system in early 2004, and NO ONE has seen any SPECIFICATION
whatsoever. Atleast I haven't and I wrote to YAHOO three times to
inquire about it - not one response. I think they are full of it.
They were testing the water to see what people will think - "Will
people accept the Alliance controlling the industry mail system?"

Anyway, in the mean time, the rest of us have to deal with current
specifications.

You have three (3) parameters to work with, actually four (4):

CIP Connection IP address
CDN Client DOMAIN provided at HELO/EHLO
RP Sender Return Path provided at MAIL FROM:
RCPT The target recipient(s).

Again, anonymous access - local mail posting only.

The way I view it is as follows:

1) If sending mail to a non-existing local user, then there is no
need to check for access priverledges.

2) It is only when a RCPT exist or the system has decided to receive
mail, should it do an "Access Check"

What is the access check capabilities for anonymous clients?

Right now, according to SMTP specs, NONE! Hence the exploitation by
spammers.

I like the CBV because it is consistent with the RFC specification
that the return path must be validate in order to bounce mail. CBV
does't care where you sending it from. CBV simply wants to verify the
address. If the remote says NOT VALID, then thats is what we are
looking for. Any other result can not be trusted and you move on per
the RFC specifications.

What the DNS proposals do is try to bring in the sending maching
access.

In my view, the best they offer is to check your own local domains.
You can't any other result.

Since it is our SYSTEM POLICY that users must authenticate when they
use our domain name, then proposals like DMP offer us a way to make
sure that spammers are not spoofing our local domain on our system.

In my view, at the protocol level, that is the best you can do with
current specifications. You can check for remote domain DMP
compliants but this is REALLY a big overhead issues because of failed
DNS lookups.

Post by Jonathan de Boyne Pollard
This is not their only fundamental problem, either.
L> AOL helps solve the problem.
... and resurrects another problem as a consequence of doing so, taking the
world back to the state that it was in several years ago. There is a
fundamental conflict. "Call back" schemes that verify envelope sender
mailboxes require, in order to be most effective, that envelope recipient
mailbox local parts be verified by the SMTP Relay server. But doing so
enables RCPT Harvesting (aka "Rumpelstilzchen attacks").

I believe that is a red herring. If you accept the address with no
feedback, as far as the SPAMMER is concern it has added weight to the
value of the email address - REGARDLESS whether it is a real address
or not that will be discarded at a later point.

Post by Jonathan de Boyne Pollard
Implementing the
measures that people have implemented to prevent RCPT Harvesting over the past
few years greatly reduces the utility of "call back" schemes.

Well, empiricically, it doesn't correlate with your view. The data
shows there is VERY little organizations accepting mail for post
processing, because they have realized they will open themselves to
open relay and scalability issues. Again, you need to think as the
outsider, the spammer. As far as the outside is concern you are honey
pot for spam and spammers will exploit your system and your domain
address across other systems to the maximum extent. So the
operational decision to accept all mail at the SMTP level, perpetuates
the problem not only for the company but for the industry as a whole.
While it wasn't a big problem in the past, today, not validation your
address will increase scalability issues on your system and eveyone
elses as YAHOO have problem to show with a majority of all spam having
spoofed YAHOO addresses. So in my view, they put on a burden on
EVERYONE else as well.

Also, as I said, its a red herring, YAHOO will validate the address
at the DATA stage. So if a spammer wanted to "havest" the address,
all they need to do is continue with the DATA transmission. They will
find out very fast whether it is good or not.

In any case, with the exception of YAHOO, the odds are very high that
if a system accepts ROUTED MAIL in an anonymous matter, it is more
than likely a open relay. If it restricts routed mail but accepts all
local addresses, well there is not much you can do here. But again,
a CBV system is not looking for true positives, it is looking for true
negatives which offer 100% and by far, the emperical data have shown
that 80% of all email are spoofed as 100% true rejected addresses.

Post by Jonathan de Boyne Pollard
L> CBV works!
No scheme to combat unsolicited bulk mail via SMTP works.

Will anything work 100%? No, but emperically, CBV has shown to
eliminate atleast 80% or higher of the spoofing spammers. Thats 100%
TRUE negatives of rejected addresses. Hey, thats a MAJOR improvement
at the protocol level.

Post by Jonathan de Boyne Pollard
They either cause
an arms race (biased in favour of the UBM senders; as is the case with
Bayesian filters, for example) that in the long run does little more than
cause SMTP to become less usable and patchy (as all of the various blocks and
the collateral damage from their false positives accumulate), or (as here)
they simply highlight one of the fundamental conflicts in SMTP-based
electronic mail.

Well, as a mail system designed, I have no interest in the
interpretation of mail content, and personally, I believe I am
ethically bound not to deal with it. I am only interested in fixing an
"anonymous access" problem at the protocol level and it JUST so
happens that by doing so, you solve 80% of the spam problem. Thats a
MAJOR improvement. It has nothing to do what SPAM is or what the mail
content really is. It is anonymous access problem.

SMTP has no business in the "interpretation of mail content" game.
That is an system policy or end-user issue. The best SMTP can do is
basically control access by enforcing the specs. We were given a
mandate by CAN-SPAM:

- Validation of the Return Path
- Labeling.

The first one is easy. The second one requires a CHANGE to the SMTP
specification, if we want to offer a SYSTEM POLICY feature to detect
ADV tags at the protocol level. Otherwise, it has to be done a the
DATA stage or post processing.

In my view, the problem is pretty simple :-) We just need to best
way to validate that address. With CURRENT specs, CBV is the only
way.

Those are my views :-)

Hector Santos, CTO
Santronics Software, Inc.

[Jeff]

I have several (Linux) systems with dynamic IP addresses that can no
longer reach friends with AOL accounts. Is there a way to modify my
postfix setup so I can get through to them? Or is the blocking of dynamic
IP addresses absolute? I'm relaying through my ISP.

-Jeff

[Jonathan de Boyne Pollard]

JdeBP> They actually have a more fundamental problem than that.
JdeBP> They attempt to correlate (the domain portions of) envelope
JdeBP> sender mailbox names with the IP addresses of SMTP Relay
JdeBP> clients. But there is simply no such correlation.

L> Sure there is.

Wrong. There is no such correlation. As I said:

JdeBP> The legitimate owner of a mailbox should be able to send
JdeBP> mail with that mailbox as the envelope sender from anywhere
JdeBP> in the world that he or she likes. Similarly, one should be
JdeBP> able to send mail via one's connection to Internet using as
JdeBP> the envelope sender whatever mailbox one might legitimately
JdeBP> own.

L> Only if he is authenticated and authorized.

That's beside the point.

L> Fundamentally, this is a "Anonymous Access" issue. [...]

Wrong. It isn't. As I said before, anonymity isn't the problem. Anonymity
is a fundamental fact of life. Concentrating upon anonymity is concentrating
upon the wrong problem.

L> Second, the basic rule of thumb with SMTP security is:
L> anonymous access - can only send mail to local final destination
L> authorized access - can send local or remote (relay)

The "DNS lookup" mechanisms, which are what are being discussed here, have
nothing to do with that model, however. That model applies to the originator
of the mail. However, the DNS lookup mechanisms apply to the machine through
which the originator is relaying the mail.

L> So the problem to be solved is how to control the high cost
L> and REAL ABUSE of anonymous access clients on the industry
L> at all levels using the current SMTP specification.

No, it isn't. The problem is unsolicited bulk mail. Anonymity isn't the
problem.

L> So lets focus on "Anonymous access" [...]

The next 33 paragraphs of your message are based upon the false premise that
anonymity is the problem. I'll skip them, because it isn't. Only two need to
be addressed on other grounds:

L> if you listen to Dave Crocker, the "Father of SMTP", he says:
L>
L> "Lets solve the problem using the current SMTP specification.
L> Lets not break it!"

Dave Crocker is actually participating in the "mail-ng" discussions.

L> AOL helps solve the problem.

JdeBP> ... and resurrects another problem as a consequence of doing
JdeBP> so, taking the world back to the state that it was in several
JdeBP> years ago. There is a fundamental conflict. "Call back"
JdeBP> schemes that verify envelope sender mailboxes require, in
JdeBP> order to be most effective, that envelope recipient mailbox
JdeBP> local parts be verified by the SMTP Relay server. But doing
JdeBP> so enables RCPT Harvesting (aka "Rumpelstilzchen attacks").

L> I believe that is a red herring.

No. It's an important point. Several of the various anti-UBM schemes have
fundamental paradigm conflicts with one another. This is one example of
that. Employing the measures singly incurs false positives. Employing them
in combination, however, incurs damage to SMTP-based Internet mail itself as a
result of the paradigm conflicts.

JdeBP> Implementing the measures that people have implemented to
JdeBP> prevent RCPT Harvesting over the past few years greatly
JdeBP> reduces the utility of "call back" schemes.

L> Well, empiricically, it doesn't correlate with your view.

"It"? Please provide antecedents for your pronouns.

L> The data shows there is VERY little organizations accepting mail
L> for post processing, because they have realized they will open
L> themselves to open relay and scalability issues.

Nonsense. Preventing RCPT Harvesting does not create open relay issues.

L> the operational decision to accept all mail at the SMTP level,
L> perpetuates the problem not only for the company but for the
L> industry as a whole.

Nonsense. Preventing RCPT Harvesting does not "perpetuate the problem"
(whatever "the problem" is - again you give no antecedent, so it's not clear)
"for the industry as a whole".

L> While it wasn't a big problem in the past, today, not validation
L> your address will increase scalability issues on your system and
L> eveyone elses as YAHOO have problem to show with a majority of
L> all spam having spoofed YAHOO addresses. So in my view, they put
L> on a burden on EVERYONE else as well.

You're confused. Preventing RCPT Harvesting has nothing to do with "spoofed"
envelope sender mailbox names. (Is the abbreviation "RCPT" in the name not
clue enough that it involves envelope _recipients_ ?) It appears that your
entire argument is based upon not understanding what RCPT Harvesting, and what
the mechanisms people have adopted to defeat it, actually are.

L> Also, as I said, its a red herring,

The argument that you have made up, based upon a misunderstanding of what RCPT
Harvesting is, is a red herring. But that has little bearing upon the actual
argument here.

L> YAHOO will validate the address at the DATA stage. So if a
L> spammer wanted to "havest" the address, all they need to do
L> is continue with the DATA transmission. They will find out
L> very fast whether it is good or not.

No. A response to a "DATA" command cannot be tied to a specific envelope
recipient. Indeed, it cannot even be tied to the envelope. A harvester
cannot hope to check the validity of mailbox names by looking at whether a
"DATA" command is successful.

L> In any case, with the exception of YAHOO, the odds are very
L> high that if a system accepts ROUTED MAIL in an anonymous
L> matter, it is more than likely a open relay. If it restricts
L> routed mail but accepts all local addresses, well there is not
L> much you can do here. But again, a CBV system is not looking
L> for true positives, it is looking for true negatives which offer
L> 100% and by far, the emperical data have shown that 80% of all
L> email are spoofed as 100% true rejected addresses.

Irrelevant. The point at hand is that call-back sender verification has a
fundamental paradigm conflict with the measures to prevent RCPT Harvesting
that have been adopted over the years.

L> CBV works!

JdeBP> No scheme to combat unsolicited bulk mail via SMTP works.

L> Will anything work 100%? No, [...]

The answer is only unequivocally "No." for SMTP-based Internet mail, because
of the fundamental flaw that that system has (which it has inherited from the
physical mail system). It is not necessarily "No." for other systems, which
may not have that fundamental flaw.

L> I am only interested in fixing an "anonymous access" problem
L> at the protocol level and it JUST so happens that by doing so,
L> you solve 80% of the spam problem. Thats a MAJOR improvement.
L> It has nothing to do what SPAM is or what the mail content
L> really is. It is anonymous access problem.

If you are only interested in fixing anonymous access, then you are only
interested in fixing _the wrong problem_. Anonymity isn't the problem.
(Anonymity, as I said, is a fundamental fact of life.) Your entire argument
here is based upon the flawed anti-UBM approach, that has failed time and
again, of looking for some element that is common to some number of
unsolicited bulk mail messages but that is not actually directly related to
their undesirable unsolicited and bulk qualities ("it just so happens that 80%
of unsolicited mail has quality 'X'"), and blocking all messages with that
element, unsolicited bulk ones or no. You are continuing to make the same old
mistake, and not learning that it _is_ a mistake.

L> SMTP has no business in the "interpretation of mail content" game.

You are creating straw men. You are the first to have even brought up the
subject of mail content.

L> We were given a mandate by CAN-SPAM:
L> - Validation of the Return Path
L> - Labeling.
L> The first one is easy. The second one requires a CHANGE to the
L> SMTP specification, if we want to offer a SYSTEM POLICY feature
L> to detect ADV tags at the protocol level. Otherwise, it has to
L> be done a the DATA stage or post processing.

Trusting the UBM senders to correctly label mail with some form of tag is
foolish. A law to demand that is as silly as a law that repeals the law of
gravity. Pursuing measures to implement such a daft law is a waste of time.

L> In my view, the problem is pretty simple :-) We just need to
L> best way to validate that address. With CURRENT specs, CBV is
L> the only way.

You are falsely presuming, in order to come to that conclusion, that address
validation is the requirement. It isn't. Anonymity isn't the problem.

[David Segall]

Post by wayne

Post by David Segall
You could save the sender some trouble by just replying with a "Thank
you for your email I will attend to it ASAP" message instead of
demanding a response. You would treat the original message as valid if
your reply did not bounce.

Even better would be to use something like Exim's "call back" system
where this is checked during the SMTP session. On my system, any
email that comes with a envelope from that is invalid gets rejected.
The domain must have an MX record, there must be a server on the IP
address, and the sending domain's MTA must be willing to accept email
from the envelope from.

How does Exim check the last of these? The only method I have read
about is an SMTP VRFY command and I am told that this is usually
disabled.

Post by wayne
While this is almost as expensive a check as generating a challenge
message or your "thank you" message, it has the advantage that no
actual message gets sent. Combined with all the advantages of
rejecting email during the SMTP session, I consider this to be well
worth the expense.
I'm pretty sure that other MTAs besides Exim have similar features.

I don't use a server at the moment but I am willing to do so if it
provides the ultimate in spam filtering.

Post by wayne
-wayne
-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----

[wayne]

Post by David Segall

[...] and the sending domain's MTA must be willing to accept email
from the envelope from.

How does Exim check the last of these? The only method I have read
about is an SMTP VRFY command and I am told that this is usually
disabled.

Exim uses a null envelope from (MAIL FROM:<>) and the RCPT TO: command
instead of the VRFY command. No email is actually sent because exim
doesn't send the DATA command. Really, disabling the VRFY command is
pretty useless because it is easily replaced, but it is unlikely that
it will ever be widely used ever again.


-wayne





-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----

[D. Stussy]

Post by David Segall
There have been a few posts recently which have advocated a
"Challenge-Response" system for minimising spam. The idea is to do
some preliminary filtering and, if you still cannot decide whether an
email is acceptable, you send an email requesting the user to respond.
If there is no response, you classify the email as spam.
You could save the sender some trouble by just replying with a "Thank
you for your email I will attend to it ASAP" message instead of
demanding a response. You would treat the original message as valid if
your reply did not bounce. This would require a little extra work to
identify bounce replies as opposed to, for example, "I'm on holidays"
replies.

Why? Bounces come from "<>" while vacation replies are likely to have the
mailbox of the recipient that is making the auto-reply.

Post by David Segall
The result would be more spam than a system requiring a response but,
given the preliminary filtering, would that be significant? It would
have the advantage, particularly for users who depend on incoming
email, that it would not discourage genuine email correspondence.

The "challenge only" system you describe above merely validates that the sending
mailbox is a valid mailbox. However, I don't see where it would confirm that
the mailbox identified as the sender is actually the sender. Spammers can forge
the headers and the SMTP envelope easily enough to get past that part. [Of
course, you mentioned that you would be trying to identify these beforehand.]

Other than verifying the username portion of the sending mailbox, I don't see
sufficient merit to this as compared to two other tests that have been proposed
in the past:

1) Simply looking up the domain name and checking for DNS resolvability to an
A, AAAA/A6, or MX record (which only validates the domain name part of course),
and its further step of actually pinging the host serving the domain to make
certain there's a real machine there, or

2) Starting to send (or use EXPN or VFRY SMTP commands, if not disabled) to the
actual sender's mailbox, but aborting the transaction before DATA (or sending an
"empty" message which may be discarded). Note that this method MAY or MAY NOT
verify the username portion - depending on how the server is configured.

Either method, the first if expanded to the point where machine contact is made,
may have a classification problem if the remote host is temporarily down or due
to a routing error or severe network congestion, effectively unreachable. The
actual sending of a message, considering SMTP retrys, doesn't suffer from those
issues.

However, how many people out there really like autoresponders to begin with?

[David Segall]

Post by D. Stussy

Post by David Segall
There have been a few posts recently which have advocated a
"Challenge-Response" system for minimising spam. The idea is to do
some preliminary filtering and, if you still cannot decide whether an
email is acceptable, you send an email requesting the user to respond.
If there is no response, you classify the email as spam.
You could save the sender some trouble by just replying with a "Thank
you for your email I will attend to it ASAP" message instead of
demanding a response. You would treat the original message as valid if
your reply did not bounce. This would require a little extra work to
identify bounce replies as opposed to, for example, "I'm on holidays"
replies.

Why? Bounces come from "<>" while vacation replies are likely to have the
mailbox of the recipient that is making the auto-reply.

I count detecting that as a "little extra work". Having thought about
it a bit more I would also count two "Mailbox full" messages 24 hours
apart as the equivalent of a bounce because most of my spam comes from
well known free domains.

Post by D. Stussy

Post by David Segall
The result would be more spam than a system requiring a response but,
given the preliminary filtering, would that be significant? It would
have the advantage, particularly for users who depend on incoming
email, that it would not discourage genuine email correspondence.

The "challenge only" system you describe above merely validates that the sending
mailbox is a valid mailbox. However, I don't see where it would confirm that
the mailbox identified as the sender is actually the sender. Spammers can forge
the headers and the SMTP envelope easily enough to get past that part.

From "my" point of view the C-R system solves this problem. However,
it is at the expense of genuine correspondents who must respond. It
makes no difference to the holder of a faked address since they will
receive an unsolicited email in either case. Unlike a C-R system I
could add a universally known standard string to my reply which could
be used to ensure that the recipient never sees the email.

Post by D. Stussy
[Of course, you mentioned that you would be trying to identify these beforehand.]
Other than verifying the username portion of the sending mailbox, I don't see
sufficient merit to this as compared to two other tests that have been proposed
1) Simply looking up the domain name and checking for DNS resolvability to an
A, AAAA/A6, or MX record (which only validates the domain name part of course),
and its further step of actually pinging the host serving the domain to make
certain there's a real machine there, or

I would use this as one of my preliminary checks but it would only
remove a tiny proportion of my spam; hotmail.com does have some MX
records.

Post by D. Stussy
2) Starting to send (or use EXPN or VFRY SMTP commands, if not disabled) to the
actual sender's mailbox, but aborting the transaction before DATA (or sending an
"empty" message which may be discarded). Note that this method MAY or MAY NOT
verify the username portion - depending on how the server is configured.
Either method, the first if expanded to the point where machine contact is made,
may have a classification problem if the remote host is temporarily down or due
to a routing error or severe network congestion, effectively unreachable. The
actual sending of a message, considering SMTP retrys, doesn't suffer from those
issues.

We seem to agree that sending a real message is the most effective way
of checking the senders address. My question remains. How much more
spam would I receive using the method I have outlined compared to a
C-R system?

Post by D. Stussy
However, how many people out there really like autoresponders to begin with?

[wayne]

Post by David Segall
From "my" point of view the C-R system solves this problem. However,
it is at the expense of genuine correspondents who must respond.

I'm not too concerned about the expense to genuine correspondents.
The problem with C-R systems is the expense they cause to random third
parties. This latter problem scales with the amount of spam being
sent and is already far more common than genuine correspondents.


-wayne


-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----

[Alan Connor]

Post by wayne

Post by David Segall
From "my" point of view the C-R system solves this problem. However,
it is at the expense of genuine correspondents who must respond.

I'm not too concerned about the expense to genuine correspondents.
The problem with C-R systems is the expense they cause to random third
parties. This latter problem scales with the amount of spam being
sent and is already far more common than genuine correspondents.
-wayne
-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----

My good man. You have been warned about posting ads on the Usenet.

Killfiled for 90 days.


AC

[Sam]

Post by Alan Connor

Post by wayne
-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----

My good man. You have been warned about posting ads on the Usenet.
Killfiled for 90 days.

Oh, no




 RUN FOR THE HILLS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

BEAVIS KILLFILED SOMEONE FOR 90 DAYS!!!!

How the world will survive? Oh the humanity, the humanity



AAAAAAARRRRRRRRRRGGGGGGGGGGGHHHHHHHHHHHHHH!!!!!!!!!!!!

[D. Stussy]

Post by Alan Connor

-----=3D Posted via Newsfeeds.Com, Uncensored Usenet News =3D-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----=3D=3D Over 100,000 Newsgroups - 19 Different Servers! =3D-----

My good man. You have been warned about posting ads on the Usenet.
Killfiled for 90 days.

Oh, no=E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6 RUN FOR THE HILL=

S!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

BEAVIS KILLFILED SOMEONE FOR 90 DAYS!!!!
How the world will survive? Oh the humanity, the humanity=E2=80=A6
AAAAAAARRRRRRRRRRGGGGGGGGGGGHHHHHHHHHHHHHH!!!!!!!!!!!!

Let alone that the guy didn't have a choice - that advertisement was one wh=
ich
was added by the news service he used, not by him as an individual.

Granted, ISPs and others which attach such to e-mail (I think hotmail.com s=
till
does that) and newsgroup posts need to be taught a lesson, but there's a pe=
rfect
example of another knee-jerk action by AC.

[D. Stussy]

Post by David Segall

Post by D. Stussy
...
Either method, the first if expanded to the point where machine contact is made,
may have a classification problem if the remote host is temporarily down or due
to a routing error or severe network congestion, effectively unreachable. The
actual sending of a message, considering SMTP retrys, doesn't suffer from those
issues.

We seem to agree that sending a real message is the most effective way
of checking the senders address. My question remains. How much more
spam would I receive using the method I have outlined compared to a
C-R system?

There's still a problem here. When one sends the challenge, he is verifying to
someone that his e-mail address is in fact valid. What is to prevent a spammer
from using [one of] his REAL addresses to receive the challenge - hiding among
many other messages that have forged return addresses? You're still VALIDATING
your address to a SPAMMER! That's a FUNDAMENTAL FLAW.

If the goal is to make your mailbox address "useless" to spammers even if they
have it, I don't see how that procedure accomplishes that fact. Is your system
smart enough (or are you, depending on how many messages you get) not to
whitelist a spammer that actually replies to your challenge message correctly?
Even if the spammer is whitelisted for a small period of time before you realize
your mistake and move him to the blacklist, you've still opened yourself to the
receipt of his spam in the meantime (since whitelisted senders bypass the spam
filters, at least according to some implementations offered here). All he needs
is to get ONE of his messages read by you to "win." (Of course, a "greater win"
is accomplished if he makes a sale of his product or service, or for his
customer.)