I honestly don't think BS will take a reputation hit, at least amongst those who it cares about. This is as good as it gets.

BS attacks spammers.
Spammers DOS BS.
DOS attack makes news.
Ordinary people think, "if spammers are attacking them, what BS are doing must be working."
Loads more people sign up for BS.
The cycle continues.

There will only ever be more spam, BS has an agenda, and VC's to appease. Nobody knows or cares about how the internet actually works, except the geeks.

Todd -- great post, and one that was much needed!

couple of things:

1. That skybox spam is almost definitely a joe-job; one in which the senders
have finally figured out how to operate a spellchecker, but a joe-job
nonetheless. It doesn't really add up in other respects.

2. 'the loss to reputation that BS has already suffered' -- on the contrary,
this DDoS suits BS' purposes perfectly. They get to appear embattled, they get
to use Churchillian "fight them on the beaches" rhetoric, and their customers
love it. The people among whom BS has lost reputation, are not the people who
would have bought their software anyway.

It's pretty clear that the story has been changing day to day and this finally sets it in stone. It's important for these articles to show that we don't accept vigilante justice on the Internet. We never have. And it's not just the geeks who care. 104K Tucows users cared.

I'm particulary enjoying Wired's coverage of "I'm the Blue Security Spammer". Right. That's about as likely as me being Santa Claus.

Good article and thanks.

It's worth noting that several SPs, including Verizon/MCI/uu.net, offer DDoS protection services.

Roland,
It seems that, based on http headers/responses, when the BS folks switched over to SixApart the SixApart folks seemed to have some form of 'mitigation service' in place with atleast one of their providers... The one that supposedly was so bad to BS... Did BS even know this? Has BS followed up with their provider on what happened/didn't-happen?

Shouting fire in a closed theater is a crime, being dumb and not asking the first hop in your ISP world what happened is similarly bad form when tossing about accusations of criminal behaviour.

Thanks Todd for the interesting writeup on this incident and for Martin pointing out the two larger issues: TuCows DNS service destruction and vigilantism being a poor solution to any problem.

Your succinct and dispassionate analysis from above the fray will be dismissed by the BS masses as another "so-called expert" and wimp who doesn't understand the genius of a BS system nobody in the world but BS and its followers can truly understand and appreciate.

You lost me on one technical detail. I don't understand how you engineered your site to avoid insults, obscenities and thinly veiled threats.

There's really nothing above the fray. Suppose you were on a mail system with 104K other users and one of them was a prolific spammer. If enough BS users complained about the single spammer, BS would cause enough collateral damage to knock down the 103,999 other legitimate users without any concern, check, or design. This is not how network operators and others choose to conduct operations. It's unnaceptable, and has been since Green Card Lawyers and beyond.

Is it fair to cause collateral damage without care? Is that in the interest in the reliability and security of the Internet? No.

As far as noting that a provider has DDoS capacity, that's irrelevant. BS was NetVision's customer. If people in the security community were advocating any sort of intervention outside of either BS->NetVision or NetVision->Provider then they are continiung their support for this illegitimate service. Interferring otherwise is impacting revenue and policy of service providers and supporting what I is the equivilant of an internet terrorist organization.

The bottom line is under no circumstances can BS justify their actions and neither can their supporters. Frankly, it makes them look even worse. If that's possible.

Please do notice how I left out the anti-spam community from this posting. If the reason why isn't obvious, it should be. They don't support Blue Security.

Martin, Martin, Martin.
Power down. You're preaching to the choir.

On my site, we've covered this story to the point we've run out of words. And still BSers comment: "Why is everybody picking on BS?" (and much worse)

Fact is, this is the first time we've received comments so obscene we were unable to doctor them enough leave them up.

In the end, we're all battling a religious sect with logic. When was the last time that worked?

BJ, no power down needed. This is how it sounds when you speak from a position of technical and political knowledge. This is what is driving the Blue Security folks mad. Facts.

I'll have to talk to Todd to see if we've received similiar un-publishable comments. It's likely. There's a parade of alleged BS users following us around these days. :-)

Best,

Martin

Hi Martin.
By use of the phrase "above the fray," I meant "from a superior viewing position." Todd has more information available to him, as well as the technical expertise to logically sort through it. As you have noted, others viewing from Todd's vantage point also surely have those abilities.

But IMHO, Todd's ability to clearly share his insights with the greater world in an engaging way sets him apart. Facts + logically applied technical knowledge + wonderful communication skills = TU.

Nevertheless, I don't believe facts have much to do with decisions made by BS followers. Take this guy, for example:

"personally i don't think BF was doing me any good but must be pissing off the spamers .. THESE PPL ARE SCUM i have been sent child porn (which i have reported) Swiss watches, Viagra, everything u could passably imagine.. i would quite happily pay some one to get their home address as i would fly half way around the world just to seek my own revenge .. what ever anti Spam groups do to these ppl is not enough .. DOS THEM, HACK THEIR WEB SITES, SUE THEM, TAKE THEIR HOUSES AND CARS OFF THEM, they have my FULL support, these spammer are total scum .. say what u like but until your business email account is shut down because u are getting flooded full of crap everyday u do not know what u are talking about .. i was not going to bother using BF any more until i got these treating emails from the spammers now i am going to send them every email i get just because i know BF pisses them off as much as there garbage dose to me far is far paybacks a b*tch !!!"

Translation: "Although I have proven to myself that BS is ineffective on my own corpus, I will continue using BS, because revenge is sweet."

True believers couldn't care less about anybody's "facts." Those are simply lies made up by the Powers That Be to undermine the Community. And as every Community knows, attackers must be attacked to protect the Community.

Having said that, we have noticed that a number of the postings are uncannily similar, on our site as well as others. Hmmmm...

dear geniuses,
i'm afraid you missed a few points. i know little/nothing about blackhole filtering, but since i have been recommending bluesecurity for a bit of time i do know a bit about that. the system through which illegal spammers receive a huge amount of complaints and unsubscribe requests is a last resort. it exists almost entirely as a deterrent. before that happens bluesecurity identifies the spammer and the spamvertised site, as they call them, and request them to sign up for a service that protects the bluesecurity members. essentially a "cease and desist" message. following that bluesecurity notifies various antispam groups, government agencies and other groups including:
-the internet registrar of the site
-the isp hosting the site
-the credit card clearance service
-the e-commerce solution provider
-law enforcement agencies
-FBI
-Interpol
-FTC
-DEA
-SEC
-MPAA
-FDA
-anti-spam groups
-Mcafee
-Symantec
-others
-BSA
-SIIA
-Microsoft
-Adobe
-Macromedia

if that doesn't work and the spammers are still up and running and not signed up for the bluesecurity service then, sometimes, they will send the huge amount of messages. As i said before this is mostly intended as a deterrent, and it's been working. Spammers representing close to 25% of spam produced on the internet have already signed up for their service, and send-safe a very popular spam sending software has added compliance with bluesecurity as one of their features:
http://community.bluesecurity.com/webx?14@943.10Yja7RllEr.1@.3c4caf92!comment=1&full=1

I prefer Illuminati over Genius. The others may like Genius. I'll leave that up to him. Genii sounds pretty cool too.

I think the Blue Security discussions are over since there is no Blue Security as released on the wires this evening.

Best,

Martin

Spin doctors at work:

http://wired.com/news/technology/0,70913-0.html?tw=wn_index_1

http://www.washingtonpost.com/wp-dyn/content/article/2006/05/16/AR2006051601873.html

Vulture Capitalists tend to not let folks just say "gosh we don't want to cause attacks!" Smells more like big money saw the burn rate for appropriately defending the service and declared it a money loser.

The Wired article shows how useless Wired continues to be: Starting May 2, a spammer known as PharmaMaster [...] hijacked a little-known Cisco Systems router feature known as "blackhole filtering" to block anyone outside Israel from accessing Blue Security's homepage.
Idiots.

At least the Post indicated it was all about 'fighting fire with fire'.

Ha, I see that the bluesecurity PR flacks and mindless sheep astroturfing of any blog that posts facts [aka points out that BS is full of.. well, BS] has begun.

Enjoy it while it lasts.

Thanks for this analysis. One
unanswered question is "who launched this attack and why?"

Oh, I'm well aware that BS
has claimed that the attack
originated with an enraged spammer and was an attempt to
thwart BS...but I don't believe
that. Not even a little bit.
No competent spammer -- and
there are many -- would be
in the least bit troubled by
the mere pinprick that BS
represents...err, represented.

Moreover, a competent attacker
wouldn't bother identifying
themself -- in any fashion,
especially not in one that
touts the "effectiveness"
of BS.

And finally, a competent attacker would have something
more than a SYN flood at their
disposal.

So who _really_ launched this
attack, and why?

I'm not sure we'll ever know
that. Maybe it *was* an enraged spammer. But I'm not
willing to accept that as the
only possible explanation,
especially when it's quite clear
that the BS propaganda machine
has been hard at work.

Todd,

In your blog, you wrote: "...skybox security, a company that focuses on helping companies simulate and survive Distributed Denial of Service (DDOS) attacks".
You have fallen a victim to the spammers' propaganda. Check our Skybox's web site. They do Security Risk Mgmt. Nothing to do with DDoS. Perhaps some other facts in your blog are a bit incorrect as well?

It's funny how different "experts" describe the same event....

"Let's clear one thing up for the press and everyone else: this event just wasn't that interesting. The attack against bluesecurity was a run-of-the-mill denial of service attack. That's actually one of the funny aspects of this story."

From the Washington Post article...

"Tucows chief executive Elliot Noss called the attack "by far the largest the company had ever seen," and said that only a handful of companies have the infrastructure in place to withstand such an assault, much less a more powerful one."

I'd like to hear your comment on this, Todd.

I believe bluesecurity has started a new chapter in the war of spammers. There actions (from an advanced users point of view) is the most drastic and controversial step ever taken to control all the garbage/crap all of us get in our e-mails. I am a proud member of blue frog and believe strongly in what they stand for, spammers should be prosecuted in my opinion, there actions are disturbing, morally wrong-->porn spammers, and most of the time illegal business practices. All of these websites (around the world) should be shut down and the supporters sued and the ones who started the website given jail time, if this was done just once or twice I believe the spammers would look for legitimate/legal & moral ways to make a living instead of being sleezy, low lives that prey on pushing there scams/junk on the rest of the world. Bluesecurity/Bluefrog will be back, COUNT ON IT their push us and get shoved back policy is just what we need. I will support anti-spam tactics whatever they are, any way I can.

Dear Todd,

Up until recently, like many people fed up with spam, I was a Blue Frog user. When the attacks started I was critical of some of your quotes reproduced in several IT news outlets. I didn't write anything mean about you on the CastleCops boards, I just said a few things that indicated I was peeved with people saying negative things about Blue Security, regardless of whether or not they were true or false. The only reason I felt that way was that BSec was the first thing to come along that was actually working for me in terms of reducing spam, and I was rooting for my "home team".

I wish the BSec concept had worked, or that it might be modified to work in the future... but my point here is that now that the dust has settled and I've calmed down, I want to tell you that I've come around to realizing that everything you've said has been on the level, well thought out, and true (your 5/8 blog and more): that some of the statements made by BSec about the details of the attack don't hold water, that they were ripe for retaliation from spammers, (and here's what really pisses me off) that they had done zero planning and had not done anything to protect themselves from the inevitable retaliation (shared server etc). I'm so disappointed in BSec for this.

Anyway, thanks for your level-headed input during this whole thing. My emotions got the best of me as I tried to root for the home team. I'm sorry.

Keep up the good work. Some day the internet's forces for good will overcome the forces of evil,

John

Isnt it possible to propagate a 'null route' (for lack of a better term at this moment) via BGP? Especally if a device was misconfigured to accept a bogus broadcast?

I am just sayin... thats the only thing I would like clarified further in your post where you say: "The key thing to note about blackhole routes is that they do not propagate from provider to provider. It only affects the single provider who installs it."

I recall in the mid 90's when Netcom screwed up and put a comma in a BGP broadcast and it hosed a nice portion of the US 'net for a few hours.

It propagated and as a result, routes got screwy. So using this logic today, one could potentially access a BGP realm and insert false information that, if taken as legitimate, could be passed on.

Everything else adds up though :) thanks for your in-depth analysis.

Quick followups:

--Skybox asked me to post saying that they were emphatically not involved in this spam and that they are not evil spammers. It's a joe job. Makes sense.

--Skybox's website has several references to "simulating network attacks". Perhaps i misread that as focusing on simulating and designing to mitigate DOSes. If so, I appologize for mis-representing the product line of a company I never claimed to know much about.

--When I said: "The attack against bluesecurity was a run-of-the-mill denial of service attack...", it had a specific context. The CEO of Blue Security was making bizarre claims about "tier-1 backbones" being hijacked and changed by a nefarious spammer. What I was trying to point out was that this was not the case. This was a simple SYN-flood DOS. The fact that it was fairly large doesn't change that.

--null-routes don't propagate via BGP. Null is the *target* of the route. It is a synthetic interface that throws away traffic. Router interface targets do not propagate via external routing protocols.

--People who write things like "I will support anti-spam tactics whatever they are, any way I can" are clearly idiots. If anti-spammers decided to kill everyone on earth in order to make sure that they got the spammers, would you support that. Again, the word "nonsense" keeps coming up in the context of this story.

I think that's it. I'm bored with this story now. Aren't you? :-)

I like BS wrong or right I use to get tons of spam by inconsiderate SOBs. I dont want to buy viagra at least for 20 or 30 more years I have a strong jonson O.K. I tried everything to get them to stop I sent emails at thier website of orgin and they still came. Until bluefrog I dont unserstand what they do but I hope they come back. I would also like to see the law actually find these people and throw them in prison for 10 years and send a message to the world that spam will not be tollerated. I mean really it takes a few minuts to add a remove link and remove non customers from their list. It is just plain rude to keep bombarding people with thier sale adds over and over and no way to make it stop. Bluefrog works and I hope to see them come back again soon. Better and faster. If you need to shut down a few servers to get the point across that we dont want spam then so be it. Shut them down. Bottom line is none of us want 30 viagra adds in our email each day every day especialy when your a femail or a male with a strong long like me. Its time to fight back

Thanks Todd :)

Not being a BGP expert by any stretch, perhaps "null" is not the correct phrase.

I was speaking generically... Cisco = "Null0" from what I recall.

Anyway I was just curious about what you stated. Cuz in Juniper one can do:
set protocols bgp group BGPGROUP export blackholeroutes

Ok enough.

But thanks for your analysis and I await the next blog entry. Theres a whole P2P thing now that obsoletes this thread and yea I am bored already :)