Microsoft antispam: incompentence squared For my sins, I am among other things a network administrator. A long time ago, I decided to maintain and support the email service for my users in two closely linked local networks spread across a handful of domains. Handling your own mail these days of course means that handling spam and malware comes with the territory. After a while, you learn to deal with it, and after doing it all for about a decade, making incremental improvements along the way, I feel that I have it pretty well covered. The last time one of my Microsoft users clicked on an attachment that turned out to be a spam sending trojan was in June 2005, and then no messages actually made it out to the rest of the world. So it will probably not surprise you that I was a bit baffled last week when I found that Microsoft, in the form of its Frontbridge.com "Hosted Exchange" mail service, is accusing me of being a spammer. Not in so many words or to my face of course. The way I found out was via a customer of ours who also happens to use Frontbridge.com for some email services, in their case the spam filtering package. This is a customer we've been working with for I forget how many years, based in another European country. The work we do for them involves among other things sending small bits of information to them at the end of some procedure. And certainly while while we are debugging something, email is our main form of communication. Around December 7 or 8, 2006, strange things started happening. Both of my main contacts at the customer site started resending messages I had replied to several hours earlier, sometimes adding "Peter, did you get this? Please respond!!!" or similar at the very top of the otherwise duplicate message. This was while we were trying to diagnose a particularly weird set of error situations in the setup procedure for the product we were working on. In that particular phase of a project (yes, there always is a debug period), we have exchanges of the type "I did X, and then Y happened while I was really expecting Z or perhaps W" followed by "OK, try going A C B instead of A B C, or perhaps skip all the way to F, see what works", and so forth. If you've been working in developent and testing, you know the exact type of email pseudo-conversations I am talking about. Only this time, I did not get the responses I was expecting. Instead, I got repeats of earlier messages with "Are you getting this? please respond!" added at the top. I finished up what I needed to do as best as I could on the Friday, wrapping up and leaving the office at about my usual time. The weekend was filled with other things, but catching up on my mail on Monday morning I noticed the increasingly concerned tone in the notes added to the resent messages which had accumulated over the weekend. Fortunately my mail server keeps its logs for seven days before rotating them out of existence, so it took me only a few minutes to check that all the messages I had sent to these addresses had in fact been accepted by their mail server. So I called the customer and told them that not only had I received all their messages, I had responded to most of them too. Over the next few days I got into the habit of printing and faxing all email messages I sent to this customer, and at some point, probably late Tuesday, one of my main contacts told me that the spam filtering system they were using had marked all datadok.no messages as spam and quarantined them. Not to worry, was the message from their admin, all they needed to do was release the messages from their holding area every now and again, and then start training the system that my messages were not spam. My reaction, fired off via email which for some reason got delivered almost like there was no spam filtering, is the strongest worded exchange with a customer I have ever had in my career. Most of it is not worth repeating, except "One false positive is a serious matter. Recurring false positives means you either fix your system so they do not happen again, ever, or you deserve to be out of business super-quick." I also had a few items like "fixed in the barnyard sense" and "you bought yourselves a steaming pile of poo" in there. After a few more exchanges, excuses, apologies, bickering and periodic releases of my messages from the holding pen, at some time Wednesday my customer's mail admin finally filed a problem report with Frontbridge, repeating my demand for all information triggering any "Spam" classification for all our domains. I was Cc:d on the message, and I added a few comments with some specifics in a followup message which was most likely ignored by Frontbridge, if it was received at all. And of course, all the while we were trying to debug that silly product, which was still not behaving as expected. Early on the Thursday I wrote in a message to my main contact at the customer site, "The morons at Frontbridge are not helping either - the most likely cause for their blacklisting is an incident on November 22, when the machine at our ISP which hosts www.datadok.no was cracked, with the perpetrators installing a paypal phishing kit as http://www.datadok.no/%7Etest1/cgi-bin/webscrcmd=_login-run/update.php. It's safe to click that link, by the way, it will show you a File not found message. On the morning of November 23, I (wearing the abuse@datadok.no hat) had received a handful of messages about this overnight, and at shortly before 10AM on November 23 the files had been removed, but preserved as evidence. The complaint we filed the next day with the local police is still in queue for investigation, and for all I know it had already been lost or fed to the resident leopard before Microsoft, in their Frontbridge incarnation, roughly two weeks later took it upon themselves to tag all mail with a datadok.no From: address as spam. This tells us that * they probably trust From: headers like gospel * they do not check their data sources * if they run checks for relaying et al, they are unable to interpret the results * but of course they will take your money anyway Of course I haven't heard back from support@frontbridge.com - they most likely follow the proud Microsoft tradition of "eat your own dogfood" and never saw my messages. Now of course if Microsoft had bothered to actually check their data, they would have gotten the same 404 you are getting now." Some of the assertions there my be inaccurate, but mostly they are consistent with available data. For example, while I did not see any signs of relay testing other than the ones I initiated myself, they may have happened at a date my preserved logs did not cover. Anyway, a little later the same day I decided to contact Frontbridge directly, by fax this time. I wrote up a fax describing the problem in my politest businesslike English, telling them to "treat the matter as a quality problem" on their part, ending with a demand that they give me any and all information in their possession relating to all our domains, "in order to correctly assess the situation and correct any misconfigurations", "at your earliest convenience but no later than Friday December 15 2006, 12:00 CET." The whois information for frontbridge.com[1] lists a fax number which is handled by something which answers but does not actually receive faxes. The phone number appears to be permanently off-hook. The administrative and technical contact email is a microsoft.com address. I ended up converting my faxes to pdf files and sending them to the administrative and technical contact address, and faxed the documents to the fax number listed at www.microsoft.com's contact information page. The rest of the Thursday I spent attending to other matters from my backlog, and I woke rather later than normal on Friday. No response from anywhere in the Frontbridge/Microsoft system. The Norwegian Microsoft subsidiary is about 500 kilometers away in Oslo, so I called them and got to speak to the their managing director's assistant, who patiently hovered near the fax machine until my Frontbridge fax got through. It took only a few minutes before they called back. This time somebody from "Response Management". I essentially repeated what was in the fax I had sent, and in response I got this gem: "I Microsoft Norge har vi ikke teknisk kompetanse." (At Microsoft Norway, we do not have technical competence). I thought the conversation had ended on a useful note with the Microsoft man promising to sort things out to the best of their ability. Instead, he appears to have fallen into a fit of panic, sending messages to my customer essentially asking "who is this madman who is threatening to go to the media unless we meet his demands?". At least that is the way I read the quoted parts from the message I got from my customer. I suppose this is the expected reaction in a marketing trained person who gets exposed to an overdose of tech talk. My customer was getting concerned that they were getting into trouble with Microsoft, and I had to calm them all down with "don't panic, hug your towel if you need to, once Frontbridge aka Microsoft fix their config the problem goes away" messages, with a side note to the Microsoft man "don't panic, please keep focused on fixing the problem". A little later a I was Cc:d with a message from frontbridge.com to the mail admin at our customer's site, indicating that they were considering looking into the problem, but at the same time claiming that "At this time we do not do any domain based blocking." Based on what we have been seeing, I find that statement a bit hard to believe, but I do not have the data yet to prove decisively that they are telling less than the truth. The statement could of course mean that they are filtering on blocks of IP addresses rather than domain names. That could explain a lot too. If a machine belonging another of our ISP's customers actually sent spam from an IP address close enough to ours, Frontbridge could have decided to block an entire /24 or /16 netblock, including our /26 and /29 address ranges as an accidental side effect. It's not pretty, but at least it is plausible and consistent with observed behavior. The message went on with a few sentences about how to operate what sounded like a hugely complicated GUI adminstration tool in order to rescue the blocked messages. All the while, they were treating this as a local problem at my customer's site. From their message it is fairly clear that they either did not understand or did not care that the data they were feeding their customer's systems was producing false positives, blocking legitimate mail. At the time I am writing this, there is still no word back from Frontbridge about progress, so in all fairness we should call this 'a matter still unresolved'. That is, if they still care. If the cracking and phishing incident is what caused this problem, it is an amazing story indeed. At the time several people wrote to me about the spamvertised phishing, one even apparently considering using the link to approve whichever operation it was about. But who in their right mind would take the domain of a url quoted in a spam message and turn it into a strong enough indicator of spamminess to block messages? We are back to the point about the hosting company's incompetence - setting up a user called test1 with a guessable password and no bruteforce protection, with predictable results - generating data which gets fed into some other incompetent's system which then by automatic arrogance classifies all mail from a domain as spam. So we have incompetence feeding incompetence in a sickening feedback loop. I suppose cruder men would be tempted to call that incest, I prefer to call it incompetence squared. Or perhaps even incompetence cubed, if we factor in the amazing marketing induced arrogance which amplifies it all. So it looks very much like what we are seeing is really an organization which treats every problem like a marketing problem and makes no effort at all to be accessible to anyone who has not already paid for the privilege. Even then, they appear not to care if the service they are selling operates correctly. I wonder if they can be made to behave like grownups, eventually. [1] 'whois frontbridge.com' from my systems produced this contact information on Thursday, December 14 14:40:57 CET 2006: FrontBridge Technologies, Inc. 4640 Admiralty Way Ste 888 Marina del Rey, CA 90292 US Domain Name: FRONTBRIDGE.COM Administrative Contact, Technical Contact: FrontBridge Technologies, Inc. cmc@microsoft.com 4640 Admiralty Way Ste 888 Marina del Rey, CA 90292 US 310-302-0522 fax: 310-301-6032 Record expires on 12-Dec-2011. Record created on 08-Dec-2004. Database last updated on 14-Dec-2006 08:43:28 EST. Notice the database last updated date. There is every reason to believe that this was supposed to be current information.