It seems very unwise for Microsoft to siphon private and sensitive links like that.
A partial defense for websites may be blocking those links with robots.txt. Bing is supposed to respect that.
Also if you’re implementing a one-time action, make sure it’s done only in response to a POST request, not just GET. The GET request should return a <form> and optionally a bit of JS to auto-post it to make it look seamless for users (bots, even ones that support JS, are not supposed to be making POST requests).
I agree that this is not a great thing for a mail service to be doing, but it’s also worth noting that email is often sent plaintext between mail relays (with opportunistic encryption, at best, and so vulnerable to downgrade attacks), so relying on it for confidentiality is also not a great idea. A lot of mail services will visit links from malware scanners, so if the user just needs to click on the link to be authorised, you might get some spurious hits. You used to be able to track where an email went quite easily by adding a link to a PDF in it and watching all of the things that fetched the PDF to scan for malware.
Note that Microsoft 365 has a ‘users with this link can access the document’ sharing mode, which will send links via email. It’s probably worth checking what this does to prevent the links from being indexed. I don’t know if rel=nofollow works in HTML emails?
These “magic links” are a form of unguessable capability token, and need to be handled carefully. For more information about this, see the W3C’s “Good Practices for Capability URLs”.
What I find shocking is that people are still shocked by this.
We apparently have to do more to educate the masses that email is 1) not secure, and 2) not private. Especially when google, microsoft, etc are involved. Small email providers may ignore user email contents, but the big players actively use it to game ad clicks, and are thus heavily incentivized to not ignore mail contents. In any case, my previous two points still apply, regardless of the provider.
I wonder if some sort of crackdown on e-mail hosting providers would work. Like, if Google, MS and all those others had to ditch their mail services and have them operate independantly, like fastmail or something.
I’ve wondered that and came to the conclusion that they would just go away, because what’s the point for any of them to run financially-free services if they don’t pay their way in data? I mean, I wouldn’t mind that, and email might be actually federated without capricious black holes dragging everything inside them, but … doesn’t seem too likely.
I think I already heard something similar years ago. If I remember correctly, back then the suspicion was that sending a link via Skype could lead to it being indexed by a crawler.
From a provider perspective it’s probably a way to index the deep web.
From a user perspective it’s an intrusion into privacy. But then again, MS is a US company and I have no trust in US companies (and US government) whatsoever.
From a security perspective the problem is the HTTP GET login link. Even when I did web development (which is lots of years ago) I learnt that sensitive information should never be included in URLs.
This is really interesting to me and explains why so many frameworks recommend a POST form of some kind on the page that gets loaded.
It also explains why email ‘one click unsubscribe’ quite often does some weird stuff. e.g. Mailchimp loads an ‘unsubscribe’ form then ‘clicks’ the button for you. I suppose otherwise people would be getting auto-unsubscribed all over the place.
I’m not seeing this, at all, with a similar setup.
I wonder if it isn’t just the Microsoft URL security/malware scanner that is visiting the URLs, rather than Bing indexing it and seeing random visitors.
When using magic links like this, always use a POST, or require it to be opened in the same client (uniquely identified by a cookie).
Our company has put in place security training, which include sending fake phishing email that we are supposed to report without clicking on the “free iPad” link. For the first few months, everybody had abysmal score of 100% malware link clicked.
Why?
Office 365 security scanner systematically followed the link to inspect it for malware. So the poorly designed software assumed that we had clicked it. We complained and the training provider fixed it. I will note that this was not a free account but a big business paid for email service. They are certainly not the only security service to do this.
So yes : “link in emails are opened by humans” is another one of those “thing programmer believe” that are completely wrong. You should avoid doing anything sensitive with them.
Sending fake phishing emails to your own employees is a good sign that your CTO has no real work to do and should be fired. It’s just obviously a waste of time. Just add the external link banner to the emails and be done with it.
I see where the bing spider accesses the link. That’s bad enough but we expect that.
But they didn’t document any actual bing search results containing those links. That would be even more concerning but are we sure bing is actually returning those links in other peoples’ searches?
It seems very unwise for Microsoft to siphon private and sensitive links like that.
A partial defense for websites may be blocking those links with
robots.txt. Bing is supposed to respect that.Also if you’re implementing a one-time action, make sure it’s done only in response to a POST request, not just GET. The GET request should return a
<form>and optionally a bit of JS to auto-post it to make it look seamless for users (bots, even ones that support JS, are not supposed to be making POST requests).I agree that this is not a great thing for a mail service to be doing, but it’s also worth noting that email is often sent plaintext between mail relays (with opportunistic encryption, at best, and so vulnerable to downgrade attacks), so relying on it for confidentiality is also not a great idea. A lot of mail services will visit links from malware scanners, so if the user just needs to click on the link to be authorised, you might get some spurious hits. You used to be able to track where an email went quite easily by adding a link to a PDF in it and watching all of the things that fetched the PDF to scan for malware.
Note that Microsoft 365 has a ‘users with this link can access the document’ sharing mode, which will send links via email. It’s probably worth checking what this does to prevent the links from being indexed. I don’t know if
rel=nofollowworks in HTML emails?I just looked at one of the 365 share emails I received recently and they do not have any
relattributes on the links that open the document.I’m in the middle of implementing such a system and my immediate concern was this.
Have you seen it happen, using a
<form>and POST?I don’t think I’ve ever seen an HTML email with a form in it.
Sorry, I wasn’t clear, the form is on the webpage. So
In theory bots don’t use POST, I’m curious about in practice whether this actually works or not
These “magic links” are a form of unguessable capability token, and need to be handled carefully. For more information about this, see the W3C’s “Good Practices for Capability URLs”.
Free email providers, mining supposedly-private data and causing problems by accident? Color me shocked.
Surprised we don’t see more of this, tbqh.
What I find shocking is that people are still shocked by this.
We apparently have to do more to educate the masses that email is 1) not secure, and 2) not private. Especially when google, microsoft, etc are involved. Small email providers may ignore user email contents, but the big players actively use it to game ad clicks, and are thus heavily incentivized to not ignore mail contents. In any case, my previous two points still apply, regardless of the provider.
I wonder if some sort of crackdown on e-mail hosting providers would work. Like, if Google, MS and all those others had to ditch their mail services and have them operate independantly, like fastmail or something.
I’ve wondered that and came to the conclusion that they would just go away, because what’s the point for any of them to run financially-free services if they don’t pay their way in data? I mean, I wouldn’t mind that, and email might be actually federated without capricious black holes dragging everything inside them, but … doesn’t seem too likely.
I think I already heard something similar years ago. If I remember correctly, back then the suspicion was that sending a link via Skype could lead to it being indexed by a crawler.
From a provider perspective it’s probably a way to index the deep web.
From a user perspective it’s an intrusion into privacy. But then again, MS is a US company and I have no trust in US companies (and US government) whatsoever.
From a security perspective the problem is the HTTP GET login link. Even when I did web development (which is lots of years ago) I learnt that sensitive information should never be included in URLs.
This is really interesting to me and explains why so many frameworks recommend a POST form of some kind on the page that gets loaded.
It also explains why email ‘one click unsubscribe’ quite often does some weird stuff. e.g. Mailchimp loads an ‘unsubscribe’ form then ‘clicks’ the button for you. I suppose otherwise people would be getting auto-unsubscribed all over the place.
I’m not seeing this, at all, with a similar setup.
I wonder if it isn’t just the Microsoft URL security/malware scanner that is visiting the URLs, rather than Bing indexing it and seeing random visitors.
When using magic links like this, always use a POST, or require it to be opened in the same client (uniquely identified by a cookie).
Fun story:
Our company has put in place security training, which include sending fake phishing email that we are supposed to report without clicking on the “free iPad” link. For the first few months, everybody had abysmal score of 100% malware link clicked.
Why?
Office 365 security scanner systematically followed the link to inspect it for malware. So the poorly designed software assumed that we had clicked it. We complained and the training provider fixed it. I will note that this was not a free account but a big business paid for email service. They are certainly not the only security service to do this.
So yes : “link in emails are opened by humans” is another one of those “thing programmer believe” that are completely wrong. You should avoid doing anything sensitive with them.
Sending fake phishing emails to your own employees is a good sign that your CTO has no real work to do and should be fired. It’s just obviously a waste of time. Just add the external link banner to the emails and be done with it.
This is really interesting to me. I’m in the middle of implementing something like this and in my research keep finding stories like yours.
I’m thinking of emailing a one-time code to the user instead of sending them a link now
[Comment removed by author]
I see where the bing spider accesses the link. That’s bad enough but we expect that.
But they didn’t document any actual bing search results containing those links. That would be even more concerning but are we sure bing is actually returning those links in other peoples’ searches?
Scribe link
I thought this was common knowledge.