I ran into an interesting problem with the Microsoft Bing search engine (used by DuckDuckGo, Yahoo, Amazon’s Alexa and many others, so about 5 – 10% of my organic traffic is affected). My websites just got erased from their search results – overnight! Simply completely disappeared from Bing’s search results. This very website, among others (all on the bikegremlin.com domain). Let’s dive in:
Table Of Contents (T.O.C.):
- The problem – pages gone from Bing SERP
- Investigating the problem
3.1. Site scan works (for now)
3.2. Indexing issue
3.3. Sort of back… but not really
3.4. Manual unblock?
- List of Bing-copy search engines
4.1. Search engines that apparently don’t rely on Bing
4.2. Search engines that use Bing’s index/discovery
- Bing broken – again!
1. The problem – pages gone from Bing SERP
Today I logged into my Microsoft Bing Webmaster dashboard and saw this:
All the pages were completely gone from Microsoft Bing’s search result pages. All the websites on my bikegremlin.com domain were affected (bike.bikegremlin.com, io.bikegremlin.com etc.). The rest of the websites were fine. Here’s a more detailed graph for one of my websites (bike.bikegremlin.com):
2. Investigating the problem
OK, it seems that the 24th of June, 2022, is when it suddenly went blank, zero. The first question to ask when something like this happens is: “what did you do/change/update?” The answer is, as far as I remember and can tell – nothing.
Not until July that is – in July I did a Setupad experiment, then tested Cloudflare Pro and WordPress APO. But this problem started in June, so I don’t think it’s related.
Next, I checked with other search engines:
- Google – all good and normal.
- Yandex – all good, all normal.
- Baidu – my website isn’t there, but I hadn’t checked before to confirm when it happened.
- DuckDuckGo – it’s a Bing-clone practically, and my sites are gone from the Duck as well!
- Yahoo – same, none of my websites there.
OK, it could be just Bing-related.
I took a dive into Bing’s dashboard:
- Checked other domain websites that also use Cloudflare – to confirm it’s not Cloudflare blocking Bing’s bots. Websites on other domains work normally.
- Checked for any error reports and similar.
- Checked and re-indexed sitemaps, though they all looked green, OK.
- Ran URL inspection for a few URL-s – all good.
- Tried Bing’s robot tester (Tools & Enhancements -> robots.txt Tester), tried a few test URL-s – all good.
- Ran Site Scan (SEO -> Site Scan). This showed one error, stopping right away, at the home page: “ERROR : Http 400-499 errors”.
That’s… “brilliant!” It would have made more sense had they provided the exact error (404, 410, or something third):
After going through the troubleshooting FAQ, I found nothing about a problem like this – so I’m writing this article. 🙂 And, I decided it’s time to contact support. Got an email confirming they have been notified and now all I can do is wait.
Another brilliant thing I noticed: Bing’s documentation/knowledge-base/FAQ has no search option! Ironic, isn’t it? 🙂
To add insult to injury, when you “Google” the term “BikeGremlin” on Microsoft Bing or DuckDuckGo, SERPs show BikeGremlin social-media accounts and results from a website “bikehow.com” that has literally copied my articles and re-published them!
Reporting this to Bing didn’t show any results, so after a few days, I contacted the hosting provider of that website, asking them to ask the webmaster of the above-noted domain to remove the content copied from my website. This resulted in the copied content removal within 24 hours. It still shows in Bing and DuckDuckGo searches but returns a 404 when clicking on the search result (the website was put behind Cloudflare a few days after that, probably in order to try and hide their hosting provider and avoid any future reports from the owners of other websites whose content they stole).
This is ironic, because Bing’s guidelines state (link to their page):
“Bing likes unique, quality content. Websites which replicate content, redirect users quickly, or provide little depth often don’t fare well in our results. Prolonged poor performance can lead us to delist websites to make room for quality websites.”
The above-mentioned website is still not delisted, though it seems to just literally copy/paste content from various websites (and monetize with Google AdSense 🙂 ).
I checked Cloudflare firewall upon testing Bing bot page scans, and SiteScan – no record of them being blocked.
Just to be on the safe side, I tried a Site Scan for a website on a different domain, that still ranks normally with Bing. It failed! That website uses Cloudflare as well. OK. What happens if I disable Cloudflare proxy?
Site Scan worked with Cloudflare disabled.
That still doesn’t explain why the bikegremlin.com domain was dropped from Bing’s SERPS while the other domains (also behind Cloudflare) are doing fine. Hell, the tested domain ranks No1 for its targeted keyword! Really strange.
What are the differences between the BikeGremlin, and the unaffected domains? As far as I can tell:
- BikeGremlin has several websites on several different bikegremlin.com subdomains. The unaffected websites use only their main domain.
- BikeGremlin has a lot more traffic.
- Cloudflare settings for bikegremlin.com have been edited in July, but not when the problem started (in June).
Looking at the SERP history in Bing’s dashboard, it seems as if this is not the first time bikegremlin.com was removed. It’s just a bit longer now:
I hadn’t noticed that first “removal.” But I Googled this problem and came across a Linkedin article explaining how Bing Exploit Allows Website Owners to Deindex Competitors. The article is from 2021. I have no idea whether this exploit has been patched, nor whether something like that could have been the problem cause (I doubt it, my websites are nonprofit and rather small).
Based on my Googling, I’m not the only one affected. There seems to be a problem with Bing. Here, you can read about Jesse Squires’ problems with Bing and DuckDuckGo (link to his article). I don’t know about the other affected websites, but mine has no problems with Google, nor have I done anything that could even remotely be considered “black hat SEO.”
There are fruitful discussions on the WebHostingTalk forum about Bing website removal (link to the WHT), as well as on the LowEndSpirit forum (LES forum link).
This leads me to believe Bing is not a good choice if I’m looking for the best answer to my search. The same goes for any other search engine using Bing’s resources (DuckDuckGo, Yahoo, Amazon’s Alexa etc.). That’s a shame – I think Google really could use some good competition.
How can I help BikeGremlin to remain independent and objective?
I got an email from a real person from Bing – saying they are looking into it. In less than 24 hours since I reported it. This was a very pleasant surprise (hoping for the best, but not getting my hopes too high 🙂 ).
One of the first things I tried was editing my “robots.txt” file to forbid crawling Cloudflare cache directories (and wasting the “crawl budget”). Thanks to Jono Anderson (link to his Tweet). Here’s what it looks like now:
# Global rules # ----------------- User-agent: * Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /cdn-cgi/ Disallow: /cdn-cgi/bm/cv/ Disallow: /cdn-cgi/challenge-platform/ Allow: /wp-admin/admin-ajax.php Sitemap: https://relevant-bikegremlin-subdomain.com/sitemap.xml # Ban bots that don't benefit us. # -------------------------------- User-agent: Nuclei User-agent: WikiDo User-agent: Riddler User-agent: PetalBot User-agent: Zoominfobot User-agent: Go-http-client User-agent: Node/simplecrawler User-agent: CazoodleBot User-agent: dotbot/1.0 User-agent: Gigabot User-agent: Barkrowler User-agent: BLEXBot User-agent: magpie-crawler Disallow: /
I tried another Bing Site Scan after this edit. Just in case. As expected, it didn’t work. This problem doesn’t look like a crawl-budget-related one because all my pages were already indexed and ranked. Nonetheless, it was worth giving it a shot, plus it might prevent any crawl-budget problems in the future.
After a few days, I tried one more thing: disabling Cloudflare Crawler Hints.
Caching -> Configuration -> Crawler Hints
What does (did) this option do? When enabled, it lets Cloudflare push signals to networks that operate large crawling operations, such as search engines, when Cloudflare thinks content on a website has changed and needs to be re-indexed. I don’t mind manually (re)submitting pages to Google and Bing in case of any additions or major edits. So I turned this off just in case it sends some wrong signals that could be confusing Bing.
Now, it’s been about a week since I reported the problem to Bing. I got an email that says the problem is “escalated to the concerned team,” and they’ll keep me posted. On the upside:
- I don’t depend on my websites to make a living.
- Most of my traffic comes from Google.
Downsides? I’m in the middle of some sponsorship negotiation, and this doesn’t look good at all. It does affect the average visitor count trend, and it shows nothing to anyone using Microsoft Edge’s default search engine (and the above-mentioned search engines that rely on Bing).
In the meantime, I’ve also contacted Cloudflare support – to confirm there’s nothing Cloudflare-related causing the problems.
3.1. Site scan works (for now)
Update of 2022, August:
It appears that SiteScan has started working. For all the websites under my Bing webmaster account – those affected by the “SERP wipe” and those unaffected. I suppose that’s good. 🙂
I don’t know when it had stopped working, since I wasn’t using the tool until I saw the SERP problem. But it wasn’t working in July, when I first tried it. Now, in August, it works.
My websites are still gone from Bing’s SERP though.
3.2. Indexing issue
I’ve tried re-submitting some pages yesterday, checking indexing and similar, but even if that does help, I would expect it to take at least a few days. What I have noticed is that pages show an indexing error, even though the live URL test shows it’s all good:
Yes, I have tried clicking on the “Request indexing” button. Will know in a few days if it works (for) now. Before having done that, I checked the “Live URL” inspection tool – that seems to be happy with my websites now:
3.3. Sort of back… but not really
Ralf from LowEndSpirit noted this Bing problem being mentioned on Hacker News.
I read the topic and saw it saying that the affected sites are back. So I checked. Yes – technically, my website now appears when I search for the exact domain (“bikegremlin.com”) or “BikeGremlin.”
However, when I search for queries, none of my articles are shown in the search results – not in the first dozen SERP pages (yes, it is a pleonasm). I.e. queries that get me in the top-3 results on Google don’t show my articles anywhere on Bing (and Bing’s “derivative” search engines – see chapter 4 for that). Instead, Bing shows the pages from the site that had copy/pasted my content and re-published it (even though the pages return 404 now, after they’d been removed from the hosting server). 🙂
I did send the info shown in chapter 3.2 to Bing’s support and asked about an update some two weeks after having reported the problem. Got no reply for now.
I tried submitting URLs – now that the indexing seems to have changed (can’t confirm it actually works, but at least my sites are no longer 100% erased from Bing… just 99.9% 🙂 ). We’ll see if that helps over the next few days… weeks… months… 🙂
Update, 2022, August 6th:
Articles in my native are back!
None of the articles in English though, not yet. Even those that ranked highly.
A link to this report has been removed and any further discussion has been blocked on Microsoft Bing forum (link to the topic).
3.4. Manual unblock?
After about 20 days since I reported the problem to Bing, I got an email notifying me that my websites were blocked by Bing, and that, after a (manual?) review they’ve decided to lift the block.
They asked me to be patient, saying it might take up to three weeks for the site to be re-indexed. Guess we’ll know by September…
They won’t disclose the reasons for the block, which is understandable – since it could help avoidance even when a block is objectively justified. However, based on my “googling” over the previous weeks, there definitely seems to be some problem with their system – blocking even legit websites. Apparently, it’s a recurring problem.
I did offer any help they could use to prevent such problems from happening again (happy to sign any NDA papers if needed). I know, it’s funny, but my philosophy is to (offer to) do what is up to me – can’t do more than that. 🙂
Update, 2022, August 18th:
Seems that my websites are slowly making a return to Bing’s search result pages:
Almost a full recovery, much sooner than I had expected. Hope it stays like this.
4. List of Bing-copy search engines
This problem is a good opportunity to figure out which search engines practically provide a (more-less edited) copy of Bing’s search results. It was enough to search for “bikegremlin,” and if I see none of my websites, and see the website that copied my content in the SERPS – bingo!
I did double-check with searches for a few other keywords my websites rank highly for.
Here’s the list:
- Bing – the original 🙂
- Amazon’s Alexa
- Search Encrypt
All the above-listed search engines practically show Bing’s results, sometimes in a bit different order (rank). I don’t see why I shouldn’t go just straight to Bing.
4.1. Search engines that apparently don’t rely on Bing
Here’s a list of search engines that were not affected by this problem, so I suppose they don’t rely on Bing:
As far as I know, Google doesn’t let other search engines “use” their indexes and SERPs – not the way Bing does. So I believe the above-listed two search engines are independent from each other.
4.2. Search engines that use Bing’s index/discovery
When my websites were erased from Bing, they had been shown and indexed before having been erased. The below-listed search engines did show my pages in their SERPs, while they were erased from Bing.
However, I published an article about Patreon, 12 days ago (and another, more recent article, about a week ago). That article was indexed and shown in Google (and Yandex) SERPs within a day. But, after the 12-days period, it’s still not indexed by Bing.
I took a look at the most popular search engines to see if it’s indexed in any of them. I also searched for specific sentence quotes from the article (in case the article was indexed, but ranked poorly). So, here is a list of search engines that can’t index a website until Bing indexes it – so they still rely on Bing, even if their SERPs aren’t practically just a copy:
- Brave – unlike Google, Brave won’t let creators from Serbia monetize their sites and YouTube channels. I hate being punished for my place of birth and residence.
- Naver (BikeGremlin doesn’t rank here, for whatever reason, but it does list my sites, and doesn’t rank the site which copied mine instead of BikeGremlin)
- Baidu (BikeGremlin doesn’t rank here, but it’s not completely removed)
Note that I’m not 100% sure about the Brave and Naver. Why? I tried with an article that is just over one month old. That article is indexed by Bing, but it is not shown in Brave and Naver search results. Do these search engines use their own crawlers, and they’re just a bit slower? Maybe. It could also be that my articles are crap, so most search engines won’t index them – even Google doesn’t index everything, and they probably have the greatest capacity. It’s hard to tell when dealing with what basically are black boxes.
As far as I know and could test, Google takes about 90% of the total searches (i.e. a huge majority of people use Google to search). Of the remaining 10%, Bing takes a lion’s share.
It appears that all the search results in the world are controlled by three corporations:
- Microsoft (the owner of Bing)
- Yandex (the Russian Google, basically)
And that’s it.
Anyone whose business depends on being visible online is hugely dependent on Google, with the other two corporations following (way, way behind it). And, based on my experience, your online presence can just disappear overnight, without you having “done anything wrong.”
Also, for all I can tell, Google and Yandex are the only two that have indexing in order. Their SERPs are sometimes lacking, so I often try Brave as well when searching for information, but they have stuff indexed at least.
Bing (and all the other engines depending on it) seems to be incapable of doing its job – it won’t even index stuff properly.
6. Bing broken – again!
September, 26th, 2022:
Well, it appears that Bing is not working, again. It seems my currently indexed pages aren’t getting removed (at least for now, at least not many of them), but Bing won’t index any new pages.
Whatever they’re doing – it’s bad. And, it appears most (all?) search engines, besides Google and Yandex rely on Bing, at least to get the newly indexed pages (see chapter 4 for details).
I have contacted Bing’s support about this. We’ll see what happens.
19 thoughts on “Microsoft Bing – site gone from SERP overnight!”
Hi, try deleting IndexNow plugin if you have it installed.. We had all these issues when we added that plugin to a lot of our websites.
We noticed that some of the sites without that plugin installed didn’t lose ranking. So few weeks ago we deleted that plugin and now some of the site have thier ranking back again.
See if doing that helps after few days or weeks.
I have never used that plugin, but I have been using Cloudflare’s “Crawler Hints” option, which does a similar thing.
However, though all of my websites used that option, only one domain is affected with the Bing SERP removal.
Still, to be on the safe side, I’ve disabled the Cloudflare Crawler Hints for the affected website about 10 days ago.
So far, nothing seems to help.
Hi Relja, I’ve been observing some weird things around the same time as you, and ran a bunch of tests over on my blog. I don’t want to link-spam you so I’ll summarize as briefly as I can.
I discovered it as we were using DDG for our internal search for one of our sites. In late May or early June, thousands of our pages disappeared. Bing will do between 10 and 50 pages now for a site: search.
I initially wondered if we were doing something wrong, but there are two findings that suggest there is something very wrong with Bing.
I ran a bunch of site: searches using Google, Mojeek (which has its own index) and Bing. Let’s take one extreme example of site:nytimes.com. Here are the results:
But pretty much every site: search you can throw at them, except for site:microsoft.com, sees Bing dead last, and with a tiny fraction of the number of pages Google has.
Secondly, head to WorldWideWebSize.com. This site tries to estimate the size of the web by throwing queries at search engines. And in Bing’s case, since late May, the index appears to have tanked. At best, Bing has about 1·5 million pages in its index, if their methodology is accurate. That’s smaller than Inktomi and Alltheweb.com in the early 2000s. In fact, their latest graph shows the line just stopping.
No one there is talking about this, but there is something very wrong with that search engine.
Finally, Cloudflare Crawler Hints. I had that switched on as well, and it sent the least relevant pages (some were nearly 20 years old) to Bing. It didn’t reflect at all the pages I know were being read on our site.
I’ve disabled the Cloudflare Crawler Hints after having noticed this problem with Bing. Just to be sure Bing is not getting spammed by page-submission requests.
It is a black box after all, so all I can do is try to minimize the amount of “unknown” and unpredictable stuff, and track the little feedback I can get.
The number of indexed pages is not the whole problem IMO. Sure “SEO” piles up the Internet with garbage, and it’s fine if a search engine decides to just ditch a lot of that stuff. However, what I found to be most appaling is that I have some articles that rank top 3, or top 1+2 on Google, for the cycling-related stuff. Those have been removed from Bing (and the “Bing-copy-search-engines“)!
To add insult to injury, Bing SERP showed a website that literally copy/pasted my articles (the whole site is comprised of copy-pasted articles from various cycling-related websites, and uses AdSense for monetizing… so Google isn’t golden either, but at least it doesn’t rank the site on its SERPs). I had submitted the report to Bing, but no action was taken. Then I asked the “thief” website’s hosting provider to sort it out, and that was done in 24 hours, so the pages stolen from my site were showing 404 and will probably get dumped from Bing’s SERPa in a few years time… 🙂 Maybe then my articles will come back.
Again – I’m fortunate these websites don’t pay my bills. But it is a bit scary – just getting “turned off” overnight, and basically nothing you can do about it. Online-based businesses should start building brick&mortar shops ASAP. 🙂
Check if your website access logs contains prod.uhrs.playmsn.com in refarrals, then your site has been manually banned, by some guy from india or south america, that system provides low-paid clickworker reviews metrics without feedbacks.
Bing now looks like mafia.
To me, it looks like a search engine service that’s not very competent – but brave enough to let many other search engines depend on it (like DuckDuckGo).
Incompetence combined with such optimism provides for some really fun/scary (depending on how you look at it) results.
It’s good they don’t run an air-traffic company though. 🙂
Hi Relja, I totally agree with your thoughts—the small index is the consequence and not the cause, and I read your addendum about the splog appearing on Bing and not your site. Like you, I’m fortunate that the sites don’t pay my bills, either—they are a shop window to what we really offer—but the traffic was among my bragging rights. With 6 per cent, maybe more, traffic gone (Bing, and its licensees like DDG, Qwant, and others) we have to stop making certain claims. And it doesn’t look like things will get fixed any time soon—as Microsoft would then have to admit something is wrong, and that wouldn’t go down well with its licensees.
What really surprises me is how quiet the licensees are. In the early days, I would always get a response from Gabriel Weinberg at Duck Duck Go. More recently I pointed out to him that Bing had collapsed. You’d think that was pretty serious for a site that’s heavily reliant on Microsoft (way, way more than before; there’s nothing from Yandex on it, as far as I can make out). Nothing, just crickets.
Fair points Jack.
Though, see the chapter 4.1. “Search engines that apparently don’t rely on Bing”.
I.e, Yandex seems to have its own index (perhaps in addition to using Bing).
DuckDuckGo on the other hand seems to be 100% reliant on Bing’s service (4. “List of Bing-copy search engines”).
I agree it all seems very quiet. Nothing from Microsoft (and the other Bing-dependent search engines), nor from any SEO-specific websites/news-sites (maybe because “no one uses Bing” – it wouldn’t make for a click-bait article). If I hadn’t Googled, I would have thought the problem is only with my websites and my bad luck.
I see your site is back on bing. How did you solve? Facing the same issue since July and not got a reply yet
I wrote it all down (documented it) in this very article. Did nothing beyond what’s published here.
I am facing exactly the same issue since July. I have contacted support several times now, and they are always giving the same useless generic response:
After further review, it appears that your site coolplaydevcom did not meet the standards set by Bing the last time it was crawled.
I have the feeling they aren’t even checking the site manually and just checking it with the same algorithm that is blocking it.
Any advice on how I can get them to manually check my site?
Hi Oliver, until Bing admits it has a problem, my sense is that it’s “business as usual” for these folks. I have a 25-year-old site that is on there, but getting Bing to index anything after, say, 2007 is incredibly tough. A handful of articles make it, but when I do a site: search, no more than 55 pages (out of thousands) will ever show for us (and 36 per cent of those are repeated over different search engine results’ pages), and other independent sites are in the same boat. The Bing index has collapsed from what I can make out—it’s probably smaller than Inktomi’s and Alltheweb’s in the first half of the 2000s.
Oh, God.. Glad to find atleast some insights here in this article, I was just scared if I am the only guy left with blocking his site on bing overnight.
Sad thing: not recovered yet from 5 days after doing all the mentioned steps, will try some other steps suggested by you in this article. Especially the Cloudflare one.
You have done a fabulous job.. At least I am having a little bit of hope, that i am not the only guy, dealing with the same problem.
Let’s hope for the best, I was pretty sure that there is some bug in their system which helped me lost my 80% of traffic overnight without much efforts.
Hi, any updates on this? I am facing same issues.
My new pages/articles take months to get indexed, if ever. And that’s all good, business as ususal, as far as Bing is concerned. 🙁
On the bright side, it seems like my sites are no longer completely erased from Bing’s SERPs. For now.
So, which one thing actually helped your pages to be re-indexed?
Is it manual resubmitting? Or is it the Microsoft Team that lifted the block?
Have you got your normal traffic back today?
Do you have any more updates on the indexation process for your newer pages?
Thanks a lot!
Good questions which I may not have answered clearly in the article.
It appears there is more than one problem.
1. My sites did get deindexed by mistake (according to Bing’s support). They sort of fixed that part. So now my sites are visible on Bing again, though the performance (and the ranking) seems to have dropped a bit, compared to the pre-deindexing period (I have no way of determining whether that is justified, i.e. has Bing found better answers to queries on other sites etc.).
So yes, it looks as if the Microsoft team have lifted the block.
2. Another problem is that my new content doesn’t get indexed for months.
Bing’s support says that’s normal and I should just wait. Well, I think even one month is a pretty long time for a new article to get indexed.
This could be just about incompetence, lack of infrastructure etc, so they may be prioritizing site indexing based on the website’s “rating.” Google does that kind of prioritization too – hugely popular sites get new pages indexed within minutes, while my sites take a day or even two. However, with Bing, my articles take more than two months to get indexed (I’m still not sure see if they’ve just stopped indexing any of my new articles altogether).
So, as far as I can tell – Bing is a mess. And they don’t communicate clearly about the problems.
That means that practically all the other search engines, except Yandex, are also having indexing problems, since they all seem to rely on Bing to get the results. Some just regurgitate Bing’s ranking, while others rely on Bing to get the info about the existing content.
For example: if a site gets removed from Bing, Brave search will still show it in its results, while DuckDuckGo will also remove it. However, if a page doesn’t get indexed with Bing at least once, no other search engine (apart from Google and Yandex) will show it – ever.
Not just Google and Yandex, Mojeek indexes your site. In fact I see io.bikegremlin.com was indexed today, but I don’t know the novelty of the other pages they have. They claim to have 410 pages from your domain indexed, about two-fifths of what Google and Yandex each has. It’s up from Bing, which claims 90 pages (although if you go through the results, many are repeated from page to page, so I wonder if it really has 90). More than us at our busiest domain: they will only show 55, c. 40 per cent of which are repeated between different pages!
On your point about waiting for new content to be indexed, most of the pages Bing has indexed for us are pre-2009. Before the index’s collapse earlier in 2022, it found newer pages. Now, we’d be lucky if it can locate anything from the 2010s or 2020s. Last I looked, it found exactly one page from 2022. The Bing proxies are a mess—like you, I find some parallel Bing exactly, others might retain removed entries but eventually they catch up. I’ve given up on Bing and its various skin jobs like DDG, Qwant, Yahoo, etc. Microsoft won’t acknowledge how near death its search engine is, while its proxies are keeping their mouths shut.
This is an interesting read. Like you, my well performing site was delisted in Bing.
I saw lots of false directories / files that are apparently in Bing’s index of this site. These don’t exist, and never have. They seem to be basic exploit attempts. Some examples:
So the question was why are these being associated with my site? Well, it seems the answer is Cloudflare’s Crawler Hints as under IndexNow in Bing Webmaster Tools I see exploit URLs being submitted by Cloudflare.
I’ve reached out to Cloudflare about this, and also have a ticket open with Bing.