Microsoft Bing – site gone from SERP overnight!

I ran into an interesting problem with the Microsoft Bing search engine (used by DuckDuckGo, Yahoo, Amazon’s Alexa and many others, so about 5 – 10% of my organic traffic is affected). My websites just got erased from their search results – overnight! Simply completely disappeared from Bing’s search results. This very website, among others (all on the bikegremlin.com domain). Let’s dive in:

In a separate article, I wrote about “the state of search” in 2024 with an emphasis on “the other” search engine: Is Google In Trouble?

Table Of Contents (T.O.C.):

1. The problem – pages gone from Bing SERP
2. Investigating the problem
3. Troubleshooting
   3.1. Site scan works (for now)
   3.2. Indexing issue
   3.3. Sort of back… but not really
   3.4. Manual unblock?
4. List of Bing-copy search engines
   4.1. Search engines that apparently don’t rely on Bing
   4.2. Search engines that use Bing’s index/discovery
5. Conclusion?
6. Bing broken – again!

1. The problem – pages gone from Bing SERP

Today I logged into my Microsoft Bing Webmaster dashboard and saw this:

All the pages were completely gone from Microsoft Bing’s search result pages. All the websites on my bikegremlin.com domain were affected (bike.bikegremlin.com, io.bikegremlin.com etc.). The rest of the websites were fine. Here’s a more detailed graph for one of my websites (bike.bikegremlin.com):

– T.O.C. –

2. Investigating the problem

OK, it seems that the 24th of June, 2022, is when it suddenly went blank, zero. The first question to ask when something like this happens is: “what did you do/change/update?” The answer is, as far as I remember and can tell – nothing.
Not until July that is – in July I did a Setupad experiment, then tested Cloudflare Pro and WordPress APO. But this problem started in June, so I don’t think it’s related.

Next, I checked with other search engines:

• Google – all good and normal.
• Yandex – all good, all normal.
• Baidu – my website isn’t there, but I hadn’t checked before to confirm when it happened.
• DuckDuckGo – it’s a Bing-clone practically, and my sites are gone from the Duck as well!
• Yahoo – same, none of my websites there.

OK, it could be just Bing-related.

I took a dive into Bing’s dashboard:

• Checked other domain websites that also use Cloudflare – to confirm it’s not Cloudflare blocking Bing’s bots. Websites on other domains work normally.
• Checked for any error reports and similar.
• Checked and re-indexed sitemaps, though they all looked green, OK.
• Ran URL inspection for a few URL-s – all good.
• Tried Bing’s robot tester (Tools & Enhancements -> robots.txt Tester), tried a few test URL-s – all good.
• Ran Site Scan (SEO -> Site Scan). This showed one error, stopping right away, at the home page: “ERROR : Http 400-499 errors”.

That’s… “brilliant!” It would have made more sense had they provided the exact error (404, 410, or something third):

After going through the troubleshooting FAQ, I found nothing about a problem like this – so I’m writing this article. 🙂 And, I decided it’s time to contact support. Got an email confirming they have been notified and now all I can do is wait.

Another brilliant thing I noticed: Bing’s documentation/knowledge-base/FAQ has no search option! Ironic, isn’t it? 🙂

To add insult to injury, when you “Google” the term “BikeGremlin” on Microsoft Bing or DuckDuckGo, SERPs show BikeGremlin social-media accounts and results from a website “bikehow.com” that has literally copied my articles and re-published them!

Reporting this to Bing didn’t show any results, so after a few days, I contacted the hosting provider of that website, asking them to ask the webmaster of the above-noted domain to remove the content copied from my website. This resulted in the copied content removal within 24 hours. It still shows in Bing and DuckDuckGo searches but returns a 404 when clicking on the search result (the website was put behind Cloudflare a few days after that, probably in order to try and hide their hosting provider and avoid any future reports from the owners of other websites whose content they stole).

This is ironic, because Bing’s guidelines state (link to their page):
“Bing likes unique, quality content. Websites which replicate content, redirect users quickly, or provide little depth often don’t fare well in our results. Prolonged poor performance can lead us to delist websites to make room for quality websites.”
The above-mentioned website is still not delisted, though it seems to just literally copy/paste content from various websites (and monetize with Google AdSense 🙂 ).

I checked Cloudflare firewall upon testing Bing bot page scans, and SiteScan – no record of them being blocked.

Just to be on the safe side, I tried a Site Scan for a website on a different domain, that still ranks normally with Bing. It failed! That website uses Cloudflare as well. OK. What happens if I disable Cloudflare proxy?

Site Scan worked with Cloudflare disabled.

That still doesn’t explain why the bikegremlin.com domain was dropped from Bing’s SERPS while the other domains (also behind Cloudflare) are doing fine. Hell, the tested domain ranks No1 for its targeted keyword! Really strange.

What are the differences between the BikeGremlin, and the unaffected domains? As far as I can tell:

• BikeGremlin has several websites on several different bikegremlin.com subdomains. The unaffected websites use only their main domain.
• BikeGremlin has a lot more traffic.
• Cloudflare settings for bikegremlin.com have been edited in July, but not when the problem started (in June).

Looking at the SERP history in Bing’s dashboard, it seems as if this is not the first time bikegremlin.com was removed. It’s just a bit longer now:

I hadn’t noticed that first “removal.” But I Googled this problem and came across a Linkedin article explaining how Bing Exploit Allows Website Owners to Deindex Competitors. The article is from 2021. I have no idea whether this exploit has been patched, nor whether something like that could have been the problem cause (I doubt it, my websites are nonprofit and rather small).

Based on my Googling, I’m not the only one affected. There seems to be a problem with Bing. Here, you can read about Jesse Squires’ problems with Bing and DuckDuckGo (link to his article). I don’t know about the other affected websites, but mine has no problems with Google, nor have I done anything that could even remotely be considered “black hat SEO.”

There are fruitful discussions on the WebHostingTalk forum about Bing website removal (link to the WHT), as well as on the LowEndSpirit forum (LES forum link).

This leads me to believe Bing is not a good choice if I’m looking for the best answer to my search. The same goes for any other search engine using Bing’s resources (DuckDuckGo, Yahoo, Amazon’s Alexa etc.). That’s a shame – I think Google really could use some good competition.

– T.O.C. –

3. Troubleshooting

I got an email from a real person from Bing – saying they are looking into it. In less than 24 hours since I reported it. This was a very pleasant surprise (hoping for the best, but not getting my hopes too high 🙂 ).

One of the first things I tried was editing my “robots.txt” file to forbid crawling Cloudflare cache directories (and wasting the “crawl budget”). Thanks to Jono Anderson (link to his Tweet). Here’s what it looks like now:

# Global rules
# -----------------

User-agent: *
Disallow: /wp-json/
Disallow: /wp-admin/
Disallow: /cdn-cgi/
Disallow: /cdn-cgi/bm/cv/
Disallow: /cdn-cgi/challenge-platform/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://relevant-bikegremlin-subdomain.com/sitemap.xml

# Ban bots that don't benefit us.
# --------------------------------

User-agent: Nuclei
User-agent: WikiDo
User-agent: Riddler
User-agent: PetalBot
User-agent: Zoominfobot
User-agent: Go-http-client
User-agent: Node/simplecrawler
User-agent: CazoodleBot
User-agent: dotbot/1.0
User-agent: Gigabot
User-agent: Barkrowler
User-agent: BLEXBot
User-agent: magpie-crawler
Disallow: /

I tried another Bing Site Scan after this edit. Just in case. As expected, it didn’t work. This problem doesn’t look like a crawl-budget-related one because all my pages were already indexed and ranked. Nonetheless, it was worth giving it a shot, plus it might prevent any crawl-budget problems in the future.

After a few days, I tried one more thing: disabling Cloudflare Crawler Hints.
Caching -> Configuration -> Crawler Hints

What does (did) this option do? When enabled, it lets Cloudflare push signals to networks that operate large crawling operations, such as search engines, when Cloudflare thinks content on a website has changed and needs to be re-indexed. I don’t mind manually (re)submitting pages to Google and Bing in case of any additions or major edits. So I turned this off just in case it sends some wrong signals that could be confusing Bing.

Now, it’s been about a week since I reported the problem to Bing. I got an email that says the problem is “escalated to the concerned team,” and they’ll keep me posted. On the upside:

• I don’t depend on my websites to make a living.
• Most of my traffic comes from Google.

Downsides? I’m in the middle of some sponsorship negotiation, and this doesn’t look good at all. It does affect the average visitor count trend, and it shows nothing to anyone using Microsoft Edge’s default search engine (and the above-mentioned search engines that rely on Bing).

In the meantime, I’ve also contacted Cloudflare support – to confirm there’s nothing Cloudflare-related causing the problems.

– T.O.C. –

3.1. Site scan works (for now)

Update of 2022, August:
It appears that SiteScan has started working. For all the websites under my Bing webmaster account – those affected by the “SERP wipe” and those unaffected. I suppose that’s good. 🙂

I don’t know when it had stopped working, since I wasn’t using the tool until I saw the SERP problem. But it wasn’t working in July, when I first tried it. Now, in August, it works.

My websites are still gone from Bing’s SERP though.

– T.O.C. –

3.2. Indexing issue

I’ve tried re-submitting some pages yesterday, checking indexing and similar, but even if that does help, I would expect it to take at least a few days. What I have noticed is that pages show an indexing error, even though the live URL test shows it’s all good:

Yes, I have tried clicking on the “Request indexing” button. Will know in a few days if it works (for) now. Before having done that, I checked the “Live URL” inspection tool – that seems to be happy with my websites now:

– T.O.C. –

3.3. Sort of back… but not really

Ralf from LowEndSpirit noted this Bing problem being mentioned on Hacker News.

I read the topic and saw it saying that the affected sites are back. So I checked. Yes – technically, my website now appears when I search for the exact domain (“bikegremlin.com”) or “BikeGremlin.”

However, when I search for queries, none of my articles are shown in the search results – not in the first dozen SERP pages (yes, it is a pleonasm). I.e. queries that get me in the top-3 results on Google don’t show my articles anywhere on Bing (and Bing’s “derivative” search engines – see chapter 4 for that). Instead, Bing shows the pages from the site that had copy/pasted my content and re-published it (even though the pages return 404 now, after they’d been removed from the hosting server). 🙂

I did send the info shown in chapter 3.2 to Bing’s support and asked about an update some two weeks after having reported the problem. Got no reply for now.

I tried submitting URLs – now that the indexing seems to have changed (can’t confirm it actually works, but at least my sites are no longer 100% erased from Bing… just 99.9% 🙂 ). We’ll see if that helps over the next few days… weeks… months… 🙂

Update, 2022, August 6th:
Articles in my native are back!
None of the articles in English though, not yet. Even those that ranked highly.

A link to this report has been removed and any further discussion has been blocked on Microsoft Bing forum (link to the topic).

– T.O.C. –

3.4. Manual unblock?

After about 20 days since I reported the problem to Bing, I got an email notifying me that my websites were blocked by Bing, and that, after a (manual?) review they’ve decided to lift the block.

They asked me to be patient, saying it might take up to three weeks for the site to be re-indexed. Guess we’ll know by September…

They won’t disclose the reasons for the block, which is understandable – since it could help avoidance even when a block is objectively justified. However, based on my “googling” over the previous weeks, there definitely seems to be some problem with their system – blocking even legit websites. Apparently, it’s a recurring problem.

I did offer any help they could use to prevent such problems from happening again (happy to sign any NDA papers if needed). I know, it’s funny, but my philosophy is to (offer to) do what is up to me – can’t do more than that. 🙂

Update, 2022, August 18th:
Seems that my websites are slowly making a return to Bing’s search result pages:

Almost a full recovery, much sooner than I had expected. Hope it stays like this.

– T.O.C. –

4. List of Bing-copy search engines

This problem is a good opportunity to figure out which search engines practically provide a (more-less edited) copy of Bing’s search results. It was enough to search for “bikegremlin,” and if I see none of my websites, and see the website that copied my content in the SERPS – bingo!
I did double-check with searches for a few other keywords my websites rank highly for.

Here’s the list:

• Bing – the original 🙂
• Amazon’s Alexa
• DuckDuckGo
• Yahoo
• Ecosia
• Qwant
• Swisscows
• Search Encrypt
• OneSearch
• giveWater
• Ekoru
• Neeva

All the above-listed search engines practically show Bing’s results, sometimes in a bit different order (rank). I don’t see why I shouldn’t go just straight to Bing.

– T.O.C. –

4.1. Search engines that apparently don’t rely on Bing

Here’s a list of search engines that were not affected by this problem, so I suppose they don’t rely on Bing:

• Google – of course
• Yandex

As far as I know, Google doesn’t let other search engines “use” their indexes and SERPs – not the way Bing does. So I believe the above-listed two search engines are independent from each other.

– T.O.C. –

4.2. Search engines that use Bing’s index/discovery

When my websites were erased from Bing, they had been shown and indexed before having been erased. The below-listed search engines did show my pages in their SERPs, while they were erased from Bing.

However, I published an article about Patreon, 12 days ago (and another, more recent article, about a week ago). That article was indexed and shown in Google (and Yandex) SERPs within a day. But, after the 12-days period, it’s still not indexed by Bing.

I took a look at the most popular search engines to see if it’s indexed in any of them. I also searched for specific sentence quotes from the article (in case the article was indexed, but ranked poorly). So, here is a list of search engines that can’t index a website until Bing indexes it – so they still rely on Bing, even if their SERPs aren’t practically just a copy:

• Brave – unlike Google, Brave won’t let creators from Serbia monetize their sites and YouTube channels. I hate being punished for my place of birth and residence.
• StartPage
• Gibiru
• Naver (BikeGremlin doesn’t rank here, for whatever reason, but it does list my sites, and doesn’t rank the site which copied mine instead of BikeGremlin)
• Ask
• Baidu (BikeGremlin doesn’t rank here, but it’s not completely removed)

Note that I’m not 100% sure about the Brave and Naver. Why? I tried with an article that is just over one month old. That article is indexed by Bing, but it is not shown in Brave and Naver search results. Do these search engines use their own crawlers, and they’re just a bit slower? Maybe. It could also be that my articles are crap, so most search engines won’t index them – even Google doesn’t index everything, and they probably have the greatest capacity. It’s hard to tell when dealing with what basically are black boxes.

– T.O.C. –

5. Conclusion

As far as I know and could test, Google takes about 90% of the total searches (i.e. a huge majority of people use Google to search). Of the remaining 10%, Bing takes a lion’s share.

It appears that all the search results in the world are controlled by three corporations:

• Google
• Microsoft (the owner of Bing)
• Yandex (the Russian Google, basically)

And that’s it.

Anyone whose business depends on being visible online is hugely dependent on Google, with the other two corporations following (way, way behind it). And, based on my experience, your online presence can just disappear overnight, without you having “done anything wrong.”

Also, for all I can tell, Google and Yandex are the only two that have indexing in order. Their SERPs are sometimes lacking, so I often try Brave as well when searching for information, but they have stuff indexed at least.

Bing (and all the other engines depending on it) seems to be incapable of doing its job – it won’t even index stuff properly.

– T.O.C. –

6. Bing broken – again!

September, 26th, 2022:
Well, it appears that Bing is not working, again. It seems my currently indexed pages aren’t getting removed (at least for now, at least not many of them), but Bing won’t index any new pages.

Whatever they’re doing – it’s bad. And, it appears most (all?) search engines, besides Google and Yandex rely on Bing, at least to get the newly indexed pages (see chapter 4 for details).

I have contacted Bing’s support about this. We’ll see what happens.

– T.O.C. –

Author

Relja Novović

Relja is an avid cyclist, a pretty good bike mechanic, and a professional computer systems administrator – with decades of experience in both those fields (computers and bikes 🙂 ). Relja Novović’s credentials and a short biography

For any comments regarding this article, please use this BikeGremlin Forum thread:
https://www.bikegremlin.net/threads/seo-microsoft-bing-site-gone-from-serp-overnight-article-comments.98/