BikeGremlin search...
Start » Web sites » Maintenance » Microsoft Bing – site gone from SERP overnight!

Microsoft Bing – site gone from SERP overnight!

I ran into an interesting problem with the Microsoft Bing search engine (used by DuckDuckGo, Yahoo, Amazon’s Alexa and many others, so about 5 – 10% of my organic traffic is affected). My websites just got erased from their search results – overnight! Simply completely disappeared from Bing’s search results. This very website, among others (all on the bikegremlin.com domain). Let’s dive in:

Table Of Contents (T.O.C.):

  1. The problem – pages gone from Bing SERP
  2. Investigating the problem
  3. Troubleshooting
    3.1. Site scan works (for now)
    3.2. Indexing issue
    3.3. Sort of back… but not really
    3.4. Manual unblock?
  4. List of Bing-copy search engines
    4.1. Search engines that apparently don’t rely on Bing


1. The problem – pages gone from Bing SERP

Today I logged into my Microsoft Bing Webmaster dashboard and saw this:

Bing SERP problem - site gone overnight!
Bing SERP problem – site gone overnight!
Picture 1

All the pages were completely gone from Microsoft Bing’s search result pages. All the websites on my bikegremlin.com domain were affected (bike.bikegremlin.com, io.bikegremlin.com etc.). The rest of the websites were fine. Here’s a more detailed graph for one of my websites (bike.bikegremlin.com):

Full website performance stats from the Bing's dashboard
Full website performance stats from Bing’s dashboard
Picture 2

– T.O.C. –


2. Investigating the problem

OK, it seems that the 24th of June, 2022, is when it suddenly went blank, zero. The first question to ask when something like this happens is: “what did you do/change/update?” The answer is, as far as I remember and can tell – nothing.
Not until July that is – in July I did a Setupad experiment, then tested Cloudflare Pro and WordPress APO. But this problem started in June, so I don’t think it’s related.

Next, I checked with other search engines:

  • Google – all good and normal.
  • Yandex – all good, all normal.
  • Baidu – my website isn’t there, but I hadn’t checked before to confirm when it happened.
  • DuckDuckGo – it’s a Bing-clone practically, and my sites are gone from the Duck as well!
  • Yahoo – same, none of my websites there.

OK, it could be just Bing-related.

I took a dive into Bing’s dashboard:

  • Checked other domain websites that also use Cloudflare – to confirm it’s not Cloudflare blocking Bing’s bots. Websites on other domains work normally.
  • Checked for any error reports and similar.
  • Checked and re-indexed sitemaps, though they all looked green, OK.
  • Ran URL inspection for a few URL-s – all good.
  • Tried Bing’s robot tester (Tools & Enhancements -> robots.txt Tester), tried a few test URL-s – all good.
  • Ran Site Scan (SEO -> Site Scan). This showed one error, stopping right away, at the home page: “ERROR : Http 400-499 errors”.

That’s… “brilliant!” It would have made more sense had they provided the exact error (404, 410, or something third):

Bing's site scan reporting "brilliance!"
Bing’s site scan reporting “brilliance!”
Picture 3

After going through the troubleshooting FAQ, I found nothing about a problem like this – so I’m writing this article. 🙂 And, I decided it’s time to contact support. Got an email confirming they have been notified and now all I can do is wait.

Another brilliant thing I noticed: Bing’s documentation/knowledge-base/FAQ has no search option! Ironic, isn’t it? 🙂

To add insult to injury, when you “Google” the term “BikeGremlin” on Microsoft Bing or DuckDuckGo, SERPs show BikeGremlin social-media accounts and results from a website “bikehow.com” that has literally copied my articles and re-published them!

Microsoft Bing and DuckDuckGo SERP for term "BikeGremlin"
Microsoft Bing and DuckDuckGo SERP for the term “BikeGremlin”
Picture 4

Reporting this to Bing didn’t show any results, so after a few days, I contacted the hosting provider of that website, asking them to ask the webmaster of the above-noted domain to remove the content copied from my website. This resulted in the copied content removal within 24 hours. It still shows in Bing and DuckDuckGo searches but returns a 404 when clicking on the search result (the website was put behind Cloudflare a few days after that, probably in order to try and hide their hosting provider and avoid any future reports from the owners of other websites whose content they stole).

This is ironic, because Bing’s guidelines state (link to their page):
“Bing likes unique, quality content. Websites which replicate content, redirect users quickly, or provide little depth often don’t fare well in our results. Prolonged poor performance can lead us to delist websites to make room for quality websites.”
The above-mentioned website is still not delisted, though it seems to just literally copy/paste content from various websites (and monetize with Google AdSense 🙂 ).

I checked Cloudflare firewall upon testing Bing bot page scans, and SiteScan – no record of them being blocked.

Just to be on the safe side, I tried a Site Scan for a website on a different domain, that still ranks normally with Bing. It failed! That website uses Cloudflare as well. OK. What happens if I disable Cloudflare proxy?

Site Scan worked with Cloudflare disabled.

That still doesn’t explain why the bikegremlin.com domain was dropped from Bing’s SERPS while the other domains (also behind Cloudflare) are doing fine. Hell, the tested domain ranks No1 for its targeted keyword! Really strange.

What are the differences between the BikeGremlin, and the unaffected domains? As far as I can tell:

  • BikeGremlin has several websites on several different bikegremlin.com subdomains. The unaffected websites use only their main domain.
  • BikeGremlin has a lot more traffic.
  • Cloudflare settings for bikegremlin.com have been edited in July, but not when the problem started (in June).

Looking at the SERP history in Bing’s dashboard, it seems as if this is not the first time bikegremlin.com was removed. It’s just a bit longer now:

BikeGremlin's Bing SERP history
BikeGremlin’s Bing SERP history
Picture 5

I hadn’t noticed that first “removal.” But I Googled this problem and came across a Linkedin article explaining how Bing Exploit Allows Website Owners to Deindex Competitors. The article is from 2021. I have no idea whether this exploit has been patched, nor whether something like that could have been the problem cause (I doubt it, my websites are nonprofit and rather small).

Based on my Googling, I’m not the only one affected. There seems to be a problem with Bing. Here, you can read about Jesse Squires’ problems with Bing and DuckDuckGo (link to his article). I don’t know about the other affected websites, but mine has no problems with Google, nor have I done anything that could even remotely be considered “black hat SEO.”

There are fruitful discussions on the WebHostingTalk forum about Bing website removal (link to the WHT), as well as on the LowEndSpirit forum (LES forum link).

This leads me to believe Bing is not a good choice if I’m looking for the best answer to my search. The same goes for any other search engine using Bing’s resources (DuckDuckGo, Yahoo, Amazon’s Alexa etc.). That’s a shame – I think Google really could use some good competition.

– T.O.C. –


3. Troubleshooting

I got an email from a real person from Bing – saying they are looking into it. In less than 24 hours since I reported it. This was a very pleasant surprise (hoping for the best, but not getting my hopes too high 🙂 ).

One of the first things I tried was editing my “robots.txt” file to forbid crawling Cloudflare cache directories (and wasting the “crawl budget”). Thanks to Jono Anderson (link to his Tweet). Here’s what it looks like now:

# Global rules
# -----------------

User-agent: *
Disallow: /wp-json/
Disallow: /wp-admin/
Disallow: /cdn-cgi/
Disallow: /cdn-cgi/bm/cv/
Disallow: /cdn-cgi/challenge-platform/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://relevant-bikegremlin-subdomain.com/sitemap.xml

# Ban bots that don't benefit us.
# --------------------------------

User-agent: Nuclei
User-agent: WikiDo
User-agent: Riddler
User-agent: PetalBot
User-agent: Zoominfobot
User-agent: Go-http-client
User-agent: Node/simplecrawler
User-agent: CazoodleBot
User-agent: dotbot/1.0
User-agent: Gigabot
User-agent: Barkrowler
User-agent: BLEXBot
User-agent: magpie-crawler
Disallow: /

I tried another Bing Site Scan after this edit. Just in case. As expected, it didn’t work. This problem doesn’t look like a crawl-budget-related one because all my pages were already indexed and ranked. Nonetheless, it was worth giving it a shot, plus it might prevent any crawl-budget problems in the future.

After a few days, I tried one more thing: disabling Cloudflare Crawler Hints.
Caching -> Configuration -> Crawler Hints

What does (did) this option do? When enabled, it lets Cloudflare push signals to networks that operate large crawling operations, such as search engines, when Cloudflare thinks content on a website has changed and needs to be re-indexed. I don’t mind manually (re)submitting pages to Google and Bing in case of any additions or major edits. So I turned this off just in case it sends some wrong signals that could be confusing Bing.

Now, it’s been about a week since I reported the problem to Bing. I got an email that says the problem is “escalated to the concerned team,” and they’ll keep me posted. On the upside:

  • I don’t depend on my websites to make a living.
  • Most of my traffic comes from Google.

Downsides? I’m in the middle of some sponsorship negotiation, and this doesn’t look good at all. It does affect the average visitor count trend, and it shows nothing to anyone using Microsoft Edge’s default search engine (and the above-mentioned search engines that rely on Bing).

In the meantime, I’ve also contacted Cloudflare support – to confirm there’s nothing Cloudflare-related causing the problems.

– T.O.C. –


3.1. Site scan works (for now)

Update of 2022, August:
It appears that SiteScan has started working. For all the websites under my Bing webmaster account – those affected by the “SERP wipe” and those unaffected. I suppose that’s good. 🙂

I don’t know when it had stopped working, since I wasn’t using the tool until I saw the SERP problem. But it wasn’t working in July, when I first tried it. Now, in August, it works.

My websites are still gone from Bing’s SERP though.

– T.O.C. –


3.2. Indexing issue

I’ve tried re-submitting some pages yesterday, checking indexing and similar, but even if that does help, I would expect it to take at least a few days. What I have noticed is that pages show an indexing error, even though the live URL test shows it’s all good:

Bing indexing error report
Bing indexing error report
Picture 6

Yes, I have tried clicking on the “Request indexing” button. Will know in a few days if it works (for) now. Before having done that, I checked the “Live URL” inspection tool – that seems to be happy with my websites now:

Bing's Live URL report says it's all good!?!
Bing’s Live URL report says it’s all good!?!
Picture 7

– T.O.C. –


3.3. Sort of back… but not really

Ralf from LowEndSpirit noted this Bing problem being mentioned on Hacker News.

I read the topic and saw it saying that the affected sites are back. So I checked. Yes – technically, my website now appears when I search for the exact domain (“bikegremlin.com”) or “BikeGremlin.”

However, when I search for queries, none of my articles are shown in the search results – not in the first dozen SERP pages (yes, it is a pleonasm). I.e. queries that get me in the top-3 results on Google don’t show my articles anywhere on Bing (and Bing’s “derivative” search engines – see chapter 4 for that). Instead, Bing shows the pages from the site that had copy/pasted my content and re-published it (even though the pages return 404 now, after they’d been removed from the hosting server). 🙂

I did send the info shown in chapter 3.2 to Bing’s support and asked about an update some two weeks after having reported the problem. Got no reply for now.

I tried submitting URLs – now that the indexing seems to have changed (can’t confirm it actually works, but at least my sites are no longer 100% erased from Bing… just 99.9% 🙂 ). We’ll see if that helps over the next few days… weeks… months… 🙂

The latest Bing crawl errors report
The latest Bing crawl errors report
Picture 8

Update, 2022, August 6th:
Articles in my native are back!
None of the articles in English though, not yet. Even those that ranked highly.

A link to this report has been removed and any further discussion has been blocked on Microsoft Bing forum (link to the topic).

– T.O.C. –


3.4. Manual unblock?

After about 20 days since I reported the problem to Bing, I got an email notifying me that my websites were blocked by Bing, and that, after a (manual?) review they’ve decided to lift the block.

They asked me to be patient, saying it might take up to three weeks for the site to be re-indexed. Guess we’ll know by September…

They won’t disclose the reasons for the block, which is understandable – since it could help avoidance even when a block is objectively justified. However, based on my “googling” over the previous weeks, there definitely seems to be some problem with their system – blocking even legit websites. Apparently, it’s a recurring problem.

I did offer any help they could use to prevent such problems from happening again (happy to sign any NDA papers if needed). I know, it’s funny, but my philosophy is to (offer to) do what is up to me – can’t do more than that. 🙂

– T.O.C. –


4. List of Bing-copy search engines

This problem is a good opportunity to figure out which search engines practically provide a (more-less edited) copy of Bing’s search results. It was enough to search for “bikegremlin,” and if I see none of my websites, and see the website that copied my content in the SERPS – bingo!
I did double-check with searches for a few other keywords my websites rank highly for.

Here’s the list:

All the above-listed search engines practically show Bing’s results, sometimes in a bit different order (rank). I don’t see why I shouldn’t go just straight to Bing.

– T.O.C. –


4.1. Search engines that apparently don’t rely on Bing

Here’s a list of search engines that were not affected by this problem, so I suppose they don’t rely on Bing:

  • Google – of course
  • Brave – unlike Google, Brave won’t let creators from Serbia monetize their sites and YouTube channels. I hate being punished for my place of birth and residence.
  • Yandex
  • StartPage
  • Gibiru
  • Naver (BikeGremlin doesn’t rank here, for whatever reason, but it does list my sites, and doesn’t rank the site which copied mine instead of BikeGremlin)
  • Ask
  • Baidu (BikeGremlin doesn’t rank here, but it’s not completely removed)

Fortunately, my websites aren’t removed from Google, but as far as I know, Google doesn’t let other search engines “use” their indexes and SERPs – not the way Bing does. So I believe the above-listed search engines are independent.

– T.O.C. –

8 thoughts on “Microsoft Bing – site gone from SERP overnight!”

  1. Hi, try deleting IndexNow plugin if you have it installed.. We had all these issues when we added that plugin to a lot of our websites.
    We noticed that some of the sites without that plugin installed didn’t lose ranking. So few weeks ago we deleted that plugin and now some of the site have thier ranking back again.

    See if doing that helps after few days or weeks.

    Reply
    • Hi Emmanuel,

      I have never used that plugin, but I have been using Cloudflare’s “Crawler Hints” option, which does a similar thing.

      However, though all of my websites used that option, only one domain is affected with the Bing SERP removal.

      Still, to be on the safe side, I’ve disabled the Cloudflare Crawler Hints for the affected website about 10 days ago.

      So far, nothing seems to help.

  2. Hi Relja, I’ve been observing some weird things around the same time as you, and ran a bunch of tests over on my blog. I don’t want to link-spam you so I’ll summarize as briefly as I can.

    I discovered it as we were using DDG for our internal search for one of our sites. In late May or early June, thousands of our pages disappeared. Bing will do between 10 and 50 pages now for a site: search.

    I initially wondered if we were doing something wrong, but there are two findings that suggest there is something very wrong with Bing.

    I ran a bunch of site: searches using Google, Mojeek (which has its own index) and Bing. Let’s take one extreme example of site:nytimes.com. Here are the results:

    Google: 36,500,000
    Mojeek: 2,879,513
    Bing: 2,723

    But pretty much every site: search you can throw at them, except for site:microsoft.com, sees Bing dead last, and with a tiny fraction of the number of pages Google has.

    Secondly, head to WorldWideWebSize.com. This site tries to estimate the size of the web by throwing queries at search engines. And in Bing’s case, since late May, the index appears to have tanked. At best, Bing has about 1·5 million pages in its index, if their methodology is accurate. That’s smaller than Inktomi and Alltheweb.com in the early 2000s. In fact, their latest graph shows the line just stopping.

    No one there is talking about this, but there is something very wrong with that search engine.

    Finally, Cloudflare Crawler Hints. I had that switched on as well, and it sent the least relevant pages (some were nearly 20 years old) to Bing. It didn’t reflect at all the pages I know were being read on our site.

    Reply
    • Hi Jack,

      I’ve disabled the Cloudflare Crawler Hints after having noticed this problem with Bing. Just to be sure Bing is not getting spammed by page-submission requests.

      It is a black box after all, so all I can do is try to minimize the amount of “unknown” and unpredictable stuff, and track the little feedback I can get.

      The number of indexed pages is not the whole problem IMO. Sure “SEO” piles up the Internet with garbage, and it’s fine if a search engine decides to just ditch a lot of that stuff. However, what I found to be most appaling is that I have some articles that rank top 3, or top 1+2 on Google, for the cycling-related stuff. Those have been removed from Bing (and the “Bing-copy-search-engines“)!

      To add insult to injury, Bing SERP showed a website that literally copy/pasted my articles (the whole site is comprised of copy-pasted articles from various cycling-related websites, and uses AdSense for monetizing… so Google isn’t golden either, but at least it doesn’t rank the site on its SERPs). I had submitted the report to Bing, but no action was taken. Then I asked the “thief” website’s hosting provider to sort it out, and that was done in 24 hours, so the pages stolen from my site were showing 404 and will probably get dumped from Bing’s SERPa in a few years time… 🙂 Maybe then my articles will come back.

      Again – I’m fortunate these websites don’t pay my bills. But it is a bit scary – just getting “turned off” overnight, and basically nothing you can do about it. Online-based businesses should start building brick&mortar shops ASAP. 🙂

  3. Hello,

    Check if your website access logs contains prod.uhrs.playmsn.com in refarrals, then your site has been manually banned, by some guy from india or south america, that system provides low-paid clickworker reviews metrics without feedbacks.

    Bing now looks like mafia.

    Reply
    • Hi Max,

      To me, it looks like a search engine service that’s not very competent – but brave enough to let many other search engines depend on it (like DuckDuckGo).

      Incompetence combined with such optimism provides for some really fun/scary (depending on how you look at it) results.

      It’s good they don’t run an air-traffic company though. 🙂

  4. Hi Relja, I totally agree with your thoughts—the small index is the consequence and not the cause, and I read your addendum about the splog appearing on Bing and not your site. Like you, I’m fortunate that the sites don’t pay my bills, either—they are a shop window to what we really offer—but the traffic was among my bragging rights. With 6 per cent, maybe more, traffic gone (Bing, and its licensees like DDG, Qwant, and others) we have to stop making certain claims. And it doesn’t look like things will get fixed any time soon—as Microsoft would then have to admit something is wrong, and that wouldn’t go down well with its licensees.

    What really surprises me is how quiet the licensees are. In the early days, I would always get a response from Gabriel Weinberg at Duck Duck Go. More recently I pointed out to him that Bing had collapsed. You’d think that was pretty serious for a site that’s heavily reliant on Microsoft (way, way more than before; there’s nothing from Yandex on it, as far as I can make out). Nothing, just crickets.

    Reply
    • Fair points Jack.

      Though, see the chapter 4.1. “Search engines that apparently don’t rely on Bing”.

      I.e, Yandex seems to have its own index (perhaps in addition to using Bing).
      DuckDuckGo on the other hand seems to be 100% reliant on Bing’s service (4. “List of Bing-copy search engines”).

      I agree it all seems very quiet. Nothing from Microsoft (and the other Bing-dependent search engines), nor from any SEO-specific websites/news-sites (maybe because “no one uses Bing” – it wouldn’t make for a click-bait article). If I hadn’t Googled, I would have thought the problem is only with my websites and my bad luck.

Leave a Comment