The Low-down on IndexNow From Mr. Bingbot (Fabrice Canel and Jason Barnard)

Fabrice Canel talks with Jason Barnard about the low-down on IndexNow.

Fabrice Canel is a Principal Program Manager leading the crawling, processing, and indexing team at Bing. He is a veteran at Microsoft Bing that has dealt with crawling web pages since the beginning of Bingbot and now deals with the hundreds of billions of new or updated web pages every day!

In this episode, Fabrice Canel and Jason Barnard discuss IndexNow – a stunningly great change that is happening to search engine indexing.

One of the ultimate goals of search engines is delivering timely information to its users. However, crawling websites may take days or weeks so updated, added and deleted information often takes some time to reflect in search engines. This is a big problem for website owners and one they often struggle to deal with.

So to relieve this pain point, Microsoft Bing has introduced IndexNow, a new protocol allowing websites to easily notify search engines whenever they make changes to website content. Because it’s new, there are a lot of questions that need to be answered, and Jason asks Fabrice all the right questions, of course 🙂

How does it work?
Can it instantly crawl our websites?
Is Google also adopting IndexNow?
What CDN have adopted IndexNow?
What CMS systems have adopted IndexNow?
How does the API work?

Fabrice (aka Mr. Bingbot) answers all these questions simply, clearly and extensively. That makes this episode a must-listen…. Especially as Fabrice makes it very clear he will not stop until he gets 80% adoption of IndexNow. He is absolutely convinced that this initiative will make everyone happy: website owners, users, and search engines. Tune in!

What you’ll learn from Fabrice Canel

00:00 Fabrice Canel and Jason Barnard
00:35 The beginning of Bingbot
01:59 Can Bingbot keep up with the infinite pages that need to be crawled?
05:55 Fabrice Canel’s Brand SERP on Bing and Google
07:18 IndexNow: What it is, how it works and why it’s helpful
08:55 Why an API layer is better than a web form
11:37 The early adopters of IndexNow
14:52 Getting the CMS and the CDNs to do the work
15:57 How long will it take for IndexNow to catch on?
19:30 Will IndexNow be integrated into the core of WordPress?
23:13 Why using IndexNow can reduce crawl on a website
24:27 Why IndexNow matters to small websites
30:05 Does IndexNow guarantee instant indexing?
32:20 How fast can websites be caught for abusing IndexNow?
33:21 Will this make XML sitemaps redundant?
35:07 The IndexNow API is super simple

This episode was recorded live on video November 2nd 2021

Recorded live at Kalicube Tuesdays (Digital Marketing Livestream Event Series). Watch the video now >>

Full Corrected Transcript for The Low-down on IndexNow From Mr. Bingbot (Fabrice Canel and Jason Barnard)

Jason Barnard: Hi everybody and welcome to the second part of this double bill of Kalicube Tuesdays with Fabrice Canel. A quick hello and we’re good to go. Welcome to the show, Fabrice Canel!

Fabrice Canel: Thank you. I will just speak — no song from me.

Jason Barnard: Right. Today we’re talking about IndexNow. I’ve been calling you Mr. Bingbot, which is probably terribly rude. What’s your official title?

Fabrice Canel: I am a Principal Program Manager at Microsoft. I’ve been working at Microsoft for 24 years.

Jason Barnard: A very long time. A very interesting time. I was looking you up on LinkedIn and you started at Microsoft in the late nineties. Did you actually start Bingbot?

Fabrice Canel: Yes. We started Bingbot together — I even created the Wikipedia page for it. It was the evolution of MSN Bot: we started with MSN Search, then Live Search, then Windows Live Search, then Bing, and now Microsoft Bing.

Jason Barnard: You’ve been on the team the whole time. Have you been programme lead from the beginning?

Fabrice Canel: Not at first, but now I’m in charge of managing part of the team delivering the best index on the internet.

Jason Barnard: You must be the only person in the industry who has been developing a crawling bot for 24 years or more — since the very beginning of the crawling era of the internet?

Fabrice Canel: Yes, in the industry. I’ve seen all the evolution. When we started, there were maybe 20 pages and it was really easy.

Jason Barnard: And now it’s infinity.

Fabrice Canel: Yes, infinity. The scale of the internet is growing year by year at an unknown rate. Some people estimate 60% year-over-year growth. The frontier keeps expanding because people now generate content using content management systems rather than Notepad, and they can automate the creation of websites entirely. And then you have spammers generating millions and billions of pages of useless content automatically, every day.

Jason Barnard: And if it’s growing at 60% a year, does that mean you can’t keep up?

Fabrice Canel: All search engines are managing a business. We are business-driven, obviously to satisfy customers, but we have to pick and choose. We cannot crawl everything on the internet — there are an infinite number of URLs out there. You have pages with calendars that go to the next day forever. It’s really about detecting what is most useful to satisfy a Microsoft Bing customer.

Jason Barnard: And a large chunk of this content is useless — the infinite calendar problem, spam, spun content. Would you say most content on the web is not useful, or is that exaggerating?

Fabrice Canel: That’s a little exaggerated. We are guided by key pages that are important on the internet, and we follow links to understand what comes next. If we focus on those key domains and key pages, that guides us to quality content. Our view of the internet is not to go deep forever and crawl useless content. It’s about keeping the index fresh and comprehensive — containing all of the most relevant content on the web.

Jason Barnard: And the key insight is that you prefer going wide over going deep. So if I have a site that’s already near the top, you’ll tend to focus on me rather than constantly searching for new things you don’t already know about?

Fabrice Canel: It depends. If you have a site that is specialised and covers an interesting topic that customers care about, we may absolutely go deep. But this is not me selecting where to go deep — nor my team. It’s the machine. Machine learning is selecting to go deep or deeper based on what we feel is important for a Bing customer.

Jason Barnard: That’s something we tend to lose sight of. I’m talking to Fabrice Canel and I get the idea that you’re programming this machine to come and look at my site — but you’re not. The machine is deciding for itself. You’re just giving it the goals you want it to achieve.

Fabrice Canel: Absolutely. The main input we give the machine learning algorithms is: satisfy Bing customers. We look at various dimensions to do that. If you query for Facebook, you want the Facebook link at the top. You don’t want some random blog talking about Facebook.

Jason Barnard: Very good point. And when you say “customer” rather than “user,” that changes our perspective on what we’re asking you to do. The search results are a product, and you’re thinking about what’s good for your customer. Which leads us to the main topic today: IndexNow. How you solve the crawling problem and help manage what gets indexed. But first — a tradition on the show — let’s look at your Brand SERP.

[Brand SERP segment — Fabrice’s personal Brand SERP on Bing is shown, including his photo and a large teddy bear in the image]

Jason Barnard: There’s a teddy bear in there. Is there a story behind it?

Fabrice Canel: It’s a family story. When I met my wife, I bought this bear in the US and brought it back. It’s a big teddy bear we have at home. Family-related.

Jason Barnard: Brilliant. And it’s a real teddy bear — that makes it all the more fun. Now, on to IndexNow. Can we start with what it is, how it works, and why it’s helpful?

Fabrice Canel: Great question. We have a single fundamental problem: my platform and other search engines don’t really know when Jason will post the next thing on the internet — the next post, the next video. Jason knows, but I don’t. The current model of crawling is to try to figure out when things are changing. When will Jason post again? We can try to model it, but we really don’t know. So we pull and pull and crawl and crawl to see if something has changed. We may learn from links, but at the end of the day, we go to the home page and try to figure it out. That model needs to change. We need input from the website owner. Jason can tell us via a simple API that his website content has changed — helping us discover the change, sending the crawler, and getting the latest content. It’s an industry shift: from crawling and crawling to discover if something has changed, to the website simply notifying us that something has changed.

Jason Barnard: That takes us right back 24 years to when all of this started — when we used to submit our web pages directly to search engines. Are we coming back to that?

Fabrice Canel: The early days had the right model, but the wrong implementation: you had to go to a form and submit your URL to each search engine individually. What we’ve done now is create an API layer — an open interface — that lets you, your content management system, or your CDN tell us what has changed on your website. Ideally, it shouldn’t even be the webmaster doing the notifying. It should be the underlying layer — the CMS — telling search engines automatically that something has changed. An API layer is better than a web form because people forget, and people don’t want to deal with every search engine separately. This is an open protocol that all search engines can adopt, where websites notify us of every change once.

Jason Barnard: If it’s integrated into the CMS, it just gets pinged out automatically. I create a page or update a page and it gets pinged immediately — by WordPress, Duda, Wix, or whatever. One of the big challenges, as I understand it, is that not all CMS providers are as enthusiastic about this as you are?

Fabrice Canel: We have started engaging with all the CMS companies and CDN companies to explain what IndexNow is and to get them to adopt it. In the following weeks and months, we’ll be making a series of announcements — not only more search engines adopting IndexNow, but also content management systems, CDNs, and other dimensions I can’t speak about yet.

Jason Barnard: CDNs — that’s something I hadn’t really considered. You can effectively sidestep some of the CMS discussion by getting CDNs involved, because an awful lot of content now goes through CDNs. If they’re submitting to you because they know something changed, that’s a significant chunk of the problem solved, isn’t it?

Fabrice Canel: Yes. Cloudflare has already adopted IndexNow. They’re providing useful information about what’s changed on every Cloudflare site. It’s an opt-in model — you click a button in the Cloudflare console to notify search engines (Yandex and Bing for now, with more coming) about changes on your sites.

Jason Barnard: I use Cloudflare and when you said that I thought “Brilliant, Fabrice already knows about all my new content!” But in fact I haven’t activated it. Why did they make it opt-in rather than the default?

Fabrice Canel: This is something we’re reviewing with CMS companies and CDNs: what’s the right model? I do believe that most of the internet will move to an opt-out model over time, where it’s built in by default and people can opt out. Due to regulations worldwide, people currently prefer an opt-in model — but I think at some point they’ll switch to opt-out as people become more comfortable with IndexNow.

Jason Barnard: There are two problems with opt-in. One is how lazy we are — you’re now telling me I have to go to Cloudflare and click a button for each site. That already tires me out. And the second problem is that I didn’t even know this existed until you just told me. How do you get that message out beyond shows like this?

Fabrice Canel: This was part of the blog post announcement, and Cloudflare communicated it as well. We’re getting visibility because a large number of URLs are already coming in through the new system — a lot of sites have already adopted. Shows like Jason’s help get the message out and tell people to go enable this feature in their Cloudflare settings.

Jason Barnard: We’ve got about 50 people watching. If those 50 people each control six sites, that’s 300 sites. Cloudflare has rather more than that. You said “a lot” — how much is a lot?

Fabrice Canel: Sites like W3Techs track CDN usage, and they estimate about 20% of the internet is on Cloudflare today. That’s a huge volume of URLs coming in. I can’t disclose the exact numbers, but we’re very pleased with what Cloudflare has done as an early adopter, and webmasters who have adopted IndexNow via Cloudflare are happy. And Cloudflare isn’t the only CDN: Akamai has disclosed they’re looking at adopting IndexNow, and other CDNs will be announced later.

Jason Barnard: So it looks to me like you, Fabrice, are personally solving the crawling problem with IndexNow. A lot of this came from you personally, if I understand correctly?

Fabrice Canel: At the end of the day, this is really solving your problem, Jason — and the problem of every website. Crawling is a blunt instrument. But if you tell me what has changed, my crawler becomes smart and efficient: it comes to pick up the latest content instead of pulling and pulling trying to get it from you. You know when the content has changed. I don’t. Please tell me. Why wait for the crawler to find it in an hour or a day? Tell search engines when the content has changed. It really is that simple.

Jason Barnard: And what strikes me is that you’re not really asking me to tell you. You’re getting the CMS and CDNs to do the work for me. I’ll be informing you without even knowing it, which is rather clever.

Fabrice Canel: Exactly.

Jason Barnard: Now the big question: Google hasn’t adopted it yet. What’s happening on that front?

Fabrice Canel: I think you should ask them — do your journalist work. What I can say is that this is an open protocol. When you ping one search engine, we relay the notification behind the scenes, so all participating search engines are aware. The overall benefit is being indexed in all search engines, being fresher, having the latest content indexed everywhere. And critically, it minimises crawl load on every site. Instead of each search engine crawling independently, the site tells us the content has changed, and we can each be smart about going to pick it up — ideally just once.

Jason Barnard: What I hear between the lines is that Google would be foolish not to adopt, because: A, it saves them money on crawl costs, and B, if they don’t, they’ll have a less fresh index than you. I’m not saying that’s what you said.

Fabrice Canel: What I can say is that IndexNow benefits us enormously right now. Top sites like eBay, LinkedIn, and GitHub have already adopted. Knowing what changed on eBay? Good luck figuring that out by crawling. If you post something on eBay, the best way for us to know is through an IndexNow notification. This helps us have the latest content, fewer dead links, and a more useful index — which satisfies Bing customers and satisfies eBay and its customers too. eBay gets more quality traffic from Bing and from Yandex. So the coverage we’re building is: search engines first, then content management systems, then CDNs, then large sites. We want to cover the whole internet — well, maybe 80% of it. About 20% may never adopt IndexNow, and that’s fine. On the 80%, we’ll be fresh and crawl load will be minimised.

Jason Barnard: The 80/20 rule. Before this conversation, I’d have said 80% is insane and impossible. But the way you’ve laid it out, I’m beginning to believe it. This is actually possible, isn’t it?

Fabrice Canel: Yes, it is possible. Not within the next month — it will take one or two years to get there. But with major content management systems, major CDNs like Cloudflare and Akamai plus others coming, and with top sites like eBay and many other large websites, you cover a very large part of the internet.

Jason Barnard: On WordPress specifically: do you think IndexNow could get integrated into WordPress core, or are you relying on plugins like Yoast?

Fabrice Canel: We do believe there’s interest. There’s a public thread open with WordPress where I’ve laid out this feature request. We’ve offered help with coding since it’s open source, including some example code that could support IndexNow. The value proposition for all WordPress users is clear — 40% of the internet runs on WordPress.

Jason Barnard: I’d heard 42%, actually — which is remarkable. And the challenge is a lot of WordPress posts are almost never updated. Updated once or twice a month, or even once a year. You don’t know when, and you can’t crawl each WordPress site daily or hourly. So the big benefit there is less about crawl load and more about getting content indexed quickly when something is actually posted?

Fabrice Canel: Exactly. Getting the content indexed quickly is the key value proposition for smaller sites. You post something, you want it indexed in search engines without waiting one day or even five days. You care about that content reaching people.

Jason Barnard: My frustration is always when I update something and I don’t get around to submitting it to Bing or Google by hand, and then four or five days later I’m wondering why nobody’s found it. The answer is: no IndexNow.

Fabrice Canel: I could monitor Jason’s page every five seconds with my crawler, but you wouldn’t like me for it — and it would be a waste. We should be smarter. If you tell me when you fix a typo, update something, or add a new page, the search engine comes to get the latest content — hopefully just once — and everyone is happy.

Jason Barnard: You’re saving me effort, bandwidth, and server load. But for a lot of small sites, that simply doesn’t matter. And small sites with rarely updated content might wonder: why would I bother?

Fabrice Canel: Freshness is value proposition number one: getting the latest content or the latest fix reflected quickly in search engines. Value proposition number two: as more search engines adopt IndexNow, the combined crawl volume from all of them should start to fall. We should collectively crawl less often, and the total crawl load across the internet should diminish significantly.

Jason Barnard: And you’ll also be feeding all of this into your machine learning, so the crawler can learn the patterns of how sites change and reduce its own footprint even further. Which leads me to the next question: does using IndexNow cause me to lose crawl? If I tell you when I update, do you stop crawling me at other times and I miss something?

Fabrice Canel: As sites adopt IndexNow, the machine learning behind Bing learns: “this site has IndexNow enabled, great.” It’s easy to detect because we know you’re publishing through IndexNow. What’s slightly harder is understanding whether you’re notifying us for the whole site or only part of it — you might have folders covered by a different CMS, or you might accidentally miss notifying us about a specific page. That learning is ongoing. As we go, the machine learning around crawling will become more intelligent and crawl less on sites that have adopted IndexNow.

Jason Barnard: Sticking with that: Yoast — who are massive in the WordPress world — have said for now they’re not adopting it. Their reasoning is that smaller sites don’t really need it, and since Google hasn’t adopted it yet, they don’t see the point. Do you feel that underestimates the importance of IndexNow, and also underestimates the fact that Bing — even though it’s smaller than Google — still accounts for 8 to 10% of organic website visits?

Fabrice Canel: What people often forget about Bing is that our market share is not zero. And there are a lot of companies leveraging Bing’s technology. If you sum all of that, yes, we’re in the 8–10% range across mobile and desktop combined — and higher on desktop alone. But the value is also about freshness, not just about reach. If you care about satisfying your customers by getting your latest content quickly indexed, IndexNow delivers that. I think we’ll see Yoast adopt at some point. We’re speaking to them. We’ll also see other WordPress plugins adopting. There are ongoing discussions. I am totally convinced the whole industry will adopt.

Jason Barnard: What I’m hearing is that you’re absolutely determined. You’re not going to let this go until everyone picks it up — because you’re so convinced it’s important, both for crawling and for the health of the internet.

Fabrice Canel: Absolutely. I’m convinced this is the right fight for the whole industry, for the internet. It’s about notifying what changed, helping the latest content get indexed, and removing unnecessary crawl load from websites. We’ve been hearing these complaints for years: “my latest content didn’t get indexed” and “you’re crawling me too much.” Both problems. If we can reduce overall crawl volume across all sites, everybody benefits — website owners and search engines alike. My teenagers want to see the latest content indexed. They don’t care about content from five days ago. They want the latest. That’s what IndexNow makes possible for the whole internet.

Jason Barnard: A little off topic, but the Facebook rebranding to Meta is an interesting example of freshness. I did a study of their Brand SERP to understand how they managed the rebrand. Turns out they bought a company called Meta five years earlier, so they’d been planning it for years. But what struck me was how fast both Bing and Google updated their content to reflect Meta in the Brand SERP — within hours of Zuckerberg’s announcement. That freshness at scale is genuinely difficult to achieve.

Fabrice Canel: When there’s a huge volume of content talking about something, search engines pick up that signal and the best content surfaces quickly. But people forget there’s also a lot of valuable content that doesn’t get picked up quickly. When something major happens, we get CNN and other big sites — that content surfaces. But think about your local school. They post the latest calendar updates, COVID restrictions changing the schedule. Search engines might miss that for weeks, for months. With IndexNow, the school can tell search engines immediately: “We’ve updated our schedule.” That’s the real value — getting that kind of timely, local, practical content reflected quickly everywhere.

Jason Barnard: You mentioned COVID. Did that accelerate your thinking on IndexNow — the need to get up-to-date information about school closures, shop closures, lockdowns — or had you already been planning this for years?

Fabrice Canel: We’d been planning this for a long time. The Bing Webmaster submission API — which was proprietary to Bing — was a first instance of really learning at scale how to manage this kind of notification from top sites. We benefited enormously from it. The thinking was: we have this API, let’s scale it to support the whole internet and offer it as an open protocol to all search engines and the whole community. It was a proprietary solution to learn from. Now we’re opening it up and extending it to the whole internet.

Jason Barnard: I love the idea of building something to learn from it, then opening it up once you know it works. One question people ask constantly: if I ping IndexNow, does that guarantee indexing?

Fabrice Canel: Obviously not — that would be too easy for spammers. It’s a hint that content has changed. We make our own decision about whether to crawl. You could notify us every second that your homepage has changed, and that’s almost certainly not true. We have basic heuristics and more advanced machine learning to understand whether the notification is from a useful source or not. But for quality sites, it’s a genuinely strong hint that gets the latest content indexed quickly. For a large site like eBay, it helps with the tail pages — the less prominent listings that we might otherwise take a long time to discover. We know eBay is publishing these URLs from a content management system, not manually. That means the links coming in are generally high quality.

Jason Barnard: What are the limits placed on websites? Is there an absolute cap, or is it machine-driven — understanding what a site would normally submit and flagging anomalies?

Fabrice Canel: We don’t accept a billion URLs from Jason’s site every second. But we can support tens of thousands of URLs per batch, and at scale that equates to billions of URLs per day across large sites. For a WordPress blog, it might be one or two URLs every few days. For a very large site, it could be millions of URLs daily. We handle both ends of that spectrum.

Jason Barnard: When will a site get caught for abuse? And is that assessed at domain level or at the CDN level — so would Cloudflare, for example, carry an overall abuse policy, or is each domain assessed independently?

Fabrice Canel: We look at each domain individually. We can’t tell you exactly what we check — that would be an open door for spammers — but it follows Bing Webmaster Guidelines: what we consider acceptable and what we don’t, per site. If your site isn’t getting indexed and you believe it should be, open a support ticket. We do look at them, and if there’s a problem on our side, we’ll fix it.

Jason Barnard: Can I now forget about my XML sitemap and just use IndexNow?

Fabrice Canel: Excellent question. XML sitemaps are a great way to tell search engines on a daily basis what URLs exist on your site. At least Bing visits your sitemaps once a day to collect all those URLs. It’s a reliable daily check. IndexNow is about “now”: you shouldn’t have to wait a day or two for a URL to be discovered via sitemaps. If you fix a link, delete a link, update a link, tell us via IndexNow immediately. The delete scenario in particular is poorly handled by sitemaps — it’s hard for search engines to understand what’s been removed. IndexNow provides a clear signal about what’s changed. And if we miss a ping, or there’s a system issue, you still have the sitemap as a backup. The two work together: IndexNow for immediate notification, sitemaps for daily verification of everything on the site.

Jason Barnard: Brilliant. I think we’ve covered the main ground. Anything I’ve missed?

Fabrice Canel: Just this: it’s all about “now.” IndexNow is a very easy API to use. Please look at the documentation. It’s a simple GET request you can literally type into your browser to notify all participating search engines of a change. It’ll be around for a very long time — just like sitemaps and other established protocols between search engines. Simple design ensures longevity.

Jason Barnard: You’ve just said it’s incredibly simple and will be around for 50 years. Does that mean you think you’ve nailed it? Will the functionality evolve, or is it essentially fixed — a ping to create, update, or delete?

Fabrice Canel: It’s a simple ping. There’s no need to specify whether it’s an update or a delete. It’s just: this URL has changed. Search engines figure out the rest. Maybe we don’t have the URL at all, and for us it’s new. Maybe we have it and it’s an update. Maybe the content is gone and we’ll mark it as a dead link. All of that is handled on our side. We’ve collected extensive feedback from many search engines before announcing the protocol, and from the learning we got from the Bing Webmaster Tool API and the broad industry discussions. We believe this protocol is well-established and will not change for a very long time.

Jason Barnard: So the conclusion I’m drawing is: whether we like it or not, this is coming. And actually, it is helpful, it does keep content fresh, and however much we might want to resist, it’s going to happen anyway.

Fabrice Canel: Yes, it will. With Cloudflare covering 20% of the internet, major content management systems, Akamai and other CDNs coming on board, plus top sites like eBay and many others — you’re covering a really large part of the internet. And maybe a large search engine will adopt someday, and we will all contribute to the quality and health of the whole internet.

Jason Barnard: I’m going to stick my neck out and say I’d bet my bottom euro that Google adopts within the next few months. And then the whole thing will explode and we’ll all be very happy. Thank you so much, Fabrice. That was absolutely brilliant.

[Baton pass to next guest]

Jason Barnard: Fabrice, could you pass the baton to our next guest, Luke?

Fabrice Canel: I will ask Luke whether he plans to use IndexNow for e-commerce websites. eBay and many others are already adopting. E-commerce is a perfect use case: if you have a product listing, if you sell something, and it doesn’t get indexed, that’s a problem. If your price changes, if Cyber Monday is coming and you want your latest offers indexed everywhere — IndexNow delivers that. Please use IndexNow for e-commerce sites.

Jason Barnard: I love that — I asked what big point I’d missed, and you were saving it for the baton pass. Absolute genius, Fabrice. Thank you very much.