Many people are silent when it comes to SEO for Bing because there isn’t much information available. Surprisingly, many cutting-edge technologies and techniques were used at Bing prior to Google. Fabrice Canel, Principal Program Manager at Bing, recently shared a wealth of knowledge with Jason Barnard of Kalicube about how search engines work in general.
Fabrice is in charge of the Bingbot Crawler, URL Discovery and Selection, Document Processing, and Bing Webmaster Tools. He’s an excellent resource for information on search engines, particularly crawling and page selection.
Fabrice describes the crawling process here, and the key takeaway for me is that he says Bing is picky about what it chooses to index.
Many people believe that every page on their website deserves to be ranked. However, neither Google nor Bing index everything.
They tend to leave certain types of pages behind.
The first feature of a page that Bing would want to index is usefulness.
Fabrice Canel explained:
“We are business-driven obviously to satisfy the end customer but we have to pick and choose.
We cannot crawl everything on the internet there is an infinity number of URLs out there.
You have pages with calendars. You can go to next day forever.
So it’s really about detecting what is the most useful to satisfy a Microsoft Bing customer.”
Key Domains and Bing
Fabrice then discusses Key Domains and how they are guided by key pages on the Internet to show them quality content.
This sounds like an algorithm that incorporates a seed set of trusted sites, and the further a site is from the key websites, the more likely it is to be spam or useless (Link Distance Ranking Algorithms)
I don’t want to put words in Fabrice’s mouth; the foregoing is simply my observation.
I’ll leave it to Fabrice to speak for himself.
Jason asked:
“Would you say most content on the web is not useful or is that exaggerating?”
Fabrice answered:
“I think it’s a little bit exaggerated.
We are guided by key pages that are important on the internet and we follow links to understand what’s next.
And if we really focus on these key domains (key pages), then this is guiding us to quality content.
So the view that we have of the internet is not to go deep forever and crawl useless content.
It’s obviously to keep the index fresh and comprehensive, containing all of the most relevant content on the web.”
What Causes Bing to Dig Deep Into Websites
Jason then inquires about websites that are thoroughly crawled. Obviously, getting a search engine to index all of a site’s pages is critical.
Fabrice walks you through the steps.
Jason asked:
“Right. And then I think that’s the key. You prefer going wide and going deep.
So if I have a site that’s at the top of the pile, you will tend to focus more on me than on trying to find new things that you don’t already know about?”
Fabrice provided a nuance answer, reflecting the complicated nature of what gets chosen for crawling and indexing:
“It depends. If you have a site that is specialized and covers an interesting topic that customer cares about then we may obviously go deep.”
Machines Select What to Crawl
We occasionally anthropomorphize search engines, saying things like “The search engine doesn’t like my site.”
In reality, algorithms have nothing to do with liking or trusting.
Machines are not trusting.
Search engines are machines that are pre-programmed with objectives.
Read 4 Ways A Fireplace Installer Can Boost Their Digital Marketing.
Fabrice explains how Bing chooses to crawl deep or not crawl deep:
“This is not me selecting where we go deep and not deep. Nor is it my team.
This is the machine.
Machine learning that is selecting to go deep or deeper based on what we feel is important for a Bing customer.”
That part about what is important for the customer is something to take note of. The search engine, in this case Bing, is tuned to identify pages that are important to customers.
When writing an article or even creating an ecommerce page, it might be useful to look at the page and ask, “How can I make this page important for the those who visit this web page?”
Jason then asked a question to elicit more information about what goes into deciding what is important to site visitors.
Jason asked:
“You’re just giving the machine the goals you want it to achieve?”
Fabrice responded:
“Absolutely. Yes.
The main input we give the the Machine Learning algorithms is satisfying Bing customers.
And so we look at various dimensions to satisfy Bing customers.
Again, if you query for Facebook. You want the Facebook link at the top position. You don’t want some random blogs speaking about Facebook.”
Crawling for search results is broken and needs to be updated
Jason inquires of Fabrice as to why IndexNow is beneficial.
Fabrice responds by explaining what crawling is today and how this nearly thirty-year-old method of finding content to index is in need of an update.
The traditional and current method of crawling is to visit a website and “pull” data from it, even if the web pages are the same and haven’t changed.
Search engines must visit the entire indexed web on a regular basis to see if any new pages, sentences, or links have been added.
Fabrice believes that the way search engines crawl websites should be changed because there is a better way to do so.
He explained the fundamental problem:
“So the model of crawling is really to learn, to try to figure out when things are changing.
When will Jason post again? We may be able to model it. We may be able to try to figure it out. But we really don’t know.
So what we are doing is we are pulling and pulling and crawling and crawling to see if something has changed.
This is a model of crawling today. We may learn from links, but at the end of the day, we go to the home page and figure it out. So this model needs to change.”
Fabrice next explained the solution:
“We need to get input from the website owner Jason and Jason can tell us via a simple API that the website content has changed, helping us to discover this change – to be informed of a change, to send the crawler and to get latest content.
That’s an overall industry shift from crawling and crawling and crawling and crawling to discover if something has changed…”
The Current State of the Search
Google refers to them as users or people who visit their website. Bing introduces the concept of searchers as customers, along with all of the little aphorisms about customers that are implicit in a customer-first approach, such as the customer is always right, and gives the customer what they want.
Steve Jobs once said about customers and innovation, which is relevant to Bing’s IndexNow but also to publishers:
“You can’t just ask customers what they want and then try to give that to them. By the time you get it built, they’ll want something new.”
Is Push the Future of Search?
IndexNow, a new push technology from Bing, has been released. It’s a method for publishers to alert search engines to new or updated web pages. This saves hosting and data center resources such as electricity and bandwidth. It also makes it easier for publishers to know that with a push method, the search engine will come and get the content sooner rather than later, as with the current crawl method.
Need help with our free SEO tools? Try our free Plagiarism Checker, Article Rewriter, Word Counter.
Learn more from SEO and read BigCommerce SEO Features & Ecommerce Success Tips.
