NewsSEO

IndexNow elicits a variety of reactions, including skepticism

Some members of the search community believe IndexNow provides no value, while others believe it is the future of search indexing.

While major companies such as Wix, Duda, and Cloudflare announced enthusiastic adoption of the new open-source IndexNow indexing protocol, which promises instant indexing, it was also met with questions about whether IndexNow was necessary, worth the trouble of adopting, and whether any of its benefits would be realized.

What exactly is IndexNow?

IndexNow is an open-source project that aims to improve how search engines discover and index content for the benefit of both search engines and publishers.

IndexNow also promises to significantly accelerate the process of re-indexing previously updated content as well as indexing newly created content.

Finally, IndexNow promises to be more energy efficient because it will crawl websites less frequently.

For decades, the traditional method of discovering web pages and indexing content has remained largely unchanged.

Crawlers repeatedly check web pages on the Internet, regardless of whether they are unchanged or do not exist.

While search engines use strategies to reduce the number of pages crawled, crawling an unchanged web page can be viewed as a waste of electricity, similar to watering a lawn during a rainstorm.

IndexNow claims to solve this problem by eliminating redundant crawls and focusing solely on new or updated pages.

However, not everyone is convinced.

Some members of the search community have expressed skepticism about every benefit IndexNow claims to provide, including the claim that fewer pages will be crawled.

IndexNow is not being adopted by Yoast

Joost de Valk, the creator of the Yoast WordPress SEO plugin, tweeted a series of statements that called IndexNow’s value into question. He stated that the IndexNow standard would not be adopted by the Yoast plugin.

He tweeted about why Yoast isn’t using IndexNow:

“We haven’t added support for it to @yoast SEO yet, simply because we don’t see the value yet.”

Yoast: IndexNow Isn’t Necessary

Joost then argued that IndexNow is attempting to solve a problem that already exists.

He claimed that XML sitemaps already do what IndexNow is attempting.

He wrote on Twitter:

“Ever since XML sitemaps were conceived, URL discovery as far as we’re concerned is a “solved problem”.

For most sites, getting search engines to crawl content is not a problem.

Yoast SEO generates XML sitemaps and it can even ping search engines when those change.”

IndexNow Is Beneficial for Big Websites

Someone replied to Joost, saying that the Bing API for fast indexing was useful for sites they worked on, such as Groupon and GitHub.

Joost recognizes the value of IndexNow for large sites.
He wrote on Twitter:

“I see the value for large sites like that.”

But doing this and making it the default for 12 million sites, the majority of which have fewer than 100 pages, seems excessive.”

Duda explains why IndexNow is valuable

Duda, a professional site-building platform that competes with WordPress and Wix, is one of the companies that is supporting IndexNow.

Duda is a website creation platform that enables web professionals to create a web presence that works well right out of the box.

However, it is flexible in the same way that WordPress is, allowing customers to create custom widgets, embed the website builder within their software (SaaS), or migrate legacy sites entirely on a large scale.

Read How to Run an Outreach Link Building Campaign?

Joost recognizes the value of IndexNow for large sites.

He wrote on Twitter:

“I see the value for large sites like that.”

But doing this and making it the default for 12 million sites, the majority of which have fewer than 100 pages, seems excessive.”

Duda explains why IndexNow is valuable

Duda, a professional site-building platform that competes with WordPress and Wix, is one of the companies that is supporting IndexNow.

Duda is a website creation platform that enables web professionals to create a web presence that works well right out of the box.

However, it is flexible in the same way that WordPress is, allowing customers to create custom widgets, embed the website builder within their software (SaaS), or migrate legacy sites entirely on a large scale.

“There is a delay between publishing, crawl, index, and finally showing in search results.

Right now search engines must re-crawl all pages of a site to find new content and not on the update schedule of an actual site update.

The difference here is that you’re telling Bing to only crawl these updated pages and do it now. A push, rather than a pull, approach is a widely accepted best practice of updating data/content between different systems efficiently.”

I reminded Russ of Joost’s insistence that XML sitemaps were adequate and that IndexNow was a better value for large sites.

Russ responded, saying:

“I do agree with Joost that for a lot of websites, it is not the most important capability to have at the moment.

But, I think this approach to indexing is a lot better than the traditional sitemap system and the industry should move towards ‘pushing’ content to search engines for them to crawl.”

While I waited for Google to adopt IndexNow, I sought the advice of Ammon Johns (@Ammon Johns), a long-time UK search marketing expert. He, like Joost, was unimpressed with IndexNow and did not believe it provided any value.

Ammon acknowledged the elephant that was not present in the IndexNow room, namely Google.

He stated:

“Unless and until Google sign up for a Bing led protocol, it’s not a huge deal.

But if Google do go along for some reason (rather than make their own), we’ll certainly all have to revisit a lot of understanding about crawl prioritization”

The promise of Less Crawling in Doubt Ammon downplayed any benefit of less crawling, saying that the freed-up resources would likely be used to crawl lower priority web pages that would not normally be crawled.

He stated:

“Even if this were to free up some resource allocated to crawling more than can be crawled (the state of the web currently, and growing exponentially), I can’t see the search engines not simply crawling more of the lower priority, or re-crawling the absolute highest with more frequency.

…Microsoft don’t even actually promise they’ll grab it at once, only that it will be indexed ‘soon’ or ‘quickly’.”

He then pointed out that using the IndexNow protocol will remove a latent signal of which pages Google or Bing consider to be the most important.

IndexNow is considering removing a data point.

Ammon remarked:

“One of the major reasons I never, EVER, use submission tools is that seeing when content gets crawled on its own is far too useful.

There’s no other way to see ‘importance’”

Links are one of several factors that influence how frequently a web page is crawled. The frequency with which a page or section of a website changes can also influence how frequently a web page is crawled.

It is up to the individual and their priorities whether fast indexing is worth foregoing the ability to see which pages a search engine considers to be more “important.” Ammon believes it isn’t worth it for him.

Will IndexNow be adopted by the web?

According to Joost’s initial post, they haven’t decided whether or not to adopt IndexNow because they don’t see the value in it.

One of the most important features of the Yoast SEO plugin is the ability to automatically generate XML sitemaps, which Joost believes is adequate for most websites.

If Google adopts the IndexNow protocol, which currently appears unlikely, XML sitemaps may lose some of their importance, dimming the shine on what is currently an important feature of the Yoast plugin.

Should Google use Index?

It now increases the likelihood that WordPress will include it in the core, eliminating the need for Yoast to turn it into a plugin feature.

As a result, it appears that IndexNow provides little to no value to Yoast.

It is up to the search community to make their own decision. IndexNow faces some criticism and skepticism.

However, many large companies, such as Cloudflare, are enthusiastically integrating IndexNow.

Joost acknowledged that IndexNow is useful for large corporations, but he believes it has less value for smaller websites.

As Ammon mentioned, for some in the search community, particularly smaller sites, unless Google adopts IndexNow, the decision will feel optional rather than mandatory.

Learn more from SEO and read Why Do Search Console and Analytics Differ in Discover Reporting?

Related Articles

Leave a Reply

Your email address will not be published.

Back to top button