Consumers over bots: Why site crawlers don’t hold all the answers

Posted by

Advances in technology have driven efficiency in SEO, where site crawlers such as Botify, DeepCrawl and ScreamingFrog have flourished.

These tools are an essential part of the SEO toolbox and are great at uncovering and visualizing technical issues such as broken links, 404 errors and invalid canonical tags. They are becoming the default source of technical performance analysis for SEOs, which means they spend less time interacting with, and analyzing, websites in a browser and/or site analytics.

On the surface, this doesn’t look like anything to be concerned about; we’re getting vast amounts of technical analysis at speed with tools.

However, these tools are bots — they analyze the site’s source code looking for identifiable issues against an audit checklist which, while useful, won’t necessarily correspond to the issues consumers face.

Search results are centered around the consumer

As I discussed in a previous column, search engines are advancing at the fastest rate we’ve seen in many years, thanks to machine learning. These advances have allowed search engines to focus on providing the most relevant results, not only in terms of content, but also in terms of user experience (UX).

Studies from SEMrush and SearchMetrics both also reference user signals and the consumer experience, including mobile-friendliness, content relevancy, site speed, bounce rate/search sequence, time on site and content format as key ranking factors.

However, with site crawlers becoming the default for website analysis and reducing the time that SEOs spend analyzing physical websites, the consumer experience is being neglected, resulting in untapped opportunities to improve performance.

Additionally, as of last November, consumers accessing the web via mobile devices overtook desktop for the first time. This further disconnects site crawlers from consumer behavior, and while site crawlers are catching up, they predominantly still default to desktop analysis.

With this in mind, it is critical to analyze and diagnose websites in the same way consumers interact with them, in addition to bot usage.

Prioritize user signals and consumer experience

A consumer-first approach is essential to succeed in the search results, now and in the future. To deliver this, we need to put ourselves in the shoes of the consumer and interact with websites, as well as analyze site analytics and Google Search Console, rather than purely relying on site crawlers.

[Read the full article on Search Engine Land.]

Some opinions expressed in this article may be those of a guest author and not necessarily Marketing Land. Staff authors are listed here.

About The Author

David, is Business Director – Owned Performance at Ecselis, Havas Group and is responsible for leading and developing their propositions and service solutions, specifically around SEO, Content and CRO in-order to deliver long term strategic results for clients. David has over ten years’ experience in delivering business growth across start-ups to multi-national enterprise level clients, as well as launching brands’ web presence in new markets. Having worked client side across paid, earned and owned media and held senior SEO agency side roles, David brings thought leadership from multiple viewpoints and a leading performance-driven approach. David was named the 12th most influential search marketer in The Drum’s Search Top 50 in 2014. In addition, David speaks at conferences such as SES and Search Leeds.


Leave a Reply

Your email address will not be published. Required fields are marked *