{"id":2020,"date":"2021-12-10T09:01:00","date_gmt":"2021-12-10T09:01:00","guid":{"rendered":"https:\/\/seopolarity.com\/blog\/?p=2020"},"modified":"2021-12-10T16:47:47","modified_gmt":"2021-12-10T16:47:47","slug":"how-to-use-python-to-automate-seo-keyword-clustering-based-on-search-intent","status":"publish","type":"post","link":"https:\/\/seopolarity.com\/blog\/how-to-use-python-to-automate-seo-keyword-clustering-based-on-search-intent\/2021\/","title":{"rendered":"How to Use Python to Automate SEO Keyword Clustering Based on Search Intent"},"content":{"rendered":"\n

There’s a lot to learn about search intent, from using deep learning to infer search intent by classifying text and breaking down SERP titles with Natural Language Processing (NLP) techniques, to clustering based on semantic relevance, with the benefits explained.<\/p>\n\n\n\n

Not only do we understand the benefits of deciphering search intent, but we also have a number of techniques for scale and automation at our disposal.<\/p>\n\n\n\n

However, this frequently entails creating your own AI. What if you don’t have the time or knowledge to do so?<\/p>\n\n\n\n

In this column, you’ll learn how to use Python to automate keyword clustering based on search intent.<\/p>\n\n\n\n

SERPs Provide Insights Into Search Intent<\/h2>\n\n\n\n

Some methods necessitate extracting all of the copy from the titles of the ranking content for a given keyword, then feeding it into a neural network model (which you must then build and test), or perhaps you’re using NLP to cluster keywords.<\/p>\n\n\n\n

There is another method that allows you to use Google’s AI to do the work for you instead of scraping all of the SERPs content and building an AI model.<\/p>\n\n\n\n

Assume that Google ranks site URLs in descending order based on the likelihood of the content satisfying the user query. As a result, if the intent for the two keywords is the same, the SERPs will most likely be similar.<\/p>\n\n\n\n

Read What Is Rank Tracker in SEO? The Advantages of the Rank Tracking Tool<\/a>.<\/p>\n\n\n\n

To stay on top of Core Updates, many SEO professionals have compared SERP results for keywords to infer shared (or shared) search intent for years, so this is nothing new.<\/p>\n\n\n\n

The value-add here is the comparison’s automation and scaling, which provides both speed and greater precision.<\/p>\n\n\n\n

How To Cluster Keywords At Scale Based On Search Intent Python Programming (With Code)<\/h2>\n\n\n\n

Begin by downloading your SERPs results as a CSV file.<\/p>\n\n\n\n

1. Import The List Into Your Python Notebook<\/h2>\n\n\n\n
import pandas as pd\nimport numpy as np\n\nserps_input = pd.read_csv('data\/sej_serps_input.csv')\nserps_input<\/pre>\n\n\n\n
The SERPs file has now been imported into a Pandas dataframe, as shown below.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
2. Data Filtering for Page 1<\/h2>\n\n\n\n
We’d like to compare the Page 1 results of each SERP for different keywords.<\/p>\n\n\n\n
Because we want to filter at the keyword level, we’ll split the dataframe into mini keyword dataframes to run the filtering function before recombining into a single dataframe:<\/p>\n\n\n\n
# Split \nserps_grpby_keyword = serps_input.groupby(\"keyword\")\nk_urls = 15\n\n# Apply Combine\ndef filter_k_urls(group_df):\n    filtered_df = group_df.loc[group_df['url'].notnull()]\n    filtered_df = filtered_df.loc[filtered_df['rank'] <= k_urls]\n    return filtered_df\nfiltered_serps = serps_grpby_keyword.apply(filter_k_urls)\n\n# Combine\n## Add prefix to column names\n#normed = normed.add_prefix('normed_')\n\n# Concatenate with initial data frame\nfiltered_serps_df = pd.concat([filtered_serps],axis=0)\ndel filtered_serps_df['keyword']\nfiltered_serps_df = filtered_serps_df.reset_index()\ndel filtered_serps_df['level_1']\nfiltered_serps_df<\/pre>\n\n\n\n
3. Converting Ranking URLs To Strings<\/h2>\n\n\n\n
Because there are more SERP result URLs than keywords, we must compress those URLs into a single line to represent the SERP for the keyword.<\/p>\n\n\n\n
Here’s how it works:<\/p>\n\n\n\n
# convert results to strings using Split Apply Combine\nfiltserps_grpby_keyword = filtered_serps_df.groupby(\"keyword\")\ndef string_serps(df):\n    df['serp_string'] = ''.join(df['url'])\n    return df    \n\n# Combine\nstrung_serps = filtserps_grpby_keyword.apply(string_serps)\n\n# Concatenate with initial data frame and clean\nstrung_serps = pd.concat([strung_serps],axis=0)\nstrung_serps = strung_serps[['keyword', 'serp_string']]#.head(30)\nstrung_serps = strung_serps.drop_duplicates()\nstrung_serps<\/pre>\n\n\n\n
The SERP for each keyword is shown below, compressed into a single line.<\/p>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n
4. Examine SERP Similarity<\/h2>\n\n\n\n
To perform the comparison, we now need every keyword SERP combination paired with other pairs:<\/p>\n\n\n\n
# align serps\ndef serps_align(k, df):\n    prime_df = df.loc[df.keyword == k]\n    prime_df = prime_df.rename(columns = {\"serp_string\" : \"serp_string_a\", 'keyword': 'keyword_a'})\n    comp_df = df.loc[df.keyword != k].reset_index(drop=True)\n    prime_df = prime_df.loc[prime_df.index.repeat(len(comp_df.index))].reset_index(drop=True)\n    prime_df = pd.concat([prime_df, comp_df], axis=1)\n    prime_df = prime_df.rename(columns = {\"serp_string\" : \"serp_string_b\", 'keyword': 'keyword_b', \"serp_string_a\" : \"serp_string\", 'keyword_a': 'keyword'})\n    return prime_df\n\ncolumns = ['keyword', 'serp_string', 'keyword_b', 'serp_string_b']\nmatched_serps = pd.DataFrame(columns=columns)\nmatched_serps = matched_serps.fillna(0)\nqueries = strung_serps.keyword.to_list()\n\nfor q in queries:\n    temp_df = serps_align(q, strung_serps)\n    matched_serps = matched_serps.append(temp_df)\n\nmatched_serps<\/pre>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n
The preceding table displays all of the keyword SERP pair combinations, preparing it for SERP string comparison.<\/p>\n\n\n\n
Because there is no open source library that compares list objects by order, the function below was written for you.<\/p>\n\n\n\n
The’serp compare’ function compares the overlap of sites and the order of those sites across SERPs.<\/p>\n\n\n\n
import py_stringmatching as sm\nws_tok = sm.WhitespaceTokenizer()\n\n# Only compare the top k_urls results \ndef serps_similarity(serps_str1, serps_str2, k=15):\n    denom = k+1\n    norm = sum([2*(1\/i - 1.0\/(denom)) for i in range(1, denom)])\n\n    ws_tok = sm.WhitespaceTokenizer()\n\n    serps_1 = ws_tok.tokenize(serps_str1)[:k]\n    serps_2 = ws_tok.tokenize(serps_str2)[:k]\n\n    match = lambda a, b: [b.index(x)+1 if x in b else None for x in a]\n\n    pos_intersections = [(i+1,j) for i,j in enumerate(match(serps_1, serps_2)) if j is not None] \n    pos_in1_not_in2 = [i+1 for i,j in enumerate(match(serps_1, serps_2)) if j is None]\n    pos_in2_not_in1 = [i+1 for i,j in enumerate(match(serps_2, serps_1)) if j is None]\n    a_sum = sum([abs(1\/i -1\/j) for i,j in pos_intersections])\n    b_sum = sum([abs(1\/i -1\/denom) for i in pos_in1_not_in2])\n    c_sum = sum([abs(1\/i -1\/denom) for i in pos_in2_not_in1])\n\n    intent_prime = a_sum + b_sum + c_sum\n    intent_dist = 1 - (intent_prime\/norm)\n    return intent_dist\n# Apply the function\nmatched_serps['si_simi'] = matched_serps.apply(lambda x: serps_similarity(x.serp_string, x.serp_string_b), axis=1)\nserps_compared = matched_serps[['keyword', 'keyword_b', 'si_simi']]\nserps_compared<\/pre>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n
We can begin clustering keywords now that the comparisons have been completed.<\/p>\n\n\n\n
We will treat any keywords that have a weighted similarity of 40% or higher.<\/p>\n\n\n\n
# group keywords by search intent\nsimi_lim = 0.4\n\n# join search volume\nkeysv_df = serps_input[['keyword', 'search_volume']].drop_duplicates()\nkeysv_df.head()\n\n# append topic vols\nkeywords_crossed_vols = serps_compared.merge(keysv_df, on = 'keyword', how = 'left')\nkeywords_crossed_vols = keywords_crossed_vols.rename(columns = {'keyword': 'topic', 'keyword_b': 'keyword',\n                                                                'search_volume': 'topic_volume'})\n\n# sim si_simi\nkeywords_crossed_vols.sort_values('topic_volume', ascending = False)\n\n\n# strip NANs\nkeywords_filtered_nonnan = keywords_crossed_vols.dropna()\nkeywords_filtered_nonnan<\/pre>\n\n\n\n
We now have a potential topic name, keywords with SERP similarity, and search volume for each.<\/p>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n
Keyword and keyword b have been renamed to topic and keyword, respectively.<\/p>\n\n\n\n
Using the lamdas technique, we’ll iterate over the columns in the dataframe now.<\/p>\n\n\n\n
The lamdas technique is more efficient than the.iterrows() function for iterating over rows in a Pandas dataframe because it converts rows to a list.<\/p>\n\n\n\n
Here we go:<\/p>\n\n\n\n
queries_in_df = list(set(keywords_filtered_nonnan.topic.to_list()))\ntopic_groups_numbered = {}\ntopics_added = []\n\ndef find_topics(si, keyw, topc):\n    i = 0\n    if (si >= simi_lim) and (not keyw in topics_added) and (not topc in topics_added): \n        i += 1     \n        topics_added.append(keyw)\n        topics_added.append(topc)\n        topic_groups_numbered[i] = [keyw, topc]          \n    elif si >= simi_lim and (keyw in topics_added) and (not topc in topics_added):  \n        j = [key for key, value in topic_groups_numbered.items() if keyw in value]\n        topics_added.append(topc)\n        topic_groups_numbered[j[0]].append(topc)\n\n    elif si >= simi_lim and (not keyw in topics_added) and (topc in topics_added):\n        j = [key for key, value in topic_groups_numbered.items() if topc in value]        \n        topics_added.append(keyw)\n        topic_groups_numbered[j[0]].append(keyw) \n\ndef apply_impl_ft(df):\n  return df.apply(\n      lambda row:\n        find_topics(row.si_simi, row.keyword, row.topic), axis=1)\n\napply_impl_ft(keywords_filtered_nonnan)\n\ntopic_groups_numbered = {k:list(set(v)) for k, v in topic_groups_numbered.items()}\n\ntopic_groups_numbered<\/pre>\n\n\n\n
A dictionary with all of the keywords clustered by search intent into numbered groups is shown below:<\/p>\n\n\n\n
{1: ['fixed rate isa',\n  'isa rates',\n  'isa interest rates',\n  'best isa rates',\n  'cash isa',\n  'cash isa rates'],\n 2: ['child savings account', 'kids savings account'],\n 3: ['savings account',\n  'savings account interest rate',\n  'savings rates',\n  'fixed rate savings',\n  'easy access savings',\n  'fixed rate bonds',\n  'online savings account',\n  'easy access savings account',\n  'savings accounts uk'],\n 4: ['isa account', 'isa', 'isa savings']}<\/pre>\n\n\n\n
Let’s put that in a dataframe:<\/p>\n\n\n\n
topic_groups_lst = []\n\nfor k, l in topic_groups_numbered.items():\n    for v in l:\n        topic_groups_lst.append([k, v])\n\ntopic_groups_dictdf = pd.DataFrame(topic_groups_lst, columns=['topic_group_no', 'keyword'])\n                                \ntopic_groups_dictdf<\/pre>\n\n\n\n
\"\"<\/figure><\/div>\n\n\n\n
The search intent groups shown above are a good approximation of the keywords contained within them, which an SEO expert would most likely achieve.<\/p>\n\n\n\n
Despite the fact that we only used a small set of keywords, the method can clearly be scaled to thousands (if not more).<\/p>\n\n\n\n
Activating the Outputs to Improve Your Search<\/h2>\n\n\n\n
Of course, the preceding could be taken a step further by using neural networks to process the ranking content for more accurate cluster and cluster group naming, as some commercial products already do.<\/p>\n\n\n\n
For the time being, you can use this output to:<\/p>\n\n\n\n