How to Optimize Content Using GenAI Powered Search Analytics?

Keyword-Based Analytics Vs. Prompt-Based Analytics

Given the proliferation of GenAI technology, many organizations have deployed ChatGPT-like search on their knowledge base to answer the questions raised by their customers. Many modern-age customers prefer ChatGPT-like search compared to traditional Google-like search based on keywords. The analytics involved in analyzing “prompts/questions” which have different characteristics compared to keywords. The tables below show the nature of lexical keyword and prompt-based analytics

Factors to Consider in Prompt-Based Analytics

Here are the factors you need to look for when enhancing content engagement and user journeys using prompt-based analytics

Prompt Analysis

Technical writers can access the list of questions (prompts) that have been raised by their customers which gives them better clarity on

• What kind of questions that my customers type in
• What are the key business keywords that are often used by your customers, that relate to your business glossary?
• What types of questions are commonly asked, and what types of similar question
• Why are those questions being typed in, and how do they correlate with other business activities
• What information is most commonly been sought

This helps technical writers better understand the customers’ intent, leading to better business outcomes. 

Topical Analysis

Topical analysis refers to the process of analyzing questions (prompts) to determine the topics they cover. This allows technical writers to get into the general themes customers are most frequently searching for. This analysis can be used to identify trends, understand customer needs, and guide content creation or product improvements.

1. Holistic topical analysis

Analysis of all questions leads to knowing what questions are trending and helps to delve into “why”. These topics can be correlated with any recent product launches, any changes to products or services, and so on. Introducing a time dimension to this topical analysis offers a perspective on trends and patterns in how your customers utilize your knowledge base.

2. “Unanswered questions” topical analysis

If you can filter out only those questions (prompts)where the GenAI-based assistive search could not respond because of lack of information or context, then undertaking topical analysis on these “unanswered questions” leads to creating new knowledge base content. This analysis will reveal the new information your customers are seeking but that you currently lack content for.

Citation Analysis

Usually, source articles that are used to create a response via GenAI are displayed as citations. These citations provide a trust factor to the customers. If citation articles for each response can be analyzed holistically, then we can understand the high-value content of your knowledge base. This analysis helps technical writers identify critical articles of the knowledge base and take steps to ensure those articles are always up to date. Stale content in the knowledge base can also be identified in the knowledge base using this analysis.

Usage Analysis

Search volume along with positive and negative feedback sliced across different time dimensions shows the usage statistics of your GenAI assistive search tool. Successful responses and “unanswered question” responses show the adoption of the GenAI assistive search tool as a primary means to interact with your knowledge base. The list of all questions, along with its feedback, can also be made available to technical writers if they choose to dive into individual questions over time!

Also read: Guidelines for Structuring code snippets in technical writing for GenAI-based agents