New search technologies are revolutionising the power of site search.
Site search, the way users search your website, has historically been keyword-based. It’s also usually not been very good. Users' familiarity with search engines like Google leads them to expect excellent search experiences when searching a website, expectations which tend to be disappointed by the realities of site search.
The fundamental issue is that most site search experiences have been based on keyword matching, whereas a search experience like Google has been built with a multi-billion-dollar investment in indexing and ranking across a range of measures like backlink profile, recency, degree of engagement, and more.
It doesn’t really need stating that website teams in charities, museums, universities and public bodies don’t have Google-sized budgets for their site search experiences. So site search has generally been the ‘problem child’ of web projects, often failing to deliver useful results if users don’t use common keywords.
That is all changing now. Two developments in AI are changing what site search can do: semantic search, and natural language interfaces that let people search by asking questions in a chat dialogue which allows the answers to be refined. Between them they hold the prospect of totally revolutionising the effectiveness of site search experiences, taking them from frequent disappointment to useful tool.
Semantic search
Semantic search is changing what is possible with site search experiences.
This approach is also known as vector search, and it works by turning every page of your content into a set of ‘vectors’ - numbers that capture its meaning, and relationship to other information and concepts. Every search query a user enters is also processed in the same way. The system then returns the content whose meaning is closest to the search term given. In this way, even if content does not contain keywords matching the search term, it will still be surfaced if it’s highly relevant to the topic in question.
For example, if a user searches ‘preventing boating accidents’, a page on that site about ‘maritime safety’ might not be surfaced in a traditional keyword search, if it does not contain those keywords used in the search term. However, with semantic search, the system knows that ‘maritime safety’ and ‘boating accidents’ are highly related terms, and would surface the page on maritime safety.
We’ve been experimenting with semantic search on Kingston University’s course finder. In this experiment, we tried using the sort of phrase a real prospective student might use: “how to build websites and apps”. ‘Basic’ keyword search returned five courses, and none of them had anything to do with building websites or apps. Semantic search returned much more relevant results, including a Game Development MSc and a User Experience Design MSc. Nobody had ever tagged those courses with the phrase “build websites and apps”. The search found them because it understood what the words meant.
This shows how semantic search can improve the user's experience whilst saving web teams time. Editors can do far less manual upkeep, because there is less need to ensure content hits certain keywords or rely on manual tagging for taxonomy.
Natural language search
The second shift is natural language, or conversational, search, powered by large language models (LLMs). Rather than forcing someone to guess the right keywords, it lets them say what they want in a sentence, then have a back-and-forth conversation. This is an experience users are increasingly familiar with thanks to the rapid adoption of LLMs like ChatGPT.
To return to the university course search example: your user might be a prospective student who isn’t sure what course is for them, but knows they're interested in working with computers. The interface can ask clarifying questions to tease out other preferences, and then provide information on courses on games development, software engineering, and computer science. The chat interface lets people refine their thinking as they go, getting to more relevant results than a single ‘one-shot’ request ever could.
This shift is underway now
This is not a distant prospect. The move to AI-powered site search experiences, whether semantic search or LLM-based chat interfaces, is well underway. This is just as well, as the rise of AI-generated answers means site search deserves more attention now rather than less. This is because as external AI answers mop up the easy questions, so the people who still land on your own search box arrive with sharper intent, more complex queries, and higher expectations.
If you’re considering investing in improving your site search, there’s an obvious place to start. Review your own search logs: the queries that return nothing, and the ones people abandon. That list helps make the argument for change, as it shows real users being failed by your current search approach.
If you want to explore what your options are with site search, let’s have a chat. We can set up semantic or conversational search against your own content, even if we’re not currently looking after your website. We’d love to show you what it turns up.
