Have AI search your website and return cited content

January 2024
1 min read
by Scott Euser

In a constantly evolving digital landscape, we’re always keeping an eye on how things are changing and what new developments might mean for the knowledge-led organisations we work with. Over the past couple of years, it’s been the rapid advances in Artificial Intelligence (AI) and Machine Learning (ML) that have captured a lot of our attention.

Back in August, OpenAI announced GPTBot. GPTBot is a web crawler that uses the content on your website to train its AI models. Website owners have the option to allow the crawler to access their website content or to limit or restrict it.

What should think tanks do? We’ve had conversations with a number of our clients – and the general consensus is that it’s better to let AI be informed by valuable, reliable research than for it to miss that information and give less accurate answers. However, AI models not citing content sources when providing an answer has been flagged repeatedly as a major concern.

So it was good to see the recent progress coming from Google’s Gemini – their own AI model (not yet available in Europe) – which has been benchmarked to match and slightly exceed the quality of results of OpenAI’s ChatGPT 4.

To test Gemini out, we asked it what the IFS has to say about early years funding for 2023. We were pleased to get a few summary bullet points along with links to three key articles from the IFS, where we could then go to read more information.

In the future, we hope to see other AI models start to provide links to sources and quotes from reputable organisations. We’ll continue to monitor developments closely and work with our clients to integrate new solutions that AI offers, as well as navigate its challenges.

If you’d like to find out more about how you can use AI solutions on your website, get in touch with us at [email protected].

Scott is Director of web development at Soapbox and an active contributor to Drupal. He is based in London.

Like what you see? Let’s work together

Get in touch