We’ve been discussing planning an AI based project for students this summer. One of the things we discussed was using AI to create a natural language interface to Duke’s knowledge base of technical information. As a test, I wanted to see what information already existed based on ChatGPT “scraping” the internet for public info.
I asked ChatGPT what media production services exist at Duke University.
The response provided several outdated and no longer accurate information. Duke Studios ceased operation in 2020, DDI no longer exists and Duke Photography also no longer operates.
We know that ChatGPT-3 is based on old information and is not connected to the internet so this was not unexpected.
I then thought to try to ask the same question to Bing. I’m not clear if Bing is using ChatGPT-4 and/or how the whole “connection to the internet” works, but I expected a better answer. Unfortunately, I got less of an answer and still outdated info.
The cool thing about Bing was that it provided sources of information leading me to believe it WAS connected to the internet. So why was the information still incorrect? Well, it kinda wasn’t. I clicked on the first link and found this page still exists on the internet:
The AI couldn’t understand that while the information is still there, there’s also a disclaimer saying the service is now closed. Pretty clearly indicates a limitation of the technology.
For our project, we’d like to be able to control the corpus of information that feeds the AI engine to make sure it is all accurate. It’s my thought that businesses would be more interested in using this type of interface against a specific set of data, that is perhaps secure, rather than adding it to all the data on the internet and hoping it finds the most accurate and up to date info. Stay tuned.
