ChatGPT from OpenAI has garnered significant attention from various sectors, including academia and journalism, with opinions ranging from excitement about its potential uses to concerns about information integrity and skilled workers’ displacement. Nimonik has been investigating the implications of this new AI tool for regulatory compliance and will be discussing them in a series of articles.
In our first article, we explored whether ChatGPT could be an assistive or disruptive tool in creating themed lists of legal documents and extracting or summarizing critical information from these documents.
Before delving into our findings, it’s crucial to understand some key details about ChatGPT:
- It can’t search the internet for information. The date it uses is based on the Internet prior to 2021.
- A user can’t give data to ChatGPT as a basis for its answers.
- Its results are expected to rapidly improve with user feedback.
Nimonik put ChatGPT to the test for the following use cases:
- Creating lists of top-ranked documents based on combinations of jurisdiction, topic, and applicable industrial sector; for example, “What are the most important legal documents that Canadian mining companies need to comply with?” or “What are the essential worker safety regulations in France?”
- Creating lists of top-ranked obligations for companies within a specified legal document; for example, “What are the key obligations in the Canadian Labour Code?” or “What are the most important requirements in the United Kingdom’s Merchant Shipping Act, 1995?”
- Summarizing the contents of a specific document; for example, “What is the Quebec Environment Quality Act about?” or “Summarize the Canadian Onshore Pipeline Regulations”.
- Creating extracts of key clauses from a specific legal document; for example, “Provide the text of the ten most important requirements in Ireland’s Explosives Act 1875” or “Extract the text of five key clauses from Ontario’s Smoke-Free Places Act.”
Evaluating the results according to the data quality dimensions of accuracy, timeliness, comparability, usability, and relevance, we came to the following conclusions:
- ChatGPT’s output is well-written and easy to read. It creates lists and summaries that are well-organized and easy to understand.
- ChatGPT can give appropriate answers to a wide variety of questions related to document lists and contents.
- The output is out-of-date. In a world where legislation is constantly changing and new legal documents are published every day, ChatGPT’s utility is limited by its non-inclusion of recently published legislation, and its inability to consider recent changes to existing legislation.
- ChatGPT can’t extract (quote from) document text, and therefore can’t provide the text of key clauses or obligations.
- The output sounds accurate, but is routinely incorrect. We found many factual errors in document lists and summaries. Users won’t know whether each assertion is true unless they verify it, and the readability of ChatGPT’s text can lull the user into thinking that the information is correct because it sounds correct.
- ChatGPT’s references to documents and clauses often lack information that the user needs to easily access the source material. Without links or section numbers, and with document titles that don’t always match the official title, users who want to refer to the source material (e.g. the full text of a regulation mentioned in a list) have to do a lot of guessing and legwork.
With its ability to write text that’s well-organized and sounds accurate, ChatGPT can be a useful ally for drafting the outline of lists or summaries related to legal compliance. But unless users have a high tolerance for error and don’t mind if their data stops in 2021, ChatGPT has far to go before it can make life easier for companies that want to understand their legal obligations. However, it is clear that the technology to process, analyze and provide plain language answers has evolved tremendously. It is only a matter of time before substantial low level analysis work is replaced with technology similar to ChatGPT.