As a follow-up to our previous post on contracting pointers for services incorporating the use of artificial intelligence (AI), we discuss below some of the key questions to ask vendors that may be using AI in the provision of their services.
Some of the considerations discussed herein—especially as they relate to an AI model’s output—may not be relevant for nongenerative AI, which are AI systems that do not involve the generation of new content or data, but rather are systems for analyzing, processing, and making decisions based on existing data.
Type of Technology
What is the source of the vendor’s AI model? AI models may be built internally from a proprietary base, built internally from an open-source base, or licensed from a third party, or a third party may be a subprocessor through which the vendor’s team passes data to the model and receives back results.
Given that some AI vendors will not be developing their own AI models, it is important to understand the source of the vendor’s AI model to determine whether company data will be passed through to a third party, the AI model’s source and accuracy, and the standard of service one may be able to expect from the vendor.
Training Data
What type of training data does the AI model use? Does it use scraped data sets? The output of an AI model depends on the data that was used to train the model. As such, it is imperative to know whether the model is trained on reliable sources and the quantity of data that is used to train the model.
Similarly, it is also important to know how often the vendor refreshes its training data sets. Scraped data sets can lead to potential infringement claims, so understanding if a vendor obtains a license to use its training data will assist when assessing the potential for liability in using the vendor’s data.
Privacy
Does the vendor use personal data as part of its training data? If so, is personal data de-identified or anonymized before being used as training data? How does the vendor prevent re-identification of personal data used as input/training data sets?
What levels of consent did the vendor obtain from data subjects for processing personal data in the training data used by the AI model?
The inquiries regarding privacy are important because a company will want to know if the vendor will use any personal information provided by the company to the vendor as part of its training data. From a liability perspective, companies will not want to work with a vendor that is not complying with applicable privacy laws in its use and disclosure of personal information.
Model Explainability and Bias
How does the AI system mitigate inaccurate, biased, and underrepresented outputs? Transparency is crucial in AI as companies will not want to rely on information from an AI system that is not easily understandable.
It is also important to understand how a vendor tests and validates its AI model to ensure that the output of the model is accurate. Has the vendor established a risk management system related to the AI system with human oversight for the same? Companies can also request that the vendor provide case studies and test results that show the accuracy of its AI model’s output.
Integration, Training, and Support
Evaluate how seamlessly the vendor’s AI solutions can integrate with existing systems and infrastructure. Companies will also want to inquire if the AI solution can be customized to meet their business needs.
Initial and ongoing training is also key to the implementation of a successful AI solution for a company’s business. AI solutions are generally offered as SaaS models and, as with other hosted solutions, the vendor should provide certain performance metrics such as a service-level agreement for availability of the solution and error resolution.
Conclusion
Selecting an AI vendor requires a thorough assessment of the AI solution and the vendor’s practices in conjunction with a company’s business goals. The foregoing is a nonexhaustive list of some of the key considerations in selecting an AI vendor.