Texas Attorney General Ken Paxton has reached an agreement with Pieces Technologies, a Dallas-based artificial intelligence healthcare technology company. The agreement addresses allegations that the company deployed its products in several Texas hospitals following false and misleading statements regarding their accuracy and safety.
At least four major Texas hospitals provided real-time healthcare data to Pieces Technologies, enabling its generative AI product to "summarize" patients' conditions and treatments for hospital staff. An investigation by the Texas Attorney General alleged that Pieces made deceptive claims about the accuracy of its healthcare AI products, posing risks to public interest. The company developed metrics claiming high accuracy for its AI products, including an advertised error rate or "severe hallucination rate" of "<1 per 100,000."
Attorney General Paxton's investigation concluded that these metrics were likely inaccurate and may have misled hospitals about the safety and reliability of Pieces' products.
As part of the agreement , Pieces Technologies has agreed to accurately disclose the true extent of its products' accuracy. The company will also ensure that hospital staff using its generative AI products are fully informed about how much they should rely on these tools.
"AI companies offering products used in high-risk settings owe it to the public and to their clients to be transparent about their risks, limitations, and appropriate use. Anything short of that is irresponsible and unnecessarily puts Texans’ safety at risk," said Attorney General Paxton. "Hospitals and other healthcare entities must consider whether AI products are appropriate and train their employees accordingly."
Pieces Technologies, however, says it was disappointed Paxton's press release misrepresents the assurance of voluntary compliance it entered into. It denies wrongdoing and said it accurately set forth its hallucination rate.
Pieces' statement says the press release makes these specific misrepresentations.
● The AVC makes no mention of the safety of Pieces products, nor is there evidence indicating that the public interest has ever been at risk.
● The AVC focuses solely on the company’s reporting of hallucination rates in the context of an independently developed risk-classification system that is based on severity.
Importantly, there is no industry-wide risk classification system for generative AI hallucinations for inpatient clinical summarization that exists today. Pieces is, in fact, a trailblazer in the development of a risk classification system, which took several years to build and is an effective way to monitor and ensure highly reliable and quality outcomes of its AI-generated working summaries.
● This is not a financial settlement, nor is there any financial penalty associated with this matter. This is an Assurance of Voluntary Compliance into which Pieces entered with the Texas OAG to demonstrate leadership, transparency, and collaboration. It sunsets in five years, which is not the norm, and can be rescinded by the OAG as early as one year from its effective date upon request.
"Pieces strongly supports the need for additional oversight and regulation of clinical generative AI, and the company signed this AVC as an opportunity to advance those conversations in good faith with the Texas OAG," the company says.
"Despite the disappointing and damaging misrepresentation of this agreement in the Texas OAG’s press release, Pieces will continue to work collaboratively at both state and national levels with organizations that share a common commitment to advancing the delivery of high quality and safe health care across communities."