OctoML says it has ended its business relationship with Civitai days after an investigation by revealed the text-to-image platform was being used to generate images that “could be categorized as child pornography.” Civitai previously relied on OctoML’s OctoAI for image generation. While OctoML initially indicated it would continue working with Civitai and introduced new measures to curb the creation of harmful images, reported on Saturday that it has now decided to cut ties with the platform altogether.
According to 404 Media’s December 5 report, internal communications showed that OctoML was aware some Civitai users were creating sexually explicit material that included nonconsensual images of real people and pornographic depictions of children. In a followup report this weekend, the publication noted that OctoML rolled out a filter to block the generation of all NSFW content on Civitai before announcing its decision to pull out. Civitai also added new moderation methods in response to the investigation earlier this week, including a mandatory embedding called Civitai Safe Helper (Minor) that bars the model from generating images of children if “a mature theme or keyword is detected,” according to 404.
Andreessen Horowitz-backed Civitai has previously that challenges users to generate realistic images of real people for rewards. In November, found it was being used to make nonconsensual deepfakes of celebrities, influencers and even a private citizen that were, in many cases, sexual in nature. The subjects were primarily women. According to 404 Media, Civitai has since added a filter to prevent the creation of NSFW content featuring “certain” celebrities.
Nevertheless, OctoML — which uses Amazon Web Services’ servers — no longer wants to be involved. In a statement to 404 Media on Saturday, OctoML said, “We have decided to terminate our business relationship with Civitai. This decision aligns with our commitment to ensuring the safe and responsible use of AI.”