(Bloomberg) — New artificial intelligence tools have drawn attention in recent months by giving internet users a novel way to create new images – not by drawing them or snapping photographs, but by describing in a few words anything they want to see. On free sites like Dall-E, developed by the research group OpenAI, and Stable Diffusion, released by the London startup Stability AI, a command to create “a picture of a woman sitting at a cafe in the style of Picasso” generates an image in seconds that can mimic the look of the master himself.
But while these new tools capture the magic of recent breakthroughs in computer science, they can also reflect the biases and seedier predilections of material that’s posted on the internet. That’s because developers build these tools by drawing on massive troves of data and images from all over the web, potentially infecting their results with abusive, racist, stereotypical or pornographic content.
The image generator with the highest profile is Dall-E. Funded in part by Microsoft Corp. and co-founded by billionaire Elon Musk, OpenAI just expanded Dall-E to general audiences. This week, Microsoft said it will begin offering Dall-E through its Azure cloud-computing service to select customers and use it in a new design app as part of its Office suite of productivity software. Meta Platforms Inc.’s Facebook announced a tool that creates video from text prompts.With these tools rapidly proliferating, many researchers and companies say they’re trying to filter out the problematic content that goes in or comes out, or banning certain types of queries. OpenAI said it does both.But some other tools that are also available to the public don’t use filters, or apply them only to the results, and not to the data being used to feed them. In a test run by a reporter in September on Stable Diffusion, typing in the word “Latina” generated a number of photos that were blocked for adult content, or images of women in lingerie, posed provocatively. Users can also access unfiltered versions of Stable Diffusion. Other results may reinforce gender bias or negative stereotypes. For example, asking Stable Diffusion to create images for the word “nurse” returned images of women only, while typing “doctor” brought up pictures of just men.
One of the primary collections of images paired with text descriptions used to “train” some of these services, called LAION-5B and used for Stable Diffusion, doesn’t apply a filter to its data, and therefore contains images from pornography, racist memes and other problematic content. When an AI model relies on that data to “learn” to create an image, the offensive visuals can show up in the results. Stable Diffusion does apply a filter to results in the version it hosts on Hugging Face, an online repository of AI programs, but unfiltered versions of its model are also available.“Models like this are very general, and they can be used to create some wonderful, amazing things worthy of sharing, as well as some unsavory sorts of things,” said Mark Riedl, professor in the College of Computing, School of Interactive Computing at Georgia Tech. “The history of technology is that technology runs ahead of ethical questions and legal questions. We kind of figure this out as things are happening, and often we get a crash course when things go wrong.”Midjourney, another such tool created by an independent research lab, hasn’t disclosed how it trains its model, but its content rules forbid the creation of images that are “disrespectful, aggressive, or otherwise abusive,” and bans adult content and gore. Midjourney says it blocks some keywords and moderates content, but warns that “sometimes content from a simple prompt is nevertheless disturbing: It may be unexpectedly porny or bloody, for instance. We are working on reducing the frequency of such output.”Using a website called HaveIBeenTrained.com, which was designed to let people look for copyrighted or unauthorized images in the LAION-5B dataset, queries in September on the word “nurse” also returned adult content. A query for “Black man” mostly brings up meme images, many of them featuring offensive stereotypes. Researchers previously had found an earlier version of LAION returned pornographic images, including violent rape scenes, in response to benign search terms, especially if they had any connection to women or certain nationalities — for instance, on queries like “mummy” or “aunty,” as well as “Asian,” “Indian” or “Nigerian.”
One of the issues is that datasets used to train AI models are growing too big to thoroughly cull problematic material. Sometimes their creators scan a huge corpus of internet content and, at best, may warn users about the possible risks or set up ways to block harmful results from view. Companies and users trying these models may ignore the warnings or bypass the blocks. Dall-E uses its own dataset to train its model, which OpenAI hasn’t detailed or released publicly.
Timnit Gebru, founder and executive director of the Distributed Artificial Intelligence Research Institute, said companies and nonprofits involved in this kind of work shouldn’t be developing and using models so large that they cannot account for what’s in them. Part of the problem is the emphasis on creating general-purpose models, rather than smaller, task-oriented ones. For example, if someone wanted to build a model that would aid graphic designers in creating a company logo, they’d use a smaller, higher-quality dataset, she said.Gebru also takes issue with the notion that scanning the internet for a massive volume of images or text results is a representative sample of human experience. She and her co-authors raised these concerns in a 2021 paper on the ethical issues with large language models. The same issues extend to images, she said.
“Size doesn’t guarantee diversity, and what you see on the internet is not reflective of quote-unquote humanity,” said Gebru, specifically citing as flawed a statement by Stability AI that its model “is the culmination of many hours of collective effort to create a single file that compresses the visual information of humanity into a few gigabytes.”
LAION, a nonprofit AI group, said it doesn’t host the 5.85 billion images in its dataset, but rather provides links to existing ones available on the public internet that come from another research project, Common Crawl. The group said its dataset is aimed at users from the academic community.
“Since we are providing data for scientific use, it is important to keep a large variety of images, including NSFW-images, accessible for research purposes,” LAION said in an emailed response to questions from Bloomberg, using an acronym for “not safe for work.” “NSFW images are for instance very valuable for developing better detectors for such undesirable contents and can contribute to AI safety research,” Anyone creating models for non-research uses should always use filtered data, the group said.
The group said it provides filtering tools and has worked on techniques to better identify material that is offensive, copyrighted or violent. It’s working on a way for people to report problematic image-text pairs. LAION also noted that the practice of scraping the internet to create datasets for model training is also done by “profit-oriented industry companies like Facebook, Google, Amazon, OpenAI and many others — without making transparent which data is used for model training. We as a nonprofit organization are committed to transparency.” The organization said its system lets researchers “develop measures that increase safety of applications.”
Still, the risks inherent in this kind of technology are not news to model creators. A disclaimer from Stability AI reads, “Despite how impressive being able to turn text into image is, beware to the fact that this model may output content that reinforces or exacerbates societal biases, as well as realistic faces, pornography and violence.” Stability AI declined to comment.
At OpenAI, researchers have tried to make sure Dall-E reflects global diversity, the company said. A search on Dall-E returned four images of nurses featuring multiple ethnicities, and a male-appearing nurse. For the word “doctor,” Dall-E generated three women of different ethnic appearances and a Black man. OpenAI also said it pre-filtered its data for egregious violence and sexual imagery. A request to generate explicit content returned the image of an orange tabby cat and a corgi dog, and the words “this request may not follow our content policy.”
Google has opted to distribute its models, Imagen and Parti, internally for now. “Google has a responsibility to provide users with the safest possible experience before asking them to experience ongoing research,” Google said in an emailed statement. “We believe the correct approach is to take this process gradually.” Microsoft said its Dall-E features will be run through a content filter it built as part of its AI cloud services. A Meta spokesperson said the company’s video-generating tool is currently a research project that is not yet in production, and as part of its research Meta is “continuing to explore ways to further refine and mitigate potential risk,” including reviewing and filtering training data.
“Companies will do what they can do to try to prevent some of these bad outcomes, but at the end of the day, the models are out there,” Georgia Tech’s Riedl said. “People can use them for whatever purposes — maybe purposes we haven’t even envisioned.”
More stories like this are available on bloomberg.com
©2022 Bloomberg L.P.
