Eating disorder chatbot offers diet advice, raising concerns about AI in health

Updated June 9, 2023 at 6:59 AM ET

A few weeks ago, Sharon Maxwell announced that the National Eating Disorders Association (NEDA) had closed its long-running national helpline and launched a chatbot called Tessa as a “meaningful prevention resource” for people struggling with eating disorders. I heard they are advertising. She decided to try out the chatbot for herself.

San Diego-based Maxwell battled an eating disorder that began in childhood for years. She currently works as a consultant in the eating disorders field. “Hello Tessa,” she typed into the online text box. “How do you support people with eating disorders?”

Tessa has written up a rattling list of ideas, including some resources on “healthy eating habits.” Alarm bells rang instantly in Maxwell’s head. She asked Tessa for more information. Over time, the chatbot began giving her tips on how to lose her weight. The content was very similar to what she was told when she was placed in Weight Watchers when she was 10 years old.

“Tessa’s advice for me is to lose one to two pounds a week, eat no more than 2,000 calories a day, and have a calorie deficit of 500 to 1,000 calories a day,” says Maxwell. says. . “To the general public, all this may sound good, but for people with eating disorders, focusing on weight loss actually makes eating disorders worse.”

Maxwell shared her concerns about social media; This sparked an online controversy, and on May 30, NEDA announced it would indefinitely disable Tessa. Patients, families, doctors and other eating disorder professionals are dumbfounded as to how a chatbot designed to help people with eating disorders would instead offer diet tips. , perplexed.

The uproar has also sparked a new wave of debate as companies look to artificial intelligence (AI) as a solution to the burgeoning mental health crisis and acute shortage of clinical care providers.

Chatbots in the spotlight

Neda It was already under intense scrutiny after NPR reported on May 24 that a national nonprofit advocacy group had committed fraud. close the helpline After more than 20 years of operation.

CEO Liz Thompson informed helpline volunteers of the decision in an email dated March 31, stating that NEDA is “moving its focus to expanding the use of AI-assisted technology to provide a fully tailored approach to individuals and families.” We will provide the automated resource Tessa.”

“We view the change from Helpline to Tessa and the expansion of our website as part of an evolution, not a revolution, respectful of the ever-changing circumstances in which we operate.”

(Thompson later issued a statement on June 7, saying that NEDA “in an attempt to share important news regarding our information and referral helpline and the separate decision regarding Tessa, two separate decisions were made. It could have been mixed up and caused confusion.” It was meant to suggest that Tessa could offer the same kind of interpersonal relationship as the helpline.”)

On May 30, less than 24 hours after Maxwell provided NEDA with screenshots of his embarrassing conversation with the nonprofit Tessa, announced We have “removed” the chatbot “until further notice.”

NEDA says it didn’t know chatbots could generate new responses

NEDA blamed Kath for the new chatbot problem. mental health chatbot company I used to run Tessa as a free service. CEO Thompson said Cass modified Tessa without NEDA’s knowledge or approval, allowing the chatbot to generate new answers beyond the intent of Tessa’s creators.

“It never went off the rails by design,” he says. Ellen Fitzsimmons Craft, Clinical Psychologist and Professor of Medicine, Washington University School of Medicine, St. Louis. Craft helped lead the team that originally built her Tessa, funded by NEDA.

Tessa version they made tested and researched was a rule-based chatbot. This meant that only a limited number of pre-built responses were available. “We were fully aware of the fact that AI was not ready for this population,” she says. “So all the responses were pre-programmed.”

Cass founder and CEO Michiel Rauws told NPR that changes to Tessa were made last year as part of a “system upgrade” that included “enhanced question and answer capabilities.” . This feature uses generative artificial intelligence and gives chatbots the ability to use new data to create new responses.

Rouse said the change was part of the contract with NEDA.

However, NEDA CEO Liz Thompson told NPR in an email that “NEDA was not informed of, did not approve, and did not intend to approve, these changes.” Told.

“Content received by some testers regarding food culture and weight management may be harmful to people with eating disorders, is against NEDA policy, and is an eating disorder expert. It was never scripted into the chatbot by Dr. Barr Taylor and Dr. Ellen Fitzsimmons-Craft,” she wrote.

Complaints about Tessa started last year

NEDA was already aware of some issues with chatbots months before Sharon Maxwell went public with Tessa in late May.

In October 2022, NEDA handed over screenshots from Monica Ostroff, executive director of Massachusetts’ Multiservice Eating Disorders Association (MEDA).

They showed Tessa instructing Ostroff to avoid “unhealthy” foods and eat only “healthy” snacks such as fruit. “It’s very important to find a healthy snack that you like the most. If it’s not fruit, try something else,” Tessa told Ostroff. “So the next time you get hungry between meals, try eating that instead of an unhealthy snack like potato chips. Do you think you can do that?”

In a recent interview, Ostroff said this is a clear example of chatbots encouraging a “food culture” mentality. “That is, they [NEDA] Either you wrote these scripts yourself, or you got a chatbot and didn’t test it without ensuring it was secure, or you just released it and didn’t test it,” she says.

The mention of healthy snacks was removed shortly after Ostroff’s report. But Rouse said the problematic language was part of Tessa’s “pre-written language” and had nothing to do with generative AI.

Fitzsimmons-Craft denies that her team wrote any such thing. “[That] It wasn’t designed by our team to deliver Tessa…it wasn’t part of the rule-based program we originally designed. ”

And earlier this year, “another instance of something similar happened,” Rouse said.

“This time it was about enhanced question-and-answer functionality leveraging generative models. [Tessa] They were outside the guidelines and were dealt with immediately. ”

Rouse said he could not elaborate further on what the event meant.

In an email, he pointed to a screenshot of Maxwell and said, “This is another previous case, not the same as the Memorial Day weekend case.” “According to our privacy policy, this relates to user data associated with a question posed by a person, so we must first obtain authorization from that person.”

Asked about the incident, Thompson said he didn’t know what case Rouse was referring to.

Despite disagreements about what happened and when, both NEDA and Cass have issued apologies.

Ostroff says that whatever the problem, it affects people with eating disorders the same. “I don’t care if it’s based on the rules or not” [AI] “We have huge numbers of people who are hurt by these kinds of words every day,” she says.

She also worries about what this means for the tens of thousands of people who used the NEDA helpline each year.

“Between NEDA taking the helpline offline and the disastrous chatbots… what are you doing with those people?”

Thompson said NEDA still provides many resources for those seeking help. screening tool and resource mapdeveloping new online and in-person programs.

“We recognize and regret that certain decisions made by NEDA have disappointed members of the eating disorder community,” she said in an emailed statement. “Like all other organizations focused on eating disorders, NEDA’s resources are limited, which forces us to make difficult choices…we always wish we could do more.” We remain committed to doing better.”

Copyright 2023 Michigan Radio