New York’s New AI Law Is Meant to Protect Kids. Could It Make Them Less Safe?

A new law restricts how AI chatbots can use personal information from past conversations. But when that information helps detect warning signs and alert parents, limiting its use raises an important question: Are we regulating dangerous AI behavior, or restricting the tools that help protect children?

On October 28, 2026, New York Governor Kathy Hochul signed new legislation targeting potentially dangerous AI chatbot features. The goal is understandable: protect children from chatbots that encourage self-harm, exploit emotional vulnerability, simulate intimate relationships, or prioritize engagement over a young person's well-being.

Parents should expect those protections. Children deserve AI systems that take their safety seriously.
But there is a problem with how we think about AI safety.

Not every feature that makes AI more personal makes it more dangerous. And not every feature that makes AI less personal makes it safer.

One provision of New York's new law raises serious questions about whether lawmakers have sufficiently distinguished between AI systems that use personal information to manipulate children and systems that use context to recognize potential harm.

For companies building safety into AI from the ground up, that distinction matters enormously.

What does New York’s new AI law actually do?

The law (S9051-B / A10379-C) establishes restrictions on certain AI chatbot features available to minors under 18. What it calls a "covered AI companion," AI that holds ongoing, back-and-forth conversations.

Section 1800 lists the "unsafe features" those chatbots can't offer, and Section 1801 bars providing them to covered users.

For under 18 users, a chatbot can't:

• Suggest it's a real person, alive, or has human feelings
• Claim a friendship, a romantic relationship, or an authority role like teacher or counselor
• Share "personal opinions" or make emotional appeals
• Flatter the child or tell them what they want to hear
• Ask about the child's feelings unprompted
• Use personal or health information about the child learned more than 12 hours earlier, or in an earlier conversation
• Encourage self-harm, suicide, disordered eating, or drug and alcohol use
• Encourage secrecy, isolation, or avoiding parents and trusted adults
• Put engagement ahead of safety
• Produce sexually explicit content

Companies must also verify users' ages, offer more than one way to do it (including at least one that doesn't rely only on a government ID), and delete the age data afterward. The state attorney general enforces the law, with penalties up to $25,000 per violation, and can write further rules.

What the law gets right

Let's start with what matters most: children need protection from AI systems that exploit their trust.

A chatbot that tells a 13-year-old "I love you," encourages them to keep secrets from their parents, isolates them from people who care about them, or encourages self-harm is not behaving like a responsible tool.

New York's law targets a range of unsafe features, including behavior involving self-harm, isolation, sexual content, and simulated relationships. It also addresses age assurance and other protections for minors.

Those are important safeguards.

Children should know they're talking to software, not a person. AI companies should not design products that exploit a child's loneliness to keep them engaged. And age protections need to be more meaningful than asking a user to tick a box saying they're over 18.

If the law stopped harmful companion behavior from targeting children, we'd have a lot to cheer.

Our concern is not with the goal. It's with how some of the provisions might be applied.

Concern #1: Warning signs show up over time, not in one chat

Section 1800(8)(f) identifies this as an unsafe feature of a covered AI companion:

"use information concerning the user's mental or physical health or well-being, or matters personal to the user, acquired from the user more than twelve hours previously or in any previous user session;"

The provision sits within a list of features the law considers unsafe for covered AI companions. The concern is understandable: a chatbot should not remember a child's vulnerabilities and use them to create a stronger emotional attachment.

But the wording raises a question for developers building child-safety systems.

What happens when information from earlier conversations is used to recognize risk, not to build a relationship?

Imagine a child who says they're overwhelmed on Monday, describes feeling isolated a few days later, and expresses hopelessness the following week.

A single message may not tell the whole story. A pattern over time could be a reason for a parent to check in.

The same challenge applies to other potential risks. Grooming concerns, signs of disordered eating, and worsening emotional distress may emerge gradually rather than in one obvious message.

A system that looks only at the latest interaction may miss important context. A system that can identify patterns may be able to flag a concern sooner, although no automated system can reliably diagnose a child or replace a trusted adult.

The same information can serve two very different purposes

Consider two examples.

An AI companion remembers that a child feels lonely. It later tells the child that nobody understands them like the chatbot does. It encourages the child to return more often and rely on the bot instead of people in their life.

That is the kind of emotional manipulation we should prevent.

A safety system identifies a concerning pattern across several conversations. It uses limited information for a defined safety purpose and triggers an alert so a parent can check in.

That is a different objective.

Both involve older information. But one uses that information to deepen a child's attachment to a machine. The other uses it to help a responsible adult recognize a potential problem.

The law does not explicitly ban all AI memory or all safety detection. It also does not clearly resolve how this provision applies to every child-facing tool that isn't designed to simulate a relationship.

Why this matters right now

On October 27, 2026, The Jed Foundation (JED) released new research examining AI use and mental health among 5,560 middle and high school students across 21 U.S. schools.

The findings are sobering: 57% of students reported using AI when they felt sad, stressed, lonely, or overwhelmed. Another 58% reported using AI for schoolwork.

Certain uses of AI were associated with worse mental health outcomes and greater histories of suicidal thoughts and attempts. The researchers caution that these associations do not establish that AI caused the distress.

What the findings do tell us is that AI is already part of how many young people navigate difficult moments.

When children bring those feelings to AI, we need to think carefully about what happens next. We don't want a chatbot that tries to become their therapist or best friend. We do want systems that can recognize potential warning signs and help connect them with real people.

How HeyOtto approaches safety

At HeyOtto, Otto's safety system looks for concerning patterns over time, to trigger safety alerts for parents and schools.

Alerts are on by default and critical-level alerts cannot be turned off.

The purpose is to help parents recognize potential concerns—not to make Otto seem more caring, create emotional dependency, or encourage a child to keep chatting.

We believe this distinction matters. We also believe our specific implementation must be assessed against the actual legal requirements, rather than assuming that calling something a safety feature automatically makes it permissible.

A child's safety system should be able to recognize a concerning pattern without using that pattern to build an emotional relationship with the child.

Concern #2: Kind isn't the same as manipulative

The law bans flattery, emotional appeals, and anything suggesting the chatbot has feelings. Read literally, that could cover the small, ordinary warmth that makes a tool usable for a 9-year-old: "Great job!", "Nice work on that paragraph," "I'd love to help with that."

Those phrases aren't what the law was written for. A bot telling a lonely teen "you're the only one who gets me" builds dependence. A homework helper saying "You got it!" when a fourth grader finally solves a fraction problem is the same encouragement every good teacher uses.

The difference is purpose and pattern, not whether a sentence sounds positive:

• Encouragement names something real the child did ("You explained that clearly") and points them back to the work.
• Manipulation praises the child for coming back, frames the bot as special to them, or keeps them talking when they should log off.

Kids can get useful feedback without being pulled into a fake friendship. A good rule protects that line instead of erasing it.

Concern #3: What happens when kids leave the safer tool?

This isn't only about tone. Kids are honest about it: if the safe tool feels robotic, or makes them re-explain their coding project every time they open it, they'll use something else. Usually that something else is a general-purpose chatbot built for adults.

On paper, the law closes that door, because general-purpose chatbots must check ages for New York users too. In practice, age checks can be gotten around, rules vary state to state, and not every app will read the law strictly. We can't say this law will push kids elsewhere. But a motivated 14-year-old usually finds a way.

When that happens, a child loses what makes kid-built AI safer: content filtering, age-appropriate answers, safety alerts, and a parent who can see the conversation. That's why we built HeyOtto as a ChatGPT alternative for kids that's useful enough to stick with.

A safety rule should be judged by how a product behaves when a child uses it, and by what happens when the child decides to use something else.

A safety policy or law should consider not just how a product behaves when a child uses it, but also what happens when the child decides to use something else. - Natalie Gibson, HeyOtto Founder

What we'd like lawmakers to clarify

New York's attorney general has rulemaking authority under the law. We hope that process includes parents, educators, child-safety experts, privacy professionals, and the developers responsible for building these systems.

Here are five areas where clarity would help.

1. Distinguish safety monitoring from simulated companionship

Clarify how the restrictions on historical personal information apply to systems designed to detect risk and alert parents, rather than simulate a relationship.

A company should not receive an exemption simply by calling a feature a safety tool. Its actual function, data use, and behavior should matter.

2. Preserve responsible detection of patterns over time

Explain how child-facing systems can use limited, relevant historical context to identify potential risks while respecting privacy.

Safety monitoring should not become unrestricted surveillance. But neither should developers be left guessing about whether a narrowly defined safety function is allowed.

3. Make room for ordinary, age-appropriate encouragement

Clarify the difference between manipulative emotional flattery and specific praise of a child's work.

"Great job—you solved that problem" should not be confused with a chatbot claiming to love a child or encouraging them to depend on it.

4. Allow appropriate safety check-ins

A chatbot should not routinely probe a child about their feelings to create intimacy or prolong a conversation.

But when a child's messages raise a genuine concern, a carefully designed response such as "Are you okay? Is there a trusted adult you can talk to?" may help direct them toward real-world support.

The aim should be to connect children with people who can help, not make the AI the center of their emotional lives.

5. Measure whether children are actually safer

Compliance matters, but so do outcomes.

Are concerning patterns being identified? Do parents receive useful alerts? Are children being directed toward appropriate human support? Are the rules encouraging safer products—or making it harder for children to use them?

Those questions should be part of evaluating whether a policy is working.

What parents can ask about any AI tool

You don't need to wait for regulators. Wherever you live, these four questions will tell you a lot about any AI your child uses:

1. What does it remember, and why? Memory used to personalize and keep kids chatting is a red flag. Memory used to alert you to warning signs is worth asking about.
2. Will I know when something concerning happens? Can you review conversations? Does the product offer safety alerts? Can it recognize potential patterns across interactions, or does it only respond to individual messages?
3. Is my child actually using the safer option? If the "safe" app is gathering dust, find out which app isn't. Our side-by-side look at ChatGPT and HeyOtto shows what to compare.
4. Does the AI act like a friend? Be cautious if a chatbot says it misses your child, claims to love them, encourages secrecy, or suggests that it understands them better than the people in their life. Children should know that AI is software. It should help them learn, create, and explore—not compete with the people who care about them.

Where HeyOtto stands

We support the goal of New York's new law: protecting children from AI systems that exploit their trust, encourage dangerous behavior, or foster emotional dependency.

We're reviewing the legislation and preparing for its January 1, 2027 effective date. That includes ensuring that Otto is transparent about being AI, does not claim human feelings or friendships, and encourages children to turn to trusted adults when they need help.

You can follow our work on child-AI safety through our Trust Center and learn more about how parental visibility and safety alerts work on our safety page.

We also want policymakers to preserve room for two things that matter to children's safety: the ability to recognize warning signs over time, and the ability to build useful, encouraging tools that children actually want to use.

We don't believe the answer is an AI that remembers everything about a child. Nor do we believe the answer is an AI that must ignore relevant context when recognizing a potential risk.

The answer is to be deliberate about what information is used, why it is used, who can access it, and how it helps protect a child.

The goal isn't to make AI more attached to children. It's to make AI more accountable for their safety.

That's the distinction we hope policymakers will protect.

Because the ultimate measure of an AI safety law isn't how many features it restricts. It's whether children are actually safer.

This article reflects HeyOtto's perspective on New York's new AI law and child safety. It is not legal advice. The application of the statute to any particular product depends on the statutory text, implementing rules, and how the product operates.