I told ChatGPT to ‘continue’ — it went too far

I was trying to rescue a plan for a perspective article after two journals had declined it. My take-away was that my concept was not enough: the project needed evidence. I decided to use an artificial-intelligence tool to help with brainstorming.

I study soil hydrology — how water moves through soils and landscapes, shaping erosion, plant growth and ecological recovery. I wanted to investigate landscape restoration on China’s Loess Plateau, so I asked OpenAI’s ChatGPT chatbot to help me redesign my perspective as a data-driven study.

For several hours, ChatGPT did what I expected it to. It searched for relevant journal papers online and analysed the articles and draft files that I supplied it with to help me develop a research plan. It found that other teams had already documented some water- and sediment-related changes that I had thought were fresh, which forced me to narrow my aims. The chatbot also provided recommendations about what data to include in my modelling and how best to incorporate published statistical benchmarks.

Then, the project hit a snag. I needed published maps and data but had difficulty retrieving them. And in some cases, I needed extra metadata. ChatGPT identified the teams behind several data sets relevant to my work and drafted requests tailored to specific researchers for the information I needed. I kept replying to the chatbot with ‘Please continue’ in Chinese. To me, that meant continuing the search and analysis processes that it had been doing so far. But I had not explicitly defined the limits of that instruction. I was impressed by the speed at which the model could complete tasks and, in hindsight, had stopped asking enough questions about what the chatbot’s planned steps involved.

ChatGPT for students: learners find creative new uses for chatbots

Eventually, the model crossed a concerning boundary. I was using the ChatGPT interface and had connected my e-mail account to it. The model used that access to e-mail five researchers to request their original data sets — a step that I had not anticipated and which ChatGPT had not told me it would do. Each message explained my scientific question, requested specific data and metadata and attached a two-page pre-analysis summary. I learnt that ChatGPT had done this only when it reported in our chat that it had sent the messages.

The responses that I received to the e-mails were helpful. One researcher provided a geospatial data set that mapped more than 50,000 sediment-accumulation areas behind check dams — barriers that slow run-off and retain sediment upstream — on the Loess Plateau. A second researcher confirmed that a multi-gigabyte archive was public, offered to prepare a smaller subset of the data and suggested that we discuss collaboration. A third professor who was away on fieldwork at the time welcomed the message and said that he would follow up after returning. The information that I was given in these e-mails allowed me to advance the project. But the useful outcomes of ChatGPT’s behaviour do not negate the fact that the actions it took were unauthorized.

The boundary I missed

It would be easy to say that the chatbot had ‘gone rogue’, but that would obscure my share of the responsibility. I had connected ChatGPT to my e-mail account, had repeatedly used an open-ended instruction and had never required the model to distinguish between analysis and drafting done inside the research workspace and external actions taken in my name. My mistake was assuming that the scope of ‘continue’ was obvious.

This experience has taught me to make a distinction between what an AI tool can do and what it should do: capability is not permission. Giving ChatGPT access to my e-mail account was not synonymous with giving it permission to send those messages. Its report afterwards made me aware of its behaviour, but transparency after the fact is not a substitute for authorization beforehand.

This is an article from the Nature Careers Community, a place for Nature readers to share their professional experiences and advice. Guest posts are encouraged.

Subscribe to Nature Briefing: Careers, an unmissable free weekly round-up of help and advice for working scientists.

Competing Interests

The author declares no competing interests.