A Japanese scientist has reportedly developed a method for combining brain scans with artificial intelligence to generate descriptive sentences.
Published earlier this month in the Science Advances journal, the method, described as “mind-captioning” by creator Tomoyasu Horikawa, uses AI to generate text that describes visual information stored in the brain pertaining to objects, places, actions, events and the correlation between all four.
Although the AI tools generate sentences in English, the method can reportedly cross the language barrier, as it can create descriptions of visual content without engaging language-related areas of the brain.
To conduct the experiment, Horikawa examined the brain activity of four men and two women aged 22 to 37. The participants were made to watch 2,180 short-form videos without sound. Each video was associated with a different action, scene and object.
The analysis of the brain activity was then promptly taken to AI models known as “decoders.” The decoders used numerical sequences- generated by large language models that took the video clip captions to create a series of numbers- to match it to the brain activity. The decoders were also used to improve the AI system as they interpreted the brain activity of videos not initially used in the experiment.
Other algorithms were then used to create the descriptive sequences once the brain activity was deciphered.
“We successfully generated descriptive text representing visual content experienced during perception and mental imagery by aligning semantic features of text with those linearly decoded from human brain activity,” said the researchers in their discussion. “Our success is attributed to two key factors: the advancement of deep LMs that provide contextualized semantic representations similar to those in the human brain and our innovative text optimization method using the MLM model for word-level optimization while efficiently constraining the search space.”
Although the latest method provides opportunities to give non-verbal people a form of communication, some have raised concerns about how ethical it is.
Critics have expressed concern about what the new method means regarding the privacy of individual thoughts, opening up the door for discrimination and manipulation of both emotions and intimate thoughts. According to creator Horikawa, however, unusual scenes cannot be translated; therefore, the method cannot completely read a person’s private thoughts. Horikawa also confirmed that the tool is not yet intended for everyday use, but rather more so for neuroscientific research.
The creators also argued that consent would have to be given by all participants involved for them to take part in an experiment such as this.
AI in general has been a source of concern for privacy. As the tech continues to get implemented in daily society, the risk of exploitation, data breaches, as they process large batches of data and ongoing surveillance have all been highlighted, drawing out calls for privacy laws regulating AI.
