Sounding like something out of a sci-fi movie, researchers from the University of California, Berkeley used fMRI and special computer models to reconstruct moving images from brain scans of people watching movie clips. In the words of Jack Gallant, a UC Berkeley neuroscientist and coauthor of the study, they’ve managed to reconstruct “the movies of our minds”.
The experiment involved three subjects watching two sets of Hollywood movie trailers. As the subjects watched the trailers, fMRI was used to measure the amount of blood flow through each subject’s visual cortex. The fMRI scans were fed into a computer that divided the brain images and processed them into three-dimensional pixels, or “voxels”. The processing essentially helps a computer program learn to associate shape and motion information from the movie clips with brain activity.
A second set of clips, which consisted of 18 million seconds of random YouTube videos, was used to test the reconstruction algorithm that the computer program developed. The program used the algorithm to predict the brain activity that each clip would evoke. It then chose the best 100 clips that resulted in brain activity most similar to what the subject was seeing and combined them into a final movie.
While low-resolution and blurry, the results, shown in the video below, clearly match the actual clips that the subjects watched:
Such a discovery is groundbreaking as it could someday give researchers a better understanding of what goes on in the minds of people who cannot communicate verbally, or who suffer from neurological diseases. It could also someday lead to brain-machine interfaces that would allow users to control a computer with just their thoughts.
Perhaps someday we’ll also be able to visualize our dreams and memories.
More from UC Berkeley: Scientists use brain imaging to reveal the movies in our mind…
Research lab website: The Gallant Lab at UC Berkeley
Journal Article from Current Biology (PDF): Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies