fMRI Used to Read What The Brain is Seeing

Generalization to new letter classes. Each letter is predicted using models trained on BOLD data for the remaining letter classes. Extending the prior to include the new letter class is shown to substantially improve the reconstructions.

Functional MRI (fMRI) has been used in a variety of ways to understand how the brain functions by monitoring changes in blood flow in response to a stimulus. The brain is pretty complicated though, so any ability to actually detect and interpret specific acts that the brain performs can go a long way to learning a lot more about our favorite organ.

Researchers at Radboud University Nijmegen in The Netherlands have developed special software that analyses fMRI data and can actually read what letters a person is seeing by interpreting changing blood flow within the visual cortex. The software monitored 1200 2x2x2 mm voxels (3D version of pixels) that were carefully mapped to the visual two-dimensional map that the subject was seeing. After a bit of tuning, the team is now able to transcribe individual letters, but plans are in the works to move up to 15,000 voxels that should allow the system to “see” the face a person is looking at.

From Radboud University Nijmegen:

By combining all the information about the pixels from the voxels, it became possible to reconstruct the image viewed by the subject. The result was not a clear image, but a somewhat fuzzy speckle pattern. In this study, the researchers used hand-written letters.

‘After this we did something new’, says lead researcher Marcel van Gerven. ‘We gave the model prior knowledge: we taught it what letters look like. This improved the recognition of the letters enormously. The model compares the letters to determine which one corresponds most exactly with the speckle image, and then pushes the results of the image towards that letter. The result was the actual letter, a true reconstruction.’

‘Our approach is similar to how we believe the brain itself combines prior knowledge with sensory information. For example, you can recognise the lines and curves in this article as letters only after you have learned to read. And this is exactly what we are looking for: models that show what is happening in the brain in a realistic fashion. We hope to improve the models to such an extent that we can also apply them to the working memory or to subjective experiences such as dreams or visualisations. Reconstructions indicate whether the model you have created approaches reality.’

Radboud University Nijmegen: Computer can read letters directly from the brain…

Study in NeuroImage: Linear reconstruction of perceived images from human brain activity

(hat tip: Gizmodo)