As we saw in our discussion of orientation decoding in V1, even when we do know the underlying functional architecture, how a classifier exploits information in neural data is deeply opaque. To further illustrate the greed of linear classifiers, consider that in psychology some have noted that linear decision-making models can be surprisingly good even when feature weightings are assigned more or less arbitrarily Dawes .
To emphasize a similar point, when using MVPA there is not even a guarantee that classifiers are detecting multivariate signals. In a simulation study, Davis et al. Although a classifier linear or non-linear may, through training, come to discriminate successfully between activity patterns associated with different experimental conditions, the information the classifier uses as the basis for this discrimination is not constrained to be the information the brain actually exploits to make the distinction that is, they are informationally greedy.
Importantly, it is evidence about the latter and not the former that is critical for zeroing in on the contents of neural representations. Hence, decodability does not entail that the features being combined, or their method of combination, bears any connection to how the brain is decoding its own signals. Of course, one might use decodability, converging with other lines of evidence, to make inferences about what information might be represented in a brain ROI cf.
Kriegeskorte and Bandettini  , p. At best, MVPA-based decoding shows that information about experimental conditions is latent in neural patterns. It cannot show that this information is used, or is even usable, by the brain.
Decoding the dynamic representation of musical pitch from human brain activity
This is the deep reason why the dictum is false. In this section we consider and respond to some objections to our criticism. When criticizing inferences in cognitive neuroscience, it is common for the philosopher to be informed that no working scientist really makes the sort of inference. Yet it is the descriptive claim that really matters—for philosophical critique matters only insofar as it identifies areas of actual methodological friction.
Do scientists really believe something like the dictum? Our reconstruction of the theoretical basis of the dictum already suggests that they do. At the same time, enumeration is also illuminating. Here are just a few of many possible illustrative examples where the dictum is either overtly referenced or strongly implied: Kamitani and Tong  was one of the first studies showing that orientation information is decodable from voxels in early visual cortex, including V1.
Hung et al. In an early review of studies that included Kamitani and Tong  ; Hung et al. Woolgar et al.
Mental representation - Wikipedia
In a review of this method, King and Dehaene [ ], p. Another tempting reply to our argument goes as follows: Classifier performance is graded, so it makes sense to talk about different brain regions having more or less decodable information. For example, although early visual cortex contains some information about object category, decodability is typically much worse than it is in inferior temporal cortex, a region heavily implicated in the representation of object categories Kiani et al. So perhaps the dictum is true if we restrict ourselves to the best or most decodable regions.
The problem with this reply is that it faces the same objection elaborated in detail above. What makes a given region the best or most decodable might have little or nothing to do with the information that is available to and used by the brain. This is why decoding results can be and often are at odds with the answers derived from other methods. Seymour et al. Their caution appears to embody the same concern that decoding results may reflect arbitrary differences to which the classifier is sensitive, without guaranteeing that these results track real differences in neural representation.
Decoding—excellent or otherwise—is not a reliable guide to representation. Another problem with this suggestion is that it entails that poor decodability or even failure to decode provides evidence that the information is not represented in a region. But this is false. Non-significant decoding does not entail the absence of information. One might have simply chosen the wrong classifier or stimuli, or the particular code used by the brain might not be read out easily by a linear classifier.
Dubois et al. They compared single-unit recordings with fMRI decoding in the face patch system of the macaque brain—a system known to possess face-sensitive neurons. In agreement with the single-unit data, face viewpoint was readily decodable from these regions. However, in the anterior face patches, face identity could not be decoded, even though single unit data show that it is strongly represented in these areas.
These results indicate how poor decodability provides a thin basis upon which to mount negative claims about what a given region does not represent. In sum, one cannot appeal to any level of classifier performance—good or bad—to preserve the dictum. Though not always carried out, the ability to connect classifier performance to behaviour has been highlighted as one of the strengths of decoding methods Naselaris et al.
To be sure, a deep problem with the dictum is that decodability fails to show that information is formatted in a way that is used, or usable, by the brain Cox and Savoy  , while connecting decoding to behaviour helps make the case for functional utilization Tong and Pratte . If behavioural performance can be predicted from the structure present in brain activation patterns, this would provide more compelling evidence that decodable information is used or at the very least is usable by the brain, and hence neurally represented.
The simplest way to connect decoding and behaviour is to show that classifier and human performance are highly correlated. Minimally, if this obtains for some activation patterns more than others, this provides some relatively weak evidence that the patterns that correlate with behaviour represent information that is used in the guidance of behaviour. Williams et al. They analysed the spatial pattern of the fMRI response in specific task-relevant brain regions while subjects performed a visual shape-discrimination task. They hypothesized that if decodable shape category information is behaviourally relevant, then decodability should be higher on correct trials than on incorrect trials.
Critically, they showed that although both retinotopic cortex and lateral occipital cortex LOC in humans contain decodable category information, only the LOC shows a difference in pattern strength for correct as compared to incorrect trials. Specifically, category information was decodable in correct but not incorrect trials in the LOC. This was not true for retinotopic cortex. This pattern of results suggests that only the information in LOC might drive behaviour.
It is also possible to quantify the relationship between decodability and behaviour more precisely. While connection to behaviour supplies valuable evidence, we still think that it is not enough to warrant inferences to representational content. As we noted earlier, there are cases where decodability appears to show something about functional processing rather than the content of neural representations. Again, V1 provides a useful test case.
Since we know that V1 primarily encodes information about low-level visual features such as luminance or orientation and does not encode higher-level visual features such as shape or object category , any decoding of higher-level visual features is unlikely to reflect genuine representational content. This is true even if decoded information can be linked with behavioural performance. For example, Haynes and Rees [ ] found that V1 activity was predictive of whether or not subjects were perceiving a visual illusion, and Kok et al.
In these cases, the connection is that early processing modulates later processing that determines behaviour. Note that the problem is not one of spurious correlation. In an important sense, it is quite the opposite problem. There is plenty of information, even in V1, which a clever decoding algorithm can often pick up on. More generally, a brain region might carry information that is reliably correlated with the information that is actually used, but which is not itself used in behaviour. This is because the information in a region might need to be transformed into a more appropriate format before it is read out.
- Decoding the dynamic representation of musical pitch from human brain activity | Scientific Reports;
- Human white matter and knowledge representation.
- The Chinese Economy: Transitions and Growth;
- Cardiac imaging: a multimodality approach!
- On single cell abstract representation in the brain;
- Bibliographic Information.
As DiCarlo and Cox [ ], p. In summary, merely predicting behaviour using decodable information is not enough to revive the dictum. However, we are not pessimists about decoding. Rather, we think the right conclusion to draw is that decoding must be augmented in order to provide good evidence about neural representation. If linear classifiers are greedy, then they cannot function as a surrogate for the sort of linear read-out carried out by the brain. Instead, we need some additional assurance that a particular decoding result relies on information stemming from neural representations.
This need not be knock-down evidence, but decodability alone is not enough to do the job as the dictum suggests. In the previous section, we considered one form of augmentation—linking decoding results to behavioural outcomes—and argued that it was insufficient. The problem was that linkages to behaviour do not show that the information is actually formatted in a useable way. Framing it this way, however, already suggests a solution. The dictum relies on the idea that the biological plausibility of linear classifiers allows them to function as a kind of surrogate—the classifier-as-decoder takes the place of the brain-as-decoder in showing that information that is latent in neural activity is used, or usable cf.
De Wit et al. We have shown that it cannot play this role. But if the information latent in patterns of neural activity can be used to predict observer behaviour based on a psychological model, then we would have a stronger evidential basis for drawing conclusions about neural representation. For unlike classifier performance, observer behaviour is clearly dependent on how the brain decodes its own signals. In other words, this approach depends on offering a psychologically plausible model of how observers through down-stream processing exploit the information found in patterns of neural activity cf.
Ritchie and Carlson . And as it happens, such an approach is already on offer. There is a long tradition in psychology of modelling behavioural performance using psychological spaces Attneave  ; Shepard . Models within this tradition characterize representations for individual stimuli or experimental conditions as points in a space, and observer behaviour such as choice or reaction time RT is modelled based on the relationship between different representations in these spaces. Thus, familiar categorization models from cognitive psychology such as prototype models, exemplar models, and decision boundary models all predict observer behaviour based on different distance metrics applied to a reconstructed psychological space Ashby and Maddox .
A virtue of some MVPA methods, like representational similarity analysis RSA , is that they help to focus attention on structure in activation spaces, rather than simply the discriminability between activity patterns in these spaces, as is the case when using linear classifiers Kriegeskorte and Kievit  ; Haxby et al. In RSA, the pair-wise dis similarity for patterns of activity for different conditions is computed, and this can be used to reconstruct an activation space from multivariate neural data.
One hypothesis that many have considered is that if an activation space implements a psychological space, then one can apply psychological models or hypotheses to the activation space directly in order to predict behaviour Edelman et al. Note that this approach is importantly different from the dictum, as it does not rely on using linear classifiers as a surrogate indeed, it might not involve the use of classifiers or decoding at all.
Furthermore, the approach achieves both biological and psychological plausibility through a linkage between the structure of the reconstructed activation space and the structure of behaviour Ritchie and Carlson . And since it makes use of MVPA in conjunction with established techniques for modelling behaviour, it also takes advantage of some of the strengths of MVPA we have already mentioned. Here we offer two examples of research that adopts this sort of approach. One illustration of this approach is provided by the results of Sha et al. The similarity space constructed from these judgements was then directly related to the similarity structure of activation spaces from throughout the brain, measured using fMRI.
They found that activation spaces that correlated with the behavioural similarity space were best accounted for by a single dimension, which seemed to reflect an animacy continuum rather than a categorical difference between the neural patterns for animate and inanimate objects Kiani et al. Second, some work has focused on the psychological plausibility of activation spaces by using them to predict the latency of behaviour. In their experiments they were explicitly motivated by the idea that linear classifiers are structurally identical to the model of an observer under signal detection theory Green and Swets .
A natural extension of signal detection theory is that distance from an evidential boundary negatively correlates with RT Ashby and Maddox . As predicted, they found that RT negatively correlated with distance from the decision boundaries, suggesting a level of psychological plausibility to even simple linear classifiers. Crucially, in these sorts of studies it is implausible to suppose that the information is present but not correctly formatted, because the decoded format of the information in activation space is precisely what is being used to predict behaviour in a psychologically plausible manner.
We do not mean to suggest that the results we have summarized suffice for drawing conclusions about neural representation, but we do believe that they help point the way forward. Significant decoding does not warrant an inference that the decoded information is represented. However, we do believe that if behaviour can be connected to the structure of activation space in a psychologically plausible manner, then this may warrant the sort of inference researchers have had in mind. And we should stress that we do not think the above shows that decoding is immaterial.
Indeed, as we have suggested, MVPA is crucial for connecting activation spaces to behaviour. Rather, as we have argued, appealing to the dictum obscures not only the true import of decoding as a tool in cognitive neuroscience, but also what sort of evidence is required for making claims about neural representation. A continuous semantic space describes the representation of thousands of object and action categories across the human brain.
Caramazza A, Shelton JR. Domain-specific knowledge systems in the brain the animate-inanimate distinction. J Cogn Neurosci. Martin A. The representation of object concepts in the brain. Annu Rev Psychol. Mahon BZ, Caramazza A. What drives the organization of object knowledge in the brain? Where is the semantic system? A critical review and meta-analysis of functional neuroimaging studies.
Cereb Cortex. Where do you know what you know? The representation of semantic knowledge in the human brain. Nat Rev Neurosci. Nature Publishing Group; ;8: Martin A, Chao LL. Semantic memory and the brain: structure and processes. Curr Opin Neurobiol. Caramazza A, Mahon BZ. Cogn Neuropsychol. The neurobiology of semantic memory. The organization of conceptual knowledge: the evidence from category-specific semantic deficits. Attention during natural vision warps semantic representation across the human brain.
Miller GA. Commun ACM. View Article Google Scholar The evolution of face processing networks. Parallel distributed processing [Internet]. Huk AC.
Thinking in graphs
Multiplexing in the primate motion pathway. Vision Res. From regions to connections and networks: new bridges between brain and behavior. Bassett DS, Sporns O. Network neuroscience. Semantic representation in the white matter pathway. PLoS Biol. Representational similarity analysis—connecting the branches of systems neuroscience. Front Syst Neurosci. Diedrichsen J, Kriegeskorte N. Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis.
PLoS Comput Biol. Kriegeskorte N, Mur M. Front Psychol. Hayashi S, Pestilli F. Catani M, Ffytche DH. The rises and falls of disconnection syndromes.
alexacmobil.com/components/hovaqow/zuhoc-software-spia-android.php Geschwind N. Disconnexion Syndromes in Animals and Man. Selected Papers on Language and the Brain. Springer, Dordrecht; The language network. Reworking the language network. Elsevier; [Cited April 13 ]. Caiafa CF, Pestilli F. Multidimensional encoding of brain connectomes. Sci Rep. Evaluation and statistical inference for human connectomes. Nat Methods. Tract profiles of white matter properties: automating fiber-tract quantification. PLoS One. The visual white matter: The application of diffusion MRI and fiber tractography to vision scienceRokem et al. J Vis. The Association for Research in Vision and Ophthalmology; ; 4—4.
Cerebral white matter—historical evolution of facts and notions concerning the organization of the fiber pathways of the brain. J Hist Neurosci. The vertical occipital fasciculus: a century of controversy resolved by in vivo measurements. The posterior arcuate fasciculus and the vertical occipital fasciculus. Charcot JM. New Sydenham Society; [Cited April 13 ]. Mental health research priorities for Europe. Lancet Psychiatry. DiLuca M, Olesen J. The cost of brain diseases: a burden or a challenge? Cost of disorders of the brain in Europe Eur Neuropsychopharmacol.
Though seemingly contradictory, the two findings may not be mutually exclusive. Because the dominant has a distinctive harmonic function relative to the tonic, it is likely that trained listeners possess a schema with which to separate the two pitches. In contrast, the augmented 4 th and minor 2 nd have no such functional harmonic relationship.
In the conical model however, distance reflects the degree to which two tones are perceived to be musically associated. In such a framework, the tonic and dominant may indeed be related. Both classes have a high probability of joint occurrence within tonal passages or as constituent pitches of chords Conversely, out-of-key classes may be considered musically unrelated; the occurrence of an out-of-key tone is likely to be followed by a resolution to the nearest in-key rather than another out-of-key pitch-class.
By distinguishing between these two concepts — the distinctiveness of two pitches as opposed to how well they musically fit — the current study helps to clarify the complex nature of representational distance. It should also be noted that the current investigation presented single tones following a tonal context, whereas the conical model arose from similarity ratings of tone pairs presented within a tonal context. Nonetheless, the conical model provides a general account of the psychological similarity of pitch-classes. Therefore, it is valuable to assess the extent to which this model can be generalized beyond the specific behavioral methodology from which it arises.
As such, we tested the hypothesis that perceptually dissimilar tones as described by the conical model also have dissimilar patterns of brain activation when presented in isolation. For example, it may be the case that when all pitch-classes are considered, a mixed model of sensory and schematic features may be needed to account for neural distinctions. We assessed the ability of classifiers to discriminate between MEG responses evoked by tones differing in their tonal functions. Using classification accuracy as a measure of representational distance, we characterized a representation for a set of musical pitches and showed that their collective representational structure correlates with the respective differences in their perceived tonal stability.
Our results provide a crucial link between musical pitch perception and the underlying neural activity from which it materializes. The current results strengthen this notion by showing consistency in the relations between tones across neural and perceptual domains.
The sample size was not pre-determined, but rather testing was terminated once trends in the decoding analyses displayed sufficient statistical power see statistical analysis below. All subjects reported having no known hearing loss or brain abnormalities, and did not possess absolute pitch. The study was approved beforehand by the Human Research Ethics Committee at Macquarie University REF and all methods were carried out in accordance with the stated guidelines. Informed consent was obtained prior to testing, after all experimental details and potential risks were explained.
Each trial consisted of a tonal context followed by a probe-tone. Contexts were either in the key of C major or F major, and consisted of four major chords written in four-part harmony outlining an I-IV-V-I harmonic progression. Subsequent probe-tones were either C4 Within each key, two versions of the tonal context were presented: one in which chords contained tones that were also probe-tones for example, in a C major context the C4 and G4 were both physically present in the preceding chords , and an alternate version in which these constituent tones were transposed an octave above or below their original position in the chords i.
The inclusion of this alternate tonal context enabled us to assess the effect, if any, of the acoustic spectral overlap between context and probe. All reported results are therefore based on an analysis of trials grouped across the two versions of the tonal context. Stimuli were piano tones recorded at This temporal separation was intended to prevent the sensory processing of the context from influencing the evoked response to probe-tones see Fig. Prior to testing, all probe-tones were passed through a time varying loudness model 32 to normalize for differences in perceived loudness.
For each tone, the maximum short-term-loudness STL max was computed and normalized to the mean value of all four tones.
Differences in STL max between the four probe-tones did not exceed 3 phones. The session was sub-divided into 8 testing blocks separated by one-minute breaks, during which subjects watched and listened to a movie. Each block consisted of 80 trials in a single tonal context C major or F major , with adjacent blocks alternating between the two keys. The two versions of each tonal context with and without shared probe-tone pitches were presented in randomized fashion with equal probability within a block. ERP studies indicate that increases in the probability of syntactically irregular trials results in decreased effect sizes 7.
We therefore opted for an in-key to out-of-key presentation ratio of , resulting in a total of in-key and out-of-key observations within each tonal context. This was done to ensure participants were attending to the stimuli Participants used their left and right thumbs to register the two different responses. Once the response was registered, inter-trial-intervals were randomly jittered between 0. Before testing, subjects completed a training session consisting of 20 trials with an identical behavioral task to that of the MEG recording session.
No trial-by-trial feedback was provided during the MEG recording; however, subjects were informed of their accuracy after each block. MEG data was bandpass filtered online from 0. Participants were in a supine position in the scanner and were instructed to direct their gaze at a fixation cross. Sound stimuli were delivered via Etymonic ER insert headphones at a sampling frequency of Data was epoched from 0. On average, PCA reduced the dimensionality of the dataset from sensor channels to 26 principle components. Generalization of the classifier was evaluate using k-fold cross validation with a training to test ratio.
In this procedure, the MEG data for all trials corresponding to the two classes being decoded were randomly assigned into 10 bins of equal size, with matched numbers of observations across the two classes in each bin. Nine of the bins were pooled to train the classifier, and the trials in the remaining bin were used to test the classifier. This procedure was repeated 10 times such that each trial was included in the test bin exactly once.
Decoding was performed with a sliding time-window to assess the time-varying ability of classifiers to discriminate between neural activity corresponding to two given pitch-classes. This meant that each classification run was based on data from the 5 most recent points in the timeseries. MDS aims to spatially represent the RDM whilst preserving the original distances as much as possible. The loss function or stress of the solution indicates how faithfully MDS preserves the distances.
Typically, the stress is minimized with higher-dimensional solutions. We constructed five model RDMs based on perceptual and acoustic properties that may account for the neural dissimilarities observed. Stimulus spectrograms were computed by passing the raw audio through a model of the auditory periphery The model consisted of three main stages: 1 a cochlear filter bank comprised of asymmetric filters uniformly distributed along a logarithmic frequency axis 2 , a hair cell stage consisting of a low-pass filter and a nonlinear compression function, and 3 a lateral inhibitory network approximated by a first-order derivative along the tonotopic axis followed by a half-wave rectifier.
We constructed an RDM of Spectral Distance between probe-tones by calculating the Euclidean distance between the respective spectrograms across all frequency bands. For each context, a Spectral Overlap RDM was computed by calculating the differences in the Euclidean distance between the context spectrogram and each of the probe-tone spectrograms. Finally, the Proximity RDM was based on the semitone difference in pitch-height between probe-tones. Decoding performance is reported in terms of balanced accuracy the mean of percent-correct in class A and percent-correct in class B.
Time-series and time-averaged decoding performance was tested for significance using a two-sided Wilcoxon signed rank test. Significance of model correlations at each time-point was assessed using randomization testing. Briefly, the class labels on RDMs being compared were randomly re-assigned before computing correlation, and this process was repeated 10, times to define the null distribution at each time-point.
Krumhansl, C. Quantification of the hierarchy of tonal functions within a diatonic context. Journal of experimental psychology: Human Perception and Performance 5 4 , Bharucha, J. Tonality and expectation in Musical Perceptions eds Aiello, R.
The psychological representation of musical pitch in a tonal context. Cognitive psychology 11 3 , — Brattico, E. Musical scale properties are automatically processed in the human auditory cortex. Brain research 1 , — Krohn, K. Neural representations of the hierarchical scale pitch structure.
Music Perception 24 3 , — Pearce, M. Unsupervised statistical learning underpins computational, behavioural, and neural manifestations of musical expectation. NeuroImage 50 1 , — Koelsch, S. Neural substrates of processing syntax and semantics in music. Current opinion in neurobiology 15 2 , — Toward the neural basis of processing structure in music.
Annals of the New York Academy of Sciences 1 , 15—28 Psychophysiology 39 5 , — Differences in electric brain responses to melodies and chords. Journal of Cognitive Neuroscience 22 10 , — Psychophysiology 46 1 , — Maess, B. Nature neuroscience 4 5 , — Foo, F.
Frontiers in human neuroscience 10 Bianco, R. Neural networks for harmonic structure in music perception and action. NeuroImage , —