Event-Related Potentials (ERPs) are a class of electroencephalographic (EEG) recordings, usually measured on the scalp, when a subject is presented with stimuli (or events). Strictly defined experimental protocols guide the administration of specific stimuli, according to psychological paradigms or tests. ERP study is focused on parts of the electrical potential waveform containing significant local maxima and minima, called components or peaks of the ERP. The high temporal resolution of ERPs allows for non-invasive observation of electrical activity changes in the brain, as reflected in scalp recordings, during the processing of information related to the presentation of the stimuli, therefore permitting the concurrent study of psychological measures of cognitive operations and of electrophysiological measures of brain functioning [1, 2]. When subjects commit incorrect actions, a negative deflection of the ERPs is produced, peaking at around 80 msec after the initiation of the incorrect response, called error-related negativity (ERN) [3, 4]. A positivity following the ERN, the so-called error positivity (Pe) has also been described [5], showing a maximum between 200 and 500 msec after the initiation of the incorrect response. Research has shown that ERN is elicited when there is a mismatch between representations of the actual response and the correct response [6, 7]. Pe have been linked to error awareness [8, 9]. It therefore seems likely that ERN and Pe are related to mechanisms reflecting error detection, a key component of performance monitoring [10]. Recently, the focus of ERN-research has been extended to include also the mechanisms related to the observation of errors committed by others, in an effort to elucidate whether the mechanisms responsible for learning ‘by doing’ are similar to mechanisms of observational learning [11]. In that work, an ERN was also found in a condition where subjects observed the incorrect actions of another person, albeit with a lower amplitude than the ERN for self-generated errors and a later occurrence of the peak.

Classification algorithms to discriminate between ERPs have been developed for various technical [12-15] and clinical applications [6]. Scalar autoregressive coefficients, extracted from biosignals and treated as feature vectors in classification methods, have been widely used for designing classifiers in biomedical systems. Tsoi et al. in [16] used one channel of EEG to estimate a feature vector consisting of the autoregressive (AR) model coefficients. By inputting the features to a multilayer neural network, differentiation between normal subjects and subjects diagnosed as suffering from severe obsessive-compulsive disorder (OCD) and from severe schizophrenia was achieved. Roberts and Tarassenko in [17] investigated the different human sleep states using AR model coefficients extracted from a single-channel EEGs. Vasios et al. used AR model features of ERPs as input to a neural network for the classification of patients suffering from first-episode schizophrenia against control subjects [18]. Almost all model-orders between 3 and 10, according to the Schwartz order-selection criterion, resulted in classification results up to 100% for electrode (C4-T6)/2.

When information can be extracted from multiple, simultaneously recorded waveforms, the AR model can be replaced by the Multivariate Autoregressive (MVAR) model. In the MVAR model, features are extracted from a number of input signals by using the multivariate analogue of the AR model. Mental task discrimination has been investigated using scalar and MVAR models of the EEG [19]. Order 3 was found to be the optimum using the Akaike information criterion. The features were inputted to a feed-forward neural network, resulting in a classification accuracy of 91.4%. In another multi-channel study, an MVAR model was computed to investigate the synchronization of brain structures, the degree of coupling between channels, the estimation of phase delays and eventually the direction of brain activity spreading [20]. An order of 6 was found to be adequate to parameterize four-channel data. Application of the MVAR model to the current study requires the construction of an associated feature vector and the definition of several parameters such as the number and kind of electrodes and the order of the model used. Exhaustive search for the selection of the values of these parameters is not feasible, especially in multi-channel studies recording tens of electrodes. Therefore, methods to reduce the computational load should be used. An example was proposed in [21], where the MVAR model was combined with a global optimization method, the Simulated Annealing (SA) technique to detect optimum combinations (number and kind) of electrodes and model order. The Fuzzy C-Means (FCM) algorithm was used as well as classification with a Multi-layer Perceptron (MLP) Artificial Neural Network (ANN).

The primary aim of the present study was the development and implementation of a classification system for discriminating correct and incorrect responses, as well as observations of correct and incorrect responses, based on scalp-recorded ERPs of actors and observers, using features extracted through the MVAR model in combination with the SA technique. The present work is based on subjects’ averages and is a first step towards future single-trial classification. We aimed at investigating the discrimination ability of signals including only the ERN, both for the execution and the observation condition, and signals including both the ERN and the Pe. The study also aimed at investigating and comparing the performance of two classifiers, one based on a MLP ANN classifier and the other on the FCM method.

The grand average curves for representative electrodes of SR-1 for the actors’ ERPs correct and incorrect responses are presented in Fig. (3a), and for the observers’ ERPs in Fig. (3b). Classification results obtained with the MVAR/SA method, for both actors and observers (correct or incorrect actions), for both classification systems (the MLP-ANN classifier and the FCM method) are presented in Table 1.

Fig. (3). The grand average curves for electrodes 1,8,9,11,13-15,17,19: (a) for actors’ ERPs correct (dashed lines) and incorrect (solid lines) responses, and (b) for ERPs of observers’ observations of correct (dashed lines) and incorrect (solid lines) responses. The vertical axis has negative values upwards. The number above each pair of curves indicates the electrode.

Concerning the classifiers built for discriminating correct and incorrect actions based on actors’ ERPs, as can be seen in Table 1, the best overall classification performance for sub-region SR-1 was 86% for every time interval investigated, using an order of p=4. Results were similar for both the FCM classifier and the ANN classifier using cross-validation. For sub-region SR-2, classification performance again reached 86%, but only for the -6 to 500 msec time window, again for an order of p=4. Nevertheless, the classification performance reached levels equal or higher than 80% for all time intervals used in SR-2, and again the two classification systems (the ANN with cross-validation and the FCM method) proved to provide similar overall classification rate, when results are inspected overall.

Concerning the classifier built for discriminating the observation of correct and incorrect actions based on observers’ ERPs, again classification rates higher or equal to 80% were reached, for both sub-regions, SR-1 and SR-2, but only for the intervals from -6 to 500 msec and -6 to 700 msec. In both cases, the optimal order was p=5. Classification performance based on the window selected to fit the observation ERN fell just below 80%, reaching a classification performance of 78% for both sub-regions, SR-1 and SR-2, respectively. The best classification performance (87%) was reached for sub-region SR-2 using the FCM method for the time interval -6 to 500 msec. The truth tables for the best classification performance for both actors and observers are presented in Table 2.

By inspecting the truth Table 2a for the case of the best overall classification performance for actors’ actions, using sub-region SR-1, in the time window -6 to 146 msec, which was achieved using the MLP-ANN classifier with cross validation, we must note the excellent sensitivity (100%) and excellent negative predictive value (NPV) (100%), while the system generates specificity and positive predictive (PPV) values that are lower than 80%. The inspection of the truth Table 2b for the best overall classification performance for actors’ actions in sub-region SR-2, which occurred using the MLP-ANN classifier with cross validation, for the time window -6 to 500 msec, indicates a sensitivity and NPV greater than 90%, while the specificity and PPV remains at levels higher than 80%. Truth Table 2c for the case of best overall classification performance for observations of actions by the observers, which occurred for sub-region SR-1, in time window -6 to 700 msec, using the MLP-ANN classifier with cross validation, indicates a sensitivity and NPV greater than 90%. Finally, the truth Table 2d for the case of best overall classification performance for observations of actions by the observers, which occurred for sub-region SR-2, in time window -6 to 500 msec, using the FCM method, is very similar to the truth table of Table 2b and indicates again a sensitivity and NPV greater than 90%, while the specificity and PPV remains at levels higher than 80%.

The combinations of electrodes providing the best classification results, whose truth tables have been presented above, are shown in Fig. (4). The squares present the electrodes whose AR parameters were used in the MVAR/SA model providing the best overall classification rate of actors’ actions and of observers’ observations for specific time intervals and for the sub-regions SR-1 and SR-2, which are included within the thick dashed and dotted lines, respectively. For the actors, in region SR-1, in the case providing the best classification rate, i.e. the classifier based on the -6 to 146 msec interval, electrodes used included anterio-frontal, central and centro-parietal and a left parietal electrode. For the SR-2 region, in the case providing the best classification rate, i.e. the classifier based on the -6 to 500 msec interval, electrodes used included only two right central and centro-posterior electrodes. In the observation condition, for SR-1, in the case providing the best classification rate, i.e. the classifier based on the -6 to 700 msec interval, electrodes used included anterior-frontal ones bilaterally, as well as two right central and fronto-central electrodes and the same left parietal electrode as in the actors’ SR-1 case above. For the observation condition and the focused search space (SR-2), the best classification rate was provided for a classifier using features from the ERP at the time interval at -6 to 500 msec and using fronto-central, centro-parietal and a midline parietal electrode. It should be noted, however, that the above approach provides only indirect and two-dimensional indications about relations between brain regions and ERP classification accuracy. More direct inferences about brain regions related to the differential response of both actors and observers, to correct and incorrect actions, can only be gained by using inversion techniques for the three-dimensional localization of brain activity, and applying the methodology of the present study to the time series of the intracranial quantities that these techniques compute [21, 37, 38].

Fig. (4). Visual representation of the electrode combinations (indicated by squares) corresponding to the best classification rate achieved for actors’ actions and observers’ observations and for the two sub-regions SR-1 and SR-2 (regions within the thick dashed and dotted lines, respectively). Electrode combinations for the best classification rate achieved for actors’ actions: (a) in the time window -6 to 146 msec for SR-1, and (b) in the time window -6 to 500 msec for SR-2, using both the ANN classifier with cross validation. Electrode combinations for the best classification rate achieved for observers’ observations: (c) in the time window -6 to 700 msec for SR-1 using the ANN classifier with cross-validation, and (d) in the time window -6 to 500 msec for SR-2 using the FCM method.

The robustness of the MVAR/SA method, in terms of classification rate, was tested against two parameters: the model order and the number and kind of electrodes that were combined. Concerning model order, criteria for order selection, such as the Schwarz Bayesian Criterion and the Final Prediction Error Criterion [24], were examined, yielding that the best representation accuracy was achieved with model order varying from 3 to 8, for all electrode combinations. Additionally, previous research [18, 19] and experimental trials have pointed out that the optimum classification rate is achieved using similar values of model order. To be complete, however, model orders ranging from 3 to 13 were examined in this study. In Fig. (5a), the robustness of the MVAR/SA model is presented in terms of the classification rate using the MLP-ANN classifier for different model orders p and for both correct and incorrect actors’ actions in the time window -6 to 146 msec using electrodes [1 5 7 11 20 21 30] for the SR-1 sub-region. From the results, it is evident that the best classification rate was achieved for a model order of p=4 for this specific case as seen in Table 1. Similarly, in Fig. (5b), a model of order p=5 achieved the best classification rate for both correct and incorrect observed actions in the time window -6 to 700 msec using electrodes [8 11 19 23 24 30] for the same sub-region (SR-1).

Fig. (5). Performance of the classification system for different model orders used by the MVAR/SA method and the MLP-ANN classifier for the sub-region SR-1. (a) using actors’ actions in the time window -6 to 146 msec and (b) using observers’ Observation of actions in the time window -6 to 700 msec.

Apart from using a sub-set of the available electrodes for creating the search space, during the implementation process, the proposed system allowed a maximum number of 8 electrodes to be combined for the calculation of the MVAR coefficients. This decision was also taken in order to reduce the computational complexity of the search process. After several trials, it was found that optimal classification results corresponded to combinations consisting of less than 8 electrodes. As a result, further expansion of the process, by combining more than 8 electrodes, was deemed not necessary. Furthermore, robustness testing of the MVAR/SA method, with respect to the number and kind of electrodes that were combined, indicated that deviation from the combinations selected by the MVAR/SA method leads to deterioration of classification performance. A characteristic result concerning actors’ actions had as follows: According to Table 1, the optimal combination of electrodes provided by the proposed methodology was electrode combination [1 5 7 11 20 21 30], for actors’ actions for a model order of 4 resulting in a classification rate of 86% for the sub-region SR-1. If one of the electrodes, i.e. electrode 21, is omitted the performance is reduced to 83%. If one of the electrodes, i.e. electrode 21, is replaced by another electrode, i.e. electrode 34, the performance is reduced to 80%. Similarly, if a new electrode, i.e. electrode 34, is added to the initial combination of electrodes, a classification rate of 73% is obtained.

This paper reports an ERP classification method capable of discriminating between an actor’s brain potentials that accompany correct and incorrect responses as well as the brain potentials that occur in an observer looking at those responses. The requirement to use information existing in concurrently recorded ERP waveforms, leads to the creation of an unpractical large search space for selecting the MVAR/SA model providing the best classification rate. The combination of the MVAR model with the SA optimization technique, as proposed in the present work, provides a principled way to reduce the computational complexity of the search process. Nevertheless, even using the SA technique, we followed a gradual approach with respect to the degrees of freedom allowed in the search process, so that a compromise between satisfactory classification rates and computational complexity was reached.

In this framework, the MVAR/SA model in conjunction with two classifiers were implemented towards the classification of correct and incorrect responses, as well as observation of correct and incorrect responses, based on scalp-recorded ERPs of actors and observers. For the classification of actors’ responses, the best overall classification performance reached 86%. The classifier used for discriminating responses viewed by observers achieved an overall classification performance of 87%. In particular, the sensitivity and consequently the NPV reached 100% for the case of actors’ actions in the time interval -6 to 146 msec using the MLP-ANN classifier with cross-validation, indicating that the proposed system was able to completely discriminate all the incorrect actions of actors without any loss. From the obtained results, it is noticeable that in general, both the sensitivity and the NPV remain in high rates (up to 90%) for both actors and observers whereas the specificity and the PPV remains constant in lower rates (up to 80%).

An important aspect of our study was the selection of specific time windows from ERPs of actors and observers. Specifically, for the actors’ data, the time interval -6 to 146 msec was selected to include the ERN as reported in the original study of van Schie et al. [11]. A time interval from 90 to 318 msec was selected to fit the observers’ ERN in conformance to the original findings ([11]). An additional interval from -6 to 500 msec was included to investigate the contribution of ERP components occurring later than the ERN, i.e. the Pe component. A final interval from -6 to 700msec was included to investigate the robustness of the classification with additional noise introduced with longer data windows. Results indicate that the classification of responses based exclusively on the ERN is possible. Furthermore, results indicate that the inclusion of the time window overlapping with the ERN and the Pe provides a small improvement, as e.g. indicated in the case of SR-2 for the execution actors’ condition, and in a more pronounced way in both sub-regions in the observation condition. The use of the longest data window, which was hypothesized to introduce signals unrelated to the cognitive processes under investigation, provided mixed results. In some cases a slight deterioration of performance was apparent, such as in SR-2 for both conditions, while in the case of SR-1 in the observation condition an improvement was noted. This suggests that classification performance maintains its robustness when more prolonged data intervals are analyzed.

The search space for electrodes used in the MVAR/SA modeling procedure was reduced to two sub-regions of the total 47 electrodes available: The sub-region SR-1 corresponded to 34 electrodes, excluding the electrodes placed on the external circumference of the 47 electrode montage and sub-region SR-2 focused on electrodes located around the vertex. The aim of this search space reduction was primarily to reduce the computational complexity of the feature extraction and classification problem, without affecting the classification performance. The reduction was based on the fact that scalp ERN difference-wave distributions ([11]), for both execution and observation conditions, are subdued at peripheral electrode positions. Results indicate that there were no clearly discernible differences between the classifiers built, on the basis of electrode features of the two sub-regions. Furthermore, little consistency was found between the location of electrodes for SR-1 and SR-2 solutions of actor and observer data, suggesting that both the selection of sub-regions and the nature of the error signal (actor/observer generated) may largely influence the optimal choice of electrodes that may be used for classification.

Additionally, an investigation of the robustness of the MVAR-SA method was performed, in terms of classification rate, for the selection of the model order and the number and kind of electrodes that were combined. The analysis indicated that model orders of p=4 and p=5 provided the best classification results for the execution and observation condition respectively, and deviation from the electrode combinations selected by the MVAR/SA method resulted in deterioration of classification performance.

The performance of the proposed classification system was tested with two different classifiers, i.e. the MLP-ANN classifier and the FCM algorithm. From the results in Table 1, it can be seen that both classifiers achieved similarly high classification rates. In the actor condition, the MLP-ANN classifier using the cross-validation method slightly outperformed the FCM method. It must be pointed out that the MLP-ANN classifier for the actors’ ERPs in the time window -6 to 146 msec provided excellent sensitivity (up to 100%) and consequently excellent NPV. For the observers’ actions, the FCM method achieved slightly better classification rates than the MLP ANN classifier, especially for the time intervals -6 to 500 msec and the sub-region SR-2.

In conclusion, the present study showed that the MVAR approach can be used reliably for the classification of brain potentials accompanying erroneous and correct actions, both for actors and observers. The additional SA technique may be used confidently to minimize the search space for the selection of relevant features such as the amount and set of electrodes, the model order, and the data interval, leading to the best classification performance. The two classifiers that were investigated, the MLP ANN method and the FCM approach yielded similarly high level performance rates, suggesting that both models may be used interchangeably for classification of brain potentials associated with correct and incorrect behaviors. The analysis of different time windows revealed comparable performance for all three intervals, suggesting that the MVAR model is largely unaffected when time intervals are prolonged. Results suggest a slight improvement for the classification of brain potentials of observers when the Pe is included in the data interval in addition to the ERN. For the actor condition the ERN provides an optimal signal for classification, with little or no improvement from inclusion of the Pe. Both the selection of sub-regions (30 electrodes or 16 electrodes) and the nature of the signal (from the actor or from the observer) may largely influence the optimal selection of electrodes for classification. Among other benefits, automated classification systems, through the features providing best classification results, provide indications about underlying mechanisms related to pathological processes, when the systems are applied to neuropsychiatric disorders. In this perspective, future research will include the evaluation of the classification system in data from patient populations that had been investigated in the framework of error processing research such as patients suffering from Attention deficit hyperactivity disorder, obsessive-compulsive disorder, depression and schizophrenia [39]. Insights might be provided both about potential mechanisms of the illnesses and error monitoring [10]. Finally, with knowledge gained from the medical applications, in a more general context, our study may foster the future development of systems capable of automatically detecting erroneous actions in human-human and human-artificial agent interactions. In this perspective, future work will include the performance evaluation of the above classification algorithms on single-trial recordings of both actors and observers performing controlled joint-action tasks, for the implementation of classification systems, that will eventually enable the discrimination between such actions in real-time, with applications in fields were human performance monitoring is critical, such as flying, driving and industrial process monitoring.