Optimizing SSVEP-Based BCI System towards Practical High-Speed Spelling

    The brain–computer interface (BCI) spellers based on steady-state visual evoked potentials (SSVEPs) have recently been widely investigated for their high information transfer rates (ITRs). This paper aims to improve the practicability of the SSVEP-BCIs for high-speed spelling. The system acquired the electroencephalogram (EEG) data from a self-developed dedicated EEG device and the stimulation was arranged as a keyboard. The task-related component analysis (TRCA) spatial filter was modified (mTRCA) for target classification and showed significantly higher performance compared with the original TRCA in the offline analysis. In the online system, the dynamic stopping (DS) strategy based on Bayesian posterior probability was utilized to realize alterable stimulating time. In addition, the temporal filtering process and the programs were optimized to facilitate the online DS operation. Notably, the online ITR reached 330.4 ± 45.4 bits/min on average, which is significantly higher than that of fixed stopping (FS) strategy, and the peak value of 420.2 bits/min is the highest online spelling ITR with a SSVEP-BCI up to now. The proposed system with portable EEG acquisition, friendly interaction, and alterable time of command output provides more flexibility for SSVEP-based BCIs and is promising for practical high-speed spelling.
    480 views
    0 comments
    0 likes
    ORIGINAL POST
    By Jiabei Tang, Minpeng Xu, Jin Han, Miao Liu, Tingfei Dai, Shanguang Chen and Dong Ming
    components
    Hardware Components
    WIZnet W5500
    X 1
    details

    URL

    Figure 1. Overview of the system framework.

    1. Introduction

    Brain–computer interfaces (BCIs) allow users to communicate with the external devices by converting brain signals into commands [1,2]. BCIs can help people with neuromuscular diseases to improve the life quality [3] or help special appliance operators like astronauts whose movements were restricted by the environment to work more efficiently [4,5]. As a kind of brain signal that owns high temporal resolution and convenience of acquisition, electroencephalogram (EEG) is welcomed by BCI researchers. Event-related potentials (ERPs) [6,7], steady-state visual evoked potentials (SSVEPs) [8,9], and event-related desynchronization/synchronization (ERD/ERS) [10,11] are typical EEG features used in BCI researches.
    Of these features SSVEPs that were induced by repetitive stimuli are widely employed in cognitive research and high-speed BCI systems for their high stability and signal-to-noise ratio (SNR) [12,13]. Researchers payed great efforts on the performance improvement of SSVEP-based BCIs in recent years, which mainly focused on the number of targets and target recognition algorithms. In order to increase the number of targets, a variety of novel coding methods were proposed, e.g., frequency shift keying (FSK) method that encodes commands into binary digits with two frequencies [14], intermodulation frequencies method that uses additional modulation frequencies [15], and hybrid coding methods that combine other EEG features such as P300 [16,17], etc. In particular, joint frequency-phase modulation (JFPM) method has been proved to improve the separability between targets and achieve high-speed SSVEP-BCI systems [18]. With regard to algorithms, various kinds of identification algorithms were applied in the SSVEPs-based BCIs [19], e.g., canonical correlation analysis (CCA) [20] and its various optimizations [21,22], multivariate synchronization index (MSI) [23], maximal-phase-locking value and minimal-distance (MP and MD, respectively) [24], and task-related component analysis (TRCA) [25], etc. Thanks to the endeavor in the encoding and decoding methods, the SSVEP-based BCIs have achieved the highest information transfer rate (ITR) among the noninvasive BCI paradigms.
    Although many high-speed systems based on SSVEPs have been established in previous studies, it is meaningful to enhance the practicability of the systems for spelling in real life. For example, most of these studies used EEG devices designed for research in the laboratory [9,21,22,25], such as Neuroscan Synamps2 system. The research-grade devices possess excellent signal amplification performance and diverse functions, whereas most of the functions are superfluous for a practical BCI system and push up the cost. In addition, the 5 × 8 matrix layout was popular in previous SSVEP-based spellers and the users needed to remember the location of each command before the experiment, which increases the workload and slows down the spelling. Furthermore, these systems used fixed stimulating time, i.e., the fixed stopping (FS) strategy. As is known in the P300-based BCIs, the dynamic stopping (DS) strategy enables the self-check of recognition confidence for a BCI system so as to quicken the output when it is confident about the result, whereas keep acquiring data when the correctness of the decision is not sure [26,27]. A few studies have tested the performance of DS strategy in an SSVEP-based BCI [28,29,30], but it was evaluated by offline analyzing the offline collected EEG data like an online experiment. The feasibility of real-time DS strategy in a practical online system needs further verification.
    The goal of this study was to design a high-speed SSVEP-based BCI system for practical use. Figure 1 is the block diagram of our system. The novelty of our system was reflected from several aspects. Firstly, we simplified the acquisition by developing a dedicated low-cost EEG amplifier with self-designed circuit and optimized the stimulation by arranging the instructions like a keyboard that would be familiar to users. Secondly, in order to extract the SSVEPs more effectively for a high-speed system, the standard forward filtering was applied instead of the frequently-used zero-phase filtering for reliable online noise reduction and the TRCA spatial filter was modified (named mTRCA) to enhance the target recognition. Last, but not the least, the DS strategy based on Bayesian posterior probability was incorporated into the system to obtain flexible stimulating time and improve the ITRs.
    Figure 1. Overview of the system framework. The stimulation was controlled by a computer, and the electroencephalogram (EEGs) were acquired by a self-developed EEG amplifier. The stimulus onset triggers and the EEG data were sent and received with user datagram protocol (UDP) and transmission control protocol/internet protocol (TCP/IP), respectively. The EEG data were decoded by the computer and the results were presented on the screen. The lower right is the circuit board of the EEG amplifier, and the upper right shows the innovation of the EEG decoding.
    As the trial duration of a SSVEP-BCI is much shorter than that of a P300-BCI due to the different coding schemes, three issues were concerned for real-time DS in an online SSVEP-BCI. The first concern is the unfixed stimulating time caused by the immediate stopping of stimulus after satisfying the output condition in DS strategy. As most users have been used to fixed stimulating time in normal FS BCIs, the sudden stopping of stimulating might distract the attention of users and delay the shifting to the next target; thus, leading to the performance decline. Therefore, this study used unfixed stimulating time in the offline calibration experiment to imitate the DS in online operation so that the subjects could adjust to the unfixed timing, and the variation of EEGs could also be covered by the calibration data. The second concern focused on the output condition of DS due to the fact that the probability distribution might change with the stimulus frequency and data length. We raised an adaptive threshold generating method that was easy to implement so as to fit the variation of probability distribution. Another concern is that the real-time DS requires the system to perform the recognition algorithm in very short time. To this end, the programs of some key processes for recognition were ported to C Mex from MATLAB for accelerating execution to ensure the real-time performance.

    2. Materials and Methods

    2.1. Experimental Protocol

    2.1.1. Stimulus Design for Practical Spelling

    Figure 2 a illustrates the design of the stimulation. The participants were seated at a distance of 60 cm from a liquid-crystal display (LCD) monitor with the refresh rate of 60 Hz. In order to make the system friendly and practical for users, forty targets were rearranged as the pattern of a keyboard with each target subtended 2° of visual angle. In particular, the keys of Backspace and Space were set longer than on general keyboards. An output box was placed above the targets. The frequencies ranged from 8.0 to 15.8 Hz with an interval of 0.2 Hz and the phase interval between two neighboring frequencies was 0.35 π, which were in accordance with the JFPM method in previous studies. In addition, the frequency approximation approach proposed by Wang et al. [31] was used to modulate frequencies and phases in the monitor. The stimulation was developed on the MATLAB platform using the Psychtoolbox 3 [32], and the stimulating onset triggers were sent to the EEG amplifier via user datagram protocol (UDP).
    Figure 2. The stimulus design of the 40-target steady-state visual evoked potential (SSVEP)-based brain–computer interface (BCI) system. (a) Schematic of the stimulation, in which the 40 targets were distributed according to the layout of a keyboard and the frequency and phase of each target were marked out at upper left corner. The cue was presented to subjects as a yellow rectangle around the target. (b) Trial timing diagram of the experiment. (c) Three kinds of EEG epochs corresponding to different data lengths were obtained after offline experiment.

    2.1.2. Experimental Procedure

    Twelve healthy subjects (five males and seven females) aged 20 to 26 years old with normal or corrected normal sight participated in this study. The study was conducted in accordance with the Declaration of Helsinki and the experimental procedures were approved by the Institutional Review Board at Tianjin University. The participants provided written consent after the details of the experiment were explained. All the subjects participated in both the offline and online experiments.
    Figure 2b shows the trial timing of the experiments. Each trial started with a rest period for 0.5 s, followed by a flash stage. A yellow box would appear around the target as a cue. The subjects were asked to shift their gaze to the target as soon as possible within the rest stage and focus on the dot displayed at the center of the target within the flash stage. In order to acquire a model that were fit with the unfixed stimulating time in DS situation, three kinds of stimulating time (0.32 s, 0.6 s, and 1 s) were randomly used in offline stimulation. The offline calibration experiment consisted of six rounds, with three kinds of stimulating time used for each of the 40 targets per round. Hence, the total trials were 40 × 3 × 6 = 720 and the total experimental time was 0.5 × 720 + (0.32 + 0.6 + 1) × 40 × 6 = 13.68 min. The 720 trials were divided into 12 blocks with 60 trials in each block. After offline blocks, we could obtain 18 EEG epochs between 0.2 s to 0.32 s, 12 epochs between 0.32 s to 0.6 s, and 6 epochs between 0.6 s to 1 s for each target, as shown in Figure 2c.
    The online experiment contained two sub-experiments. Firstly, a cue-guided experiment including 5 blocks was conducted and the subjects were asked to complete 40 trials corresponding to all 40 targets in each block. When a stopping trigger was received from the algorithm (introduced in 2.4) during the flash stage, the stimulation program would stop flashing and prompt the next target with the result displayed in the output box at the same time. If the stimulating time reached 1 s without a stopping trigger (trial 3 in Figure 2b), the program would treat the target as incorrect and prompt the next one. A free spelling experiment was conducted following the cue-guided experiment. The subjects were asked to input “TIANJIN UNIVERSITY 1895” two times without visual cues. The result of each trial would be displayed in the output box and reported by voice as feedback. When an incorrect input happened, the subjects should stare at the key of Backspace to remove the wrong character. The gaze shifting time was determined according to the self-feeling of subjects after a tentative block was completed.
    Source: mdpi.com

    URL

    Figure 1. Overview of the system framework.

    1. Introduction

    Brain–computer interfaces (BCIs) allow users to communicate with the external devices by converting brain signals into commands [1,2]. BCIs can help people with neuromuscular diseases to improve the life quality [3] or help special appliance operators like astronauts whose movements were restricted by the environment to work more efficiently [4,5]. As a kind of brain signal that owns high temporal resolution and convenience of acquisition, electroencephalogram (EEG) is welcomed by BCI researchers. Event-related potentials (ERPs) [6,7], steady-state visual evoked potentials (SSVEPs) [8,9], and event-related desynchronization/synchronization (ERD/ERS) [10,11] are typical EEG features used in BCI researches.
    Of these features SSVEPs that were induced by repetitive stimuli are widely employed in cognitive research and high-speed BCI systems for their high stability and signal-to-noise ratio (SNR) [12,13]. Researchers payed great efforts on the performance improvement of SSVEP-based BCIs in recent years, which mainly focused on the number of targets and target recognition algorithms. In order to increase the number of targets, a variety of novel coding methods were proposed, e.g., frequency shift keying (FSK) method that encodes commands into binary digits with two frequencies [14], intermodulation frequencies method that uses additional modulation frequencies [15], and hybrid coding methods that combine other EEG features such as P300 [16,17], etc. In particular, joint frequency-phase modulation (JFPM) method has been proved to improve the separability between targets and achieve high-speed SSVEP-BCI systems [18]. With regard to algorithms, various kinds of identification algorithms were applied in the SSVEPs-based BCIs [19], e.g., canonical correlation analysis (CCA) [20] and its various optimizations [21,22], multivariate synchronization index (MSI) [23], maximal-phase-locking value and minimal-distance (MP and MD, respectively) [24], and task-related component analysis (TRCA) [25], etc. Thanks to the endeavor in the encoding and decoding methods, the SSVEP-based BCIs have achieved the highest information transfer rate (ITR) among the noninvasive BCI paradigms.
    Although many high-speed systems based on SSVEPs have been established in previous studies, it is meaningful to enhance the practicability of the systems for spelling in real life. For example, most of these studies used EEG devices designed for research in the laboratory [9,21,22,25], such as Neuroscan Synamps2 system. The research-grade devices possess excellent signal amplification performance and diverse functions, whereas most of the functions are superfluous for a practical BCI system and push up the cost. In addition, the 5 × 8 matrix layout was popular in previous SSVEP-based spellers and the users needed to remember the location of each command before the experiment, which increases the workload and slows down the spelling. Furthermore, these systems used fixed stimulating time, i.e., the fixed stopping (FS) strategy. As is known in the P300-based BCIs, the dynamic stopping (DS) strategy enables the self-check of recognition confidence for a BCI system so as to quicken the output when it is confident about the result, whereas keep acquiring data when the correctness of the decision is not sure [26,27]. A few studies have tested the performance of DS strategy in an SSVEP-based BCI [28,29,30], but it was evaluated by offline analyzing the offline collected EEG data like an online experiment. The feasibility of real-time DS strategy in a practical online system needs further verification.
    The goal of this study was to design a high-speed SSVEP-based BCI system for practical use. Figure 1 is the block diagram of our system. The novelty of our system was reflected from several aspects. Firstly, we simplified the acquisition by developing a dedicated low-cost EEG amplifier with self-designed circuit and optimized the stimulation by arranging the instructions like a keyboard that would be familiar to users. Secondly, in order to extract the SSVEPs more effectively for a high-speed system, the standard forward filtering was applied instead of the frequently-used zero-phase filtering for reliable online noise reduction and the TRCA spatial filter was modified (named mTRCA) to enhance the target recognition. Last, but not the least, the DS strategy based on Bayesian posterior probability was incorporated into the system to obtain flexible stimulating time and improve the ITRs.
    Figure 1. Overview of the system framework. The stimulation was controlled by a computer, and the electroencephalogram (EEGs) were acquired by a self-developed EEG amplifier. The stimulus onset triggers and the EEG data were sent and received with user datagram protocol (UDP) and transmission control protocol/internet protocol (TCP/IP), respectively. The EEG data were decoded by the computer and the results were presented on the screen. The lower right is the circuit board of the EEG amplifier, and the upper right shows the innovation of the EEG decoding.
    As the trial duration of a SSVEP-BCI is much shorter than that of a P300-BCI due to the different coding schemes, three issues were concerned for real-time DS in an online SSVEP-BCI. The first concern is the unfixed stimulating time caused by the immediate stopping of stimulus after satisfying the output condition in DS strategy. As most users have been used to fixed stimulating time in normal FS BCIs, the sudden stopping of stimulating might distract the attention of users and delay the shifting to the next target; thus, leading to the performance decline. Therefore, this study used unfixed stimulating time in the offline calibration experiment to imitate the DS in online operation so that the subjects could adjust to the unfixed timing, and the variation of EEGs could also be covered by the calibration data. The second concern focused on the output condition of DS due to the fact that the probability distribution might change with the stimulus frequency and data length. We raised an adaptive threshold generating method that was easy to implement so as to fit the variation of probability distribution. Another concern is that the real-time DS requires the system to perform the recognition algorithm in very short time. To this end, the programs of some key processes for recognition were ported to C Mex from MATLAB for accelerating execution to ensure the real-time performance.

    2. Materials and Methods

    2.1. Experimental Protocol

    2.1.1. Stimulus Design for Practical Spelling

    Figure 2 a illustrates the design of the stimulation. The participants were seated at a distance of 60 cm from a liquid-crystal display (LCD) monitor with the refresh rate of 60 Hz. In order to make the system friendly and practical for users, forty targets were rearranged as the pattern of a keyboard with each target subtended 2° of visual angle. In particular, the keys of Backspace and Space were set longer than on general keyboards. An output box was placed above the targets. The frequencies ranged from 8.0 to 15.8 Hz with an interval of 0.2 Hz and the phase interval between two neighboring frequencies was 0.35 π, which were in accordance with the JFPM method in previous studies. In addition, the frequency approximation approach proposed by Wang et al. [31] was used to modulate frequencies and phases in the monitor. The stimulation was developed on the MATLAB platform using the Psychtoolbox 3 [32], and the stimulating onset triggers were sent to the EEG amplifier via user datagram protocol (UDP).
    Figure 2. The stimulus design of the 40-target steady-state visual evoked potential (SSVEP)-based brain–computer interface (BCI) system. (a) Schematic of the stimulation, in which the 40 targets were distributed according to the layout of a keyboard and the frequency and phase of each target were marked out at upper left corner. The cue was presented to subjects as a yellow rectangle around the target. (b) Trial timing diagram of the experiment. (c) Three kinds of EEG epochs corresponding to different data lengths were obtained after offline experiment.

    2.1.2. Experimental Procedure

    Twelve healthy subjects (five males and seven females) aged 20 to 26 years old with normal or corrected normal sight participated in this study. The study was conducted in accordance with the Declaration of Helsinki and the experimental procedures were approved by the Institutional Review Board at Tianjin University. The participants provided written consent after the details of the experiment were explained. All the subjects participated in both the offline and online experiments.
    Figure 2b shows the trial timing of the experiments. Each trial started with a rest period for 0.5 s, followed by a flash stage. A yellow box would appear around the target as a cue. The subjects were asked to shift their gaze to the target as soon as possible within the rest stage and focus on the dot displayed at the center of the target within the flash stage. In order to acquire a model that were fit with the unfixed stimulating time in DS situation, three kinds of stimulating time (0.32 s, 0.6 s, and 1 s) were randomly used in offline stimulation. The offline calibration experiment consisted of six rounds, with three kinds of stimulating time used for each of the 40 targets per round. Hence, the total trials were 40 × 3 × 6 = 720 and the total experimental time was 0.5 × 720 + (0.32 + 0.6 + 1) × 40 × 6 = 13.68 min. The 720 trials were divided into 12 blocks with 60 trials in each block. After offline blocks, we could obtain 18 EEG epochs between 0.2 s to 0.32 s, 12 epochs between 0.32 s to 0.6 s, and 6 epochs between 0.6 s to 1 s for each target, as shown in Figure 2c.
    The online experiment contained two sub-experiments. Firstly, a cue-guided experiment including 5 blocks was conducted and the subjects were asked to complete 40 trials corresponding to all 40 targets in each block. When a stopping trigger was received from the algorithm (introduced in 2.4) during the flash stage, the stimulation program would stop flashing and prompt the next target with the result displayed in the output box at the same time. If the stimulating time reached 1 s without a stopping trigger (trial 3 in Figure 2b), the program would treat the target as incorrect and prompt the next one. A free spelling experiment was conducted following the cue-guided experiment. The subjects were asked to input “TIANJIN UNIVERSITY 1895” two times without visual cues. The result of each trial would be displayed in the output box and reported by voice as feedback. When an incorrect input happened, the subjects should stare at the key of Backspace to remove the wrong character. The gaze shifting time was determined according to the self-feeling of subjects after a tentative block was completed.
    Source: mdpi.com
    documents
    Others
    PDF
    https://www.mdpi.com/1424-8220/20/15/4186/pdf

    COMMENTS

    Please Login to comment
      Subscribe  
    Notify of
    POSTED BY
    kitwan
    Reusable S/W
    Others