Master Thesis (I)

Topic: Development of interfaces for the use of communication and entertainment applications by people with a serious motor and verbal disability.

Responsible: Nicolas Simon

The aim of this project is to develop a user interface for people who cannot move any parts neck downwards of their body and in addition are seriously limited in their verbal communication. For these people using daily applications like writing an email, browsing, watching videos and listening to music becomes nearly impossible without supplementary tools. Using a keyboard and a mouse is not possible so other approaches must be considered. Every interaction can also be done by the Sweep-and-Click-Method introduces by Lopes [1]. GUI-elements like buttons, lists, etc. are visually highlighted after one another (sweeping) and if the desired GUI-element is highlighted the user can perform a selection (click). The sweeping is done automatically by the program, so the important part is to think about how the user, who has no movable body part downwards his neck, can execute this selection.
For this problem a lot of different solutions of the following categories were proposed.

  • Using vision based approaches like a camera with object recognition algorithms,
  • Using BCI approaches by analyzing EEG signals,
  • Using the remaining muscle activity with EMGs or

Other creative ideas like the IntegraMous [2], where mouse clicks are simulated by sucking and blowing.
The problem of these applications is that they consider just one or a subset of these categories. For example the user can navigate and control applications just by EEG or just by EMG.
In this approach an event layer is added between the hardware (e.g. an EEG neuro headset) and the software (e.g. a multimedia and communication center). The software is just receiving events like a selection event or focusing the next GUI-element. And the hardware with the signal processing application provides these events, for example a selection event each time the user thinks of an imaginary arm movement.
The whole concept is shown by figure 1.

Schaubild1_v4_english

 

The GUI-Application is the multimedia and communication center, the Eventgenerator-Application does the signal processing and sends the events. The Configuration-Client updates the configuration used by the GUI-Application and handles things like font size, account information for the e-mail application etc.

The main user interface shows figure 2. The following entertainment and communication applications are implemented.

  • An alarm function to call for help.
  • A food application to select breakfast, lunch and supper.
  • A video player to watch locally stored movies.
  • A communication app to show configured sentences in a dialog box.
  • A music player to play locally stored music and listen to configured online radio station
  • A web browser for surfing in the internet.
  • An e-mail client to receive and send simple text e-mails.
  • A gaming application. Just now only windows pinball is playable.

GuiApp_Startscreen

 

The whole interface is controlled by events and the following are supported:

  • The “selection”-event to select a GUI -element like a button or a list.
  • The ”next”-event to highlight the next GUI -element.
  • The “previous”-event to highlight the previous GUI -element.
  •  The “back”-event to quit dialogs.
  • The “up”- and “down”-events are used for games or scrolling in lists and the browser.
  •  The “alarm”-event for immediately triggering the alarm function.

One big advantage of the event based approach is that not just one but a lot event generator applications can be connected, also simultaneously. So for example it becomes possible to have one device (e. g. EMG) for just triggering the “alarm”-event and a second device (e.g. the EEG) for other events like the “selection”-event.

The application was first tested with healthy people to remove bugs and correct wrong behaviors. So a next step will be the testing with disabled people e.g.  people with Locked- In-Syndrom to evaluate the real use of this application.

References:

  1.  Designing User Interfaces for Severely Handicapped Persons. João Brisson Lopes, 2001, http://dl.acm.org/citation.cfm?id=564553
  2.  “IntegraMouse”, http://www.rehavista.de/?at=Produkte&p=40870
Posted in Projects | Leave a comment

How to add new Module/Box to designer in OpenViBE?

In the previews article we talked about how to generate .h and .cpp files by using Skeleton Generator for OpenViBE. Here we want to learn how to add this new module to designer and make it visible. There are some few steps which should be done and some points that you have to check them to not having further errors in time of compilation.

Step 1: Including the module to ovp_main.cpp

Open the OpenViBE folder in your system >> “openvibe-plugins”, in this folder there are 15 folders, which each of them contains codes for different categories of plugins. If you open any of them and check the folder “trunk” >> “src”, you will see that, there are two files “ovp_defines.h” and “ovp_main.cpp” with the same name in all of plugin categories. These two file contains the information about modules header file and class ID specifically for correspond plugin and its modules. In our case we don’t need to add any code in “ovp_defines.h”, because if you check the “ovpCBoxAlgorithmMomentaryFrequency.h” file, you will see that, in the lines 12 and 13 the class ID already defined for our module. In the other hand, we should include two lines to “ovp_main.cpp”. Open the “ovp_main.cpp” file and add   #include “ovpCBoxAlgorithmMomentaryFrequency.h” in the line 12 and also add OVP_Declare_New(OpenViBEPlugins::SignalProcessingGpl::CBoxAlgorithmMomentaryFrequencyDesc); to the line 75. Save the file and exit.

Note: In the files “ovpCBoxAlgorithmMomentaryFrequency.h” and “ovpCBoxAlgorithmMomentaryFrequency.h”, our module defined as sub class of “SignalProcessing” namespace, but actually we located the .h and .cpp files of module in “Signal-processing-gpl” plugin, that’s why, in these two files, instead of “SignalProcessing” add “SignalProcessing Gpl”.

Step 2: Make module visible

In the previews step we add necessary information to make our recognizable and the module be compile with other modules in time of compilation, but still we cannot see the Momentary Frequency module in the designer. For this purpose we have to change the parameter of “Designer_ShowUnstable” to “true” in the “openvibe.conf” file in the “dist” >> “share” folder.

At this point come back to “OpenViBE” folder >> “scripts” and build the source code again. After finishing the compilation, open the OpenViBE designer from “dist” folder and you can see the Momentary Frequency module under “Signal processing” >> “Filtering”. Drag and drop the Momentary Frequency to the left side and double click on it, the “Configure Momentary Frequency setting” window will show up and in that window, you can see the two setting values for “AdapTime_FreqEstimation” and “AdapTime_MeanEstimation”.

 

Posted in Development, General, Software | Leave a comment

How to create a new module/Box in OpenViBE (Momentary Frequency Module)?

This article is about creating a module or box in OpenViBE by using Skeleton Generator. Skeleton generator is a nice application for .h and .cpp file generating which specially designed for OpenViBE by OpenViBE team.

OpenViBE itself has some predefined variable type and functions which you can see most of them inside all modules .h and .cpp files. These variables and functions help the core application of OpenViBE to recognize the behavior of the modules. In this purpose, they provide skeleton generator to create the necessary functions and variables for specific module. On the way of explaining the procedure of module creation, we are going to create a module for calculating Momentary Frequency Module.

Open the OpenViBE folder >> dist >> double click on “ov-skeleton-generator.cmd”. The “OpenViBE Skeleton-Generator” will show up.

There are two radio box “Driver skeleton” and “Box skeleton”, by default the “Driver skeleton” is checked, click on “Box skeleton”, because we do not want to create any driver (there are some devices which OpenViBE cannot connect and reads data from them, in that purpose you can use Driver skeleton to create drivers to connect and read data for specific devices). In the “Author” and “Company“ text boxes write the information which you want to show up about your module, then click on “OK” button. In the next window, there are seven tabs correspond to module information and behavior.

General Tab:

This tab is for module/box name, class name in .h and .cpp file, version of box, category which the module belongs and will show up in the designer, short and detailed description of the module/box and at the end icon of the box.

 

Inputs Tab:

In this tab, we chose how many and what type of inputs does the box need. In case of Momentary frequency, it needs one input with type of signal. Click on the “Add” button and add one input with name “Signal” and type “signal”. There are two check boxes in this tab, for this module, there is no need to any modification or add/remove option for user, uncheck both of them.

Output Tab:

This tab is about the output of the box, Momentary frequency has one output with type of signal, so, click on “Add” button and add one output with name “Momentary Signal” and type “signal”. In case of output also the module doesn’t need to be edit, add or remove, uncheck both check boxes.

Settings Tab:

If you already have started to explore the existed modules in designer, perhaps you see that some of boxes have some setting like necessary coefficients or address of file to be read or right. Momentary frequency has two setting value, Adaptation time for the mean estimator and frequency estimator. Add two setting by names of “AdapTime_FreqEstimation” and “AdapTime_MeanEstimation” with type “Float” and Default value “0.002” and uncheck the both check boxes too.

Codec Algorithm Tab:

Codec Algorithm tab is the place to add predefined functions which provided by OpenViBE team. Here, we add two functions “Signal stream decoder” and ”Signal stream encoder”. OpenViBE defined a special “signal” variable which contains a batch of information about signal data. In Momentary Frequency, we want to work on signal data, so, we use the “Signal stream decoder” to extract the part which we need for Momentary frequency calculation and put it inside another special variable “Matrix”. By “Matrix” we can access to necessary data directly. Because, the output of Momentary frequency is a type “signal” again, we add a “Signal stream encoder” to encode calculated data into a “signal”type. In this tab it recommended to use codec toolkit, let the check box be checked.

Processing Method Tab:

In this tab, we define how the module will call by OpenViBE core application. As you can see, there are two check boxes, for Momentary frequency, it needs to be called for all chunk of input (OpenViBE sends data as input by blocks/chunks of data, each chunk contains 32 values of signal data), so, click on the first check box to be check. The second check box is about calling the module in case of specific frequency, in this module, it doesn’t need this option, let it be unchecked.

Box Listener Tab:

By this tab, you can define a Box Listener for the module; they will run just in case their condition be true, their statement could be modified inside the code of module. For Momentary frequency, it doesn’t need any Box Listener, let them all be inactive.

At this moment we are ready to check and then generate the code of Momentary frequency module/box. Click on the “Check” button. You should see the following information at the bottom of Skeleton Generator window:

At the end, click on “Generate” button to generate the code. In the “Select the destination folder” window, go to the OpenViBE folder >> openvibe-plugins >> signal-processing-gpl >> trunk >> src and click on “OK” button. This address is the location which we want the module code be generate be compile.

If you check the given address, you can see that the “ovpCBoxAlgorithmMomentaryFrequency.h” and “ovpCBoxAlgorithmMomentaryFrequency.cpp” files are generated.

Posted in Development, Software | Leave a comment

How to install/compile OpenViBE in Windows

In this article we are going to explore how to install and compile OpenViBE source code in Windows. Of course, if you don’t want to implement any extra Module in OpenViBE for any purpose related to your research, you just need to download the OpenVibe installer and follow the instructions to install it and start to create new scenarios. For the purpose of implementing new module or edit the existing module you need to install and compile the OpenViBE source code. In below we explain some few easy steps to do it in short time and correctly.

Step 1: Installing Visual Studio:

For compiling OpenViBE source code you need a compiler on your system, because OpenViBE by itself doesn’t have any compiler and use the default compiler of system to compile its code. In Windows operating based systems they recommend to use Visual Studio 2008 or 2010 which provides CL compiler.

Step 2: Installing CMake:

After finishing the installation of Visual Studio on your system, you need to install CMake (Cross Platform Make) too. They used CmakeList file in OpenVIBe, so in this way compiler needs CMake, surprisingly, in time of installing of dependencies of OpenVIBE you can see that, they provide CMake there also, but in some cases it doesn’t work and in time of compile it gives some error related CMake.

Step 3: Installing OpenViBE source code:

First make sure that, you download the latest stable version of OpenViBE source code. Then extract the ZIP file to a location which you want to keep OpenViBE in your system. Then open the OpenViBE folder and open the “scripts” folder, you should see the below picture:

OpenViBE also has some prerequisites which should be install before being compile. For this purpose double click on “win32-install_dependencies.exe”. In the “OpenViBE dependencies Setup” window, click on “Compilation platform” to be expand.

 

You will see by default the Visual C++ 2010 is selected, so, for not having further error in compilation select the one which you installed in your system too. Then click on “Install” button.

In the “scripts” folder there are two files with names: “win32-build.cmd” for building the OpenViBE source code and “win32-clean.cmd” for cleaning all built data. For compiling OpenViBE source code double click on “win32-build.cmd”, depend to your system performance, it will takes few minutes to finish compilation process. After finishing the compilation, if you check the OpenViBE folder, there two more folders “dependencies” and “dist” added.

For the first run, open the “dist” folder and double click on “ov-designer.cmd” file and the “OpenViBE Designer” window will show up.

In the menu bar, click on the File >> New, then from the right side of the OpenViBE Designer, in the “Boxes”,  click on “Samples” folder to be expand and drag “Sinus oscillator” and drop it in the left side of window. This module will generate sinus signal. For display the sinus signal, click on the “Visualisation” >> “Basic” , then drag “Single display” module and drop it in left side of window under “Sinus oscillator” module. Then connect the pink triangle of Sinus oscillator to green triangle of Signal display and from the menu bar click on start scenario button with (play). you should see the following window:

 

 

Posted in Development, Software | Leave a comment

The long way to implement a TCP server box in OpenVIBE

Hi guys here is something I want to share with you on my way to implement a new Box in OpenVIBE.

The goal was to run a TCP sever in a box which sends a command string when a specific stimulation arrives. Waiting for clients and the sending takes place in a thread for which I use the CreateThread() function from windows.h.

First I tried to integrate an outside of OpenVIBE working version with windows sockets. But because I had to include ws2tcpip.h I got a lot of errors of the kind:

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\winsock2.h(1818) : error C2375: ‘recv’ : redefinition; different linkage

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\winsock.h(790) : see declaration of ‘recv’

and

C:\Program Files\Microsoft SDKs\Windows\v6.0A\include\ws2tcpip.h(761) : error C3861: ‘WSASetLastError’: identifier not found

I tried to link WS2_32.Lib as they did with winmm.lib in ‘<OpenVIBE Path>\openvibe-plugins\stimulation\trunc\CMakeLists.txt’  but the errors were still there.

 

Another way was to use the socket functions from boost/asio.hpp because boost is already included in OpenVIBE as a third party module. But here also as soon as I included the header I got the error.

C:\openvibeDev\dependencies\boost\include\boost/asio/detail/socket_types.hpp(27)

: fatal error C1189: #error : WinSock.h has already been included

So again one of the windows header files makes problems. A tip to include all the related headers after windows.h I already tried out.

 

So up to now the only way to get my server box running is to use the example client server implementation already included in OpenVibe (<OpenVIBE Path>\socket\trunc\test).

So if you have any suggestions how to use boost asio or windows sockets please let me know.

Posted in Development | Leave a comment

Interesting article (German)

Hi,

I found this article this morning and thought it might be interesting to the community.

http://heise.de/-1578581

The arcticle is in German language and deals with emotion computing research at MIT and Tufts University.

Enjoy.

Posted in References | Leave a comment

Emotiv EPOC headset

The Emotiv headset is a consumer EEG headset.  In common slang, it’s a brain-computer interface.

We decided to use this kind of hardware for our emotion computing lab. For doing so we purchased

the research editions of Emotiv EPOC.

 

Other information resources:

OpenViBE

Emotiv

Epoc Neuroheadset

Using Your Thought-Controlled I Phone to Dial Home

Mind Reading (Neural Decoding) Goes Mainstream

 

Posted in Hardware | 2 Comments

openVibe Software toolkit

OpenViBE is a software platform dedicated to designing, testing and using brain-computer interfaces. The package includes a Designer tool to create and run custom applications, along with several pre-configured and demo programs which are ready for use. [see Wikipedia also]

OpenViBE is software for realtime neuroscience. It can be used to acquire, filter, process, classify and visualize brain signals in real time. We are using openVibe for the our work.

OpenVibe is the first library of functions written in C++ of this type developed by INRIA - Institut national de recherche en informatique et automatique (France) – it can be integrated and applied quickly and easily .

For further information, please have a look at:

OpenViBE: Open-Source Software for Brain-Computer Interfaces ERCIM News, accessed March 2011

or directly on the openVibe homepage:

http://openvibe.inria.fr

Posted in Software | Leave a comment

R & D Project (II)

Topic: “Analysis and Evaluation of the application possibilities of an EPOC neuro-headset from Emotiv.

Responsible: Nicolas Simon

Abstract:

The Emotiv EPOC neuro headset offers many possibilities to analyse facial expressions, emotions and thoughts. Within this work the Emotiv neuroheadset will be analysed and evaluated with respect to its application possibilities. A demo application will be developed that allows for differentiation of the neuro commands for “left” and “right” and hence map those commands onto an external device which is a picture flow application running on an Android phone. In addition the level of current concentration shall be displayed in the Android device as well. The communication between the phone and the analysis computer will be implemented by using a conventional TCI/IP connection.

 

Note:  This R & D  Project is a project being carried out as a preparation for the Master Thesis project at the Bonn Rhein Sieg University of Applied Sciences.

Posted in Projects | Leave a comment

R & D Project (I)

Topic: “Exploration of various classification algorithms for Sensory Motor Rhythm (SMR) analysis of a ‘Turn Left’ command for a Care-o-Bot simulator using an Emotiv EPOC headset.

ResponsibleRamesh Kumar Natarajan

 

Abstract:

Emotiv Epoc is one of the best BCI devices which offers a special variant for researchers to develop BCI applications. Moreover, it works based on a traditional 10-20 EEG system with 16 electrodes and a gyroscope. The interface for the device is implemented by an EPOC control panel that comes along with the device. The users are trained to use the device by providing a cube interface with a separate training profile for each user. The trained data sets can be used only for the specific user profile and this leads to the device to be not usable by an untrained user. For the device to be applied in the field of robotics, the brain signals from the users are to be made generic by applying various pattern recognition algorithms to study the common features of the brain signals of various individuals of a particular command.

 

Note:  This R & D  Project is a project being carried out as a preparation for the Master Thesis project at the Bonn Rhein Sieg University of Applied Sciences.

Posted in Projects | Leave a comment