Helicopter controlled with muscle signals

Video 1: Gerhard's first tries to control a Syma S107G Helicopter

Show Text Alternative

The video shows a man, who at first is unable to control an RC-helicopter, because he is sitting in a wheelchair due to tetraplecic paralysis.

At the beginning of the video, he ist looking down sadly to his helicoper, which of course won't move.

After a screen transition, with the caption "The AsTeRICS rescue team will help", the members of the AsTeRICS project come running into the screen, and set up their devices.

A mouthstick, a laptop, an Arduino based IR Transmitter and a modded OpenEEG to measure EMG signals. They connect the OpenEEG with the electrodes on the mans schoulders.

After that, the helicopter flies and the pilot is happy. The video then shows how he flies for a few minutes.

One week later, after flying a lot, he gets a muscle cramp in his shoulders. Not even a massage was able to help, so the AsTeRICS rescue team has to help again.

This time, he gets a sip/puff control - and again, the helicopter flies. After some more flight scenes, the screen fades to a frontshot of the pilot with the whole KI-I helicopter-fleet on his chair - then the credits appear.

Cast:
Pilot	Gerhard Nussbaum
AsTeRICS-Team	Stefan Parker Roland Ossmann David Thaller
Camera	Stefanie Chea Stefan Parker Roland Ossmann
Cut	Stefanie Chea
Music by	The Dead Rocks - My Nurse Wanna Dance with ME The Dead Rocks - One Million Dollar Theme The Modern Fighting Vehicles - Neon LukHash - Make It Away La Guerra delle Formiche - Rain Mad Mav - Morbit

In the course of the AsTeRICS project, a mobile hardware platform and an open source software framework have been developed to make various ICT-based services and solutions accessible to people with severe physical disabilities, especially with problems in their upper limbs. AsTeRICS is based on a flexible plugin system which already contains about 100 plugins like speech recognition, environment control (KNX, FS20, IR-Trans), mouse/keyboard emulation, PS3 gamepad emulation etc.. These plugins can be used to create new assistive applications by combining the components within a graphical user interface. For an introduction on how to use the system see the AsTeRICS Quickstart guide.

During the last few weeks, the main work was to find ways to make toys accessible to people with motor disabilities. One outcome of this effort was the integration of the toy helicopter Syma S107G into the system. In this article we describe two models which were built with the AsTeRICS Configuration Suite (ACS) to control the helicopter. The first model uses a Bluetooth wheelchair joystick and a modded version of the OpenEEG device to measure EMG signals from the shoulders to allow the control of the helicopter. The second model uses a sip/puff sensor instead of the OpenEEG because the user got a cramp in his shoulders after a few days of flying 18 hours a day (watch the video ;)) For all the hardware we developed ourselves, we provide the schematic in the download package.

Download

The following package contains the plugins for the OpenEEG called P2Parser and documentation about the AsTeRICS software. Additional to the models used in the video it includes the following model:

the model 3_channel_joystick.acs allows you to fly the Syma S107G helicopter with a gamepad. It uses the left analog stick to control the throttle. Gamepad buttons 1-4 are used to control yaw and pitch.

At the beginning you should read the AsTeRICS Quickstart guide which explains how to build a webcam based headmouse and how to setup the software. For information about how to flash the Arduino and create your own plugins for the system see the Developer manual on the AsTeRICS webpage.

Software Requirements

Java 7 32 Bit, the bin directory must be set in the system path variable
Microsoft Visual C++ 2010 Redistributable Package
Microsoft .NET Framework 4

Installation

Step 1: Download the AsTeRICS Runtime
Step 2: Unzip the downloaded package
Step 3: Download the Helicopter package
Step 4: Extract the Helicopter package
Step 5: Copy the p2parser.jar file from the Helicopter package into the AsTeRICS_Runtime/ARE folder of the AsTeRICS software
Step 6: Append the following line to the AsTeRICS_Runtime/ARE/profile/loader.ini file

asterics.sensor.p2_parser.jar
Step 7: Start the ACS.exe in the AsTeRICS_Runtime/ACS/ folder
Step 8: Start the start.bat batch file in the AsTeRICS_Runtime/ARE/ folder
Step 9: After the ARE has started click on the "System/Connect To ARE" button in the ACS (in the options the correct ip has to be set)
Step 10: If the ACS asks to override the model and go into run mode press "No"
Step 11: In the ACS click on the "System/Download Component Collection" button and confirm the dialog with "Yes"
Step 12: Confirm the last dialog with "OK"
Step 13: In the ACS click the "Misc/Component Collection Manager" Button and then the "Set Active Collection as Autostart" button

Now the p2parser Plugin for the OpenEEG was successfully integrated into the ACS and ARE, and you can start building the hardware (see the following section) to rebuild the systems used in the Video.

Hardware Used

Helicopter

We are using the indoor helicopter S107G from Syma, because there is already a library from Tom Gallacher available for the Arduino (https://github.com/tomgco/Arduino_Helicopter) to control it and we already found various articles (e.g. Hackaday) about building systems to control this helicopter with devices like the Kinect or the WiiMote.

Figure 1: Syma S107G remote controlled helicopter

Arduino

An Arduino Duemilanove handles the communication between the AsTeRICS Runtime Environment (ARE) and the Infrared-LED shield. The code is based on Tom Gallachers code to interface with the Syma S107G helicopter and the communication between the Arduino and the AsTeRICS Runtime Environment uses the hardware UART of the microcontroller. The transmission protocol is based on packets with a size of 7 bytes each. Figure 2 shows the packet layout.

Figure 2: Serial protocol to transfer control data to the IR-Sender

IR-LED Shield

schematic showing the curcuit to build the IR-Sender shield

Figure 3: Schematic of the IR-Sender shield

To communicate with the helicopter we are using 6 Infrared-LED's with a peak wavelength of 940nm. Five of them are high-power LED's from Vishay type TSAL6100 and one is from type TSAL6400 which has a higher angle of half intensity. Since the maximum output-current of this pin is 40mA and one LED needs about 150mA, we integrated a Darlington circuit which amplifies the current between base and collector of the transistor efficiently. Finally we put the Arduino and the circuit in a fancy casing like Figure 5 shows. In this setup the sender reaches an emitting range of up to 25 meters.

A more simple approach for a sender with limited range would be to simply connect the anode of the IR-Led with Vcc and the cathode of the led to Pin 8 of the Arduino using a resistor.

picture showing the red casing of the IR-sender and the IR-Leds

Figure 5: Casing of the Arduino based IR-Sender

Model 1 EMG with modded OpenEEG

The setup consists of two different kinds of sensors. To control yaw and throttle of the Helicopter we use the joystick from the weelchair-steerage. More information about the model can be found at http://www.pgdt.com/products/r-net/index.html. The joystick can connect to a PC via Bluetooth and then acts as an HID-Mouse. The AsTeRICS model catches the mouse-events, processes them and sends them to the helicopter.

drawing showing the layout of the R-Net joystick

Figure 6: R-Net Bluetooth joystick

The pitch of the Helicopter is controlled by muscular activity of the user. Therefore we used a small, 2-channel EEG-device which was modified by Markus Winter in his Master thesis. For more information about the OpenEEG project have a look here: http://openeeg.sourceforge.net/doc/ Since the frequency spectrum of EEG-waves is about 0-60Hz and lower than that of usual EMG-signals the bandwidth of the channels had to be adjusted to about 75-105Hz. In addition the signal-amplification was reduced from 16dB to 4.5dB.

Figure 7: OpenEEG device

The values of the passive high-pass C229 and R231 are changed to 1kOhm and 220nF. → Cut-off frequency = 73,3Hz

The new values of the Sallen-Key low pass are: R232+R228 = 150kOhm, R229+R230 = 8,2KOhm, C234+C231=10nF; → Cut-off frequency = 106,1Hz

picture showing the schematic of a band pass filter

Figure 8: Schematic of the band pass

As an alternative to the modded OpenEEG you can also use two EMG-Shields from Olimex, see https://www.olimex.com/dev/shield-ekg-emg.html. A model to use them with an Arduino, which has the AsTeRICS CIM firmware on it, is inside the download package.

Model 2 Sip/Puff

In this model yaw and throttle is controlled by the joystick of the weelchair-steerage. The Sip/Puff Sensor is used to control the pitch channel. Blowing into the tube means fly forward and sucking means backwards. The sip/puff module is based upon an MPXV7007G piezoresistive silicon pressure sensor which is interfaced with an Arduino ADC input. The sensor delivers an accurate high level analog output signal that is proportional to the applied pressure. We used the reference circuit shown in Figure 9.

picture showing the reference circuit of the design schematic for the IC MPXV7007G

Figure 9: Schematic for the Sip-Puff sensor

The Arduino must have the AsTeRICS CIM firmware flashed. This firmware transmits the ADC data to the AsTeRICS plugin of the ARE over the hardware UART, where the data can be processed further to control the helicopter. See the "3_channel_sip_puff.acs" model in the download package and the quickstart guide for details.

Project AsTeRICS

The AsTeRICS project is partially funded by the European Commission under the Seventh Framework Programme for Research and Technological Development (FP7 - 2007-2013). G.A.No. 247730

More information about the project can be found at http://www.asterics.eu