Adversarial attacks on deep neural networks for time series classification
This is the companion web page for our paper titled "Adversarial attacks on deep neural networks for time series classification".
This paper has been presented at the IEEE International Joint Conference on Neural Networks (IJCNN19).
The source code
The software is developed using Python 3.5. We trained the models on a cluster of more than 60 GPUs. You will need the UCR archive to re-run the experiments of the paper. The source code can be downloaded here.
To run the code you will also need to download seperatly and install the following dependencies:
Pre-trained models
You can download the pre-trained models here.
The archive contains a folder for each dataset. In each of these folders you will find the model in a HDF5 format (best_model.hdf5 file).
This file contains both the architecture and the pre-trained weights which can be loaded using the function load_model in Keras.
A full tutorial on how to load models, weights and architectures using the Keras library is available here.
Adversarial time series for the UCR Archive
We provide a set of perturbed time series for each dataset's test set in the UCR archive.
This would allow time series data mining practitioners to test the robustness of their machine learning models against adversarial attacks.
The set of adversarial time series generated by the FGSM and the BIM attack can be found here.
The zip file contains two folders (one for each attack method).
For each dataset we provide a txt file that contains the perturbed time series as well as its corresponding true original label, thus preserving the same format as the original testing file.
Raw results
The folllowing table shows the accuracy over the 85 datasets with and without adversarial perturbation, using both attacks FGSM and BIM for two models ResNet (white-box mode) and FCN (black-box mode).
The raw csv results can be found here.
For example column 'resnet_ori' shows the original accuracy of ResNet over the 85 datasets, while column 'resnet_fgsm_adv' shows the accuracy after performing the FGSM attack.
| dataset_name | resnet_ori | resnet_fgsm_adv | resnet_bim_adv | fcn_ori | fcn_fgsm_adv | fcn_bim_adv |
|---|---|---|---|---|---|---|
| 50words | 73.2 | 17.1 | 8.8 | 45.5 | 7.7 | 8.8 |
| Adiac | 83.1 | 3.1 | 1.5 | 84.7 | 2.8 | 2.0 |
| ArrowHead | 85.1 | 33.1 | 14.3 | 82.3 | 41.7 | 29.1 |
| Beef | 76.7 | 20.0 | 10.0 | 70.0 | 26.7 | 36.7 |
| BeetleFly | 85.0 | 15.0 | 15.0 | 90.0 | 25.0 | 25.0 |
| BirdChicken | 95.0 | 55.0 | 15.0 | 100.0 | 60.0 | 45.0 |
| Car | 93.3 | 21.7 | 6.7 | 90.0 | 21.7 | 11.7 |
| CBF | 98.9 | 86.1 | 84.8 | 99.4 | 95.3 | 94.7 |
| ChlorineConcentration | 83.5 | 12.3 | 11.8 | 82.4 | 12.3 | 12.5 |
| CinC_ECG_torso | 83.8 | 25.4 | 23.3 | 83.8 | 25.7 | 23.6 |
| Coffee | 100.0 | 50.0 | 35.7 | 100.0 | 75.0 | 64.3 |
| Computers | 81.2 | 40.8 | 24.0 | 81.6 | 58.4 | 30.8 |
| Cricket_X | 79.0 | 35.4 | 20.8 | 79.5 | 43.8 | 34.1 |
| Cricket_Y | 80.5 | 24.9 | 13.8 | 76.7 | 28.5 | 20.8 |
| Cricket_Z | 81.5 | 27.7 | 16.2 | 80.3 | 35.4 | 26.2 |
| DiatomSizeReduction | 30.1 | 46.7 | 34.6 | 30.4 | 43.1 | 57.8 |
| DistalPhalanxOutlineAgeGroup | 79.8 | 16.0 | 17.0 | 82.8 | 16.8 | 17.5 |
| DistalPhalanxOutlineCorrect | 82.0 | 35.2 | 20.7 | 79.8 | 35.8 | 25.3 |
| DistalPhalanxTW | 74.8 | 9.8 | 12.5 | 75.8 | 11.2 | 12.2 |
| Earthquakes | 78.6 | 51.2 | 48.8 | 78.3 | 68.9 | 69.6 |
| ECG200 | 89.0 | 61.0 | 46.0 | 89.0 | 74.0 | 66.0 |
| ECG5000 | 93.5 | 76.1 | 36.4 | 93.9 | 90.0 | 88.0 |
| ECGFiveDays | 96.2 | 30.2 | 3.9 | 99.0 | 51.2 | 31.4 |
| ElectricDevices | 73.5 | 48.6 | 31.2 | 70.9 | 50.3 | 48.9 |
| FaceAll | 85.5 | 76.7 | 72.5 | 95.7 | 90.2 | 89.6 |
| FaceFour | 95.5 | 71.6 | 43.2 | 92.0 | 71.6 | 70.5 |
| FacesUCR | 95.3 | 79.4 | 76.1 | 94.7 | 86.4 | 85.9 |
| FISH | 97.7 | 12.6 | 4.0 | 96.0 | 12.6 | 9.7 |
| FordA | 91.8 | 33.9 | 21.6 | 90.1 | 59.6 | 57.3 |
| FordB | 91.1 | 27.8 | 14.3 | 88.2 | 70.0 | 67.7 |
| Gun_Point | 99.3 | 31.3 | 6.7 | 100.0 | 62.0 | 16.0 |
| Ham | 80.0 | 21.0 | 20.0 | 71.4 | 27.6 | 27.6 |
| HandOutlines | 86.0 | 36.2 | 36.2 | 74.6 | 36.2 | 36.2 |
| Haptics | 51.6 | 19.2 | 14.6 | 48.7 | 18.8 | 17.9 |
| Herring | 64.1 | 43.8 | 35.9 | 65.6 | 59.4 | 57.8 |
| InlineSkate | 37.8 | 14.9 | 12.5 | 32.4 | 9.6 | 11.1 |
| InsectWingbeatSound | 50.6 | 17.7 | 15.7 | 39.3 | 11.5 | 12.1 |
| ItalyPowerDemand | 95.9 | 92.5 | 91.6 | 96.1 | 89.8 | 89.6 |
| LargeKitchenAppliances | 90.4 | 74.7 | 65.3 | 89.6 | 66.4 | 63.5 |
| Lighting2 | 77.0 | 42.6 | 42.6 | 73.8 | 41.0 | 39.3 |
| Lighting7 | 78.1 | 50.7 | 35.6 | 80.8 | 57.5 | 54.8 |
| MALLAT | 96.6 | 33.0 | 4.6 | 97.0 | 32.6 | 24.2 |
| Meat | 98.3 | 35.0 | 1.7 | 81.7 | 1.7 | 31.7 |
| MedicalImages | 76.2 | 52.1 | 28.7 | 77.9 | 60.9 | 57.6 |
| MiddlePhalanxOutlineAgeGroup | 74.2 | 58.0 | 12.8 | 72.8 | 62.0 | 54.0 |
| MiddlePhalanxOutlineCorrect | 80.5 | 29.8 | 19.5 | 80.7 | 25.8 | 20.2 |
| MiddlePhalanxTW | 60.9 | 13.3 | 14.5 | 58.4 | 21.1 | 24.3 |
| MoteStrain | 92.4 | 74.3 | 68.8 | 93.4 | 80.5 | 77.4 |
| NonInvasiveFatalECG_Thorax1 | 94.6 | 5.5 | 2.4 | 95.6 | 7.4 | 5.1 |
| NonInvasiveFatalECG_Thorax2 | 94.4 | 5.2 | 1.2 | 95.6 | 4.4 | 1.6 |
| OliveOil | 86.7 | 20.0 | 3.3 | 86.7 | 13.3 | 13.3 |
| OSULeaf | 97.9 | 15.7 | 0.0 | 98.3 | 17.4 | 6.6 |
| PhalangesOutlinesCorrect | 85.7 | 36.8 | 16.2 | 81.5 | 35.9 | 24.9 |
| Phoneme | 33.3 | 15.0 | 10.3 | 32.1 | 21.0 | 15.5 |
| Plane | 100.0 | 81.0 | 56.2 | 100.0 | 58.1 | 56.2 |
| ProximalPhalanxOutlineAgeGroup | 83.9 | 46.3 | 8.3 | 81.5 | 46.8 | 9.8 |
| ProximalPhalanxOutlineCorrect | 91.4 | 32.0 | 10.7 | 91.1 | 35.7 | 20.6 |
| ProximalPhalanxTW | 77.8 | 10.2 | 11.8 | 81.0 | 15.0 | 13.0 |
| RefrigerationDevices | 51.7 | 32.0 | 30.1 | 50.7 | 38.4 | 40.0 |
| ScreenType | 60.8 | 31.7 | 25.9 | 60.8 | 36.5 | 28.0 |
| ShapeletSim | 100.0 | 53.9 | 36.1 | 75.6 | 60.0 | 58.3 |
| ShapesAll | 91.7 | 5.2 | 1.0 | 89.5 | 6.7 | 6.3 |
| SmallKitchenAppliances | 78.9 | 40.5 | 21.9 | 78.7 | 47.5 | 28.8 |
| SonyAIBORobotSurface | 96.8 | 83.9 | 82.2 | 96.0 | 85.0 | 84.2 |
| SonyAIBORobotSurfaceII | 98.6 | 89.2 | 88.7 | 98.1 | 91.5 | 91.6 |
| StarLightCurves | 97.2 | 58.8 | 57.7 | 96.6 | 73.0 | 60.1 |
| Strawberry | 96.2 | 21.9 | 3.8 | 95.8 | 14.4 | 13.7 |
| SwedishLeaf | 95.4 | 31.2 | 16.0 | 97.3 | 34.6 | 30.4 |
| Symbols | 92.7 | 36.6 | 12.9 | 94.3 | 58.4 | 28.3 |
| synthetic_control | 100.0 | 94.3 | 94.0 | 98.3 | 94.7 | 95.3 |
| ToeSegmentation1 | 96.9 | 62.3 | 46.9 | 96.1 | 63.2 | 57.5 |
| ToeSegmentation2 | 91.5 | 63.8 | 53.8 | 90.8 | 54.6 | 52.3 |
| Trace | 100.0 | 58.0 | 52.0 | 100.0 | 47.0 | 52.0 |
| TwoLeadECG | 100.0 | 5.3 | 0.4 | 100.0 | 13.0 | 5.2 |
| Two_Patterns | 100.0 | 98.2 | 96.7 | 86.8 | 82.9 | 82.6 |
| UWaveGestureLibraryAll | 86.2 | 21.8 | 7.1 | 81.7 | 25.3 | 22.3 |
| uWaveGestureLibrary_X | 78.0 | 32.1 | 11.1 | 75.7 | 32.7 | 27.2 |
| uWaveGestureLibrary_Y | 66.7 | 27.7 | 14.9 | 63.9 | 29.6 | 22.4 |
| uWaveGestureLibrary_Z | 75.0 | 37.0 | 14.0 | 72.0 | 27.1 | 21.0 |
| wafer | 99.8 | 86.6 | 7.3 | 99.7 | 64.3 | 81.2 |
| Wine | 61.1 | 38.9 | 38.9 | 55.6 | 38.9 | 38.9 |
| WordsSynonyms | 62.5 | 15.7 | 13.5 | 55.0 | 9.7 | 12.7 |
| Worms | 64.6 | 27.6 | 19.9 | 66.9 | 27.1 | 23.8 |
| WormsTwoClass | 74.6 | 45.3 | 31.5 | 74.6 | 55.8 | 44.8 |
| yoga | 87.2 | 45.4 | 12.8 | 84.1 | 44.9 | 19.2 |
Last update : March 2019