FLAIRS 2019 Tutorial: How to Determine the Optimal Anomaly Detection Method For Your Application
We introduce a framework for automating the classification of time series and choice of anomaly detection method. The early detection of anomalies is vital for ensuring undisrupted business and efficient troubleshooting. Unfortunately, there is no one best way to detect all anomalies across a variety of domains; such a methodology is a myth given that time series can display a wide range of behaviors. In addition, what behavior is anomalous can differ from application to application. In this tutorial we introduce a framework that helps you determine the best anomaly detection method for your application based on the characteristics the time series possesses (e.g. seasonality, concept drift, etc.). Participants will get hands-on experience applying various anomaly detection methods to several example datasets. We will then discuss the results they observe and how best to evaluate them.
- FLAIRS 2019 Tutorial: How to Determine the Optimal Anomaly Detection Method For Your Application
This tutorial is hands-on and participants will be able to run the code and experiment with the datasets. In order to participate, a laptop with a current version of Windows, macOS, or Linux will be required. Participants without laptops will still be able to learn the concepts and methods from the presentation, but will miss out on the hands-on experience.
VirtualBox will be used for the exercises portion of the tutorial. Please install VirtualBox to run the virtual machine provided. It has also been tested to work directly in VMWare Fusion 8+, and may be imported into other virtualization software such as Hyper-V, but this is beyond the scope of the tutorial. You will need about 10GB of hard disk space free for VirtualBox and the VM image.
Slides from talk
VirtualBox installers and the following VM image will distributed on USB drives during the tutorial as well to prevent network bottlenecks.
Download the following:
Open VirtualBox and click Add
Select the FLAIRS2019Tutorial.vbox config file from where you saved it
Start the VM, the Jupyter notebook server and Firefox will automatically start up in the VM. For each exercise, navigate to the corresponding folder and click on the ipynb file inside to launch the exercise notebook.
During the introduction, attendees will be installing supplied VM image containing Jupyter notebooks and datasets to follow along. Help will be provided as needed to make sure everyone is able to use the materials.
(15 min) Who are the speakers? What is an anomaly and why is its detection important? What has anomaly detection been used in? Why is anomaly detection difficult?
Time Series Characteristics and How to Detect Them
(70 min) Define time series characteristics and how to detect them:
- Autocorrelation Plots
- Exercise 1: Autocorrelation Plots with Python (10 minutes of instruction + 5 minutes of Hands-On Time)
- Deterministic Trend vs Stochastic Trend
- Augmented Dickey-Fuller and Cox-Stuart Test
- Detrending and Differencing
- Exercise 2: Trend Detection with Python and R (10 minutes of instruction + 5 minutes of Hands-On Time)
- Concept Drift
- Exercise 3: Bayesian Online Change Point Detection with Python (10 minutes of instruction + 5 minutes of Hands-On Time)
- Missing Time Steps (Regularly and Irregularly Sampled)
- Exercise 4: Missing Time Steps with Python (10 minutes of instruction + 5 minutes of Hands-On Time)
(5min) Go over valuable resources for time series datasets
Break (30 min)
Anomaly Detection Methods
(80 min) Introduce and briefly discuss the following anomaly detection methods:
- Exercise 5: STLPLUS with R (10 minutes of instruction + 5 minutes of Hands-On Time)
- Exercise 6: SARIMA with Python (10 minutes of instruction + 5 minutes of Hands-On Time)
- Facebook Prophet
- Exercise 7: Prophet with Python (10 minutes of instruction + 5 minutes of Hands-On Time)
- Gaussian Processes
- Exercise 8: Gaussian Processes with Python (10 minutes of instruction + 5 minutes of Hands-On Time)
- Exercise 9: RNNs with Python (10 minutes of instruction + 5 minutes of Hands-On Time)
- Hierarchical Temporal Memory Networks (HTMs)
- HTM Studio
(5 min) Define several evaluation strategies:
- Numenta Anomaly Benchmark Scoring (NAB)
- Window Based F-Score
Results and Conclusion
(5 min) Give guidelines on what anomaly detection methods work better on certain characteristics than others.
- Characteristic Corpora
- Which methods are more promising given a characteristic?
- Which evaluation strategy should I use?
Open Question and Discussion (time permitting)
Cynthia Freeman1 - Research Engineer Verint Intelligent Self-Service
Bio: Cynthia Freeman is a Research Engineer at Verint, a developer of conversational AI systems. She is currently pursuing her PhD in Computer Science at the University of New Mexico, where she works on time series analysis and developing new anomaly detection methods. She holds an MS in Applied Mathematics from the University of Washington and a BS in Mathematics from Gonzaga University.
Ian Beaver, PhD2 - Lead Research Engineer Verint Intelligent Self-Service
Bio: Ian Beaver has been publishing discoveries in the field of AI since 2005 on topics surrounding human-computer interactions such as gesture recognition, user preference learning, and communication with multi-modal automated assistants. Ian has presented his work at various academic and industry conferences and authored numerous patents within the field of human language technology. Ian is currently focused on dialog systems and means to detect and resolve misunderstandings between humans and machines. He is also leading a team in finding ways to optimize human productivity by way of automation and augmentation, using symbiotic relationships with machines. Ian received his PhD in Computer Science from the University of New Mexico. He also holds a BS and MS in Computer Science from Eastern Washington University.