Advances in Change Point Detection
Achray Cyber Security's approach to AI is to teach the computer to deploy the best available tools from statistics to the cyber security threat landscape and to do so quickly, rigorously, and transparently. Key among those tools is Change Point Detection (CPD). Most statistical procedures aim to estimate a quantity such as an event rate or variability, usually at or after a given point in time. CPD cares only about finding a point in time; a point where one thing stopped, and something new started. By focusing on the "when" of something changing, CPD can find it in less data and therefore earlier than statistical procedures that fit complex models.
We were introduced to change point detection by Prof Bill Ziemba of the University of British Columbia. Its roots go back to Prof Abraham Wald, whose 1947 book "Sequential Analysis" is still in print (for good reason, it is exceptionally clearly written). Wald had discovered a procedure so powerful that his work was embargoed until well after the end of WW2. Research on the topic continues by people such as Prof Tartakovsky of the Moscow Institute of Physics and Technology. His latest volume of work is one influence feeding into the improvements Achray has made.
CPD it is only looking for one thing: a point in time. It needs less data than other statistical procedures but it can still benefit from our proprietary advances in statistics. To demonstrate, we generated 800,000 periods of stylized, synthetic event data in which we placed changes in the behaviour of a single observation. We compared how quickly the conventional statistical procedures identified the change and compared this to the procedures with our enhancements.
Conventional change point detection applied to 800,000 time periods of synthetic data. Red dots indicate when a change, shown on the horizontal plane, has not been detected in a given number of days. Time is given on the vertical axis. The bigger the dot, the less certain the procedure is of a change. Green dots indicate 90% or better confidence, the blue axis indicates the original process.
The red dots on the plots indicate the points where the procedures did not identify a change with 90% confidence in a given time. The green dots indicate correct identification at the 90% confidence level. (Technically we are measuring Type II errors, roughly speaking, missed changes are red, true positives, green.) The conventional approach rarely picks up a change in fewer than 5 event measurements.
Achray's proprietary statistical method is incorporated into change point detection applied to 800,000 time periods of synthetic data. Note the green dots, smaller indicating higher confidence and lower error, and that the procedure identifies changes much more quickly.
Our enhanced procedure is up to 10 times faster to identify a stastically significant change. That it respects statistical significance is key: that means the user can control the rate of false positives explicitly. This, in turn, means that the precious time of the best cyber security personnel is allocated judiciously.
AI practitioners might recognize our CPD algorithm as an advanced temporal version of a blind separation algorithm. Bayesian techniques figure heavily in the approach as well. That said, we emphasize that this is a tool for our AI to work with, not a "tuned" or "trained" procedure. It is completely transparent, and a product of analysis applied to data science. What's more, because we know exactly how and why this works, we know how and where to improve it.