An extended version of a previous post, based on my presentation at the ISA ISSS-ISAC Joint Annual Conference 2013, Washington D.C. last week.
I first wondered what the implications would or could be about algorithmic interaction while attending the “Governing Algorithms” conference at NYU this past spring. The conference was a very interesting mix of presentations from many different fields, including computer science, the digital humanities, finance, and so on. In particular, an idea was suggested by Paul Dourish’s presentation, in which he offered the idea of “ecosystems of algorithms” for consideration. How would we map such an ecosystem? Algorithms are usually studied either individually (e.g.; the algorithm that determines whether or not you trade a particular stock) or vertically in combination with the programmer, data, software, hardware, network, and final purpose to which it is put. What would it mean to study these algos as they interact with each other and with data?
The premise of this working paper is that security studies can learn a great deal about cybersecurity by watching what happens in the financial sector. The whole-hearted embrace of algorithmic trading has precipitated several situations in which the security of either data, information, or systems has been compromised.
First, some definitions: security as it’s defined by different fields, and then algorithms. In “traditional” security and “human” security, security is defined along spectra in answer to four questions: security for whom? security from what? how severe is the threat? how fast is the threat? For example, traditional bombs-n-bullets security answers those questions this way: 1. for the territorial integrity of the sovereign state, 2. from invasion or attack, 3. killing a lot of the state’s citizens, 4. (usually) very suddenly. Human security, on the other hand, answers them thusly: 1. for the human being, 2. from physical harm to bodily integrity, such as rape, 3. may impact smaller numbers of people within a state, or larger numbers of people in a region, 4. may be low-grade but persistent, such as poverty.
This is very different from the field of financial investment/risk management, where the term “security” refers to a financial instrument that represents either an ownership or creditor position in relation to the issuing entity – in other words, a stock or bond – while it is the term “risk,” or a quantitative measure of the probability that an investment’s return will be lost or less than expected, that captures what the traditional and human security fields term “security.” However, the macro-level term for security in finance is “stability.” This is too often confused for stasis, or unchangingness, rather than a more accurate reading, which would reflect the connotations of homeostasis, or volatility within a well-defined range.
Information technology combines the categories of security studies (both traditional and human) with the clarity of the finance definition. It defines a security threat as “a person or event that has the potential for impacting a valuable resource in a negative manner,” a security vulnerability as the “quality of a resource or its environment that allows a threat to be realized,” and a security incident as unauthorized access or activity on a system, denial of service, non-trivial probing for an extended period of time, including damage caused by a virus or other malicious software. Risk assessment is conducted similarly to finance in order to identify vulnerabilities and opportunities for mitigation.
Algorithms are step-by-step problem-solving procedures, especially an established, recursive computational procedure for solving a problem in a finite number of steps. Algorithms are used in all aspects of life, whether or not these systems are automated. For example, figuring out whether or not you should eat something involves the following two-step process: 1. taste something 2. if it tastes bad, spit it out.
We can define an algorithm as a procedure which is “precise, unambiguous, mechanical, efficient, [and] correct,” with two components: a logic component specifying the relevant knowledge and a control component specifying the problem-solving strategy. “The manner in which the logic component is used to solve problems constitutes the control component” and can be made more or less efficient.
The classic formulation is “Algorithm = Logic + Control.” Andrew Goffey in Software Studies reminds us that the formula captures both its abstract nature as a set of instructions as well as an implemented entity embodied in a programming language for a particular machine architecture, with real effects on end users. Therefore, even though an algorithm can be modeled using mathematical notation, it is real in a way that an equation is not: “algorithms bear a crucial, if problematic, relationship to material reality.”
There are at least two problem with their use. First, algorithms ossify social relations at the moment they are incorporated into the algorithms’s equations/process – which does not reflect the dynamic nature of reality. As Zeynep Tufekci points out, big data pattern recognition requires using algos to pull recognizable patterns, and that only works if you know the pattern you’re looking for – by definition, it won’t be the rare event. Furthermore, algorithms are only as good as their assumptions! To sift through that much data, the algorithms will rely on the same shortcuts that the humans who write them do: stereotypes.
Which leads us to questions for the future. What happens when “cyber” and physical reality interact? Unless your systems are air-gapped (with a backup power source!), they will be interacting with each other. Therefore it’s not a question of which is “more” dangerous, because they act together. What are the security implications of the growing use of algorithms in automating all these fields? What are the implications for military communications, including command and control, as well as infrastructure and finance? Who has ultimate responsibility for these algorithms? Industry-specific situational awareness? Finance does NOT provide a great example of self-policing harmful systemic behavior or structure.
And finally, how will governing algorithms behave if/when they interact? An algorithm that runs on a really huge dynamic data set will not only find new (previously unknowable) patterns, but it may also produce data itself — on which other algorithms will run. It is difficult to map the possible networks of interaction even theoretically, to do so for networks of algorithms may be an “unknowable unknown.” Does it make sense to map algorithmic interaction as a two-mode network, in which we have the algorithms in one group, and they interact only with objects from another group? Or does it make more sense to map the interactions, and see what groupings emerge? The former might be more useful for understanding the theory, but the latter might be more useful for taking action. It would also be useful to closely examine the way biologists model epistasis (gene-gene interaction).
This is no longer a theoretical question. DoD algorithms may be interacting even more in the future: the plan is to make a “joint information environment” or JIE out of some 15,000 disparate networks in order to create a more secure system architecture that will not be as vulnerable to leakers. Such centralization would be to allow interaction between competing and incompatible algorithms “baked in” to the existing networks.
And that is without even considering the coming “internet of things”, which at least in the CIA’s view would be a heaven of total surveillance, according to then-Director David Petraeus. It is also not clear that human involvement in interaction would be a mitigating factor, if it’s even possible, given the timescales.
An example of algorithmic interaction is the AP Twitter Hack. In April of 2013, the Syrian Electronic Army hacked the feed of the the Associated Press’s Twitter account, sending a tweet saying the White House had been hit by two explosions and that President Obama was injured. Because many traders rely on machine-reading the news, the stock market crashed briefly before the AP could correct it. The “AP Twitter Hack,” as it became known, is the most important example because it demonstrates the INTERACTION of at least two different algorithms: the one(s) that the ETF(s) relies on to buy and sell stocks, the one that “reads” the AP Twitter feed, and the ones that govern whether or not to shut down trading on an exchange. (Possibly also the one that was used to crack the AP Twitter feed.) These algorithms are processed much faster than humans can react, and can interact with unforeseen consequences. Financial markets are growing used to this sort of thing, perhaps because the consequences there are (relatively) easily rectified: trading is shut down, trades are unwound, etc. What happens if the algorithms in question are the ones that control weapons systems? Or critical infrastructure? In the “internet of things,” all of these systems can interact, and an introduced deviation can have severe consequences.
With all this in mind, here are a few preliminary policy prescriptions. We need a culture of rule of law. Some call for a “centralized cyber policy.” However, this is a fool’s errand for several reasons. First, the technology changes too swiftly to even formulate (let alone enforce) a policy for an entity of any size. Forget the entire federal government, it would be impossible to enforce at just the NSA, with all its concomitant contractors. It’s not a policy that’s needed so much as a value system that promotes the rule of law.
And we have to learn to expect “normal accidents” as Charles Perrow warned almost 30 years ago. Algorithms are possibly the most tightly-coupled technology of all, because their processing time is not on a human scale, making their interactions seamless from our point of view. Resilience of components should be fostered, because ensuring the robustness of the entire network may not always be possible.
October 14th, 2013 4:54pm
annelies kamran 