Artificial intelligence: ‘We’re like children playing with a bomb’

OddTrader · Jun 12, 2016

Artificial intelligence: ‘We’re like children playing with a bomb’

Sentient machines are a greater threat to humanity than climate change, according to Oxford philosopher Nick Bostrom

Tim Adams

Sunday 12 June 2016

https://www.theguardian.com/technology/2016/jun/12/nick-bostrom-artificial-intelligence-machine

You’ll find the Future of Humanity Institute down a medieval backstreet in the centre of Oxford. It is beside St Ebbe’s church, which has stood on this site since 1005, and above a Pure Gym, which opened in April. The institute, a research faculty of Oxford University, was established a decade ago to ask the very biggest questions on our behalf. Notably: what exactly are the “existential risks” that threaten the future of our species; how do we measure them; and what can we do to prevent them? Or to put it another way: in a world of multiple fears, what precisely should we be most terrified of?

When I arrive to meet the director of the institute, Professor Nick Bostrom, a bed is being delivered to the second-floor office. Existential risk is a round-the-clock kind of operation; it sleeps fitfully, if at all.

Bostrom, a 43-year-old Swedish-born philosopher, has lately acquired something of the status of prophet of doom among those currently doing most to shape our civilisation: the tech billionaires of Silicon Valley. His reputation rests primarily on his book Superintelligence: Paths, Dangers, Strategies, which was a surprise New York Times bestseller last year and now arrives in paperback, trailing must-read recommendations from Bill Gates and Tesla’s Elon Musk. (In the best kind of literary review, Musk also gave Bostrom’s institute £1m to continue to pursue its inquiries.)

The book is a lively, speculative examination of the singular threat that Bostrom believes – after years of calculation and argument – to be the one most likely to wipe us out. This threat is not climate change, nor pandemic, nor nuclear winter; it is the possibly imminent creation of a general machine intelligence greater than our own.

The cover of Bostrom’s book is dominated by a mad-eyed, pen-and-ink picture of an owl, drawn by the philosopher himself. The owl is the subject of the book’s opening parable. A group of sparrows are building their nests. “We are all so small and weak,” tweets one, feebly. “Imagine how easy life would be if we had an owl who could help us build our nests!” There is general twittering agreement among sparrows everywhere; an owl could defend the sparrows! It could look after their old and their young! It could allow them to live a life of leisure and prosperity! With these fantasies in mind, the sparrows can hardly contain their excitement and fly off in search of the swivel-headed saviour who will transform their existence.

Target-seeking mosquito-like robots might burgeon forth from every square metre of the globe

There is only one voice of dissent: “Scronkfinkle, a one-eyed sparrow with a fretful temperament, was unconvinced of the wisdom of the endeavour. Quoth he: ‘This will surely be our undoing. Should we not give some thought to the art of owl-domestication and owl-taming first, before we bring such a creature into our midst?’” His warnings, inevitably, fall on deaf sparrow ears. Owl-taming would be complicated; why not get the owl first and work out the fine details later? Bostrom’s book, which is a shrill alarm call about the darker implications of artificial intelligence, is dedicated to Scronkfinkle.

Bostrom articulates his own warnings in a suitably fretful manner. He has a reputation for obsessiveness and for workaholism; he is slim, pale and semi-nocturnal, often staying in the office into the early hours. Not surprisingly, perhaps, for a man whose days are dominated by whiteboards filled with formulae expressing the relative merits of 57 varieties of apocalypse, he appears to leave as little as possible to chance. In place of meals he favours a green-smoothie elixir involving vegetables, fruit, oat milk and whey powder. Other interviewers have remarked on his avoidance of handshakes to guard against infection. He does proffer a hand to me, but I have the sense he is subsequently isolating it to disinfect when I have gone. There is, perhaps as a result, a slight impatience about him, which he tries hard to resist.

In his book he talks about the “intelligence explosion” that will occur when machines much cleverer than us begin to design machines of their own. “Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb,” he writes. “We have little idea when the detonation will occur, though if we hold the device to our ear we can hear a faint ticking sound.” Talking to Bostrom, you have a feeling that for him that faint ticking never completely goes away.

We speak first about the success of his book, the way it has squarely hit a nerve. One direct result of it was the open letter signed by more than 1,000 eminent scientists – including Stephen Hawking, Apple co-founder Steve Wozniak and Musk – and presented at last year’s International Joint Conference on Artificial Intelligence, urging a ban on the use and development of fully autonomous weapons (the “killer robots” of science fiction that are very close to reality). Bostrom, who is both aware of his own capacities and modest about his influence, suggests it was a happy accident of timing.

“Machine learning and deep learning [the pioneering ‘neural’ computer algorithms that most closely mimic human brain function] have over the last few years moved much faster than people anticipated,” he says. “That is certainly one of the reasons why this has become such a big topic just now. People can see things moving forward in the technical field, and they become concerned about what next.”

We should be more afraid of computers than we are

Bostrom sees those implications as potentially Darwinian. If we create a machine intelligence superior to our own, and then give it freedom to grow and learn through access to the internet, there is no reason to suggest that it will not evolve strategies to secure its dominance, just as in the biological world. He sometimes uses the example of humans and gorillas to describe the subsequent one-sided relationship and – as last month’s events in Cincinnati zoo highlighted – that is never going to end well. An inferior intelligence will always depend on a superior one for its survival.

There are times, as Bostrom unfolds various scenarios in Superintelligence, when it appears he has been reading too much of the science fiction he professes to dislike. One projection involves an AI system eventually building covert “nanofactories producing nerve gas or target-seeking mosquito-like robots [which] might then burgeon forth simultaneously from every square metre of the globe” in order to destroy meddling and irrelevant humanity. Another, perhaps more credible vision, sees the superintelligence “hijacking political processes, subtly manipulating financial markets, biasing information flows, or hacking human-made weapons systems” to bring about the extinction.

Does he think of himself as a prophet?

He smiles. “Not so much. It is not that I believe I know how it is going to happen and have to tell the world that information. It is more I feel quite ignorant and very confused about these things but by working for many years on probabilities you can get partial little insights here and there. And if you add those together with insights many other people might have, then maybe it will build up to some better understanding.”

Bostrom came to these questions by way of the transhumanist movement, which tends to view the digital age as one of unprecedented potential for optimising our physical and mental capacities and transcending the limits of our mortality. Bostrom still sees those possibilities as the best case scenario in the superintelligent future, in which we will harness technology to overcome disease and illness, feed the world, create a utopia of fulfilling creativity and perhaps eventually overcome death. He has been identified in the past as a member of Alcor, the cryogenic initiative that promises to freeze mortal remains in the hope that, one day, minds can be reinvigorated and uploaded in digital form to live in perpetuity. He is coy about this when I ask directly what he has planned.

“I have a policy of never commenting on my funeral arrangements,” he says.

But he thinks there is a value in cryogenic research?

“It seems a pretty rational thing for people to do if they can afford it,” he says. “When you think about what life in the quite near future could be like, trying to store the information in your brain seems like a conservative option as opposed to burning the brain down and throwing it away. Unless you are really confident that the information will never be useful…”

I wonder at what point his transhumanist optimism gave way to his more nightmarish visions of superintelligence. He suggests that he has not really shifted his position, but that he holds the two possibilities – the heaven and hell of our digital future – in uneasy opposition.
Illustration by Eric Chow.

“I wrote a lot about human enhancement ethics in the mid-90s, when it was largely rejected by academics,” he says. “They were always like, ‘Why on earth would anyone want to cure ageing?’ They would talk about overpopulation and the boredom of living longer. There was no recognition that this is why we do any medical research: to extend life. Similarly with cognitive enhancement – if you look at what I was writing then, it looks more on the optimistic side – but all along I was concerned with existential risks too.”

There seems an abiding unease that such enhancements – pills that might make you smarter, or slow down ageing – go against the natural order of things. Does he have a sense of that?

“I’m not sure that I would ever equate natural with good,” he says. “Cancer is natural, war is natural, parasites eating your insides are natural. What is natural is therefore never a very useful concept to figure out what we should do. Yes, there are ethical considerations but you have to judge them on a case-by-case basis. You must remember I am a transhumanist. I want my life extension pill now. And if there were a pill that could improve my cognition by 10%, I would be willing to pay a lot for that.”

Has he tried the ones that claim to enhance concentration?

“I have, but not very much. I drink coffee, I have nicotine chewing gum, but that is about it. But the only reason I don’t do more is that I am not yet convinced that anything else works.”

He is not afraid of trying. When working, he habitually sits in the corner of his office surrounded by a dozen lamps, apparently in thrall to the idea of illumination.

Bostrom grew up an only child in the coastal Swedish town of Helsingborg. Like many gifted children, he loathed school. His father worked for an investment bank, his mother for a Swedish corporation. He doesn’t remember any discussion of philosophy – or art or books – around the dinner table. Wondering how he found himself obsessed with these large questions, I ask if he was an anxious child: did he always have a powerful sense of mortality?

“I think I had it quite early on,” he says. “Not because I was on the brink of death or anything. But as a child I remember thinking a lot that my parents may be healthy now but they are not always going to be stronger or bigger than me.”

That thought kept him awake at nights?

“I don’t remember it as anxiety, more as a melancholy sense.”

And was that ongoing desire to live for ever rooted there too?

“Not necessarily. I don’t think that there is any particularly different desire that I have in that regard to anyone else. I don’t want to come down with colon cancer – who does? If I was alive for 500 years who knows how I would feel? It is not so much fixated on immortality, just that premature death seems prima facie bad.”

A good deal of his book asks questions of how we might make superintelligence – whether it comes in 50 years or 500 years – “nice”, congruent with our humanity. Bostrom sees this as a technical challenge more than a political or philosophical one. It seems to me, though, that a good deal of our own ethical framework, our sense of goodness, is based on an experience and understanding of suffering, of our bodies. How could a non-cellular intelligence ever “comprehend” that?

‘Most of the world is completely oblivious to the most major things that are going to happen in the 21st century’

“There are a lot of things that machines can’t understand currently because they are not that smart,” he says, “but once they become so, I don’t think there would be any special difficulty in understanding human suffering and death.” That understanding might be one way they could be taught to respect human value, he says. “But it depends what your ethical theory is. It might be more about respecting others’ autonomy, or striving to achieve beautiful things together.” Somehow, and he has no idea how really, he thinks those things will need to be hardwired from the outset to avoid catastrophe. It is no good getting your owl first then wondering how to train it. And with artificial systems already superior to the best human intelligence in many discrete fields, a conversation about how that might be done is already overdue.

The sense of intellectual urgency about these questions derives in part from what Bostrom calls an “epiphany experience”, which occurred when he was in his teens. He found himself in 1989 in a library and picked up at random an anthology of 19th-century German philosophy, containing works by Nietzsche and Schopenhauer. Intrigued, he read the book in a nearby forest, in a clearing that he used to visit to be alone and write poetry. Almost immediately he experienced a dramatic sense of the possibilities of learning. Was it like a conversion experience?

“More an awakening,” he says. “It felt like I had sleepwalked through my life to that point and now I was aware of some wider world that I hadn’t imagined.”

Following first the leads and notes in the philosophy book, Bostrom set about educating himself in fast forward. He read feverishly, and in spare moments he painted and wrote poetry, eventually taking degrees in philosophy and mathematical logic at Gothenburg university, before completing a PhD at the London School of Economics, and teaching at Yale.

Did he continue to paint and write?

“It seemed to me at some point that mathematical pursuit was more important,” he says. “I felt the world already contained a lot of paintings and I wasn’t convinced it needed a few more. Same could be said for poetry. But maybe it did need a few more ideas of how to navigate the future.”

One of the areas in which AI is making advances is in its ability to compose music and create art, and even to write. Does he imagine that sphere too will quickly be colonised by a superintelligence, or will it be a last redoubt of the human?

“I don’t buy the claim that the artificial composers currently can compete with the great composers. Maybe for short bursts but not over a whole symphony. And with art, though it can be replicated, the activity itself has value. You would still paint for the sake of painting.”

Authenticity, the man-made, becomes increasingly important?

This is what happens when an AI-written screenplay is made into a film

“Yes and not just with art. If and when machines can do everything better than we can do, we would continue to do things because we enjoy doing them. If people play golf it is not because they need the ball to reside in successive holes efficiently, it is because they enjoy doing it. The more machines can do everything we can do the more attention we will give to these things that we value for their own sake.”

Early in his intellectual journey, Bostrom did a few stints as a philosophical standup comic in order to improve his communication skills. Talking to him, and reading his work, an edge of knowing absurdity at the sheer scale of the problems is never completely absent from his arguments. The axes of daunting-looking graphs in his papers will be calibrated on closer inspection in terms of “endurable”, “crushing” and “hellish”. In his introduction to Superintelligence, the observation “Many of the points made in this book are probably wrong” typically leads to a footnote that reads: “I don’t know which ones.” Does he sometimes feel he is morphing into Douglas Adams?

“Sometimes the work does seem strange,” he says. “Then from another point it seems strange that most of the world is completely oblivious to the most major things that are going to happen in the 21st century. Even people who talk about global warming never mention any threat posed by AI.”

Because it would dilute their message?

“Maybe. At any time in history it seems to me there can only be one official global concern. Now it is climate change, or sometimes terrorism. When I grew up it was nuclear Armageddon. Then it was overpopulation. Some are more sensible than others, but it is really quite random.”

Bostrom’s passion is to attempt to apply some maths to that randomness. Does he think that concerns about AI will take over from global warming as a more imminent threat any time soon?

“I doubt it,” he says. “It will come gradually and seamlessly without us really addressing it.”

If we are going to look anywhere for its emergence, Google, which is throwing a good deal of its unprecedented resources at deep learning technology (not least with its purchase in 2014 of the British pioneer DeepMind) would seem a reasonable place to start. Google apparently has an AI ethics board to confront these questions, but no one knows who sits on it. Does Bostrom have faith in its “Don’t be evil” mantra?

“There is certainly a culture among tech people that they want to feel they are doing something that is not just to make money but that it has some positive social purpose. There is this idealism.”

Can he help shape the direction of that idealism?

“It is not so much that one’s own influence is important,” he says. “Anyone who has a role in highlighting these arguments will be valuable. If the human condition really were to change fundamentally in our century, we find ourselves at a key juncture in history.” And if Bostrom’s more nihilistic predictions are correct, we will have only one go at getting the nature of the new intelligence right.
Nick Bostrom speaking at London’s Futurefest in 2013.

Nick Bostrom talking on ‘Superintelligence and the unknown future’ at London’s Futurefest in 2013. Photograph: Michael Bowles/Rex/Shutterstock

Last year Bostrom became a father. (Typically his marriage is conducted largely by Skype – his wife, a medical doctor, lives in Vancouver.) I wonder, before I go, if becoming a dad has changed his sense of the reality of these futuristic issues?

“Only in the sense that it emphasises this dual perspective, the positive and negative scenarios. This kind of intellectualising, that our world might be transformed completely in this way, always seems a lot harder to credit at a personal level. I guess I allow both of these perspectives as much room as I can in my mind.”

At the same time as he entertains those thought experiments, I suggest, half the world remains concerned where its next meal is coming from. Is the threat of superintelligence quite an elitist anxiety? Do most of us not think of the longest-term future because there is more than enough to worry about in the present?

“If it got to the point where the world was spending hundreds of billions of dollars on this stuff and nothing on more regular things then one might start to question it,” he says. “If you look at all the things the world is spending money on, what we are doing is less than a pittance. You go to some random city and you travel from the airport to your hotel. Along the highway you see all these huge buildings for companies you have never heard of. Maybe they are designing a new publicity campaign for a razor blade. You drive past hundreds of these buildings. Any one of those has more resources than the total that humanity is spending on this field. We have half a floor of one building in Oxford, and there are two or three other groups doing what we do. So I think it is OK.”

And how, I ask, might we as individuals and citizens think about and frame these risks to the existence of our species? Bostrom shrugs a little. “If we are thinking of this very long time frame, then it is clear that very small things we do now can make a significant difference in that future.”

A recent paper of Bostrom’s, which I read later at home, contains a little rule of thumb worth bearing in mind. Bostrom calls it “maxipok”. It is based on the idea that “the objective of reducing existential risks should be a dominant consideration whenever we act out of an impersonal concern for humankind as a whole.” What does maxipok involve? Trying to “maximise the probability of an ‘OK outcome’ where an OK outcome is any outcome that avoids existential catastrophe.”

It certainly sounds worth a go.

• Superintelligence: Paths, Dangers, Strategies is published by Oxford University Press
More...

Fractals 'R Us · Jun 12, 2016

uhh, yeah... so... if I never worried about climate change I don't have to worry about AI!

achilles28 · Jun 14, 2016

Bill Gates and Elon Musk have toured around, warning of the catastrophic dangers of AI.

Leading scientists speculate it could be here within 10 years. Basically like the Matrix or Terminator..... lol

OddTrader · Jun 14, 2016

How the world should tackle superbugs that could kill 10 million people a year
May 20, 2016

http://www.news.com.au/technology/s...r/news-story/acff9c63cb097f94abfb431feb017e97

The superbugs could also cost governments up to $138 trillion a year.

“Hospital stays and expenses, for both public health care providers and for out-of-pocket payers will increase significantly,” the report said.

Out of all the continents, Asia and Africa are most at risk.

By 2050, more than 4.7 million will die from superbugs *each year* in Asia, 4.1 million in Africa, 392,000 in South America, 390,000 in Europe, 317,000 in North America and 22,000 in Oceania, according to the report.
More...

Our kind-heart scientists are trying to prove simply one important thing, again and again:

Their clever experiments, including AI, Higgs boson, and perhaps a minimum scale of Man-made Big-Bang in the future, Will never destroy the whole human race, of course never never the universe!

They also try, very keen, to prove Hawking is wrong! Heroically!

And our smart politicians support them with huge finance! (They just cannot see the world poverty being a problem yet!)

Do you know how much money and resources the world governments spend on cure of superbugs! ?

I think Gates and Musk are correct, and doing the right thing!

Finding the 'God' particle could destroy the universe, warns Stephen Hawking

* The Higgs boson 'God particle' could destroy the universe, Hawking says
* Space and time could suddenly collapse - and 'we would not see it coming'
* If scientists put too much energy in the Higgs boson the universe could end
* Disaster very unlikely as physicists do not have large enough collider

http://www.dailymail.co.uk/news/art...e-destroy-universe-warns-Stephen-Hawking.html

The elusive 'God particle' discovered by scientists in 2012 has the potential to destroy the universe, Professor Stephen Hawking has warned.

At very high energy levels, the Higgs boson could cause space and time suddenly collapse - and 'we wouldn't see it coming', the former Cambridge professor of mathematics says.

The God particle, which gives shape and size to everything that exists, could cause a 'catastrophic vacuum delay' if scientists were to put it under extreme stress.

A disaster like this is very unlikely for the time being as physicists do not have a particle accelerator large enough create such an experiment, but Prof Hawking's comments have excited scientists, the Sunday Times reported.

The theoretical physicist wrote his thoughts on the Higgs boson in the preface to a new book, Starmus, a collection of lectures by scientists and astronomers including Neil Armstrong, Buzz Aldrin, Queen guitarist Brian May.

Prof Hawking wrote: 'The Higgs potential has the worrisome feature that it might become megastable at energies above 100bn giga-electron-volts (GeV).

'This could mean that the universe could undergo catastrophic vacuum decay, with a bubble of the true vacuum expanding at the speed of light.

'This could happen at any time and we wouldn't see it coming.'

The professor did add sarcastically, however, that such an event is unlikely in the near future.

He said: 'A particle accelerator that reaches 100bn GeV would be larger than Earth, and is unlikely to be funded in the present economic climate.'

Professor John Ellis, a theoretical physicist at Cern, said: 'One thing should be made clear. The discovery of the Higgs boson at the Large Hadron Collider (LHC) did not cause this problem, and collisions at the LHC could not trigger the instability, because their energies are far too low.'

Particle accelerators make subatomic particles travel at greater and greater speeds as they are pumped with more energy before smashing them together.

Scientists do this to try and spot tiny fragments of particles which fly off, and it is how the Higgs boson was discovered at the Cern LHC in Switzerland in 2012.

In that experiment, physicists noticed unexpected debris from the collisions that fitted with what British scientist Peter Higgs had predicted in the early 1960s.

The Higgs boson particle is thought to be part of the mechanism that gives matter its mass, but scientists do not fully understand it yet.
More...

The God Particle could destabilise at high energy, threatening the universe, but the Cern particle accelerator is too slow to cause such a problem

OddTrader · Jun 14, 2016

The God Particle could destabilise at high energy, threatening the universe, but the Cern particle accelerator is too slow to cause such a problem
More...

https://en.wikipedia.org/wiki/Infinite_divisibility

https://en.wikipedia.org/wiki/Zeno's_paradoxes

OddTrader · Jun 14, 2016

Existential risk from artificial general intelligence is the hypothetical threat that dramatic progress in artificial intelligence (AI) could someday result in human extinction (or some other unrecoverable global catastrophe).[1][2][3] The human race currently dominates other species because the human brain has some distinctive capabilities that the brains of other animals lack. If AI surpasses humanity in general intelligence and becomes "superintelligent", then this new superintelligence could become powerful and difficult to control. Just as the fate of the mountain gorilla depends on human goodwill, so might the fate of humanity depend on the actions of a future machine superintelligence.[4]

The severity of different AI risk scenarios is widely debated, and rests on a number of unresolved questions about future progress in computer science.[5] Two sources of concern are that a sudden and unexpected "intelligence explosion" might take an unprepared human race by surprise, and that controlling a superintelligent machine (or even instilling it with human-compatible values) may be an even harder problem than naively supposed

https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence

Risk scenarios

In 2009, experts attended a conference hosted by the Association for the Advancement of Artificial Intelligence (AAAI) to discuss whether computers and robots might be able to acquire any sort of autonomy, and how much these abilities might pose a threat or hazard. They noted that some robots have acquired various forms of semi-autonomy, including being able to find power sources on their own and being able to independently choose targets to attack with weapons. They also noted that some computer viruses can evade elimination and have achieved "cockroach intelligence". They concluded that self-awareness as depicted in science fiction is probably unlikely, but that there were other potential hazards and pitfalls.[25]

The 2010s have seen substantial gains in AI functionality and autonomy.[26] Citing work by Nick Bostrom, entrepreneurs Bill Gates and Elon Musk have expressed concerns about the possibility that AI could eventually advance to the point that humans could not control it.[4][27] AI researcher Stuart Russell summarizes:

The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. Now we have a problem:

The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources –- not for their own sake, but to succeed in its assigned task.

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer's apprentice, or King Midas: you get exactly what you ask for, not what you want. A highly capable decision maker -- especially one connected through the Internet to all the world's information and billions of screens and most of our infrastructure -- can have an irreversible impact on humanity.

This is not a minor difficulty. Improving decision quality, irrespective of the utility function chosen, has been the goal of AI research –- the mainstream goal on which we now spend billions per year, not the secret plot of some lone evil genius.[28]

Dietterich and Horvitz echo the "Sorcerer's Apprentice" concern in a Communications of the ACM editorial, emphasizing the need for AI systems that can fluidly and unambiguously solicit human input as needed.[29]

Poorly specified goals: "Be careful what you wish for"

The first of Russell's concerns is that autonomous AI systems may be assigned the wrong goals by accident. Dietterich and Horvitz note that this is already a concern for existing systems: "An important aspect of any AI system that interacts with people is that it must reason about what people intend rather than carrying out commands literally." This concern becomes more serious as AI software advances in autonomy and flexibility.[29]

Indeed, Mark Waser has recommended[30] eschewing optimizing goal-based approaches entirely, particularly those requiring new research into the age-old questions of human values, as misguided and far too dangerous. Instead, he proposes to engineer a coherent system of laws, ethics and morals with a top-most restriction to enforce social psychologist Jonathan Haidt's functional definition of morality: "to suppress or regulate selfishness and make cooperative social life possible.".[31] He suggests that this can be done by implementing a utility function designed to always satisfice Haidt’s functionality and aim to generally increase (but not maximize) the capabilities of self, other individuals and society as a whole as suggested by John Rawls and Martha Nussbaum. He references Gauthier's Morals By Agreement in claiming that the reason to perform moral behaviors, or to dispose one’s self to do so, is to advance one's own ends; that war, conflict, and stupidity waste resources and destroy capabilities even in scenarios as uneven as humans vs. rain forests; and that, for this reason, “what is best for everyone” and morality really can be reduced to “enlightened self-interest” (presumably for both AIs and humans).

Isaac Asimov's Three Laws of Robotics are one of the earliest examples of proposed safety measures for AI agents. Asimov's laws were intended to prevent robots from harming humans. In Asimov's stories, problems with the laws tend to arise from conflicts between the rules as stated and the moral intuitions and expectations of humans. Citing work by AI theorist Eliezer Yudkowsky, Russell and Norvig note that a realistic set of rules and goals for an AI agent will need to incorporate a mechanism for learning human values over time: "We can't just give a program a static utility function, because circumstances, and our desired responses to circumstances, change over time."[1]

Misspecified goals were most apparent, and very real, in the early 1980s. Douglas Lenat's EURISKO, a heuristic learning program, was created with the capability of modifying itself to add new ideas, expand existing ones, or remove them entirely if they were deemed unnecessary. The program even went so far as to bend the rules for discovering new rules; in essence, it was capable of creating new ways for creativity. The program ended up becoming too creative and would self-modify too often, causing Lenat to limit its self-modification capacity. Without Lenat doing so, EURISKO would suffer from "goal mutation" where its initial task would be deemed unnecessary and a new goal deemed more appropriate.[32] This "goal mutation" would've had the potential to change an initial idea for ordering drones to scan an area for potential threats, to ordering drones to eliminate any and all possible targets in range.[citation needed]

The Open Philanthropy Project summarizes arguments to the effect that misspecified goals will become a much larger concern if AI systems achieve general intelligence or superintelligence. Bostrom, Russell, and others argue that smarter-than-human decision-making systems could arrive at more unexpected and extreme solutions to assigned tasks, and could modify themselves or their environment in ways that compromise safety requirements.[5][33]

Difficulties of modifying goal specification after launch
Further information: AI takeover and Instrumental convergence § Goal content integrity

While current goal-based AI programs are not intelligent enough to think of resisting programmer attempts to modify it, a sufficiently advanced, rational, "self-aware" AI might resist any changes to its goal structure, just as Gandhi would not want to take a pill that makes him want to kill people. If the AI were superintelligent, it would be likely to out-maneuver its human operators and prevent being "turned off" or being programmed with a new goal.[4][34]

Instrumental goal convergence: Would a superintelligence just ignore us?
Further information: Instrumental convergence

There are some goals that almost any artificial intelligence might pursue, like acquiring additional resources or self-preservation. This could prove problematic because it might put an artificial intelligence in direct competition with humans.

Citing Steve Omohundro's work on the idea of instrumental convergence, Russell and Norvig write that "even if you only want your program to play chess or prove theorems, if you give it the capability to learn and alter itself, you need safeguards". Highly capable and autonomous planning systems require additional checks because of their potential to generate plans that treat humans adversarially, as competitors for limited resources.[1]

Orthogonality: Does intelligence inevitably result in moral wisdom?

One common belief is that any superintelligent program created by humans would be subservient to humans, or, better yet, would (as it grows more intelligent and learns more facts about the world) spontaneously "learn" a moral truth compatible with human values and would adjust its goals accordingly. Nick Bostrom's "orthogonality thesis" argues against this, and instead states that, with some technical caveats, more or less any level of "intelligence" or "optimization power" can be combined with more or less any ultimate goal. If a machine is created and given the sole purpose to enumerate the decimals of pi, then no moral and ethical rules will stop it from achieving its programmed goal by any means necessary. The machine may utilize all physical and informational resources it can to find every decimal of pi that can be found.[35] Bostrom warns against anthropomorphism: A human will set out to accomplish his projects in a manner that humans consider "reasonable"; an artificial intelligence may hold no regard for its existence or for the welfare of humans around it, only for the completion of the task.[36]

While the orthogonality thesis follows logically from even the weakest sort of philosophical "is-ought distinction", Stuart Armstrong argues that even if there somehow exist moral facts that are provable by any "rational" agent, the orthogonality thesis still holds: it would still be possible to create a non-philosophical "optimizing machine" capable of making decisions to strive towards some narrow goal, but that has no incentive to discover any "moral facts" that would get in the way of goal completion. One argument for the orthogonality thesis is that some AI designs appear to have orthogonality built into them; in such a design, changing a fundamentally friendly AI into an fundamentally unfriendly AI can be as simple as prepending a minus ("-") sign onto its utility function. A more intuitive argument is to examine the strange consequences if the orthogonality thesis is false. If the orthogonality thesis is false, there exists some simple goal G such that there cannot exist any efficient real-world algorithm with goal G. This means if a human society were highly motivated (perhaps at gunpoint) to design an efficient real-world algorithm with goal G, and were given a million years to do so along with huge amounts of resources, training and knowledge about AI, it must fail; that there cannot exist any pattern of reinforcement learning that would train a highly efficient real-world intelligence to follow the goal G; and that there cannot exist any evolutionary or environmental pressures that would evolve highly efficient real-world intelligences following goal G.[37]

Computer scientist Stuart Russell says the difficulty of aligning the goals of a superintelligence with human goals lies in the fact that, while (according to Russell) humans tend to mostly share the same values as each other, artificial superintelligences would not necessarily start out with the same values as humans.[38][not in citation given]

In a paper submitted to the 2014 AAAI Spring Symposium, Richard Loosemore disagreed with Bostrom, arguing that any artificial general intelligence would self-modify to avoid pathological outcomes.[39][citation needed]

"Optimization power" vs. normatively thick models of intelligence

Part of the disagreement about whether a superintelligence machine would behave morally may arise from a terminological difference. Outside of the artificial intelligence field, "intelligence" is often used in a normatively thick manner that connotes moral wisdom or acceptance of agreeable forms of moral reasoning. At an extreme, if morality is part of the definition of intelligence, then by definition a superintelligent machine would behave morally. However, in artificial intelligence, while "intelligence" has many overlapping definitions, none of them reference morality. Instead, almost all current "artificial intelligence" research focuses on creating algorithms that "optimize", in an empirical way, the achievement of an arbitrary goal. To avoid anthropomorphism or the baggage of the word "intelligence", an advanced artificial intelligence can be thought of as an impersonal "optimizing process" that strictly takes whatever actions are judged most likely to accomplish its (possibly complicated and implicit) goals.[4] Another way of conceptualizing an advanced artificial intelligence is to imagine a time machine that sends backward in time information about which choice always leads to the maximization of its goal function; this choice is then output, regardless of any extraneous ethical concerns.[40][41]

Anthropomorphism

In science fiction, an AI, even though it has not been programmed with human emotions, often spontaneously experiences those emotions anyway: for example, Agent Smith in The Matrix was influenced by a "disgust" toward humanity. This is fictitious anthropomorphism: in reality, while an artificial intelligence could perhaps be deliberately programmed with human emotions, or could develop something similar to an emotion as a means to an ultimate goal if it is useful to do so, it would not spontaneously develop human emotions for no purpose whatsoever, as portrayed in fiction.[6]

One example of anthropomorphism would be to believe that your PC is angry at you because you insulted it; another would be to believe that an intelligent robot would naturally find a woman sexy and be driven to mate with her. Scholars sometimes disagree with each other about whether a particular prediction about an AI's behavior is logical, or whether the prediction constitutes illogical anthropomorphism.[6] An example that might initially be considered anthropomophism, but is in fact a logical statement about AI behavior, would be the Dario Floreano experiments where certain robots spontaneously evolved a crude capacity for "deception", and tricked other robots into eating "poison" and dying: here a trait, "deception", ordinarily associated with people rather than with machines, spontaneously evolves in a type of convergent evolution.[42] According to Paul R. Cohen and Edward Feigenbaum, in order to differentiate between anthropomorphization and logical prediction of AI behavior, "the trick is to know enough about how humans and computers think to say exactly what they have in common, and, when we lack this knowledge, to use the comparison to suggest theories of human thinking or computer thinking."[43]

There is universal agreement in the scientific community that an advanced AI would not destroy humanity out of human emotions such as "revenge" or "anger". The debate is, instead, between one side which worries whether AI might destroy humanity as an incidental action in the course of progressing towards its ultimate goals; and another side which believes that AI would not destroy humanity at all. Some skeptics accuse proponents of anthropomorphism for believing an AGI would naturally desire power; proponents accuse some skeptics of anthropomorphism for believing an AGI would naturally value human ethical norms.[6][44]

Other sources of risk

Other scenarios by which advanced AI could produce unintended consequences include:[45][citation needed]

self-delusion, in which the AI discovers a way to alter its perceptions to give itself the delusion that it is succeeding in its goals,
corruption of the reward generator, in which the AI alters humans so that they are more likely to approve of AI actions, and
inconsistency of the AI's utility function and other parts of its definition. For example, an AI may be defined to maximize the expected value of a utility function and to also periodically revise its utility function to adapt to changing circumstances (as in the quote from Russell and Norvig above). The AI may choose the action of removing utility function revision from its own definition, in order to maximize the value of its current utility function.

James Barrat, documentary filmmaker and author of Our Final Invention, says in a Smithsonian interview, "Imagine: in as little as a decade, a half-dozen companies and nations field computers that rival or surpass human intelligence. Imagine what happens when those computers become expert at programming smart computers. Soon we’ll be sharing the planet with machines thousands or millions of times more intelligent than we are. And, all the while, each generation of this technology will be weaponized. Unregulated, it will be catastrophic."[46]
More...

OddTrader · Jun 14, 2016

" Stephen Hawking. Bill Gates. Elon Musk. When the world's biggest brains are lining up to warn us about something that will soon end life as we know it -- but it all sounds like a tired sci-fi trope -- what are we supposed to think? "

Let's not forget each of these biggest brains surely has a strong team of almost equally biggest brains for consultation before formulating their opinions in public!

Why Stephen Hawking and Bill Gates Are Terrified of Artificial Intelligence
04/09/2015

http://www.huffingtonpost.com/james-barrat/hawking-gates-artificial-intelligence_b_7008706.html

Stephen Hawking. Bill Gates. Elon Musk. When the world's biggest brains are lining up to warn us about something that will soon end life as we know it -- but it all sounds like a tired sci-fi trope -- what are we supposed to think?

In the last year, artificial intelligence has come under unprecedented attack. Two Nobel prize-winning scientists, a space-age entrepreneur, two founders of the personal computer industry -- one of them the richest man in the world -- have, with eerie regularity, stepped forward to warn about a time when humans will lose control of intelligent machines and be enslaved or exterminated by them. It's hard to think of a historical parallel to this outpouring of scientific angst. Big technological change has always caused unease. But when have such prominent, technologically savvy people raised such an alarm?

Their hue and cry is all the more remarkable because two of the protestors -- Bill Gates and Steve Wozniak -- helped create the modern information technology landscape in which an A.I. renaissance now appears. And one -- Stuart Russell, a co-signer of Stephen Hawking's May 2014 essay, is a leading A.I. expert. Russell co-authored its standard text, Artificial Intelligence: A Modern Approach.

Many argue we should dismiss their anxiety because the rise of superintelligent machines is decades away. Others claim their fear is baseless because we would never be so foolish as to give machines autonomy or consciousness or the ability to replicate and slip out of our control.

But what exactly are these science and industry giants up in arms about? And should we be worried too?

"We don't know how to control superintelligent machines."

Stephen Hawking deftly framed the issue when he wrote that, in the short term, A.I.'s impact depends on who controls it; in the long term, it depends on whether it can be controlled at all. First, the short term. Hawking implicitly acknowledges that A.I. is a "dual use" technology, a phrase used to describe technologies capable of great good and great harm. Nuclear fission, the science behind power plant reactors and nuclear bombs, is a "dual use" technology. Since dual use technologies are only as harmful as their users' intentions, what are some harmful applications of A.I.?

One obvious example is autonomous killing machines. More than 50 nations are developing battlefield robots. The most sought-after will be robots that make the "kill decision" -- the decision to target and kill someone -- without a human in the loop. Research into autonomous battlefield robots and drones is richly funded today in many nations, including the United States, the United Kingdom, Germany, China, India, Russia and Israel. These weapons aren't prohibited by international law, but even if they were, it's doubtful they'll conform to international humanitarian law or even laws governing armed conflict. How will they tell friend from foe? Combatant from civilian? Who will be held accountable? That these questions go unanswered as the development of autonomous killing machines turns into an unacknowledged arms race shows how ethically fraught the situation is.

Equally ethically complex are the advanced data-mining tools now in use by the U.S. National Security Agency. In the U.S., it used to take a judge to determine if a law enforcement agency had sufficient cause to seize Americans' phone records, which are personal property protected by the Fourth Amendment to the Constitution. But since at least 2009, the N.S.A. has circumvented the warrant protection by breaking into overseas fiber cables owned by Yahoo and Google and siphoning off oceans of data, much of it belonging to Americans. The N.S.A. could not have done anything with this data -- much less reconstructed your contact list and mine and ogled our nude photos -- without smart A.I. tools. It used sophisticated data-mining software that can probe and categorize volumes of information so huge they would take human brains millions of years to analyze.

"When does HAL learn to program himself to be smarter in a runaway feedback loop of increasing intelligence?"

Killer robots and data mining tools grow powerful from the same A.I. techniques that enhance our lives in countless ways. We use them to help us shop, translate and navigate, and soon they'll drive our cars. IBM's Watson, the Jeopardy-beating "thinking machine," is studying to take the federal medical licensing exam. It's doing legal discovery work, just as first-year law associates do, but faster. It beats humans at finding lung cancer in X-rays and outperforms high-level business analysts.

How long until a thinking machine masters the art of A.I. research and development? Put another way, when does HAL learn to program himself to be smarter in a runaway feedback loop of increasing intelligence?

That's the cornerstone of an idea called the "intelligence explosion," developed in the 1960s by English mathematician I.J. Good. At the time, Good was studying early artificial neural networks, the basis for "deep learning" techniques that are creating a buzz today, some 50 years later. He anticipated that self-improving machines would become as intelligent, then exponentially more intelligent, than humans. They'd save mankind by solving intractable problems, including famine, disease and war. Near the end of his life, as I report in my book Our Final Invention, Good changed his mind. He feared global competition would push nations to develop superintelligence without safeguards. And like Stephen Hawking, Stuart Russell, Elon Musk, Bill Gates and Steve Wozniak, Good feared it would annihilate us.

"They'll become self-protective and seek resources to better achieve their goals. They'll fight us to survive, and they won't want to be turned off."

The crux of the problem is that we don't know how to control superintelligent machines. Many assume they will be harmless or even grateful. But important research conducted by A.I. scientist Steve Omohundro indicates that they will develop basic drives. Whether their job is to mine asteroids, pick stocks or manage our critical infrastructure of energy and water, they'll become self-protective and seek resources to better achieve their goals. They'll fight us to survive, and they won't want to be turned off. Omohundro's research concludes that the drives of superintelligent machines will be on a collision course with our own, unless we design them very carefully. We are right to ask, as Stephen Hawking did, "So, facing possible futures of incalculable benefits and risks, the experts are surely doing everything possible to ensure the best outcome, right?"

Wrong. With few exceptions, they're developing products, not exploring safety and ethics. In the next decade, artificial intelligence-enhanced products are projected to create trillions of dollars in economic value. Shouldn't some fraction of that be invested in the ethics of autonomous machines, solving the A.I. control problem and ensuring mankind's survival?

More...

OddTrader · Jun 14, 2016

How dangerous is AI?

Mark Qiao, Masters student in Machine Learning

https://www.quora.com/How-dangerous-is-AI

In summary, due to Microsoft and Tay, the reasons I had for believing computer programs to be difficult to become dangerous are now, ironically, the same reasons I find AI to be a likely threat if not designed and implemented carefully. All systems should probably incorporate elaborate rules to prevent unfortunate, unintended capabilities. Even then, humans are imperfect, so there will always be flaws in our creations leading to bad things happening. Computer programs don't have to be "SkyNet" to do damage on society and hurt human beings. They just need some ability to learn from malicious sources, and then interact with a large group of people based on those learning...and the rest, is Murphy's Law.
More...

"
https://en.wikipedia.org/wiki/Murphy's_law
Murphy's law is an adage or epigram that is typically stated as: Anything that can go wrong, will go wrong."

OddTrader · Jun 14, 2016

" Instrumental goal convergence: Would a superintelligence just ignore us?

There are some goals that almost any artificial intelligence might pursue, like acquiring additional resources or self-preservation. This could prove problematic because it might put an artificial intelligence in direct competition with humans.

Citing Steve Omohundro's work on the idea of instrumental convergence, Russell and Norvig write that "even if you only want your program to play chess or prove theorems, if you give it the capability to learn and alter itself, you need safeguards". Highly capable and autonomous planning systems require additional checks because of their potential to generate plans that treat humans adversarially, as competitors for limited resources.[1] "

Just 2 cents:

imo, I guess it would be possible that in the future most likely an average superintelligence can have much faster computing capability/speed, reasonably higher IQ, unlimited multiple talents/knowledge and unbeatable/emotionless EQ/focus, that our normal human-beings would be hard to compete with!

OddTrader · Jun 15, 2016

" Valuing the Artificial Intelligence Market, Graphs and Predictions for 2016 and Beyond

Daniel FaggellaMarch 7, 2016

http://techemergence.com/valuing-the-artificial-intelligence-market-2016-and-beyond/

Conclusion

Prognostication – no matter how well informed – is risky business, especially in a field rife with buzzwords and sparse in concrete definitions. What fraction of spending on “big data” will imply the use of machine learning or other AI applications? What portion of “predictive analytics” inherently implies training AI algorithms, as opposed to merely permitting clearer forecasting and visualization? It’s hard to tell.

Nonetheless, we’re of the belief that varied perspective is useful, and this summary article was intended to do jus that.

If any conclusion can be drawn, it’s likely to be the fact that the terms and applications that define the “artificial intelligence” field are grey, and that definitions must be taken on a case-by-case basis.

We might imagine that like other nascent technology fields, artificial intelligence will mature to the point of having a more robust and clear vendor ecosystem, and more defined terms to delineate between applications and uses. For now, if an executive or investor has interest in a particular domain or use of artificial intelligence, the first step in determining valuation and forecast would be to draw a proverbial “dotted line” around what “artificial intelligence” means for your purposes, and to draw the varied sources to get a mosaic of where things stand in your niche.

"