Engineering Ethics

fh00od

responsibility.zip

Home >English homework help >Engineering Ethics

environmental+responsibility.pdf

Pg1.pdf
Pg2.pdf
Pg3.pdf
Pg4.pdf
Pg5.pdf
Pg6.pdf
Pg7.pdf
Pg8.pdf
Pg9.pdf
Pg10.pdf
Pg11.pdf

__MACOSX/._environmental+responsibility.pdf

Milton+Friedman+_+Social+Responsibility.ppt

Milton Friedman
& Social Responsibility
`The Social Responsibility Of Business Is To Increase Its Profits’

Friedman

“The businessmen believe that they are defending free enterprise when they declaim that business is not concerned `merely’ with profit but also with promoting desirable `social’ ends; that business has a `social conscience’ and takes seriously its responsibilities for providing employment, eliminating discrimination, avoiding pollution and whatever else may be the catchwords of the contemporary crop of reformers. In fact they are … preaching pure and unadulterated socialism.”

Fallacy Alert

At this point Friedman commits the fallacy of abusive ad hominem. … name calling instead of a rational argument.

Friedman’s argument

“What does it mean to say that `business’ has responsibilities? Only people can have responsibilities.’

A Businessperson’s Responsibility Is To The Owners Of The Corporation

Fiduciary responsibility to owners.
“ In a free -enterprise, private-property system, a corporate executive is an employee of the owners of the business. That responsibility is to conduct the business in accordance with their desires, which generally will be to make as much money as possible while conforming to the basic rules of the society, both those embodied in law and those embodied in ethical custom.”

Taxation without representation?

“Here the businessman … is to be simultaneously legislator, executive, and jurist. He is to decide whom to tax by how much and for what purpose, and he is to spend the proceeds – all this guided only by general exhortations on high to restrain inflation, improve the environment, fight poverty and so on and on.”

Huh?

“If they are to impose taxes and make expenditures to foster `social’ objectives, then political machinery must be set up to guide the assessment of taxes and to determine through a political process the objectives to be served.”
But Friedman does not object to the spending of money to influence the political system toward a less ethical direction.
If it is wrong for businesspeople to attempt to spend money to bring about positive social change, why isn’t it wrong to spend to either resist positive change or to promote negative change?

What about the taxation without representation involved in polluting the environment?

Environmental pollution costs us money.
A corporate choice to avoid the responsibility of cleaning up your own mess in order to make a profit places the costs on all of us. Only those who reap high profits benefit.
Isn’t it fundamentally unjust for those not reaping the benefits to carry the burden?

“The political principle that underlies the market mechanism is unanimity.”

“In an ideal free market resting on private property, no individual can coerce any other, all cooperation is voluntary, all parties to such cooperation benefit or they need not participate. There are no `social’ values, no `social responsibilities in any sense other than the shared values and responsibilities of individuals. Society is a collection of individuals and of the various groups they voluntarily form.”

No Coercion?????

Is ours an `ideal’ free market?
Is there really no coercion?
Do all parties in the market really benefit?
Do we really have the option of not participating?

Are we ever released from our social or ethical responsibility?

Hannah Arendt and her study of evil and responsibility
Eichmann in Jerusalem; A Report on the Banality of Evil

Trial of Adolph Eichmann

Eichmann became the head of the section of the Nazi SS section charged with the `Jewish Question’
Israeli agents tracked him down in Argentina and captured him in 1960.

Eichmann was taken to Jerusalem to stand trial for `Crimes Against Humanity”

Eichmann was found guilty and hanged in 1962.

Arendt analyzed the trial and the personality of Eichmann

She concluded that he was unable to think from another’s point of view.
Evil is not bigger than life. It is Banal.
In many ways, Eichmann was the guy next door.

My Conclusion

There is no occupation or context that releases us from our Ethical and Social Responsibilities.
`I was only following orders’, is not an excuse.
`I was just doing business’ is not an excuse.

__MACOSX/._Milton+Friedman+_+Social+Responsibility.ppt

reason%2C+relativity+_+responsibility+in+computer+ethics.pdf

Pg1
Pg2
Pg3
Pg4
Pg5
Pg6
Pg7
Pg8
Pg9
Pg10

__MACOSX/._reason%2C+relativity+_+responsibility+in+computer+ethics.pdf

THE+SOCIAL+RESPONSIBILITY+OF+BUSINESS+IS+TO+INCREASE+ITS+PROFITS.doc

THE SOCIAL RESPONSIBILITY OF BUSINESS IS TO INCREASE ITS PROFITS

By Milton Friedman

When I hear businessmen speak eloquently about the "social responsibilities of business in a free-enterprise system," I am reminded of the wonderful line about the Frenchman who discovered at the age of 70 that he had been speaking prose all his life. The businessmen believe that they are defending free enterprise when they declaim that business is not concerned "merely" with profit but also with promoting desirable "social" ends; that business has a "social conscience" and takes seriously its responsibilities for providing employment, eliminating discrimination, avoiding pollution and whatever else may be the catchwords of. the contemporary crop of re- formers. In fact they are-or would be if they or anyone else took them seriously-preaching pure and unadulterated socialism. Businessmen who talk this way are unwitting puppets of the intellectual forces that have been undermining the basis of a free society these past decades. The discussions of the "social responsibilities of business" are notable for their analytical looseness and lack of rigor. What does it mean to say that "business" has responsibilities? Only people can have responsibilities. A corporation is an artificial person and in this sense may have artificial responsibilities, but "business" as a whole cannot be said to have responsi- bilities, even in this vague sense. The first step toward clarity to examining the doctrine of the social responsibility of business is to ask precisely what it implies for whom. Presumably, the individuals who are to be responsible are busi- nessmen, which means individual proprietors or corporate executives. Most of the discussion of social responsibility is directed at corporations,.so in what follows I shall mostly neglect the individual proprietors and speak of corporate executives. In a free-enterprise, private-property system, a corporate executive is an employee of the owners of the business. He has direct responsibility to his employers. That responsibility is to conduct the business in accordance with their desires, which generally will be to make as much money as possi- ble while conforming to the basic rules of the society, both those embodied in law and those embodied in ethical custom. Of course, in some cases his employers may have a different objective. A group of persons might estab-

lish a corporation for an eleemosynary purpose-for example, a hospital or a school. The manager of such a corporation will not have money profit as his objectives but the rendering of certain services. In either case, the key point is that, in his capacity as a corporate executive, the manager is the agent of the individuals who own the corpo- ration or establish the eleemosynary institution, and his primary responsi- bility is to them. Needless to say, this does not mean that it is easy to judge how well he is performing his task. But at least the criterion of performance is straight- forward, and the persons among whom a voluntary contractual arrange- ment exists are clearly defined. Of course, the corporate executive is also a person in his own right. As a person, he may have many other responsibilities that he recognizes or assumes voluntary-to his family, his conscience, his feelings of charity, his church, his clubs, his city, his country. He may feel impelled by these responsibilities to devote part of his income to causes he regards as worthy, to refuse to work for particular corporations, even to leave his job, for example, to join his country's armed forces. If we wish, we may refer to some of these responsibilities as "social responsibilities." But in these re- spects he is acting as a principal, not an agent; he is spending his own money or time or energy, not the money of his employers or the time or energy he h as contracted to devote to their purposes. If these are "social responsibilities," they are the social responsibilities of individuals, not of business. What does it mean to say that the corporate executive has a "social responsibility" in his capacity as businessman? If this statement is not pure rhetoric, it must mean that he is to act in some way that is not in the interest of his employers. For example, that he is to refrain from increasing the price of the product in order to contribute to the social objective of preventing inflation, even though a price increase would be in the best interests of the corporation. Or that he is to make expenditures on reduc- ing pollution beyond the amount that is in the best interests of the corpora- tion or that is required by law in order to contribute to the social objective of improving the environment. Or that, at the expense of corporate profits, he is to hire "hardcore" unemployed instead of better qualified available workmen to contribute to the social objective of reducing poverty. In each of these cases, the corporate executive would be spending someone else's money for a general social interest. Insofar as his actions in accord with his "social responsibility" reduce returns to stockholders, he is spending their money. Insofar as his actions raise the price to customers, he is spending customers' money. Insofar as his actions lower the wages of some employees, he is spending their money. The stockholders or the customers or the employees could separately spend their own money on the particular action if they wished to do so. The executive is exercising a distinct "social responsibility," rather than

serving as an agent of the stockholders or the customers or the employ- ees, only if he spends the money in a different way than they would have spent it. But if he does this, he is in effect imposing taxes, on the one hand, and deciding how the tax proceeds shall be spent, on the other. This process raises political questions on two levels: principle and con- sequences. On the level of political principle, the imposition of taxes and the expenditure of tax proceeds are governmental functions. We have es- tablished elaborate constitutional, parliamentary and judicial provisions to control these functions, to assure that taxes are imposed so far as possibl e in accordance with the preferences and desires of the public-after all, "taxation without representation" was one of the battle cries of the Ameri- can Revolution. We have a system of checks and balances to separate the legislative function of imposing taxes and enacting expenditures from the executive function of collecting taxes and administering expenditure pro- grams and from the judicial function of mediating disputes and interpreting the law. Here the businessman-selfselected or appointed directly or indi- rectly by stockholders-is to be simultaneously legislator, executive and jurist. He is to decide whom to tax by how much and for what purpose, and he is to spend the proceeds-all this guided only by general exhortations from on high to restrain inflation, improve the environment, fight poverty and so on and on. The whole justification for permitting the corporate executive to be selected by the stockholders is that the executive is an agent serving the interests of his principal. This justification disappears when the corporate executive imposes taxes and spends the proceeds for "social" purposes. He becomes in effect a public employee, a civil servant, even though he re- mains in name an employee of a private enterprise. On grounds of political principle, it is intolerable that such civil servants-insofar as their actions in the name of social responsibility are real and not just window dressing- should be selected as they are now. If they are to be civil servants, then they must be elected through a political process. If they are to inil>ose taxes and make expenditures to foster "social" objectives, then political ma- chinery must be set up to make the assessment of taxes and to determine through a political process the objectives to be served. This is the basic reason why the doctrine of "social responsibility" involves the acceptance of the socialist view that political mechanisms, not market mechanisms, are the appropriate way to determine the allocation of scarce resources to alternative uses. On the grounds of consequences, can the corporate executive in fact discharge his alleged "social responsibilities"? On the one haiid, suppose he could get awav with spending the stockholders' or customers' or employees' money. How is lie to know how to spend it? He is told that he iiiust coiitrib- ute to fighting inflation. How is he to ktiow what action of his will cuittrib-

ute to that end? He is presumably an expert in running his company-in producing a product or selling it or financing it. But nothing about his selection makes him an expert on inflation. Will his holding down the price of his product reduce inflationary pressure? Or, by leaving more spending power in the hands of his customers, simply divert it elsewhere? Or, by forcing him to produce less because of the lower price, will it simply con- tribute to shortages? Even if he could answer these questions, how much cost is he justified in imposing on his stockholders, customers, and employ- ees for this social purpose? What is his appropriate share and what is the appropriate share of others? And, whether he wants to or not, can he get away with spending his stockholders', customers' or employees' money? Will not the stockholders fire him? (Either the present ones or those who take over when his actions in the name of social responsibility have reduced the corporation's profits and the price of its stock.) His customers and his employees can desert him for other producers and employers less scrupulous in exercising their social responsibilities. This facet of "social responsibility" doctrine is brought into sharp re- lief when the doctrine is used to justify wage restraint by trade unions. The conflict of interest is naked and clear when union officials are asked to subordinate the interest of their members to some more general purpose. If union officials try to enforce wage restraint, the consequence is likely to be wildcat strikes, rank-and-file revolts and the emergence of strong competi- tors for their jobs. We thus have the ironic phenomenon that union leaders-at least in the U.S.-have objected to Government interference with the market far more consistently and courageously than have business leaders. The difficulty of exercising "social responsibility" illustrates, of course, the great virtue of private competitive enterprise-it forces people to be responsible for their own actions and makes it difficult for them to "exploit" other people for either selfish or unselfish purposes. They can do good-but only at their own expense. Many a reader who has followed the argument this far may be tempted to remonstrate that it is all well and good to speak of Government's having the responsibility to impose taxes and determine expenditures for such "social" purposes as controlling pollution or training the hard-core unemployed, but that the problems are too urgent to wait on the slow course of political processes, that the exercise of social responsibility by businessmen is a quicker and surer way to solve pressing current problems. Aside from the question of fact-I share Adam Smith's skepticism about the benefits that can be expected from "those who affect to trade for the public good"-this argument must be rejected on grounds of principle. What it amounts to is an assertion that those who favor the taxes and expenditures in question have failed to persuade a majority of their fellow citizens to be of like mind and that they are seeking to attain by undem-

ocratic procedures what they cannot attain by democratic procedures. In a free society it is hard for "evil" people to do "evil," especially since one man's good is another's evil. I have, for simplicity, concentrated on the special case of the cor- porate executive, except only for the brief digression on trade unions. But precisely the same argument applies to the newer phenomenon of calling upon stockholders to require corporations to exercise social responsibility (the recent C.M. crusade for example). In most of these cases, what is in effect involved is some stockholders trying to get other stockholders (or customers or employees) to contribute against their will to "social" causes favored by the activists. Insofar as they succeed, they are again imposing taxes and spending the proceeds. The situation of the individual proprietor is somewhat different. If he acts to reduce the returns of his enterprise in order to exercise his "social responsibility," he is spending his own money, not someone else's. If he wishes to spend his money on such purposes, that is his right, and I cannot see that there is any objection to his doing so. In the process, he, too, may impose costs on employees and customers. 14owever, because he is far less likely than a large corporation or union to have monopolistic power, any such side effects will tend to be minor. Of course, in practice the doctrine of social responsibility is fre- quently a cloak for actions that are justified on other grounds rather than a reason for those actions. To illustrate, it may well be in the long-run interest of a corporation that is a major employer in a small community to devote resources to pro- viding amenities to that community or to improving its government. That may make it easier to attract desirable employees, it may reduce the wage bill or lessen losses from pilferage and sabotage or have other worthwhile effects. Or it may be that, given the laws about the deductibility of cor- porate charitable contributions, the stockholders can contribute more to charities they favor by having the corporation make the gift than by doing it themselves, since they can in that way contribute an amount that would- otherwise have been paid as corporate taxes. In each of these-and many sirnilar-cases, there is a strong tempta- tion to rationalize these actions as an exercise of "social responsibility." In the present climate of opinion, with its widespread aversion to "capitalism," 44 profits," and "soulless corporation" and so on, this is one way for a corpo- ration to generate goodwill as a by-product of expenditures that are entirely justified in its own self-interest. It would be inconsistent of me to call on corporate executives to re- frain from this hypocritical window-dressing because it harrns the founda- tions of a free society. That would be to call on them to exercise a "social responsibility"! If our institutions, and the attitudes of the public make it in their self-interest to cloak their actions in this way, I cannot summon much indignation to denounce them. At the same time, I can express admiration

for those individual proprietors or owners of closely held corporations or stockholders of more broadly held corporations who disdain such tactics as approaching fraud. Whether blameworthy or not, the use of the cloak of social responsi- bility, and the nonsense spoken in its name by influential and prestigious businessmen, does clearly harm the foundations of a free society. I have been impressed time and again by the schizophrenic character of many businessmen. They are capable of being extremely far-sighted and clearheaded in matters that are internal to their businesses. They are in- credibly short-sighted and muddle-headed in matters that are outside their businesses but affect the possible survival of business in general. This short- sightedness is strikingly exemplified in the calls from many businessmen for' wage and price guidelines or controls or income policies. There is nothing that could do more in a brief period- to destroy a market system and replace it by a centrally controlled system than effective governmental control of prices and wages. The short-sightedness is also exemplified in speeches by businessmen on social responsibility. This may gain them kudos in the short run. But it helps to strengthen the already too prevalent view that the pursuit of profits is wicked and immoral and must be curbed and controlled by external forces. Once this view is adopted, the external forces that curb the mar will not be the social consciences, however highly developed, of the pon- tificating executives; it will be the iron fist of Government bureaucrats. Here, as with price and wage controls, businessmen seem to me to reveal a suicidal impulse. The political principle that underlies the market mechanism is unani- mity. In an ideal free market resting on private property, no individual can coerce any other, all cooperation is voluntary, all parties to such coopera- tion benefit or they need not participate. There are no values, no "social" responsibilities in any sense other than the shared values and responsibili- ties of individuals. Society is a collection of individuals and of the various groups they voluntarily form. The political principle that underlies the political mechanism is con- formity. The individual must serve a more general social interest-whether that be determined by a church or a dictator or a majority. The individual may have a vote and say in what is to be done, but if he is overruled, he must conform. It is appropriate for some to require others to contribute to a general social purpose whether they wish to or not. Unfortunately, unanimity is not always feasible. There are some re- spects in which confomiity appears unavoidable, so I do not see how one can avoid the use of the political mechanism altogether. But the doctrine of "social responsibility" takei-i seriously would ex- tend the scope of the political i-nechanisni to every human activity. It does not differ in philosophy from the most explicitly collectivist doctrine. It differs onlv by professing to believe that collectivist ends can be attaiiied

without collectivist means. That is why, in my book Capitalism and Freedom, I have called it a "fundamentally subversive doctrine" in a free society, and I have said that in such a society, "there is one and only one social responsibility of business-to use its resources and engage in activi- ties designed to increase its profits so long as it stays within the rules of the game, which is to say, engages in open and free competition without de- ception or fraud." KENN-ETH J. ARRow

from others. It sets prices to its customers, and so enters into an economic relation with them. The firm typically sets working conditions, including- of greatest importance-conditions that affect the health and possibility for accident within the plant. We are reminded in recent years that the firm, as well as the private individual, is a contributor to pollution. Pollution has a direct effect on the welfare of other members of the economy. Less men- tioned, but of the same type, are the effects of economic activity on con- gestion. Bringing a new plant into an already crowded area is bound to create costs, disservices, and disutilities to others in the area if by nothing else than by crowding the streets and the sidewalks and imposing additional burdens on the public facilities of the area. Indeed, although congestion has not been discussed as much as has pollution, it may have greater eco- nomic impact and probably even greater health costs. Certainly the num- ber of automobile deaths arising from accidents far exceeds the health haz- ards arising from automobile pollution. The firm affects others through determining the quality of its products, and again, among the many aspects of product quality we may especially single out the qualities of the product with respect to its pollution-creating ability, as in the case of automobiles, and with respect to its safety, the hazards it poses to its user. The question of social responsibility takes very different forms with regard to the different items on this varied list. It is not a uniform characteristic at all. Let us first. consider the case against social responsibility: the assump- tion that the firms should airn simply to maximize their profits. One strand of that argument is empirical rather than ethical or normative. It simply states that firms will maximize their profits. The impulse to gain, it is argued, is very strong and the incentives for selfish behavior are so great that any kind of control is likely to be utterly ineffectual. This argument has some force but is by no means conclusive. Any mechanism for enforc- ing or urging social responsibility upon firms must of course reckon with a profit motive, with a desire to evade whatever response of controls are imposed. But it does not mean that we cannot expect any degree of respon- sibility at all. One finds a rather different argument, frequently stated by some economists. It will probably strike the noneconomist as rather strange, at least at first hearing. The assertion is that firms ought to maximize profits; not merely do they like to do so but there is practically a social obligation to do so. Let me briefly sketch the argument: Firms buy the goods and services they need for production. What they buy they pay for and therefore they are paying for whatever costs they impose upon others. What they receive in payment by selling their goods, they receive because the purchaser considers it worthwhile. This is a world of voluntary contracts; nobody has to buy the goods. If he chooses to buy it, it must be that he is getting a benefit measured by the price he pays. Hence, it is argued, profit really represents the net contribution that the firm makes to the social good, and the profits should therefore be made as

large as possible. When firms compete with each other, in selling their goods or in buying labor or other services, they may have to lower their selling prices in order to get more of the market for themselves or raise their wages; in either case the benefits which the firm is deriving are in some respects shared with the population at large. The forces of compe- tition prevent the firms from engrossing too large a share of the social benefit. For example, if a firm tries to reduce the quality of its goods, it will sooner or later have to lower the price which it charges because the pur- chaser will no longer find it worthwhile to pay the high price. Hence, the consumers will gain from price reduction at the same time as they are losing through quality deterioration. On detailed analysis it appears the firm will find it privately profitable to reduce quality under these circurn- stances only if, in fact, quality reduction is a net social benefit, that is, if the saving in cost is worth more to the consumer than the quality reduc- tion. Now, as far as it goes this argument is sound. The problem is that it may not go far enough. Under the proper assumptions profit maximization is indeed efficient in the sense that it can achieve as high a level of satisfaction as possible for any one consumer without reducing the levels of satisfaction of other con- sumers or using more resources than society is endowed with. But the limits of the argument must be stressed. I want to mention two well-known points in passing without making them the principal focus of discussion. First of all, the argument assumes that the forces of competition are suffi- ciently vigorous. But there is no social justification for profit maximization by monopolies. This is an important and well-known qualification. Second, the distribution of income that results from unrestrained profit maxirniza- tion is very unequal. The competitive maximizing economy is indeed ef- ficient-this shows up in high average incomes-but the high average is accompanied by widespread poverty on the one hand and vast riches, at least for a few, on the other. To many of us this is a very undesirable consequence. Profit maximization has yet another effect on society. It tends to point away from the expression of altruistic motives. Altruistic motives are mo- tives whose gratification is just as legitimate as selfish motives, and the expression of those motives is something we probably wish to encourage. A profit-maximizing, self-centered form of economic behavior does not pro- vide any room for the expression of such motives. If the three problems above were set aside, many of the ways by which -firms affect others should not be tampered with. Making profits by competition is, if anything, to be encouraged rather than discouraged. Wage and price bargains between the firm and uncoerced workers and customers represent mutually beneficial exchanges. There is, therefore, no reason within the framework of the discussion to interfere with them. But these examples far from exhaust the list of interactions with which we started. The social desirability of profit maximization does not extend to all

the interactions on the list. There are two categories of effects where the arguments for profit maximization break down: The first is illustrated by pollution or congestion. Here it is no longer true (and this is the key to these issues) that the firm in fact does pay for the harm it imposes on others. When it takes a person's time and uses it at work, the firm is paying for this, and therefore the transaction can be regarded as a beneficial ex- change from the point of view of both parties. We have no similar mecha- nism by which the pollution which a firm imposes upon its neighborhood is paid for. Therefore the firm will have a tendency to pollute more than is desirable. That is, the benefit to it or to its customers from the expanded activity is really not as great, or may not be as great, as the cost it is irnpos- ing upon the neighborhood. But since it does not pay that cost, there is no profit incentive to refrain. The same argument applies to traffic congestion when no change is made for the addition of cars. or trucks on the highway. It makes everybody less comfortable. It delays others and increases the probability of accidents; in short, it imposes a cost upon a large number of members of the society, a cost which is not paid for by the imposer of the cost, at least not in full. The person congesting is also congested, but the costs he is imposing on others are much greater than those he suffers himself. Therefore there will be a tendency to over-utilize those goods for which no price is charged, particularly scarce highway space. There are many other examples of this kind, but these two will serve to illustrate the point in question: some effort must be made to alter the profit-maximizing behavior of firms in those cases where it is imposing costs on others which are not easily compensated through an appropriate set of prices. The second category of effects where profit maximization is not so- cially desirable is that in which there are quality effects about which the firm knows more than the buyer. In my examples I will cite primarily the case of quality in the product sold, but actually very much the same con- siderations apply to the quality of working conditions. The firm is fre- quently in a better position to know the consequences (the health hazards, for example) involved in working conditions than the worker is, and the considerations I am about to discuss in the case of sale of goods have a direct parallel in the analysis of working conditions in the relation of a firm to its workers. Let me illustrate by considering the sale of a used car. (Simi- lar considerations apply to the sale of new cars.) A used car has potential defects and typically the seller knows more about the defects than the buyer. The buyer is not in a position to distinguish ai-nong used cars, and therefore lie will be willing to pay the same amouiit for two used cars of differing quality because he cannot tell the difference between them. As a result, there is aii inefficiency ii-i the sale of used cars. If somehow or other the cars were distinguished as to their quality, there would be some buyers who would prefer a cheaper car with iiiore defects because they intend to

use it very little or they only want it for a short period, while others will want a better car at a higher price. In fact, however, the two kinds of car are sold indiscriminately to the two groups of buyers at the same price, so that we can argue that there is a distinct loss of consumer satisfaction imposed by the failure to convey information that is available to the seller. The buyers are not necessarily being cheated. They may be, but the prob- lern of inefficiency would remain if they weren't. One can imagine a situa- tion where, from past experience, buyers of used cars are aware that cars that look alike may turn out to be quite different. Without knowing whether a particular car is good or bad, they do know that there are good and bad cars, and of course their willingness to pay for the cars is influ- enced accordingly. The main loser from a monetary viewpoint may not be the customer, but rather the seller of the good car. The buyer will pay a price which is only appropriate to a lottery that gives him a good car or a bad car with varying probabilities, and therefore the seller of the good car gets less than the value of the car. The seller of the bad car is, of course, the beneficiary. Clearly then, if one could arrange to transmit the truth from the sellers to the buyers, the efficiency of the market would be greatly improved. The used-car illustration is an example of a very general phe- nomenon. Consider now any newly produced complex product, such as a new automobile. The seller is bound to know considerably more about its properties than all but a very few of its buyers. In order to develop the car, the producer has had to perform tests of one kind or another. He knows the outcome of the tests. Failure to reveal this knowledge works against the efficiency of satisfying consumers' tastes. The argument of course applies to any aspect of the quality of a product, durability or the ability to perform under trying circumstances or differing climatic conditions. Perhaps we are most concerned about the safety features of the automobile. The risks in- volved in the use of automobiles are not trivial, and the kind of withholding of safety information which has been revealed to exist in a number of cases certainly cannot be defended as a socially useful implication of profit maxi- mization. The classical efficiency arguments for profit maximization do not apply here, and it is wrong to obfuscate the issue by invoking them. Perhaps even more dramatic, though on a smaller scale, are the re- peated examples of misleading information about the risks and use of pre- scription drugs and other chemicals. These again manifest the same point. Profit maximization can lead to consequences which are clearly socially injurious. This is the case if the buyers are on the average deceived-if, for example, they have higher expectations than are in fact warranted. They are also injured when on the average they are not deceived but merely uncertain, although here the argument is more subtle. One consequence may be the excessively limited use of some new drugs, for example. If the users of the drugs become fully aware of the risks involved but are not able to assess the risk with respect to any particular drug, the result may be an

indiscriminate rejection of new treatments which is rational from the point of view of the user; this, in the long run, may be just as serious an error as the opposite. Defenders of unrestricted profit maximization usually assume that the consumer is well informed or at least that he becomes so by his own experi- ence, in repeated purchases, or by information about what has happened to other people like him. This argument is empirically shaky; even the ability of individuals to analyze the effects of their own past purchases may be limited, particularly with respect to complicated mechanisms. But there are two further defects. The risks, including death, may be so great that even one misleading experience is bad enough, and the opportunity to learn from repeated trials is not of much use. Also, in a world where the products are continually changing, the possibility of learning from experi- ence is greatly reduced. Automobile companies are continually introducing new models which at least purport to differ from what they were in the past, though doubtless the change is more external than internal. New drugs are being introduced all the time; the fact that one has had bad experiences with one drug may provide very little information about the next one. Thus there are two types of situation in which the simple rule of maximizing profits is socially inefficient-. the case in which costs are not paid for, as in pollution, and the case in which the seller has considerably more knowledge about his product than the buyer, particularly with regard to safety. In these situations it is clearly desirable to have some idea of social responsibility, that is, to experience an obligation, whether ethical, moral, or legal. Now we cannot expect such an obligation to be created out of thin air. To be meaningful, any obligation of this kind, any feeling or rule of behavior has to be embodied in some definite social institution. I use that term broadly: a legal code is a social institution in a sense. Exhorta- tion to do good must be made specific in some external form, a steady reminder and perhaps enforcer of desirable values. Part of the need is simply for factual information as a guide to individual behavior. A firm may need to be told what is right and what is wrong when in fact it is polluting, or which safety requirements are reasonable and which are too extreme or too costly to be worth consideration. Institutionalization of the social re- sponsibility of firms also serves another very important function. It provides some assurance to any one firm that the firms with which it is in compe- tition will also accept the same responsibility. If a firm has some code im- posed from the outside, there is some expectation that other firms will obey it too and therefore there is some assurance that it need not fear any exces- sive cost to its good behavior. Let me then turn to some alternative kinds of institutions that can be considered as embodying the possible social responsibilities of firms. First, we have legal regulation, as in the case of pollution where laws are passed about the kind of burning that may take place, and about setting maximum

standards for emissions. A second category is that of taxes. Economists, with good reason, like to preach taxation as opposed to regulation. The movement to tax polluting emissions is getting under way and there is a fairly widely backed proposal in Congress to tax sulfur dioxide emissions from industrial smokestacks. That is an example of the second kind of institutionalization of social responsibility. The responsibility is made very clear: the violator pays for violations. A third very old remedy or institution is that of legal liability-the liability of the civil law. One can be sued for damages. Such cases appar- ently go back to the Middle Ages. Regulation also extends back very far. There was an ordinance in London about the year 1300 prohibiting the burning of coal, because of the smoke nuisance. The fourth class of institutions is represented by ethical codes. Re- straint is achieved not by appealing to each individual's conscience but rather by having some generally understood definition of appropriate be- havior. Let me discuss the advantages and disadvantages of these four insti- tutions. In regard to the first two, regulation and taxes, I shall be rather brief because these are the more familiar. We can have regulations governing pollution. We can also regulate product safety. We may even have stand- ards to insure quality in dimensions other than safety. The chief drawback of direct regulation is associated with the.fact that it is hard to make regula- tions flexible enough to meet a wide variety of situations and yet simple enough to be enforceable. In addition, there is a slowness in response to new situations. For example, if a new chemical, such as a pesticide, comes on the market and after a period of time is recognized as a danger, it requires a long and complicated process to get this awareness translated into legal action. One problem is that legislative time is a very scarce factor; a proposal to examine the problems involved in some pesticide may at any given time be competing with totally different considerations for the atten- tion of the legislature or regulatory body. In short, there is considerable rigidity in most regulatory structures. For certain purposes it is clear that regulation is best but it is equally clear that it is not useful as a universal device. In the case of taxes on the effects, rather than on the causes, there is a little more bugt-in flexibility. To combat pollution, taxation is probably the most appropriate device; a tax is imposed on the emission by the plant, whether in water or in air. Now this means the plant is free to find its own way of minimizing the tax burden. It is not tcild it must do one thing, such as raising smokestacks to a certain height. It is free to try to find the cheapest way of meeting the pollution problem. It may well decide that the profitabflity situation is such that it will continue to pollute and sell the product presumably at a somewhat higher price. This decision is not neces- sarily bad; it implies that the product is in fact much desired and it provides an automatic test of the market to see whether it is worth polluting or not, because in effect the consumer is ultimately paying for the pollution he

induces. However, it is difficult to see how this method, useful though it is in the case of pollution, would have any relevance to safety, to see how one could frame a tax which would make very much sense. Taxation appears to be a rather blunt instrument for controlling product safety. Legal liability can be and has been applied; i.e., courts have allowed damages in cases arising out of pollution or out of injury or death due to unsafe products. The nature of the law in this area is still evolving; under our system this means that it is being developed by a sequence of court decisions. just exactly what the company or its officers have to know before they can be regarded as liable for damages due to unsafe products is not yet clear. No doubt it would certainly be held even today that if officers of a company were aware that a product had a significant probability of a dan- gerous defect and they sold it anyway without saying so, and if the defect occurred, legal liability would be clear. But it is frequently hard to establish such knowledge. No doubt if society wants to use the route of legal liability as a way of imposing social responsibility, then it can change the principles on which the decision is based. For example, one might throw the burden of proof on the company, so that in the case of any new product they have to run tests to show positively that it is safe. Their failure to make such tests would be an indication of their liability. One could imagine changes of this kind which would bring the law more into line with what is desirable. But there are some intrinsic defects in the liability route which, in my opinion, make it unsuitable in its present form as a serious method of achieving social control or of imposing responsibility on profit-making firms. First, litigation is costly. In many cases there are social wrongs or social ineffi- ciencies which are quite significant in the aggregate and are perceived by a large number of people, each of whom bears a small part of the cost. This is characteristic of pollution and may be the case with certain kinds of quality standards. It really does not pay any particular person to sue, and if a few people do sue it does not really do the company much hann. Another problem is that the notion of liability in law is really too simple a concept. Legal liability tends to be an all-or-none proposition. Consider a product such as plastic bags. They are perfectly all right for storing clothes or food but there is a risk that small children will misuse them, with serious consequences. One would hardly want to say that there is any legal liability ascribable to the plastic-bag makers, for even the safest product can be misused. On the other hand, one might argue that a prod- uct that can be misused ought to be somewhat discouraged and perhaps some small degree of responsibility should be imposed, particularly if no adequate warning is issued. The law does not permit any such distinctions. Thus, in an automobile case, one party or the other must be found wrong, even though in fact a crash may clearly be due to the fact that both drivers were behaving erratically, and it would be reasonable to have some splitting of responsibility. At present, with some minor exceptions, the law does not permit this, and I suppose it would confuse legal proceedings irreparably to

start introducing partial causation. Economists are accustomed to the idea that almost nothing happens without the cooperation of a number of fac- tors, and we have large bodies of doctrine devoted to imputing in some appropriate way the consequences of an action to all of its causes. It is for these reasons that this kind of crude liability doctrine seems to be unsuit- able in many cases. A number of other problems with litigation could be mentioned. Con- sider very high-risk situations that involve a very low probability of death or other serious adverse consequence, as in the case of drugs, or possible radiation from nuclear power plants. The insurance companies are willing to insure because the probability is low. But once insurance is introduced the incentive to refrain from incurring the risk is dulled. If you are insured against a loss you have less of an incentive to prevent it. In the field of automobile liability, it has become clear that the whole system of liability has to a very great extent broken down. The result is a widespread move- ment toward no-fault insurance, which in effect means people are compen- sated for their losses but no attempt is made to charge damages to the persons responsible. Responsibility is left undecided. Finally, litigation does not seem suitable for continuing problems. Pollution will be reduced but not eliminated; indeed it is essentially impos- sible to eliminate it. There remain continuous steady damages to individ- uals. These should still be charged to firms in order to prevent them from polluting more. But enforcement by continuous court action is a very ex- pensive way of handling a repetitious situation. It is silly to keep on going to court to establish the same set of facts over and over again. For these reasons taxes which have the same incentive effects are superior. l,et me turn to the fourth possibility, ethical code. This may seem to be a strange possibility for an economist to raise. But when there is a wide difference in knowledge between the two sides of the market, recognized ethical codes can be, as has already been suggested, a great contribution to economic efficiency. Actually we do have examples of this in our everyday lives, but in very limited areas. The case of medical ethics is the most striking. By its very nature there is a very large difference in knowledge between the buyer and the seller. One is, in fact, buying precisely the service of someone with much more knowledge than you have. To make this relationship a viable one, ethical codes have grown up over the centu- ries, both to avoid the possibility of exploitation by the physician and to assure the buyer of medical services that he is not being exploited. I am not suggesting that these are universally obeyed, but there is a strong pre- sumption that the doctor is going to perform to a large extent with your welfare in mind. Unnecessary medical expenses or other abuses are per- ceived as violations of ethics. There is a powerful ethical background against which we make this judgment. Behavior that we would regard as highly reprehensible in a physician is judged less harshly when found among businessmen. The medical profession is typical of professions in

general. All professions involve a situation in which knowledge is unequal on two sides of the market by the very definition of the profession, and therefore there have grown up ethical principles that afford some protection to the client. Notice there is a mutual benefit in this. The fact is that if you had sufficient distrust of a doctor's services, you wouldn't buy them. Therefore the physician wants an ethical code to act as assurance to the buyer, and he certainly wants his competitors to obey this same code, partly because any violation may put him at a disadvantage but more espe- cially because the violation will reflect on him, since the buyer of the rnedi- cal services may not be able to distinguish one doctor from another. A close look reveals that a great deal of economic life depends for its viability on a certain limited degree of ethical commitment. Purely selfish behavior of individuals is really incompatible with any kind of settled economic life. There is almost invariably some element of trust and confidence. Much business is done on the basis of verbal assurance. It would be too elaborate to try to get written commitments on every possible point. Every contract depends for its observance on a mass of unspecified conditions which sug- gest that the performance will be carried out in good faith without insis- tence on sticking literally to its wording. To put the matter in its simplest form, in almost every economic transaction, in any exchange of goods for money, somebody gives up his valuable asset before he gets the other's, either the goods are given before the money or the money is given before the goods. Moreover there is a general confidence that there won't be any violation of the implicit agreement. Another example in daily life of this kind of ethics is the observance of queue discipline. People line up; there are people who try to break in ahead of you, but there is an ethic which holds that this is bad. It is clearly an ethic which is in everybody's interest to preserve; one waits at the end of the line this time, and one is protected against somebody's coming in ahead of him. In the context of product safety, efficiency would be greatly enhanced by accepted ethical rules. Sometimes it may be enough to have an ethical compulsion to reveal all the information available and let the buyer choose. This is not necessarily always the best. It can be argued that under some circumstances setting minimum safety standards and simply not putting out products that do not meet them would be desirable and should be felt by the businessman to be an obligation. Now I've said that ethical codes are desirable. It doesn't follow from that that they will come about. An ethical code is useful only if it is widely accepted. Its implications for specific behavior must be moderately clear, and above all it must be clearly perceived that the acceptance of these ethical obligations by everybody does involve mutual gain. Ethical codes that lack the latter property are unlikely to be viable. How do such codes develop? They may develop as a consensus out of lengthy public discussion of obligations, discussion which will take place in legislatures, lecture halls, business journals, and other public forums. The codes are communicated

by the very process of coming to an agreement. A more formal alternative would be to have some highly prestigious group discuss ethical codes for safety standards. In either case to become and to remain a part of the economic environment, the codes have to be accepted by the significant operating institutions and transmitted from one generation of executives to the next through standard operating procedures, through education in business schools, and through indoctrination of one kind or another. If we seriously expect such codes to develop and to be maintained, we might ask how the agreements develop and above all, how the codes remain stable. After all, an ethical code, however much it may be in the interest of all, is, as we remarked earlier, not in the interest of any one firm. The code may be of value to the running of the system as a whole, it may be of value to all firms if all firms maintain it, and yet it will be to the advantage of any one firm to cheat-in fact the more so, the more other firms are sticking to it. But there are some reasons for thinking that ethical codes can develop and be stable. These codes will not develop completely without institutional support. That is to say, there wfll be need for focal organizations, such as government agencies, trade associations, and consumer defense groups, or all combined to make the codes explicit, to iterate their doctrine and to make their presence felt. Given that help, I think the emergence of ethical codes on matters such as safety at least, is possible. One positive@ factor here is something that is a negative factor in other contexts, namely that our economic organization is to such a large extent composed of large firms. The corporation is no longer a single individual; it is a social organization with internal social ties and internal pressures for acceptability and esteem. The individual members of the corporation are not only parts of the corpo- ration but also members of a larger society whose esteem is desired. Power in a large corporation is necessarily diffused; not many individuals in such organizations feel so thoroughly identified with the corporation that other kinds of social pressures become irrelevant. Furthermore, in a large, com- plex firm where many people have to participate in any decision, there are likely to be some who are motivated to call attention to violations of the code. This kind of check has been conspicuous in government in recent years. The Pentagon Papers are an outstanding illustration of the fact that within the organization there are those who recognize moral guilt and take occasion to blow the whistle. I expect the same sort of behavior to occur in any large organization when there are well-defined ethical rules whose vio- lation can be observed. One can still ask if the codes are likely to be stable. Since it may well be possible and profitable for a minority to cheat, will it not be true that the whole system may break down? In fact, however, some of the pressures work in the other direction. It is clearly in the interest of those who are obeying the codes to enforce them, to call attention to violations, to use the ethical and social pressures of the society at large against their less scrupu- lous rivals. At the same time the value of maintaining the system may well

be apparent to all, and no doubt ways will be found to use the assurance of quality generated by the system as a positive asset in attracting consumers and workers. One must not expect miraculous transformations in human behavior. Ethical codes, if they are to be viable, should be limited in their scope. They are not a universal substitute for the weapons mentioned earlier, the institutions, taxes, regulations, and legal remedies. Further, we should ex- pect the codes to apply only in situations where the firm has superior knowledge of the situation. I would not want the firm to act in accordance with some ethical principles in regard to matters of which it has little knowledge. For example, with quality standards which consumers can ob- serve, it may not be desirable that the firm decide for itself, at least on ethical grounds, because it is depriving the consumer of the freedom of choice between high-quality, high-cost and low-quality, low-cost products. It is in areas where someone is typically misinformed or imperfectly in- formed that ethical codes can contribute to economic efficiency.

__MACOSX/._THE+SOCIAL+RESPONSIBILITY+OF+BUSINESS+IS+TO+INCREASE+ITS+PROFITS.doc

turing_s+response+ai.pdf

Turing’s Responses to Two Objections

Darren Abramson

Received: 11 March 2007 / Accepted: 22 January 2008 / Published online: 15 March 2008

� Springer Science+Business Media B.V. 2008

Abstract In this paper I argue that Turing’s responses to the mathematical objection

are straightforward, despite recent claims to the contrary. I then go on to show that by

understanding the importance of learning machines for Turing as related not to the

mathematical objection, but to Lady Lovelace’s objection, we can better understand

Turing’s response to Lady Lovelace’s objection. Finally, I argue that by understanding

Turing’s responses to these objections more clearly, we discover a hitherto unrec-

ognized, substantive thesis in his philosophical thinking about the nature of mind.

Keywords Alan Turing � Artificial intelligence � Creativity � Halting problem � Lady Lovelace’s objection � Mathematical objection � Turing Test

Introduction

In this paper, my major goal is to convince you of a few interpretive claims

concerning the philosophical work of Alan Turing. My minor goal is to convince

you that the claims Turing makes are philosophically interesting, and quite possibly

true. Before getting down to it, though, I will suggest some reasons why we ought to

care about getting Turing right on a few of the objections he considered to his claim

that, properly understood, yes, machines can think.

Turing has significance of a few kinds to cognitive scientists. He is one of the key

figures in the invention of the modern notion of the general purpose computer,

central to the cognitive revolution.1 Turing’s views on the possibility of machine

D. Abramson (&)

Department of Philosophy, Dalhousie University, Halifax, NS, Canada B3H 4P9

e-mail: [email protected]

1 For example, Jerry Fodor writes, ‘‘… the question arises, how a machine could be rational? …the great

logician Alan Turing proposed an answer to this question. It is, I think, the most important idea about how

123

Minds & Machines (2008) 18:147–167

DOI 10.1007/s11023-008-9094-6

intelligence have been inspiring to generations of practitioners in artificial

intelligence. His work in mathematical logic laid the foundation for recursion

theory. So, although it is fallacious to appeal to his authority on various matters in

justifying interest in an accurate depiction of his claims, many of us do care a great

deal about what he actually thought.

In my attempt to complete my minor goal, however, I will argue that the

interpretive difficulties I find obscure Turing’s thoughts on issues receiving

considerable contemporary attention; and, his thoughts play an interesting, and

significant role in these matters. In particular, I will argue that Turing’s comments

on the so-called ‘mathematical objection’ and ‘Lady Lovelace’s Objection’ (both of

which are defined and explained below) have been conflated in a way which

1. misunderstands the relative importance each had for Turing;

2. mistakenly attributes Turing’s discussion of ‘learning machines’ to his response

to the mathematical objection, instead of Lady Lovelace’s objection; and

3. obscures his philosophically interesting response to Lady Lovelace’s objection.

Before discussing objections, I will remind the reader briefly of the central

argument of Turing’s (1950) paper: the question ‘Can machines think?’, it is

clamied, vague to deserve serious attention. A better question is whether a general

purpose computer can pass what is now called the ‘Turing Test’. The Turing Test

gives a judge the task of determining, using only text-based interaction with a

human being and a computer, which is the human being and which is the computer.

A computer passes the Turing Test just in case the judge can successfully

distinguish the human and the computer no better than at chance (with 50%

success).

The mathematical objection and Lady Lovelace’s objection both assert that

machines cannot think. However, they make this claim without addressing any

particular details of the Turing Test, appealing to some more general reason which,

it is claimed, justifies the view that machines lack some necessary property for

thinking.

Turing’s Response to the Mathematical Objection

Turing considers what he eventually calls ‘the mathematical objection’ in a number

of places. In this section I will argue that the term actually refers to at least two

distinct objections, both of which rely on results in (what is now called) recursion

theory. I will discuss the occurrences chronologically, as they appeared in research

reports, public lectures, and published papers, some of which have only been made

widely accessible very recently.2

Footnote 1 continued

the mind works that anybody has ever had. Sometimes I think that it is the only important idea about how

the mind works that anybody has ever had’’ (Fodor 1992, p. 6). 2 Some of the articles discussed here appeared in print first in Furukawa et al. (1999), with

B. J. Copeland editing them, and then, collected with corrected versions of other materials in which

Turing discusses machine intelligence, in Copeland (2004).

148 D. Abramson

123

In his 1947 ‘Lecture on the Automatic Computing Engine,’ Turing writes,

It might be argued that there is a fundamental contradiction in the idea of a

machine with intelligence… It has for instance been shown that with certain

logical systems there can be no machine which will distinguish provable

formulae of the system from unprovable, i.e. that there is no test that the

machine can apply which will divide propositions with certainty into these two

classes. Thus if a machine is made for this purpose it must in some cases fail to give an answer. (Turing, 1947, 393, emphasis added)

So far, this is an argument against the idea of a machine with intelligence only if

human mathematicians do not lack the ability described here to sort provable sentences

for a logical system from unprovable ones. Turing is sympathetic to this idea.

On the other hand, if a mathematician is confronted with such a problem he

would search around and find new method of proof, so that he ought

eventually to be able to reach a decision about any given formula. This would

be the argument. (ibid., 393–394)

Turing has a counterresponse to the claim that human mathematicians have an

unlimited power that machines do not to solve problems.

Against it I would say that fair play must be given to the machine. Instead of it

sometimes giving no answer we could arrange that it gives occasional wrong

answers. But the human mathematician would likewise make blunders when

trying out new techniques. It is easy for us to regard these blunders as not

counting and give him another chance, but the machine would probably be

allowed no mercy. In other words then, if a machine is expected to be

infallible, it cannot also be intelligent. (ibid.)

Turing likely has in mind Gödel’s first incompleteness result, according to which

any enumerable, consistent axiomatic system rich enough to express elementary

arithmetic is incomplete, and his own proof of the unsolvability of the halting

problem. Richard Penrose, in his Shadows of the Mind, provides an argument

against the possibility of machine intelligence that does not appeal to the

incompleteness theorem explicitly, but is likely similar to the argument Turing is

thinking of. I will paraphrase Penrose’s presentation. Some enumeration of

computers is assumed, with subscripts referring to positions in this enumeration.

1. Let C be a computer which, when provided with input n, either prints out a

correct proof that Tn(n) (the nth computer provided with input n) does not halt,

or fails to halt.

2. Let c be the index of C in the enumeration of computers.

3. Assume C(c) halts. Then it produces a correct proof that C(c) does not halt. This

is contradictory, since we assumed that C(c) halts.

4. Therefore, C(c) does not halt.

5. Therefore, any intelligent person can see that C(c) cannot halt (since they can

follow this argument to this point), and C is unable to prove this [Compare

(Penrose 1994, pp. 74–77)].

Turing’s Responses to Two Objections 149

123

Of course, if we alter the assumption of the argument such that, when C is

finished its computation it does not necessarily have a complete, correct proof of its

output, then the conclusion cannot be drawn. We might imagine a computer that

incorporates heuristics that get assigned greater or poorer likelihoods of being used

according to their past success at generating results that cannot be disproved with

some fixed, finite application of resources.

Turing returns to the issue of whether Gödel’s theorem, or a related result by

Church or Turing rules out thinking machines in his 1948 National Physical

Laboratory document ‘Intelligent Machinery’ (Turing 1948). He rejects this tersely

in a single paragraph there, first, by noting that ‘‘The argument from Gödel’s and

other theorems … rests essentially on the condition that the machine must not make

mistakes. But this is not a requirement for intelligence’’ (ibid., p. 411). The

remainder of the paragraph recounts possible failed attempts that Gauss might have

taken as a child along his way to discovering a closed form for iterated sums of

natural numbers. He does not return to the topic of the mathematical objction in the

1948 document. In the rest of this paper, I refer to the argument against machines

thinking that incorporates a premise that thinking machines must be infallible, and

then appeals to an argument like the one above, as the first mathematical objection.

Now I will examine the mathematical objection that Turing addresses in his

influential 1950 paper. This objection, I will show, is distinct from the one discussed

in his 1947 lecture and in ‘Intelligent Machinery’. It is useful to quote the entire

section from the paper.

‘‘There are a number of results of mathematical logic which can be used to

show that there are limitations to the powers of discrete-state machines. The

best known of these results is known as Gödel’s theorem, and shows that in

any sufficiently powerful logical system statements can be formulated which

can neither be proved nor disproved with the system, unless possibly the

system is inconsistent. There are other, in some respects similar, results due to

Church, Kleene, Rosser and Turing. The latter result is the most convenient to

consider, since it refers directly to machines, whereas the others can only be

used in a comparatively indirect argument for instance if Gödel’s theorem is to

be used we need in addition to have some means of describing logical systems

in terms of machines, and machines in terms of logical systems. The result in

question refers to a type of machine which is essentially a digital computer

with an infinite capacity. It states that there are certain things that such a

machine cannot do. If it is rigged up to give answers to questions as in the

imitation game, there will be some questions to which it will either give a

wrong answer, or fail to give an answer at all however much time is allowed

for a reply. There may, of course, be many such questions, and questions

which cannot be answered by one machine may be satisfactorily answered by

another. We are of course supposing for the present that the questions are of

the kind to which an answer ‘Yes’ or ‘No’ is appropriate, rather than questions

such as ‘What do you think of Picasso?’ The questions that we know the

machines must fail on are of this type, ‘Consider the machine specified as

follows… Will this machine ever answer ‘Yes’ to any question?’ The dots are

150 D. Abramson

123

to be replaced by a description of some machine in standard form, which could

be something like that used in § 5. When the machine described bears a certain

comparatively simple relation to the machine which is under interrogation, it

can be shown that the answer is either wrong or not forthcoming. This is the

mathematical result: it is argued that it proves a disability of machines to

which the human intellect is not subject.’’ (Turing, 1950, 445)

Before examining Turing’s responses to this objection, note that it is clearly

different from the earlier mathematical objection. Instead of starting with the

assumption that a given machine is infallible and moving to the conclusion that the

machine knows less than a person, Turing begins with a different claim. Regardless

of any properties of the machine in the test, including fallibility, there is at least one

question that a person can ask of the machine, knowing in advance that the machine

will fail to give a correct answer to it.

Turing appears to have in mind something like the following:

1. Suppose, in the context of the Turing Test, I am checking to see if Turing

machine T47 can think.

2. Consider primitive recursive fd, such that fd(x) = the index of a machine that

outputs, on any input y,

yes if T47 (‘Does Tx ever answer ‘yes’?’) = no,

no if T47 (‘Does Tx ever answer ‘yes’?’) = yes,

and is undefined otherwise. (The machines that fd generates exhaustively

explore each computation step for each input to T47).

3. Now consider primitive recursive (and total) h(x) such that Th(x) computes the

same function as TTxðxÞ. The index of a machine that computes h can be found

constructively.

4. Define k to be the index of a computing machine that computes the function

fd � h. Again, k can be found constructively.

5. Notice that TTkðkÞ computes the same function as Th, by the definition of h.

6. Since TkðkÞ ¼ ðfd � hÞðkÞ; TTkðkÞ computes the same function as TfdðhðkÞÞ. 7. Therefore, Th(k) computes the same function as TfdðhðkÞÞ. 8. Since h is total and constructive, and k can be constructively determined based

on 47, we can constructively find h(k) = m, for some m. So, we have that Tm

computes the same function as TfdðmÞ. 3

9. Consider what happens when we ask T47, ‘Does Tm ever answer ‘yes’?’.

10. If T47 answers yes, then, for all y, TmðyÞ ¼ TfdðmÞðyÞ = no; and if T47 answers

no, then for all y, TmðyÞ ¼ TfdðmÞðyÞ = yes.

11. So, either T47 (‘Does Tm ever answer ‘yes’?’) is either wrong or undefined.

Therefore, as Turing claims, given any machine, I can come up with a number n such that the machine cannot correctly answer the question ‘will machine n answer

yes to any question’. Suppose, also, that we add the requirement that T47 only

answers questions in a manner consistent with arithmetic. Then we will know the

3 Notice that the generation of m involves applying a similar method to one used in a proof of what is

often called the recursion theorem for computability (see, for example, Rogers 1967, p. 180).

Turing’s Responses to Two Objections 151

123

answer to whether Tk ever answers ‘yes’: the answer is no. However, this last

observation is not mentioned by Turing, I argue, for at least two reasons.

First, if we suppose that the answers of the machine are consistent with

arithmetic, then we can make a stronger claim. The stronger claim is the conclusion

of the first mathematical objection, which, as we saw, Turing dismissed on the

grounds that we should not suppose that an intelligent machine is infallible.4

Second, we cannot, in general, check whether an arbitrary machine is infallible.

This can easily be seen by enumerating machines that are undefined except for a

response to a single instance of the special halting problem.

It must be insisted that Turing, in the 1950 paper, limits the mathematical

objection by ruling out any premise to the effect that the machine we are presented

with is known to produce answers consistent with arithmetic. He emphasizes the

claimed deficiency a number of times. Recall from above: ‘‘…there will be some

questions to which [the machine] will either give a wrong answer, or fail to give an

answer at all… When the machine described bears a certain comparatively simple

relation to the machine which is under interrogation, it can be shown that the answer

is either wrong or not forthcoming’’ (ibid., emphasis added).

Turing makes two responses to what I am calling the second mathematical objection.

1. The mathematical objection relies on the arrogant assumption that machines are

known to be fallible, while people are known to be infallible. Turing says,

however, ‘‘I do not think think [the mathematical objection] can be dismissed so

lightly’’ (ibid.).

2. The mathematical objection does not rule out that machines can think. It merely

signifies an intellectual victory over particular machines. Turing says this:

‘‘Whenever one of these machines is asked the appropriate critical question,

and gives a definite answer, we know that this answer must be wrong, and this

gives us a certain feeling of superiority … our superiority can only be felt on

such an occasion in relation to the one machine over which we have scored our

petty triumph. There would be no question of triumphing simultaneously over

all machines. In short, then, there might be men cleverer than any given

machine, but then again, there might be other machines cleverer again, and so

on.’’ (Turing, 1950, 450, emphasis added)

Given the above understanding of the second mathematical objection, and its

distinction from the first mathematical objection, these comments can be made

clear. When presented with a machine and its program, we can come up with a

question that it cannot answer correctly. However, we cannot, in general, know

whether the machine we are trying to outsmart will merely fail to answer the

question, instead of giving an incorrect answer (in which case we would be able to

answer a question that the machine could not).

4 Here, as in general (and as Turing did), by infallible, I mean infallible for questions that can be

expressed as theorems of first order arithmetic, as can statements of the halting of a particular Turing

machine on a particular input.

152 D. Abramson

123

An ability to triumph over all machines would only result if we could construct

questions which we knew before hand would be answered incorrectly by any

machine with which we are presented. But, because Turing rejects the premise that

any candidate machine for intelligence must be infallible, we cannot in general

know whether machines will fail to answer or give a wrong answer to our best

attempts to outsmart them. So, since we cannot outsmart every machine

simultaneously, we will only ever have outsmarted some finite set of machines.

However, there will always be some other machine that can answer each of the

questions we have generated to outsmart machines successfully. This follows from

the fact that the characteristic function of any finite set is primitive recursive. Then,

of course, there may be a person who has constructed an unanswerable question for

this machine, etc. ‘‘In short, then, there might be men cleverer than any given

machine, but then again there might be other machines cleverer again, and so on’’

(ibid.). Turing’s response amounts to showing that once we reject the premise that

only infallible machines could be intelligent, we lack a general method for

outsmarting any candidate machine we are presented with.

In a moment I will discuss other interpretations of Turing’s response to the two

mathematical objections. First I want to argue for a strength of my interpretation of

Turing. On this interpretation, Turing’s response anticipates a long history of

responses by logicians to the first mathematical objection. Stewart Shapiro

discusses, in great detail, various versions of the first mathematical objection, all

of which depend, in one way or another, on the premise that we can ascertain, with

mathematical certainty, that any candidate for a thinking machine can be proven to

have outputs consistent with arithmetic (Shapiro 2003). What he alternately calls the

‘‘Gödel–Kreisel–Benacerraf’’ maneuver (ibid., p. 39), analysis (ibid., p. 25), and

conclusion (ibid., p. 32), is just the claim that we cannot know, in general, and with

mathematical certainty, that candidate machines are infallible. So, on the

interpretation I am offering, Turing formulates and rejects a mathematical objection

to thinking machines in good (later) company, and in his only work on the matter

published in his lifetime, offers a weaker argument that omits the premise found

dubious by the community of logicians considering the objection.5

Another Interpretation

Now I will consider a different interpretation of Turing’s responses to the

mathematical objections. Jack Copeland (2004) and Gualtiero Piccinini (2003) have

argued that to understand Turing’s response, we must take into account other of

Turing’s writings, especially his writings on so-called ‘learning machines’.

Copeland’s argument is limited to textual analysis of Turing’s 1951 BBC Radio

Lecture entitled ‘Intelligent Machinery: A Heretical Theory’ (Turing 1951b) and

Turing’s letter to Newman (Turing 1940). Copeland writes:

5 Perhaps, then, it should be the ‘Turing–Gödel–Kreisel–Benacerraf maneuver/analysis/conclusion’.

Turing’s Responses to Two Objections 153

123

In ‘Intelligent Machinery, A Heretical Theory’ Turing passes immediately

from his remarks on the Mathematical Objection to a discussion of machine

learning. This juxtaposition perhaps indicates that Turing’s view was this it is

the possibility of a machine’s learning new methods and techniques that

ultimately defeats the Mathematical Objection… The learning machine

successively mutates from one proof-finding Turing machine into another,

becoming capable of wider sets of proofs as new, more powerful methods of

proof are acquired. (Copeland, 2004, 470)

Piccinini’s longer account is in agreement with Copeland on this analysis, and

finds evidence in yet other of Turing’s writings for understanding Turing’s response

to the mathematical objection in this manner. I now turn to examine some of that

claimed textual evidence.

Lessons from Ordinal Logics

In addition, Piccinini says we must understand Turing’s response to the mathemat-

ical objection in the context of Turing’s 1938 paper, ‘‘Systems of Logic Based on

Ordinals’’ (Turing 1939). In short, this early paper by Turing investigates the

possibility that Gödel incompleteness can be avoided by human mathematicians.

According to Piccinini,

Turing was far from stating that the mind is not a machine. Quite the contrary

he wanted to show that by using many formal systems, whose proofs could be

checked by as many machines, one could form stronger and stronger logical

systems that would allow one to prove more and more arithmetical theorems,

and the whole sequence of such logical systems would be complete (Piccinini

2003, p. 33).

Therefore, machines do not succumb to the mathematical objection because ‘‘[if]

the machine could modify its instruction tables by itself, without following a

uniform method, Turing saw no reason why it could not reach, and perhaps surpass,

the intelligence of human mathematicians’’ (ibid., p. 39).

‘Puzzling Comments’ by Turing

Recall that, in the 1947 lecture, Turing responds to the mathematical objection by

claiming that intelligence precludes infallibility: ‘‘In other words then, if a machine

is expected to be infallible, it cannot also be intelligent. There are several

mathematical theorems which say almost exactly that’’ (Turing 1947, p. 394).

Piccinini’s interpretation of this comment, which he claims is ‘‘often cited but

never clearly explained…’’:

…has to do with the unsolvability result proved by Turing… If … the

machine were allowed to give ‘the wrong answer,’ viz. an output that is not the

correct answer to the original question, then there was no limit to what the

machine could ‘learn’ by changing its instruction tables. In principle, like a

154 D. Abramson

123

human mathematician with an ordinal logic, it could reach mathematical

completeness. (Piccinini, 2003, 38)

As we have seen, the mathematical objection, in its standard form, has as its

premise that only machines that are infallible (for statements of mathematics) can be

candidates for intelligence. This premise is crucial to drawing the conclusion that

human beings are superior to machines. Turing’s comment reflects his understand-

ing of the first mathematical objection, and, far from inexplicable, it is a reflection

of a response of generations of logicians to the standard mathematical objection.

An Initial Problem with this Interpretation

In his 1950 paper, Turing is very clear in his thesis: machines can think. Each word

other than ‘can’ here receives a special definition. Here is what he thinks we should

replace the question ‘can machines think?’ by.

Let us fix our attention on one particular digital computer C. Is it true that by

modifying this computer to have an adequate storage, suitably increasing its

speed of action, and providing it with an appropriate programme, C can be

made to play satisfactorily the part of A in the imitation game, the part of B

being taken by a man? (Turing, 1950)

If Turing’s answer to the mathematical objection is that machines that follow a

non-uniform method can ‘be cleverer’ than some humans, and can therefore survive

it, he has given away his thesis. Another problem with Piccinini’s account is that it

ignores the possibility that Turing had in mind the relatively isolable responses I

have given, relying either on the claim that machines need not be infallible, or

showing that our apparent triumph over machines is fleeting. The Piccinini/

Copeland interpretation may have the benefit of generating philosophical continuity

both between Turing’s earlier and later work, and his apparent early views on the

mathematical ability of humans, on the one hand, and machines on the other, but the

cost is simply too high.

Turing’s Response to the Earlier Version

In understanding how Copeland and Piccinini go wrong, it is useful to look again at

Turing’s response to the first version of the mathematical objection in his 1947

lecture.

To continue my plea for ‘fair play for the machines’ when testing their I.Q. A

human mathematician has always undergone an extensive training. This

training may be regarded as not unlike putting instruction tables into a machine.

One must therefore not expect a machine to do a very great deal of building up

of instruction tables on its own. No man adds very much to the body of

knowledge, why should we expect more of a machine? Putting the same point

differently, the machine must be allowed to have contact with human beings in

order that it may adapt itself to their standards. (Turing, 1947, 394)

Turing’s Responses to Two Objections 155

123

Here, Turing attempts to identify circumstances in which one would be forced to

allow ‘errors of performance’, as opposed to ‘errors of competence’, in an equal

fashion for machines and humans. Such a circumstance is one in which the machine’s

errors are just like human errors. In other places Turing argues at great length that, to

build thinking machines, it is essential that they not be programmed in a manner that

contains explicit instructions on how to behave in given circumstances, but instead

be taught in some fashion. I discuss these comments in detail below.

Before investigating other reasons for building learning machines, let me

reiterate at this point that Copeland and Piccinini miss Turing’s straightforward

responses to the mathematical objections. Turing’s responses to these objections are

connected to his claim that learning machines can think only insofar as teaching

them instead of programming them might be a way for machines to imitate our

patterns of mistakes. I will argue below that Turing’s main motivation in identifying

intelligent machines with learning machines, is, in contrast to the Piccinini/

Copeland interpretation, his consideration of Lady Lovelace’s objection.

An Old Misunderstanding

Copeland and Piccinini are not alone in misinterpreting Turing. John Lucas,

responsible for a great deal of the discussion of the mathematical objection in the

last four decades, misunderstands the distinction between the two mathematical

objections that Turing considers. Lucas clearly believes that, in his Mind paper,

Turing is dealing with the first mathematical objection.

[Turing] argues that the limitation to the powers of a machine do not amount

to anything much. Although each individual machine is incapable of getting

the right answer to some questions, after all each individual human being is

fallible also and in any case ‘‘our superiority can only be felt on such an

occasion in relation to the one machine over which we have scored our petty

triumph. There would be no question of triumphing simultaneously over all

machines.’’ But this is not the point. We are not discussing whether machines

or minds are superior, but whether they are the same. In some respect

machines are undoubtedly superior to human minds; and the question on

which they are stumped is admittedly, a rather niggling, even trivial, question.

But it is enough, enough to show that the machine is not the same as a mind.

True, the machine can do many things that a human mind cannot do: but if

there is of necessity something that the machine cannot do, though the mind can, then, however trivial the matter is, we cannot equate the two, and cannot

hope ever to have a mechanical model that will adequately represent the mind.

(Lucas, 1961, 117–118, emphasis added)

Recall from above our discussion of T47. We ask it, ‘Does machine Tm ever

answer ‘yes’ to any question?’ knowing that the machine will either fail to answer,

or give a wrong answer. Suppose we have waited a few minutes for an answer. We

then must decide whether to continue waiting. And, if we do decide to wait, we must

reconcile ourselves with the fact that we do not have a general method for all

156 D. Abramson

123

machines that will tell us whether an answer is forthcoming. Once again, on the

assumption that we had such a method, we would also have a solution to the special

halting problem. So, supposing us to be in the position of deciding whether to wait

for an answer that may never come, we cannot claim that we can do something the

machine cannot do, namely, correctly answer the question ‘Does machine Tm ever

answer ‘yes’ to any question?’.

In other words, the second mathematical objection, which Turing felt was the

strongest objection one could have to the claim that machines can think since it did

not require a dubious premise, does not have as its conclusion that there is, ‘of

necessity, something that the machine cannot do, though the mind can’. It only

shows that there is something that the machine cannot do, that any particular mind

may not be able to do either. In this light, Lucas clearly misunderstands Turing’s

discussion of the mathematical objection. In his defence, though, Lucas spends a

great deal of energy defending the extra assumption with which we can pass from

the second mathematical objection to the first.6

Lady Lovelace’s Objection

Definition and First Response

An objection that occupies Turing at great length in his 1950 paper is Lady

Lovelace’s objection: ‘‘The Analytical Engine has no pretensions to originate anything. It can do whatever we know how to order it to perform’’ [from Lovelace’s

memoir, quoted by Turing (1950, p. 450, emphasis in original)].

Turing’s first response is that computers surprise him all the time, not only due to

carelessness when programming, but also due to unforeseen consequences of the

progam that one writes for a computer to run. Turing outlines a possible source of

this objection.

‘‘The view that machines cannot give rise to surprises is due, I believe, to a

fallacy to which philosophers and mathematicians are particularly subject.

This is the assumption that as soon as a fact is presented to a mind all

consequences of that fact spring into the mind simultaneously with it. It is a

very useful assumption under many circumstances, but one too easily forgets

that it is false. A natural consequence of doing so is that one then assumes that

there is no virtue in the mere working out of consequences from data and

general principles.’’ (ibid., 451)

Although this paragraph concludes the section in which Turing introduces Lady

Lovelace’s objection, it does not signify the end of his discussion of the objection.

He returns to Lady Lovelace’s objection, to the exclusion of the other eight

objections he considers, in the final, lengthy section of the paper. For now, note that,

in this paragraph, Turing appears to believe that Lady Lovelace’s objection can be

responded to on empirical grounds, although he does not say exactly what those

might be. Later, I will return to the issue of Turing’s empirical commitments.

6 See, for example, ibid., p. 123.

Turing’s Responses to Two Objections 157

123

A More Considered Response: Learning Machines

This does not, however, exhaust Turing’s response to Lady Lovelace’s objection.

The final, lengthy section of the 1950 paper begins with a reconsideration of the

objection that ‘‘the machine can only do what we tell it to do’’ (Turing 1950,

p. 454). He claims that the problem with creating machines that appear to do more

than a few simple things that they have been explicitly forced to do is ‘‘mainly one

of programming’’ (ibid., p. 455).

Turing considers building a thinking machine by hard coding it. ‘‘… about 60

workers, working steadily through … 50 years might accomplish the job, if nothing

went into the waste-paper basket. Some more expeditious method seems desirable’’

(ibid.).

In the remaining pages of this section, he describes a model of biomorphic

computation: initial states of learning machines are unorganized and random.

Such machines then learn using methods of reward and punishment: instructions

in the machine that lead to correct, or desirable responses are reinforced, and

instructions that lead to incorrect, or undesirable responses are degraded or

destroyed. Turing compares the process a learning machine might undergo to

evolution. Turing’s motivations for relying on learning machines in trying to build

intelligent computers is not limited, however, to considerations of convenience

and speed. In a moment, I will argue for what I take to be a deeper reason Turing

believes that learning machines will allow him to address Lady Lovelace’s

objection.

Two Objections at Once

Early writings on Lady Lovelace’s Objection

Now I want to look in greater depth at the conflation, in Piccinini’s 2003 paper, of

Turing’s responses to the mathematical objection and Lady Lovelace’s objection.

Piccinini tries to ‘‘reconstruct the mathematical objection’’ by, in his words,

following ‘‘the chronological order of Turing’s writings’’ (Piccinini 2003).

Unfortunately, the earlier writings do not as clearly separate Lady Lovelace’s

objection from the mathematical objection as do the later papers.

Piccinini quotes a long section in a 1947 lecture with much discussion of learning

machines, emphasizing programming not by hard coding but by instruction, and

correctly notes the importance of, for Turing, learning and unpredictability for

intelligence. However, the first sentence of the paragraph quoted is this: ‘‘It has been

said that computing machines can only carry out the processes that they are instructed to do’’ (Turing 1947, p. 392, emphasis added).

Immediately following this paragraph, Piccinini notes, Turing presents one of the

mathematical objections. However, as explained earlier, Turing’s response to this is

merely that machines must be allowed to make the same kinds of errors as human

mathematicians.

158 D. Abramson

123

Lady Lovelace and the Mathematical Objection Begin to Separate

Piccinini refers to the 1948 document in which Turing both considers the first

mathematical objection and discusses Lady Lovelace’s objection. Piccinini cites

Turing’s comment that ‘‘The argument from Gödel’s and other theorems rests

essentially on the condition that the machine must not make mistakes’’ (Turing 1948,

p. 411). He concludes that ‘‘the key was, once again, that the machine must be able to

learn from trial and error, an issue to which much of the paper was devoted’’ (Piccinini

2003, p. 38). This is mistaken. The response really just is that intelligence does not

preclude fallibility, and intelligent machines can be built that make mistakes.

Immediately following the quoted section, Turing goes on to give an example of

intelligent mistakes that people have made. Discussion of learning machines in

Turing’s 1948 paper begins in response to the following objection: ‘‘In so far as a

machine can show intelligence this is to be regarded as nothing but a reflection of

the intelligence of its creator’’ (Turing 1948, p. 411).

Nowhere in the 1948 paper does Turing discuss mistakes that can be made for the

purpose of novel axiomatic proof procedures. Much discussion, however, is devoted

to more and less expedient ways for creating intelligent machines. By the 1950

paper, Turing has moved to the other version of the mathematical objection, likely

because it seems more devastating. After all, he has twice dismissed the infallibility

version by arguing that intelligence is compatible with making mistakes. Also, in

the 1950 paper, Turing explicitly addresses the claim that machines cannot make

mistakes in a section that contains no mention of any version of the mathematical

objection (in the ‘arguments from various disabilities’, in which he distinguishes

errors of conclusion from errors of functioning).

There is much discussion of learning machines in the 1948 lecture, but, pseudo-

random elements (methods of generating series of digits that do not follow an

obvious pattern, but are in fact generated by algorithm) are discussed, thereby

preventing any need for non-uniform procedures. Therefore, it is by conflating Lady

Lovelace’s objection with the mathematical objection that Piccinini derives, I argue,

a mistaken interpretation of Turing’s response to the mathematical objection. In

summary, it should be noted that the first mathematical objection is considered

twice by Turing, and rejected each time because of the unjustifiable premise that

machines cannot make mistakes. Then, when considering the second mathematical

objection, he has a different, straightforward response. And, as described above,

since Turing’s death the mathematical objection has, in general, appeared in the first

version when presented by other authors, and the responses by critics have been

virtually identical to Turing’s.7

7 I have not presented all of the logicians and philosophers who have taken the response to the first

mathematical objection that Turing does. A notable early example of such a response is made by Putnam

in 1960: ‘‘Given an arbitrary machine T, all I can do is find a proposition U such that I can prove (3) If T

is consistent, U is true, where U is undecidable by T if T is in fact consistent. However, T can perfectly

well prove (3) too! And the statement U, which T cannot prove (assuming consistency), I cannot prove

either (unless I can prove that T is consistent, which is unlikely if T is very complicated)!’’ (Putnam 1960,

p. 77, emphasis in original). It is therefore preferable to have some name for this response other than

hyphenating the names of luminaries who make this response to the first mathematical objection.

Turing’s Responses to Two Objections 159

123

The Significance of Lady Lovelace’s Objection

Significance to Turing

Now I will discuss the significance of Lady Lovelace’s objection to Turing. I hope

to show the increasing significance he gave to it. Turing gives a 1951 BBC

broadcast which is almost entirely preoccupied with Lady Lovelace’s objection

(Turing 1951a). Here we see an explicit development: not only are learning

machines a more expedient way to create intelligence, but also they seem to satisfy a

prerequisite for thinking.

Turing repeats his claim that although machines only do what we program them

to, we may not know what the consequences of our instructions are.

‘‘If we give the machine a programme which results in its doing something

interesting which we had not anticipated, I should be inclined to say that the

machine had originated something, rather than to claim that its behaviour was

implicit in the programme, and therefore that the originality lies entirely with us’’

(ibid., p. 485).

In a 1952 radio discussion with three other scientists, Turing discusses how

analogy-making might be implemented in a computer. He comes up with a sketch of

an algorithm that he thinks human brains might use, but laments that ‘‘as soon as

one can see the cause and effect working themselves out in the brain, one regards it

as not being thinking, but a sort of unimaginative donkey-work’’ (Turing et al. 1952,

p. 500).

Then Turing offers what I will call the ‘epistemic–limitation condition on

intelligence’:

‘‘From this point of view one might be tempted to define thinking as consisting of

‘those mental processes that we don’t understand’. If this is right, then to make a

thinking machine is to make one which does interesting things without our really

understanding quite how it is done’’ (ibid.).

It is conceivable that these comments are merely tongue-in-cheek. However,

there is ample textual evidence that Turing believed what he says here quite

seriously. To argue for this, I turn to passages in which Copeland seems to find

greatest evidence that Turing introduces learning machines to respond to the

mathematical objection. In a short 1951 BBC radio address Turing does mention

Gödel’s theorem, but, I would argue, only to show that it provides motivation for

allowing intelligent machines to make mistakes.

Turing claims that it is possible to build machines that ‘‘simulate the behavior of

the human mind very closely’’ (Turing 1951b, p. 472). In defending this claim, he

describes machines that are not programmed directly, but instead given experiences

from which to learn. Turing says ‘‘It would be quite easy to arrange the experiences

in such a way that they automatically caused the structure of the machine to build up

into a previously intended form, and this would obviously be a gross form of

cheating, almost on a par with having a man inside the machine’’ (ibid., p. 473). In

this passage, we see that Turing’s focus is on the epistemic relationship between the

creator of the machine and the machine created. Turing clearly is committed to the

view that in order for the actions of a machine to be truly its own, and not the

160 D. Abramson

123

achievements of its creator, there must be an epistemic–limitation on the creator

with respect to the machine’s behavior.

There is further evidence that Turing’s guide to the building of intelligent

machines is in no way intended to address the mathematical objection directly, but

only Lady Lovelace’s objection. Satisfaction of the epistemic–limitation condition

by a machine amounts to performance by the machine in ways unforeseen by its

creator (or someone with access to the resources its creator did). Near the end of this

1951 address, he suggests a final feature to be incorporated in the design of learning

machines.

Each machine should be supplied with a tape bearing a random series of

figures, e.g. 0 and 1 in equal quantities, and this series should be used in the

choices made by the machine. This would result in the behaviour of the

machine not being by any means completely determined by the experiences to

which it was subjected, and would have some valuable uses when one was

experimenting with it. (ibid., 475)

Notice that in this case the behavior of the machine is completely determined by

the machine’s experience and the initial sequence of 1s and 0s on its tape. The only

lack of determination that is introduced is in the perspective of the educators of the

machine. For example, the teacher may use a series of lessons that has hitherto

created dull, uninteresting machines, but can remain hopeful that in this particular

case interesting behavior will result. Furthermore, this inclusion of a pseudo-random

element yields machines which will obviate attempts to ‘cheat’ in the way just

mentioned.

Before discussing the epistemic–limitation condition, I want to present the

general form of Turing’s argument in favour of the epistemic–limitation condition.

1. Creativity involves the ability to originate at least something.

2. Following a set of rules intended by the creator of the rules to bring about a

particular behavior does not involve originating anything.

3. Therefore, machines that are programmed to have intended behavior are not

creative.

4. Creativity is essential for intelligence.

5. Therefore, intelligent machines must have behaviors not intended by their

creators.

6. Therefore, intelligent machines must not be programmed as conventional

computers are.

The mathematical objection is relevant to this argument only insofar as most

computers are intended by their programmers not to make mistakes. However, as

Turing insisted, there is no inconsistency in the notions of a computer that makes

mistakes, and a computer that makes mistakes unintended by its creator.

A natural first reaction to this argument is to notice its mixing of metaphysical

and epistemological commitments. Here, Turing is asking us to require, for

intelligence, a restriction on others’ ability to understand the processes that lead to

behavior: a kind of ‘anti-intentional stance’. I want to note, however, that Turing is

not advocating that intelligent beings always behave in inexplicable ways. Also,

Turing’s Responses to Two Objections 161

123

Turing’s point of view is not entirely different from one advocated in recent

comments by a philosopher of cognitive science, the champion of the intentional

stance.

In his 2000 address to the Eastern APA, Daniel Dennett describes a general

theory of mental events and its impact on the concept of creativity. In essence, he

claims that the brain recapitulates evolution from moment to moment. Possible

thoughts and actions compete with one another for control of the body, and our

notion of an identifiable self that persists is an illusion; he says ‘‘It is important to

recognize that genius is itself a product of natural selection and involves generate-

and-test procedures all the way down’’ (Dennett 2001, p. 25). Dennett asks,

Does this realization amount to a loss – an elimination – of selfhood, of

genius, of creativity? Those who are closest to the issue – the artisitic and

scientific geniuses who have reflected on it – often confront this discovery

with equanimity. Mozart is reputed to have said of his best musical ideas

‘‘Whence and how do they come? I don’t know and I have nothing to do with

it.’’ The painter Philip Guston is equally unperturbed by this evaporation of

visible self when the creative juices start flowing:

‘When I first come into the studio to work, there is this noisy crowd which

follows me there; it includes all of the important painters in history, all of my

contemporaries, all the art critics, etc. As I become involved in the work, one

by one, they all leave. If I’m lucky, every one of them will disappear. If I’m

really lucky, I will too.’ (ibid., 26)

In his closing remarks, Dennett acknowledges a long list of philosophers,

scientists, and artists as co-authors, to illustrate the rich effect that his ‘teachers’

have had on his unconscious, Darwinian processes. Dennett here takes a position

adopted in early philosophical literature responding to Turing’s claim that machines

can think. For example, Richard Purtill writes that if we find that human behavior is

completely determined by our initial state and our experience, and we are thus

mechanical, then ‘‘it would so alter our conception of ourselves as to rather make us

say that men did not think than that computers did’’ (Purtill 1971, p. 294).

Geoffrey Sampson responds that, due to the facts that computers are designed by

humans but humans are not, computers are easier to open up and investigate, and

humans are much more complex than computers, ‘‘computer behavior is known to

be determined while human behavior is not can [sic] easily be explained’’ (Sampson

1973, p. 593). Sampson concludes that thinking is whatever we are discovered to

do, mechanical or not.

Sampson’s comments are important because they separate out two forms of the

epistemic–limitation condition. We have seen one form, in which the condition is

satisfied just in case it is impossible to anticipate the behavior of a machine by

investigating its initial state, and input. Distinct from this is the claim that there is

some fundamental mystery present in the functional operation of the machine (or,

put differently, that the behavior of the machine on some input results from

something other than prior determination). Recent critics have tried to show that no

mere computer could satisfy the first epistemic–limitation condition; but, I will

162 D. Abramson

123

show, they have picked on the wrong version of it. First, though, I will address a

possible criticism of my interpretation of Turing on Lady Lovelace’s objection.

Necessary and Sufficient Conditions for Thinking

Suppose that I am right to attribute to Turing the epistemic–limitation condition on

thinking. A familiar problem arises immediately. Turing’s central purpose in his

1950 paper is to offer a sufficient condition on thinking. If I claim now that, implicit

in this paper and his other musings, we find a different, independent necessary

condition on thinking, then one could make the same objection to me that I made

earlier to Piccinini and Copeland: Turing has to thereby give up the central thesis of

the 1950 paper.

To see this, suppose that a particular machine does not satisfy the epistemic–

limitation condition; suppose it merely consists in a look-up table in the sense

described by Purtill. Furthermore, suppose that this machine passes the Turing Test.

Then Turing must deny that we can call this machine a thinking being, contrary to

the main thesis of the 1950 paper.

However, there is another interpretation of the relationship between the necessary

and sufficient conditions for thinking that are under discussion here. Turing could

have held that, as a matter for empirical investigation, satisfaction of his sufficient

condition for intelligence implies satisfaction of his necessary condition for

intelligence. In the paper ‘Psychologism and Behaviorism’, Ned Block argues for

the claim that passing a Turing Test cannot be sufficient for intelligence, since a

look-up table (a machine with a simple program that emits canned responses for

every possible question in a sequence of questions) could pass the test, and is not

intelligent (Block 1981).

However, notice that Block only argues for the logical possibility of such a

machine: ‘‘My argument requires only that the machine be logically possible, not

that it be feasible or even nomologically possible… Could it be an empirical hypothesis that intelligence is the capacity to emit sensible sequences of outputs

relative to input sequences?’’ (ibid., p. 30). Block’s argument fails as a criticism of

Turing’s own views, since Turing explicitly offers his test only as a sufficient

condition for intelligence. Nevertheless, on the reading I have presented, Turing has

a response to Block’s criticism that the Turing Test lacks empirical content.8 On the

reading I am presenting, Turing makes the empirical claim that, as we try to build

machines that pass the Turing Test, we will have poor success with machines that

contain surveyable procedures for anticipating questions. This, in fact, has been the

case.9

8 Arguments that Turing intended his test to have related, but different empirical content can be found in

Dennett’s 1985 postscript to his paper ‘Can Machines Think’ (Dennett 1985, p. 21). 9 The transcripts of the competitors for the ‘Loebner Prize’ are striking instances of these; see links from

http://www.loebner.net/Prizef/loebner-prize.html for examples.

Turing’s Responses to Two Objections 163

123

http://www.loebner.net/Prizef/loebner-prize.html

Significance to Later Commentators

In a recent article, Selmer Bringsjord et al. argue for a version of the epistemic–

limitation condition on intelligence, at least with respect to creative intelligence.

Instead of finding such a condition in Turing’s writing’s, they argue that such a

condition can be used to demonstrate that machines fail the Lovelace objection. If

correct, given the interpretation of Turing I have offered here, we again, as with

Piccinini’s comments, must find a crucial tension in Turing’s position.

In their article, Bringsjord et al. introduce a test that they claim captures Lady

Lovelace’s insight. They define the ‘Lovelace Test’ such that machine M created10

by human H passes it if and only if

1. M outputs o;

2. M’s outputting o is not the result of a fluke hardware error, but rather the result

of processes M can repeat;

3. H (or someone who knows what H knows, and has H’s resources) cannot

explain how A produced o by appeal to Ms architecture, knowledge-base, and

core functions (adapted from Bringsjord et al., 2001, p. 12).

For the moment, I will grant that this definition of a ‘Lovelace test’ corresponds

to Turing’s intuition that thinking must satisfy the epistemic–limitation condition.

Bringsjord et al. claim to have a proof that no conventional computer can pass this

test. If this is correct, then it seems that Turing’s intuitions that machines could

satisfy the epistemic–limitation condition by teaching as opposed to explicit

programming is misguided.

Here is a sketch of the attempted proof. The so-called ‘representability theorem’ in

logic says that if some function can be computed effectively, then one can come up

with a set of axioms in arithmetic from which one can derive all and only the values

of that function. Call this set of axioms U; then, say Bringsjord and his coauthors,

You can think of U in this case as a knowledge-base. But then there is no

longer any ‘‘thinking for itself’’ going on, for if we assume a computer

scientist to be in command of the knowledge-base U and the relevant

deduction from it, the reasons for this scientist to declare the [learning-

machine] a puppet are isomorphic to the reasons that compel the designer of

knowledge-based systems like [a system we created for understanding stories]

to admit that such a system originates nothing. (Bringsjord, 2001, 19–20)

It is important to see why this argument fails. Applying the representability theorem

to a Turing machine, or conventional computer, simply says that if the machine

halts on an input and generates an output, then we can ‘represent’ that behavior in

axiomatic fashion by deriving statements in arithmetic from sets of statements

about arithmetic. Where the argument goes wrong is by suggesting that this is

anything like the working out of behavior from the knowledge-base of a computer.

10 I use the word ‘created’ here instead of ‘designed’, as Bringsjord et al. do, since design seems to give

away at least some of what is at stake. Things that are designed typically are intended to behave in some

way conceived by the designer, whereas created objects may not—e.g. the described creation, by God, of

free human beings in Genesis.

164 D. Abramson

123

To see this, I give a reductio of the claim that we can, in general, ascertain the

behavior of an arbitrary computer by working out the consequences of its

knowledge-base, architecture, and core functions. Consider a sequence of Turing

machines. The nth machine, on any input, checks to see if Tn(n) halts. If it does, then

it begins to enumerate theorems of Peano arithmetic. Notice that in one sense, the

knowledge-base of such a computer consists in its program, as I have just described

it.

I would claim that, in a more natural sense, the output of each of these machines

is determined by either the knowledge-base consisting of the empty set, in case

Tn(n) does not halt, or Peano arithmetic. And, since the special halting function is

not solvable, we cannot in general determine what the (more naturally described)

knowledge-base of an arbitrary machine is. The representability theorem merely

points out that, if a machine does halt on some input, then we can work out, step by

step, the sequence by which the machine reached the halting state. This says

nothing, in general, about our ability to determine if a machine will halt, or, even if

we observe a series of halting behaviors, what the knowledge-base is that has

determined these behaviors. To put it another way, one cannot in general, based on

the program of a machine, determine future performances, and relevant parts of the

program for future performances. Calling the U that the representation theorem

gives us the ‘knowledge base’ is an equivocation between the rules that govern the

machine’s functioning and the rules that govern the machine’s competence at a task

(say, enumerating theorems of Peano arithmetic).

Bringsjord et al., in their paper, considers a few different instances of attempts to

build intelligent computers. These examples are misleading because in each, the

programmer’s knowledge of the basic workings of the computers matches their

knowledge of the competence of the computer in question In each one, a software

engineer, or group of engineers, has attempted to write explicit instructions for a

computer to solve a problem. So, Bringsjord et al. ignore the fact that the first epistemic–limitation condition cuts across different sorts of computers, even though

the second epistemic–limitation condition does not. To put it yet another way, the

representability theorem, even more generally, says that there is no miraculous step

along the way in a computer’s calculating a response to the question. However,

Turing was aware that no such miraculous step is required for the computer’s

behavior to be, in principle, unpredictable. I take Turing’s warning, against the

fallacy that computer scientists and philosophers are particularly subject, to be quite

serious. It is for similar reasons that in the 10th Turing Award Lecture, computer

scientists Newell and Simon, in defending a version of the claim that regular

computers can think, spend considerable energy arguing that computer science is an

empirical science (Newell and Simon 1976).

Conclusion

If one asks, ‘why did Turing spend so much time and energy discussing learning

machines’, I believe the obvious answer, with more than ample textual support, is

‘because of his commitment to the epistemic–limitation condition’. This

Turing’s Responses to Two Objections 165

123

commitment is given disguise in the only presentation of his philosphical views on

thinking machines published in Turing’s lifetime: twice, within a few sentences, he

writes in his 1950 paper that teaching learning machines will be a ‘more

expeditious’ way to create intelligent machines. However, even amidst those

comments, Turing returns to Lady Lovelace’s objection explicitly, claiming that

‘‘An important feature of a learning machine is that its teacher will often be very

largely ignorant of quite what is going on inside… Intelligent behavior presumably

consists in a departure from the completely disciplined behavior involved in

computation, but a rather slight one (Turing 1950, pp. 462–463)’’.

This epistemic–limitation condition, as I have called it, can be easily conflated

with something like a ‘mysterian’ position, such as the one held by some in

explaining consciousness. However, it is distinct from it in that it allows for a

scientific understanding of the mechanisms underlying mental phenomena, and

suggests methods for creating artificial minds.

In closing, I want to mention that researchers in ‘biomorphic’ computation,

following in broad strokes the method that Turing urged (pseudo-random initial

state, iterated procedures involving testing and preservation of success without

explicit programming), have attempted to make precise the notion that machines

have performed in ways competitive with human beings. As of recently, such

machines had produced results that

1. were patented as an invention in the past, was an improvement over a patented

invention, or would qualify today as a patentable new invention

2. were equal to or better than a result that was accepted as a new scientific result

at the time when it was published in a peer-reviewed journal

3. were publishable in their own right as a new scientific result (independent of the

fact that the result was mechanically created)

4. solved a problem of indisputable difficulty in its field (adapted from Koza

(1999, pp. 5-6).

In none of these cases could the programmers simply anticipate, in advance, a

solution to the problem posed to the machine, or even whether the machine could

find a solution. So, Turing’s empirical claim, resulting from his struggle with Lady

Lovelace’s objection, has been confirmed both by positive and negative examples:

computers that satisfy the epistemic–limitation condition show signs of creativity,

and despite computer scientists’ best efforts, computers that fail the condition are

clearly unintelligent. Turing’s empirical claim only becomes apparent once we

understand Turing’s motivations for discussing learning machines, and his adoption

of a response to the first mathematical objection common among later logicians.

References

Block, N. (1981). Psychologism and behaviorism. The Philosophical Review, 90(1), 5–43.

Bringsjord, S., Bello, P., & Ferrucci, D. (2001). Creativity, the Turing Test, and the (better) Lovelace

Test. Minds and Machines, 11, 3–27.

Copeland, B. J. (Ed.) (2004). The essential Turing. Oxford University Press.

166 D. Abramson

123

Dennett, D. (1998/1985). Can machines think with postscripts 1985 and 1997?. In Brainchildren: Essays on designing minds. MIT Press.

Dennett, D. (2001). In Darwin’s wake, where am I? Proceedings and Addresses of the American Philosophical Association, 75, 11–30.

Fodor, J. (1992). The big idea: Can there be a science of mind? Times Literary Supplement, 5–7.

Furukawa, K., Michie, D., & Muggleton, S. (Eds.) (1999). Machine intelligence, Vol. 15. Oxford

University Press.

Koza, J. R. (1999). Genetic programming III: Darwinian invention and problem solving. Morgan

Kaufmann.

Lucas, J. (1961). Minds, machines, and Gödel. Philosophy, XXXVI, 112–127.

Newell, A., & Simon, H. A. (1997/1976). Computer science as empirical inquiry: Symbols and search,

Vol. 19. MIT Press. Presented as the Tenth Turing Award Lecture, Published in the Communi-

cations of the Association for Computing Machinery, pp. 113–126.

Penrose, R. (1994). Shadows of the mind. Vintage.

Piccinini, G. (2003). Alan Turing and the mathematical objection. Minds and Machines, 13, 23–48.

Purtill, R. L. (2004/1971). Beating the imitation game. MIT Press. Originally Published 1971 in Mind, 80,

318, 290–294.

Putnam, H. (1964/1960). Minds and machines. In A. R. Anderson (Ed.), Minds and machines. Prentice-

Hall. Originally Published in Dimensions of Mind, Sidney Hook (Ed.).

Rogers, H. (1967). Theory of recursive functions and effective computability. MIT Press.

Sampson, G. (1973). In defence of Turing. Mind, 82(328), 592–594.

Shapiro, S. (2003). Mechanism, truth, and Penrose’s new argument. Journal of Philosophical Logic, 32,

19–42.

Turing, A. (1939). Systems of logic based on ordinals. Proceedings of the London Mathematical Society, 45, 161–228.

Turing, A. (2004/1940). Letters on logic to Max Newman. In B. J. Copeland (Ed.), The essential Turing.

Oxford University Press.

Turing, A. (2004/1947). Lecture on the automatic computing engine. In B. J. Copeland (Ed.), The essential Turing. Oxford University Press.

Turing, A. (2004/1948). Intelligent machinery. In B. J. Copeland (Ed.), The essential Turing. Oxford

University Press.

Turing, A. (1950). Computing machinery and intelligence. Mind, 59(236), 433–460.

Turing, A. (2004/1951a). Can digital computers think? In B. J. Copeland (Ed.), The essential Turing.

Oxford University Press.

Turing, A. (2004/1951b). Intelligent machinery, a heretical theory. In B. J. Copeland (Ed.), The essential Turing. Oxford University Press.

Turing, A., Braithwaite, L. C., Jefferson, A. A., & Newman, E. (2004/1952). Can automatic calculating

machine be said to think? In B. J. Copeland (Ed.), The essential Turing. Oxford University Press.

Turing’s Responses to Two Objections 167

123

Turing’s Responses to Two Objections

Abstract
Introduction
Turing’s Response to the Mathematical Objection
Another Interpretation

Lessons from Ordinal Logics
‘Puzzling Comments’ by Turing
An Initial Problem with this Interpretation
Turing’s Response to the Earlier Version
An Old Misunderstanding

Lady Lovelace’s Objection

Definition and First Response
A More Considered Response: Learning Machines

Two Objections at Once

Early writings on Lady Lovelace’s Objection
Lady Lovelace and the Mathematical Objection Begin to Separate

The Significance of Lady Lovelace’s Objection

Significance to Turing
Necessary and Sufficient Conditions for Thinking
Significance to Later Commentators

Conclusion
References

<< /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles true /AutoRotatePages /None /Binding /Left /CalGrayProfile (None) /CalRGBProfile (sRGB IEC61966-2.1) /CalCMYKProfile (ISO Coated v2 300% \050ECI\051) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Error /CompatibilityLevel 1.3 /CompressObjects /Off /CompressPages true /ConvertImagesToIndexed true /PassThroughJPEGImages true /CreateJDFFile false /CreateJobTicket false /DefaultRenderingIntent /Perceptual /DetectBlends true /ColorConversionStrategy /sRGB /DoThumbnails true /EmbedAllFonts true /EmbedJobOptions true /DSCReportingLevel 0 /SyntheticBoldness 1.00 /EmitDSCWarnings false /EndPage -1 /ImageMemory 524288 /LockDistillerParams true /MaxSubsetPct 100 /Optimize true /OPM 1 /ParseDSCComments true /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveEPSInfo true /PreserveHalftoneInfo false /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts false /TransferFunctionInfo /Apply /UCRandBGInfo /Preserve /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true ] /NeverEmbed [ true ] /AntiAliasColorImages false /DownsampleColorImages true /ColorImageDownsampleType /Bicubic /ColorImageResolution 150 /ColorImageDepth -1 /ColorImageDownsampleThreshold 1.50000 /EncodeColorImages true /ColorImageFilter /DCTEncode /AutoFilterColorImages false /ColorImageAutoFilterStrategy /JPEG /ColorACSImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /ColorImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /JPEG2000ColorACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000ColorImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasGrayImages false /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 150 /GrayImageDepth -1 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /GrayImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000GrayACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000GrayImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasMonoImages false /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 600 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict << /K -1 >> /AllowPSXObjects false /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (None) /PDFXOutputCondition () /PDFXRegistryName (http://www.color.org?) /PDFXTrapped /False /Description << /ENU <FEFF004a006f0062006f007000740069006f006e007300200066006f00720020004100630072006f006200610074002000440069007300740069006c006c0065007200200036002e000d00500072006f006400750063006500730020005000440046002000660069006c0065007300200077006800690063006800200061007200650020007500730065006400200066006f00720020006400690067006900740061006c0020007000720069006e00740069006e006700200061006e00640020006f006e006c0069006e0065002000750073006100670065002e000d0028006300290020003200300030003400200053007000720069006e00670065007200200061006e006400200049006d007000720065007300730065006400200047006d00620048> /DEU <FEFF004a006f0062006f007000740069006f006e007300200066006f00720020004100630072006f006200610074002000440069007300740069006c006c0065007200200036002e000d00500072006f006400750063006500730020005000440046002000660069006c0065007300200077006800690063006800200061007200650020007500730065006400200066006f00720020006f006e006c0069006e0065002e000d0028006300290020003200300030003800200053007000720069006e006700650072002d005600650072006c0061006700200047006d006200480020000d000d0054006800650020006c00610074006500730074002000760065007200730069006f006e002000630061006e00200062006500200064006f0077006e006c006f006100640065006400200061007400200068007400740070003a002f002f00700072006f00640075006300740069006f006e002e0073007000720069006e006700650072002e0063006f006d000d0054006800650072006500200079006f0075002000630061006e00200061006c0073006f002000660069006e0064002000610020007300750069007400610062006c006500200045006e0066006f0063007500730020005000440046002000500072006f00660069006c006500200066006f0072002000500069007400530074006f0070002000500072006f00660065007300730069006f006e0061006c0020003600200061006e0064002000500069007400530074006f007000200053006500720076006500720020003300200066006f007200200070007200650066006c00690067006800740069006e006700200079006f007500720020005000440046002000660069006c006500730020006200650066006f007200650020006a006f00620020007300750062006d0069007300730069006f006e002e> >> >> setdistillerparams << /HWResolution [2400 2400] /PageSize [5952.756 8418.897] >> setpagedevice