The basic concept behind the experimental method is explained in greater detail in chapter 2, where we are given clear advice on how treatment and outcome should be specified and differences interpreted. In the remaining chapters important issues for the evaluator responsible for designing, implementing and analysing the results of a social experiment are discussed. Chapter 3 presents alternative models for random assignment – dependent on whether the programme is ongoing or a special demonstration and on the number of different policies to be evaluated. Chapter 4 deals with random assignment models, including issues such as how to calculate the sample size and select sites to obtain optimal statistical power. In chapter 5 practical advice on implementation and data collection is given. In this phase of the experiment it is of vital importance to gain the co-operation of the programme staff.

As demonstrated in chapter 6, the ultimate strength of social experiments is the simplicity with which results can be analysed. Nevertheless, potential biases must be taken into consideration. The author points to the loss of follow-up data because of survey non-response, this being probably the most common departure from the ideal in real-world social experiments. In addition, the fact that a programme or social policy is found to have intended effects is not necessarily sufficient to justify adoption, which is why we are told how to calculate benefits and costs.

The last chapter of the book (chapter 7) is the least necessary of them all. However, the discussion on how social experiments can – and, according to the author, should – inform the process of policy making is neither uninteresting nor wasted. It applies, however, as much to research and evaluations in general as to social experiments in particular.

I find the book well written, well organized and easy to follow. It is illustrated sufficiently, rather than excessively, by instructive examples. The author’s knowledge and insight into the issue is beyond doubt. The vital question, however, is whether we need a book like this. As mentioned, only 195 experiments have been evaluated in the USA over the last thirty years and the number in Europe is unlikely to be any greater (we hardly know of any real experiments from Norway). The experimental method is truly exclusive. This might result in few readers, as this method rarely suits the evaluation situation – not least because it is heavy on resources and time. At the same time, this exclusiveness could function as an argument in favour of a practical guide like this. And since most researchers lack practice in the field of experimental methods it is of vital importance that experienced practitioners, such as Orr, share their knowledge.

Orr presents the experimental method as the ‘gold standard’ of programme evaluation. It is his experience that the use of ordinary econometric techniques with their ‘clouds of confusing correlations ...make the inference of causality ...hazardous’. He therefore underlines the superiority of the method in deciding what causes what in the real world and in estimating the exact magnitude of the impact with measurable confidence. This is the case, however, only in properly designed and implemented experiments and numerous pitfalls in the practice of the experimental method are demonstrated in the book. For instance, difficulties may be confronted when trying to obtain a random sample, stating the real content of the experiment, and obtaining reliable data. Most of the biases potentially caused by such problems can be handled by adjusting the estimates without seriously distorting the condition of random assignment – but not all of them. For example, when it comes to the common problem of loss of follow-up data as a result of survey non-response, the experimenter is in much the same position as the non-experimental analyst. Orr concludes that, in fact, it is impossible to know how well the treatments and control groups are matched, that is, whether the estimates are biased. Thus, the general view that knowledge in social science is uncertain definitively holds true also for the experimental method. It is hard for me to see why we do not risk making estimates and inferences of causality which are just as hazardous when using the experimental method as any other adequate method. In the ideal world the experimental method may be by far the best; in the real world this is less obvious.

It is symptomatic that the word ‘theory’ is not found in the index of the book. The claim from some quarters that experimental methods are unsuitable for dealing with the question of why or how a programme is working, is not touched upon at all. Even though a discussion on this subject would have been highly interesting, it would be unfair to criticize Orr for questions he does not raise. In experimental methods, theory is obviously not needed. The book does what it promises, therefore: to be a practical guide to social experiments.