Public relations measurement and evaluation

Public relations measurement and evaluation has long been a major practice subject. From the late 1970s onwards it has been identified as an important issue for research and practice implementation (McElreath, 1980, 1989; Synnott & McKie, 1997; Watson, 2008; Watson & Noble, 2007). The evolution of public relations measurement starts much earlier, with some suggesting that media monitoring practices can be identified from the late 18th century onwards (Lamme & Russell, 2010). It is, however, from the beginning of the 20th century, when ‘public relations’ began to be widely used as the description for a set of communication activities, that measurement practices can be identified. This paper traces that development which parallels public relations’ holistic beginnings through to its transformation into a communication practice, with strong publicity influences. Along the way, there has been the worldwide expansion of public relations practices, services and education; the growth of measurement and evaluation services; and the influence of academic thinking.

Struggling to meet your deadline ?

Get assistance on

Public relations measurement and evaluation

done on time by medical experts. Don’t wait – ORDER NOW!

Meet My Deadline

This paper uses a timeline narrative to describe and discuss the evolution of public relations measurement and evaluation over more than a century. In many ways this evolution has similarities to the development of public relations as an emerging and then extensive communications practice. Like public relations, it starts with elements of both social science research, especially opinion polling, and of a practice emphasis on publicity through media channels. By the mid-20th century, this moves much more towards a publicity-led practice with the use of media analytics becoming far more important than social science methods. However, but the beginning of the 21st century, the balance was moving back towards more sophistication

∗ Correspondence address: The Media School, Bournemouth University, Poole, Dorset BH12 5BB, United Kingdom. Tel.: +44 01202 961986; fax: +44 01202 965530.

E-mail address: twatson@bournemouth.ac.uk

0363-8111/$ – see front matter © 2011 Elsevier Inc. All rights reserved. doi:10.1016/j.pubrev.2011.12.018

T. Watson / Public Relations Review 38 (2012) 390– 398 391

in measurement and the wider alignment of public relations communication objectives with organisational objectives, especially in corporate public relations where new techniques such as scorecards (Zerfass, 2005) are being used. Ironically, this area of public relations is adopting whole-of-organisation (holistic) approaches to organisational communication similar to those promoted in the 1920s and 1930s.

2. The beginnings

The timeline starts before the term, public relations, came into use. Lamme and Russell’s monograph, Removing the spin: Towards a new theory of public relations history (Lamme & Russell, 2010) argues that from George Washington onwards, US presidents monitored newspapers in order to gain intelligence on what was being said about them and the views of fellow citizens. In the 19th century, many industries and groups also tracked media coverage and public opinion. They ranged from railroads to temperance societies and evangelists. In the US and UK, news cuttings agencies were established in the latter part of the century. From some of these, there is lineage to today’s international computer-based evaluation companies.

As the 20th century opened, the first publicity agencies were formed in the United States. Cutlip (1994) dates the first to the Publicity Bureau in Boston in 1900. One of its major clients from 1903 onwards was the telecommunications busi- ness, American Telephone & Telegraph Company (AT&T) which was based in the same city. AT&T and its agency “early saw the need for systematically gauging public opinion . . . collected and studied newspaper clippings from the nation’s press” (Cutlip, 1994, p. 18). It found that 90% were antagonistic. By modifying company behaviour and disseminating “real infor- mation through the press,” AT&T gradually reduced the negative coverage to “sixty percent and lower” (ibid, p. 18). Working for railroad interests, the Publicity Bureau developed a system of monitoring and influencing press coverage. Called The Barometer, it developed a card index of the attitudes of editors, gained from visits, and media usage of publicity material. This allowed the agency to “whether a paper is “Good” or “Bad” from the standpoint of the railroads” (ibid, p. 21) Cutlip commented, ironically, that “public relations research is not as new as some think,” (ibid, p. 21).

Amongst the founders of public relations practice in the US, Ivy L. Lee, who formed two of the earliest public relations advisory firms, took the view that he was engaged in an art whereas another pioneer, Edward L. Bernays, saw public relations as an applied social science (Ewen, 1996; Tye, 1998). Lee, according to his biographer, considered that his activity was nondefinable and nonmeasurable. It only existed through him and was thus not comparable (Hiebert, 1966). Cutlip says Lee “constantly referred to his work as an art. In fact, Bernays was quoted in November 1940 as saying, “He (Ivy Lee) used to tell me that this was an artist’s field, that what he was doing would die with him” (Cutlip, 1994, p. 59).

Bernays presented public relations as an applied social science to be planned through opinion research and precisely evaluated. Ironically, there is very little discussion of measurement and evaluation of campaign effectiveness in Bernays’ books and papers. His first book, Crystallizing Public Opinion, (Bernays, 1923) set the foundations for a systematic approach to public relations (Pavlik, 1987). Unlike Lee, who was in practice before the First World War, Bernays was on the staff of the Committee on Public Information (The Creel Committee) which was “organised to unite public opinion behind the war at home and propagandize American peace aims abroad” and “(whose) demonstration of propaganda was to have a profound effect on American culture and on the future of public relations” (Cutlip, 1994, p. 106). Advertising and the publicity side of public relations both expanded rapidly in the 1920s driven along by the self-publicising efforts of pioneers and several business books on publicity and public relations. Bernays’ books and Lee’s privately published Publicity Some of the Things It Is and Is Not (Lee, 1925), which Cutlip says attracted John W. Hill into public relations, were well known along with R.H. Wilder and K.L. Buell’s Publicity (Wilder & Buell, 1923) and several other books. The latter two publicists defined publicity as “the organised and deliberate effort to enlist the support of the public for an idea, sponsored by any given group for any given purpose,” (ibid, p. 109).

Although cuttings agencies monitored press coverage for clients, there was little discussion of the measurement and eval- uation of publicity or public relations activity. It was a former magazine editor, Arthur W. Page, who brought the disciplines of opinion research into public relations and organisational communication at American Telephone and Telegraph (AT&T). In the 1920s, the journalist and commentator Walter Lippmann’s (1922) book Public Opinion had a major influence on all forms of communication. He identified the role of public opinion in legitimising governments and organisations. Although AT&T had started using opinion polling shortly before Arthur Page joined it, he championed the use of surveys which were to be an important factor in developing a customer-facing culture at the telecommunications giant. “He deserves credit for recognizing the need for feedback and encouraging development of systems to gauge the moods of AT&T’s publics. Integra- tion of formal feedback systems into the public relations function is one of his contributions to public relations practise” (Griese, 2001, p. 122). AT&T continued to monitor media, although the examples are less prevalent than the use of opinion surveys. Page’s biographer, Noel Griese, has identified two studies of the use by newspapers of “clipsheets” (broadsheets with several AT&T news items which editors would select and send to typesetters) in 1932 and 1933. These were measured by the number of items published and the total of column inches of coverage. “Whilst the column inches of publicity a corporation gets are not a reliable indicator of the amount of good will being built, these studies show AT&T’s practice of systematically evaluating public relations devices” (Griese, 2001, p. 153).

Page created a “public relations laboratory where PR successes and failures were gathered, studied and the lessons learnt passed on to his colleagues at AT&T” (Broom & Dozier, 1990, p. xi). This approach continued after his retirement in 1947 until the telephone monopoly was broken up in the late 1970s. It is notable, however, that AT&T was not measuring the results of communication activity (outcomes) (Tedlow, 1979). Page used the term ‘public relations’ in an organisationally holistic

392 T. Watson / Public Relations Review 38 (2012) 390– 398

manner with a strong emphasis on the corporation understanding “the overall relations with the public it served” (Griese, 2001, p. 195). In 1938 he explained it further: “The task which business has, and which it has always had, of fitting itself to the pattern of public desires, has lately come to be called public relations” (ibid, p. 195).

3. 1930s and 1940s

By the late 1930s, a wide range of measurement and evaluation methods were being used in the United States, notably by various levels of government. Batchelor, writing in 1938, gave two examples of the monitoring and interpretation of media publicity.

The Roosevelt Administration gives close attention not merely to the technique of publicity dissemination but also to the manner of its reception. In other words, it watches carefully all changes in the political attitudes of a community. The sum of these numerous local impressions constitutes, of course, a barometer of national opinion that possesses great value. (p. 212) [The method of data collection is not identified by Batchelor]

He also discussed Toledo Associates, which was a “cooperative publicity effort, sponsored by local business interests” set up to promote the city of Toledo, Ohio during the Great Depression.

Toledo’s experiment in cooperative industrial publicity became an unqualified success. Ninety-one per cent of more than 72,000 clippings, representing newspaper circulations totalling more than one and half millions, were regarded as favourable to the city’s interests (p. 214).

So it can be seen that at high level, measurement and evaluation were taking place using methods that are still in place today.

Although publicity had always continued as a practice, it was seen as a delivery sub-set of public relations. The mid- century view, expressed by Griswold and Griswold (1948), was that public relations was a management function to create relationships and “earn public understanding and acceptance” (p. 4). Plackard and Blackmon (1947) separated public rela- tions as “the administrative philosophy of an organisation” which “stems from corporate character and over-all operations” from publicity which was “the art of influencing opinion by special preparation and dissemination of news” (p. 14). Com- munication or publicity were thus delivery and dialogue processes but not conceived as public relations itself. That view changed quickly as consumer products were developed and notions of corporate and product brands grew. Public relations lost that holistic concept and became typified by publicity practices. L’Etang (2004), writing about the 1960s, summarised the changed situation as: “business managers saw public relations as a cheap way of getting media coverage in comparison with advertising.” The impact on public relations measurement and evaluation was a move away from the social science-led emphasis on public opinion research to a more pragmatic analysis of media coverage, which was to dominate the second half of the 20th century.

4. Media evaluation practices to 1950

From early times, PR practitioners and organisations had monitored press coverage of their own and others activities. In 1942, Harlow wrote that public relations practitioners and their employers “should not be impressed by sheaves of press clippings” (p. 43) as a volume indicator of what was going on. Most books on public relations across the initial 40–50 year period discussed measurement of the volume of coverage, its length in column inches and whether it was positive or negative. The creation of the clippings or cuttings book became an art form with thick card paper on which clippings were mounted. Plackard and Blackmon gave this ethically dubious advice in 1947: “The publicist must learn the art of “pepping up” publicity results. Publicity clippings as such are not sufficiently interesting to show to a client. However, they can be dressed up or dramatized in unusual ways” (p. 299). Examples given included “trick photography” by blowing cuttings up and then printing large sheets of folded card on which they were placed; graphic presentation of cuttings beneath newspaper mastheads; and displays on large display boards, especially in hall corridors, all in order to emphasise the volume.

5. The UK

Public relations by mid-century was well-established in the United States, but in the UK, it was a post-World War 2 phenomenon. The first press agency, Editorial Services, had been set up by Basil Clarke in London in 1924 (L’Etang, 2004) but the establishment and real growth of public relations came as a result of journalists and propaganda experts coming out of government and the armed forces in 1945 with knowledge of news management and propaganda methods. The Institute of Public Relations (IPR) was set up in 1948, mainly by governmental communicators in information officer posts, as the first step to professionalise their area of activity (ibid.). The first IPR conference was in 1949 and the first British book, a ‘how-to’ guide entitled Public relations and publicity by J.H. Brebner (1949), appeared in the same year.

From its outset, issues of evaluating public relations were discussed in the IPR’s Journal: mostly as methods of collation of cuttings and transcripts, and how to do it cheaply (J. L’Etang, personal communication, January 10, 2011). Unlike the US with its interest in social sciences and university education, there was a strong anti-intellectual streak in the IPR. This

T. Watson / Public Relations Review 38 (2012) 390– 398 393

was expressed by its 1950 President Alan Hess who inveighed against “a tendency for too much intellectualisation and too much market research mumbo-jumbo” (L’Etang, 2004, p. 75). The IPR was not to produce its own book on PR until 1958 and training support for members was slow to start.

6. Evaluation scholarship

The first edition of Scott Cutlip and Alan Center’s long standing and still published PR text, Effective Public Relations, addresses measurement and evaluation mainly through the routes of public opinion research. Some commentators (notably Lindenmann, 2005) consider that the first edition of Cutlip and Center (1952) was the first scholarly book to mention the measurement and evaluation of public relations programmes. In later editions, they introduced their PII (Preparation, Implementation, Impact) model of planning and measuring PR programmes. It was the most widely taught process model until the late 1990s.

Analysis of the ‘program research and evaluation’ sub-section of Cutlip’s bibliography of public relations research shows that of the 159 articles listed from 1939 to the early 1960s, the largest group (67) were concerned with opinion research, including employee studies (Cutlip, 1965). This was followed by a cluster of papers with topics such as public relations, promotional activity (including advertising), publicity research and measurement (31) and research methods and surveys (28). Media measurement (including press, film, TV, radio and mass media in general), which was soon to become the dominant area of public relations measurement and evaluation, had only produced 15 papers in a quarter of a century. Within the range of papers, there was little discussion of the methodology of measuring public relations activity or programmes, with the main emphasis on objective setting based on opinion research. Cutlip’s summaries did not offer any references to specific methodology, other than one example of a rating system. The bibliography thus demonstrates the change in the practices of public relations and its measurement at the period of change from the social science-led approach to planning to the publicity-led communications that have been identified by L’Etang (2004) and others.

7. But it’s very difficult

Despite the emphasis placed on measurement by the IPR in the UK and leading US texts, many pre-1980 texts reveal great reluctance by practitioners to evaluate the outcomes of their activity. James E. Grunig commented in 1983 that practitioners are “not scientists at all although they should (but few do) use theories and research on public relations and communications” (p. 28). To illustrate the reluctance of the times, here are some statements drawn from the literature ranging from the 1930 to the mid-1960s. Some say, like Ivy Lee, that PR cannot be measured whilst others like Marston, whose 1963 textbook was widely used, say time and cost deter them.

“The counselor works to better a firm’s reputation, but the improvement can rarely be satisfactorily measured” (Tedlow, 1979, p. 160 writing about 1930s and 1940s). “Few practitioners will claim they can prove their efforts have paid off for their clients or companies” (Finn, 1960, p. 130). “Most public relations men, faced with the difficulty and cost of evaluation, forget it and get on with the next job” (Marston, 1963, p. 176). “Measuring public relations effectiveness is only slightly easier than measuring a gaseous body with a rubber band” (Burns W. Roper, cited in Marston, 1963, p. 289).

In the UK, views were very similar. The first is from James Derriman, later a president of the IPR in 1973–1974. Two others come from the most prolific British writers of PR texts – Frank Jefkins and Sam Black, the latter becoming an honorary professor at Stirling for his role in establishing its MSc in Public Relations.

“It is often hard to assess (achievement of objectives) with precision or identify effects of public relations” (Derriman, 1964, p. 198). ““Results” is something of a dirty word in PR” (Jefkins, 1969, p. 219). “The results of public relations activity are very difficult to measure quantitatively . . . it may be uneconomic to devote too much time and too many resources” (Black, 1971, p. 98).

The reluctance to evaluate was a feature of studies of public relations practice over coming decades. Watson (1994) found similar attitudes in a large-scale study of UK practitioners which included comments such as: “PR is not a science; most practitioners are inadequate; clients are too thick,” and “the best evaluation of results is when the client is pleased, satisfied, happy and renews the contract. All else is meaningless” (Appendix 2). Although practitioners expressed the desire to evaluate, the reality was that they lacked the knowledge, time and budget to undertake the task, much like their predecessors 30 years earlier.

8. 1950s and 1960s

The Institute of Public Relations published its first book – A guide to the practice of public relations in 1958. Although it stated that public relations is “an essential part of management” (p. 17), the book was mostly concerned with craft aspects

394 T. Watson / Public Relations Review 38 (2012) 390– 398

such as writing, media relations, event creation and management. It gave one short paragraph to monitoring press enquiries and handed the chapter on market research to a non-PR market research specialist. In a slightly later book, then IPR President Alan Eden-Green, writing the foreword in Ellis and Bowman’s Handbook of Public Relations in Eden-Green (1963), posits PR as being “primarily concerned with communication” (Ellis & Bowman, 1963, foreword). Other texts at the time also focus on processes, but not planning, measurement or outcomes.

In Germany, Albert Oeckl in 1964 proposed three methods of research – publics and how they use media, content analysis and research on media effects. He was much more linked to the Bernaysian social science of PR than were UK practitioners (Oeckl, 1964).

However, beyond texts and articles, Advertising Value Equivalents (AVE) was used to put a value on media coverage, which emphasised the craft nature of PR. The first warning against AVE came in a 1949 edition of the IPR Journal (J. L’Etang, personal communication, January 10, 2011). Plackard and Blackmon (1947) also refer to it in the US, with both indicating that it was an established practice by mid-century. It did not, however, surface in professional or quasi-academic literature till the late 1960s.

9. Increasing discussion

The late 1960s and the 1970s were periods when books and articles addressing public relations evaluation started to appear. Measuring and Evaluating Public Relations Activities was published by the American Management Association (1968). It had seven articles on methods of measuring public relations results. It is notable that it came from the American Manage- ment Association, and not a public relations professional body. Soon after, Robinson’s Public Relations and Survey Research (Robinson, 1969) was published. Pavlik says that “(Robinson) predicted that PR evaluation would move away from seat- of-the-pants approaches and towards “scientific derived knowledge” (Pavlik, 1987, p. 66). He added that Robinson was suggesting practitioners would no longer rely on anecdotal, subjective measures of success, such as feedback from per- sonal contacts or winning awards; they “would begin to use more systematic measures of success, primarily social science methods such as survey research” (ibid.). Academics then began taking the lead. A conference in 1977 at the University of Maryland chaired by James Grunig, partnering with AT&T, was followed by the first scholarly special issue, ‘Measur- ing the Effectiveness of Public Relations,’ in Public Relations Review’s Winter 1977 edition, which featured papers from the conference.

10. Rise of PR service industries

US industry veteran Mark Weiner has recently commented (M. Weiner, personal communication, February 16, 2011) that a key reason for the introduction of measurement services was that industry growth in the 1960s and 1970s could support it. By then, the US public relations consultancy networks pioneered by John Hill of Hill & Knowlton (Miller, 1999) and Harold Burson of Burson-Marsteller were widening their spread of offices and services to work for US-owned multi- nationals. They needed world-wide monitoring and management systems that gave systematic data back to HQs. Consumer public relations rapidly developed in the 1950s and 1960s in the post war economic boom, aided by the widespread access to television which had also fostered advertising’s expansion. University studies which had started in the US in the 1940s, although Edward Bernays claims to have taught the first public relations class at New York University in 1923 (Bernays, 1952), were growing in North America and other countries. These developments – national and international expansion of public relations activity and budgets, along with education and training to meet demand – led to the emer- gence of the service industries, especially in the measurement of PR activity. One of the first evaluators was PR Data, which was formed from an internal General Electric operation by Jack Schoonover and the first to use computer based analysis – using punch-cards and simple programmes (Tirone, 1977). It was soon followed by other providers, mainly press cuttings agencies which became evaluators. The UK development in this field did not, however, come for another 20 years.

11. 1980s – academic input

Following on from the initial conference and academic journal discussion late in the previous decade, US journals came alive in the 1980s with papers from leading academics such as Glenn Broom (Broom & Dozier, 1983), David Dozier (Dozier, 1984, 1985), James Grunig (Grunig and Hickson, 1976; Grunig, 1979, 1983). From the consultancy side, Lloyd Kirban of Burson Marsteller (Kirban, 1983) and Walter Lindenmann of Ketchum (Lindenmann, 1979, 1980) were prolific and drove the subject higher on the practitioner agenda, whilst from the media analysis side, Katie Delahaye Paine announced her first publicity measurement system in 1987 and went on to establish the Delahaye measurement business. In the UK, White (1990) undertook the first study of practitioner attitudes amongst member consultancies of the Public Relations Consultants Association (PRCA) and offered recommendations on ‘best practice.’ In 1990 Public Relations Review had a seminal special edition on evaluation, ‘Using Research to Plan and Evaluate Public Relations’ (Public Relations Review – Summer, 1990). Widely cited, it showed that measurement and evaluation were consistently part of academic and professional discourse. All these authors emphasised the need for public relations to be researched, planned and evaluated using robust social science

T. Watson / Public Relations Review 38 (2012) 390– 398 395

techniques. It was particularly fostered by Broom and Dozier’s influential Research Methods in Public Relations (Broom & Dozier, 1990).

12. 1990s – pace increases

As the 1990s proceeded, evaluation became a major professional and practice issue that was addressed by research and education activity in many countries which produced books, methods of analysis and proliferating international initiatives. In the US, the Institute for Public Relations Research and Education (now the Institute for Public Relations, IPR), harnessing Walter Lindenmann’s enthusiasm, published research and commentaries on establishing objectives and assessing results. The International Public Relations Association (IPRA) published its Gold Paper No. 11: Public Relations Evaluation: Profes- sional Accountability in 1994. In Europe, the German public relations association and the International Communications Consultants Organisation (ICCO) held a pan-European summit on evaluation in 1996, whilst the Swedish PR body, Svenska Informationsforening, moved ahead of the debate at the time to report on Return on Communication, a form of Return on Investment that considered the creation of non-financial value through communications (SPRA, 1996).

The 1990s was also a decade when Quality Assurance (QA) approaches to production and the BS5750 or ISO9000 process standards became part of management language and discourse. Companies with QA certification wanted their suppliers, including public relations advisers, to also have the same standards of operation. In the US, the Six Sigma quality production- led business management system was applied to public relations and communication, notably at Motorola (where it was developed), General Electric and other manufacturing majors (Poole, 2000; Weiner, 2004).

The first UK consultancy to gain BS5750 was Countrywide Communications (now Porter Novelli) in 1993 (P. Hehir, personal communication, April 9, 2011), but other consultancies were slow to follow as the new QA standards were prepared for production-oriented businesses, rather than service industries like consultancy. To promote the discussion, a spin-off from IPRA, the International Institute for Quality in Public Relations (IQPR) was formed and prepared the Quality in Public Relations paper (Berth & Sjoberg, 1997). It included a section on measurement and evaluation as integral to the management of a public relations operation. By the end of the decade, the UK’s Public Relations Consultants Association (PRCA) developed the Consultancy Management Standard (CMS) with the assistance of a leading international QA certification body. This took the place of BS5750/ISO9000 as it had been prepared with the aim of improving the management and operations of consultancies, a different emphasis to the early QA certifications. It included an assessable commitment to the systematic use of measurement of programmes, thus embedding practices within the consultancy sector. CMS has been adopted world-wide by industry organisations (PRCA, 2011).

The late 1990s also saw the launch of extensive national campaigns to promote best practice in measurement and evaluation. Lindenmann’s paper on public relations measurement was widely used in the US. It established the terminology of three stages of evaluation – Output, Out-take and Outcome – that are almost universally used (Lindenmann, 2006). The public relations consultancy bodies, PRCA and ICCO, its international offshoot, published their own booklets and were followed by other industry bodies separately or cooperatively. The major UK initiative was PRE-fix, a partnership between PRCA and IPR (UK) with PR Week, the weekly trade magazine. It ran for three years and was accompanied by seminars, research, online resources and best practice case studies. AMEC, then the Association of Media Evaluation Companies, was formed as a UK trade body. It is now the International Association for Measurement and Evaluation of Communications with members in 38 countries, which also indicates the expansion of the measurement and media analysis service industry. In the US, the IPRRE formed the Commission on Public Relations Measurement and Evaluation in 1999, which plays a major role in undertaking practice based research and disseminating it.