Social science research endeavours to "(...) help citizens and policy-makers to understand the world better, with an eye to changing that world" (Gerring 2015: 47). Many social science disciplines aim at guiding policy. As an applied science, public administration fills this role by focusing on both the implementation of public policies and the management of public sector entities. Essentially, there are two routes whereby academic research which focuses on Canadian cases can influence public managers, decision makers, and broadly contribute to the accumulation of knowledge. First, the direct route consists of analyzing specific policies, programs and initiatives deployed by the Canadian federal government, provincial ministries and agencies, as well as municipal organizations. Second, the indirect route consists of participating among the international community of scholars who develop and test theories. Over time, these theories hopefully find their way into public administration textbooks, mandatory readings in MPA programs and other scientific articles, where they might later be applied by alums working in governments. The present research is focused on the second route: public administration scholarship about Canada that could have an eventual and more subtle influence. A study in 2000 found that Canadian public sector innovations were replicated worldwide at a much wider rate than American innovations (Borins 2000: 68-69). More recently, a report which assessed civil service effectiveness ranked Canada first among OECD countries (fourth after being adjusted for GDP) (InCiSE 2017). However, Canada is perceived as playing a less important part in exporting ideas from the Anglosphere than the UK, all the while trailing behind New Zealand and Australia (Pollitt 2015: 4). Considering this perception, the present article provides an assessment on the extent to which Canadian scholarship, as well as a scholarly inquiry about Canada, is contributing to the international conversation on public administration and the effort to improve how governments work.

Theories of public administration are generated and tested in various contexts, sometimes in comparative studies, but more often in single-country studies. Their applicability to the management of policies and programs is contingent upon their boundaries (Ashkanasy 2016). Hypothetically, a policymaker wishing to develop a theory-informed program or evidence-based policy in Canada, would certainly prefer to ensure that the core causal mechanism embedded within his or her prospective theory, would not be hampered by any foreign contextual elements which could jeopardize the planned implementation process. However, as previous studies show--as well as the results from our first analyses will suggest--for most recent public administration theoretical advances, there are few empirical studies from Canada and fewer studies included in articles taking stock of the field. Hence, our hypothetical policymaker would be cornered into assuming that theories developed within British, Australian, Dutch or Danish contexts, hopefully, also apply to Canada.

Something happened in 2000, and it does not look like a bug

The reader can consider the following three phenomena about Canadian scholarship in public administration. First, as revealed by the Web of Science (WoS) database, the five most cited articles to ever come from Canadian Public Administration (CPA) are Kernaghan (1993), Howlett and Rayner (1995), Anderson (1996), Lindquist (1992) and Boase (2000). (1) The same exercise reveals that the most cited articles in the history of the Australian Journal of Public Administration (AJPA) are more recent O'Flynn (2007), Head (2008), Bishop and Davis (2002), Rhodes and Wanna (2007), and Hodge (2004). Not only are they more recent, they are also cited more often: this pattern holds true if we continue past the lists' top five as well. Indeed, if we were to rank the pooled articles from CPA and AJPA in terms of citations, Kernaghan (1993) would be 12th instead of 1st in the CPA-only ranking, Howlett and Rayner (1995), would be at 30th instead of 2nd, Anderson (1996) would be 32nd instead of 3rd, Lindquist (1992) would be 35th instead of 4th, and Boase (2000) would be tied at 36th instead of 5th. Second, according to the regional breakdown of contributors to Public Administration Review since the 1960s (Ni, Sugimoto, and Robbin 2017 (2)), the relatively higher number of contributions from Canadian scholars over the Australians and Dutch stopped around the 2000s, and then was reversed. Third, in the past decade, a puzzling phenomenon occurred. Across the board, journals in public administration grew in scholarly influence, as measured by the growth of their impact factor. Figure 1 presents the trend for some of the current 47 journals in the SSCI; we left out other top journals who saw their already high impact factor double or triple over that decade.

The reasons for the stable impact factor of Canadian Public Administration is especially puzzling when compared to its Australian counterpart: the Australian Journal of Public Administration. The similarities in historical, political, cultural and administrative structures between the two countries have remained constant. AJPA's impact factor followed the international trend; CPA's has not. With three independent strategies, the remainder of this article pursues these puzzles by examining the state of Canadian public administration, the field and the journal (3), following a golden age that ended in the late 1990s.

Previous research on Canadian public administration research

Public administration is one of eighteen social science disciplines listed by Gerring (2012: 438). This study focuses solely on public administration and sets public policy aside, as we do not wish to revisit the ground covered by Montpetit, Rothmayr Allison and Engeli (2016: 771) who analyzed: "five generalist public policy journals with the highest H-index in the 2013 Public Administration ranking produced by Thomson Reuters": Policy Sciences, Journal of Public Policy, Policy Studies Journal, Journal of European Public Policy, and Governance. The authors found that "the article outputs of Canadians and Australians are similar in many ways" (2016: 774). In contrast, as this article shows, the same cannot be said for the state of Canadian public administration scholarship.

Few studies have analysed the state of public administration research in Canada in the past ten years. One study canvassed 48 public administration journals in 2004 and 2005 to identify the main topics of interest in the field (McConkey and Dutil 2006). The authors included public management studies and excluded policy studies. A total of 950 articles were surveyed. One hundred categories organized around ten topics were generated from that scanning of the literature. McConkey and Dutil (2006) also surveyed 312 Institute of Public Administration of Canada (IPAC) members to compare their priorities with the trending topics in academic research and observed some overlap. Although Canadian and foreign journals were analysed in that study, the findings were not broken down along national lines. Thus, direct comparisons between national communities were not possible.

The second study analysed every article published during the first fifty years of Canadian Public Administration. Wake Carroll and Kpessa (2007) provide common themes found in research, and present them by time period and editorial era. Since they analyzed a single journal, they could not compare the Canadian themes with the foreign ones. However, they compared their themes with the ones found by McConkey and Dutil. Thus, Wake Carroll and Kpessa (2007: 487) concluded that seven out of ten themes identified internationally did not correspond to the ones they found in CPA. This held true even when they limited their sample to recent articles: the research presented in CPA was not in sync with global public administration research. Research published in Canada was qualitatively different.

A third more recent study initially comparing the themes and key authors between eight generic management and business journals and public administration from 2000-2010 (Vogel 2014) were re-analyzed to assess the relative productivity of Canadian business schools and public administration schools. A total of 1025 articles were coded by Vogel (2014), (4) that is 489 articles in eight journals in public administration and eight in business. Bear in mind that Vogel's effort was not to present national contributions to those two disciplines. Our own analysis of Vogel's data shows that out of 489 articles from eight top public administration journals on both sides of the Atlantic, only three had a mention of "Canadian" or "Canada" in either the title, the abstract, or the keywords. More telling still, almost 10% of the 536 articles in Management or Business journals included at least one coauthor with an affiliation to a Canadian university. Finally, only 1.2% of the 489 articles in the public administration journals had at least one coauthor with an affiliation in a Canadian institution.

How influential is Canadian research compared to other national strands?

Recent studies on the research productivity of universities in public administration have converging results with the trends identified in the previous section. For instance, a public administration ranking of universities was done in 2014 (Williams, Slagle and Wilson 2014). Productivity was measured by a complex weighted algorithm which took into consideration journal impact factor as measured in the SSCI between 2006 and 2010, and the number of author affiliations. The authors included 40 journals in the public administration category, including public management and public policy journals. CPA and Canadian Public Policy were included in this list. The weighted Institutional Impact Final Ranking featured 100 universities, from the 1,078 accredited institutions. Dutch (6[R], 20th, 28th, 61st 64th, 83rd, 89th, 90th) and Australian universities (11th, 46th, 47th, 50th, 65th) fared better than Canadian ones (U of T 22nd, UBC 86th, SFU 93rd, U de M 98th). This ranking extends outside of the core public administration discipline to include policy journals as well. As such, the authors commented that contrary to frequent remarks about the inherent national biases and blind spots of international rankings, the findings did not seem biased towards universities from one nation (Williams, Slagle and Wilson 2014: 402).

Van de Walle and van Delft (2015) produced a more focused ranking than the previous one, setting aside policy journals in favour of core public administration journals. Thus, they attributed two lists of articles to the universities affiliated by the authors. The first list was generated by using articles in the SSCI public administration journals in 2012, from 2009 to 2013, which totalled to 7071 articles. The second list included 4409 articles from the SSCI public administration journals in 2006. The authors then ranked the top twenty universities according to the number of published articles. "The top 20 using the 2012 list contains (...) six British [institutions], two Dutch, and one each from Hong Kong, Australia, Denmark, and Canada" (Van de Walle and van Delft 2015: 99). One Canadian school does appear in the Ranking Based on the SSCI 2006 Journal List: the University of Toronto at fifteenth. No Canadian university can be found in the Ranking Based on the SSCI 2012 Journal List. Furthermore, three other top-20 lists are generated by restricting the articles from the top-4 journals or from reputable journals that do not have impact factors. There is no mention of Canadian schools among these lists. It should be noted that the unit of analysis is universities and not departments, and that the numbers are not adjusted for the size of the faculty. The authors concluded that "(...) the SSCI-based analysis highlights that public administration research is now a global enterprise, with institutions from across the world included in the ranking" (Van de Walle and van Delft 2015:102).

These comprehensive studies about the productivity and scholarly influence of universities and countries rest on citations and impact factors. Citation patterns have been used to study the interconnectedness of public administration to other social sciences (Wright 2011), and to organization theory in particular (Andrews and Esteve 2015). Citations reflect the individual choices of thousands of researchers who chose to select the specific articles upon which to build their research. Inversely, these choices also reflect the behaviour of researchers in a given field, who decided not to use an article in their own research. Thus, a citation count does signify that academics found a certain piece of research relevant enough to reference it in their own work (Meier and O'Toole 2012: 889). More importantly, it discriminates between researchers who are prolific, but not influential (Ruscio 2016: 905). As such, citations and impact factors are quantitative indicators of both quality and scholarly influence. Like any one indicator, their limits are well-documented (Lariviere, Gingras and Archambault 2006: 520-521). Some academics will simply interpret the metrics, instead of using their professional judgement and read the much-(or not so much)-cited research (Schrodt 2015:29). Additionally, "investigators who work in hot areas, whose work is particularly controversial, or who write about topics that appeal to very broad audiences have an advantage, perhaps sometimes an undue advantage, in number of citations" (Sternberg 2016: 879). Unlike CPA, many (but not all) of the most influential public administration journals have an extensive and suspiciously retentive list of forthcoming articles that take years before being assigned to an issue. The unforeseen and advantageous effect of this incubation period is the inflation of citations such articles accumulate before the two-year measuring period officially starts.

Typically, there are two sources of definitions for reputable journals: "lists of journals based on the Social Sciences Citation Index's (SSCI) Journal Citation Reports, and journal reputation surveys" (Van de Walle and van Delft 2015: 87). A reputational score was created from a 2007 survey of 185 editors, associate editors or managing editors and editorial board members of 39 public administration journals (Bernick and Krueger 2010), which replicated Forrester and Watson (1994). The results show that CPA scaled down to 17th out of 39 journals in 2007, from 9th out of 35 journals in 1994; while A]PA slightly climbed from 20th out of 35 journals in 1994 to 19th out of 39 journals in 2007. We are not aware of a more recent reputation ranking of public administration journals.

Simply put, impact factors are the average number of citations for each article published in one journal. It is calculated by: "dividing the number of current citations a journal receives to articles published in the two previous years by the number of articles published in those same years" (Amin and Mabe 2000: 2). It has the advantage of pooling the judgement of all the researchers in a field, which reduces individual biases (Ni, Sugimoto, and Robbin 2017: 496), rather than fewer well-connected editors and editorial board members. If circumscribed correctly, it is considered a measure of journal influence (Amin and Mabe 2002: 6). As such, an impact factor can enable inter-journal comparisons of relative influence in a given field, by assessing the number of citations its articles receive. However, comparisons are less straightforward across disciplines, as the size of the community is one factor that can influence the average citation count.

Main methods and data: a systematic review of systematic reviews

To push our analysis further, we present our own three-pronged approach in assessing the scholarly influence of public administration research done in Canada. Our main analysis looks at the proportion of studies with Canadian samples (qualitative and quantitative) that make their way into systematic reviews of literature on various topics in public administration journals. To that effect, we identified systematic literature reviews and meta-analyses in public administration journals published in 2013,2014,2015 and 2016, as well as articles that were forthcoming at the time of data collection in February of 2017. Systematic literature reviews and meta-analyses play important roles for practitioners and academic scholars alike. Bedard and Ouimet (2017: 178) observe that "in contexts where research findings can provide helpful policy inputs, systematic literature reviews and meta-analyses might be the best means for providing a valid viewpoint on the state of the literature on a precise question." Indeed, we are interested in the proportion of articles which analyze a sample of Canadian public agencies, ministries, programs, etc. Systematically identifying and analyzing systematic literature reviews and meta-analyses provides us with a panorama of panoramas of sorts. This method enables us to capitalise on their overarching reach to cast a wide enough net to avoid, as much as possible, any omissions. Most systematic, systemized or umbrella reviews include quality assessment of sources, while meta-analyses have quality assessment that "may determine inclusion/exclusion and/or sensitivity analyses" (Grant and Booth 2009: 94). This is the main element of our analysis. The advantage of letting dozens of scholars prune the initial results into a more restrictive list controlling for quality, is that the potential biases of authors are not carried from one review to the next. Thus, the results of our main analyses do not rest on citation counts or average citation counts that make up impact factors.

Results: the share of Canadian studies in systematic reviews and meta-analyses

First, we selected the 17 core journals form the public administration category in the SSCI; like Van de Walle and van Delft (2015), we left aside more policy-oriented journals. We then manually searched on the respective websites of our selected body of journals for papers published from January 2013 onwards to February 2017. We searched on the journal websites and retained papers with the following string: "systematic" or "literature" or "review" or "meta," in all fields OR title OR abstract OR keywords. Some titles did not mention those keywords but seemed relevant enough to merit further investigation. Those specific articles were downloaded and a manual CTRL+F search was done to assess if the article was relevant for this review or not. Generally, we found that running our string under the options of all fields and relevance generated the most relevant articles. To confirm our assumption, we then searched in title OR abstract OR keywords using the same search string. Those three search options usually narrowed the number of articles to less than one hundred. The relevant articles previously identified in the all fields search option would systematically reappear, while new potential articles surfaced.

In total, the first step generated 77 potential articles from 14 journals which were retrieved for our review. To be included, the related authors mentioned that they had produced a literature review and/or systematic literature review and/or meta-analysis at least in the title, abstract, keywords or methodology section of the article. From this list, 52 articles from 12 journals were selected after manually reviewing the primary pool of 77 articles. We further rejected non-systematic literature reviews. This step limits the possibility that some authors produce narrative reviews that leave aside Canadian content. Most of the systematic literature reviews did not present the national breakdown of their data. We corresponded with the authors to either get the identity of the article using Canadian data, or to get their dataset and identify the Canadian data ourselves. Collaboration was productive with authors, as only five did not respond to our queries or did not have the information we needed. The final sample includes 24 articles from ten journals.

The characteristics of the 24 systematic literature reviews and meta-analyses are presented in Table 1. Systematic literature reviews can include essays and empirical studies, qualitative or quantitative. Meta-analyses only compile and pool quantitative studies. At times, authors selected studies via databases. Other times, their selections were narrowed to particular journals. The number of articles about Canada is presented next to the number of articles analysed in the systematic literature reviews or meta-analyses. We also scanned the article to see if the words "Canada" or "Canadian" were mentioned. In the last column, additional contextual information pertinent to our research goals are provided.

Table 1 shows that studies about Canada are having a hard time carving out their share of scholarly influence or noticeability within systematic literature reviews and meta-analyses published in public administration journals. First, on a total of 3452 articles canvassed by the systematic reviews and meta-analyses, only 49 articles were about Canadian public organisations (1.42%). Furthermore, we must consider the possibility that references to Canada could be fortuitous rather than central to a given study. As we did in the total number, we correct for this risk by subtracting studies focused on South Asia, Africa, Europe and Eastern Europe. The ratio shifts to 45 studies with Canadian data out of 2211 articles (2.06%): inquiries about Canada remain far from being a significant part of the conversation. An example will illustrate our findings. Amid our results is featured Walker and Andrews' systematic review of local government performance, which was awarded the 2015 Beryl Radin Award for the best article in JPART. The authors canvassed a total of 86 articles among 490 empirical articles in SSCI journals, including CPA, between 1970 and 2012. One finding concerned where some studies came from: "(...) 28 based in the United Kingdom (England and Wales), 6 in other European countries, and one each in Israel, Pakistan, and South Korea" (Walker and Andrews, 2015:108).

Furthermore, studies using data from municipalities in Italy, Norway and Pakistan also contributed to taking stock of theories of performance. Canada is nowhere to be found, and although research has been done in Canada on the topic. It was excluded because Canadian articles did not meet the inclusion criteria set in the study: in this case, having a multiple regression analysis where performance is the dependent variable. This is reflects the wider tendencies that meta-analyses or systematic literature reviews centred on quantitative studies gathered even fewer studies than the more general systematic reviews that can include qualitative studies as well. Second, the discussion on public administration and public management rarely talks about Canada. Indeed, as only three mentions of "Canada" or "Canadian" are reported in the vast reach of the systematic reviews, one can presume that Canadian data are rarely present on the empirical front. This also indicates that the Canadian context is seldom included when the state of knowledge is crystalized in a systematic literature review. Third, in the event that the systematic reviews utilized a journal selection sampling method and not databases, Canadian Public Administration was included only three times, and was added as one of three non-UK/USA-oriented journals.

As presented earlier, this article seeks to assess if Canadian public administration scholarship is less or more influential than other strands, not why it is so. Some commentators to this research offered comments to the effect that contrary to other countries, much of the research about the public sector would not be present in public administration journals, but in journals in other disciplines, primarily political science. The argument is that in Canada, public administration scholars would be "trained outside of public administration and tend to publish in non-public administration journals," instead of being "trained in public administration and tend to publish in public administration journals" (Rodgers and Rodgers 2000: 435). For simplicity's sake, let's label that idea the "multidisciplinary displacement hypothesis": like a squeezed balloon in one's palm, research about the public sector that is absent from public administration "bulges" in other disciplines. The next two sections will empirically tackle this hypothesis, to see if there are indeed signs that this is the case.

Secondary methods and data

To complete our study, we analyse the raw data of citations produced by St.Clair, Hicks and Isett (2017) of the 70 articles with the most citations and the highest average yearly citations. Those then recent 70 articles from various public administration journals had been cited a total of 13,154 times. We focus our attention on the thirty journals citing these 70 articles with the words Canad* or Australia* in their title (ex. Canadian Public Administration, Australian Journal of Political Science, etc.). This provides a proxy for the relative connection of Canadian social science journals to the mainstream in public administration. Later, for our third analysis, we compare the relative impact factor of Canadian and Australian journals among twelve social sciences ranked by Thompson Reuters SSCI in 2016.

Secondary results: how connected to mainstream public administration are Canadian and Australian social science journals?

Short of extending previous CPA analyses by manually analysing a plethora of other journals, (5) an efficient compromise consists in taking a focused sample of all publications by targeting the fields' best sellers. Thus, a most-likely case consists in observing to what degree the most influential articles are referred to by articles published in Canadian Public Administration and other social science journals. To get a sense of the task, we asked St. Clair, Hicks and Isset (2017) (6) and obtained the raw data used to support a recent research which examined the articles with both the most citations and the most yearly citations from all journals in public administration from 1997 to 2015. This creates a proxy to assess the level of connectedness of Canadian and Australian social scientists to mainstream global public administration. These 70 most-cited articles were quoted for a total of 13,154 times. Of those, 218 citations with either the word Canada or Canadian or Australia or Australian in their title originated from one of the thirty journals. To that effect, we sampled "national" social sciences journals. (7) Table 2 presents the relative frequencies. This citation-count proxy of influential studies provides us with a broad idea of how social science journals are connected to contemporary mainstream public administration.

The first line of Table 2 presents the number of times these 70 highest-cited articles appeared in articles published respectively in the Australian Journal of Public Administration, 85 times, and in Canadian Public Administration, 43 times. This amounts to almost twice as much articles published in AJPA mentioning these articles compared to CPA's articles. A possible rebuttal would point to the "multidisciplinary displacement hypothesis" and opine that Canadian PA scholarship is more multi-disciplinary than in other countries. The bottom half of Table 2 suggests otherwise: the relative lack of connection to mainstream PA found in Canadian Public Administration does not appear in equal number to Australian journals in other Canadian journals. Indeed, there appear to be no "bulges" in other Canadian journals to compensate for the relative lack of connection between articles published in CPA and mainstream research. That said, there is some merit in the observation that political science in Canada is more preoccupied with public administration than in Australia. However, this additional attention is relatively small in absolute numbers, and does not compensate for the numbers of mainstream public administration research in Canada. Overall, all of the Canadian journals were less connected to these 70 highest cited papers in public administration, not just CPA. This highly targeted analysis reverberates with the findings from earlier research about the disconnection of themes between Canadian research and the rest of the international community (McConkey and Dutil 2006; Wake Carroll and Kpessa 2007).

Tertiary analysis: the relative impact factor of Canadian and Australian journals

A second test for the "multidisciplinary displacement hypothesis," alleging that research in public administration is displaced and featured in other disciplinary journals, is to look if the lack of scholarly influence in CPA is counterbalanced by an excess in other disciplines. This particular section is the sole empirical analysis which rests upon impact factors. Table 3 presents the ratio between the impact factors of Canadian to Australian journals in twelve social sciences in 2016. For example, in the first line of the table, the impact factor for Canadian Public Administration is three times smaller as the Australian Journal of Public Administration: CPA's impact factor measures to 31.1% of AJPA's. By comparing this ratio, we can either get a sense if the hypothesised phenomenon is limited to public administration published in CPA, or if it is present in other Canadian flagship social sciences journals as well.

The first obvious result is that the relative place of Canadian Public Administration to the Australian Journal of Public Administration is an acute case of a wider Canadian phenomenon. Across twelve disciplines, there is only one example where a Canadian journal is cited more widely on average than its Australian counterpart. The relative gap between the influence of Australian and Canadian journals is less pronounced in some disciplines than in others, but it is nevertheless a constant. For several disciplines like social work, international relations, education and anthropology, the Canadian journals are not even indexed in the SSCI. The opposite is not true: there were no cases of Canadian journals who could not be compared to their Australian counterparts because the latter were not ranked. Results consequent with the "multidisciplinary displacement hypothesis" would have shown that the large impact factor difference in public administration is counterbalanced by excesses in other social sciences. This is not what we find.

Discussion and conclusion

The aim of this article has been to assess the scholarly influence of public administration research done about Canada in respect to other national strands using the Australian example as a most similar case to start the comparison. After presenting these measures, we now hope to launch a debate about the discipline in Canada and get a better understanding of the meaning of the numbers presented, and, eventually, hopefully get Canadian public administration back to punching in its rightful weight class. Our main analysis is centred on the share of studies about Canada by performing a systematic review of recent systematic reviews. One must keep in mind that systematic reviews and meta-analyses document topics where a sizable body of work already exists. In the event where Canadian researchers have been doing pioneering work in less widely-known or popular areas of inquiry, their efforts would not be registered within the reviews listed in Table 1. Systematic reviews can exclude articles that do not meet inclusion criteria, but they cannot include articles that have not been written. (8) That covered research about Canada no matter where it is published.

Our second and third analyses were centred on cpa, the flagship Canadian journal in public administration, but also on other Canadian journals in the social sciences. That covers research about Canada and published in Canada. Our results from these two analyses, as different as they were, converge among themselves, as well as with the literature. Research done about Canada is seemingly of a different nature and is not cited often. Our findings should not be misconstrued as a critique of the flagship journal or its editorial staff. They are managing content and selecting manuscripts; they are not producing said manuscripts.

Almost twenty years ago, Savoie (1999: 6) lamented that public administration suffered from theoretical malnutrition. It is unclear if it was true then, but it is not true now. If Savoie's (1999) malnutrition metaphor still holds true today, Canadian studies and empirical data seldom appear in the list of ingredients for mainstream theories found in public administration journals. The literature and our results point to the conclusion that in today's debates about what works, what should be done and what has been learned in the discipline of public administration, it is done with relatively few Canadian inputs.

How could we explain this situation? At this point, we cannot. Before addressing causality, one must first build a description of the phenomenon and subsequently, establish the presence of a correlation. The goal of our research was to assess if Canadian scholarship was, or was not, influential with international scholars. We had no room in the article to inquire as to why it is so. Building our descriptive case as supplemented by our independent analyses, proved to be a hotly debated article-length endeavour. However, our second and third analyses did rule out some explanations. Time will tell if the readers of CPA will be as surprised as the reviewers who graciously commented on this manuscript. Nevertheless, at the request of our five reviewers, we propose possible explanations that are not firmly based on our results.

1. It is possible that like Canadian foreign policy research (Black and Smith 2014:147), public administration researchers from Canada and contributors to CPA might value solving real-world problems instead of testing and developing theories. Therefore, there could be a large proportion of atheoretical articles (Maliniak et al. 2011) written in and about Canada, which would explain the practical-oriented bias of Canadian scholarship as Savoie was complaining.

2. Another explanation which could limit the inclusion of studies with Canadian samples in systematic literature reviews is methodology. Contrary to British public administration research (Hood 2011), Canadian public administration research may have yet to experience "phoenix"-type developments, that is, the "refinements in method and analysis that go well beyond the traditional practico-descriptive approach to PA" (Hood 2011:132). Because systematic literature reviews filter studies according to the quality of their methodology, it is possible that the bulk of Canadian studies get sifted out. It would take further studies to systematically analyze the methods used by Canadian public administration scholars and conclude if they differ or even lag behind the methods used by their counterparts in other countries. (9)

3. The Transylvanian Review of Administrative Sciences, AJPA and CPA were described as having a "strong regional focus" (Van de Walle and van Delft 2015: 94). However, this regional-oriented bias does not explain why AJPA is more heavily cited, and more in-tuned with international themes than Canadian Public Administration, as our second analysis revealed. In its aims and scopes, it is mentioned that CPA "(...) focuses mainly on Canadian issues but also welcomes manuscripts which compare Canadian public sector institutions and practices with those in other countries or examine issues in other countries or international organizations which are of interest to the public administration community in Canada." This policy contrasts with another journal like Public Administration Review, under the umbrella of the American Society for public administration, where the "the majority of 2016 PAR authors were not from the United States" (Ni, Sugimoto and Robbin 2017: 504). CPA's editorial policy has invited the publication of articles about Canada, but it might also shield Canadian scholars from expectations, methodological and otherwise, present in other journals. Such a trade-off between local concerns and openness to international topics has effects.

4. As the main outlet for public administration research about Canada, an element that potentially limits citation counts in Canada might be the language gap with Administration publique du Canada. There is a stark difference in citation patterns between French and English-language articles in CPA-APC. Since 2007, among the fifty most cited articles, only one of them is in French, at the 49th rank. There are few articles in French published in APC. The fact that the Parenteau prize is not awarded every year is but one proof, aside from counting French-language articles. These few articles in French are not numerous enough to weight down CPA's impact factor.

5. Is it that the Canadian scholarly community in public administration is small, which impacts its influence? Currently, there are 221 members in IPAC's academic section. That is hard to determine if that is a small number or a large number. Accordingly, we listed all the unique authors who published articles in CPA and in AJPA in 2009,2011,2013,2015 and 2017. The 136 articles published in CPA over that period were coauthored by 212 unique authors (1.56 unique author/article); the 169 articles published in AJPA during these same five years were authored by 292 unique authors (1.72 unique author/article). We do not think the size of the community can explain the many gaps we illustrated in this article between Canadian and Australian research.

6. The extent of research funding could be linked to the results of our third analysis. According to Web of Science, since 2000, there have been 46 articles which mention the support of funding in A/PA, compared to 21 articles in CPA. This is a stark contrast. However, these funded studies represent 6.9% of studies in AJPA and 4.8% of studies in CPA for that period. Nevertheless, given that citation patterns are typically highly asymmetrical--where a few 'best sellers' get a lot of attention and citations, as opposed to most studies that do not get cited at all--it is possible that the additional funded studies in AJPA could have generated more attention. This explanation rests on the assumption that this funding produced superior articles with expensive added-value characteristics, such as expanding data coverage or enabling a longitudinal aspect to a study, instead of simply delegating menial tasks to research assistants. One source of funding in Canada are SSHRC programs refereed by way of "political science and public administration" committees. Fifteen years ago, Borins (2003: 252) opined that "(u)ntil the current year, SSHRC considered public administration research proposals in the same committee as proposals in law and political science, and the feeling among public administration researchers is that the committee has not been especially receptive to their proposals." It remains unknown if public administration scholars' treatment has changed since then. However, it is telling that according to Web of Science, in the last ten years, eight articles in CPA declared funding from the SSHRC, compared to 23 in the Canadian Journal of Political Science and 19 in Canadian Public Policy.

If proven viable, these six possible explanations could mitigate our findings, but they cannot explain them away. We have yet to come across an explanation that could account for the different trends identified earlier in the article, as well as in the literature, and the results of our analyses. We hope to spark a debate about how to improve the visibility of research on Canadian public administration.

Etienne Charbonneau is Canada Research Chair in Comparative Public Management, ENAP, Montreal, Quebec. Luc Bernier is Jarislowsky Chair in Public Sector Management, University of Ottawa. Nicholas Bautista-Beauchesne is doctoral candidate at ENAP.

Notes

(1) We scoured the recent literature and the back issues from CPA for sources. We tried to put our hands on as many databases as we could; we tried to study the issue from many angles. What we present here, and the additional analyses that were submitted to reviewers and later withdrawn, constitute what we found. We did not cherry pick our evidence or our references.

(2) We thank Ni, Sugimoto and Robin for sharing for answering our query.

(3) We follow Dwight Waldo's rule of using public administration to identify the practice, and public administration to identify the academic discipline. Also, the italicized Canadian Public Administration refers to the academic journal, while the non-italicized version refers to the day-to-day in the Canadian state.

(4) We thank Rick Vogel for sharing his dataset with us.

(5) This would be an article on its own. We would know, as we started doing just that.

(6) We thank Rebekah St. Clair, Diana Hicks, and Kimberley Isett for sharing their dataset with us.

(7) It bears mentioning here that although national journals are not all that national, neither is national research performance. As found by Schneider and Sorensen (2015: 10), in all sciences (including hard, medical, social sciences, but also humanities), close to half of articles published in 2010 had transnational authorship. We thank Schneider and Sorensen for sharing this insight with us.

(8) We ran analyses in WoS, and in the past decade, Canadians scholars wrote much fewer articles in SSCI journals than Australians scholars. This fact does not impact our second and third analyses, but it plays a role in the results of our first analysis.

(9) This would be an article on its own. We would know, as we did just that.

References

Amin, M., and Mabe, M. 2000. "Impact factors: Use and abuse." Medicina 63 (4): 347-354.

Anderson, George. 1996. "The new focus on the policy capacity of the federal government." Canadian Public Administration 39 (4): 469^88.

Andrews, R., and Esteve, M. 2015. "Still like ships that pass in the night? The relationship between public administration and management studies." International Public Management Journal 18 (1): 31-60.

Ashkanasy, N. M. 2016. "Why we need theory in the organization sciences." Journal of Organizational Behavior 37 (8): 1126-1131.

Bedard, P.-O., and Ouimet, M. 2017. "Awareness and use of systematic literature reviews and meta-analyses by ministerial policy analysts." Canadian Public Administration 60 (2): 173-191.

Bernick, E., and Krueger, S. 2010. "An assessment of journal quality in public administration." International Journal of Public Administration 33 (2): 98-106.

Bishop, Patrick, and Davis, Glyn. 2002. "Mapping public participation in policy choices." Australian Journal of Public Administration 61 (1): 14-29.

Black, D. R., and Smith, H. A. 2014. "Still Notable: "Reassessing Theoretical 'Exceptions' in Canadian Foreign Policy Literature"." International Journal 69 (2): 133-151.

Boase, Joan Price. 2000. "Beyond government: The appeal of public-private partnerships." Canadian Public Administration 43 (1): 75-92.

Borins, S. 2000. "What border? Public management innovation in the United States and Canada." Journal of Policy Analysis and Management 19 (1): w46-74.

Borins, S. 2003. "From Research to Practice: A Survey of Public Administration Scholars in Canada." Canadian Public Administration, 46 (2): 243-256.

Forrester, J. P., and Watson, S. S. 1994. "An Assessment of Public Administration Journals: The Perspective of Editors and Editorial Board Members." Public Administration Review 54 (5): 474-482.

Gerring, J. 2012. Social Science Methodology: A Unified Framework. 2nd ed. New York, NY: Cambridge University Press.

--. 2015. "The relevance of relevance." In In The Relevance of Political Science, edited by B. Gerry Soker, Guy Peters and Jon Pierre; 36-49. Basingstoke, UK: Palgrave Macmillan.

Grant, M. J., and Booth, A. 2009. "A typology of reviews: An analysis of 14 review types and associated methodologies." Health Information and Libraries Journal 26 (2): 91-108.

Head, Brian W. 2008. "Three lenses of evidence-based policy." Australian Journal of Public Administration 67 (1): 1-11.

Hodge, Graeme A. 2004. "The risky business of public-private partnerships." Australian Journal of Public Administration 63 (4): 37-49.

Hood, C. 2011. "It's public administration, Rod, but maybe not as we know it: British public administration in the 2000s." Public Administration 89 (1): 128-139.

Howlett, Michael, and Rayner, Jeremy. 1995. "Do ideas matter? Policy network configurations and resistance to policy change in the Canadian forest sector." Canadian Public Administration 38 (3): 382^10.

InCiSE. 2017. The International Civil Service Effectiveness (InCiSE) Index. Oxford, UK: Blavatnik School of Government & the Institute for Government.

Kernaghan, Kenneth. 1993. "Partnership and public-administration: Conceptual and practical considerations." Canadian Public Administration 36 (1): 57-76.

Lariviere, V., Gingras, Y., and Archambault, E. 2006. "Canadian collaboration networks: A comparative analysis of the natural sciences, social sciences and the humanities." Scientometrics 68 (3): 519-533.

Lindquist, Evert A. 1992. "Public Managers and policy communities: Learning to meet new challenges." Canadian Public Administration 35 (2): 127-159.

Maliniak, D., Oakes, A., Petgrson, S., and Tierney, M. J. 2011. "International relations in the US academy." International Studies Quarterly 55 (2): 437-464.

McConkey, M., and Dutil, P. 2006. The Top Ten Topics in Public Administration Scholarship: An International Perspective. Ottawa, ON: New Directions. Institute of Public Administration of Canada.

Meier, K. J., and O'Toole, L. J. J. 2012. "Influence: What kind? And how to know?" Administration & Society 44 (7): 888-893.

Montpetit, E., Rothmayr Allison, C., and Engeli, I. 2016. "Has Simeon's vision prevailed among Canadian policy scholars?" Canadian Journal of Political Science 49 (4): 763-785.

Ni, C., C. R. Sugimoto, and Robbin, A. 2017. "Examining the evolution of the field of public administration through a bibliometric analysis of public administration review." Public Administration Review 77 (4): 496-509.

O'Flynn, Janine. 2007. "From new public management to public value: Paradigmatic change and managerial implications." Australian Journal of Public Administration 66 (3): 353-366.

Pollitt, C. 2015. "Towards a new world: Some inconvenient truths for Anglosphere public administration." International Review of Administrative Sciences 81 (1): 3-17.

Rhodes, R. A. W., and Wanna, John. 2007. "The Limits to Public Value, or Rescuing Responsible Government from the Platonic Guardians." Australian Journal of Public Administration 66 (4): 406-421.

Rodgers, R., and Rodgers, N. 2000. "Defining the boundaries of public administration: Undisciplined mongrels versus disciplined purists." Public Administration Review 60 (5): 435-445.

Ruscio, J. 2016. "Taking advantage of citation measures of scholarly impact: Hip Hip h Index!." Perspectives on Psychological Science 11 (6): 905-908.

Savoie, D. J. 1999. Governing from the Centre: The Concentration of Power in Canadian Politics. Toronto, ON: University of Toronto Press.

Schneider, J. W., and Serensen, M. P. 2015. Measuring Research Performance of Individual Countries: The Risk of Methodological Nationalism. Montreal, QC: Paper presented at the European Consortium for Political Research.

Schrodt, P. A. 2015. If I Ran the University.. Penn State working paper. PA: State College.

St. Clair, R., Hicks, D., and Isett, K. R. 2017. "An investigation into the characteristics of papers with high scholarly citations in public administration: The relativity of theory and method." Review of Public Personnel Administration 37 (3): 323-350.

Sternberg, R. J. 2016. ""Am I Famous Yet?" Judging Scholarly Merit in Psychological Science: An Introduction." Perspectives on Psychological Science 11 (6): 877-881.

Van de Walle, S., and van Delft, R. 2015. "Publishing in public administration: Issues with defining, comparing, and ranking the output of universities." International Public Management Journal 18 (1): 87-107.

Vogel, R. 2014. "What happened to the public organization? A bibliometric analysis of public administration and organization studies." American Review of Public Administration 44 (4): 383-408.

Wake Carroll, B., and Kpessa, M. W. 2007. "Enduring, ephemeral and emerging issues in public administration in Canada: Trends in Canadian public administration over fifty years (1958-2007)." Canadian Public Administration 50 (4): 477-491.

Walker, R. M., and Andrews, R. 2015. "Local Government Management and Performance: A Review of Evidence." Journal of Public Administration Research b Theory 25 (1): 101-133.

Williams, A. M., Slagle, D. R. and Wilson, D. 2014. "Ranking universities for scholarship in public administration research 2006-2010." Journal of Public Affairs Education 20 (3): 393-412.

Wright, B. E. 2011. "Public administration as an interdisciplinary field: Assessing its relationship with the fields of law, management, and political science." Public Administration Review 71 (1): 96-101.

Caption: Figure 1. Evolution of the Impact Factor of National Flagship Journals in Public Administration in the Past Decade.

Table 1. The Canadian content of systematic literature reviews and
meta-analyses in public administration, 2013-early 2017

Journal   Year     Author(s)         Title

ARPA      2014     Walker, Brewer    Public Administration
                   & Choi            Research in East and
                                     Southeast Asia: A
                                     Review of the
                                     English Language
                                     Evidence, 1999-2009
ARPA      2017     Munoz, Bolivar    Transparency in
                   &                 Governments: A
                   Hernandez         Meta-Analytic
                                     Review of Incentives
                                     for Digital Versus
                                     Hard-Copy Public
                                     Financial
                                     Disclosures
ARPA      2014     Gao               Public Administration
                                     Research in Hong
                                     Kong and Macau: A
                                     Review of Journal
                                     Articles Published
                                     From 1999 to 2009
A&S       2014     Kennedy           Unraveling
                                     Representative
                                     Bureaucracy: A
                                     Systematic Analysis
                                     of the Literature
IPMJ      2017     Cappellaro        Ethnography in Public
                                     Management
                                     Research:A
                                     Systematic Review
                                     and Future
                                     Directions
IRAS      2017     Wessels &         The Eligibility of Public
                   Visagie           Administration
                                     Research for Ethics
                                     Review: A Case
                                     Study of Two
                                     International
                                     Peer-Reviewed
                                     Journals
IRAS      2016     Meijer &          Governing the Smart
                   Rodriguez         City: A Review of the
                   Bolivar           Literature on Smart
                                     Urban Governance
IRAS      forth-   Vigan &           Job Satisfaction in
          com-     Giauque           African Public
          ing                        Administrations: A
                                     Systematic Review
JPART     2015     Walker &          Local Government
                   Andrews           Management and
                                     Performance: A
                                     Review of Evidence
JPART     2015     Tummers,          Coping During Public
                   Bekkers,          Service Delivery: A
                   Vink &            Conceptualization
                   Musheno           and Systematic
                                     Review of the
                                     Literature
PA        2016     De Vries,         Innovation in the
                   Bekkers &         Public Sector: A
                   Tummers           Systematic Review
                                     and Future Research
                                     Agenda
PA        2014     Kuipers, Higgs,   The Management of
                   Kickert,          Change in Public
                   Tummers,          Organizations: A
                   Grandia, Van      Literature Review
                   Der Voet

PAR       2015     Homberg,          Meta-Analysis of the
                   McCarthy &        Relationship
                   Tabvuma           between Public
                                     Service Motivation
                                     and Job Satisfaction
PAR       2016     Gerrish           The Impact of
                                     Performance
                                     Management on
                                     Performance in
                                     Public
                                     Organizations: A
                                     Meta-Analysis
PAR       2016     Ritz, Brewer &    Public Service
                   Neumann           Motivation: A
                                     Systematic Literature
                                     Review and Outlook

PMM       2015     Leggat,           Have process redesign
                   Bartram,          methods, such as
                   Stanton,          Lean, been
                   Bamber &          successful in
                   Sohal             changing care
                                     delivery in hospitals?
                                     A systematic review
PMR       2015     Sutton, Eborall   Patient Involvement in
                   & Martin          Patient Safety:
                                     Current experiences,
                                     insights from the
                                     wider literature,
                                     promising
                                     opportunities?
PMR       2014     Lecy, Mergel &    Networks in Public
                   Schmitz           Administration:
                                     Current Scholarship
                                     in Review

PMR       2015     Torchia,          Public-Private
                   Calabro, &        Partnerships in the
                   Morner            Health Care Sector:
                                     A Systematic Review
                                     of the Literature
PMR       2015     Dan & Pollitt     NPM Can Work: An
                                     optimistic review of
                                     the impact of New
                                     Public Management
                                     reforms in central
                                     and eastern Europe
PMR       2015     Vogel & Ma sal    Public Leadership: A
                                     review of the
                                     literature and
                                     framework for future
                                     research
PMR       2016     Overman           Great Expectations of
                                     Public Service
                                     Delegation: A
                                     Systematic Review
PPMR      2013     Pollitt & Dan     Searching for Impacts
                                     in Performance-
                                     Oriented
                                     Management Reforn
PPMR      2015     Kroll             Drivers of Performana
                                     Information Use:
                                     Systematic Literature
                                     Review and
                                     Directions for Future
                                     Research
Total     --       --                --

Journal   Period       Data            Selected journals     Total
                                                            articles

ARPA      1999-2009    Sys. Review     No 29 SSCI             309
                       Quant & Qual    journals

ARPA      1980-2015    Meta-analysis   n/a 11 databases        51
                       Quant

ARPA      1999-2009    Meta-analysis   n/a SSCI database     668 *
                       Quant

A&S       1944-2011    Sys. Review     n/a JSTOR and           93
                       Everything      Google scholar

IPMJ      1990-2014    Sys. Review     No SSCI database        70
                       Everything      and IPMJ,
                                       JPART, PA, PAR,
                                       PMR, Gov, A&S,
                                       PM&M, JPAM,
                                       RoPPA
IRAS            2012   Sys. Review     No just two             70
                       Quant & Qual    journals: IRAS,
                                       PAR

IRAS      1999-2012    Sys. Review     n/a Databases           51
                       Everything

IRAS      1990-2014    Sys. Review     n/a Two databases      22 #
                       Quant & Qual

JPART     1970-2012    Sys. Review     Yes                     86
                       Quant.          SSCI Public
                                       Administration
                                       journals
JPART     1981-2014    Sys. Review     No Google Scholar       67
                       Quant & Qual    and Gov, JPART,
                                       PS, PA, PAR

PA        1990-2014    Sys. Review     n/a two databases      181
                       Quant & Qual

PA        not          Sys. Review     Yes Databases,         133
          mentioned    Quant & Qual    Gov, JPART, PS,
                                       PA, PAR CPA,
                                       IRAS, CPAT

PAR       1990-2013    Meta-analysis   No JPART, PAR,          28
                       Quant           ARPA, IPMJ, PA,
                                       RoPPA

PAR       1988-2014    Meta-analysis   n/a Google Scholar      49
                       Quant

PAR       1990-2014    Sys. Review     Yes A&S, ARPA,         323
                       Everything      IRAS, PP&A,
                                       RoPPA, IPMJ,
                                       JPART, PA, PMR,
                                       PAR, PPM

PMM       1995-2013    Sys. Review     No Google               41
                       Quant           scholars + some
                                       healthcare
                                       journals

PMR       1990-2013    Sys. Review     n/a Scopus              27
                       Everything      database

PMR       1987-2013    Sys. Review     No APMH, MHSR,          82
                       Quant &         A&S, AJPA,
                       Qual            EJPS, Gov, IJPA,
                                       IJPSM, IPMJ,
                                       IRAS, JEPP,
                                       JPAM, JPART,
                                       JPP, NVSQ,
                                       NML, PSJ, PSR,
                                       PA, PAR, PMR,
                                       POR, PPMR,
                                       RPR, ARPA

PMR       1990-2011    Sys. Review     n/a WoS and             46
                       Quant & Qual    Ebsco databases

PMR       2001-2013    Sys. Review     No Halduskultuur,      32 *
                       Quant & Qual    J PAP, TRAS, U-A

PMR       1985-2011    3ys. Review     Yes, SSCI in PA      107(787)
                       Quant & Qual

PMR       2000-2012    5ys. Review     No database and        250
                       Quant & Qual    PAR, PA, Gov,
                                       IRAS, JPART,
                                       PMR, PS
PPMR      not          Sys. Review     n/a databases         519 #
          mentioned    Quant & Qual

PPMR      2011-2013    Sys. Review     n/a database, SSCI      25
                       Quant           journals from
                                       the Performance
                                       Information
                                       Project

Total     --                           --                     3452
                                                            (2211 #
                                                              w/o
                                                              area
                                                             focus)

Journal   Canadian     Mention       Comments
          articles     of Canad *

ARPA         0            no         Canadian
                                     Public Policy
                                     is among the
                                     29 journals

ARPA         0            no

ARPA        4 *           yes        168 studies on
                                     HK and+500
                                     on Macao

A&S          4            no

IPMJ         1            yes        14 journals
                                     with high
                                     reputation

IRAS         0            no

IRAS         0            no

IRAS        0 #           no         22 studies on
                                     Africa

JPART        0            no

JPART        2            no         2 articles, but
                                     not from PA
                                     1 psy journal
                                     1 education
                                     journal

PA           12           no

PA           1            yes        CPA added as
                                     one of three
                                     non-UK/
                                     USA-oriented
                                     journals

PAR          0            no

PAR          0            no

PAR          11           no         4 articles by
                                     Canadians, 7
                                     articles by
                                     intr scholars
                                     about
                                     Canada
PMM          0            no

PMR          0            no

PMR          0            no         "In this
                                     network,
                                     each article
                                     was cited an
                                     average of
                                     forty times.
                                     The final
                                     sample was
                                     generated by
                                     filtering all
                                     publications
                                     in the
                                     network that
                                     were cited at
                                     a below-average
                                     number of
                                     times."
PMR          1            no

PMR         0 *           no         Restricted to
                                     Eastern
                                     Europe

PMR          1            no         Co-citation
                                     analysis and
                                     bibliographic
                                     coupling

PMR          12           no

PPMR        0 #           no         Restricted to
                                     Europe

PPMR         0            no

Total     49 (45 #         3         --
          w/o area     mentions
           focus)

Table 2. Connection to 70 Articles (St. Clair, Hicks and Isset,
2017) from Australian and Canadian Flagship Journals, by Disciplines

                    Citation frequencies of the Top 70 articles in PA

Disciplines             Australian    Canadian
                        flagship      flagship
                        journals)     journal(s)

Public administration   64.4% (85)    50.0% (43)
Political science       7.6% (10)     20.0% (18)
Other disciplines       28.0% (37)    29.1% (25)
Total                   (132)         (86)

[Chi.sup.2]= 9.09, Pr = 0.011 Raw data from St.Clair, Hicks and
Isett (2017)

Table 3. Comparisons of Impact Factors between Australian and
Canadian National Flagship Journals in Twelve Disciplines in
the Social Sciences, 2016

disciplines             journals    impact factor     relative diff.
                                                      to Australian
                                                      equivalent

Public administration   AJPA        1.072             --
                        CPA-APC     0.333             31.1%
Political science       AJPS        0.688             --
                        CJPS-RCSP   0.406             59.0%
Management              AJM         1.483             --
                        CJAS-RCSA   0.268             18.1%
Anthropology            AJA         0.864             --
                        CJA-RCA     defunct in 1986   n.a.
Area studies            JAS         0.233             --
                        JCS-REC     not in SSCI       n.a.
Criminology             ANZJC       0.981             --
                        CJCCJ       0.923             94.1%
Economics               AJARE       1.826             --
                        CJAE-RCA    1.052             57.6%
Education               ADR         0.910             --
                        AJET        0.853             --
                        AJET        0.667             --
                        AJAL        0.558             --
                        EA          0.396             --
                        CJE-RCE     not in SSCI       n.a.
                        CJEAP       not in SSCI       n.a.
                        CJLT-RCAT   not in SSCI       n.a.
Geography               AG          1.115             --
                        CG-GC       0.896             80.4%
Intl' relations         AJIA        0.859             --
                        CFPJ        not in SSCI       n.a.
Social work             ASW         0.787             --
                        AJGC        0.778             --
                        CSWR-RCSS   not in SSCI       n.a.
Sociology               JS          0.841             --
                        CRS-CSS     1.022             121.5%
                        CJS-CCS     0.341             40.5%

Source: Compiled from Thompson Reuters SSCI