To what extent do government ministries regularly evaluate the effectiveness and/or efficiency of public policies and use results of evaluations for the revision of existing policies or development of new policies?
Ex post evaluations are carried out for all significant policies and are generally used for the revision of existing policies or the development of new policies.
10
UK
Evidenced-based decision-making is deeply rooted in the United Kingdom’s tradition of firm and efficient legislation, and ex post evaluations are as vital a part of public policymaking as impact and sustainability checks. The OECD ranks the United Kingdom second among its 40 members for its approach. Specialist analytical functions have recently been brought together to form the Analysis Function, further strengthening their role in policymaking and service delivery. Analytical approaches to evaluation are set out by the Magenta Book and the Green Book, supported by the Cross-Government Evaluation Group coordinated by HM Treasury. More recently the Behavioral Insights Team (formerly a team within the Cabinet Office, but now an independent entity) and the What Works Network (coordinated by the Cabinet Office) promote the increased use of evaluation methods, especially randomized controlled trials.
The notion here is to make government more efficient, and so to quickly identify and overcome unnecessary regulatory hurdles. In 2011, the previous coalition government launched the “Red Tape Challenge,” encouraging citizens to report unhelpful or burdensome legislation that could be cut or reformed. However, the main focus of the United Kingdom’s commitment to ex post evaluation is surely on facilitating business.
The Regulatory Policy Committee (RPC) assesses the quality of the government’s RIA analyses and further examines all published ex post evaluation. If the RPC submits a recommendation to the government, it is expected to be implemented into law. Further, businesses can directly address the RPC if they disagree with or feel disadvantaged by a specific governmental regulatory assessment.
The notion here is to make government more efficient, and so to quickly identify and overcome unnecessary regulatory hurdles. In 2011, the previous coalition government launched the “Red Tape Challenge,” encouraging citizens to report unhelpful or burdensome legislation that could be cut or reformed. However, the main focus of the United Kingdom’s commitment to ex post evaluation is surely on facilitating business.
The Regulatory Policy Committee (RPC) assesses the quality of the government’s RIA analyses and further examines all published ex post evaluation. If the RPC submits a recommendation to the government, it is expected to be implemented into law. Further, businesses can directly address the RPC if they disagree with or feel disadvantaged by a specific governmental regulatory assessment.
Citations:
OECD Regulatory Policy Outlook 2021
OECD Regulatory Policy Outlook 2021
9
Finland
Consultation with experts and stakeholders is a natural phase in the Finnish lawmaking process. In addition, the public is invited to comment on draft proposals online. Furthermore, all proposals for changing statutes must be accompanied by an assessment of their impact across several aspects of society (e.g., the economy and environment). However, the OECD has pointed out that although ex post evaluations are frequently carried out, Finland lacks a systematic strategy for the ex post evaluation of regulations. The pandemic has not impacted the process of parliamentary consultation.
Citations:
“Säädösehdotusten vaikutusten arvionti,” Oikeusministeriö, http://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/76082/saadosehdotusten_vaikutusten_arviointi_ohjeet.pdf?sequence=1&isAllowed=y
OECD: “Better Regulation in Europe: Finland.” http://www.oecd.org/gov/regulatory-policy/45054846.htm.
OECD (2018), OECD Regulatory Policy Outlook 2018, OECD Publishing, Paris, https://doi.org/10.1787/9789264303072-en.
Opinion service webpage: lausuntopalvelu.fi
Governments Registry for Projects and Initiatives (http://valtioneuvosto.fi/hankkeet).
“Säädösehdotusten vaikutusten arvionti,” Oikeusministeriö, http://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/76082/saadosehdotusten_vaikutusten_arviointi_ohjeet.pdf?sequence=1&isAllowed=y
OECD: “Better Regulation in Europe: Finland.” http://www.oecd.org/gov/regulatory-policy/45054846.htm.
OECD (2018), OECD Regulatory Policy Outlook 2018, OECD Publishing, Paris, https://doi.org/10.1787/9789264303072-en.
Opinion service webpage: lausuntopalvelu.fi
Governments Registry for Projects and Initiatives (http://valtioneuvosto.fi/hankkeet).
Switzerland
Article 170 of the constitution states that “(t)he federal parliament shall ensure that the efficacy of measures taken by the confederation is evaluated.” Ex post evaluations have been strongly developed and are, to varying degrees, standard in most policy fields. Evaluations are best established in the fields of development cooperation, public health, education and economy. Ex post evaluations are a most important source of information for the revision and development of policies when used by the administration. Administration experts draft laws and reports based on the available empirical evidence, which mostly consists of policy evaluations. However, the administration formulates drafts that are subject to pre-parliamentarian and parliamentarian policymaking processes that include many relevant actors that do not prioritize evidence. The main goal of policymaking in Switzerland is acceptance rather than evidence-based policy. However, as many evaluations focus on learning within the administration rather than executive control and new legislation, the impact of evaluations remains significant in Switzerland. The Swiss evaluation community is one of the most professional in Europe and evaluations are of good quality. The role of ex post evaluations in Switzerland can thus be considered important.
Citations:
Fritz Sager, Thomas Widmer, Andreas Balthasar (eds.) 2017: Evaluation im politischen System der Schweiz. Entwicklung, Bedeutung und Wechselwirkungen, Zürich: NZZ Verlag
Fritz Sager, Thomas Widmer, Andreas Balthasar (eds.) 2017: Evaluation im politischen System der Schweiz. Entwicklung, Bedeutung und Wechselwirkungen, Zürich: NZZ Verlag
Ex post evaluations are carried out for most significant policies and are used for the revision of existing policies or the development of new policies.
8
Canada
As with other aspects of Canada’s RIA regime, ex post evaluation differs between laws and regulations. While laws are not subject to systematic ex post evaluation, departments and agencies are expected to regularly review existing regulations to assess their impacts and develop refinements.
In 2016, the Treasury Board of Canada introduced a new “Policy on Results.” Evaluations of programs, policies and priorities under the policy is to be a “systematic and neutral analysis of evidence related to relevance, effectiveness and efficiency of policies and programs.” Those evaluations can be done through the Treasury Boards’ resource alignment reviews and internally by departments themselves.
In 2018, the federal government introduced the Canadian Gender Budgeting Act (S.C. 2018, c.27, s.13) which provided for government policies, programs and services to take into account “gender” and “diversity” both in their design and impacts. As a result, Treasury Board has put in place gender-based plus analysis which requires departments to annually report on the impacts of policies and programs in terms of their access and inclusiveness.
In practice, it is frequently the Office of the Auditor General of Canada (OAG) that evaluates government programs and initiatives ex post. The OAG is formally charged with so-called performance audits, which aim to provide an independent, objective and systematic assessment of whether government programs are being run with due regard for the economy, efficiency and environmental impact. The OAG has considerable discretion regarding which programs it will examine, and takes requests from parliamentary committees, members of parliament, citizens, civic groups and other parties to conduct audits in specific areas. These audits generally undertake an extensive analysis of the issues under consideration. The OAG conducts between 25 and 30 performance audits each year, publishing the results.
In 2016, the Treasury Board of Canada introduced a new “Policy on Results.” Evaluations of programs, policies and priorities under the policy is to be a “systematic and neutral analysis of evidence related to relevance, effectiveness and efficiency of policies and programs.” Those evaluations can be done through the Treasury Boards’ resource alignment reviews and internally by departments themselves.
In 2018, the federal government introduced the Canadian Gender Budgeting Act (S.C. 2018, c.27, s.13) which provided for government policies, programs and services to take into account “gender” and “diversity” both in their design and impacts. As a result, Treasury Board has put in place gender-based plus analysis which requires departments to annually report on the impacts of policies and programs in terms of their access and inclusiveness.
In practice, it is frequently the Office of the Auditor General of Canada (OAG) that evaluates government programs and initiatives ex post. The OAG is formally charged with so-called performance audits, which aim to provide an independent, objective and systematic assessment of whether government programs are being run with due regard for the economy, efficiency and environmental impact. The OAG has considerable discretion regarding which programs it will examine, and takes requests from parliamentary committees, members of parliament, citizens, civic groups and other parties to conduct audits in specific areas. These audits generally undertake an extensive analysis of the issues under consideration. The OAG conducts between 25 and 30 performance audits each year, publishing the results.
Citations:
Government of Canada, “Integrating Gender-Based Analysis Plus into Evaluation: A Primer,” 30 September 2019, https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/evaluation-government-canada/gba-primer.html.
Treasury Board of Canada, Cabinet Directive on Regulation, ttps://www.canada.ca/
en/treasury-board-secretariat/services/federal-regulatory-management/guidelines-tools/cabinet-directive-regulation.html.
Treasury Board of Canada, Policy on Results, https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=31300.
Government of Canada, “Integrating Gender-Based Analysis Plus into Evaluation: A Primer,” 30 September 2019, https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/evaluation-government-canada/gba-primer.html.
Treasury Board of Canada, Cabinet Directive on Regulation, ttps://www.canada.ca/
en/treasury-board-secretariat/services/federal-regulatory-management/guidelines-tools/cabinet-directive-regulation.html.
Treasury Board of Canada, Policy on Results, https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=31300.
Germany
Instituting ex post evaluations of legislation has become more important in Germany. Ex post analyses are widely used in labor market, education and family policy areas. A milestone for ex post labor-market research was the introduction of a legal obligation to evaluate the impact of active labor-market policies in 1998. Since then, important legislation such as labor-market and social security reforms (Hartz reforms), and later the introduction of minimum wages, have undergone far-reaching ex post evaluations (Boockmann et al. 2014).
In 2013, a concept for the systematic evaluation of new regulations was accepted. As a principle, all important laws and regulations for which compliance costs exceed €1 million have to be evaluated three to five years after being introduced. The Committee of State Secretaries delivered a decision in 2019 – the Reduction of Bureaucracy and Better Regulation – that further developed and specified this concept. As a result of the decision, federal ministers are committed to involving states, municipalities and other stakeholders impact assessments and to making the results public. While ministries conduct the ex post evaluations themselves, the quality of the analyses is to be checked by an independent authority. As a result of these requirements, the federal government expects more than 330 ex post evaluations to be conducted in the coming years (Wissenschaftlicher Dienst 2020).
In 2013, a concept for the systematic evaluation of new regulations was accepted. As a principle, all important laws and regulations for which compliance costs exceed €1 million have to be evaluated three to five years after being introduced. The Committee of State Secretaries delivered a decision in 2019 – the Reduction of Bureaucracy and Better Regulation – that further developed and specified this concept. As a result of the decision, federal ministers are committed to involving states, municipalities and other stakeholders impact assessments and to making the results public. While ministries conduct the ex post evaluations themselves, the quality of the analyses is to be checked by an independent authority. As a result of these requirements, the federal government expects more than 330 ex post evaluations to be conducted in the coming years (Wissenschaftlicher Dienst 2020).
Citations:
Boockmann, B., Buch, C. M., Schnitzer, M. (2014): Evidenzbasierte Wirtschaftspolitik in Deutschland: Defizite und Potentiale, IAW Discussion Paper Nr. 103, April 2014.
Wissenschaftlicher Dienst, Deutscher Bundestag (2020): Sachstand: Zur praktischen Umsetzung und Evaluierung von Gesetzen, WD 3 - 3000 - 298/19.
Boockmann, B., Buch, C. M., Schnitzer, M. (2014): Evidenzbasierte Wirtschaftspolitik in Deutschland: Defizite und Potentiale, IAW Discussion Paper Nr. 103, April 2014.
Wissenschaftlicher Dienst, Deutscher Bundestag (2020): Sachstand: Zur praktischen Umsetzung und Evaluierung von Gesetzen, WD 3 - 3000 - 298/19.
Norway
The Norwegian government makes use of evaluations in most policy sectors and issue areas. Each ministry has the responsibility of evaluating policy results in its area.Evaluations are carried out by external experts or internal ministerial review bodies. Evaluations are sometimes intended to measure the effect of reforms, although more frequently evaluations serve as a starting point for a future reform process. There is broad support for evidence-based policymaking and the results of policy evaluations tend to attract considerable attention.
Sweden
Ex post evaluations take various forms in the Swedish system. For the past 10 to 15 years, performance measurement and management has been an integral part of public management in Sweden, as in most other countries. Also, audits conducted by the Swedish supreme audit institution (Riksrevisionen) are important evaluation instruments. There is, however, a tendency to focus more on institutions and cost efficiency (the audit approach) than on programs and impact (the evaluation approach) – a trend that is increasingly noticeable across many Western countries. That said, both approaches are useful as feedback on public policy (Pierre, Peters and de Fine Licht, 2018; Peters and Pierre, 2019).
Sweden’s pandemic policies are evaluated by an independent commission composed mainly of academics. The first partial reports of this evaluation are already available (https://coronakommissionen.com/).
Sweden’s pandemic policies are evaluated by an independent commission composed mainly of academics. The first partial reports of this evaluation are already available (https://coronakommissionen.com/).
Citations:
Pierre, Jon. B. Guy Peters and Jenny de Fine Licht. 2018. “Is Auditing the New Evaluation? Can it be? Should it be?,” International Journal of Public Sector Management 31:726-39.
Peters, B. Guy and Jon Pierre. 2019. “From Evaluation to Auditing and from Programs to Institutions?: Causes and Consequences of the Decline of the Program Approach.” Governance: An International Journal of Policy, Administration, and Institutions. 33(3), 585-597.
Pierre, Jon. B. Guy Peters and Jenny de Fine Licht. 2018. “Is Auditing the New Evaluation? Can it be? Should it be?,” International Journal of Public Sector Management 31:726-39.
Peters, B. Guy and Jon Pierre. 2019. “From Evaluation to Auditing and from Programs to Institutions?: Causes and Consequences of the Decline of the Program Approach.” Governance: An International Journal of Policy, Administration, and Institutions. 33(3), 585-597.
7
Denmark
Existing regulations of RIAs do not require ex post evaluations. Such evaluations are part of the ongoing political process. Political agreements either explicitly or implicitly settle policies for some period. Though various events and developments may require subsequent action, which would provide a chance for citizens, media outlets and policy entrepreneurs to advocate policy reforms. This happens regularly in the Danish political system and very often the government or the opposition will suggest new policies. Expert committees are often appointed to analyze the issues. Most significant policies are regularly debated and policy reforms are common.
The annual budget preparation is one occasion for evaluating policies. The parliament’s Rigsrevision (auditor general) also issues an annual report, which may lead to policy reforms. In some cases, an assessment is made an explicit part of a political agreement (e.g., labor market policy).
The annual budget preparation is one occasion for evaluating policies. The parliament’s Rigsrevision (auditor general) also issues an annual report, which may lead to policy reforms. In some cases, an assessment is made an explicit part of a political agreement (e.g., labor market policy).
Citations:
Vejledning om konsekvensanalyser Maj 2005 (afsnit 3.2, 7.1 og 7.2 revideret i 2018). https://modst.dk/media/19528/vejledning-om-konsekvensanalyser-maj-2005-afsnit-3-2-7-1-og-7-2-revideret-i-2018.pdf (Accessed 10 October 2018),
Jørgen Grønnegård Christensen et al, Politik og forvaltning. 4. udg. Reitzels Forlag, 2017, p.300.
Vejledning om konsekvensanalyser Maj 2005 (afsnit 3.2, 7.1 og 7.2 revideret i 2018). https://modst.dk/media/19528/vejledning-om-konsekvensanalyser-maj-2005-afsnit-3-2-7-1-og-7-2-revideret-i-2018.pdf (Accessed 10 October 2018),
Jørgen Grønnegård Christensen et al, Politik og forvaltning. 4. udg. Reitzels Forlag, 2017, p.300.
Estonia
The Strategy Unit of the Government Office is responsible for the overall quality of policymaking, including the evaluation of policy effectiveness and development of a knowledge base for future reforms. In general, ex post evaluations take place three to five years after the implementation of the regulation, and cover areas of competition, administrative burden and regulatory overlap. The first ex post evaluations were undertaken in 2018. More recently, in-depth reviews have begun to be conducted in some policy areas, but the evaluation framework is not fully established yet. The publication of ex post evaluations remains at the discretion of the relevant minister and the use of analyses results is not systemic. The objective to increase the proportion of ex post evaluations set out in the new strategy document Principles for Legislative Policy until 2030, which was adopted in November 2020.
France
There is no practice of systematic evaluation, except for policies or laws in which the respective constitutive act stipulates the need for an evaluation. However, over the past 25 years, the Court of Accounts, which previously exerted a legalistic type of oversight, has transformed its mission and adapted its methods so as to evaluate public policies from a political, social, economic and financial point of view. The Court’s reports have become reference documents not only for the political authorities (government and parliament), but also for the opposition, the media and the broader public. The reports are usually characterized by rich analysis and accurate criticisms, and the recommendations are usually well received. The parliament and the government rarely challenge the courts’ conclusions and recommendations, which often become the basis for new legislation. Since Sarkozy’s time in office, the nominee for president of the court has always been a former politician from the opposition (at the time of appointment). This pattern has strengthened the legitimacy of the court, and allowed for the adoption of more policy-oriented evaluations. This dimension is not negatively perceived, as the Court is not seen as biased in its conclusions; indeed, its pragmatic suggestions are seen as useful in the preparation of new legislation. The last appointed president (a former socialist minister and EU commissioner) added one more instrument to the tool-box of the Court by publishing preliminary assessments or analyses of hot issues or government proposals within a few weeks or months. The Court acts on its own initiative.
A recent example of the Court’s critical attitude regarding shortcomings in current evaluation procedures came in a report about important public investment programs in the Sarkozy and Hollande era involving more than €57 billion from 2010 to 2018. The Court pointed out the limited nature of the assessment procedures concerning these investment plans, given the huge sums involved. Following President Macron’s announcement of a new €30 billion investment plan to foster the French industrial sector in October 2021, the Court said that “the moment has come to open a thorough analysis” about this kind of public-financed investment plan, and “about its place in the overall strategy of public investment.”
A recent example of the Court’s critical attitude regarding shortcomings in current evaluation procedures came in a report about important public investment programs in the Sarkozy and Hollande era involving more than €57 billion from 2010 to 2018. The Court pointed out the limited nature of the assessment procedures concerning these investment plans, given the huge sums involved. Following President Macron’s announcement of a new €30 billion investment plan to foster the French industrial sector in October 2021, the Court said that “the moment has come to open a thorough analysis” about this kind of public-financed investment plan, and “about its place in the overall strategy of public investment.”
Citations:
“France 2030: les erreurs à ne pas refaire,” lefigaro.fr, 14 October 2021
“France 2030: les erreurs à ne pas refaire,” lefigaro.fr, 14 October 2021
Japan
Government ministries evaluate their policies on an ex post basis. The Administrative Evaluation Bureau (AEB) conducts inspections, and each ministry carries out independent evaluations of the effects of its own policies. The AEB supports such activities, for instance by encouraging ministries to share methodologies and experiences. It also works to standardize and prioritize policy evaluations, and reviews ministry and agency evaluations.
Japan ranked comparatively low in an OECD ex post evaluation index for 2014. However, things started to improve with the introduction of the Basic Program on Reducing Administrative Burden in 2017. The program required regulatory enforcement ministries and agencies, which had previously mostly evaluated regulations themselves, to engage in ex post evaluations.
Japan ranked comparatively low in an OECD ex post evaluation index for 2014. However, things started to improve with the introduction of the Basic Program on Reducing Administrative Burden in 2017. The program required regulatory enforcement ministries and agencies, which had previously mostly evaluated regulations themselves, to engage in ex post evaluations.
Citations:
Council for Promotion of Regulatory Reform, Third Report by the Council for Promotion of Regulatory Reform – For New Era to Come, Provisional Translation, 4 June 2018
Ministry of Internal Affairs and Communications (Administrative Counseling Division), Japanese Ombudsman System, Tokyo, March 2018
OECD Regulatory Policy Outlook 2021, OECD, https://www.oecd-ilibrary.org/governance/oecd-regulatory-policy-outlook-2021_196 ce20a-en
Council for Promotion of Regulatory Reform, Third Report by the Council for Promotion of Regulatory Reform – For New Era to Come, Provisional Translation, 4 June 2018
Ministry of Internal Affairs and Communications (Administrative Counseling Division), Japanese Ombudsman System, Tokyo, March 2018
OECD Regulatory Policy Outlook 2021, OECD, https://www.oecd-ilibrary.org/governance/oecd-regulatory-policy-outlook-2021_196 ce20a-en
Lithuania
Government ministries sometimes evaluate the effectiveness and/or efficiency of public policies, but most evaluations are related to the use of EU funds; it is mandatory to evaluate the implementation of operational programs financed from EU structural and investment funds. For instance, 63 evaluations were performed during the programming period from 2007 to 2013. Many evaluations were executed during the 2014 to 2020 period. For instance, annual evaluation plans contain about 10 evaluations each year.
The implementation of recommendations derived from these evaluations is monitored on a regular basis, but a 2013 study revealed that only about 60% of all recommendations provided by evaluators had been implemented by Lithuanian ministries or other state institutions. This average rate of implementation was attributed to insufficient institutional and staff capacities in the administration; this in turn reduces the demand for evaluations, hinders quality-assurance efforts and limits the use of evaluation results. The administration has also showed limited progress in implementing National Audit Office recommendations.
The ex post system of evaluation is still in its early stages in Lithuania, as in many other OECD countries. According to Lithuanian legislation, “if a law regulates a previously unregulated field or amends it significantly, or has a high impact to a specific policy area, individuals or groups of them, an ex post evaluation clause should be included.” However, the law does not describe sources of funding or data provision (OECD). Thus, there have been no ex post evaluations implemented yet in accordance with this legislation. In its study presented in late 2021, the OECD recommended that the ex post evaluation system be made more robust by providing clear processes, financial resources and data collection provisions. Furthermore, improvements can and should be made with regard to enhancing the coherence and compatibility of ex ante and ex post evaluations. In 2021, the government finalized a new methodology for conducting ex post impact assessments. However, it remains to be seen how this will be implemented in practice.
The implementation of recommendations derived from these evaluations is monitored on a regular basis, but a 2013 study revealed that only about 60% of all recommendations provided by evaluators had been implemented by Lithuanian ministries or other state institutions. This average rate of implementation was attributed to insufficient institutional and staff capacities in the administration; this in turn reduces the demand for evaluations, hinders quality-assurance efforts and limits the use of evaluation results. The administration has also showed limited progress in implementing National Audit Office recommendations.
The ex post system of evaluation is still in its early stages in Lithuania, as in many other OECD countries. According to Lithuanian legislation, “if a law regulates a previously unregulated field or amends it significantly, or has a high impact to a specific policy area, individuals or groups of them, an ex post evaluation clause should be included.” However, the law does not describe sources of funding or data provision (OECD). Thus, there have been no ex post evaluations implemented yet in accordance with this legislation. In its study presented in late 2021, the OECD recommended that the ex post evaluation system be made more robust by providing clear processes, financial resources and data collection provisions. Furthermore, improvements can and should be made with regard to enhancing the coherence and compatibility of ex ante and ex post evaluations. In 2021, the government finalized a new methodology for conducting ex post impact assessments. However, it remains to be seen how this will be implemented in practice.
Citations:
STRATA/OECD, Strengthening Decision-Making and Policy Evaluation for Long-Term Development, 2021, https://strata.gov.lt/lt/apie-mus/projektai/irodymais-gristos-politikos-formavim as-ir-politikos-vertinimas-vyriausybes-centre
OECD, Mobilising Evidence at the Centre of Government in Lithuania. Strengthening decision-making and policy evaluation for long-term development, Paris: OECD, 2021.
STRATA/OECD, Strengthening Decision-Making and Policy Evaluation for Long-Term Development, 2021, https://strata.gov.lt/lt/apie-mus/projektai/irodymais-gristos-politikos-formavim as-ir-politikos-vertinimas-vyriausybes-centre
OECD, Mobilising Evidence at the Centre of Government in Lithuania. Strengthening decision-making and policy evaluation for long-term development, Paris: OECD, 2021.
South Korea
In a recent (2021) OECD assessment of ex post evaluation, Korea was ranked fifth. Recent changes include making ex post evaluation mandatory for all regulations developed by the executive and central ministries; requiring a quality control process for packaged reviews of ex post evaluations; and conducting reviews with the aim of reducing burdens on new industries and SMEs. While there is an effective evaluation system in place, the effect on policy revision has been limited to date. In recent years, real estate policy has been broadly unsuccessful, contributing to historically high and skyrocketing prices by repeatedly imposing low-quality short-term regulations without effective ex post evaluation. Soaring prices have produced adverse effects by increasing economic inequality.
Citations:
OECD, Government at a Glance Database,
https://www.oecd-ilibrary.org/sites/9789264303072-en/1/2/2/index.html?itemId=/content/publication/9789264303072-en&_csp_=46cd5ed37844c9dc6cf21c84716ab307&itemIGO=oecd&itemContentType=book
OECD Regulatory Policy Outlook 2021, https://www.oecd-ilibrary.org/sites/38b0fdb1-en/index.html?itemId=/content/publication/38b0fdb1-en
OECD, Government at a Glance Database,
https://www.oecd-ilibrary.org/sites/9789264303072-en/1/2/2/index.html?itemId=/content/publication/9789264303072-en&_csp_=46cd5ed37844c9dc6cf21c84716ab307&itemIGO=oecd&itemContentType=book
OECD Regulatory Policy Outlook 2021, https://www.oecd-ilibrary.org/sites/38b0fdb1-en/index.html?itemId=/content/publication/38b0fdb1-en
USA
The United States has extensive and highly sophisticated capabilities for evaluating the impact of public policies – within government, in a large sector of think tanks and consulting firms, and in the extensive public policy-oriented academic community. In normal times, however, U.S. government is relatively inflexible, and slow to respond to evidence of the need for change. This is a widely recognized consequence of the separation-of-powers constitutional system, which was designed to inhibit policy change.
The lack of interest in information about the consequences of policies reached a new and quite extraordinary level during the Trump presidency. In some cases, the Republican leadership overrode congressional rules by refusing to wait for CBO analysis before voting on bills. Effective use of evaluation information was limited to issues that were not on presidential or partisan agendas.
Early on, the Biden administration broke away from the Trump administration’s practices when, on January 27, 2021, the new president signed the memorandum which created a Task Force on Scientific Integrity while promoting evidence-based policymaking, which relies extensively on policy evaluation, on the regulatory front and beyond. This memorandum is part of a broader push by the Biden administration to restore the importance of evidence and policy evaluation within the federal decision-making process.
The lack of interest in information about the consequences of policies reached a new and quite extraordinary level during the Trump presidency. In some cases, the Republican leadership overrode congressional rules by refusing to wait for CBO analysis before voting on bills. Effective use of evaluation information was limited to issues that were not on presidential or partisan agendas.
Early on, the Biden administration broke away from the Trump administration’s practices when, on January 27, 2021, the new president signed the memorandum which created a Task Force on Scientific Integrity while promoting evidence-based policymaking, which relies extensively on policy evaluation, on the regulatory front and beyond. This memorandum is part of a broader push by the Biden administration to restore the importance of evidence and policy evaluation within the federal decision-making process.
Citations:
https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/27/memorandum-on-restoring-trust-in-government-through-scientific-integrity-and-evidence-based-policymaking/
https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/27/memorandum-on-restoring-trust-in-government-through-scientific-integrity-and-evidence-based-policymaking/
6
Australia
Ex post evaluation of public policies is not a mandated or standard part of policy implementation in Australia. However, Australia was ranked first in ex post evaluation in the OECD Government at a Glance database. While an effective evaluation system is in place, the effect on the revision of policies or creation of new policies has so far been limited. Periodically, policies are announced with explicit accommodation for their evaluation, but this is the exception rather than the norm. In general, evaluation is more likely to occur for policies developed at the departmental or agency level (as opposed to the top level of government) or for policies that are not politically important/sensitive.
For policies that are evaluated, the quality of evaluation is variable, and rarely is the “gold standard” of a randomized controlled trial used.
For policies that are evaluated, the quality of evaluation is variable, and rarely is the “gold standard” of a randomized controlled trial used.
Citations:
https://de.slideshare.net/OECD-GOV/effective-ex-post-evaluation-purpose-and-challenges
https://de.slideshare.net/OECD-GOV/effective-ex-post-evaluation-purpose-and-challenges
Chile
Since 1997, the Ministry of Finance’s Budget Office (Dirección de Presupuestos, DIPRES) has had the power to assign specific budgets to line ministries for the contracting of external consultants to carry out ex post evaluations of their government programs (Evaluación de Programas Gubernamentales, EPG). Programs or institutions to be evaluated are agreed on with the Congress annually, with the instruction coming via ministerial decree. The evaluation results are normally made publicly and freely available.
Citations:
Evaluation of government programs:
Budget Office (Dirección de Presupuesto, DIPRES), Evaluación de Programas Gubernamentales (EPG), http://www.dipres.gob.cl/598/w3-article-111762.html
Budget Office (Dirección de Presupuesto, DIPRES),”Evaluación Focalizada de Ámbito EFA”, October 2016, http://www.dipres.gob.cl/598/w3-article-154357.html, last accessed: 13 January 2022.
Evaluation of government programs:
Budget Office (Dirección de Presupuesto, DIPRES), Evaluación de Programas Gubernamentales (EPG), http://www.dipres.gob.cl/598/w3-article-111762.html
Budget Office (Dirección de Presupuesto, DIPRES),”Evaluación Focalizada de Ámbito EFA”, October 2016, http://www.dipres.gob.cl/598/w3-article-154357.html, last accessed: 13 January 2022.
Czechia
Ex post evaluations are a regular part of the RIA process. However, they are usually carried out internally by individual departments and, save for measures financed with EU funds, have not been done systematically so far. Intending to change this situation, the Government Office presented a comprehensive proposal in mid-2018 to improve the quality of ex post evaluations. An interdepartmental working group was established to draft new formal guidelines for ex post evaluations that would make findings public and make the whole process more transparent. However, the new formal guidelines are still under discussion.
Israel
According to OECD reports, which are based on surveys with Israeli civil servants, Israel does not have recommended or binding guidelines for the assessment of ex ante or ex post policies. In addition, in Israel there is no procedural framework for evaluating the performance and effectiveness of the state budget. In a survey, which was conducted in 2016, Israel scored 1.7 on ex post policy evaluation (out of four). This score was similar to that OECD average, but significantly lower than the average of 3.08 for EU member states.
Citations:
“Israel could gain from increased competition, public spending, OECD says “, Times of Israel, 2018 (Hebrew):
https://www.timesofisrael.com/israel-could-gain-from-increased-competition-public-spending-oecd-says/
“OECD economic scenarios to 2060 illustrate the long-run benefits of structural reforms,” OECD Website,
http://www.oecd.org/economy/oecd-economic-scenarios-to-2060-illustrate-the-long-run-benefits-of-structural-reforms.htm
“OECD Regulatory Policy Outlook 2015,” OECD Website,
https://www.oecd.org/gov/regulatory-policy/Israel-web.pdf
“OECD Regulatory Policy Outlook 2018“, OECD WEBSITE, 2018
https://www.oecd-ilibrary.org/docserver/9789264303072-en.pdf?expires=1540849342&id=id&accname=ocid195467&checksum=2F91C900404B34AA4EE7659685539C79
“Government at a Glance 2017 – Israel,” OECD Website,
https://www.oecd.org/gov/gov-at-a-glance-2017-israel.pdf
“Israel could gain from increased competition, public spending, OECD says “, Times of Israel, 2018 (Hebrew):
https://www.timesofisrael.com/israel-could-gain-from-increased-competition-public-spending-oecd-says/
“OECD economic scenarios to 2060 illustrate the long-run benefits of structural reforms,” OECD Website,
http://www.oecd.org/economy/oecd-economic-scenarios-to-2060-illustrate-the-long-run-benefits-of-structural-reforms.htm
“OECD Regulatory Policy Outlook 2015,” OECD Website,
https://www.oecd.org/gov/regulatory-policy/Israel-web.pdf
“OECD Regulatory Policy Outlook 2018“, OECD WEBSITE, 2018
https://www.oecd-ilibrary.org/docserver/9789264303072-en.pdf?expires=1540849342&id=id&accname=ocid195467&checksum=2F91C900404B34AA4EE7659685539C79
“Government at a Glance 2017 – Israel,” OECD Website,
https://www.oecd.org/gov/gov-at-a-glance-2017-israel.pdf
Latvia
Ex post evaluations are carried out in Latvia for development planning documents. The introduction of the TAP portal and the standardized annotation form also affects ex post evaluation, which has been specifically highlighted and supplemented in the new form. Institutions now need to assess whether a piece of draft legislation provides for an ex post evaluation; when it does, it must also specify the results and indicators that will be used to assess the achievement of the objective of the act (or any part of it). The goal here is to link the ex ante and ex post evaluations, as recommended by the OECD.
Citations:
1. Methodology for developing and evaluating the results and performance indicators for ministries and other central state institutions (regulation), Available at: https://likumi.lv/doc.php?id=200935, Last accessed: 10.01.2022.
2. Cabinet of Ministers (2016), Report on Ex-post Evaluation Implementation (in Latvian), Available at: http://tap.mk.gov.lv/lv/mk/tap/?pid=40386136, Last accessed: 10.01.2022.
3. Informative report on the improvement of the impact assessment system (2021) Available (in Latvian) at: (http://tap.mk.gov.lv/mk/tap/?pid=40495849), Last accessed: 10.01.2022.
1. Methodology for developing and evaluating the results and performance indicators for ministries and other central state institutions (regulation), Available at: https://likumi.lv/doc.php?id=200935, Last accessed: 10.01.2022.
2. Cabinet of Ministers (2016), Report on Ex-post Evaluation Implementation (in Latvian), Available at: http://tap.mk.gov.lv/lv/mk/tap/?pid=40386136, Last accessed: 10.01.2022.
3. Informative report on the improvement of the impact assessment system (2021) Available (in Latvian) at: (http://tap.mk.gov.lv/mk/tap/?pid=40495849), Last accessed: 10.01.2022.
Malta
In recent years, ex post evaluations have been carried out for most significant policies. Various tools are used, and supported by enhanced digital processes. Improvements in ministerial coordination have also contributed to this development, along with ongoing review by the Office of the Principal Permanent Secretary. A “mystery shopper” for government departments was introduced in 2017, tasked with identifying shortcomings in service delivery so that they could be remedied accordingly. The National Audit Office performs audits to determine whether government entities have adequate systems of internal controls in place, with follow-up audits conducted to determine whether identified weaknesses have been dealt with. In 2021, NAO notes that “in the case of the audits included in this Report, 66% of our recommendations were either fully implemented or very significant progress was made thereon. 25% of the recommendations were partly implemented. Thus, a total of 91% of our recommendations were implemented to varying degrees. On the other hand, 9% of the recommendations were not implemented or little progress was made.” These audits are based on international standards. Progress has been made since the 2019 OECD report on regulatory policy and governance, which indicated that Malta was well below the OECD average in terms of the ex post evaluation of regulations. But Malta still lacks an entity that can take legal or regulatory action against consultants who present flawed reports, or who mislead the Environment and Resources Authority or Planning Authority.
Citations:
Follow Up Reports by the National Audit Office 2018 http://nao.gov.mt/en/recent-publications
Malta Today 03/07/2018 Environment Impact Assessments still unregulated after 20 years
https://nao.gov.mt/en/press-releases/4/1260/follow-up-reports-by-nao-2020-vol-2
Follow Up Reports by the National Audit Office 2018 http://nao.gov.mt/en/recent-publications
Malta Today 03/07/2018 Environment Impact Assessments still unregulated after 20 years
https://nao.gov.mt/en/press-releases/4/1260/follow-up-reports-by-nao-2020-vol-2
Mexico
Overall, Mexican policies are supposed to subjected to ex post evaluation and, at least at the national level, a comparatively strong culture of ex post evaluation has grown over the last two decades. This phenomenon is rooted in two different ideological streams. On the one hand, the traditional planning euphoria from a left-leaning, corporatist system has embraced the idea of ex post evaluation as an integral part of a well-mastered policy cycle. On the other hand, market liberal reforms and the influence of international finance organizations have introduced forms of new public management, with rigorous ex post evaluation seen as a crucial way to guarantee the efficiency and effectiveness of public policies.
Since 2012, mandatory guidelines require the use of ex post evaluations. As a consequence, Mexico has established one of the most developed evaluation policies in the field of social policy, which is executed by a constitutionally anchored evaluation agency, CONEVAL. Created in 2014, CONEVAL is established as an autonomous constitutional organization with a very high level of technical and management autonomy. Its task is to coordinate and manage the ex post evaluation of national social policies, and it does so with a very high level of sophistication. It consists of the head of the Ministry of Social Development, six well-known academics chosen by the National Social Development Commission and an executive secretary who is in charge of the council. In the international realm, CONEVAL has been an institutional innovation in poverty measurement and the evaluation of social public policies. So far, the independent provision of scientifically based evaluations has had a substantial impact on technically improving social policy programs in Mexico.
While CONEVAL represented a substantial move forward in the evaluation of social policies, a major setback occurred in mid-2019 when the head of CONEVAL was replaced after having criticized cuts to the agency’s budget. Given his prior critiques on the governments’ austerity policies, the replacement was perceived as a serious blow to the agency’s independence. It remains to be seen whether CONEVAL will continue to act as a critical but constructive evaluation agency in an environment where the central government, at least, has decreased its interest in evidence-based policymaking.
Beyond the field of social policy, however, other policy fields are subjected to less scientific ex post evaluations and, at the subnational level, much more remains to be done. However, the ongoing presence of many organizations of international development cooperation in Mexico as well as promising dynamics at the subnational level is favorable to the development of an evaluation-friendly environment. In this regard, there have been some initiatives focusing on improving evaluations of climate change adaptation, for instance.
Since 2012, mandatory guidelines require the use of ex post evaluations. As a consequence, Mexico has established one of the most developed evaluation policies in the field of social policy, which is executed by a constitutionally anchored evaluation agency, CONEVAL. Created in 2014, CONEVAL is established as an autonomous constitutional organization with a very high level of technical and management autonomy. Its task is to coordinate and manage the ex post evaluation of national social policies, and it does so with a very high level of sophistication. It consists of the head of the Ministry of Social Development, six well-known academics chosen by the National Social Development Commission and an executive secretary who is in charge of the council. In the international realm, CONEVAL has been an institutional innovation in poverty measurement and the evaluation of social public policies. So far, the independent provision of scientifically based evaluations has had a substantial impact on technically improving social policy programs in Mexico.
While CONEVAL represented a substantial move forward in the evaluation of social policies, a major setback occurred in mid-2019 when the head of CONEVAL was replaced after having criticized cuts to the agency’s budget. Given his prior critiques on the governments’ austerity policies, the replacement was perceived as a serious blow to the agency’s independence. It remains to be seen whether CONEVAL will continue to act as a critical but constructive evaluation agency in an environment where the central government, at least, has decreased its interest in evidence-based policymaking.
Beyond the field of social policy, however, other policy fields are subjected to less scientific ex post evaluations and, at the subnational level, much more remains to be done. However, the ongoing presence of many organizations of international development cooperation in Mexico as well as promising dynamics at the subnational level is favorable to the development of an evaluation-friendly environment. In this regard, there have been some initiatives focusing on improving evaluations of climate change adaptation, for instance.
New Zealand
Despite the dominant role of ex ante RIAs, steps are also taken to implement ex post RIAs. However, the quality of ex post evaluation of public policies remains unclear. They are not mandatory and there is no established methodology for conducting them. A 2021 OECD report ranks New Zealand below the OECD average in terms of ex post evaluation of regulations (OECD 2021).
Citations:
https://treasury.govt.nz/publications/legislation/regulatory-impact-assessments
Gill, Derek 2016. Rgulatory Coherence: The Case of New Zealand. ERIA Discussion Paper Series 2016-12. Wellington: University of Wellington.
OECD Regulatory Policy Outlook 2015 Country profile New Zealand. https://www.oecd.org/gov/regulatory-policy/New%20Zealand-web.pdf
OECD (2021) New Zealand: Indicators of Regulatory Policy and Governance 2021. https://www.oecd.org/gov/regulatory-policy/new-zealand-country-profile-regulatory-policy-2021.pdf
Kupiec, Tomasz 2015. Regulatory Impact Analysis Practice in New Zealand in the Light of Models of Evaluation Use – Inspiration for the Polish Government. „Management and Business Administration. Central Europe” 23(2), pp. 109–128.
https://treasury.govt.nz/publications/legislation/regulatory-impact-assessments
Gill, Derek 2016. Rgulatory Coherence: The Case of New Zealand. ERIA Discussion Paper Series 2016-12. Wellington: University of Wellington.
OECD Regulatory Policy Outlook 2015 Country profile New Zealand. https://www.oecd.org/gov/regulatory-policy/New%20Zealand-web.pdf
OECD (2021) New Zealand: Indicators of Regulatory Policy and Governance 2021. https://www.oecd.org/gov/regulatory-policy/new-zealand-country-profile-regulatory-policy-2021.pdf
Kupiec, Tomasz 2015. Regulatory Impact Analysis Practice in New Zealand in the Light of Models of Evaluation Use – Inspiration for the Polish Government. „Management and Business Administration. Central Europe” 23(2), pp. 109–128.
Slovenia
Ex post evaluations are regularly carried out for the most significant policies, but rarely for all other policies. When carried out, ex post evaluations are mostly used for the improvement of existing policies rather than for the development of new policies.
Spain
The 2015 law on the common administrative procedure includes a provision regarding ex post evaluation, making it compulsory to evaluate the application of rules and norms in order to ascertain whether they have met the objectives pursued, and whether costs have been justified. In 2017, the Institute for Evaluation of Public Policies replaced previous structures designed to produce internal policy evaluations of initiatives requested by the government.
The Office on Regulatory Coordination and Quality supervises the initial definition of the objectives and methodology for the ex post evaluation of regulations covered by RIAs, but does not scrutinize ex post evaluations themselves. The Independent Authority for Fiscal Responsibility (AIReF) is responsible for the evaluation of public spending and the efficiency of public policies. The AIReF analyses key expenditure and investment items such as education, healthcare and public works. The reports are not binding, but if the administration chooses not to follow the recommendations, it must justify this decision.
According to OECD recommendations published in 2021, ex post evaluations of regulations are still not performed in a systematic manner. Developing targeted guidance and standard evaluation techniques would contribute to more widespread and consistent evaluation of how regulations actually work in practice.
Most autonomous communities have developed additional, systematic ex post evaluation programs for their public policies. However, the practical impact of these measures has to date been limited.
The Office on Regulatory Coordination and Quality supervises the initial definition of the objectives and methodology for the ex post evaluation of regulations covered by RIAs, but does not scrutinize ex post evaluations themselves. The Independent Authority for Fiscal Responsibility (AIReF) is responsible for the evaluation of public spending and the efficiency of public policies. The AIReF analyses key expenditure and investment items such as education, healthcare and public works. The reports are not binding, but if the administration chooses not to follow the recommendations, it must justify this decision.
According to OECD recommendations published in 2021, ex post evaluations of regulations are still not performed in a systematic manner. Developing targeted guidance and standard evaluation techniques would contribute to more widespread and consistent evaluation of how regulations actually work in practice.
Most autonomous communities have developed additional, systematic ex post evaluation programs for their public policies. However, the practical impact of these measures has to date been limited.
Citations:
OECD (2021), Indicators of Regulatory Policy and Governance: Spain
https://www.oecd.org/gov/regulatory-policy/spain-country-profile-regulatory-policy-2021.pdf
Gobierno de España (2021): Informe de diagnóstico de la evaluación en la Administración General del Estado 2021. Available at: https://www.mptfp.gob.es/portal/funcionpublica/evaluacion-politicas-publicas/Informes-de-Evaluacion.html
OECD (2021), Indicators of Regulatory Policy and Governance: Spain
https://www.oecd.org/gov/regulatory-policy/spain-country-profile-regulatory-policy-2021.pdf
Gobierno de España (2021): Informe de diagnóstico de la evaluación en la Administración General del Estado 2021. Available at: https://www.mptfp.gob.es/portal/funcionpublica/evaluacion-politicas-publicas/Informes-de-Evaluacion.html
Netherlands
The General Audit Chamber (Algemene Rekenkamer) scrutinizes ex post policy evaluations by ministerial departments. Since 2000, the chamber has reported its findings to parliament on the third Wednesday in May each year. In 2012, the government introduced the Regulation for Regular Evaluation Studies, which specifies research criteria for assessing policy efficiency, goal achievement, evidence-based policymaking and subsidy-based policies. Yet, time and again, the chamber has reported deficits in goal achievement and weaknesses in goal formulation, which undermine the quality of ex post evaluation research. Other weaknesses in policy evaluation studies include the lack of citizen perspectives, inability to accurately calculate societal costs and benefits, overreliance on input from implementing organizations for evidence and lack of public access to many evaluations. In line with the general trend toward more instrumental advice, over the last couple of years, the General Audit Chamber has focused its attention on specific points in departmental agendas.
Moreover, there are a wide range of additional non-obligatory evaluations produced by ministerial departments, parliament, government-sponsored knowledge institutes, the ombudsman, implementation bodies and quasi-independent non-governmental bodies. In response to the coronavirus crisis, several evaluations were undertaken, including a review of impacts on different groups of citizens. The PBL did an ex ante evaluation of the sustainability impacts of proposals in the party platforms of six political parties. In response to worries about the use of algorithms in governance, and anticipating the abuse of algorithms by the tax authorities in the child benefits affair, the General Audit Chamber developed an ex ante evaluation framework for the design and use of algorithms. Since evaluation findings are just one factor in designing new or adjusting existing policies, it is not clear how much policy learning from formal and informal evaluations actually occurs. A recent study commissioned by the minister of finance assessed past evaluations and their use. The study confirmed that although “no other country evaluates so many of its policies,” policymaking civil servants and members of parliament are less sensitive to the outcomes of previous policies than to images and incidents (in the press). Moreover, obstruction and disinterestedness contribute to methodological weaknesses in many of the evaluation studies, this assessment found. For example, although the government agreement stipulates that a new policy decriminalizing the use of hashish may be experimentally tested at the local level, interference in the study’s design has already made a politically unbiased evaluation of results as good as impossible.
Dutch ex post evaluators closely follow international trends of “evidence informed” and “behavioral knowledge” evaluation studies. There has been a tendency to move away from a focus on single, case-specific ex post evaluation studies to a focus on the construction of broader, more balanced departmental knowledge portfolios, in which ex post evaluation studies are embedded as elements in a larger body of knowledge accessible to policymakers and other participants in policy subsystems. It is not yet clear to what extent such trends in evaluation research really inform evaluation practices.
Moreover, there are a wide range of additional non-obligatory evaluations produced by ministerial departments, parliament, government-sponsored knowledge institutes, the ombudsman, implementation bodies and quasi-independent non-governmental bodies. In response to the coronavirus crisis, several evaluations were undertaken, including a review of impacts on different groups of citizens. The PBL did an ex ante evaluation of the sustainability impacts of proposals in the party platforms of six political parties. In response to worries about the use of algorithms in governance, and anticipating the abuse of algorithms by the tax authorities in the child benefits affair, the General Audit Chamber developed an ex ante evaluation framework for the design and use of algorithms. Since evaluation findings are just one factor in designing new or adjusting existing policies, it is not clear how much policy learning from formal and informal evaluations actually occurs. A recent study commissioned by the minister of finance assessed past evaluations and their use. The study confirmed that although “no other country evaluates so many of its policies,” policymaking civil servants and members of parliament are less sensitive to the outcomes of previous policies than to images and incidents (in the press). Moreover, obstruction and disinterestedness contribute to methodological weaknesses in many of the evaluation studies, this assessment found. For example, although the government agreement stipulates that a new policy decriminalizing the use of hashish may be experimentally tested at the local level, interference in the study’s design has already made a politically unbiased evaluation of results as good as impossible.
Dutch ex post evaluators closely follow international trends of “evidence informed” and “behavioral knowledge” evaluation studies. There has been a tendency to move away from a focus on single, case-specific ex post evaluation studies to a focus on the construction of broader, more balanced departmental knowledge portfolios, in which ex post evaluation studies are embedded as elements in a larger body of knowledge accessible to policymakers and other participants in policy subsystems. It is not yet clear to what extent such trends in evaluation research really inform evaluation practices.
Citations:
A. Knottnerus, Van casus-specifieke beleidsevaluatie naar systematische opbouw van kennis en ervaring, Beleidsonderzoek Online, May 2016
Meyken Houppermans, ‘Wat is de toegevoegde waarde van de onafhankelijk
deskundige bij beleidsdoorlichtingen?’, Beleidsonderzoek Online juni 2018,
DOI: 10.5553/BO/221335502018000005001
SEO Economisch Onderzoek, december 2018. Beleidsdoorlichtingen belicht (SEO-rapport nr. 2018-110, sep.nl, accessed 8 November 2019
Algemene Rekenkamer,26-01-2021. Betere kwaliteitscontrole en meer inzicht voor burgers nodig bij algoritmes overheid
Platform O, Klieverik and Zwetsloot, 22 March 2021, Overheid, positioneer algoritmes als oplossing
FD, Daan Ballegeer Jean Dohmen 16 mrt 2021. ‘Er wordt veel beleid gemaakt waarvan we niet weten of het werkt’
PBL, 01-03-2021. Analyse leefomgevingseffecten verkiezingsprogramma’s 2021-2025
A. Knottnerus, Van casus-specifieke beleidsevaluatie naar systematische opbouw van kennis en ervaring, Beleidsonderzoek Online, May 2016
Meyken Houppermans, ‘Wat is de toegevoegde waarde van de onafhankelijk
deskundige bij beleidsdoorlichtingen?’, Beleidsonderzoek Online juni 2018,
DOI: 10.5553/BO/221335502018000005001
SEO Economisch Onderzoek, december 2018. Beleidsdoorlichtingen belicht (SEO-rapport nr. 2018-110, sep.nl, accessed 8 November 2019
Algemene Rekenkamer,26-01-2021. Betere kwaliteitscontrole en meer inzicht voor burgers nodig bij algoritmes overheid
Platform O, Klieverik and Zwetsloot, 22 March 2021, Overheid, positioneer algoritmes als oplossing
FD, Daan Ballegeer Jean Dohmen 16 mrt 2021. ‘Er wordt veel beleid gemaakt waarvan we niet weten of het werkt’
PBL, 01-03-2021. Analyse leefomgevingseffecten verkiezingsprogramma’s 2021-2025
Ex post evaluations are rarely carried out for significant policies and are rarely used for the revision of existing policies or the development of new policies.
5
Italy
Italian governments and parliaments do not regularly conduct in-depth ex post evaluations of policies before promoting a revision of existing or the development of new policies. Any new bill is accompanied by a technical report, which typically discusses existing policies and explains the benefits of the new policy. But only rarely is this the result of an in-depth evaluation process conducted through a public and open debate. It may happen for some of the most controversial reforms after some years of implementation. For instance, this was the case under the Draghi government for the pension reform and the citizenship income, which had been introduced by the first Conte government. In sum, ex post evaluation has not yet become a regularly applied policy instrument, but is conducted in a non-systematic way.
4
Greece
Ex post evaluation is a brand-new concept in Greece’s regulatory process. It was only legislated as late as 2019, after the change in government in 2019. It is now a requirement that laws are evaluated five years after their adoption. It is too early for ex post evaluation to be implemented, but the new legal provision represents a vast improvement in policymaking compared to the past.
Citations:
Law 4622/2019 art. 56 introduced ex post evaluation of regulations.
Law 4622/2019 art. 56 introduced ex post evaluation of regulations.
3
Austria
Ex post evaluation is a rather unknown field in Austrian politics. The lack of any systematic ex post evaluation tradition and the tendency of political actors to prioritize the next election over all other perspectives makes it highly unlikely that the present government or parliament will establish a structure of ex post evaluations. The absence of long-term strategies, beyond traditional vague ideologies (like social justice or defending Austrian identity), prevent any reasonable systematic ex post evaluation.
The only systematic ex post evaluation is provided by the Austrian Court of Audit., However, the court’s activities tend to focus on the financial aspects of specific government or government-sponsored projects. Nevertheless, ex post evaluation constitutes a major objective for Austrian scientific bodies (outside of ministries) such as the Austrian Institute of Economic Research (WIFO), Institute for Advanced Studies Vienna (his) and other Austrian university actors.
The only systematic ex post evaluation is provided by the Austrian Court of Audit., However, the court’s activities tend to focus on the financial aspects of specific government or government-sponsored projects. Nevertheless, ex post evaluation constitutes a major objective for Austrian scientific bodies (outside of ministries) such as the Austrian Institute of Economic Research (WIFO), Institute for Advanced Studies Vienna (his) and other Austrian university actors.
Citations:
https://www.rechnungshof.gv.at/rh/home/home_1/home_2/Taetigkeitsbericht_des_Rechnungshofes_2020_BF.pdf
https://www.rechnungshof.gv.at/rh/home/home_1/home_2/Taetigkeitsbericht_des_Rechnungshofes_2020_BF.pdf
Bulgaria
The rules for impact assessments in Bulgaria established in 2016 require an ex post evaluation of policies and their effects within five years of their implementation. By the end of the review period in 2021, only two such evaluations had been published through the government’s public-consultation portal.
In 2020-2021, the legislative process was often motivated by COVID-19 considerations and almost exclusively referred to the state of emergency situation. Ministerial executive orders were evaluated only by the next cabinet.
In 2020-2021, the legislative process was often motivated by COVID-19 considerations and almost exclusively referred to the state of emergency situation. Ministerial executive orders were evaluated only by the next cabinet.
Croatia
The process of Europeanization and Croatia’s EU membership have opened the space for the use of evaluation studies in Croatian public administration at the central and local government levels. Methods and theoretical approaches to evaluation are exchanged through the Croatian Evaluation Network, which is comprised of experts interested in evaluation practice. However, Croatian policymakers are yet to introduce effective ex post evaluation mechanisms for the already passed legislative and regulatory measures. Thus, ex post evaluations of significant policies are rarely carried out, and are even more rarely used by policymakers as a source of evidence and inspiration. If ex post evaluations are carried out, the success indicators tend to be too general and insufficiently precise.
Hungary
In Hungary, there is a legal framework for carrying out ex post evaluations since 2011. In practice, however, such evaluations are rarely carried out (OECD 2021).
Citations:
OECD (2021): Hungary: Indicators of Regulatory Policy and Governance 2021. Paris (https://www.oecd.org/gov/regulatory-policy/hungary-country-profile-regulatory-policy-2021.pdf).
OECD (2021): Hungary: Indicators of Regulatory Policy and Governance 2021. Paris (https://www.oecd.org/gov/regulatory-policy/hungary-country-profile-regulatory-policy-2021.pdf).
Ireland
Prior to the pandemic, the extent of overspending in healthcare totaled more than €2 billion over the previous four years, according to the Irish Fiscal Advisory Council. Such budgetary over-runs suggest that there is little ex post evaluation of policy in this significant budgetary area. Each year the government is presented with an over-run that is tacitly accepted and paid for out of buoyant tax revenues. However, if tax revenues fall, the ability of the government to fund such over-runs will create significant political tensions.
Notably, the central bank calculated that the cost of all measures deployed by the government to fight the pandemic in 2020 was €24.6 billion (Conefrey et al 2021), with an Exchequer deficit of €7.5 billion recorded in 2021 (DOF, 2022).
Notably, the central bank calculated that the cost of all measures deployed by the government to fight the pandemic in 2020 was €24.6 billion (Conefrey et al 2021), with an Exchequer deficit of €7.5 billion recorded in 2021 (DOF, 2022).
Citations:
Conefrey, T., Hickey, R. & McInerney, N. (2021) COVID-19 and the Public Finances in Ireland, Economic Letter, Vol. 2021, No. 3.
DOF (2022) Exchequer deficit of €7½ billion recorded in 2021: Corporation tax receipts at similar levels to VAT, €13½ billion in Covid related expenditure to support recovery – Ministers Donohoe & McGrath, Department of Finance, 05 January, available at: https://www.gov.ie/en/press-release/badf9-exchequer-deficit-of-712-billion-recorded-in-2021-corporation-tax-receipts-at-similar-levels-to-vat-1312-billion-in-covid-related-expenditure-to-support-recovery-ministers-donohoe-mcgrath/
Conefrey, T., Hickey, R. & McInerney, N. (2021) COVID-19 and the Public Finances in Ireland, Economic Letter, Vol. 2021, No. 3.
DOF (2022) Exchequer deficit of €7½ billion recorded in 2021: Corporation tax receipts at similar levels to VAT, €13½ billion in Covid related expenditure to support recovery – Ministers Donohoe & McGrath, Department of Finance, 05 January, available at: https://www.gov.ie/en/press-release/badf9-exchequer-deficit-of-712-billion-recorded-in-2021-corporation-tax-receipts-at-similar-levels-to-vat-1312-billion-in-covid-related-expenditure-to-support-recovery-ministers-donohoe-mcgrath/
Poland
According to the RIA rules, ex post evaluations can be requested by the Council of Ministers and subsidiary bodies; since 2019 by the Center for Strategic Analysis or the Ombudsman for SMEs; and since 2020 by the president of the Government Legislative Center. By the end of 2020, no evaluation had been conducted according to these recent procedures (OECD 2021: 2). In general, primary laws and subordinate regulations are rarely evaluated. Even when such procedures are carried out, the review’s outcome is not used to improve the quality of laws.
Citations:
OECD (2021): Poland. Indicators of Regulatory Policy and Governance 2021. Paris (https://www.oecd.org/gov/regulatory-policy/poland-country-profile-regulatory-policy-2021.pdf).
OECD (2021): Poland. Indicators of Regulatory Policy and Governance 2021. Paris (https://www.oecd.org/gov/regulatory-policy/poland-country-profile-regulatory-policy-2021.pdf).
Portugal
A recent study noted that ex post evaluation is very weakly institutionalized in Portugal. The ex post evaluations that take place are largely the result of external pressures (notably, EU requirements) and take place within a limited set of policy areas – usually those that legally require ex post evaluation. This was also confirmed by the OECD’s Indicators of Regulatory Policy and Governance 2021, which recommend that “Portugal could consider introducing systematic requirements to undertake ex post evaluation as well as introducing ‘in-depth’ reviews in particular sectors or policy areas to identify core reforms to Portugal’s regulatory framework.”
Citations:
Ana Diogo (2018), “Hacia un sistema integrado de M&E para los objetivos de desarrollo sostenible en Portugal” [dissertation], available online at: https://www.somosiberoamerica.org/investigaciones/la-evaluacion-de-la-agenda-2030/
OECD (2021), “Portugal: Indicators of Regulatory Policy and Governance 2021,” available online at: https://www.oecd.org/gov/regulatory-policy/portugal-country-profile-regulatory-policy-2021.pdf
Ana Diogo (2018), “Hacia un sistema integrado de M&E para los objetivos de desarrollo sostenible en Portugal” [dissertation], available online at: https://www.somosiberoamerica.org/investigaciones/la-evaluacion-de-la-agenda-2030/
OECD (2021), “Portugal: Indicators of Regulatory Policy and Governance 2021,” available online at: https://www.oecd.org/gov/regulatory-policy/portugal-country-profile-regulatory-policy-2021.pdf
Romania
While the institutionalization of ex post evaluations has been announced several times, they have remained the exception rather than the rule. If such evaluations have been done, their impact on decision-making has been intransparent.
Slovakia
Ex post evaluations have been rare so far. The RIA 2020 Better Regulation Strategy, approved by the cabinet in January 2018, has envisaged comprehensive ex post evaluations. A draft methodology was approved in 2019 and underwent pilot testing. However, the final methodology has yet to be published.
Ex post evaluations are generally not carried out and do not play any relevant role for the revision of existing policies or the development of new policies.
2
Belgium
The typical strategy is to pick the data that justify the decisions that have been made. For instance, Brussels changed its speed limit from 50 to 30 kph in 2021, and the regional government claimed that accident figures have dropped substantially. However, these are most likely attributable to the COVID-19 lockdowns since, due to a lack of speed controls, the actual driving speed in the city has barely changed.
This approach has led to counterproductive decisions in the areas of, for example, education, energy conservation, subsidies for solar panels and immigration. There may be some scattered ex post evaluations undertaken on the initiative of individual line ministries, but these evaluations have no direct impact on the revision of existing policies since they are not seriously considered by ministerial cabinets, where all strategic policy choices are initiated and arbitrated. The management of the health crisis, however, increased the use of expert panels for a short-term monitoring of impact but not systematic ex post evaluation.
This approach has led to counterproductive decisions in the areas of, for example, education, energy conservation, subsidies for solar panels and immigration. There may be some scattered ex post evaluations undertaken on the initiative of individual line ministries, but these evaluations have no direct impact on the revision of existing policies since they are not seriously considered by ministerial cabinets, where all strategic policy choices are initiated and arbitrated. The management of the health crisis, however, increased the use of expert panels for a short-term monitoring of impact but not systematic ex post evaluation.
Cyprus
Given the deficiencies in strategic planning, ex post evaluation has not been part of government practice. However, in reformed framework of commitments for public administration, since 2012, the government has solicited studies from the World Bank and others. These studies on public administration and line ministries aimed to restructure and reform public administration and its procedures, while also reviewing general policies. However, specific policies in ministries are not subject to evaluations and progress achieved on the basis of studies undertaken by external bodies has been slow.
While acknowledging the need that sectoral reviews should form part of strategic planning, the government still has not introduced it.
While acknowledging the need that sectoral reviews should form part of strategic planning, the government still has not introduced it.
Iceland
No regular ex post evaluations of the effectiveness or efficiency of public policies appear to be conducted by any government ministry. However, that does not mean that no such evaluations take place.
Luxembourg
According to the OECD, “ex post evaluations have been undertaken in Luxembourg although they remain an inconsistently applied regulatory management tool.” However, legislation is rarely modified through a process of ex post evaluation. High-quality evaluation studies are rarely carried out. There is also a shortage of personnel able to design legal texts. Rigorous state-commissioned evaluation of legislation is thus unlikely in the near future.
The Ministry of the Civil Service and Administrative Reform is the central oversight body responsible for quality control of regulatory management tools in Luxembourg. Its oversight functions apply to stakeholder engagement, RIA and ex post evaluations, and the entity is additionally responsible for other functions including the evaluation and coordination of regulatory policy. The Council of State is in charge of providing legal scrutiny of regulatory proposals.
The Ministry of the Civil Service and Administrative Reform is the central oversight body responsible for quality control of regulatory management tools in Luxembourg. Its oversight functions apply to stakeholder engagement, RIA and ex post evaluations, and the entity is additionally responsible for other functions including the evaluation and coordination of regulatory policy. The Council of State is in charge of providing legal scrutiny of regulatory proposals.
Citations:
“Indicators of Regulatory Policy and Governance Europe 2019 - Luxembourg.” OECD (2020). https://www.oecd.org/gov/regulatory-policy/indicators-of-regulatory-policy-and-g overnance-2019-luxembourg.pdf. Accessed 14 January 2022.
“Better Regulation in Europe. Luxembourg.” OECD (2019). https://www.oecd.org/gov/regulatory-policy/46547003.pdf. Accessed 14 January 2022.
“Indicators of Regulatory Policy and Governance Europe 2019 - Luxembourg.” OECD (2020). https://www.oecd.org/gov/regulatory-policy/indicators-of-regulatory-policy-and-g overnance-2019-luxembourg.pdf. Accessed 14 January 2022.
“Better Regulation in Europe. Luxembourg.” OECD (2019). https://www.oecd.org/gov/regulatory-policy/46547003.pdf. Accessed 14 January 2022.
Turkey
Generally, public policy assessment institutions in Turkey lack knowledge regarding evidence-based instruments, both theoretically and practically. However, there are various social and political barriers to the evaluation of public policies. For example, the parliament and judiciary cannot effectively supervise and review executive actions. It is unlikely that an effective evaluation mechanism will be developed in the near future. However, there are a few academic studies that have evaluated certain public policies.
The Annual Presidential Program of 2021 makes references to monitoring and assessment but not to the systematic ex post evaluation of public policies. There are a few ex post analyses of public policies, mainly in the health sector, that address citizen/patient satisfaction, self-assessment, and the assessment of action plans. The EU Commission draws attention to the absence of systematic ex post monitoring and public reporting on the implementation of key government programs. There is also a lack of public scrutiny, which further limits accountability.
The Annual Presidential Program of 2021 makes references to monitoring and assessment but not to the systematic ex post evaluation of public policies. There are a few ex post analyses of public policies, mainly in the health sector, that address citizen/patient satisfaction, self-assessment, and the assessment of action plans. The EU Commission draws attention to the absence of systematic ex post monitoring and public reporting on the implementation of key government programs. There is also a lack of public scrutiny, which further limits accountability.
Citations:
Aydın, U. S. T. A. (2014). Yeni Kamu Yönetiminde Politikaların Değerlendirilmesi: Tipolojiler, Boyutlar ve Ölçütler. Sayıştay Dergisi, (94), 5-27.
European Commission. “Turkey Report 2021. Commission Staff Working Document.” October 19, 2021. https://ec.europa.eu/neighbourhood-enlargement/turkey-report-2021_en
Gazete Duvar. “Duvar Arkası… ‘Beştepe’de iki insan boyu evrak bekliyor’.” August 16, 2020. https://www.gazeteduvar.com.tr/politika/2020/08/16/duvar-arkasi-bestepede-iki-insan-boyu-evrak-imza-bekliyor
Aydın, U. S. T. A. (2014). Yeni Kamu Yönetiminde Politikaların Değerlendirilmesi: Tipolojiler, Boyutlar ve Ölçütler. Sayıştay Dergisi, (94), 5-27.
European Commission. “Turkey Report 2021. Commission Staff Working Document.” October 19, 2021. https://ec.europa.eu/neighbourhood-enlargement/turkey-report-2021_en
Gazete Duvar. “Duvar Arkası… ‘Beştepe’de iki insan boyu evrak bekliyor’.” August 16, 2020. https://www.gazeteduvar.com.tr/politika/2020/08/16/duvar-arkasi-bestepede-iki-insan-boyu-evrak-imza-bekliyor
1
---
---