New assessment unit at Treasury Board makes good on business innovation program reform

A new unit at the Treasury Board of Canada Secretariat (TBS) is using data from Statistics Canada to better evaluate the performance of business support programs across government.

Following the horizontal review of federal innovation and clean technology programs initiated in Budget 2017 and completed in January the following year, Budget 2018 allocated $2 million for the creation of the Central Performance and Impact Assessment (CPIA) unit at the TBS, to provide a firm-level view on government support for businesses.

CPIA teamed up with Statistics Canada to share data that will enable comparison and benchmarking across programs. The data from Statistics Canada includes anonymized information from the business register, corporate tax data, patents, payroll data, the import/export register, industry surveys, and others.

Learning from the horizontal business review

[rs_related_article slug="federal-innovation-support-programs-focus-of-ground-breaking-treasury-board-inventory/"]

The horizontal business review, led by then-Assistant Secretary Neil Bouwer, made several important findings: 74% of business support programs had some degree of mandate overlap; one third of funding ultimately went to third-party service providers; and program evaluations weren’t able to provide adequate insight into the impact of the support.

Following the review, the Business Innovation Program Reform in Budget 2018 proposed changes based on three principles: 1) adopting a business-centric lens, 2) growing high-potential firms, and 3) strengthening accountability for results. The CPIA unit addreses the first and third priorities, by providing a firm-level view of the impact that business support programs are actually making.

"We're helping to make data available that can inform the improvement to the design and delivery of these programs that the firms use,” said Greg Bridgett, Acting Director of Innovation Impact Assessment at TBS, in an interview with RE$EARCH MONEY.

The horizontal review was designed to provide a foundation for simplifying business support programs. For example, the reform recommended a two-thirds reduction of the 92 existing program streams. As a first step, TBS needed to confront the complex array of programs for businesses. “We have to look across a suite of programs that are delivered across many departments at many times, and pursue various policy objectives designed around different outcomes,” said Bridgett.

“We found a bit of a fruit basket of outcomes and indicators,” Bridgett observed. “The way they define success is all a little bit different.” In order to reduce the number of duplicated or overlapping programs, untangle programs that work at cross-purposes, and help simplify processes that confuse the program users, TBS needed to create what Bridgett calls a “coordinating frame” for all the program streams.

Finding the opportunity in the data

When the CPIA undertook to create a better tool for assessing program impact, they found two fantastic puzzle-pieces, according to Bridgett. One was the administrative data that the programs generate — that is, the type, value and time of support — and the other was “the capability of Statistics Canada to link administrative datasets in order to unlock the analytical value,” said Bridgett.

[rs_quote credit="Greg Bridgett" source="Treasury Board of Canada Secretariat"]We can take cross-sections of firms... and look at the role that the programs play in advancing a policy.[/rs_quote]

George Sciadas, director at the Centre for Special Business Projects at StatsCan, explained the process in an email to RE$EARCH MONEY. By using data from 22 federal departments, agencies and Crown corporations, representing 97 program streams over the 2007-2016 period, and then matching the data to the Business Register, StatsCan was able to identify roughly 44,000 unique enterprises. StatsCan then obtained information through its Linkable File Environment (LFE) about the enterprises along variables such as revenues, employment, and R&D.

Based on this dataset, StatsCan produced descriptive tables for TBS that profiled the supported enterprises by attributes like program stream, geographic region, industry sector, business size, for-profit or not-for-profit status, and more. The statistical agency also conducted pre- and post-support analysis, as well as econometric analysis to compare the performance of enterprises that received support and those that didn’t.

StatsCan has long been in the business of linking administrative data sets, and some departments already do their own impact analysis, but “what hasn't been done is the horizontal approach,” Bridgett observed. He noted that this creates a novel analytical opportunity, enabling the programs to see their performance through the lens of the firms they support. "We can take cross-sections of firms, like high-growth firms for example, and look at the role that the programs play in advancing a policy."

Creating a tool for deeper analysis

The CPIA continues to work with other departments across government to get as much buy-in as possible, to ensure the program is truly horizontal in its perspective and scope. “We've had everyone sign on, which is great. Now we're dealing with putting processes into place and getting everyone to feed in the data so that we can actually start achieving some results,”  said Glenn Purves, Assistant Secretary Designate, Expenditure Management Sector at TBS, in conversation with RE$EARCH MONEY.

The aim is to use all those inputs to create a longitudinal database that will go back 10 years or so and link up the programs' administrative data with the Statscan data, to see whether what was contributed in terms of support had an impact on performance, Purves explained.

Currently under development with StatsCan, the longitudinal study is “the asset upon which all of our work is going to happen,” said Bridgett. This dataset will become a tool to enable descriptive analysis that will allow the programs to see themselves through the firms that they support. The tool will also support the use of statistical and econometric methods to identify causal links between program activities and the impact on growth, said Bridgett.

The longitudinal dataset that CPIA is creating in coordination with StatsCan will be made available to researchers. “We'll be working with researchers to bring their expertise to bear to help strengthen the methods that we use,” said Bridgett. “It's a key aspect of the design of our unit and our mandate.” The CPIA will launch an outreach program to advertise the value of the dataset and the types of work that can be done on it. “We expect to be contracting some of our interesting question and seeking expertise of the research community to help us advance the development of our knowledge and practices,” said Bridgett.

Setting the stage for performance-informed budgeting

Building on the Policy on Results launched in August, 2016, the federal government has been taking steps toward performance-informed budgeting. The goal is to make decisions about how resources are allocated based on performance evaluations. But in order to make performance-informed budgeting a reality, the government needs to be able to measure the achievements programs make over time. The assessments conducted by the CPIA team at TBS will play a role in reaching that goal.

"The results from the Central Performance and Impact Assessment unit is leading-edge analysis that will be an important additional tool to inform future federal budgets," said Anna Arneson, spokesperson for the Department of Finance, in an email to RE$EARCH MONEY. "Once the first annual results from this new and innovative unit are available, we will begin to use them to inform the budgetary process."

"The government has a priority to grow the number of high-growth firms and lots of programs interact with firms that become high growth firms,” said Bridgett. “But it could be that there are a couple of programs that are really linked to the conversion of a firm from one that has high potential to one that meets the criteria for high-growth. If the government wants to step on the gas and achieve more results on enabling the creation of more high-growth firms, then they know where to put their dollars."

R$