How can developers measure a project’s impacts?

Project evaluations should measure both direct and indirect impacts. A mini-grid’s direct impacts might include increased access to electricity, increased use of electricity, improved energy services, new productive uses of energy and decreased fuel costs. Indirect impacts might include economic growth, increased incomes and improved health.

It’s not always possible to measure an impact directly, so evaluators use indicators, measurable variables that serve as proxies for desired results. To evaluate improved access to electricity, for example, the developer might use indicators such as the number of new household connections, increase in kWh of energy consumed or percentage of very poor households that receive energy services. Indicators of the quality of energy services might include number of hours of available electricity or number and duration of outages.

Measuring indirect impacts is more difficult. The evaluation must control for other variables that could influence results. For example, individuals in the sample who own land might increase their incomes by using electricity in agriculture. The mini-grid project made the increase in income possible, but some of the increase was due to land resources, not increased access to electricity. So evaluators would need to control for the variable of land use to measure the mini-grid’s impact on income from agriculture.

At the beginning of the project, developers need to collect baseline data, the starting point for each indicator. During project assessment, the evaluator measures the indicators again and compares the values to the baseline data. The difference between the two measurements indicates the impact of the project.

Impact evaluations are identified by the type of methods used to generate the counterfactual analysis. Impact evaluations that use formal experimental designs yield the most accurate results. In an evaluation with an experimental design, the evaluator randomly selects a sample of individuals from a community and randomly assigns half of the individuals to a treatment group and half to a control group. For a mini-grid project, the treatment group would receive the energy services while the control group would not. To measure the project’s impact, the evaluator compares the benefits gained by the treatment group against those gained by the control group.

Experimental designs have drawbacks. First, they’re expensive, because they require skilled evaluators to design the experiment, collect accurate data and evaluate results. Measuring impact with an experimental design may increase project costs for mini-grid developers. Perhaps more importantly, many development practitioners would consider it unethical to provide energy services to some members of the community but deny them to others simply to measure the project’s impact more accurately. The control group may be reluctant to participate since they would not receive energy services.

Non-experimental impact evaluation techniques such as cross-sectional comparisons and panel designs focus on comparing intervention groups before and after implementing an intervention. Though not as rigorous as counterfactual studies, non-experimental approaches are simple, practical and less expensive ways to assess the impact of mini-grid projects. The developer can collect individual data throughout the life of the project and compare it to aggregate socio-economic or energy data from local development authorities or national agencies.

Cross-sectional Comparisons

Cross-sectional comparisons measure impact through individual surveys at the end of project implementation. For mini-grid projects, cross-sectional comparisons can show how broader project outcomes—like increasing a community’s total energy supply—impact individuals. Surveys can identify the type and quality of energy services an individual household receives. Evaluators also collect data about improvements to health, education and income.

Panel Designs

Panel designs also measure impact on the individual level, but evaluators measure impacts before, during and after project implementation. Evaluators engage directly with individuals, such as holding a face-to-face meeting, to determine how their livelihoods change as a result of energy services. By directly engaging with individuals over time, evaluators can identify and control for the impacts of other development projects, leading to a more accurate picture of a mini-grid's impact.

To ensure equity, the mini-grid developer needs to assess a project’s impact on different members of the community, including women and other marginalized groups. The project’s monitoring and evaluation system should track impacts by gender, income and other socio-economic factors. Otherwise, developers might not realize that some groups are benefitting significantly more than others.

For example, a general impact evaluation might identify increased economic activity as a result of new energy services. A gender-focused analysis, however, might reveal that only men are benefiting from that economic growth. The project might be providing energy services for productive uses across the community, but if women don’t have the financial resources or access to credit to use that energy, they will not benefit equally from the project.

While monitoring project impact, the mini-grid developer must watch for unintended consequences. Mini-grid projects might create or exacerbate inequalities in a community. For example, in an area where income levels vary widely, a uniform tariff might limit electricity access to the wealthier members of the community. Families that couldn’t afford the tariff would not gain access to energy.

The impacts of energy projects on development goals can be difficult to measure. Energy is not usually an end in itself, but rather an input into other sectors. The chain of impact from the energy service to livelihood improvements can be long, convoluted and difficult to track. Access to electricity impacts many different sectors.

In addition, the benefits of energy projects may take years to emerge. Communities might see the benefits of energy services after the project itself has ended. To measure real impact, an evaluation might have to continue long after the life of the project.

Furthermore, to achieve a desired impact, an energy project might depend on developments in other sectors. For example, increasing energy services can spur economic growth, but only if the local economy has raw materials, machinery for production, financing for economic activities and markets. Project developers must understand how energy is linked to these factors in order to assess a project’s economic development impact.