by Scott Bayley

The challenges of evaluating social services programs

Social services programs


Typical examples of state government services include: health programs, education programs, employment services, services for people with a disability, programs for indigenous peoples, housing programs, child protection, and mental health services.

The evaluation’s potential areas of focus

  • The changing nature of government policies over time (funding levels, desired outcomes, strategies).

  • The volume of outputs delivered, service coverage, adequacy of access.

  • Service quality.

  • The program’s impact on stakeholders (e.g. clients, their families, the broader community, both intended and unintended effects, interaction with other programs).

  • Equity of funding allocation across geographic regions and across different client groups.

  • Performance monitoring, evaluation and accountability mechanisms.

Systemic challenges

  • The department has not developed a program logic model or theory of intervention to specify how the program is supposed to work, any theories of action are implicit not explicit.

  • The target group cannot be serviced at agreed standards within current resource allocation. This leads to waiting lists, queuing, the “unofficial” raising of eligibility requirements or narrowing of the intended target group.

  • Infinite demand – no clear target to measure against, demand management (i.e. demand reduction) becomes management’s focus, not meeting demand.

  • No clear measure of successful outcomes or of appropriate service provision is available (ie unclear criteria and standards for service delivery).

  • Workforce issues are common – staff shortages, a need for training, high turnover, low morale. This often results in an inadequate quality of services.

  • Large amounts of administrative program data are collected, but data is of poor quality, limited ad hoc analysis is undertaken of what the data actually means. Client files are often incomplete.

  • Decentralised service provision coupled with unclear roles and responsibilities between the central administration and regional offices.

  • Contracted service delivery with inadequate contract management and quality oversight.

  • State governments often have limited control (the national government has a policy and funding role and alternative service providers are also operating in the sector) but the state has overall responsibility for ensuring good outcomes. The state's role becomes not the efficient and effective delivery of an entire service, but of efficiently and effectively plugging gaps in available services without providing perverse incentives.

Methodological problems

  • Limitations in the available data, IT systems are not user friendly

  • Privacy/ethical considerations

  • Identifying all relevant desired outcomes for the client group

  • Evaluators often have a need for independent ‘expert’ advice on substantive matters, for example when assessing the quality-of-service delivery

  • Sensitivities relating to evaluators second-guessing “clinical” judgements (although there are ways around this)

  • Decentralized service provision means time consuming and expensive evaluations with large samples

  • When service delivery has been contracted out evaluators will need the authority to examine the contracted operations (the ability to follow the money).

A big thank you to my former supervisor Pam Williams who first alerted me to many of these issues.

Scott Bayley, Managing Director of Scott Bayley Evaluation Services and former Principal Consultant for Monitoring Evaluation and Learning at Oxford Policy Management (OPM) for the Asia Pacific region.

Scott Bayley is Senior Principal Specialist, MEL at Oxford Policy Management (OPM).
Scott leads OPM Australia’s monitoring, evaluation and learning (MEL) work for the Australian Department of Foreign Affairs and Trade and the New Zealand Ministry of Foreign Affairs and Trade.

Call Scott now

on:

 +61 452 509 756

or,

 Email me

Scott Bayley Evaluation Services - Continuous Improvement

Affiliations

Fellow of the:

Australian Evaluation Society logo

Ask a question


A quick question or,

Make an appointment

Please type your full name

This field is required

Please supply a valid email address

This field is required

Please type your phone number

This field is required

Ask for a quote or ask a question.

This field is required