Social services programs
Published on Linkedin, 31 May, 2021. https://www.linkedin.com/pulse/challenges-evaluating-social-services-programs-scott-bayley/
Typical examples of state government services include: health programs, education programs, employment services, services for people with a disability, programs for indigenous peoples, housing programs, child protection, and mental health services.
The changing nature of government policies over time (funding levels, desired outcomes, strategies).
The volume of outputs delivered, service coverage, adequacy of access.
The program’s impact on stakeholders (e.g. clients, their families, the broader community, both intended and unintended effects, interaction with other programs).
Equity of funding allocation across geographic regions and across different client groups.
Performance monitoring, evaluation and accountability mechanisms.
The department has not developed a program logic model or theory of intervention to specify how the program is supposed to work, any theories of action are implicit not explicit.
The target group cannot be serviced at agreed standards within current resource allocation. This leads to waiting lists, queuing, the “unofficial” raising of eligibility requirements or narrowing of the intended target group.
Infinite demand – no clear target to measure against, demand management (i.e. demand reduction) becomes management’s focus, not meeting demand.
No clear measure of successful outcomes or of appropriate service provision is available (ie unclear criteria and standards for service delivery).
Workforce issues are common – staff shortages, a need for training, high turnover, low morale. This often results in an inadequate quality of services.
Large amounts of administrative program data are collected, but data is of poor quality, limited ad hoc analysis is undertaken of what the data actually means. Client files are often incomplete.
Decentralised service provision coupled with unclear roles and responsibilities between the central administration and regional offices.
Contracted service delivery with inadequate contract management and quality oversight.
State governments often have limited control (the national government has a policy and funding role and alternative service providers are also operating in the sector) but the state has overall responsibility for ensuring good outcomes. The state's role becomes not the efficient and effective delivery of an entire service, but of efficiently and effectively plugging gaps in available services without providing perverse incentives.
Limitations in the available data, IT systems are not user friendly
Identifying all relevant desired outcomes for the client group
Evaluators often have a need for independent ‘expert’ advice on substantive matters, for example when assessing the quality-of-service delivery
Sensitivities relating to evaluators second-guessing “clinical” judgements (although there are ways around this)
Decentralized service provision means time consuming and expensive evaluations with large samples
When service delivery has been contracted out evaluators will need the authority to examine the contracted operations (the ability to follow the money).
A big thank you to my former supervisor Pam Williams who first alerted me to many of these issues.
Scott Bayley is Senior Principal Specialist, MEL at Oxford Policy Management (OPM).
Scott leads OPM Australia’s monitoring, evaluation and learning (MEL) work for the Australian Department of Foreign Affairs and Trade and the New Zealand Ministry of Foreign Affairs and Trade.