Evaluation Design

The overall objective of NEYAI can be stated in relatively simple terms: to improve the quality of early childhood care and education (ECCE), particularly through building the capacity of staff, in order to improve outcomes for children and their parents. This is also the overall objective of Síolta, the National Quality Framework for Early Childhood Education which was published in 2006, and is designed to define, assess and support the improvement of quality across all aspects of practice in ECCE settings for children from 0-6 years. It follows logically that a comparison of these two strands of work provide a unique opportunity to assess the effectiveness of different ways of promoting quality and outcomes in the ECCE sector.

The questions at the heart of the evaluation can also be stated in relatively simple terms:

  1. Does NEYAI and Siolta QAP have a net impact on staff, children and parents, when all measured variation is taken into account?
  2. If NEYAI or Siolta QAP, or both, have a net impact, what are the ‘active ingredients’ associated with better outcomes for staff, children and parents? In particular, does the quality improvement aspects of NEYAI and Siolta QAP have an impact over and above the generic aspects of childcare?
  3. What is the distribution of ‘active ingredients’ associated with better outcomes between NEYAI centres and Siolta QAP centres?
  4. In those centres which have some, or all, of the ‘active ingredients’ associated with better outcomes, what are the more qualitative aspects of some of those centres, as revealed through case studies?

In order to answer the first question, the evaluation involves two related elements:

  • assessing participants (staff, children, parents) in NEYAI and Siolta at baseline (before the intervention) and at follow-up (immediately after the intervention) in order to measure any changes in the intervening period;
  • comparing changes to participants in NEYAI and Siolta in order to establish the difference in outcomes between NEYAI and Siolta, thereby measuring the net impact of the intervention.

In order to answer the second and third questions, the evaluation uses advanced statistical analysis to identify all possible sources of statistically-significant variation in outcomes between NEYAI and Siolta, including the relative strength of each source of variation. Statistical analysis, however sophisticated, cannot interpret results much less infer that a statistically-significant association between variables implies that they are causally related. That is why the interpretation of results is a key part of the evaluation since establishing a causal link between inputs, pathways and outcomes is fundamentally a matter of interpretation and requires that verifiable criteria are met in order to have confidence that the interpretation is valid.

In order to answer the fourth question, the evaluation will undertake case studies in those settings where statistical analysis has already indicated that significantly better outcomes were achieved in any of the domains of child and parent well-being. This will involve site visits, consultations and a review of relevant documentation to add further insights into the ‘active ingredients’ already identified.

The focus of the evaluation is on interventions which are are deemed by projects themselves to be substantial and sustained since, otherwise, they are unlikely to produce a measurable impact. This follows logically from the fact that the best programmes for children and their parents tend to produce relatively small effects.

The overall envelope of resources enables the evaluation questions to be answered through a sample of 35 children and their parents in both NEYAI and Síolta, assessed at baseline and follow-up. This sample size is adequate for the evaluation but is also close to the minimum required. Overall, the research design will enable the evaluation to identify impacts, where they occur, and the ‘active ingredients’ associated with them. This is the type of knowledge that the initiative was set up to produce in order to establish ‘a strong evidence-base to contribute to improvements in practice and policy’.

For a download of the Research Protocol, click here.