Poster #44 - Assessing the Reliability of the HOVRS-3 by Comparing Live and Video Recorded Home Visit Observations

• In Event: Poster Session 11

Abstract

Home visiting is a service delivery model where a trained professional provides support to
parents in their homes. Research indicates that home visiting promotes favorable child and family outcomes by offering individualized supports and services to families living in poverty (Duffee et al 2017; Duggan et al 2018). For home visits to be effective, home visitors must engage parents in supporting children’s development and research has identified specific home visiting practices associated with improved outcomes for parents and children (e.g., Peterson et al 2018; Zajicek-Farber 2010). Roggman and associates (2006) developed the Home Visit Rating Scales (HOVRS) to provide an objective measure of home visit quality and identify areas for improvement. The HOVRS has undergone several iterations and has become a commonly used tool to assess home visit quality and provide professional development to home visitors.

Although the actual HOVRS observations are usually conducted live, observers are certified by measure developers through video recordings of home visits. In live observations, an observer accompanies the home visitor, chooses a location with a clear view, and as unobtrusively as possible assesses the quality of the visit. This method allows observers to move around and ensure they can hear and view interactions. However, assessing home visit quality using video recordings can offer several advantages, including cost savings (associated with travel for observers), flexibility (videos can be collected and coded later), improved reliability (recorded videos can be coded multiple times to improve inter-rater agreement), and potentially reduce interference in the visit (without an observer physically present). Although the HOVRS-3 has been used to code home visits both live and through videos, its validation study did not compare reliability between modes (Roggman et al 2019). We seek to explore whether there are any meaningful differences between live HOVRS scores versus video.

For the Early Head Start Family and Child Experiences Study (Baby FACES 2022), we used the Home Visiting Practices Scales (Table 1) from the HOVRS-3 to assess home visit quality in Early Head Start. While most visits were only coded live with an observer physically present, we also video recorded 21 live-coded home visit observations.

As these data were collected in spring 2022, we have live-coded the observations, but have not yet coded the video-recordings. Table 2 outlines our method for coding these videos. Analytically, we will calculate correlations, examine differences in means, agreement, item to total correlations, and Cronbach’s alphas between the two modes of observing home visits.

Despite the small sample size, this study will make an important contribution by providing preliminary evidence on performance of the HOVRS-3 in assessing the quality of home visits using video-recorded data. This will be especially important moving forward, as there may be ongoing limits to allowing multiple people into families’ homes in the aftermath of the COVID-19 pandemic. This also may be a more feasible approach for assessing home visit quality, particularly in large-scale research studies. We will discuss implications of the findings in the context of the aforementioned advantages/disadvantages of each mode.

Authors

• Alex Hollister, Mathematica Policy Research - Massachussetts
  Presenting Author

• Harshini K Shah, Mathematica Policy Research - District of Columbia

• Cheri Vogel, Mathematica Policy Research - New Jersey