Using Integrated Data to Assess and Monitor Regional Third Sector Initiatives

• In Event: Mapping the third sector

Abstract

The evaluation of community wide strategies can benefit from access to data of various types, particularly individual-level administrative data maintained by governmental and nonprofit entities. One approach involves integrated data systems (IDS) to bring together data from various sources, reflecting service usage, health status, and well-being outcomes. Records are linked at the individual level and reveal patterns of experience that can inform practice and policy. This presentation explores a 20 year undertaking of data integration in Cuyahoga County, Ohio (Cleveland).

Governmental and nonprofit agencies routinely maintain administrative data, and these are often used to describe patterns of service use, risk factors, and outcomes. But individuals often utilize other services at the same time, and they traverse various public systems and nonprofit programs over time. The integration of administrative records across agencies and time has the potential to provide new types of information that can be utilized by decision makers to evaluate outcomes, target resources and gain understanding of how the collective work of agencies and systems are addressing the needs and concerns in their communities. Technical advances in data transfer, management, analysis and visualization now make it possible for this type of longitudinal, cross-system information to be made available in a timely fashion.

The challenges of working with large administrative data sets are well documented, and as such numerous strategies have been developed to maximize their effective use. Electronic individual records are matched using a combination of deterministic and probabilistic matching techniques and the resulting longitudinal files serve as the data sources for analyses. Logistic and analytic challenges include maintaining access to records and data security, ensuring the reliability of initial matching procedure, and constantly improving methods for resolving data discrepancies and redundancies.

The Child Household Integrated Longitudinal Data (CHILD) System was developed in partnership with a group of data steward organizations and funding entities. The system serves as a key resource in that it enables data-driven planning and decision making. It is also used to monitor access to services, scope and reach of services, and to enable and support evaluation activities that are both program specific and system wide. Administrative records are matched together from more than 35 data sources to construct individual-level records. The protected data are maintained in a highly secure computing environment and data analysis files are stripped of identifying information.

The work to date demonstrates (1) integration of administrative data provides an opportunity to better understand the cross-system experiences and outcomes of those served; (2) challenges to data integration include access to source data, maintaining data security, and the technical aspects of data linkage; and (3) regional data integration presents an opportunity to inform local decision making as it relates to understanding community needs and tracking the effects of interventions. Integrated data systems can play an important role in informing local programs and policies and evaluating their effectiveness. Effectively used, an IDS can serve as a tool to support the undertaking of innovative approaches to improve social outcomes and better evaluate their success.

References

Actionable Intelligence for Social Policy Network. (2017). Philadelphia: University of Pennsylvania.
Allard, S. W., Wiegand, E.R., Schlecht, C., Datta, A. R., Goerge, R. M. & Weigensberg, E. (2018). State Agencies’ Use of Administrative Data for Improved Practice: Needs, Challenges, and Opportunities. Public Administration Review 78 (2): 240-250.
Annie E. Casey Foundation. 2018. Using Integrated Data to Identify and Protect Children from Educational Risk: A Cleveland Case Study. (December 13, 2018).
Arthur, K. C., Luceko, B.A., Sharkova, I.V., Xing, J. & Mangione-Smith, R. (2018). Using State Administrative Data to Identify Social Complexity Risk Factors in Children. Annals of Family Medicine 16 (1): 62-69.
Commission on Evidence-based Policymaking. Evidence-based Policymaking Commission Act of 2016. P.L. 114-140.
Coulton, C., Richter, F., Kim, S. J., Fischer, R. L., & Cho, Y. (2016). Temporal Effects of Distressed Housing on Early Childhood Risk Factors and Kindergarten Readiness. Children and Youth Services Review 68: 59-72.
Crosas, M., King, G., Honaker, J., & Sweeney, L. (2015). “Automating Open Science for Big Data.” Annals of the American Academy of Political and Social Sciences 659 (1): 260-273.
Culhane, D. P., Fantuzzo, J., Rouse, H. L., Tam, V. & Lukens, J. (2010). Connecting the Dots: The Promise of Integrated Data Systems for Policy Analysis and Systems Reform. Intelligence for Social Policy, Available at http://repository.upenn.edu/spp_papers/146
Fantuzzo, J. & Culhane, D. (eds.). 2015. Actionable Intelligence: Using Integrated Data Systems to Achieve a More Effective, Efficient, and Ethical Government. New York: Palgrave MacMillan.
Fischer, R. L., Lalich, N., & Coulton, C. (2008). Taking it to Scale: Evaluating the Scope and Reach of a Community-Wide Initiative on Early Childhood. Evaluation and Program Planning 31: 199-208.
Fischer, R. L., Richter, F. G. C., Anthony, E. A., Lalich, N., & Coulton, C. (2019). Leveraging administrative data to better serve children and families. Public Administration Review, 79(5), 675-683.
Gomatam, S., Carter, R., Ariet, M. & Mitchell, G. (2002). An Empirical Comparison of Record Linkage Procedures. Statistics in Medicine 21: 1485–1496.
Groves, R. M., & Schoeffel, G. J. (2018). Use of Administrative Data in Evidence-Based Policymaking. The Annals of the American Academy of Political and Social Sciences 678: 71-80.
Harron, K., Dibben, C., Boyd, J., Hjern, A., Azimaee, M., Barreto, M.L., & Goldstein, H. (2017). Challenges in Administrative Data Linkage for Research. Big Data & Society 4 (2): 1-12.
Kitzmiller, E M., & Burnett, T.C. (2015). The AISP Network: Three Organizational Models for Building, Using, and Sustaining Integrated Data Systems. In Fantuzzo, John and Culhane, Dennis P. (Eds.) Actionable Intelligence: Using Integrated Data Systems to Achieve a More Effective, Efficient, and Ethical Government. Pp. 169-190. New York: Palgrave McMillan.
Mayer, D. J., & Fischer, R. L. (2023). Exploring data use in nonprofit organizations. Evaluation and Program Planning, 97. April.
Mergel, I., Rethemeyer, R. K., & Isett, K. (2016). Big Data in Public Affairs. Public Administration Review, 76 (6): 928-937.
Petrila, J. (2011). Legal Issues in the Use of Electronic Data Systems for Social Science Research. Actionable Intelligence for Social Policy (AISP), University of Pennsylvania.
Roos, L. L., Walld.R., Wajda, R. et al. (1996). Record Linkage Strategies, Outpatient Procedures, and Administrative Data. Medical Care, 34 (6): 570-582.
Saunders, R C., & Heflinger, C. A. (2004). Integrating Data from Multiple Public Sources: Opportunities and Challenges for Evaluators. Evaluation, 10 (3): 349–365.

Author

• Robert L Fischer, Case Western Reserve University