A comparison of agency-based and self-report methods of measuring services across an urban environment by a drug-abusing homeless population

The purpose of this paper is to advance the methodology for studying service assessment by comparing self-report and agency-generated methods. This study compares 30-day self-reported service use for homeless individuals (N = 229) randomly recruited from a single urban environment (St Louis, Missouri) with similar data collected from a broad array of service agencies providing homeless, substance abuse and outpatient mental health services across the same environment. Comparisons were made between self-report and agency-based data on shelter use, outpatient mental health service use (case management, psychiatric treatment, group therapy), outpatient substance abuse service use (case management, counselling, group therapy) and drop-in/day treatment use. Consistently low levels of kappa scores (all under 0.4) and correlation coefficients (only shelter use demonstrated significant agreement) were found. Findings demonstrated that the two methods of collecting service data are generally not concordant at the individual level. Certain demographic characteristics (increased age, being male, non-white ethnicity) and diagnoses (cocaine abuse/dependence, mania, schizophrenia) were associated with decreased reliability between the two methods of data collection. The two methods of assessment appeared to capture overlapping but not identical information. Each method of assessment has different utility to researchers and providers wishing to assess service use.