Advancing family-centred care measurement in adult critical care: psychometric testing of the MPOC-SP(A) tool in South Africa

2.1 Study design and participants

This cross-sectional study included doctors, nurses, and allied health professionals who have work experience in adult intensive care units (ICUs) in South Africa. Patients, administrative staff, and students were excluded from the study. A total of 134 participants from both public and private hospitals in South Africa were recruited. The sample size was calculated as 10% of the total number of participants required for the third phase of this study (national evaluation). This total was calculated using the freely available tool StatCalc by Epi Info™.

Participants completed the 24-item MPOC-SP(A) tool via Research Electronic Data Capture (REDCap). The MPOC-SP(A) was delivered electronically to potential participants via the Critical Care Society of Southern Africa, the researchers’ network and at professional conferences. The data were collected between 22 August 2024, and 27 October 2024.

Digitally signed informed consent was obtained from all participants before any online data collection. Participant data were safeguarded by regulations outlined in the Protection of Personal Information (POPI) Act. Codes were utilized to anonymise participant identities, and no personally identifiable information was used during data analysis and dissemination of results.

Participants completed the online survey and were also given the option to participate in a retest, which was used to determine the tool’s reliability. The responses received were exported as nominal measurements, where 1 represented “not at all” and 7 represented “to a very high extent”. Each item in each domain was coded numerically to simplify the data analysis.

2.2 Data analysis

The data were extracted from REDCap, cleaned, and pre-processed in Microsoft Excel (Version 2506 (Build 16.0.18925.20076), Microsoft Corporation, Redmond, WA, USA) before being transferred to RStudio for further analysis. Descriptive statistics were used to summarise sample characteristics such as age, profession, and years of experience. The mean score and standard deviation were calculated for each domain and individual item.

The factor structure of the 24-item MPOC-SP(A) was examined using confirmatory factor analysis (CFA) by comparing various plausible hypothesised models (Figs. 1,2,3,4,5 below). This was done by comparing the models’ respective goodness-of-fit indices to identify which model best represented the test’s factor structure and subsequently assessing measurement invariance across groups. The analysis was run in RStudio, following the recommendations given by Fischer et al. [20], with the code provided in the Supplementary material.

Fig. 1.Unidimensional model.

Fig. 2.Four-factor model.

Fig. 3.Five-factor model.

Fig. 4.2nd order model (four-factor).

Fig. 5.2nd order model (five-factor).

The goodness of fit indices used were:

● A chi-square value below 5;

● Comparative fit index (CFI) and Tucker-Lewis index (TLI) values of 0.90 or greater;

● Root mean square error of approximation (RMSEA) and Standardized Root Mean Square Residual (SRMR) values between 0.03 and 0.08, and;

● A loading factor of at least 0.50 [12, 21, 22, 23].

Before CFA, the data underwent normality testing to confirm its suitability for CFA and MGCFA assumptions. Both univariate and multivariate normality were assessed separately for public and private sector participants using item scores. Univariate normality was evaluated using several tests, including the Shapiro-Wilk and Anderson-Darling tests [24, 25]. For multivariate normality, the Mardia test, Henze-Zirkler test, and Quantile-Quantile (Q-Q) plots were applied, all performed using the “MVN” package in R [26, 27, 28, 29]. Employing multiple methods aligns with best practices in normality testing, as no single method is considered definitive. Univariate normality does not guarantee multivariate normality; therefore, both assessments were necessary [30, 31]. Detailed results of the normality tests are presented in the Supplementary material.

Following CFA, measurement invariance testing was conducted both at the overall sample level and between public and private groups. The adequate factor structure identified through CFA was subjected to increasing levels of invariance constraints, progressing through configural, metric, and scalar invariance, as per a multi-group confirmatory factor analysis (MGCFA) approach [19, 20].

Furthermore, modification indices were examined for the most robust CFA model at each level where MGCFA results indicated non-invariance. Where necessary, model constraints were adjusted based on these indices. Items were either deleted or allowed to correlate, both within and across domains [32]. The raw item scores and aggregated domain scores were utilized at each level of analysis to ensure model accuracy.

If the data failed to meet the normality criteria, appropriate adjustments were applied in the CFA to account for the non-normality. Specifically, the Satorra-Bentler Chi-Square Correction was applied, accompanied by robust standard errors for evaluating individual parameters in the model. These modifications were carried out using the MLM estimator provided by the “lavaan” package in R [33, 34].

2.3 Hypothesized models for internal structure testing

Five different models were tested to determine which model provided the best fit. Two models were originally proposed by the MPOC tool developers, while three additional models were hypothesized based on the premise that all items ultimately measure a single overarching construct: family-centred care (FCC). These models were evaluated at both the overall sample level and within the private sector groups. Figs. 1,2,3,4,5 depict the five models; their definitions and descriptions are as follows:

1. The Unidimensional model:

This model hypothesizes that all 24 items load onto a single factor, family-centred care (FCC). This hypothesis assumes that all MPOC-SP(A) items assess different aspects of FCC. If this model demonstrates a good fit, it suggests that a single factor adequately explains all items, forming a unified, meaningful construct.

2. The Four-factor model:

The second model, originally proposed by the tool developers, hypothesizes a four-factor structure in which each item loads onto one of four distinct factors:

● Showing Interpersonal Sensitivity (8 items);

● Providing General Information (4 items);

● Communicating Specific Information (3 items);

● Treating People Respectfully (9 items).

This model aligns with the theoretical foundation of the MPOC-SP(A) tool, which differentiates between these four key domains of FCC [7].

3. The Five-Factor Model:

The third model tested a five-factor structure, as suggested by the developers of the original MPOC tool. This structure extends the categorization of FCC-related dimensions, distinguishing an additional factor beyond the four-factor model [5].

4. The Second-Order factor model (four-factor):

This model hypothesizes that all items first load onto the four-factor model, which then loads onto a single overarching factor—FCC. This structure suggests that the four dimensions are interrelated and contribute collectively to a higher-order construct.

5. The Second-Order factor model (five-factor):

Like the four-factor second-order model, this model assumes that all items first load onto a five-factor structure, which then loads onto a single overarching FCC factor. This model assesses whether an additional dimension improves the overall explanatory power of the instrument.

2.4 Reliability

3.1 Demographics

134 healthcare providers participated in the study, comprising 89 (66.4%) from the public sector and 45 (33.6%) from the private sector. The demographic characteristics of the participants are outlined in Table 3.

3.2 Factor structure

3.3 Reliability testing

4.1 Normality assessment

The assessment of multivariate normality using Mardia’s tests yielded mixed results. While Mardia’s Kurtosis indicated multivariate normality, Mardia’s Skewness and the Henze-Zirkler statistic suggested deviations from multivariate normality. Additionally, univariate normality tests using Shapiro-Wilks and Anderson-Darling tests revealed non-normality for each item.

These findings suggest that the data do not adhere to the assumptions of normality, which is a common prerequisite for CFA [26, 27]. The lack of normality may impact the reliability of the CFA results, particularly the chi-square statistic, which is sensitive to deviations from normality. Due to deviations from normality, CFA was conducted using robust estimation methods.

4.2 Factor structure and model fit

The confirmatory factor analysis supported a four-factor model with reduced constraints and item modifications (Model 8) as the best-fitting solution. This model demonstrated good fit indices meeting established benchmarks for acceptable model fit:

● CFI = 0.933,

● TLI = 0.918,

● RMSEA = 0.079,

● SRMR = 0.072.

When compared to similar validation studies, our findings align with a study conducted in the United Arab Emirates, where the four-factor structure of the 27-item MPOC-SP tool initially showed a poor model fit (Chi-square = 4.92 (minimum chi-square (CMIN)= 2638.66/df = 536), CFI = 0.77, TLI = 0.71, RMSEA = 0.14 and SRMR = 0.06). However, after deleting items and including covariance errors, the resulting 19-item MPOC-SP tool showed an improved fit (Chi-square = 2.62 (CMIN = 375.15/df = 143), CFI = 0.92, TLI = 0.90, RMSEA = 0.10, and SRMR = 0.06) [12].

Other validation studies did not conduct confirmatory factor analysis but instead relied on Pearson’s and Spearman’s correlation coefficient, which have more recently been deemed inappropriate tests for determining the factor structure [7, 8, 10, 11]. The improved fit indices of our model suggest that the modifications made to the MPOC-SP(A) have enhanced its structural validity in the South African context.

The four-factor structure aligns with the theorised framework of the original MPOC-SP(A) tool [7, 8]. However, the model fit varied between the public and private sectors, with the model fit being notably better in the public sector compared to the private sector, a finding that appears counterintuitive given the resource constraints and systemic challenges typically associated with public healthcare settings. This inconsistency suggests that the observed differences may not solely reflect genuine variation in FCC perceptions or practice but could be influenced by unequal sample sizes between the two groups, potentially affecting the sensitivity and stability of the model fit indices. The larger sample size from the public sector likely increased the statistical power of the analysis, thereby facilitating more stable parameter estimation and enhancing overall model performance. These findings highlight the need for sector-specific strategies to support FCC implementation and underscore the importance of contextualising psychometric assessments within diverse healthcare settings.

4.3 Internal consistency and reliability

The MPOC-SP(A) demonstrated strong internal consistency, with an overall Cronbach’s alpha of 0.93. Three of the four domains showed robust reliability (α = 0.86–0.90), while the Communicating Specific Information domain showed moderate reliability (α = 0.69).

Cronbach’s values ranging between 0.70 and 0.90, typically indicate good consistency, suggesting that items within each domain measure the same construct. However, higher alpha values may imply redundancy, where some items overlap in measuring similar aspects [15].

The pattern of consistency observed in our study aligns with findings from the original validation study on the MPOC-SP(A) tool, which reported alpha values between 0.67 to 0.88 [8]. The lower internal consistency in the Communicating Specific Information domain may be attributed to the small number of items in this domain [8].

Test-retest reliability analysis resulted in an overall ICC(A,1) of 0.416, with a 95% confidence interval of 0.155 to 0.625, indicating poor to moderate reliability. However, the domain-specific ICCs ranged from 0.520 to 0.582, suggesting moderate reliability across the different domains. The F-test demonstrated:

● F(46,47) = 2.46, p = 0.001, confirming statistical significance and temporal stability, despite moderate agreement levels.

These moderate ICC suggest acceptable test-retest reliability, particularly given the complex and dynamic nature of ICU settings. In contexts where perceptions of care may be influenced by daily variability in workflow, communication, and patient interactions, moderate agreement reflects both the tool’s stability and the natural fluctuation in responses. These values highlight the tool’s utility while also indicating potential areas for refinement or adaptation to specific clinical environments.

Despite these limitations, the statistically significant F-test results support the tool’s potential utility in assessing FCC practices in adult ICUs. Other validation studies have reported ICC values ranging from 0.22 to 0.95 [7, 9, 10, 37].

It is essential to highlight that while many studies have tested test-retest reliability using ICC, only two explicitly specified the type of ICC employed.

● Kim et al. [37] utilised a two-way mixed-effects model with absolute agreement (ICC(A,1)), which is the same method employed in our study. Their study reported ICC values ranging from 0.22 to 0.78 (n = 20), indicating poor to good reliability [37]. In contrast, our study found that each domain exhibited moderate reliability.

● Himuro et al. [9] used ICC(2,1), reporting ICC values ranging from 0.68 to 0.95 (n = 20). However, this study assumed that raters were randomly selected, making their results generalizable to a broader population [9].

The remaining two studies did not specify which ICC model was used, but both reported high ICC values:

● Woodside et al. [7]: ICC 0.79 to 0.99 (n = 29);

● Siebes et al. [10]: ICC 0.83 to 0.89 (n = 13).

4.4 Sample size considerations in ICC calculations

One important factor to consider in the four studies is their sample size, which was relatively smaller than that used in our study. According to Bujang and Baharum, the Intraclass Correlation Coefficient (ICC) can be estimated based on the sample size. They argue that when observations of a subject are conducted on two occasions, as in these studies, the minimum sample size required to achieve ICC values of:

● 0.7 is 10,

● 0.8 is 7,

● 0.9 is 5 [38].

This suggests that smaller sample sizes increase the likelihood of observing moderate to excellent reliability.

Although our ICC values are lower than preferred, this may be influenced by the sample size, as well as the dynamic nature of healthcare service delivery rather than any instability in the measurement instrument. The behaviours and approaches of providers can vary significantly across different patient interactions and clinical contexts, which may contribute to the variability observed over time.

4.5 Implications for practice

The validation of the MPOC-SP(A) provides healthcare organizations with a psychometrically robust instrument for assessing the quality-of-service delivery. The strong internal consistency suggests the tool reliably captures distinct aspects of service provision, while the moderate test-retest reliability indicates sensitivity to temporal variations in service delivery patterns.

The differential performance across public and private sectors underscores the need for sector-specific considerations in implementation and interpretation. These findings suggest that healthcare organizations should tailor their approaches based on the unique structural and operational differences within public and private ICU settings.

4.6 Limitations and future directions

A limitation of this study is the unequal sample sizes between the public (n = 89) and private (n = 45) sectors. While both meet the minimum requirements for validation, the smaller private sector sample may limit generalizability. Nonetheless, including both sectors adds contextual relevance. Future research should aim for larger, more balanced samples to strengthen external validity.