CHALLENGES OF GOAL EVALUATION

The measurement of short-term goal realization poses inevitable problems resulting from the complexity of goals, their personal character, and their dynamics. First, a person may be engaged in the realization of multiple personal goals at the same time (Little, 2006). A measure intended to assess goal realization should take this into account, allowing for the evaluation of multiple goals (e.g., Koestner et al., 2008). Second, each person establishes, maintains, and strives to attain a different set of personal goals (Laguna et al., 2017; Nurmi et al., 2009; Salmela-Aro et al., 2000), specific to this person only and expressing his or her personality in specific life situations (Little, 2006). As autonomous motivation is related positively to goal progress (Koestner et al., 2008), analysis of self-selected personal goals seems important when studying goal realization (Sheldon et al., 2010). Therefore, a measure of personal goal realization has to allow for the expression of person-specific goals and to evaluate progress in their realization. Third, people may change their engagement in particular goal realization over time, changing their goal priorities, which is especially prominent in the case of short-term goals. A measure should therefore assess shot-term engagement in goal realization and allow researchers to evaluate it repetitively over time.

Based on all these considerations, we propose a measure of short-term goal realization: the Daily Goal Realization Scale (DGRS). It was built with the intention to enable the idiographic evaluation of progress in the realization of goals chosen by a particular person (Little, 2006). The DGRS was intended to enable multiple goal evaluations which, additionally, could be repeated many times (i.e., daily), as needed in intensive longitudinal research (Bolger & Laurenceau, 2013). To serve these purposes, the measure should be reliable but very short, so as to reduce respondents’ fatigue when enquired repetitively.

To demonstrate why a new measure is needed, it is worth analyzing the problems faced when evaluating personal goals. Usually, with some notable exceptions (e.g., Milyavskaya et al., 2012; Sheldon et al., 2010), single items were used for goal evaluations. As an illustrative example we use the Personal Projects Analysis (PPA; Little & Gee, 2007). Applying this flexible tool consisting of several modules, a researcher chooses the modules they want to apply. Usually, personal goals are assessed in a two-step procedure, applying the first two PPA modules. First, participants write down the goals they consider most relevant and personally valid for them. Second, they rate each goal on several dimensions selected for a particular study. Each of these dimensions is a single item with an 11-point scale (e.g., plan clarity: from 0 – have no plan of project realization to 10 – have a clear and concrete plan). Different sets of PPA dimensions were used in different studies (Nurmi et al., 2009; Salmela-Aro et al., 2000) and similar evaluations were applied in other research (e.g., Koestner et al., 2008). Because reliability of single item evaluations is usually unknown (test-retest correlations are rarely available), as a remedy some researchers combined several PPA dimensions into scales, for example, using the total score on five single-item dimensions (Laguna et al., 2017). A similar solution was applied in other studies (e.g., Milyavskaya et al., 2012; Sheldon et al., 2010). Thus developed, the scales were then evaluated psychometrically; their factorial structure and reliability were estimated. Such measures, however, were merely ad hoc solutions developed for a specific study, and none of them was systematically tested psychometrically. Moreover, there has been no specific measure of short-term goal realization. Therefore, a measure with known psychometric properties that enables the assessment of short-term personal goal realization is proposed.

CURRENT RESEARCH

In this paper we present the development and initial validation of the brief measure of daily goal realization, the DGRS, allowing evaluation of personally chosen idiographic goals. Three studies are reported. In Study 1 we tested the internal structure of the initial set of items, selected the final set of items for the DGRS, and tested its internal consistency. Next, in Study 2, the DGRS was applied in a daily diary allowing the multilevel evaluation of its factorial structure and testing its reliability both on the goal and on the person level. Thus, Study 2 also demonstrates the usefulness of the DGRS in investigations concerning the daily realization of personal goals. Study 3 demonstrates its usefulness in investigations of multiple personal goals. We also tested the reliability and construct validity, established the measurement invariance, and estimated the criterion validity of the measure, thus conducting an initial validation of the DGRS.

When estimating criterion validity of the DGRS, we expected that daily goal realization would be positively correlated with other goal scales, selected from the PPA (Little & Gee, 2007) – namely, with the evaluations of progress in goal achievement, engagement, likelihood of success, and goal importance. Of all the dimensions proposed in the PPA, we selected the ones that were most likely to be related to the daily level of goal realization. According to Little and Gee (2007), the progress dimension is the current level of success in achieving the goal. This is why we assumed that there would be a positive correlation between DGRS scores and this PPA dimension. Engagement (also called absorption) is defined as the actual extent of involvement in the project (Little & Gee, 2007), which has been found in many studies to be associated with persistence in goal realization (Locke & Latham, 2019). We therefore expected that engagement would be positively correlated with daily goal realization. Likelihood of success is the belief in successful goal achievement (Little & Gee, 2007). Research shows that people are more likely to pursue the goals that they perceive as attainable compared to those in which they have little hope of succeeding (Milyavskaya & Werner, 2018). We therefore expect positive correlations between this PPA dimension and DGRS scores. The last goal dimension that we selected is goal importance – the extent to which the achievement of a goal is important to a person (Little & Gee, 2007). Goal importance is directly linked to goal realization, because assessing one goal as more important than another leads to prioritization and greater perseverance in pursuing that goal (Austin & Vancouver, 1996); it also stimulates motivation to engage in the pursuit of valued goals (e.g., Savina, 2013). Consequently, we expected positive correlations between goal importance and daily goal realization as measured by the DGRS.

PARTICIPANTS AND PROCEDURE

In sample 1A the participants were 72 university students of different majors (58 women), aged 19 to 29 years (M = 22.38, SD = 1.73). We invited participants during their classes at the university and via social media. Those who agreed to take part in the study provided their email addresses and were then invited by e-mail to complete an online questionnaire. Participation in the study was voluntary and no compensation was provided.

Sample 1B consisted of 318 participants aged 25 to 55 years (M = 40.81, SD = 8.49; 172 women). They lived in cities with more than 200,000 inhabitants (28.9%), 16.0% lived in cities with 100,000-200,000 inhabitants, 23.9% in cities with 20,000-100,000 inhabitants, 11.0% in cities with up to 20,000 inhabitants, and 20.1% lived in the countryside. In terms of educational attainment, 0.9% had completed primary education, 6.9% vocational education, 34.3% secondary education, 57.6% had a postgraduate degree, and 0.3% had some other qualification. The sample was gathered on-line using the research panel of participants who were rewarded with points in a loyalty program. Participants were recruited using quota sampling to guarantee a sufficient number of individuals in each age and gender range.

MEASURE

Based on the theoretical assumptions outlined in the introduction, we developed seven items, all of them describing a single construct, i.e. daily progress in goal realization (see examples in Supplementary material). In sample 1A participants were asked to list their three personal goals that they currently pursued (i.e., on the day of the study), and in sample 1B they listed one personal goal. Then, they rated the daily engagement in the process of realization for each of these goals, responding to seven items from the initial pool on a scale from 1 (strongly disagree) to 7 (strongly agree).

DATA ANALYSIS

Using the data from sample 1A we performed principal component analysis (PCA) with Oblimin rotation (delta = 0) and Kaiser normalization on a pool of answers provided for each of the seven items.

Using the data from sample 1B we performed confirmatory factor analysis (CFA) using maximum likelihood estimation. When evaluating model fit we applied the chi square (χ²), root mean square error of approximation (RMSEA), comparative fit index (CFI), and standardized root mean square residual (SRMR). Values below .08 for the RMSEA and .09 for the SRMR, and values higher than .90 for the CFI indicate an acceptable fit (Schweizer, 2010).

Cronbach’s α coefficient was used to estimate the reliability of the measure.

PCA RESULTS

Evaluations of three personal goals by each participant from sample 1A resulted in 215 responses to each of the initial seven items (one person listed and evaluated only two personal goals). A preliminary analysis of the data showed their relevance for the intended analyses (KMO = .84, Bartlett test: χ² = 888.74, df = 21, p < .001). The Kaiser criterion (eigenvalue greater than 1) indicated two components which together explained 74.88% of the variance in all variables. Both factors concern daily goal realization. The first factor explained 56.44% of the variance and comprised four items (see Supplementary material) with factor loadings from .75 to .97; none of them had cross loadings higher than .30. Cronbach’s α of this scale was .92. The second factor comprised three items (e.g., “Today I have been motivated to accomplish my goal”) with factor loadings from .77 to .83. Cronbach’s α of the second factor was .73.

The aim of the study was to develop a brief, unidimensional and reliable measure of daily goal realization. These requirements were fully met by the items belonging to the first factor, as the scale had higher reliability and its items have high factor loadings. Thus, these four items were selected for the Daily Goal Realization Scale (DGRS).

Each of the DGRS items was positively correlated with the total score (calculated excluding that item); item-total correlations ranged from .77 to .88, showing high discrimination of all items.

CFA RESULTS

Using data from sample 1B, the unidimensional model built from four items – selected based on the PCA results – was fitted to the data. The CFI and SRMR model fit indices indicated acceptable fit, while RMSEA did not: χ²(2) = 36.58, p < .001, RMSEA = .234, CFI = .955, SRMR = .035. Therefore, as suggested by the modification indices, we added one covariance between the error terms of items 2 and 3, which resulted in acceptable model fit: χ²(1) = 1.48, p = .224, RMSEA = .039, CFI = .999, SRMR = .006. The factor loadings in this model ranged from .71 to .83. Cronbach’s α = .89.

CONCLUSIONS

Study 1 allowed for development of a brief four-item DGRS (see Supplementary material), which may be easily applied in studies with multiple measurements. The scale demonstrated high reliability.

PARTICIPANTS AND PROCEDURE

The sample included 180 participants (133 women) aged 19 to 41 years (M = 21.98, SD = 2.50).

We approached university students of different majors during their classes at the university and via social media and invited them to participate in a seven-day diary study. Participation was voluntary and no compensation was offered. Those who agreed to take part in the study provided their e-mail addresses. They were then invited by e-mail to an initial online questionnaire, including a consent form.

Each evening the following week, starting on Monday, participants received a link to an online survey. They were asked to write down a personal goal that they intended to pursue the next day. The following day, they were asked to assess the realization of that goal by completing the DGRS and to name a goal for the next day.

MEASURE

We used the four-item DGRS, developed in Study 1.

COMPLIANCE WITH INSTRUCTIONS

A total of 1,005 daily goal evaluations were collected during the seven consecutive days. The minimum number of completed daily surveys was 3 (M = 5.58, SD = 1.36). The mean daily sample size was 143.60 (SD = 20.60).

DATA ANALYSIS

We performed a multilevel confirmatory factor analysis (MLCFA), using maximum likelihood estimation with robust standard errors (MLR), to assess how well the data fitted the single-factor structure suggested by the Study 1 results. The two-level MLCFA allowed us to decompose the total sample covariance matrix into pooled within-individual (Level 1) and between-individual (Level 2) covariance matrices and to use these two matrices in the analyses of the factorial structure at each level. When evaluating model fit we applied the Satorra-Bentler scaled chi square (S-Bχ²), RMSEA, CFI, and SRMR. Data were analyzed using Mplus v.7.0 (Muthén & Muthén, 2012). We calculated composite reliability (CR), based on the MLCFA results, to estimate the internal consistency of scale scores.

DESCRIPTIVE STATISTICS, CORRELATIONS, AND VARIANCE DISTRIBUTION

The average DGRS scores obtained on each of the seven days ranged from 4.50 to 4.79 (Table 1), showing variability across days (standard deviation varied from 1.59 to 1.75). Daily DGRS scores were uncorrelated or demonstrated low intercorrelations (ranging from −.18 to .25). Such diverse correlations are due to the fact that each day participants listed and evaluated different personal goals and the level of realization of one of these goals may have been unrelated to the level of realization of a goal chosen for a different day. These results confirm that the measure captures goal-specific daily realization of personal goals.

To check if there is both within-individual and between-individual variance in daily goal realization, we checked variance distribution (Nezlek, 2012). The MLCFA results showed that there was variance in this variable at both levels (1.55 at the within-individual level and 0.63 at the between-individual level). The intraclass correlation coefficient (ICC), being the ratio of the between-individual level variance to the total variance in the DGRS scores, was .29. This demonstrates that daily goal realization differs between individuals and, even more prominently, between days. These results support the choice of MLCFA as the data analysis strategy.

FACTORIAL STRUCTURE

The MLCFA results showed that the unidimensional factorial structure of the DGRS did not fully fit the multilevel data – CFI and SRMR model fit indices indicated acceptable model fit, while RMSEA did not, S-Bχ²(4) = 59.79, p < .001, RMSEA = .118, CFI = .969, SRMR_(within) = .023, SRMR_(between) = .019. Likewise in sample 1B, as suggested by the modification indices, we added one covariance between the error terms of items 2 and 3 on the within-individual level. The model fit indices demonstrated acceptable fit of this model, S-Bχ²(3) = 20.71, p < .001, RMSEA = .077, CFI = .990, SRMR_(within) = .009, SRMR_(between) = .017. The factor loadings of all items were statistically significant and ranged from .83 to .94 on the within-individual level and from .92 to 1.00 on the between-individual level.

INTERNAL CONSISTENCY

To evaluate the internal consistency of the DGRS, we used Cronbach’s α and CR indicators (Table 1). The values of Cronbach’s α for the seven days ranged from .92 to .97. The CR values were .94 on the within-individual level and .98 on the between-individual level. Thus, both indicators showed high internal consistency of DGRS scores on both levels.

CONCLUSIONS

The results of this daily diary study confirm that the DGRS is a reliable single-factor measure and that this factorial structure well represents the personal goal realization process both on the goal (within-individual) level and on the person (between-individual) level. This demonstrates the construct validity of the measure.

PARTICIPANTS AND PROCEDURE

The participants were 56 adults (37 women) aged 18 to 51 years (M = 24.98, SD = 8.03). In the sample, 36 participants (66.03%) had secondary education, 18 participants (33.97%) had higher education, and 2 participants (3.6%) had lower secondary education.

We invited psychology students to help us gather the data. Each student was asked to deliver paper-and-pencil questionnaires and explain the instructions (described in the Measures section) to at least three persons. Participation in the study was voluntary, with no compensation.

MEASURES

We performed goal evaluation by applying the Personal Projects Analysis (PPA) method (Little & Gee, 2007). At the beginning, the participants were asked to list their personal goals, referring to actions that they were currently considering. They listed as many goals as possible. Then they were asked to choose 10 goals from their list that were central to them and that they intended to actively pursue in the coming months. Next, using 11-point scales, they rated each goal on four dimensions: progress (“How successful have you been in this project so far?”; answers from 0 – had no success at all to 10 – very successful), engagement (“To what extent do you become engrossed or deeply involved in a project?”; answers from 0 – uninvolved to 10 – generally absorbed), likelihood of success (“How successful do you believe this project will be?”; answers from 0 – total failure to 10 – entirely successful), and goal importance (“How important is this project to you?”; answers from 0 – not at all important to 10 – very important).

The DGRS was used to evaluate daily goal realization of each of the personal goals elicited from the participants using the PPA.

COMPLIANCE WITH INSTRUCTIONS

Nearly all participants evaluated their 10 personal goals; only two respondents listed and evaluated 7 goals (M = 9.84, SD = 0.67). In total, we collected 554 individual goal evaluations.

DATA ANALYSIS

To verify the factor structure of the DGRS, we performed a two-level MLCFA using MLR estimation. The CR indicator was used to evaluate the internal consistency of scores. Next, we tested the criterion validity of the DGRS using multilevel modelling (Nezlek, 2012). The multilevel approach allowed us to take into account the multilevel structure of personal goals, with personal goals as Level 1 (within-individual level) and person as Level 2 (between-individual level). Controlling for the grouping effect on Level 2, we analyzed the relationships between four PPA goal evaluations (Little & Gee, 2007) as predictors and the DGRS score as the dependent variable. Finally, we tested the measurement invariance of the DGRS across sex using multigroup confirmatory factor analysis (MGCFA; Meredith, 1993; Steenkamp & Baumgartner, 1998). We examined a series of increasingly restricted models. First, we estimated the configural invariance model, a model without any cross-group constraints. Second, we tested the metric invariance model, in which factor loadings were constrained to be equal across groups. Third, we estimated the scalar invariance model, in which factor loadings and intercepts were constrained to be equal across groups. When a measure does not achieve full invariance, it is reasonable to test for partial invariance, allowing some parameters to vary across groups (Steenkamp & Baumgartner, 1998). MLR estimation was used. To test the differences between increasingly restricted nested models, we calculated Δχ² and ΔCFI (Chen, 2007). An absolute difference in CFI smaller than .01 indicated measurement invariance. The additional criteria were differences in model fit – a change by less than .015 in RMSEA and a change by less than .030 in SRMR.

DESCRIPTIVE STATISTICS, CORRELATIONS, AND VARIANCE DISTRIBUTION

The mean DGRS score ranged from 2.96 to 3.52 (Table 2), and the mean score estimated for all goals was 3.30 (SD = 1.78). Correlations between evaluations concerning the realization of different goals ranged from −.14 to .39; only some of them were statistically significant. This demonstrates that people are engaged in the realization of their personal goals to various degrees on a specific day.

To check if there was both within-individual level and between-individual level variance in DGRS scores, we checked variance distribution (Nezlek, 2012). The MLCFA results demonstrated that there was variance at both levels (2.75 at the within-individual level and 0.45 at the between-individual level). The ICC for DGRS scores was .14. These results show that people differ in their daily realization of different personal goals and that the variance between goals is higher than the variance between people, who nevertheless do differ in their general tendency to engage in short-term goal realization.

FACTORIAL STRUCTURE

The results of MLCFA demonstrated that the two-level single-factor model had good fit indices, S-Bχ²(3) = 10.53, p < .001, RMSEA = .068, CFI = .989, SRMR_(within) = .003, SRMR_(between) = .068. Factor loadings of all items were statistically significant and ranged from .86 to .90 on the within-individual level and from .84 to 1.00 on the between-individual level. This confirms the construct validity of the scale.

INTERNAL CONSISTENCY

The values of Cronbach’s α for the DGRS evaluations of ten personal goals ranged from .90 to .97 (Table 2). CR was .93 on the within-individual level and .96 on the between-individual level. These results indicated the high internal consistency of DGRS scores on both levels.

CRITERION VALIDITY

To test the criterion validity of the DGRS, we examined its relationships to other evaluations of personal goals, namely goal importance, likelihood of success, progress, and engagement. As expected, all correlations proved to be statistically significant and positive (Table 3).

Multilevel modelling allowed us to further verify these relationships. We applied multilevel modelling (Nezlek, 2012) to test the relationships between goal evaluations and DGRS scores (within-individual level controlled for the grouping effect). We entered predictors as group-mean centered (Nezlek, 2012). In all equations, DGRS scores were treated as the dependent variable (y) explained by the effects (γ) of an intercept and a predictor and by error terms at the between-individual (u) and within-individual levels (r). The multilevel model is presented in the equation:

y = γ INTERCEPT + γ PREDICTOR + u + r

Acknowledging that personal goals belong to a specific person, the results demonstrated that each of the PPA goal evaluations was a statistically significant predictor of DGRS scores (Table 4). This means that evaluations of progress in goal achievement, engagement, likelihood of success, and the importance of a specific personal goal are positively related to evaluation of daily realization of this specific goal. These multilevel results confirm that the evaluations of daily goal realization using the DGRS are meaningfully associated and consistent with other evaluations of personal goals.

MEASUREMENT INVARIANCE

To test the measurement invariance of the DGRS across sex, we performed MGCFA, testing the single-factor model on 355 goal evaluations (DGRS scores) obtained from women and 187 from men. The configural (unconstrained) model showed a good fit to the data (χ²(2) = 5.05, p = .080, RMSEA = .075, SRMR = .004, CFI = .996). Next, we imposed equality constraints on all factor loadings across sex. The constrained metric invariance model did not differ significantly from the configural model (Δχ² = 4.68, p = .197, ΔRMSEA = .011, ΔSRMR = .022, ΔCFI = .003). In the next step, we checked for scalar invariance. The comparison of this model with the metric model showed that the difference in chi square slightly exceeded the significance level (Δχ² = 7.81, p = .049), which indicates that scalar invariance was not achieved. However, it is worth stressing that for the remaining indicators the level of the difference was not significantly exceeded (ΔRMSEA = .004, ΔSRMR = .003, ΔCFI = .007). We therefore checked for partial scalar invariance, allowing one intercept (item 4) to not be constrained as invariant. The comparison of this model with the metric model (Δχ² = 0.72, p = .699, ΔRMSEA = .007, ΔSRMR = .003, ΔCFI = .000) revealed partial scalar invariance of DGRS scores across sex.

These results confirm that meaningful cross-group comparisons not only of regression slopes but also of mean DGRS scores across men and women are possible (Chen, 2007). Such comparison of mean scores revealed no statistically significant differences (F(1, 551) = 1.54, p = .215). This shows that daily realization of different personal goals does not differ across men and women.

CONCLUSIONS

Taking into account the fact that people pursue multiple personal goals, in this study we confirmed that the DGRS allows one to meaningfully and reliably evaluate both intra-individual differences and between-person differences in goal realization. We once again confirmed the single-factor two-level model, extending the results of Study 1 and Study 2 on construct validity. Moreover, we found confirmation for the criterion validity of the measure in the form of its expected correlations with relevant goal evaluations at the goal level. The DGRS also demonstrated measurement invariance across men and women. This allows meaningful comparisons of DGRS scores; such differences were not detected, however.

LIMITATIONS AND FUTURE DIRECTIONS

Our studies provide initial evidence concerning the psychometric properties of the DGRS, which can be extended in future research. First, as the majority of participants in all our samples were women, most of them relatively young, more research is needed to test the psychometric properties of the DGRS with older people and with men. As we found measurement invariance of the DGRS across men and women, it is likely that also in other samples the results of the DGRS will be invariant. This, however, requires further studies on more diverse samples. Second, we tested the measure’s structural (construct) validity and performed an initial investigation of its criterion validity. Further research is needed to fully validate the DGRS by examining whether it correlates logically with other variables related to goal realization on both the within-individual and between-individual levels. Especially studies including objective measures of goal attainment are welcome, being at the same time the most demanding ones. Third, the DGRS instructions and items require participants to evaluate the realization of a personal goal during a specific day. Future research may test other instructions, asking participants to evaluate goal realization in a different time frame (e.g., a week, a month). Such changes in the instructions, however, would require future validation.