Introduction

Neuropsychological assessment plays a crucial role in diagnosing and subsequent treatment of acquired brain injury (ABI). It is usually structured with the primary purpose of checking individual performance to clarify the nature of various difficulties in neuropsychological domains (e.g., attention, memory, executive functioning, language, or affective functioning) and how these interact with the individual’s psychosocial environment (^{Parsons, 2011}). Besides aiming to identify neuropsychological sequela resulting from the injury, neuropsychological assessment also helps predict the severity of cognitive and affective changes in patients’ daily lives and offers pertinent information in planning individual interventions (^{Lezak et al., 2004}). Therefore, the success of rehabilitation programs depends, at least in part, on practical neuropsychological assessment.

The study of cognitive functioning in healthy individuals and individuals with ABI or neuropathology prompted the development of neuropsychological assessment and behavior prediction methods that are increasingly ecologically sound. Inferences about the individual’s functioning in daily life based on neuropsychological assessment require psychometric rigor, especially the test’s reliability, sensitivity, and clinical and ecological validity (^{Parsons, 2011}). By applying neuropsychological assessment tests, therapists can also plan a rehabilitation intervention (^{Bennett & Raymond, 2008}). However, sometimes only screening tests that lack specificity or sensitivity are used (^{Molloy et al., 2005}), increasing the false-positive or false-negative results.

Cognitive changes are difficult to identify with standard neuropsychological tests, especially in the presence of mild cognitive impairment. According to ^{Lezak (2004}), there are some lapses in the interpretation of neuropsychological tests, such as failure to demonstrate a reduced performance: the problem of false negatives. Thus, additional research is helpful to mitigate the number of false negatives. ^{Chaytor, Schmitter-Edgecombe, and Bur (2006}) stated that there is no perfect way of quantifying in cognitive abilities, and every neuropsychological battery is associated with a certain margin of error. Additionally, authors such as ^{Donovan et al. (2011}) referred that traditional tests can limit the evaluation of the impact an acquired brain injury has on the individual’s overall functioning. Their systematic literature review concluded that commonly used neuropsychological tests only evaluated individual cognitive domains instead of the traumatic brain injury’s (TBI)full impact. None of the neuropsychological tests could evaluate the full extent of skills or deficits associated with TBI or the impact of cognitive deficits in daily functioning.

The selection of a set of valid neuropsychological tests to plan the rehabilitation of patients with ABI could simply begin with studying how the patients’ performance in rehabilitation relates to the measures usually derived from these tests. Despite being based only on correlational analyses, that study would provide valuable indications about the association between a particular test’s result and the degree of individual success in the intervention, which can be helpful for neuropsychological rehabilitation. Tests that are easy to administer and provide information for therapists to develop more adequate rehabilitation plans are needed (^{Bennett & Raymond, 2008}).

Some tests that help provide information on the neurocognitive functioning of patients with TBI are often cited in the literature, such as the following: Wisconsin Card Sorting Test (WCST; ^{Berg, 1948}), ^{Wechsler Adult Intelligence Scale (WAIS-IV; Wechsler, 2008}; see also ^{Lichtenberger & Kaufman, 2009}), Trail Making Test (TMT - Forms A and B; ^{Gaudino et al., 1995}), Stroop Color-Word Test (^{Golden, 1978}; ^{Strausset al., 2006}), Wechsler Memory Scale-III (WMS-III; Wechsler, 1998), Neuropsychological Assessment Battery (NAB, ^{White & Stern, 2003}), Luria-Nebraska Neuropsychological Battery (^{Golden et al., 1985}), Mini-Mental Status Examination (MMSE; ^{Folstein et al., 1975}). However, their relationship with these patients’ performance in neuropsychological rehabilitation programs is not fully explored. Instead, professionals have focused their efforts on attempting to increase the neuropsychological assessment’s ecological validity by emulating everyday problems in items and evaluative tasks, and, more recently, by replacing the traditional paper-and-pencil tests with virtual environments, as in the case of the Virtual Reality Stroop Task (VRST; ^{Parsons et al., 2011}).

While the neuropsychological assessment is changing toward evaluation tests with greater ecological validity, it is important to continue investigating conventional tests and identify which neuropsychological assessment measures can best guide therapists in selecting neuropsychological rehabilitation tasks. This study investigates how standard neuropsychological assessment measures relate to the patients’ actual performance in cognitive rehabilitation tasks. If the test is designed to assess a particular function, we expect to observe a significant positive correlation between that function’s measure and the patient’s performance in the rehabilitation of that function. If a test is uncorrelated with the patient’s performance in the rehabilitation of that function, it has limited value for planning the rehabilitation and does not need to be administered for that purpose.

METHODS

Participants

This study comprised eight male patients admitted to a Medical Rehabilitation Center to follow a physical medicine and neuropsychological rehabilitation program after an ABI (post-acute phase). The Ethics Committee of the Rehabilitation Center approved the study, and the clinicians responsible for the cases confirmed that the patients’ sensory and motor deficits would not prevent them from participating in the neuropsychological rehabilitation program. Psychologists working at the Center were explained the procedures to implement the protocol established by the research team concerning the individual neuropsychological assessment. Data on the ABI’s severity and other clinical information relevant to the sample’s characterization were obtained from hospital discharge reports.

All participants were contacted and voluntarily participated in the study, giving their informed consent. The inclusion criteria were: 1-have acquired brain injury, 2 - with age equal to or above 18 years old, 3- motivation to integrate an intensive rehabilitation program, 4- ability and willingness to participate in group activities, 5- and no previous participation in neuropsychological rehabilitation programs. Exclusion criteria were: (1) diagnoses of psychopathological disorder; (2) problems of interpersonal relationships, irritability, or impulsivity interfering with themselves or others; (3) use/abuse of illegal substances; (4) general intellectual disability; and (5) general functional disability.

Regarding sociodemographic characteristics, all participants were male, aged 37.0 years, on average (SD = 12.17), with around 10 years of education (M = 10.37, SD = 3.35). In what concerns time after injury, it was on average of 5.09 months (SD = 4.73), with six being severe injuries and two moderates

Instruments

A neuropsychological assessment protocol comprising eight tests was individually administered to every patient before their rehabilitation program. The eight tests are described below.

The Montreal Cognitive Assessment (MoCA; ^{Nasreddine et al., 2005}; Portuguese version by ^{Freitas et al., 2010}) is a brief test aimed to screen eight cognitive domains: executive function, visuospatial ability, memory, attention, sustained attention, and working memory, language, and orientation. It entails 30 tasks, takes approximately 10 minutes to complete, and the total score ranges from 0 to 30 points, with values below 26 suggesting cognitive impairment. We used the overall score as a measure of general cognitive functioning.

The Token Test (^{De Renzi & Vignolo, 1962}) was developed to assess language comprehension. It consists of 20 pieces of different geometric shapes (squares and circles), sizes (small and large), and colors (yellow, blue, green, and white) that are laid out on a table in a defined order. The version used in this study is divided into six parts (A-F) and comprises 39 instructions for the patients to handle certain parts in a certain way. We calculated the correct number of items to assess language comprehension.

The Trail Making Test (TMT; ^{Gaudino et al., 1995}) comprises Forms A and B and evaluates attention, processing speed, mental flexibility, and space organization. Form A requires that the patient to draw lines to connect the numbers 1 to 25, randomly spatially distributed. Form B requires alternately connecting numbers (1 to 13) and letters (A to L) in a sequence (e.g., 1-A-2-B-3-C-4-D). The most common scores are the times taken to complete each task. We calculated the total time in Form B in seconds as a measure of attention and processing speed.

The D2 Attention Test (^{Brickenkamp, 2002}) was used to assess selective attention and sustained attention. It consists of 14 lines with 47 characters each, which correspond to the letters ‘p’ and ‘d’. The individual looks for the letters ‘d’ with certain characteristics and must identify them within a maximum time of 20 seconds per line. The test time may be up to 10 minutes. The results are measured according to the following indicators: a- total processed characters (an indicator of processing speed, productivity, capacity, and motivation); b- total score (an indicator of accuracy and effectiveness); c- total effectiveness (an indicator of attention control and the relationship between speed and thoroughness in the task); sustained attention index (an indicator of sustained attention capacity); variability index (an indicator of consistency in task execution); and error percentage (an indicator of performance quality). We used the raw scores as measures of attention.

The ^{Wechsler Memory Scale’s (WMS-III; Wechsler, 1998}) Space Location and Letters and Numbers Sequence subtests were used to evaluate working memory. In the Space Location subtest, visual stimuli are used, and patients are asked to touch the points on a three-dimensional board (numbered 1 to 10) in two sequences: in the direct sequence, the examiner sequentially points to the numbers on the board and the patient has to repeat the same sequence; in the reverse sequence, the patient has to touch the points on the board in the reverse order. The degree of difficulty increases from level 1 (two items) to level 8 (nine items). In the subtest Letters and Numbers Sequence, auditory stimuli are used, i.e., the examiner says alternating sequences of letters and numbers, and the patient has to repeat them, first the numbers in ascending order and then the letters in alphabetical order. Level 1 begins with a sequence of two items, and the top-level 7 comprises eight items. We used the raw scores from the Letters and Numbers Sequence subtest as measures of working memory.

The Hopkins Verbal Learning Test (HVLT; ^{Brandt & Benedict, 2001}) was administered to assess learning and verbal memory. The task consists of reading a list of 12 words to the patient, and consequently, who should then repeat as many words as possible. This task is repeated three times (tests 1, 2, and 3). After a 20-25-minute break, the evaluator asks the participant to recall once again the words previously read (test 4). Then, the evaluator reads a list of 24 words and asks the patient to identify those that were part of the initial list (only 12 words). Results are obtained by calculating total recall (total sum of correct answers in tests 1, 2, and 3), learning in test 3 (number of words marked correctly), delayed recall (number of correct answers in test 4), and semantic categories (corresponding to the total number of true positives and the total number of false positives). We assessed the number of items marked correctly in test 4 as a measure of delayed recall.

The Wisconsin Card Sorting Test (WCST; ^{Berg, 1948}) consists of using a deck of cards with geometric figures in different numbers, colors, and shapes, and the patient has to choose a way of pairing them with each other by using one of these categories (color, number, or shape). Patients receive feedback after each trial to have the opportunity of selecting an alternative category in the subsequent trial if their pairing decision was wrong. After ten consecutive correct trials, the category is considered completed, and the pairing criterion is switched to a new category (e.g., change from color to form) without the patient’s knowledge. Thus, the patient must find out the new pairing criteria through trial and error, together with logical reasoning. The task ends when the patient completes six categories (color - shape - number - color - shape - number) or when the 128 cards in the deck are over. This test assesses executive functioning through the following measures: number of trials; the total number of errors; the number of perseverative responses; the number of perseverative errors; the number of non-perseverative errors; the number of completed categories; and failure to maintain the attitude. The number of perseverative errors is considered a sensitive measure of executive dysfunction and was used as an executive functioning measure.

The Stroop Test (^{Golden, 1978}; ^{Strauss et al., 2006}) was administered in the paper-and-pencil version, consisting of three sheets with 100 words, each distributed in five columns of 20 items. The ‘blue, ’green,’ and ‘red’ words are randomly distributed and printed in black on the first sheet. The examiner asks the patient to read the words in vertical order quickly. The second sheet is composed of one hundred equal stimuli ‘XXXX’ randomly distributed and printed in blue, green, and red, and the task is to name the colors in which the ‘XXXX’ are printed. The third sheet evaluates the Stroop effect. It comprises words naming colors that are incongruent with the ones in which they are printed. The patient is instructed to read the words, regardless of the color in which they are printed. We computed the number of words correctly read in the word-color part and used it to measure cognitive flexibility and sustained attention.

Procedures

All tests were administered before and after the intervention, and all participants underwent rehabilitation, consisting of an online cognitive training program. The program was conducted three times per week (lasting 40 to 60 minutes per session, depending on the participants’ performance) for 16 weeks, in a total of 80 sessions (49 cognitive training and 31 affective and psychosocial intervention sessions). The neuropsychological rehabilitation was conducted with a web-based cognitive training program - the Neuropsychological Enrichment Program of the University of Minho NEP-UM® (NPL-nepum.psi.uminho.pt/).

Table 1: Structure and characterization of cognitive domains and subdomains that have developed rehabilitation tasks. 

The neuropsychological rehabilitation program included training in four neurocognitive domains: (1) attention, (2) working and short-term memory, (3) language, and (4) executive functioning. The tasks for cognitive training in each domain were organized in increasing difficulty levels, with the progress being adjusted to each participant’s performance. Each participant performed more tasks in the most compromised cognitive functions.

The level that each participant was able to reach and successfully perform in each neurocognitive training domain was considered a measure of performance or efficiency in the individual rehabilitation program. This efficiency measure was calculated according to the following formula: Efficiency = (accuracy rate/time) x 100, where the accuracy rate is the number of tasks successfully performed/total number of tasks in a particular domain, and time is the meantime, in minutes, spent on the tasks successfully performed by the patient. Therefore, the efficiency measures all training performance, comprising all the 49 rehabilitation sessions.

The Spearman’s correlation coefficient was used to investigate how the efficiency measure’s results were retrospectively correlated with the initial test scores to identify which of the neuropsychological assessment measures better related to the participant’s performance in the rehabilitation program. Since the initial neuropsychological assessment showed that language training was recommended only for one of the participants, no correlation was computed for this cognitive domain. We used the false discovery rate (FDR) method to correct the values of p resulting from multiple correlations (^{Benjamini & Hochberg, 1995}).

The level that each participant was able to reach and successfully perform in each neurocognitive training domain was considered a measure of performance or efficiency in the individual rehabilitation program. This efficiency measure was calculated according to the following formula: Efficiency = (accuracy rate/time) x 100, where the accuracy rate is the number of tasks successfully performed/total number of tasks in a particular domain, and time is the meantime, in minutes, spent on the tasks successfully performed by the patient. Therefore, the efficiency measures all training performance, comprising all the 49 rehabilitation sessions.

The Spearman’s correlation coefficient was used to investigate how the efficiency measure’s results were retrospectively correlated with the initial test scores to identify which of the neuropsychological assessment measures better related to the participant’s performance in the rehabilitation program. Since the initial neuropsychological assessment showed that language training was recommended only for one of the participants, no correlation was computed for this cognitive domain. We used the false discovery rate (FDR) method to correct the values of p resulting from multiple correlations (^{Benjamini & Hochberg, 1995}).

RESULTS

Table 2 presents the descriptive data concerning the neuropsychological assessment tests before the intervention.

In what regards to the correlations between the neuropsychological assessment tests and participants’ performance in the domains of the cognitive training program (Table 3), the WMS-III is the test that is more strongly associated with participants’ performance in the rehabilitation program’s attention tasks, followed by MoCA and Stroop, all of these correlations being positive and statistically significant. The tasks related to working memory, the D2, HVLT, and TMT are the tests that correlate more strongly with participants’ performance in this domain, the first two with positive correlations and the latter negative, all statistically significant. Thus, the longer the participants take to complete the TMT’s Form B, the worse their performance is in the tasks related to working memory rehabilitation. WMS-III also presented a strong positive association with the working memory domain. Regarding executive functioning, only the MoCA score showed a moderate negative correlation with participants’ performance despite it being non-significant. All of the other scores evidenced weak to negligible, also non-significant, associations. In addition, it is essential to note that the results of the WCST only showed a moderate negative correlation, non-significant, with the working memory domain being weakly related to the other two dimensions.

Table 2: Descriptive statistics regarding the neuropsychological assessment tests before the intervention. 

In what regards the correlations between the neuropsychological assessment tests and participants’ performance in the domains of the cognitive training program (Table 3), the WMS-III is the test that is more strongly associated with participants’ performance in the rehabilitation program’s attention tasks, followed by MoCA and Stroop, all of these correlations being positive and statistically significant. The tasks related to working memory, the D2, HVLT, and TMT, are the tests that correlate more strongly with participants’ performance in this domain, the first two with positive correlations and the latter negative, all statistically significant. Thus, the longer the participants take to complete the TMT’s Form B, the worse their performance is in the tasks related to working memory rehabilitation. WMS-III also presented a strong positive association with the working memory domain. Regarding executive functioning, only the MoCA score showed a moderate negative correlation with participants’ performance despite it being non-significant. All the other scores evidenced weak to negligible, also non-significant, associations. In addition, it is essential to note that the results of the WCST only showed a moderate negative correlation, non-significant, with the working memory domain being weakly related to the other two dimensions.

Table 3: Correlation coefficients (Spearman’s r) between neuropsychological assessment tests and performance (efficiency measure) in the various domains of the web-based cognitive training program. 

This study aimed to investigate which of the traditional neuropsychological assessment measures better relates to the performance of patients with brain injury. Different areas of cognitive training, aiming to predict their success and guide the therapist toward a better selection of neuropsychological rehabilitation tasks, were considered.

The results indicate that the WMS-III (^{Wechsler, 1998}), the MoCA (^{Nasreddine et al., 2005}), and the Stroop (^{Golden, 1978}; ^{Strauss et al., 2006}) scores are strongly related to the patients’ performance in the attention tasks of the rehabilitation program. Performance in the tasks linked to memory functioning, specifically working memory, is strongly correlated with the D2 (^{Brickenkamp, 2002}), TMT-B (^{Gaudino et al., 1995}), HVLT (^{Brandt & Benedict, 2001}), and WMS-III (Wechsler, 1998) results.

The MoCA’s results (^{Nasreddine et al., 2005}) regarding the attention measure were expected because this test is sensitive to cognitive impairment screening in the post-acute phase of ABI (^{Godefroy et al., 2011}), which was the case of the participants in the study. The WMS-III proved to be a sensitive test regarding changes in memory. It is divided into subtests is advantageous because it allows a discriminating assessment of memory, controlling for several sensory-perceptive deficits (^{Spooner & Pachana, 2006}). Its ability to provide comprehensive information on the effects of neuropsychological interventions in memory was one of the reasons for its choice (Spooner & Pachana, 2006).

The Stroop test’s (^{Golden, 1978}; ^{Strauss et al., 2006}) results confirm a positive correlation with the rehabilitation program’s attention tasks. Other studies also mention this test’s discriminating capacity regarding processing speed, especially among patients with mild TBI compared to healthy subjects (^{Mathias & Wheaton, 2007}). A meta-analysis showed that the word-color interference’s effect size in populations with TBI is small (d = .05). These results prompted authors to highlight the need for a cautious approach when choosing interference measures to assess attentional deficits after TBI (^{Dimoska-Di Marco et al., 2011}).

The results of the D2 Test (^{Brickenkamp, 2002}) were not expected to be strongly correlated with working memory. However, the results of the TMT-B (^{Gaudino et al., 1995}), WMS-III (^{Wechsler, 1998}), and HVLT (^{Brandt & Benedict, 2001}) these tests point out their utility regarding this type of memory. Concerning the HVLT (Brandt & Benedict, 2001), our results agree with ^{O’Neil-Pirozzi et al. (2012}) finding that the test-retest reliability of the delayed recall sub-score was exceptionally high in patients with TBI.

Recent studies show controversy about the TMT-B’s sensitivity in assessing ABI. However, research has reported a negative linear relationship between the brain lesion’s severity and performance in the TMT (^{Lange et al., 2005}). Additionally, ^{Lezak (1995}) stated that the TMT-B is sensitive to injuries in the prefrontal lobe, which is essential to regulate working memory.

The WCST (^{Berg, 1948}) only showed a moderate negative correlation, non-significant, with the working memory domain being weakly related to the other two dimensions. The weak correlation between the WCST’s (Berg, 1948) results and the participant’s performance in the executive functioning tasks of the neuropsychological rehabilitation program is unexpected but agrees with data from other studies (e.g., ^{Bogod et al., 2003}; ^{Chaytor et al., 2006}; ^{Norris & Tate, 2000}) that did not find significant correlations between this test and measures of patients’ functioning in daily life activities. The tasks of the rehabilitation program in this study represent such activities. The WCST test’s characteristics can explain these results since it comprises items with low ecological validity and is administered in a structured way, with no possibility of using compensatory strategies at the evaluation time (Chaytor et al., 2006).

Limitations

The main limitation of the present study is the small sample size, and it would be important to replicate this study in a randomized controlled trial that provides robustness to the results. Related to this, some significant findings were presented that can be false positives. Furthermore, this study focuses on the most common forms of ABI (e.g., TBI), which means that we did not incorporate other forms of brain injury, such as Multiple Sclerosis (MS). Finally, in this study, the commonly used assessment tools for brain injury assessment were reported, and it is necessary in a future study to include new instruments and cover other domains (e.g., language). A meta-regression analysis would provide valuable information about which population benefits more from the intervention (TBI vs. stroke), what kind of cognitive training produces better results, and in which domains (multi-domain vs. single domain). Moreover, it can provide good indicators about the patients’ performance, the effects of control groups, the generalization of the cognitive improvements to real-life settings, or a transfer effect to other cognitive domains.

CONCLUSION

This study suggests that the results of conventionally administered neuropsychological tests, such as the MoCA (^{Nasreddine et al., 2005}), D2 (^{Brickenkamp, 2002}), TMT (^{Gaudino et al., 1995}), WMS-III (^{Wechsler, 1998}), HVLT (^{Brandt & Benedict, 2001}), and the Stroop test (^{Golden, 1978}; ^{Strauss et al., 2006}), can be good indicators of the patients’ performance in the cognitive training of attention and working memory. None of the tests provide useful data regarding the patients’ ability to perform the executive functioning training tasks during neuropsychological rehabilitation. This might be because those tasks appeal to everyday situations, which are not adequately represented in the administered tests, including the WCST (^{Berg, 1948}).

Neurocognitive rehabilitation is challenging, primarily if it does not correspond to the real difficulties and compromised skills in patients with ABI. The adequacy of neurocognitive rehabilitation should be based on the neuropsychological assessment results, but this assessment must provide adequate data. A better understanding of the information obtained from neuropsychological measures allows an adequate adjustment of cognitive rehabilitation and helps establish achievable therapeutic goals for ABI patients. This study contributes to this end, indicating that some results of traditional neuropsychological tests, but not all, relate particularly well to the performance of patients with ABI in the training of attention and memory. However, tests that can serve as better performance indicators for executive functioning seem to be necessary.