The study of the diet quality should include not just the nutritional adequacy but also incorporate the multidimensional understanding and the current dietary recommendations. This study aimed to evaluate the relevance, clarity, and comprehensibility of the Scale of Quality of Diet (ESQUADA). Nutritionists’ perspectives about the relevance and clarity of the items were evaluated through focus groups between November and December 2016. An exploratory content analysis investigated the themes that emerged from the focus groups, which were organized in a structured codebook. Two reviewers applied it to the data. Inter-rater reliability was analyzed by the kappa coefficient. The coding was analyzed using the frequencies and central aspects of the sub-themes. Laypersons’ comprehension of the items was evaluated using a question regarding whether they understood each item in online questionnaire in February 2018. Descriptive statistics investigated the comprehensibility of each item. The reviewers presented adequate reliability on coding the data. Three themes emerged from the data considering the nutritionists’ suggestions. The theme Item relevance indicated that the items assessed diet quality while considering sociocultural influences. However, the nutritionists suggested attention was needed to how some items were expressed and their response options. Of relevance, the themes Item clarity and Clarity of alternatives primarily encompassed discussions regarding item text (44%) and discrimination of alternatives (24%). The study of the comprehensibility indicated that all items were readily understood. These results support the relevance of the ESQUADA for evaluating quality of diet as well the comprehensibility of all the items by individuals uneducated in public health.
Key words: Surveys and questionnaires, dietary guidelines, psychometric, comprehension, qualitative research.
El estudio de la calidad de la dieta debe incluir la adecuación nutricional además sus dimensiones y las recomendaciones dietéticas actuales. Este estudio evaluó la relevancia, claridad y comprensibilidad de la Escala de Calidad de la Dieta (ESQUADA). Las perspectivas de los nutricionistas sobre la relevancia y claridad de los ítems se evaluaron en grupos focales entre noviembre y diciembre de 2016. Un análisis de contenido exploratorio investigó los temas que surgieron de los grupos. Estos temas fueron ordenados en un libro de códigos estructurado. Dos revisores lo aplicaron a los datos. La fiabilidad inter-evaluadores fue analizada por el coeficiente kappa. La codificación se analizó utilizando las frecuencias y aspectos centrales de los subtemas. La comprensión de los ítems por personas sin conocimientos de nutrición se evaluó mediante una pregunta sobre el entendimiento de cada ítem en uno cuestionario online in febrero de 2018. Las estadísticas descriptivas investigaron la comprensibilidad de cada ítem. Los revisores presentaron adecuada fiabilidad en la codificación de los datos. Tres temas surgieron de los datos. El tema Relevancia del ítem indicó que los ítems evaluaron la calidad de la dieta mientras consideraban las influencias socioculturales. Sin embargo, los nutricionistas sugirieron la necesidad de atención en el texto de algunos ítems y sus opciones de respuesta. Los temas Claridad del ítem y Claridad de las opciones abarcaron principalmente las discusiones sobre el texto del ítem (44%) y la discriminación de las opciones (24%). El estudio de la comprensibilidad indicó que todos los ítems se entendían fácilmente. Estos resultados apoyan la relevancia de ESQUADA para evaluar la calidad de la dieta y la comprensión de todos los ítems por personas sin conocimientos de nutrición.
Palabras clave: Encuestas y cuestionarios, guías alimentarias, psicometría, comprensión, investigación cualitativa.
https://www.doi.org/10.37527/2018.68.4.003
Obesity and related diseases have been described as important public health problems with increasing prevalence in Brazil (1,2). Unhealthy diets contribute to this epidemiological status (3,4).
In Brazil, the quality of diet has been evaluated using the Healthy Eating Index and the Revised Diet Quality Index (5). These indices only focus on nutrient intake, which is insufficient to assess the complexity of the diet.
To comprehensively evaluate diet, it is necessary to extend the concept of quality of diet beyond nutritional adequacy (6,7). Quality of diet should encompass healthy and unhealthy foods, where meals are eaten, and habitual activities that take place during mealtime (7,8).
Accordingly, the 2014 Dietary Guidelines for the Brazilian Population (DGBP) (7) are based on a broad understanding of diet, incorporating recommendations for ways to eat and suitable food combinations. The guidelines also adopted the NOVA food classification, which considers industrial food-processing (9).
Public health research requires an easy-to-use instrument that comprehensively considers the quality of diet and the current dietary guidelines. Thus, the current study reported the initial stages of development of the Scale of Quality of Diet (ESQUADA), which evaluates diet considering the different settings that influence eating according to the DGBP. The present study focused on assessment of the relevance, clarity, and comprehensibility of the items of the ESQUADA.
Development of the ESQUADA was guided by recommendations for psychometric studies. The first step is a theoretical analysis of the survey items, discussing their content with experts, and assessing how laypersons interpret the items. The second step assesses the validity and accuracy of the items (10). Note that the term “item” was adopted to refer to “questions.” Additionally, the current analysis concentrated on the first step, namely qualitative and quantitative study of the relevance, clarity, and comprehensibility of each item of the ESQUADA. A subsequent study will assess the accuracy of the items for the assessment of the quality of diet (manuscript in development).
The DGBP was the theoretical reference for item development (7). The response options were based on dietary recommendations (7,9). Three nutritionists (TSSS, BSV, and MAAA) developed 56 items between January and November 2016.
Nutritionists working in public health fields were invited to participate in focus group discussions at the School of Public Health (University of São Paulo – USP) and at the Nutrition Department of the Federal University of Santa Catarina. This study aimed to assess the relevance and clarity of the developed items. The invitation was sent by email or telephone with the possibility of the recipient referring other nutritionists who might participate (11,12). The nutritionists were reminded of the focus group by the research team, the day before the group was scheduled. Each nutritionist agreed to participate in the study by signing a consent form, and completed a questionnaire that collected personal information.
After considering the nutritionists’ suggestions, Brazilian adolescents and adults were invited to complete the questionnaire in February 2018, in order to study the comprehensibility of the items. All participants were Brazilian, lived in Brazil, were 15 to 50 years old, and signed an online consent form.
Considering the objective to discuss the relevance and clarity of all items, the focus groups were organized as per three topics: 1.- food practices, 2.- unprocessed or minimally processed foods and processed culinary ingredients, and 3.- processed and ultra-processed foods. The number of focus groups was defined by the saturation criterion (11,12). Six focus groups were held between November and December 2016. Therefore, each topic was discussed in two different focus groups.
An interviewer mediated the interactions among the participants and encouraged dialogue among them.
Each focus group was conducted in the presence of an observer, who assisted with annotating the principal points of the discussions and relevant expressions of the participants. The interviewer and the observer were trained in focus group methods, and the script was previously pilot tested (11,12).
Participants were asked to introduce themselves to others and to try to express their opinions aloud in turn. Furthermore, some examples were presented of unprocessed or minimally processed foods, processed culinary ingredients, processed foods, and ultraprocessed foods, to help the participants understand the NOVA food classification.
Each of the focus groups was composed of six to eight nutritionists and carried out in an easily accessible and neutral environment (11). The groups were conducted in an atmosphere of conversation and trust so that the nutritionists would feel free to share their opinions. The discussions were guided by a semi-structured script. Some probing questions were also used to encourage discussions regarding the inclusion, exclusion, or change in the expression of the items. The discussions were audiotaped and subsequently transcribed (12).
To reduce the tedium associated with answering an extensive questionnaire, and measurement bias, the items were organized in booklets using a balanced incomplete blocks (BIB) design(13) using the package crossdress in R software. The booklets were loaded on the Survey Monkey platform and named in ascending order. The booklets were accessed through one link that organized the presentation of each booklet in ascending order. This link was shared in social media.
Each item was accompanied by an extra question that evaluated whether the participants understood the item. This question presented the following response options: I did not understand anything; I understood a little; I understood more or less; I understood almost everything, but I had some doubts; I understood almost everything; I understood perfectly, and I had no doubts(14). Besides the items that addressed the quality of diet and item comprehensibility, the booklets also included items that collected sociodemographic data.
Exploratory content analysis was used to investigate the themes and sub-themes that emerged from the focus groups’ data (15). The transcripts were read extensively by two authors (TSSS and PMS) to become familiar with the data and to identify emerging themes. Separately, they completed a first exploratory coding of the data, using excerpts as units of analysis. From those initial codes, a structured codebook was created to organize and characterize each sub-theme (15).
The transcripts and the codebook were imported into the MAXQDA version 12 software package, which assisted the analysis. Two authors (TSSS and MRC) independently applied the codebook to excerpts of the transcripts. Coding agreement was analyzed using the kappa coefficient for inter-rater reliability, as calculated using GraphPad software. Agreement was considered almost perfect when the kappa coefficient was larger than 0.8 (16). Differences in the coding were discussed until the coders reached consensus. The number and frequency of excerpts coded were also calculated. Each sub-theme was described in terms of central and peripheral aspects and the range of meanings in the data were identified.
To evaluate the comprehensibility of the items, responses to the extra question regarding the individuals’ understanding were inspected. The items with the highest frequency of response for the first three categories (I did not understand anything, I understood a little, and I understood more or less) were rewritten(14).
The Ethics Committee on Human Research at the Faculty of Public Health approved the study protocol (number 1.943.099).
Six focus groups were conducted to allow discussions among the experts regarding the 56 items. These items are listed in Table 1. The session duration ranged from 99 to 160 minutes. The focus groups included 35 nutritionists (ranging from four to eight participants per group) with a median age of 30 years (range: 23 to 71 years), 33 of whom were female.
Three themes (Item Relevance, Item Clarity, and Clarity of alternatives) and nine sub-themes emerged through the analysis (Table 2). The results of the inter- rater reliability analysis are shown in Table 3. The kappa coefficient indicated almost perfect agreement for all sub-themes. The sub-theme “Need to change the text or order of the alternatives” had the lowest inter-rater reliability (0.88).
Item relevance was composed of three sub-themes. The sub-theme “Relationship between eating behavior and consumption” considered that eating behavior might indicate the quality of diet. Regarding the item about breakfast, a related excerpt showed the item’s relevance to the relationship between eating breakfast and the quality of diet: “It already indicates that he organized the meal and reserved a time to eat…It already is an indicator of quality.” This sub-theme ranged from discussions about item relevance to discussions about the relevance of the questionnaire.
The sub-theme “Characteristics of food habits” primarily referred to item relevance for evaluating the habitual food consumption patterns of individuals, as reflected in a quote: “It is a very important item because they consume a lot of cookies.” This sub-theme ranged from opinions about the importance of the item for food consumption trends to discussions about the relevance of item as an indicator of healthy eating. For instance, a quote related to milk consumption independent of the fat content was as follows: “…I believe that when the issue is healthy eating, milk usually is associated [with it] a lot… they replaced milk with artificial juices or soda. So, I also agree that you should take out if it is full-fat or not…So, drinking milk is an indicator of healthy eating…”
The sub-theme “Marketing influence” focused on the effect of marketing on the quality of diet. The sub- theme’s range encompassed the item’s relevance for the study of the probable marketing influence on buying large portions of healthy and unhealthy foods, as in this quote: “They always think about the advantages. It is cheaper! They do not think about the quality.”
Item clarity was discussed in terms of four sub- themes. The sub-theme “Need to change the item’s expression” identified the excerpts about the need to write differently in order to guarantee better clarity. This was characterized primarily by discussions regarding poor comprehensibility of technical terms, as can be seen in a quote: “...they will not know what unnatural yogurt is...” The sub-theme’s range encompassed this need to change the terms presented in the item (including local terms or the way the item was expressed), such as in a quote related to the item “Do you usually choose the largest portion of food if there is a small difference in price?” (“…between a medium or large pizza, if the price difference is small, would you choose the large one?”).
The sub-theme “Need to include a new item” focused on the discussions about the need for new items, including the suggestion to include an item addressing the number of meals because of its relationship with daily eating practices: “And the item about how many meals a person eats? It would also be an interesting question...” The sub-theme’s range also included advice for dividing the items considering the culinary use of the foods, as in this quote: “...they never know any ingredients in the recipes. Maybe separating these items regarding what they use at the time of consumption and creating another about what they use in culinary preparations...maybe it would make it easier for them.”
The sub-theme “Difficulty with NOVA food classification” was predominately characterized by reports indicating potential difficulty for the respondents in discriminating the foods according to the NOVA food classification and urging the item to be clarified without using technical terms (“I do not think you have to use the word ‘processed’ because…it’s very technical nomenclature. So, I think if you want to facilitate the understanding of the person I think you have to avoid this kind of word.”). In contrast, this sub-theme was peripherally exemplified by encouraging the use of the term “processed,” as can be seen in this quote: “But to stimulate the use of this nomenclature we should use it in research.”
The sub-theme “Difficulty in reaching the objective” focused on discussions of items that did not addressing their objectives (“…if the objective is to check whether he adds salt or not on prepared food, the item is not related...”). This sub-theme also identified comments that an item did not clearly achieve its objective, as in this quote: “To check the habit of snacking. It is not necessarily that the snacking term in the question means goodies (cookies, candy, etc.)…he will not understand that it is necessarily goodies.” Finally, this sub-theme ranged from incoherence between the item expression and the proposed objective, to a lack of clarity in writing for younger respondents, thus making it difficult for the item to reach its objective.
Clarity of alternatives was discussed in terms of two sub-themes. The sub-theme “Need to include alternatives with a better discrimination” was primarily characterized by suggestions to include response options to better discriminate eating practices (“So, I’m also in doubt regarding whether to put none or one day…you do not know if he never eats (breakfast) or eats at least once a week…you cannot differentiate...”). In contrast, the sub- theme’s range encompassed whether to include response options unable to discriminate healthy from unhealthy consumption. This is peripherally exemplified by the quote: “…maybe I have one suggestion: ‘yes, but I do not know the kind of preparation’…maybe you would have more mistakes…I believe it is better to show this option…”
The sub-theme “Need to change the text or order of the alternatives” focused on suggestions related to the expression of the response options, as can be seen in the quote: “I think that the option ‘No’ could be substituted for: I do not usually do breakfast. So, you have to write more clearly.” This sub-theme also identified suggestions to change the order of the response options. Finally, the sub-theme’s range also included suggestions to use the same response alternatives in similar items (“…because if you follow the logic, you ask about a month or week…I think that this logic is consistent if you use the same response options for all the items.”).
This qualitative approach was based on the suggestions of the experts. In summary, the nutritionists highlighted that the items evaluated the quality of diet while considering its sociocultural influences. They also identified items that were not related to quality of diet and should be excluded. In addition to the study of the relevance of the items, the nutritionists underlined the importance of item wording and suggested changes to improve comprehensibility.
The 56 items presented to the nutritionists were changed considering the results from the exploratory content analysis. The principal changes were to the sub-themes “Need to change the text or order of the alternatives” (13.09%), “Need to include alternatives with better discrimination” (24.33%), and “Need to change the item text” (44.3%). In addition to the changes to the text, the nutritionists also suggested excluding nine items because of their irrelevance to the study of eating practices, as in this example: “it is very difficult to collect specific information about culinary ingredients if we do not explore the recipes…people do not know who cooks for them…which ingredients he/she uses.” They suggested dividing two items by the food characteristics and including another two items to address the consumption of ultra-processed meals, and the addition of processed culinary ingredients.
The remaining 52 items were organized into thirteen booklets and loaded onto the Survey Monkey platform.
Considering only complete responses, 112 individuals (63%) contributed by assessing the comprehensibility of the items. The characteristics of these individuals are shown in Table 4. Most of them were female (77%) and had no university education (51%). No item’s highest frequency of response was within the first three categories of the extra question. In general, the respondents highlighted some errors in the text and suggested using local terms and examples in the items.
One of the steps when developing research questionnaires should be contacting a group of experts on the subject to evaluate to what extent the questionnaire addresses the construct (10). While developing the ESQUADA, nutritionists were invited to discuss the items’ relevance to the study of quality of diet and the items’ clarity in order to improve the comprehensibility of the items by adolescents and adults.
The invitation to participate in focus groups was extended to nutritionists with professional experience related to public health. They worked in clinics, and universities with or without research projects. The focus groups allowed the participants to exchange their points of view in a conversational atmosphere (17). Moreover, the experience in public health yielded rich interactions regarding which items were relevant or irrelevant for studying quality of diet, and what changes were needed to assess it and to improve the comprehensibility of the ESQUADA. The ideas were coded using exploratory content analysis. The quality of this coding was confirmed by the almost perfect inter-rater reliability.
Consistent with international evidence and the DGBP, the items presented to the experts were developed based on a multidimensional understanding of the quality of diet (6,7). The sub-theme “Relationship between eating behavior and consumption” encompassed the nutritionists’ opinions about the relevance of the contextual characteristics of eating and meal regularity because of their relationship with the quality of consumption. This discussion is consistent with the studies that have identified an association between inattentive eating and greater consumption (18).
The experts also highlighted the relevance of investigating companionship while eating and cooking. Although the trend toward not cooking regularly exists, individuals who spend more time cooking the meals at home have a healthier eating and less frequently use fast-food restaurants (19). Besides the association with healthier eating (19), the frequency of consuming home- cooked meals is positively associated with better health (20). Therefore, these points reinforce the relevance of studying the regularity of cooking one’s meals.
Furthermore, the sub-theme “Relationship between eating behavior and consumption” highlighted the irrelevance of some items that assessed the use of oils, fats, and instant seasonings. The nutritionists indicated the difficulty of assessing their usual use when preparing foods because of the greater frequency of eating out (21). Therefore, these items were excluded. In contrast, one item was newly added that encompassed the addition of salt, olive oil, or vinegar to the food after it was served.
The nutritionists also suggested excluding the item about the consumption of homemade breads or biscuits (prepared without using ready-made mixes). In addition to the reduction in the customary preparation of foods at home, they indicated the indiscriminate use of the term “homemade” in food labels. The use of the term “homemade,” or similar was more frequent in the labels of ultra-processed foods, such as cake mixes and breads (22). Furthermore, consumers related that these terms might mislead them when attempting to select a food (23). Finally, the nutritionists affirmed that the item noted would not discriminate quality of diet because the term “homemade” could refer to foods either prepared at home or commercially. This item was also excluded from the pool of items.
In addition to studying the relevance of the items, the focus groups discussed the clarity of the items. The sub-theme “Need to change the item text” was the most frequent (44.3%) related to improving how the items were expressed. The frequency of this sub- theme indicates the importance of clarity to the quality of the questionnaire. When developing instruments, researchers must consider how best to express items to improve their comprehensibility (24). In the current analysis, the nutritionists also contributed to improving how the items were expressed, in accordance with their professional experience. In this sense, the experts underlined the need to improve the evaluation of fruit intake by splitting the item in two: one item considering only fruit intake and another regarding consumption of natural juices.
Some suggestions concerned the inclusion of self- exclusion response options to better discrimination of eating practices. The experts indicated that the individuals must be able to identify the option that accurately represents their eating practices to provide a correct answer. The response options must range along the construct of interest. Otherwise, misinterpretations based on the data could distort the conclusions and future recommendations (25). Therefore, the response options were changed to better discriminate the variety and frequency of eating practices.
Considering all the suggestions, 52 items remained on the questionnaire. In addition to concerns regarding the length of the questionnaire on the quality of answers, the comprehensibility of the questions was important (24). After considering the suggestions of the nutritionists, the comprehensibility of each item was directly assessed by the naïve population. Because of their good understanding, no items were excluded, highlighting the effectiveness of the focus groups in enhancing the clarity of the items.
The limitations of this study are primarily related to the lowest number of nutritionists in one of the six focus groups. However, the interviewer encouraged discussion while not directly participating in the discussions. Additionally, the study of item comprehensibility did not investigate whether the individuals experienced difficulties in using the online questionnaire. The study also has strengths. Both the interviewer and the observer were trained in the use of focus groups, and the script was pilot tested with nutritionists. The selection of participants with different professional experience in public health yielded rich interactions. The experts were helped to feel more confident with the NOVA food classification by the presentation of some examples of foods. Furthermore, the focus groups provided important discussions regarding quality of diet that improved the pool of items, and the study of the comprehensibility of the items collected the opinions of a naive population with respect to public health.
This study reported discussions regarding the items proposed for a new tool to evaluate quality of diet. The focus groups provided important information improving the content of the ESQUADA. A sample of adolescents and adults directly assessed their understanding of the tool, and the results suggested a clear understanding of the items. These results indicated adequate relevance and comprehensibility of the items for evaluating quality of diet. Therefore, the ESQUADA is ready for further research regarding its accuracy for assessing quality of diet. Finally, this study highlights the advantages of using a qualitative approach to assess experts’ suggestions when developing research questionnaires.
We gratefully thank the nutritionists and the respondents for the participation. This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brazil (CAPES) - Finance Code 001.
None.
Recibido: 13-02-2019
Aceptado: 30-04-2019