Section 1. Changes to Survey Instrument

Q1: What is the revised DataZone SEL Survey?

The revised SEL survey consists of seven competencies and 21 items with three items (or questions) in each competency. This survey also includes College & Career Readiness (CCR) questions that are positioned to the Elementary, Middle and High School student audiences. Please note that nothing in the CCR section of the survey has changed.

Previously the SEL survey sections had included 6 competencies with 60 questions. The survey was introduced during the 2021-2022 school year after extensive review and alignment (see Q2 and Q3 on how and, respectively, why the survey was updated). A brief description of each competency is provided below:

Relationship Skills (RS): Refers to social and interpersonal skills, such as getting along with others and thinking about the feelings of other students.
Responsible Decision Making (RDM): Refers to learning strategies, such as setting learning goals and making a plan to reach them.
Self-Management School Setting (SMS): Refers to coordination of schoolwork or assignments, such as keeping track of and turning in assignments on time.
Self Awareness--Future (SAF): Refers to academic aspirations, such as improving as a student and post-secondary education goals.
Self Awareness--Learning (SAL): Refers to persistence in learning new information or engaging in challenging work.
Social Awareness Class Setting (SAC): Refers to a student’s readiness to engage with others in a classroom setting, such as participating in classroom discussions, seeking assistance, and asking questions.
Social Awareness School Setting (SAS): Refers to school climate, such as feeling safe in school and being treated with respect by adults at school.

Q2: Why was the SEL survey revised and shortened? How do I know that this revised/shorter version is a sound instrument to use?

There are three reasons why the SEL survey was shortened to the 21-item survey and organized into seven competencies: (1) To reduce administrative burden (i.e., decrease the time it takes to administer the survey), (2) reduce response burden on participants (i.e., less survey fatigue), and (3) increase the likelihood that participants will provide a valid response to 100% of the survey’s items.

Between 2017 and 2020, four DataZone partner districts participated in a 4-year pilot of an SEL measure (or survey instrument) that included 60 items. When the initial SEL survey was launched in 2017, the development and research team identified a timeline to verify the "soundness" of the instrument via statistical analyses such as factor analyses. The factor analysis process can lead to discovering items that, although they were initially thought to be associated, were not typically responded to by students in similar patterns.The items that were not informative of what students thought/believed/felt were "removed" or at least identified as unhelpful and were excluded from further analyses and inclusion in the updated SEL survey instrument.

Q3: How were the recent changes to the competencies on the survey instrument determined? Why were the item/question groupings changed for the new survey?

At the conclusion of the 4-year pilot, a series of exploratory and confirmatory factor analyses were conducted on the data that was collected. The original SEL survey instrument included 61 questions. An analysis led to the initial identification of a 47-item measure* of SEL, which required a minor reorganization of the items into seven competencies. A subsequent item analysis (conducted in the Fall of 2021) involving a review of factor loadings, missing data, and item content supported the research and assisted the analysis team in identifying a shorter survey where each competency consists of only three items, resulting in a final revised SEL survey with 21 items spread over 7 competencies that are roughly aligned with the CASEL buckets.

*The 47-item measure was limited in use during the Fall 2021 school year.

Q4: What changes can I expect to see for the College and Career Readiness (CCR) items?

No analyses were conducted on the CCR items, and therefore, no changes were made to the CCR items in the Elementary, Middle, or High School surveys.

Section 2. Metric Calculations

Q5: How is a student’s overall score (overall summed score) generated? And why is the overall score different from the scale of the individual competency score?

The overall summed score is calculated by adding the summed scores for all seven competencies together. As a result, the overall score can serve as a summary or general overview of students’ SEL levels. However, it does not provide details on which competency of SEL is higher or lower than others. For example, a student high in Competency C but low in Competency G may have the same overall SEL score as a student low in Competency C but high in Competency G. As a result, it is important to use overall SEL as only a limited summary of student SEL level. Moreover, it is important to note that competencies utilize different ranges in possible values. For example, competencies can range from 0 to 12, but the overall score can range from 0 to 84. Additionally, because individual competency summed scores are adjusted, no further adjustments are made for the overall summed score. See below chart for an example of how an overall summed score is determined. See Q7 for additional information about summed scores.

(Click image to enlarge)

Q6: How are SEL item responses weighted (i.e., what is the numerical scale that applies to SEL items)?

Each item has five response options, which are assigned a numerical value and utilize a -2 to 2 scale (see table below). A student’s response to an item is re-coded into the numerical value for data visualization and analysis. This re-coded value is considered the student’s raw score for an item. Students were also provided with the option of selecting “I don’t understand this item” for each item.

For example, if a student responds…

“Strongly Disagree” to an item, the student’s raw score for the item is “-2.”
“Neither Agree nor Disagree,” the student’s raw score for the item is “0.”
“I don’t understand this item,” or if the student does not respond to the item, the student’s raw score is considered “missing.”

Response Option	Assigned Response Weight
Strongly Disagree	-2
Disagree	-1
Neither Agree nor Disagree	0
Agree	1
Strongly Agree	2
I don’t understand this item	[Response is Considered Missing]

Q7: What is a competency summed score, and how is it calculated?

A competency summed score is the basis for a student’s adjusted summed score. A competency summed score is calculated by summing all of a student’s raw scores for the items within the competency, then adjusting the summed value by increasing it by a fixed/constant value of 6 points (the maximum amount of total possible points for the competency) to the score. If a student does not respond to an item or if a student selects “I don’t understand this item,” then the student’s raw score is considered “missing” and excluded from analyses related to the competency that item is in. See Q8 for more information on how missing values affect summed scores.

The adjusted summed score (typically shortened to “summed score”) allows for easier interpretation of visuals and analyses that utilize summed scores by eliminating negative values. The chart below provides an outline of the steps taken to calculate an adjusted summed score.

(Click image to enlarge)

Example: Like all competencies in the revised SEL Survey, example Competency A has three items. If a student responds “Strongly Agree” to Item 1 in Competency A, that student’s raw score for that item is two. If the student then responds with “Agree” to the remaining two items in that competency, that student’s raw summed score for Competency A is “4” because it sums the raw score from all three item responses (2+1+1) The constant value of 6 points is then added and the student’s final, adjusted summed score becomes “10” (4+6) for Competency A.

Q8: How were missing values handled?

Missing values are determined one of two ways: a student leaves an item blank or a student selects the response option “I don’t understand this item.” Missing values are not considered valid responses and are not used in a student’s summed scores or in calculations that utilize summed scores containing that item. If a student has one missing response for an item in a competency, that student’s summed score is excluded from analyses and visuals that use summed scores for that competency. See the chart below for an example. See Q7 for additional information about summed scores.