eProve surveys
Fall 2017

Student Engagement 

Measuring Student Engagement with eProve surveys

Measuring Student Engagement with eProve surveys

Student engagement has been defined as the extent to which students fully and actively participate in the process of learning. Even educators skilled and seasoned in the art and science of engaging students in their learning often still don’t know how engaged their students really are. In an effort to empower educators with tools and information to improve teaching and learning and fueled by a desire to ensure that learners realize their full potential, AdvancED developed an easy-to-use measure of student engagement for elementary, middle and high schools.

Research has shown a strong relationship between student engagement and a variety of student success outcomes (e.g., Fredricks et al.,2004; Parsons et al., 2013). For example, Finn and Rock (1997) found that low-income minority students who were observed as engaged, were more likely to be academically successful, achieve passing grades, and graduate on time. Similarly, a study conducted by Guthrie, Wigfield et al. (2004) found that increased levels of student engagement were related to increased reading achievement. In general, student engagement impacts student achievement, regardless of the content of the course or the demographic characteristics of the student (Marks, 2000). Student engagement matters. 

Measuring Student Engagement

The AdvancED Student Engagement Survey was designed to measure elementary, middle and high school student engagement through students’ responses to items about their learning experiences. The survey delves deeply into the student experience and provides additional information and actionable data to improve teaching and learning. Unlike other approaches that are focused on the adults in the building, we focus on student engagement from the perspective of the learner. This cutting edge approach to gauging student engagement provides an opportunity for schools and systems to more deeply understand not only simply whether students are engaged or disengaged but also how and in what ways they do so. 

The survey consists of 20 items categorized into three components or domains of engagement: behavioral, cognitive and emotional (Fredricks et al., 2004). Behavioral engagement refers to a student’s efforts in the classroom, while cognitive engagement examines a student’s investment in learning, and emotional engagement measures a student’s emotions or feelings about the classroom and school, in general. Each of the domains is further broken down into three categories of engagement quality – committed, compliant and disengaged. Finally, each category consists of two distinct levels. Thus, the committed category has an “invested” or “immersed” level; the compliant category has a “strategic” or “ritual” level; and the disengaged category has a “retreatism” or “rebellion” level.

The survey is available in three forms – one for elementary (grades 3-5), middle (grades 6-8), and high school (grades 9-12) students, each with similar content adapted for the age group taking the survey. As such, each version has different characteristics psychometrically and must be considered individually when scoring. Pilot testing provided evidence that each version of the tool has both reliability and validity. (For a summary of the psychometric properties of the AdvancED Student Engagement Survey, click here).

Engagement Survey Results

In general, results are categorized by engagement type and quality of engagement. Survey results provide a useful summary of the detailed information represented in students’ responses and also provides information relative to a benchmark. A respondent who finishes the survey is labeled as “Committed,” “Compliant,” or “Disengaged” across each of the three domains. This label is based on which component of engagement the respondent answers the majority of the time within each domain. It should be noted that the Behavioral domain has six items which means it is possible that a respondent has an even number of responses across two or more components. In these cases, the respondent is labeled as having a “mixed” engagement type.

In spring 2017, AdvancED conducted a pilot study of the AdvancED Student Engagement Survey in Alabama, North Dakota and South Carolina using a purposeful randomized sampling frame based on the school-level characteristics representative of each state. While the instrument will be mostly used at the school, district and state level, this article summarizes the results of the pilot test at the individual student level aggregated across all sites. The sample consisted of 7,100 elementary, 6,880 middle, and 6,514 high school students with a total of 20,494 students. Table 1 provides the racial and ethnic breakdown of the sample.

Table 1: AdvancED Student Engagement Survey Sample Demographics

Race/Ethnicity

 

High School

Middle School

Elementary School

American Indian or Alaska Native

 

1.0%

2.0%

2.7%

Asian

 

1.5%

0.9%

0.9%

Black or African American

 

27.1%

27.6%

27.1%

Native Hawaiian or Other Pacific Islander

 

0.4%

0.4%

0.6%

Two or More Races

 

7.0%

10.8%

10.2%

White

 

59.9%

55.9%

55.2%

Unknown Race

 

3.1%

2.4%

3.3%

Hispanic or Latino

 

6.8%

7.6%

8.6%

Not Hispanic or Latino

 

89.5%

89.6%

87.03%

Unknown Ethnicity

 

3.7%

2.8%

4.1%

Actual Sample Size

 

6514

6880

7100

The percentage reported for each domain was calculated by counting the number of respondents in each domain out of the total number of respondents taking part in the survey. The percentage reported for each component of engagement was calculated in the same way.

Overall, the majority of students at all grade levels were in the “Committed” category across all domains. Table 2 shows a summary of the results. However, as students went from elementary to middle to high school there was a decrease in the percentage of students in the “Committed” category across all 3 domains, with the biggest drop, 25 percent, between elementary and high school respondents in the Emotional domain.

Table 2: Percentage of Students in Each Category Across Domains

 

Cognitive

Emotional

Behavior

 

Committed

Compliant

Disengaged

Committed

Compliant

Disengaged

Committed

Compliant

Mixed

Disengaged

Elementary School

80%

18%

2%

77%

12%

11%

68%

16%

8%

8%

Middle School

67%

67%

6%

62%

17%

21%

67%

10%

7%

16%

High School

59%

32%

9%

52%

28%

20%

47%

41%

5%

7%

*Note: Some percentages may be higher than 100% based on rounding

In general, there were no significant differences in the percentage of students within each category across domains when looking at race and ethnicity. At the same time, there were significant differences based on gender across all of the different school levels, as shown in Table 3. Note that the same pattern of a lower percentage of students in the Committed domain is present in these results as well.

Table 3: Percentage of Students in Each Category Across Domains by Gender

 

Cognitive

Emotional

Behavior

 

Committed

Compliant

Disengaged

Committed

Compliant

Disengaged

Committed

Compliant

Mixed

Disengaged

Elementary School

Female

84%

14%

2%

82%

10%

9%

72%

15%

6%

7%

Male

75%

22%

3%

73%

14%

14%

64%

17%

10%

9%

Middle School

Female

70%

25%

5%

64%

15%

21%

71%

9%

6%

14%

Male

63%

28%

8%

61%

20%

19%

63%

12%

8%

18%

High School

Female

65%

29%

6%

55%

28%

18%

52%

38%

5%

5%

Male

53%

34%

12%

49%

29%

22%

43%

42%

5%

10%

*Some percentages may be higher than 100% based on rounding

We also examined differences in the percentage of students in each category for high and low levels of students eligible for Free and/or Reduced Price Lunch (Table 4), as well as schools where minority students were the majority population of that school (Table 5). As above, the percentage of students who were in the Committed category shrunk as the students aged. In addition, there were some significant differences between groups within each domain. Notably, Cognitive, Emotional and Behavioral commitment at both the high and middle school level was significantly higher in schools with a greater percent students eligible for Free/Reduced Price Lunch.

Table 4: Percentage of Students in Each Category Across Domains by Free/Reduced Price Lunch

 

Cognitive

Emotional

Behavior

 

Committed

Compliant

Disengaged

Committed

Compliant

Disengaged

Committed

Compliant

Mixed

Disengaged

Elementary School

High

80%

18%

2%

80%***

10%

10%

68%

16%

8%

8%

Low

79%

18%

3%

73%

14%

13%

68%

16%

8%

7%

Middle School

High

69%***

25%

6%

65%***

15%

20%

68%***

10%

6%

16%

Low

63%

29%

8%

56%

22%

22%

64%

10%

8%

17%

High School

High

63%***

29%

8%

55%***

26%

19%

50%***

38%

5%

7%

Low

53%

36%

11%

47%

31%

23%

41%

44%

5%

9%

*Note: Some percentages may be higher than 100% based on rounding

*** - Difference is significant at the .0001 level

Similarly, at middle and high schools with higher numbers of minority students, significantly higher percentages of students reported Cognitive, Emotional and Behavioral commitment than their peers who attended schools with fewer minority students.

Table 5: Percentage of Students in Each Category Across Domains by Minority-Majority Student

 

Cognitive

Emotional

Behavior

 

Committed

Compliant

Disengaged

Committed

Compliant

Disengaged

Committed

Compliant

Mixed

Disengaged

Elementary School

High

80%

18%

2%

82%***

9%

9%

70%

16%

7%

7%

Low

80%

18%

3%

76%

12%

12%

67%

16%

8%

8%

Middle School

High

73%***

23%

4%

72%***

12%

16%

70%***

10%

7%

13%

Low

64%

29%

8%

58%

20%

23%

65%

11%

7%

18%

High School

High

68%***

26%

6%

57%***

26%

17%

54%***

38%

4%

4%

Low

55%

34%

11%

49%

29%

22%

44%

42%

5%

9%

Some percentages may be higher than 100% based on rounding

*** - Difference is significant at the .0001 level

In general, students reported diminishing overall engagement across all three domains as they got older. This pattern has been seen in other studies of student engagement, such as a study conducted by Gallup in 2013. While the reasons for this are beyond the scope of this review, AdvancED in future research will continue to investigate and identify best practices contributing to the higher student engagement levels in schools with high minority and free/reduced price lunch eligibility as well as why there is less engagement as student’s progress through their schooling using data gathered from the more than 34,000 schools that are part of the AdvancED Improvement Network.

Keeping students engaged in classrooms is a key component of effective teaching and learning. The more educators know about the level of engagement of their students, the greater educators’ ability to address potential problem areas. The AdvancED Student Engagement Survey, along with other eProve™ tools such as the Effective Learning Environments Observation Tool®(eleot®), the AdvancED Climate/Culture Surveys, Student and Teacher Inventories, and the School Quality Factors Diagnostic, provide needed data to create a roadmap for this crucial part of the improvement journey.

References

Finn, J. D., & Rock, D. A. (1997). Academic success among students at risk for school failure. Journal of Applied Psychology, 82(2), 221-234.

Fredricks, J.A., Blumenfeld, P.C., & Paris, A.H. (2004).  School engagement: Potential of the concept, state of the evidence.  Review of Educational Research, 74(1), 59-109.

Marks, H.M.(2000).  Student engagement in instructional activity: Patterns in the elementary, middle, and high school years. American Educational Research Journal, 37(1), 153-184.

Parsons, S.A., Nuland, L.R., & Parsons, A.W. (2014).  The ABC’s of student engagement. The Phi Delta Kappan, 95(8), 23-27.

Note:
Click here to review the psychometric properties of the AdvancED Student Engagement Survey.

 

Matt Dawson

Matt Dawson, Ph.D., is vice president for Research and Data Analysis in the Research Division at AdvancED. In this position, Dr. Dawson is responsible for directing, designing, managing and implementing the organization’s research and evaluation work, and overseeing all aspects of data analysis conducted within the Research Division. Dr. Dawson has over 15 years of experience leading research projects examining various K-12 education topics. Dr. Dawson has a B.A. from Yale University, a M.S. from the University of Georgia, and received his Ph.D. in Human Development and Family Science from The Ohio State University.

Mike McGill

Mike McGill, Ph.D., is the senior statistician in the Research Division at AdvancED. Dr. McGill is charged with the analysis of both internal and external data using statistical and psychometric methodologies. Along with numerous research projects, Dr. McGill has 10 years of teaching experience in the higher education setting. Dr. McGill holds multiple Bachelor of Science degrees from Kennesaw State University, a Master of Science degree from The University of Georgia, and a Ph.D. from Virginia Tech.