The Evaluation Journey

Every journey begins with the first step. In order to take that step, a destination is determined. The journey’s destination relies on prior travel and the desire for a new direction to explore. A G/T program services evaluation journey looks at both the places traveled and any new destination that leads to student achievement. The journey of gifted services is one that relies on past experiences, current needs, and best practices for the students and educators in the district or campus.

Evaluation Research

First steps for effective program evaluation begin with current best practice that relies on research studies and articles. Program improvement and legal requirements are two perspectives for initiating a program services evaluation suggested by Callahan, Tomlinson, Hunsaker, Bland, and Moon (1995). Whether for program improvement or legal necessity, an evaluation leads to decisions for creating effective services.

Callahan et al. (1995) offered the following guidelines for program evaluation:

• Make evaluation procedures a part of planning from the earliest stages of program development and develop a specific plan for the use of evaluation findings.
• Develop clear program descriptions and goals utilizing multiple data sources (e.g., teachers, parents, students, administrators, school board members).
• Provide adequate funding and time for evaluations while preparing staff for conducting and analyzing the results of the evaluation.
• Clearly identify all audiences who have an interest in or need for evaluation results and involve them in the evaluation process.
• Develop or select assessment tools that address the complex issues of measurement that characterize outcomes of gifted programs.
• Use a variety of data gathering methods designed to reflect the unique structure and goals of programs for gifted learners (e.g., out-of-level testing, portfolio assessment, product rating with demonstrated interrater reliability).
• Disseminate reports to all appropriate audiences in a timely fashion and with recommendations designed to encourage follow-through.

A program service evaluation guide edited by VanTassel-Baska and Feng (2004) defined survey construction and implementation, use of focus groups, assessment of classroom practices, outcome assessment, alignment with the best practice standards, and strategies for synthesizing your findings as essentials for any program services evaluation.

NAGC Pre-K–Grade 12 Gifted Education Programming Standards (Johnsen, 2012) supplied a guide to implementation of high-quality service and should be referenced throughout any approach to evaluation of services. Johnsen’s introduction to this book recommended that this work offers “coherence, structures, guidelines for professional development, and a foundation for advocacy” (p. 3). Chapters 9 (Friedman-Nimz) and 11 (Cotabish & Krisel) presented specific insight into using the NAGC standards for evaluation of services.

Friedman-Nimz’s chapter, “Using the NAGC Gifted Programming Standards to Evaluate Progress and Success: Why and How,” presented programming standards as useful tools in program services evaluation. This author recommended that two types of program evaluation result from the use of program standards: “formative and summative” (Friedman-Nimz, 2012, p. 202). Friedman-Nimz (2012) continued that “formative evaluation could be considered as evaluation for programming, while summative evaluation could be viewed as evaluation of programming” (p. 202). In this light, evaluation is considered to be ongoing in that the evaluation is used to strengthen services and evaluate outcomes of new services.

In their chapter, “Action Plans: Bringing the Gifted Programming Standards to Life,” Cotabish and Krisel (2012) referenced evidence-based standards in order to create an action plan following the completion of the evaluation process. They suggested a roadmap approach to evaluation. Their roadmap consists of a snapshot survey, gap analysis, and an action plan. “Working in concert, the three tools can assist educators in program planning and avoiding roadblocks or a dead end while carrying out an action plan” (Cotabish & Krisel, 2012, p. 233).

These researchers look at best practice for program services evaluation and offer specifics for this effort. Whether the evaluation is for legal purposes or program enhancement, these authors provide maps for the journey.

Approaches to Evaluation

As noted, an evaluation of G/T services takes a variety of forms. Fetterman (1993) suggested two approaches: (a) a self-study using district personnel or (b) employment of a third party who assesses program services with fresh eyes and without prior knowledge of services selected for evaluation. Once the choice of approach is determined, the district selects the method of evaluation that ensures improvement of district/campus G/T services for student achievement.

If self-evaluation is selected, NAGC’s (2017) Administrator Toolbox offered a checklist of program elements to facilitate the evaluation under the following topics:

• program design,
• identification,
• curriculum and instruction,
• affective needs,
• professional development, and
• program evaluation.

Each item in the 41-item checklist supplies space to comment based on “No Evidence, Some Evidence, or In Place” (NAGC, 2017, pp. 1–4). The comments section is valuable in determining where to place emphasis for a plan of action following the evaluation. This checklist also may be used with third party evaluation depending on the method that best fits district/campus needs.

Methods of Evaluation

The chosen method of evaluation steers toward improvement of services, including appropriate identification of students and their academic growth. The following are five considerations for methods of evaluation:

• A 5-Year Plan that evaluates one area of services at a time. In this case, each year the district selects one of the areas in the Texas State Plan for the Education of the Gifted/Talented (Texas Education Agency [TEA], 2009) as a focus.
• Student Outcomes determine another method aimed at strengthening of services for students. This approach makes use of all available data for determining current student outcomes. With this method, one grade span at a time is evaluated; one year for each span: elementary, middle school/junior high, high school. The order of evaluation depends on the area of greatest need as determined by self-evaluation or formal needs assessment conducted in-house or by a third party.
• District/Campus Improvement Plans (DIP/CIP) dictate another form of evaluation. Emphasis on area(s) included in the DIP/CIP guide the evaluation. (See TEA, 2009, 3.4C.)
• An evaluation that compares budget expenditures to Student Outcomes is a newer approach. This process considers how effective expenditures are in creating optimal learning experiences for gifted students. An evaluation with this focus can be combined with Student Outcomes or alignment with DIP/CIP. Audit of resources and Student Outcomes creates insight into effectiveness of services.
• Focus on Special Populations provides an opportunity to determine appropriate identification and service options for twice-exceptional students.

Any of the approaches is informed by document review, interviews, focus groups, and classroom-observation techniques to arrive at recommendations and an action plan (Gubbins, 1998). Each of the five methods are designed to ensure students are identified and served in ways that match the district/campus program design to students’ learning needs and to emphasize the need to incorporate ongoing evaluation of gifted services. The development of an evaluation plan is critical to achieve positive outcomes for gifted learners.

5-Year Plan

A 5-Year Plan is one that evaluates one area of services at a time. The district selects one of the areas in the Texas State Plan (TEA, 2009) as a focus. The five areas are Student Assessment, Service Design, Curriculum and Instruction, Professional Development, and Family/Community Involvement. After a needs assessment is conducted, a timeline for the order of evaluation is established. This method allows districts an in-depth effort into the area of most need.

Both the Texas Association for Gifted and Talented (TAGT, n.d.) and the Texas Education Agency and the Commissioner’s Advisory Council on Gifted Education (2013) suggested a form that addresses each of the five areas found in the State Plan. Each document is found on the TAGT and TEA websites. The documents, “Guiding Questions for Program Review,” lead discussion with verification questions to implement program evaluation in each of the five areas.

In determining a timeline for the five areas, consideration is given to alignment of the area to be evaluated with the other four. An example of timeline alignment is that while the natural progression from Student Assessment is Program Design, the district may find that Professional Development should follow the next year to ensure that all staff and teachers understand the characteristics of gifted learners from all populations. Or, the district may recognize the need for Family/Community Involvement for all populations to understand who are the gifted and the community’s role in finding all gifted students.

The 5-Year Plan approach applies to those districts who want to strengthen their services and do not have a pressing need that encompasses looking at all five areas simultaneously. As with any method of evaluation of services, the goal is student achievement.

Student Outcomes

Student Outcomes evaluations are similar to the 5-Year Plan in that this approach looks first at the greatest area of need to be developed. A formal needs assessment and/or reviews of data and expectations for each grade span determine the area of greatest need. For some, beginning with the end in mind determines that grades 9–12 are evaluated and addressed. For others, the enigma of middle school drives the evaluation. Many primary grade G/T services are well in place and evaluated last unless identification of special populations or new research necessitates an in-depth look at novel approaches to identification or curriculum in K–5. In some cases, Kindergarten is a stand-alone evaluation. Current research along with Pre-K implementation have shed new light on the needs of the young gifted learner.

Johnsen (2010) stated, “The number of practices used or how the schools use practices is not as important as whether or not the practice is effective with students” (p. 5). An evaluation that only seeks to check the boxes of completion will not yield information about improved student achievement. Callahan and Caldwell (1997) stated that all of the elements of good program services, identification, curriculum and instruction, and must be connected to outcomes. Friedman-Nimz (2012), in discussion about the use of standards for program services evaluation, specifically student outcomes, said, “Use of standards and evaluation strengthens the distinct identity of the field and provides a setting for continuing important conversations about providing comprehensive programming that promotes rather than suppresses extraordinary performance” (p. 212).

Benchmarks employed on a continuing basis facilitate evaluating outcomes. Benchmark data and all other available data resources that document student growth informs the area of greatest need. Questions that guide this method include the following:

• What measures of student achievement are available?
• Growth measures are determined by __? (Source)
• In what ways will achievement be documented?
• What areas of student growth will be measured? Content areas only? Affective measures?
• By what means might affective needs be measured?

A district may combine the Student Outcomes approach with their 5-year plan. For example, if the district looks at Kindergarten, only their student assessment for identification is evaluated before looking at primary grades’ student assessment as a whole. The Student Outcomes method also may be informed by the District and Campus Improvement (DIP/CIP) plans. If student achievement is stated in the DIP/CIP, it is combined with Student Outcomes.

District and Campus Improvement Plans

The Texas State Plan stated, “Provisions to improve services to gifted/talented students are included in district and campus improvement plans” (TEA, 2009, 3.4C [TEC §§11.251-11.253]). This method of evaluation takes into account the area(s) of improvement based on District and Campus Goals. All Strategies/Activities, Resources, Persons Responsible, Timelines, and Formative Evaluation are taken into account.
The DIP/CIP may also be combined with one of the other four methods for program evaluation. After looking at the DIP/CIP, the 5-Year Plan, Student Outcomes, or Special Populations may be selected for evaluation. For example, if Professional Development for all teachers is described in the DIP, the 5-Year Plan starts with Professional Development.

Budget and Student Outcomes

An evaluation that compares monies spent to Student Outcomes adds meaningful insight into both district values and expectations for gifted students and their educators. This approach considers how effectively expenditures create optimal learning experiences for gifted students. Data from Student Outcomes views student achievement alongside an audit of the program services budget. Questions that guide this method include these:

• Do expenditures match requirements in the State Plan? (See Service Design, 2.5C)
• Are expenditures prorated among all campuses? If not, why not? If so, describe.
• What percentage of the budget is from state funding? From district funding?
• What percentage of the budget partially or totally pays teacher salaries?
• What percentage of the budget partially or totally pays administrator salaries?
• What measures are used to determine the relationship of expenditures to student achievement?
• What impact do expenditures have on student achievement?

The purpose of this evaluation method is to make best use of local and state funding for achieving positive student outcomes. Audit of resources and Student Outcomes creates insight into current services. How monies are used and the accountability of their use is yet another form of evaluation whose goal is to enhance student achievement.

Special Populations

Identification of gifted students from special populations, including twice-exceptional (2E) students, is a current topic of importance to many districts. In addition to identification, appropriate services that fill gaps and/or meet specific learning needs of students from the nondominant culture, from a home language different from English, or with learning differences, such as ADHD or dyslexia, may be lacking or nonexistent.

Evaluation related to special populations requires an in-depth look into demographic data, special education documents, including ARD information, district or campus 504 information, ELL/ESL/bilingual student lists and any other documentation that reveals the need to adjust identification methods and appropriate services for twice-exceptional populations. For example, a review of who refers students for testing and the number of referrals of twice-exceptional students may reveal variations in lower numbers of 2E students being referred. Or, a data-dig of the demographic data of the campus or district compared to the demographic data of 2E students receiving services may indicate discrepancies to be considered.

When differences are found, the program services evaluation looks at means of identification, service design, and/or curriculum for identified students as each relates to district or campus 2E students. In the case of identification, if only verbal tests are used to identify all populations, non-English speakers will be at a disadvantage. If the service design is differentiation in the regular classroom and no efforts toward filling learning gaps or addressing learning needs for 2E students are used, those who are identified may exit services or be considered misidentified. Information that reveals a lack of instructional scaffolding to meet specific learning needs offers insight into reasons for inconsistency in special populations being identified and served.

Whether this approach is used or combined with any of the other four, it is important to consider twice-exceptional learners and include it in any effort toward increasing student achievement.

Another Consideration

Size of the district creates differing needs for evaluation. Any of the methods described above may be utilized, but the size of the district influences the evaluation. For example, if the district is considered large and urban, the Student Outcomes method of evaluation is more applicable for feeder patterns. If identification is selected with a focus on special populations’ access to services, the evaluation will look at the district as a whole.

Once the decision is made, Robinson, Cotabish, Wood, and Biggers (2009) provided an evaluation template for large, midsize, and small districts. Each evaluation template offers questions to guide the evaluation and space to list data sources, instrumentation, process for data collection, involved persons, timeline, and strategies. Questions include the following for each school size:

1. To what extent are the stated mission and goals of the gifted program fulfilled in their actual operation?
2. To what extent is the gifted program meeting the needs of identified students as perceived by relevant groups?
3. What evidence exists to document positive student performance trends for students participating in the gifted program?
4. What are the program strengths and weaknesses in relation to the state of the art or best practices in gifted education?
5. What are the recommendations for program improvement or revision? (Robinson et al., p. 413–432)

Even though the questions are the same for all-sized districts, the expectations for personnel and course offerings differ among each category.

Summary

The first step in the evaluation journey is the most important. Best practice found in research is imperative when designing evaluation of program services. All sources cited in this article offer valuable information when determining the process and procedures for conducting an evaluation.

The evaluation map leads to selection of methods for evaluation. The five described in this article originate from the author’s research findings of best practice, district experience from small, rural to large, urban districts, and experience conducting programs service evaluations in small, midsize, and large districts. There may be more than five ways to approach the evaluation journey. As long as the destination of the evaluation is for improved student achievement for all gifted students, the evaluation method that matches specific district or campus circumstances yields improved student services and student achievement.

References

Callahan, C. M., & Caldwell, M. S. (1997). A practitioner’s guide to evaluating programs for the gifted. Washington, DC: National Association for Gifted Children.

Callahan, C. M., Tomlinson, C. A., Hunsaker, S. L., Bland, L. C., & Moon, T. (1995). Instruments and evaluation designs used in gifted programs. Storrs: University of Connecticut, The National Research Center on the Gifted and Talented.

Cotabish, A., & Krisel, S. (2012). Action plans: Bringing the Gifted Programming Standards to life. In S. K. Johnsen (Ed.), NAGC Pre-K–Grade 12 gifted education programming standards: A guide to planning and implementing high-quality services (pp. 199–213). Waco, TX: Prufrock Press.

Fetterman, D. M. (1993). Evaluate yourself. Evaluation: Research-based decision-making series. Storrs: University of Connecticut, The National Research Center on the Gifted and Talented.

Friedman-Nimz, R. C. (2012). Using the NAGC Gifted Programming Standards to evaluate progress and success: Why and how. In S. K. Johnsen (Ed.), NAGC Pre-K–Grade 12 gifted education programming standards: A guide to planning and implementing high-quality services (pp. 231–253). Waco, TX: Prufrock Press.

Gubbins, E. J. (1998). NRC/GT’s suggestions: Evaluating your programs and services. Storrs: University of Connecticut, The National Research Center on the Gifted and Talented.

Johnsen, S. K. (2010). Principles underlying the 2010 programming standards in gifted education. Gifted Child Today, 34(2), 5, 65.

Johnsen, S. K. (Ed.). (2012). NAGC Pre-K–Grade 12 gifted education programming standards: A guide to planning and implementing high-quality services. Waco, TX: Prufrock Press.

National Association for Gifted Children. (2017). Master checklist of gifted program elements for self-assessment. Retrieved from http://www.nagc.org/sites/default/files/administrators/GT%20Program%20Master%20Checklist.pdf

Robinson, A., Cotabish, A., Wood, B. K. & Biggers, A. (2009). The Arkansas evaluation initiative in gifted education. In J. S. Renzulli, E. J. Gubbins, K. S. McMillen, R. D. Eckert, & C. A. Little (Eds.), Systems and models for developing programs for the gifted & talented (2nd ed., pp. 413–432). Waco, TX: Prufrock Press.

Texas Association for Gifted and Talented. (n.d.). Resources: Program evaluation. Retrieved from http://www.txgifted.org/program-evaluation

Texas Education Agency. (2009). Texas state plan for the education of the gifted/talented. Retrieved from https://tea.texas.gov/Academics/Special_Student_Populations/Gifted_and_Talented_Education/Gifted_Talented_Education

Texas Education Agency, & Commissioner’s Advisory Council on Gifted Education. (2013). Guiding questions for program review. Retrieved from https://tea.texas.gov/Academics/Special_Student_Populations/Gifted_and_Talented_Education/Gifted_Talented_Education

VanTassel-Baska, J., & Feng. A. X. (Eds.). (2004). Designing and utilizing evaluation for gifted program improvement. Waco, TX: Prufrock Press.

Cecelia Boswell, Ed.D., is an educator, consultant, and author. She taught migrant and gifted students, served as Advanced Academics Consultant Region 14 ESC, and was Executive Director of Advanced Academics for Waco ISD where she created a middle school gifted academy. She has consulted with the Texas and Florida Departments of Education and led research projects for International Baccalaureate. Cecelia has served as President of the Texas Association for the Gifted and Talented and as President of The Association for the Gifted. She was named the Texas State Gifted Administrator of the Year. Cecelia is the coauthor of five books about gifted education.