Progress and Promise: Shaping Long-Term Social Change Through Evaluation of Professional Development Beverly A. Parsons InSites, Boulder, CO November 1998   Dr. Parsons is Executive Director of InSites, 1460 Quince Avenue, Suite 101, Boulder, CO 80304-1157, 303-449-1190. InSites is a Colorado-based non-profit 501(c)3 organization that conducts research and evaluation and provides technical assistance to educational and social institutions/agencies and policymakers engaged in major change within their social systems. This paper was presented at the American Evaluation Association 1998 annual meeting, Chicago, IL. Ñ InSites, Boulder, CO, 1998. Progress and Promise: Shaping Long-Term Social Change Through Evaluation of Professional Development Professional development for teachers currently is viewed as a critical component for major renewal and redesign of the education system to better meet the needs of today's society. The National Education Association (NEA) and the American Federation of Teachers (AFT) are both engaged in major changes to emphasize professionalism among teachers rather than strictly the rights and welfare of teachers. NEA's foundation, the National Foundation for the Improvement of Education (NFIE), published a major report in 1996 entitled Teachers Take Charge of Their Learning: Transforming Professional Development for Student Success that made recommendations about teachers' professional development. The recommendations address finding time to build professional development into the life of schools, helping teachers to assume responsibility for their own professional development (with special emphasis on the use of technology), finding common ground by working with the community to provide high-quality professional development, and finding the revenues to support high-quality professional development. NFIE's ultimate goal is to see that every teacher in the US regularly engages in high-quality professional development that improves student achievement. In 1997, NFIE began a grantmaking initiative entitled A Change of Course (ACOC). This effort is designed to provide examples of how recommendations made in NFIE's 1996 report are put into practice with an eye toward how to bring these reforms to scale. In Spring 1997, the Foundation provided planning grants to 17 sites to design implementation of one or more of the recommendations in the report. In Fall 1997, NFIE awarded three-year implementation grants to ten sites. Nine of the grantees were primarily joint efforts between a local teachers' associations and its district with occasional other partners such as a business group. One was a state level collaboration that included both the state NEA and AFT affiliates, the state department of education, and a local school district and its teachers' association. The ACOC initiative includes an external cross-site evaluation component being conducted by InSites. During the first year of implementation, in-depth case studies were conducted in four sites. The studies focused on historical patterns that appear to impact professional development. Less intensive site visits were conducted in the other sites, along with document reviews and phone interviews. The data collection in the case study sites included extensive interviews, site visits, and review of documents. Comprehensive descriptions were written for each of these sites for use by NFIE. The cross-site evaluation of ACOC, as well as ACOC itself, has a different purpose and perspective than is often the case for grant-making initiatives. The emphasis is not on whether sites can demonstrate improved learning for a specific group of teachers and/or students but whether they have put in place the long-term support structures (e.g., policies, communications) that will lead to high-quality professional development sustained over time for all teachers. The cross-site evaluation is complemented by evaluations being conducted by each site. These internal evaluations often look specifically at improved learning of teachers and students that derives from the site's work to put in place one or more recommendations in the NFIE report. The cross-site evaluation is designed to serve three intertwined purposes: help improve the ACOC initiative; determine the impact of the initiative; and generate new knowledge about large-scale education system change focused on professional development. A key feature of the evaluation is the use of a theory of change that focuses on how systems are changed in a sustained fashion. The evaluation builds on a theory of change and uses an Inquiry Team composed of selected NFIE staff and board members, representatives from the National Education Association (NEA), evaluators, and others as determined by NFIE. The Inquiry Team both provides for ongoing flow of information between the evaluation team and the intended users of the evaluation information and serves as a means of enriching the interpretation of the data. Two key questions serve as the basis for the evaluation design: 1.What policies, practices, and conditions foster high-quality, sustained professional development for all teachers? 2.What are the implications of the ACOC work for how NFIE can increase its effectiveness in leveraging large-scale education improvement? The purpose of this paper is to present an adaptable theory of change that focuses on long term sustained professional development practices. The paper will also illustrate the process of using and revising a theory of change to continually strengthen the evaluation. Initial Theory of Change Guiding the Evaluation The ACOC evaluation is designed around an evolving theory of change that NFIE is using to guide its work with the sites. Neither making nor tracking changes in a highly complex system is a simple task. Change can occur in a multiplicity of ways. Many aspects of the change process are unpredictable and specific to one group of individuals. Yet, buried within the chaos of change are some roughly definable patterns. It is around these patterns that NFIE and the evaluation team have attempted to develop a modest theory of change that serves (1) as a communication vehicle for working with the sites and one another and (2) as the basis for interpreting evaluation data. As someone has said, "There is nothing more practical than a good theory." A key part of the evaluation approach is to have a theory of change that can be revised as the work proceeds. The theory of change is important in bringing to consciousness how NFIE perceives the change process initially. The yearly updates help to also bring to consciousness the learning that is occurring as the work proceeds. NFIE's initial statement of its theory of change was established during the planning phase for the grantees (1997). It was based on initial contacts with the sites. It had several features: Philosophical shift. It specifies a few fundamental philosophically based features of the education system that the ACOC initiative is attempting to shift. The philosophical shifts ACOC is attempting to bring about are related to teachers' roles; community involvement; time and money allocations; the link between teachers' professional development and student learning; the impact of diversity and complexity on education systems; and the designation of who determines teachers' professional development. Support structures. The theory recognizes that certain system structuresñvisible and obscure, formal or informalñhighly influence the extent to which systems accomplish their intended purposes. (These are discussed in more detail below.) Goals. It expects goals to be established for how change agents are seeking to transform the education system on a long-term basis. It articulates how these system changes contribute to the specific goals set for the teachers and students who are to benefit. For example, the goal may be to change the education system's policies to require more content-specific professional development for teacher licensure. This in turn is expected to contribute to better student learning in the content area (e.g., mathematics). Phases of change. The theory identifies four roughly defined phases of change that systems tend to move through as they shift from one type of system to another. The phases are identified as Incubation (ideas formulated), Exploration (exploration of ideas on a small scale in a variety of settings), Transition (major commitments to new policies and practices), and Broad-Based Application (new norms, expectations, and structures in place). Insiders and outsiders. It distinguishes those directly involved in the system being changed from those outside of the system. It describes the roles of both groups and the impacts of their roles. Evolving theory. NFIE views the theory of change as continually evolving and being informed by work in the sites. Initial Support Structures for Professional Development for All Teachers As noted above, one key part of the theory of change is the support structures that define the education system. Although there are many ways to categorize the structures, the evaluation work in the planning year (1997) led us to organize them around a working list of nine structures. These structures, not listed in order of importance, are described as follows: Context/Climate. The technological, demographic, economic, and social context may heavily influence the extent to which other support structures are effective. Closely tied to these contextual factors are long-standing attitudes and norms that shape the climate of the education system. Collaborations. Many types of existing and potential collaborative efforts may impact teacher professional development. They may be short- or long-term and vary widely in their degree of formality or informality. They may range from formal partnerships to informal or temporary collaborations. The formal partnerships include university-school, education-business, community-school partnerships that are designed to support teachers' professional development. Informal collaborations tend to be ones formed to accomplish specific tasks. Communications. The content and processes of communications constitute powerful system structures that affect the extent to which all teachers are reached and whether professional development is sustained. We consider the "who, what, why, when, how, and where" of communications. Leadership, Politics, and Power. Moving the education system toward support of professional development for all teachers will require significant commitments and leadership from the full range of stakeholders. These leaders include key opinion leaders, official leaders, the teacher workforce, and other stakeholders. Their understanding of power and politics, their political connections, and how these leaders interact and engender support among those they represent are important features. Initiatives for Systemic Reform. A variety of state, federal, and privately funded initiatives for reforming the full education system exist. It is important to understand how they impact teachers' professional development and the support they have from key groups that are affected by ACOC. Philosophy. Organizations have stated or unstated philosophies that are key to creating a situation that either supports, is neutral to, or undermines the kind of teacher professional development called for in the NFIE report. Policy. We use "policy" to refer to actions and written procedures that have been officially adopted by organizational entities. These entities include schools; businesses; state legislatures; city governments; private foundations; nonprofit organizations; official coalitions; district school boards; parent-teachers' associations; county boards of commissioners; state boards of education; and national, state, and local teachers' associations. State and district policies about standards, accountability, finance, and partnerships are types of state and district policies that are likely to be the most important for ACOC. The degree of policy flexibility, the distribution of power, the ways policy is circumvented, the implications of how money is actually spent, and other features of policy such as negotiated agreements between associations and governing bodies are important to understand. Procedures/Guidelines. This category includes written procedures or guidelines, developed by committees, tasks forces, or staffs of governing bodies, that have not been officially adopted as policy but are influential in shaping actions. The category also includes official regulations developed around official policies. We are particularly interested in those guidelines and procedures that are related to professional development and to mechanisms for the policy development process. Plans and Planning Processes. Five types of plans and planning processes appear to be especially relevant to the ACOC: induction; work-technology; school improvement; strategic plans; and professional development and evaluation. Findings: Themes That Impact Professional Development In looking across all ten sites, the evaluation found cross-cutting themes both within clusters of sites with a common focus and themes across all ten sites. To illustrate how the theory of change shaped the articulation of the themes and how the theory was revised, we present here the seven themes from the full cross-site analysis. Mentoring A strong pattern that emerged was the use of mentoring as a key strategy for peer assistance and review, induction, and technology support. Seven of the ten sites initiated some form of mentoring through their grant work. At several of these sites the use of mentors demonstrated a major shift in action and thinking toward more collegial work. ACOC demonstrated the importance of having designated mentors rather than counting on an informal mentoring process for peer assistance to learn new content and instructional methods. We also found that recognition, compensation, and extra training for mentors are key motivating forces for participation and excellence and can help to elevate the status and value of professional roles for teachers that are outside of the classroom. As envisioned in NFIE's Teachers Take Charge of Their Learning report, teachers are excited and challenged by new roles and grow considerably with their new responsibilities. Mentoring appears to be a strategy for change that may appeal to the large middle-group of teachers to improve their practice. Mentoring may prove to be an important entry point for initiating more sustainable, self-directed, professional development. The sites with mentoring programs have found it useful (and of financial necessity) to start with a small group of mentors and mentees who volunteer to test and refine the program at its pilot stage with the idea of expanding in the future. Challenges associated with mentoring are: funding the expansion of mentoring programs to reach all teachers, offering incentives to master teachers to continue as classroom teachers, and encouraging other new roles for teachers recommended in NFIE's report. The Role of State and Local Policy Across the ACOC sites, state policy is playing a powerful role in surfacing new ways of thinking about professional development. Local policy is having a substantial influence as well. In some cases, policy initiates a philosophical shift that results in change and/or provides a framework for local change. In other situations, policy is formulated only after state or local action takes place. Then it serves to institutionalize new ways of working. In several cases, policy and action combine in an iterative process that moves sites toward quality professional development for all teachers. At the majority of the ACOC sites, grantees are responding to various policies creatively and viewing them as opportunities to explore new ways of teaching and learning. State policy serves an important incentive for professional development when it supports strategic planning at the district level, supports teacher-determined professional development, and links licensure and professional development to standards-based curriculum. The types of state-level policies that are of most support to professional development at the sites are: teacher licensure and recertification, student achievement standards and teacher performance standards, professional development planning, and strategic planning and accountability policy. At the local level, teacher appraisal and supervision policies weld a strong influence at many of the ACOC sites, especially those that call for individual professional development plans, collaborative action, and peer assistance and review.   Strategic Planning Strategic planning at the state and local level has emerged as an important theme at some of the ACOC sites. Strategic planning is helping people to think in new ways and talk across old boundaries (e.g. teachers, administrators, and community members). Combining strategic planning and professional development appears to be an effective way to move toward high-quality, sustained professional development for all teachers because it gets people to think creatively together and moves discussions about professional development to a deeper level. At many of the sites, shared decision making between teachers, administrators, and the community is critical in the design of school-based and district-sponsored professional development programs. Where strategic planning is a collaborative venture the results are more likely to be accepted by a greater number of people. Collaboration That Supports Professional Development Through strategic planning and site-based decision making, many of the ACOC grantees have already incorporated collaboration into their mode of operation. At other ACOC sites a shift in philosophy is occurring, though tentative at some sites. It is moving them away from a hierarchical model toward more collaborative modes of operation. The sites are forging new partnerships through ACOC and understanding the reasons that make collaboration attractive and a necessity where resources are limited. These key ingredients make for viable collaborative partnerships: strong leadership, good communication, trust and respect, inclusion of new perspectives, balance between goals and flexibility, proof of results, and time to reflect and celebrate accomplishments. One key type of structure emerging across the sites to support collaboration is what we will term "boundary-crossing groups." These are formal groups convened for defined tasks that include people from many roles representing many different groups. They enrich the dialogue around professional development and move the conversation into the philosophical as well as practical realm. At three of the sites, formal policies lay the foundation for a collaborative relationship between the district and the local association and define expectations and procedures for their work together on all professional issues, including professional development. Some challenges to collaboration are: involving a range of stakeholders, finding adequate time for collaboration, building trust, working within hierarchical structures, fending off divisive politics, imposing a vision, bridging the isolation, and focusing on positive action instead of problems. Communication Both collaboration and communication are pursuits that involve building trust and strong relationships. This important interpersonal work can fortify ACOC efforts to bring high quality professional development to all teachers, but it takes time to develop effective and resilient systems to support this challenging work and reach out to all interested stakeholders. Grantees have had mixed success with communication this year. At the four ACOC technology sites people are starting to use electronic means for internal communication and other sites are making some use of technology as well. Public misunderstandings and counterproductive assumptions and beliefs about professional development still abound. To build a strong public base of support, the ACOC grantees and educators generally need to communicate to the public about their grant work and the essential nature and value of high-quality professional development. Leadership, Power, and Politics Leadership and the distribution of power are factors that are likely to affect the ACOC sites' strategy for change. Leadership exercised at the school, district, and state levels intertwines with the other cross-cutting themes impacting professional development. Among various leaders who are most influential at the sites in sustaining professional development, first and foremost are the local leaders, especially district and teachers' association leaders. Superintendents of school districts and other district administrators have welded considerable power in shaping the professional development focus, historically and now. Over time there appears to be a general shifting from primarily top-down to more collaborative decision making, but this varies considerably from site to site. When district leadership changes, it can have a disruptive impact on power relationships and planning processes. Role of Teachers' Associations and Their Leaders The role of the teachers' association appears to be important in at least three key ways. First, the association can focus attention on professional quality by taking an active role in professional issues as well as in salary-related issues. Teachers' associations nation-wide are feeling the impact of NEA's philosophical shift away from an adversarial position to a more collaborative one and its strong focus on professional quality. In light of this, both state and local associations are having to reevaluate their role and commitments, and their relationship with district administrations. The ACOC initiative supports and encourages district and association partnerships around professional issues to support a shift in thinking about collaborative action. Second, the association can be a vehicle to engage many teachers in developing a commitment to professional development by involving them in the policy making process. A trend that is emerging across the ACOC sites is that teachers want to take responsibility for directing their own professional growth in an environment of support and collaboration. ACOC and the unions are providing opportunities for mentoring and participation on committees. And third, the teachers' association can also be the vehicle to put in place policies and procedures that will ensure that professional development methods are maintained even when district leadership changes. Even in cases where professional development is primarily district-operated and is very satisfactory and where teachers are involved in decision making, it is vitally important that structures and processes that support quality professional development are written into local policy. A Revised Theory of Change After determining the themes above, we then looked at what the implementation work in the sites suggested about revisions in the theory of change. Three of the above areas were refined using the knowledge gained during the first year of implementation. Those areas are: support structures phases of change insiders and outsiders Here is how we refined these three aspects of the theory. Revised Support Structures for Professional Development for All Teachers As we looked at the findings of the evaluation over the planning period and first year of implementation, we found it useful to modify the list of support structures. Basically, the nine structures given above were collapsed into six and two new ones are added. These categories seemed more in tune with the findings and yet still are broad enough to capture new patterns that may arise. We also are considering changing the terminology from "support structures" to "system forces." Here is the revised list of factors. Context/Climate. The study of historical patterns showed that technological, demographic, economic, and social factors are heavily influencing the extent to which other support structures are effective. The influx of technology, the shifting economic conditions, and the increasingly diverse student population are especially important. Closely tied to these contextual factors are long-standing attitudes, values, philosophies and norms that shape the climate of the education system. (The earlier Philosophy category has been incorporated here.) Collaborations. Collaborative relationships are surfacing as especially important in bringing about change. These collaborations are partnerships at the organizational level (e.g., between districts, teachers' associations, and universities) as well as collaborative actions at the individual level (e.g., between the district superintendent and the association president). Teams and networking also are forms of collaboration that support professional development. Teams tend to be within single organizations, while networks cross organizational boundaries. Communications. Methods of engaging teachers in decision making (e.g., regularly scheduled surveys, committee participation) are proving to be important aspects of communication. Electronic networks are becoming increasingly important. Leadership, Politics and Power. Issues of leadership, politics, and power regularly surface as important features of the change process. It is likely that these will become even more important as sites move to embed new policies and practices (often affecting resource allocations) in their education system. Teachers' Association Roles. We began by including the teachers' association role in the leadership category. The role of the teachers' association at both the local and state levels may well be framed in ways other than leadership. The specific roles they play or choose not to play can better be tracked if treated as a separate category. Further delineation of the actions at both local and state levels is needed. Policy. Both state and local policies are surfacing as important factors. Strategic planning requirements, financial policies, licensure requirements, student and teacher standards, and performance evaluation policies are especially important. The degree of policy flexibility, the distribution of power, the ways policy is circumvented, the implications of how money is actually spent, and other features of policy such as negotiated agreements between associations and governing bodies are all features of policy that may be significant. (The previously separate categories of Initiatives for Systemic Reform and Procedures/Guidelines has been subsumed in this category.) Plans and Planning Processes. Six types of plans and planning processes appear to be relevant to the ACOC: induction; technology; school improvement; strategic plans; professional development; and evaluation. Currently, we are finding that strategic planning processes are especially important and often incorporate the other types of plans mentioned above. Learning Methods. Mentoring is emerging in the sites as a major new mode of professional development. Its presence suggests that we need to track the learning processes that are used for professional development. Others, such as computer chat rooms, may emerge over time. We also expect to see a shift away from one-shot workshops. Mentoring may prove to be an especially important change strategy for all sites. Enhancements of the Phases of Change We also found that we could refine the theory about the phases of change based on the initial evaluation work. Initial Description of Phases of Change The initial theory of change identified four roughly defined phases of change that systems tend to move through as they shift from one type of system to another. The phases are identified as Incubation (ideas being formulated), Exploration (exploration of ideas on a small scale in a variety of settings), Transition (major commitments to new policies and practices), and Broad-Based Application (new norms, expectations, and structures in place). The sites involved in the NFIE work tend to be in the Transition phase. This is a difficult period. It is in the Transition stage that initiatives coalesce and new structures are put in place that can begin to define the new connections. Problems inevitably occur when people make the switch to the new system. Typically, they will hang on to some aspects of the old system until they are comfortable with the new ways of doing things. Those who succeed in moving through this phase realize they don't have the resources to do both and that they need to make a choice between the old and the new. This is a tricky process that involves recognizing the point when one has to give up the old way and cling to the new. It involves figuring out how to balance what worked in the past against what is needed in the new context (rather than throwing everything out from the past or trying to keep all of both old and new). Also, it involves figuring out how to allocate resources to support the change. These tough decisions need to be based on a deliberate commitment to the new underlying assumptions that will anchor the system, for example, a commitment to high-quality sustained professional development for all teachers and to strategic planning. During the Transition stage, some funding is still typically needed from outside sources. However, increasingly large amounts need to be reallocated from the existing formal and informal systems. The Transition stage is a fragile one. It tends to need considerable outside support, both morally and financially. All too often, external funders and supporters pull out before people have made it through the Transition phase, leaving new practices and commitments too fragile to weather the assaults of those still holding on to their old power positions and perspectives.   Refinements of the Phases of Change The case study work was designed in part to understand more about the Transition phase as it relates to how professional development is established and sustained for all teachers. As we reviewed the literature on change, a theory that seemed especially appropriate to enrich NFIE's current understanding of the Transition phase is one that looks at how organizations deal with radical change. Indeed the changes toward greater teacher professionalism constitute a radical shift for the traditional education system. Our analysis of the data kept bringing us back to three radical externally-determined changes affecting these sites: (1) the influx of technology into nearly all aspects of society, (2) the shifts in economic conditions across the states in which the sites are located, and (3) the changing demographics which bring in students with different values, experiences, and home support for education. These changes are deep, fundamental social conditions that education systems cannot control. Rather educators need to know how to respond to these conditions in ways that allow them to accomplish their mission of high-quality education for their students. These conditions are like deep currents in a river. The ship cannot control the currents. But the captain of the ship can recognize the currents, know how to deal with them, and bring the ship to its destination. Enriching NFIE's theory of change in the Transition phase involves considering two types of radical change: sudden, radical change and gradual, radical change. Sudden radical change leads to "breakpoints," noticeable breaks in how organizations operate. Breakpoints are relatively sharp and sudden. "Turning points" refer to radical change that happens gradually. In either case, the fundamental change itself is beyond the control of the organization. However, the way people respond to it can determine whether the fundamental change will be a breakpoint or a turning point for them and what the consequences of either might be. Their response can determine whether that breakpoint or turning point becomes a positive experience for the organization. An effective organization is one that consciously decides when and how to anticipate, exploit, or create breakpoints; it decides how to shift an unhealthy breakpoint into a turning point or vice versa. Change Arena Figure 1 depicts an arena of change. The arena is defined by the forces for change and the forces of resistance to change. When resistance forces are strong and change forces are weak, there tends to be no change. When both change and resistance forces are relatively weak, change tends to be sporadic and leads to gradual turning points. If the change forces are more intense but the resistance forces are still weak, a pattern of continuous change is likely to ensue. When both the change forces and the resistance forces are strong, discontinuous change or breakpoints occur. Across the arena, when the forces of resistance are dominant, the status quo tends to win out. If the forces of change are stronger than the resistance forces, the change agents tend to dominate. Figure 1óChange Arena The situations are continually changing and unexpected forces can overwhelm those that seem important at a given point in time. The point here is not so much to determine exactly where a site is located as to provide a tool for a collective group of insiders and outsiders to look at the forces of change and resistance in a given site as a basis for developing their strategy to move toward sustained high-quality professional development for all teachers. The Transition phase is a dynamic time of forces of resistance and change moving the system back and forth. It will be important to understand the shifts in these forces and what new balances among the forces determine the sustainability of teachers' professional development. Strebel (1992), in his characterization of the four areas of the change arena, describes four intervention approaches that leaders use depending upon where their organization is in the change arena. (Refer back to Figure 2.) When leaders decide that the change forces are strong but declining and resistors are closed to change (the "no change" quadrant), the intervention strategy tends to be one of resisting making change in the organization as a whole. If leaders want innovation, they tend to do it through a separate group that is protected from the forces of the status quo. Outsiders are likely to find it futile to spend time encouraging change in the organization as a whole. If the organization tends to be in the quadrant of "sporadic change," where change forces are strong but declining and resistance is low, the strategy tends to be one of incremental targeted change. Partnerships and alliances specific to the targeted change are important. They are working on a targeted change represented by the particular recommendation that they have selected to implement. They have formed partnerships around this target. Whether or not they move toward continuous change, i.e., go beyond the particular target recommendation to other aspects of creating sustained high-quality professional development for all teachers, will depend on whether the forces of change increase. If the organization tends to be in the "continuous change" quadrant, where change forces are strong and growing and resistors are open to change, an ongoing revitalization strategy is more possible. It involves long-term investment in organizational learning with slow continuous adaptation to changing conditions. Here there is more integration of functions across the organization. In the case of the ACOC sites, those that have long-standing, solid collaborative relationships between the district and the teachers' association may well be in this quadrant. They may be able to more effectively leverage professional development along with strategic planning and student and teacher standards. They may be especially well-positioned to do the most creative work in the reallocation of resources and the establishment of policy that will foster continuous well-paced change. If an organization is in the "discontinuous change" quadrant, where change forces are strong and growing and resistors are closed to change, significant restructuring of the organization is likely to be the mechanism for handling the situation. In education, this is where we are likely to see state takeovers, major pushes for restructuring such as charter schools, and other options. It may be very difficult to use a professional development strategy as the major force for change in these situationsñat least until the crisis has resolved itself and the situation has moved into one of the other three quadrants. When looking at the support structures mentioned above, we may find that specific features in a given site may be defined as forces for change or resistance forces. As these dynamics are more fully articulated, we expect that we may be in a better position to see when the scales tip sufficiently that one can say that the site has moved from the Transition phase to the Broad-based Application phase of change. The support structures or systemic forces are especially important to understand since they are the key aspect of the work that NFIE theorizes will be necessary to sustain the work over time. These are the factors that will lead to long term social change in professional development. Insiders and Outsiders The area of attention concerning insiders and outsiders this year was the role of NFIE and its work with the sites. At the heart of the evaluation work is helping NFIE refine the roles that it can best play to support the sites and move forward the broader agenda of Teachers Take Charge of Their Learning. We looked at it in light of the theory of change discussed above. In the first annual evaluation report (at the end of the sites' planning period), we described the initial theory of change which the Inquiry Team developed to guide the work and which is being explicated through the work. One feature of the theory was distinguishing the role of insiders and outsiders. The report stated: It is also important to consider the roles of insiders and outsiders in the theory of change. Often external consultants or private funders play key roles in initiating changes in systems. This is very valuable, but if consideration is not given to how the roles and functions of these external parties are transferred to people and structures within the system, the changes often disappear when the external groups depart. Thus, the theory of change used with ACOC includes conscious attention to this factor. For example, the initiative is designed such that relatively small amounts of funding are awarded grantees and matching funds from those within the system are required. Also, NFIE staff are consciously attending to the roles they play within each site and how those roles that seem important on a long-term basis are transferred to people within the system. For example, an NFIE staff role may be to link the site with experts in peer assistance and review. This linking role may be one that the teachers' association affiliates would take on over time. In such a case, the NFIE staff member may involve the state affiliate in its work within the site as a means of transferring this role. At the end of the first year of implementation we found that NFIE was playing six key roles as outsiders to the sites. These six roles provide the basis not only for NFIE's contribution to the work in the sites but also helped to more fully articulate the theory of change in regard to the role of outsiders. The first NFIE role is providing a focus. NFIE set forth a set of recommendations regarding professionalism for teachers in Teachers Take Charge of Their Learning which was then used as the focal point in the design of the grantmaking. NFIE's calls for letters of intent and proposals not only reinforced the focus on the recommendations but said that the implementation of the recommendations needed to be done with an emphasis on sustaining professional development over time through policy and practice. The initial conference for the grantees, site visits, and other communications throughout the year also were used to maintain the focus on the recommendations and the mechanisms to ensure sustainability. A second role for NFIE is at the core of its being. It provides credibility because it is a research-based entity and affiliated with NEA. Third, NFIE provides small amounts of funding to the sites to implement one or more of the report recommendations. Fourth, NFIE provides technical assistance. This is done through a kick-off conference, site visits, helping sites gain access to research and other written materials (especially via the Internet), and providing consultants on certain issues. The fifth NFIE role plays is to help disseminate exemplars of the recommendations in action in the sites. The sixth NFIE role is to work with NEA on getting the recommendations into official NEA policy. We developed several ideas about possible refinements or shifts in roles that NFIE might consider for the coming year. The refinements were in the categories given above. The theory of change was especially helpful in generating recommended refinements in NFIE's role. We used the support structures (or system forces) along side the categories of NFIE action given above to generate suggestions that would be truer to the theory. Another observation was that it may be useful for NFIE to consider its role as a mentoring role. By changing the language from technical assistance to mentoring makes the role of NFIE more congruent with one of the fundamental recommendations that NFIE was making to the sites. We are looking to see if showing the parallel between the NFIE work and how sites operate is helpful. Summary Comments on Using a Theory of Change to Support System Transformation Both the theory itself and the process of developing it are important issues if the theory of change is going to support system transformation. A key feature of the theory described above is that it focused on the conditions outside the classroom that are highly important in determining if professional development will be sustained over time. The theory is expected to be further enhanced next year by looking more closely at key factors in the classroom. As for the process of developing the theory, a key feature of our process is the Inquiry Team. The Inquiry Team for ACOC is comprised of NFIE staff and board members, NEA staff and evaluators. The Inquiry Team established the initial theory by drawing on members' understanding of research on change as well as their own assumptions. The process of discussing and developing the initial theory was an important step in raising the interest in the evaluation and preparing team members to receives the findings. Likewise the refinement of the theory helps bring to consciousness what is being learned. Another paper by the author discusses further points about using a theory of change to find transformative themes through evaluation. <-- Back