For the Love of Learning

Volume IV: Making It Happen

Chapter 19: The Accountability of the System

Accountability in education: What does it involve?

In this era of decreasing revenues and increasing uncertainty about the future, accountability is a key concern for many people. Virtually all public institutions have been criticized for failing to meet the needs of the groups they were intended to serve. In education, the dissatisfaction is often coupled with a belief that if only schools and those who teach in them were more "accountable," the problems relating to standards of learning and effective use of tax dollars would be resolved. However, as noted by educational researcher Lorna Earl in a paper written for the Commission:

As with every other issue we addressed, people naturally assume that, because concerns about inadequacies of the system are easy enough to articulate, solutions are as easy to find. And, as with every other issue, it is simply not so on matters of accountability. The issues involved, in fact, are quite complex, and if people are serious about introducing accountability into the publicly funded education system - as this Commission is responses must be equal to the problem.

Accountability means exactly that: Who accounts to the public for what happens in schools? Equally, it could be called responsibility: Who is responsible for the performance of our schools? How do we know what we are entitled to expect from schools? How do we know whether schools are delivering on this entitlement? Whom do we hold to account - who is responsible - if we are not satisfied with the answers we get?

Accountability in the education system, then, means that information has to be available to the public, to taxpayers, and to parents, in a form that allows them to have reasonable expectations of the system, to make reasonable judgments about how well the system has performed, and to know who is responsible if they are not satisfied.

The most fundamental form of accountability is that of the classroom teacher to the parents, and of the school to the community. It has been extraordinarily difficult for parents to find out simply what the curriculum is and how their children are performing. Although our focus in this chapter is on system accountability, the ultimate concern for parents and students is, naturally, with individual student learning. But at both the individual and the system level, we should remember that accountability is not an end in itself: its function is to ensure that information about performance is actually used to improve that performance in the future. In other words, "accountability" and "reform" should always be closely linked.

Two types of accountability are relevant: fiscal and program. Below, we look briefly at each, and then discuss what additional measures should, in our view, be taken to satisfy the public that the educational system is operating as it should, and that identified problems are being addressed."

Fiscal accountability" at the school, school board, and Ministry level is addressed by the use of auditing processes to examine operations on a regular basis. As well, the Ministry conducts spot audits of boards, examining its transportation functions, verifying enrolment figures, and ensuring that provincial grants are used as specified. Such audits usually focus on whether funds are administered honestly and according to regulations. Many people told us they wanted to know more about whether the system is run as efficiently as it ought to be, and whether funds are allocated appropriately.

In his 1993 annual report, Erik Peters, the provincial auditor, looked not only at the usual fiscal issues, but also addressed "value for money" questions, suggesting areas that needed to be improved. He noted, for example, that:

We believe that such initiatives should continue, but we caution that auditing an education system is a complicated process. As we stressed in Chapter 11, on assessment procedures, the qualitative acts of teaching and learning do not easily lend themselves to quantitative measures of efficiency and effectiveness; judging schools on the basis of inappropriate tools does not contribute to public knowledge. "

Program accountability," in the sense of establishing and assuring quality of student performance, is a key priority. We agree with the many observers, both in and outside the educational system, who believe the time has come for a clear set of criteria by which performance can be judged: people need to know what students are expected to have learned by the time they complete a given course or grade (the outcomes) - and what different levels of achievement mean (standards). Such a framework can, and must, be used to monitor and enhance the progress of students and the performance of the system. The results of such monitoring must be communicated in an understandable and timely way to all stakeholders.

In Chapter 11 we addressed the need for clearer and more useful assessment of student learning - a very large part of improved program accountability. That is the purpose of the province-wide literacy tests we recommend for every student in Grades 3 and 11. And individual results to students and parents, and system results to all interested parties must be clearly communicated. That these system-wide assessments are associated with what we term "literacy guarantees" is a particularly powerful accountability mechanism. Of course, the question of the adequacy of the standards applied to the test results is also a fundamental accountability question.

We have also said that the Ministry should continue to conduct other program reviews, through testing sample groups of students across the province. Results from such reviews make it possible to judge the adequacy of the curriculum and whether the official curriculum is actually being taught and learned in schools.

Beyond that, student assessment would be primarily the responsibility of the teacher and the school board, and, as we note, it is important for all teachers to learn more about how best to assess student learning and use the results of assessments to improve instruction and program.

Who is accountable?

The education system involves both elected and appointed policy makers, and both are accountable for their actions. At the local level, trustees are accountable to the electorate every three years, although it is widely acknowledged that complications exist: there is little attention paid by the media to the activities of boards of education, little useful discussion of education issues during elections, and notoriously low voter turnouts. At the provincial level, the Minister of Education and Training is, of course, accountable to the electorate whenever an election is held, as is the government as a whole.

Although such political accountability is important, it hardly seems sufficient to us, because the information that would allow voters to make informed decisions about the system may well not be available. In terms of political accountability, policy makers at the local and provincial level must answer for the soundness of their policies, and also, to some extent, for the results of those policies.

On the administrative and managerial side, there is a need for accountability for implementing policies and for monitoring the process and the impact of implementation.

If education policy makers are going to be held accountable, they will need measures of educational quality. Without these, they cannot report reliably and meaningfully on the soundness of their policies.

Indicators of quality

The education system, like any other publicly funded system, is accountable to the public for operating effectively, efficiently, and equitably - although, as we have stressed, such accountability is far easier to demand than to deliver. If the system is to be as accountable as possible, there must be far more clarity about its purposes or objectives. We believe that considerably more information should be made available, and it should be collected regularly and presented in more consistent, understandable, and meaningful ways. This will enable members of the public to look at it and arrive at their own conclusions about how well the system is operating.

The first step in the process, as we emphasize in Chapter 17, is that the provincial government, through the Ministry of Education and Training, establish clear directions and expectations for the education system, in terms of student learning, regular assessment, parental involvement, and other important objectives.

The term "indicators" is used to refer to quantitative and qualitative data that describe various features of the school system. The obvious problem is that from an education system as a vast as Ontario's, one can derive endless statistics, and there can be indicators that tell us something about literally any part of the system - and they may refer to the student, school, board, or provincial level. Decisions about appropriate indicators of a successful system will determine what kind of information should be collected.

Student achievement is the most obvious indicator of the effectiveness of an education system. If students are doing well on measures of learning in relation to standards established locally and those established province-wide or beyond, schools and school systems are usually considered to be doing their jobs satisfactorily and providing value for taxpayers' dollars.

At the moment, there is a wide-spread sense that schools are not doing the jobs well enough, based on both anecdotal evidence and media reports of certain provincial, national, and international tests.

Questions of acceptable standards and their levels, became a particularly contentious issue in Ontario in 1994, when the results of Grade 12 writing reviews and Grade 9 reading and writing tests were released. Members of the public seem to be concerned that expectations of students are too low, and that acceptable standards are not high enough.

While this report consistently stresses the need for more challenging and rigorous learning for our children, we stress that an in-depth sense of student achievement is far more difficult to assess than the media and the public often seem to think.

Moreover, student achievement, crucial as it is, is not the only indicator of the quality of the system, and it is not the only outcome for which the system is accountable to the public.

Other indicators of educational success and quality include such factors as the proportion of students who enter college or university, or who enter employment readily; the relative representation of minority students across all achievement levels and across different programs; per-pupil costs; the drop-out rate (the percentage of students who leave school before graduating); attendance rates of both staff and students; the rate at which students progress through the school system. A different type of indicator, but an important one, relates to the way in which teacher and administrator performance is evaluated, and how the results of evaluation are used to improve performance of individuals and of the system as a whole.

There are also indicators that are not (or not directly) learning related, but also suggest the degree to which a school or system is well managed. These include cost efficiencies, implementation of fair employment practices, and the achievement of acceptable standards of workplace and school safety.

Finally, we believe that the level of satisfaction expressed by students and parents - and, to some extent, by the community - is also a useful indicator. To what extent do these groups feel their concerns are addressed, their ideas welcomed, their needs met?

Policy makers and administrators can, through regular and systematic sampling of student, parent, and public opinion, be alerted to potential problems that need to addressed. Let us be clear: we are by no means suggesting that education policy and practice should be determined by public opinion. It should not. However, if an education system is to serve its public well, the system should monitor the concerns and reactions of those it serves.

Assessment agency

Until recently, Ontario, in comparison with other jurisdictions, did not place a high priority on monitoring, assessing, and reporting various aspects of school system performance, at either the provincial or local level. The problem is that, without regular monitoring, teachers and principals do not receive the kind of feedback that allows them to adjust their instruction and curriculum planning. Nor does the public have the information on which to base reasonable judgments of schools. Assessments, therefore, must not only be carried out, but must be widely reported in understandable ways.

Although most people, including educators, are coming to agree that more monitoring of system performance is justified, there is little consensus on just how this should be done. There is particular disagreement on whether an independent agency should evaluate and report on the system, or whether the responsibility should be left with the Ministry: there is some concern about the capacity of the Ministry to carry out monitoring, or to be as open and objective as required.

In other countries, including the United States and Australia, there are models of agencies that do large-scale assessments; they usually operate nationally rather than just at the state level. They tend to be quite large institutions that develop tests, administer them, and report on the results. Such large-scale assessments are extremely expensive to develop and administer, and are not easy to change when there are major shifts in curriculum policies.

While throughout our work we have been reluctant to recommend the creation of new bureaucratic structures, largely for the reasons just cited, we found, in the end, that the argument for an outside assessment agency is persuasive.

Education policy is set by governments and, therefore, is by definition political. But in matters of assessment, public credibility is probably the overriding need. Therefore, an arm's-length agency, removed from the political arena, seems to be the inevitable solution.

We see such an agency as consisting of a small number of experts in education and assessment with overall responsibility for evaluating and reporting on the success of Ontario's education policies. As a mark of its independence, this Office of Learning Assessment and Accountability, as we have chosen to call it, would report directly to the Legislature, perhaps through the Standing Committee on Social Development.

The first job of the new office would be responsibility for the Grade 3 language and mathematics test and for the Grade 11 literacy test, as recommended in Chapter 11 for all students. To keep the office small and flexible, it would not itself develop and administer these tests, but would contract with assessment experts, preferably, but not exclusively, from Ontario.

The contract process would involve issuing a public call for proposals, to be advertised widely. We would hope that the Ontario Institute for Studies in Education (OISE) and other Ontario graduate schools of education would respond enthusiastically to the call for proposals, as would measurement experts in departments of psychology or elsewhere.

Recommendation 163 (Cf. Chapter 11, Rec. 51)

*We recommend that the government establish an Office of Learning Assessment and Accountability, reporting to the Legislature. Its first responsibility would be the Grades 3 and 11 system-wide, every-student assessments.

The Ministry and school boards have a variety of information-gathering mechanisms that can and should be adapted to give additional information on such things as drop-out rates and breakdown of data by region, language, gender, race, etc. There is no need for other agencies to develop new systems, but it is important that the existing systems be improved to ensure that necessary information is available for the Office of Learning Assessment and Accountability and for the Ministry, so they can provide accurate information to the public on the effectiveness of the entire education system.

Accountability and consistency

What is critical, and what will require some changes in data-gathering and reporting procedures, is that the data be comparable from board to board and from year to year. One of the problems in assessing today's education system is a lack of good past data for useful comparisons. Information on drop-out rates, for instance, has been difficult to get and to interpret, but the Ministry, in collaboration with several school boards, is currently developing common systems for tracking and reporting them.

Over a number of years, many school boards have developed their own systems for keeping track of information about programs, staff, students, and finances, as well as about student achievement. Not surprisingly, they are reluctant to abandon their investments by adopting new and different systems, even though these might be more useful for province-wide use.

However, we note that adoption of the Grades 3 and 11 tests will require all school boards to use a single provincial identification number for students; once that is done, developing a single database for all students in the province will be much easier. The Ministry established a Student Information System in 1986, which could be the basis of an expanded system for tracking students; it would be important to maintain data after students leave the system, in order to do longitudinal research when that is appropriate.

We have already mentioned other existing mechanisms for accountability, such as the work of the provincial auditor, and other provincial reviews and audits. We expect these mechanisms to continue to be used, but see a need for clearer guidelines, as well as for greater public scrutiny and reporting.

We firmly believe that the best way to ensure accountability is to make public the relevant information about the characteristics and performance of the school system, and to publish it in a way that is readily understood and interpreted by people. Only then can members of the public decide whether their schools are providing the kind and level of service they want.

In recent years, the Ministry has not always closely monitored boards' implementation of its policies and related programs. Monitoring is sometimes perfunctory: boards are required to file documents showing they have the required policy statement or plan (on special education, for example, or on anti-racism), but not whether that policy or plan is, in fact, being implemented in the schools - or, even more important, whether the policy is having the intended effect.

We believe that provincial policies should be developed in terms of broad directions, and should be accompanied by a clear description of how they are to be assessed. Then, the most important monitoring is of the intended results, or outcomes, leaving it up to school boards to decide the details of how they are to be achieved.

The difficult challenge is to balance central direction-setting and monitoring with local flexibility about the ways desired results are achieved, linking "top-down" and "bottom-up" strategies for reform. Because the Ministry sets the province's direction for schooling, it must articulate its sense of a shared purpose, and set clear expectations. Schools and school boards would then be responsible for deciding, within the broad guidelines, and based on their knowledge of the local context, how they will work to meet those expectations.

Although government monitoring - evaluating whether local schools are doing what they are supposed to be doing - is quite rightly seen as a key element of accountability, monitoring is expensive. Therefore, the information gathered must be available to, and actually used by, schools and school boards.

We are convinced that, in the long run, the most critical accountability mechanism is full public disclosure of all relevant data concerning school and school system performance, delivered in a meaningful form. It has been suggested to us that the Ministry ought to apply sanctions to boards that either do not comply with Ministry regulations, or whose performance is not satisfactory; withholding funds is the most frequently suggested sanction. This is difficult for the Ministry to implement, because students and parents will suffer.

We believe, however, that if data are made available to the public in ways that are understandable, consistent, and comparable, parents and the community will put pressure on schools and school boards to improve weak areas and close gaps. If they do not, trustees will not be re-elected, and it will be difficult for principals and supervisory officers to maintain any credibility. In a democracy, this is the ultimate form of accountability. In other words, we believe that if people have the information they need, they will be able to judge and act appropriately.

Reporting

We also believe that the information on the system indicators and on student assessment should be readily accessible, not only to the public, but, wherever possible, to the press.

The Minister of Education and Training and individual school boards prepare annual public reports, although we doubt that most Ontarians have ever read one. We think these reports could be considerably more valuable than is now the case.

In the first place, clear content guidelines for both the Minister's and school boards' annual reports, with a list of agreed-on indicators to be addressed, would make it easier for the public to understand and make judgments about the information and about the system. Although it is not difficult to agree on at least some indicators of a successful education system, achieving consensus on comparable ways of gathering, summarizing, and reporting such information is much more difficult. Various measures or indicators can be seen as snapshots, providing diagnostic information about many aspects of the school system. No one measure can give a full picture; several have to be examined together if members of the public are to make reasonable judgments. Questions of how indicators are to be developed and how the indices are to be used must be addressed by the users, and by the technical experts who develop the statistical indices.

The Office of Learning Assessment and Accountability should begin its work on this task by bringing together education stakeholders, including boards, federations, and faculties of education, as well as parent, student, and community groups. Working with the other groups, it would develop the lists of indicators and, with input from education stakeholders, decide how the indicators should be defined, calculated, and reported.

Recommendation 164

*We recommend that the Office of Learning Assessment and Accountability also be responsible for developing indicators of system performance, to be used at the board and provincial levels.

Indicators for school board reports would include reporting on the results of large-scale and other assessments and on audits specific to the board. Reports would also include an indication of what actions have been taken to address problems revealed by the assessment, and what further actions are planned.

The indicators used by the Ministry should also include reporting on assessments and follow-up; it would be expected that board and Ministry reports would provide summary statistics decided on by the Office of Learning Assessment and Accountability.

In our view, the Quebec Ministry of Education produces reports that may be useful as an example for Ontario. Quebec's Educational Indicators for the Elementary and Secondary Levels is analogous to Ontario's Key Statistics, but is more complete. It not only tracks indicators over time, but also comments on the most important points arising from an analysis of the indicators, all presented in an attractive and easy-to-comprehend 80-page format.(3)

Recommendation 165

Moreover, we believe that boards and the Ministry must pay more attention to providing useful information to the public on an on-going basis; they should ensure that information on policies and their intended outcomes is available, as are the results of any evaluations. Some of this will be published in annual reports, but other data, such as descriptions of policies and their outcomes, will have to be provided in a more timely way; as well, there will be occasions when it is useful to have more detail than would be appropriate for an annual report.

In order to assure the public that all information and reports are accurate, that interpretations are defensible, and that boards and the Ministry are held accountable for learning, the Office of Learning Assessment and Accountability should do spot checks of a sample of board reports, and monitor board and Ministry assessments of outcomes. The office should report publicly on these activities, and could do so, informally, by having the head of the office meet regularly with the Committee of the Legislature, and, formally, through an annual report. We stress that, to keep costs down, the review should be done on a sample of reports and assessments.

We would not want any structure we recommend to exist beyond its actual usefulness. It is not impossible that the Office of Learning Assessment and Accountability might one day prove redundant, and it is entirely plausible that its responsibilities might need to be revised.

Recommendation 166

*We therefore recommend that the work and mandate of the Office of Learning Assessment and Accountability be reviewed in five years.

In Chapter 12, we recommended the formation of an Ontario College of Teachers, as a professional self-governing body responsible for setting professional teaching standards in the province; thus, it would play a critical role in the provincial accountability framework. The college would be responsible for ensuring that high professional standards of teaching, and of teacher preparation programs, meet the needs of Ontario schools. Its duties would also include setting and monitoring the framework governing renewal of teacher certification every five years.

Because we have now recommended the addition of two new bodies to the education system, it might be helpful to summarize briefly what they would do and to whom they are accountable.

Finally, we have made recommendations concerning the education responsibilities of ministries other than that of Education and Training, and of other agencies of government. Should the government assign such duties to other government bodies, there would have to be an accountability mechanism for those agencies.

Conclusion

Until recently, issues of accountability did not receive as much attention in Ontario education as many taxpayers and members of the public would have wished. However, there have been many changes in the past few years.For instance, public reporting of the provincial Grade 9 reading and writing tests, released in the fall of 1994, not only provided board data, but school results as well.

We are of two minds about this development. On the one hand, we, of course, applaud the move to share all useful information about students' performance with the public. On the other, we remain seriously concerned that information without perspective, context, or proper interpretation can, in fact, do more harm than good. As we point out in Chapter 11, serious tests are not horse races and should not be reported or judged as such.

To appraise an entire education system on the basis of one test or a single set of tests, and to ignore the many factors that determine whether one school's students do better than another's, is an imperfect exercise at best.

We want the system to be open and accountable, and our recommendations would go far to achieving that goal. But we also want that information to be meaningful and relevant. In that context, we would hope the media will present data in a proper context in a way that enhances, rather than distorts, public understanding.

Once good information becomes available, the onus will be on the public and on parents to use it to make reasonable judgments, and to find out how schools plan to improve programs on the basis of current results.

The onus will also be on educators to work together to continue to improve their programs on the basis of the feedback represented by such results. After all, the point of developing better accountability mechanisms is to help schools to be more effective.

__________
Endnotes (Chapter 19)

1. Lorna M. Earl, "Accountability and Assessment: Ensuring Quality in Ontario Schools." Paper prepared for the Ontario Royal Commission on Learning, 1994.
2. Ontario, Office of the Provincial Auditor, 1993 Annual Report: Accounting, Accountability, Value for Money (Toronto, 1993), p. 66.
3. Quebec, Ministry of Education, Education Indicators for the Elementary and Secondary Levels (Quebec City, 1993).

ISBN 0-7778-3577-0
©Copyright 1994, Queens Printer for Ontario