Concordia Journal Home > Archives > January 15, 2009 issue >

The limits of quantitative evaluation 

By Karen Herland

Education professor Ayaz Naseem did not pull any punches in Scientism and Education: Empirical Research as Neo-Liberal Ideology (co-authored with Emery Hyslop-Margison of the University of New Brunswick). The book argues persuasively that use of traditional empirical research methods to improve the quality of education in society is not only inadequate, it might actually be an impediment.

The argument’s receptive audience is evidenced by its place among the winners of the prestigious American Educational Studies Association (AESA) 2008 Critics Choice Book Awards.

“I was a bit surprised by the recognition. This is an American award and the book is critical of policies there. It is a clarion call to challenge some popular assumptions,” says Naseem. The book was recognized for its originality of research; scholarly/ intellectual impact on the field and the significance of the topic.

The two researchers open their book by stating that the entire discipline of education is at a critical crossroads, affirming the last 150 years of scholarship represent “epistemological mistakes.” They argue the discipline should be bypassed in favour of a direction that recognizes the limits of empirical research and takes the “more enlightened choice … (that) emphasizes instead the purposes, goals and strategies of education within a democratic society.” In short, the book challenges the centrality of quantitative research that often dismisses qualitative factors as less theoretically grounded.

Although the book addresses an American trend, it is one that continues to influence Canadian practice. For instance, the U.S. federal program No Child Left Behind, introduced by the Bush administration, relies on scientific tools to measure the effectiveness of teachers and educational programs in a standardized way. These same measures are being increasingly supported by programs like SSHRC in their evaluation of Canadian research projects involving the field of education.

“The real problem with this form of measurement is that it does not take into account differences in class, gender, race or ethnicity,” argues Naseem. “It is based on the assumption that all students are on a level playing field.” He is careful to point out that this is not a liberal argument for reverse discrimination but one that needs to acknowledge very real gaps in access and opportunity.

The danger with these assumptions is that children are promoted, demoted, and (ultimately) labeled, based on the results of these supposedly objective measures.

A second argument in Scientism and Education builds on this criticism within a climate of neo-liberal policy.

Naseem and his co-author argue that educational institutions are increasingly measuring successful graduates solely by their ability to integrate into the job market. They wonder whether that is the best measure of achievement.
“Do we want our students to be workers for the global economy, or democratic citizens?” If the latter is valued, other determinants might be more appropriate markers of success.

Naseem continues to apply these ideas to his current research projects. He has a book coming out with Palgrave Macmillan that considers the constitution of gendered citizenship in developing countries using women in Pakistan as a focus. His research projects are funded by the SSHRC, the FQRSC and the Alexander von Humboldt Foundation, Germany.