screen-shot-2016-12-13-at-7-21-55-pm

In 2013 The Ontario Public School Boards Association (OPSBA) made it a priority to develop a position paper on EQAO testing in response to Provincial Policy Memorandum 155 and the position paper of the Ontario Teachers’ Federation (OTF). A survey was distributed to trustees and senior staff of all the member boards and based on that some recommendations about EQAO were developed. The OPSBA discussion paper: “EQAO and Large Scale Testing In Ontario” was released to this public on December 12, 2016.

The paper has three main sections. The survey results, the recommendations and “Key Themes” from the survey,

Here are some highlights:

The Survey

screen-shot-2016-12-13-at-7-37-15-pm

Perhaps the most stunning finding of the survey is right up front. The survey found that 76.7% of OPSBA Trustees and senior staff want to either eliminate EQAO or agree that changes are required. Putting that in EQAO parlance only 23.3% of those OPSBA surveyed found EQAO to be at or above the provincial standard.

Recommendations

There are seven main recommendations:

  1. Equity & Accessibility: Work to remove cultural biases and make the test more accessible to students with IEPs.
  2. Technology & High Order Thinking Skills: Accelerate the use of technology in testing and have fewer multiple choice questions. Make the test more like the kind of learning that happens in the classroom.
  3. OSSLT: Allow students to go directly to the OSSLT remedial course without failing the test first. Make the OSSLT writing activities more like the writing students regularly do (collaboration, peer editing, etc.)
  4. Diagnostic Assessment: EQAO isn’t the same as a diagnostic test and shouldn’t be used that way.
  5. Student Well-Being: There was some suggestion in the Ministry’s Well-Being Strategy document that EQAO would develop a measure of student well-being. OPSBA doesn’t support this.screen-shot-2016-12-13-at-8-04-49-pm
  6. Overhaul how EQAO Shares Information With The Public:
    1. Clarify that EQAO data is limited in scope and has a narrow, specific purpose.
    2. EQAO should fit into school routines, and shouldn’t be a source of stress.
    3. Find a way of sharing data that prevents ranking schools.
  7. Use of Randomized Testing: The Ministry should examine whether randomized testing could effectively replace EQAO’s current Large Scale Testing model.

Reflections

Overall this is a very exciting document. It’s wonderful to have OPSBA confirm and agree what many vocal critics of EQAO testing have been saying. It seems that the tide on standardized testing is truly starting to turn and it’s great to see OPSBA also calling for significant reforms along with Michael Fullan and Andy Hargreaves.

  • The failure of the OSSLT trial earlier this year has left many questions about EQAO’s ability to successfully manage this transition. EQAO needs to pull back from integrating technology into their testing and develop a more effective approach. Technological failures are further undermining confidence in their competence.
  • The hand wringing about Ontario students and their math scores continues. There’s every reason to wonder if part of the drop in student’s math scores is related to EQAO tests not effectively measuring the kind of math learning students are doing. Standardized tests are good at measuring procedural math but not very good at assessing mathematical thinking. Having EQAO more closely reflect classroom learning may well help with this discrepancy.
  •  The notion that EQAO would develop a Student Well-Being measure is disturbing. Students need less testing, not more. One of the most effective ways of enhancing student well-being would be to eliminate EQAO completely, so eliminating the associated stress for students.
  • It’s great to see OPSBA agree that EQAO must take steps to prevent the harmful misuse assessment data by other groups. EQAO’s stance has been to condemn the practice but to take no steps to prevent it. OPSBA recommends that they start to do that. How about collecting school specific data but not releasing it publicly? EQAO can still fulfill its mandate by publicly releasing results for the school board without stigmatizing low scoring schools.
  • I first heard Dr Joel Westheimer suggest that EQAO could move to randomized testing and get equally reliable data on student achievement. Doing so would also de-emphasize the number of students affected and the stress caused by EQAO. Schools wouldn’t know who was being tested, so no need for a lot of teaching to the test. Great to see that OPSBA agrees.

 

screen-shot-2016-10-26-at-8-11-00-pm

As previously discussed EQAO has announced that the technical failure of the online OSSLT was due to “an intentional, malicious and sustained Distributed Denial of Service (DDoS) attack—a type of cyberattack.” However some evidence has surfaced that contradicts those claims.

ICT Educator Elliot Royle tweeted  a link to the Digital Attack Map for October 20, 2016.

The Digital Attack Map “presents data gathered and published by Arbor Networks ATLAS global threat intelligence system” and shows all the DDoS attacks that happen around the world on any given day. It shows the source, target and duration of any DDoS attacks on that day.

A typical day like October 19th looks like this:

As you can see there were three DDoS attacks that day, targeting Brazil, The US and Great Britain. As is typical the sources of the attacks are routed through countries all around the world.

Here is the Digital Attack Map for October 20th, the day EQAO said the OSSLT was taken down by a DDOS:

As you can see this map looks very different. There is some distributed traffic and an attack centred on The USA, but no attacks converging on Canada. Not one.

As another point of comparison, here’s the Digital Attack Map for October 21st, the day of the massive Dyn DDoS attack, the largest ever (so far):

Again, this looks quite different than the activity of October 20th when the OSSLT was cancelled.

So what can we make of this? There are only a few options.

The Digital Attack Map may be wrong. Arbour admits that while the “data represented in the Digital Attack Map is sourced from one of the most complete data sets available, it is an incomplete picture. The data may misidentify or exclude attack activity, and is intended to present high level trends in significant attacks as they are observed by Arbor Networks”.

There may have been an insignificant attack that doesn’t show on the map, but then EQAO’s system should have been able to withstand an insignificant attack.

Another option is that the failure of the online OSSLT wasn’t caused by a DDoS at all. This would be consistent with the reports by teachers that the MIST platform failed when it was tested last March and it crashed when it was used by Tennessee schools in February.

Perhaps EQAO can explain why Arbor Networks didn’t detect a DDoS attack on the day they said one took down the OSSLT?

cvnmlvzvyaezd1g

After some trials in March of 2016, EQAO scheduled a large scale trial of The Ontario Secondary School Literacy Test (OSSLT) on October 20th and was set to accommodate up to 250,00 students in two sittings. It was a no-lose proposition for any students that chose to write it, because if they passed the online version they could avoid writing the test in its regular pencil and paper version in March, 2017. If they failed, they could still write it again later.

The day before EQAO was confident they were prepared for any potential technical issues:

But even as EQAO was wishing students good luck it became obvious they weren’t as prepared as they’d previously thought:

EQAO remained confident that this was a minor problem even though teachers assured them that it was anything but minor:

After an hour’s delay EQAO continued to insist that students wait in the assessment rooms:

It took almost 3 hours before EQAO announced (on social media) that they were cancelling the test:

At 6:23 pm on October 20th EQAO released an official statement apologizing for the cancellation, saying they didn’t know what had happened and would be investigating the cause. Over ten hours since they’d known there was a problem and EQAO still hadn’t figured out what happened.

There was no further comment from EQAO for four days, but On October 24th they announced that the test had been cancelled due to “…an intentional, malicious and sustained” type of cyberattack, a Distributed Denial of Service (DDoS) attack. Coincidentally, this was the same type of attack that took out large parts of the internet in Europe and North America on October 21st.

Since then the students and educators involved, and the province in general, have been trying to figure out exactly what happened. Education Minister Mitzie Hunter called it “extremely disappointing” and Education Critic Lisa Gretzky wondered about the costs of the failed test:

Around all this discussion is a sense of uneasiness. A feeling that we don’t have the full story about what really happened. There are so many unanswered questions.

For example:

Why did the EQAO attack last so long and take so long to identify?

The EQAO Attack lasted three hours and 10 hours later it still hadn’t been identified as a DDoS attack. By comparison the massive Dyn Attack was identified almost immediately, the public was notified right away, and it was resolved in a little over two hours. Why did it take EQAO so long?

Who investigated ?

Apparently “IT experts and a third-party forensic team spent the weekend trying to figure out what happened” but who exactly did the investigation remains a mystery. Why? When tax dollars are being used why isn’t there full and transparent disclosure?

What successful trials?

Before and after October 20th EQAO mentioned repeatedly that they had conducted “several successful tests” of the OSSLT platform. However, some of the teachers involved in those tests dispute that they were “successful”, and said that the same issues that shut down the OSSLT on October 20th were present in the March tests.

Was the testing platform stable?

The online platform that EQAO was using for the OSSLT is called Measurement Incorporated Secure Testing (MIST), sold by Measurement Incorporated, a company based in North Carolina that “provides achievement tests and scoring services for state governments, other testing companies, and various organizations”.

The same platform had problems in February, 2016 when used for school testing in Tennessee that seem awfully similar to what happened in Ontario on October 20th.

There are also many implications for EQAO moving forward:

  • While the test was eventually cancelled some students were able to complete the online test (why??). Will those completed tests still be marked? If they aren’t, isn’t that unfair to those students who persevered?
  • EQAO stores millions pieces of data on students from eight years of age and older. Given what happened on October 20th, how can Ontario parents remain confident that personal information about their children will remain private when testing becomes online?
  • Will we ever know what happened? Richard Jones, director of assessment for EQAO said “I’m not sure if this kind of thing can ever be figured out,” which won’t fill anyone with confidence.
  • Should EQAO have been better prepared for what happened. Cyber security lawyer Imran Ahmad thinks so:

screen-shot-2016-10-25-at-1-46-47-pm

Ontarians should know what happened, why it happened and what EQAO is doing to prevent it from ever happening again. Our students and families deserve that. A confused shrug from Richard Jones isn’t good enough.

 

 

 

pokemon-go-1

Welcome to the era of Pokemon Go and moral panic.

Pokemon Go has conquered over the world, and it only took a weekend. After a global rollout on July 6th Niantic‘s augmented reality game Pokemon Go has simultaneously destroyed the online gaming world, rewritten the records for active users and app downloads, and in the process become an internet and social phenomenon. Of course, Pokemon Go’s takeover of our collective consciousness didn’t happen without a few hiccups.

In less than a month Pokemon Go has, according to mainstream media, been responsible for:

Along with a whole host of other bizarre mishaps, while also having a fatwa against it as an “un-Islamic” game.

Read More

Macro shot of A- on white results sheet. Extremely shallow focus.

In June of 2016 I wrote something on the future of report cards for TVO.org. Here’s an excerpt:

The digital revolution has transformed every area of our lives by changing how we send, receive and use information. Information is plentiful, easily accessible and flows freely and at high speeds, allowing people to make better, more effective decisions. The use of information technology has transformed many of the ways we educate students, with one notable exception – the school report card.

Twice a year parents open envelopes containing folded sheets of paper and see five months of learning reduced to letter grades or percentages and a few comments. Educators decide how to effectively communicate a student’s learning journey, and which of the thousands of learning moments are significant and deserve to be shared.

If this sounds a lot like the kind of mass communication your parents would feel comfortable with, it is. Many aren’t surprised we still use such an antiquated system, because schools are often criticized for being outdated and slow to modernize. Fortunately, there’s evidence that how educators communicate about learning is finally moving into the digital age.

Click here to read the full article

Shortly after I was invited to speak on the CBC Radio show Ontario Morning to discuss digital report cards. Here is that interview:

 

 

%d bloggers like this: