Skip to main content Skip to secondary navigation

Matthew Rascoff on Empowering Students with AI

Main content start

Matthew Rascoff, vice provost for digital education at Stanford University, presented “Empowering Students with AI” at Unleashing Human Potential Powered by Technology, the Esperanza and Minerva higher education summit held in Hong Kong, October 13, 2023. Here are the video and transcript of Matthew’s talk.

Transcript

It is really a pleasure to be here. This is my first time in Hong Kong, which I’m a little bit embarrassed to say, but it’s been a long time coming and I have so admired the Asian educational cultures for so long. There’s so much curiosity in the world that I come from about the story of the educational success of the systems here, and to get to experience it firsthand with some of these experts, some of the expertise of faculty from the universities, but also the edtech innovators who are bringing their ideas to us, it is truly inspiring to me. 

I think one of the challenges of an event like this is we all come from such different backgrounds and different contexts, and education systems are still very nationally rooted despite the global efforts of organizations like Minerva and Esperanza to diffuse ideas, innovations around the world, and great edtech innovators who are cosmopolitans.

And they’re crossing borders and they’re spreading these ideas. So one preface that I would offer is that I come from the US system, I was trained in the US system. I’ve lived in other countries, I’ve studied in other countries. But I think there are some limitations, and what might be innovative in one country might be humdrum for you in another context. And so bear with me on that. This comes from a US perspective. 

When I teach at Stanford in the business school, you know, I have 50 students in my class and 40 different countries represented. And I’m teaching them, kind of, about the American education system. And I always feel a little bit uneasy about the provincialism of a curriculum that’s organized around learning about what’s happening in education.

I console myself, though, that they are the enablers of learning. They’re going to go back, most of them, to their home countries, and they’re going to spread ideas. And there is a kind of cross-fertilization that happens through that process, in which there’s kind of intellectual arbitrage, and there’s new ideas that come from the exchange of ideas in contexts like this.

So that’s my caveat on the American perspective that I’m going to bring, and gratitude for all of you for hosting me here and for listening in as as I as I share what’s happening at Stanford on this. So the title of this is “Empowering Students with AI.” And I think the message that I want to leave you with is the way to do that is to empower educators with AI, and that humans have a fundamental role that we cannot discount, rather, we should be investing in much more deeply, even as the value of human educators is being questioned. 

So I was listening very carefully this morning, and I did not hear a single time the term “personalized learning.” If we were doing this conference five years ago, personalized learning would have been the beginning, middle, and end of this. But it has not come up a single time. 

Why is that? I’m asking genuinely, why did we not talk about personalized learning today? Has anybody worked in the space long enough to know that fad, the rise of that fad, and then the decline of that fad? I see some nods. Why did personalized learning fail at the Chan Zuckerberg Foundation such that they’ve just written off a $100 million investment in their Summit Learning platform, which was supposed to be the scalable mechanism for kids to do personalized learning in K-12 schools across the United States?

They laid off all of their staff who were working on this. They’ve given up on this idea. To me, this should make us somewhat skeptical of the kind of rhetoric that we heard from Ben [Nelson] and from George [Siemens] this morning, that, you know, there are some fads and some faddish behaviors in this system, that seek to chase the latest idea and sometimes lose track of the fundamentals that do not rise and fall on a five-year cycle, but are much more oriented towards the long term.

And to me, that has to be our focus. You know, we were talking a little bit this morning about the patience that is required. John [Tsang], this came up with you, the patience that’s required to invest in this space. The investors don’t necessarily get that. They’re impatient. They want returns. Right? But if you’re an educator in this space, think about the deep impact for the long term that educators have had on you, individual teachers.

And those are not fads. And so to me, you know, Zuckerberg and Chan Zuckerberg and Bill Gates talking about personalized learning — this was the Gates Foundation’s core investment thesis — they’ve basically given up on it, and Salman Khan has talked about it, too. Personalized learning allows students to progress through content at their own pace without worrying about you being too far behind and too far ahead of their classmates.

Where’s that? Where has that gotten us? Kids on computers, in classrooms with headphones on, who are not learning with one another, who are not being socialized, who are not being helped to create an identity, who are not building a learning community, they are not progressing. And the data finally caught up. And I credit Chan Zuckerberg for at least being honest about the lack of results and being willing to write off a $100 million investment.

So to me, the core fundamental does not change over time, no matter how advanced the technology is that you’re building, is that we need to be investing in great educators and great teachers. And they actually do do personalized learning. Great teachers are listening to a student’s needs and they are doing it systematically as part of what they do.

This is an example from Dan Meyer, who’s an educator who I love, who I highly recommend. He’s a math educator who writes about math pedagogy, mostly in K-12. But I think a lot of these lessons are relevant to higher education as well. And he basically has argued that an educator like Liz Clark-Garvey in New York City public schools, she can start the lesson with a whole-class move.

She’ll ask one question for the whole class and then through the class, moving around the class, listening to what the students say, listening to how they decipher this problem, she is able to understand where students are at and to meet their needs. The problem, I think the challenge, with a context like this, is it seems to depend on heroic individual teachers like this.

And there has not been a systematic mechanism. Maybe in Singapore there is. Maybe in Hong Kong there is. But in the US there has not been a systematic mechanism to take a model like this from individual great educators and scale it to the order of the millions of teachers that we have, three million teachers in our schools, not to mention higher education.

So to me, the challenge is not, How do we give every kid a laptop and a screen and headphones? The challenge is how do we give them a great educator who cares about them, who will create a learning community in the classroom of people who will learn together, who will support one another? That to me is the precious thing and the rare thing and the thing that has become even more precious and more rare under the conditions of technology, seeming to take away some of the role for humanity in our classrooms.

So to me, if you care about personalized learning or at least the ideas behind personalized learning, the way to do that is not to take the persons out. It’s to bring the persons in, bring the humans in. But how do you do that systematically? How do you train an educator workforce to really capitalize on what makes us most human? As we were talking about earlier today, that is a nontrivial challenge for teacher training programs and for in-service teacher training for teachers who are already in the workforce, and for faculty at universities who sometimes get the least training in pedagogy, including at places like Stanford.

So that to me is that’s how we should frame the challenge. And how we frame the challenge is going to determine what kinds of solutions rise to the top. What I want to talk to you about today is a model that I see emerging at Stanford for how we can simultaneously democratize and humanize education at the same time.

So usually these are a trade-off, right? Think about, MOOCs did a great job of democratizing education, but took every last iota of humanity out of a course and totally denatured the course. They they basically turned the course into YouTube, for all intents and purposes. And even forums have been taken out of Coursera. Why? Because it doesn’t work when you have an always-on platform.

They eliminated the idea of cohorts, which were there at the beginning. That’s gone. So MOOCs did a great job of democratizing, but did a horrible job of humanizing education. They made it this cold, bloodless place to learn that most people cannot get through, and only 6% of learners can get through a MOOC for that reason. But places like, you know, the liberal arts colleges that George was talking about this morning, they do a great job of humanizing education.

I teach at Stanford this quarter. I’m teaching 14 students, two instructors, 14 students. Think about that ratio. Think about the luxury of having two instructors for 14 students. That is an incredibly wonderful privilege. That is very expensive to provide. So we figured out how to humanize, but we have not figured out how to democratize because the model of Stanford is predicated on extreme selectivity, and we want to be able to provide that to many more.

But what is the mechanism by which you can do that? So to me, this is a fundamental balance that we need to strike. And the model that I want to share comes from a project called Code in Place. So this comes from a colleague of mine, Chris Piech, and his colleague Mehran Sahami in the Computer Science Department at Stanford.

Has anybody heard of this project before? I’m so glad you haven’t, because it means I’m sharing something new with you. So this was launched during the pandemic and it’s now run in three separate cohorts, and it’s reached 30,000 students. And it has a unique pedagogical model that to me exemplifies what’s possible in the balance of democratizing and humanizing education at the same time.

So this is the structure of Code in Place. I call it the meso scale. The meso scale is kind of the middle ground between the the macro scale, the mega scale of MOOCs, which is on the order of millions and the extreme micro scale of face-to-face learning at places like Stanford, which is on the order of, you know, tens of thousands.

And in between those is the meso scale. And the way this works, the way this model works is that it’s a combination of synchronous and asynchronous learning. But all students are in a group of ten, and that ten has a human section leader, a volunteer who’s been trained by Stanford. Many of them are Stanford alums, not exclusively, and they’re responsible for creating a learning community among those ten students.

And we had them in every single time zone around the world. This was where the geographic representation of our first cohort. But this chart basically shows you how we built this kind of fractal model of scale where every group of ten is led by a section leader and then every section leader themselves has a community, and those are led by training leaders.

So what you end up with is 30,000 students, but it’s not 30,000 students who are treated as one bulk of humanity. It’s 30,000 students, each of whom is in a small section that is led by a caring individual who’s been trained to support them, and creating a cohort, creating accountability, creating community among those group of ten in every single time zone in the world, in multiple languages.

And so to teach 12,000 students in a cohort, we trained 1,200 co-instructors, maybe the largest co-taught course in history. That’s what the faculty think. Trained 1,200 co-instructors to reach 12,000 learners in a year, and that is the way to get to this kind of scale. The question, though, is, you know, what does their teaching look like, and how do you support 1,200?

So the results speak for themselves. Like 99% of our section leaders completed it, 56% of students completed. Not bad for a free noncredit course in which there is basically no skin in the game. Right. So compare that to 6% for MOOCs. 4.95 on Stanford’s five-point teaching scale.

Take that for whatever it works. But the net promoter scores, that’s an indication of whether somebody would recommend taking a course like this or teaching a course like this in the future...30% of students said they would like to lead a section if offered again. So like that, that’s the results. 

How do you scale this, though? How do you get to 1,200 co-instructors? And this, to me, is where the AI comes in. And what I’m showing you here is an intervention that we designed that is an AI feedback tool for the instructors that ran as a randomized controlled trial behind the scenes in Code in Place.

And this randomized controlled trial, it was basically some of the teaching fellows, some of the section leaders got this intervention, and others did not. And it had in the end size of 1,100 co-instructors. And this is the feedback that we were able to give those who received the experimental treatment. 

And it’s maybe a little bit hard to read, so I’ll just call out some of the text that’s in here. This feedback gives you an opportunity to reflect and to support your professional development. It is not meant as an evaluation. So this is a formative tool to provide what’s called instructional coaching. Instructional coaching is a proven method for improving teaching practices, and it is a formative mechanism that brings an expert to the back of a classroom to give advice and feedback to a teacher.

It’s usually done in the context of K-12 education. It also happens in the higher education context. I ran a program like this in my previous role at Duke University, so it’s a proven mechanism, but it’s not scalable, it’s very expensive, it’s underprovided. So it’s effective, but it’s not cost-effective. What we built here was an AI instructional coaching mechanism and the instructional coach in this AI, it was somewhat limited.

It could only understand English, so we were teaching this in some other languages, but it could only understand English. And it was basically measuring two mechanisms of student engagement, student talk time — how much do the students talk — and moments when you built on student contributions. So student talk time is an important concept to understand. This is a translation of research that was done by Rachel Lotan at Stanford.

I see some nods from the psychologists here, and it’s critical for understanding classroom equity and classroom effectiveness. And it is predicated on the idea that if students are not talking, they’re probably not learning. And Rachel Lotan did this pioneering research in labs at Stanford that basically showed that student talk time and equity of talk time is critical not just for demonstrating their learning, but for actual active learning to happen.

So this is a measure of how much students are doing the talking in their class and the quality of the uptake, the quality of the response that faculty are giving, or the section leaders in this case, are giving to the students. So are they hearing what the students are saying and responding to it? “Thank you for that point,” “I agree with you,” “Let me build on that point,” that kind of a comment. That’s a high-quality piece of feedback that’s very hard to come by, and it’s very hard to coach to do that. The AI is there to do that in a nonjudgmental way. So there’s no evaluation, there’s no supervisor listening in. It’s a little bit different than the ClassIn model where, you know, there is a kind of supervision component to this.

This is not it. And it’s, it’s basically, I think, an effective use of the AI as a coach who’s not there to judge you and just wants to help you get better. And so, but it turns out, this mechanism, it’s very, very effective for helping teachers improve. 

So what you can see here is a related model from a program called TeachFX, is a Stanford-student-created startup that’s now a commercial edtech company, that also uses a similar mechanism.

So what I just showed you previously is called Empowered, which was the basis of this randomized controlled trial that we ran inside Code in Place. This is a commercial product that any school can purchase, that basically shows how they measure the distribution of time in a classroom. And what TeachFX has found is that just giving teachers a single recording with an analysis for how much they spoke and how much the student spoke is enough to significantly affect the second time they teach.

So there’s a measurable impact on the quality of their teaching just from hearing this data once. From seeing the data once from their own classroom, they improve the next time. So it’s a very, very effective mechanism for helping them improve. So TeachFX has recently built new AI tools that have more quality measures. So not just quantity of talk time, which we know is really important, and the distribution between the teacher and the students, but also a more semantic analysis using natural language processing of what is actually happening, so similar to what you saw in the Code in Place mechanism of the uptake of a student idea.

You can see, does anyone see this information that Rakiya is talking about? You can see the distribution of who’s talking when. You can replay an audio clip in this moment. So this is feedback that’s being given to the instructor. You’re guiding a discussion about data usage in units. The AI is understanding this and getting this from the transcript of a classroom, an online or a face-to-face classroom, either one. Building on the students’ contributions, you prompt the class to find the connection between Rakiya’s comment and the topic at hand, encouraging them to think critically. You also acknowledge a correct response from Deborah, further engage in the conversation. That was written by a machine that’s listening in on the classroom and giving feedback on the richness of the discourse that’s happening in the classroom.

This, to me is the potential of AI. I don’t care about ChatGPT. If every teacher had this, we would see immediate improvements in classrooms. This to me makes ChatGPT an absolute distraction. We know this is effective. This is backed by years of data. This is backed by educational research and cognitive science research. And every teacher in the world would benefit from having an instructional approach like this. 

And we have the randomized controlled trial data that comes out of the Code in Place project to demonstrate the effectiveness of this. So this is a working paper [“Empowering Educators via Language Technology”] that my colleague Dora Demszky, who led this research and is my critical research partner in this project, she published with Chris Piech, who led Code in Place, that showed instructors’ uptake of student contributions improved 13%.

So this is gold standard research, randomized controlled trial research. I encourage you to check out this paper. That is a huge improvement in educational terms, 13% improvement in this gold standard mechanism for the increase in teacher uptake, meaning they’re pulling an idea back from the student, repeating it, affirming it, and building on it high-quality discourse in the classroom.

So to me, this is the opportunity that we have in AI. It is not actually about AI for students at all. And I think chatbots are bordering on a complete waste of time because their hallucinations are totally unreliable. They’re worse than calculators. Calculators don’t hallucinate. ChatGPT is not a reliable source of information right now. 

Somebody said this morning that there is no bias and it doesn’t care whether you are black or white. It absolutely cares whether you’re black or white because the programming in ChatGPT is known to be biased. It is known to be biased. And for most students, writing is a form of thinking. And if you’re taking away that mechanism for doing the thinking, how do you think we’re going to get more critical thinking out of students, if you basically de-skill them systematically from doing the thinking that is embedded in writing?

So I tell my students, you’re cheating yourself if you use ChatGPT. There is no good detector for it, and even OpenAI basically admits that. But I think it is a blind alley with respect to learning, but natural language processing, based on the same large language models, has enormous potential as a mechanism for empowering teachers and for giving teachers and educators, instructors, the feedback they need, and I have the data to prove it.

And that to me feels like the conversation that we need to be having about artificial intelligence and the future of learning and how we can double down on the unique human qualities that teachers have, on the impact that they have on us, on the impact that our classmates have on us, and avoid the destruction and the faddishness that you saw in the personalized learning movement that can easily happen with educational AI. And that would — the real risk there is that we throw out the whole thing and we lose the potential, the enormous potential of technologies like this to improve learning systematically.

So with Dora and our colleagues, we’re about to publish a white paper that is focused on how to empower educators with AI. And what we’re laying out is a set of principles and a set of possible strategic directions that are informed by the educational research, that are not driven by commercial hype from AI companies that know nothing about teaching and should not be trusted with your student data, but are driven by what we know about learning.

Driven by the cognitive science, the computer science, the learning science. This is a group of faculty, not just at Stanford, but from other institutions that are going to put this white paper out. And I will circulate it to this community when it is published. But the basic idea is that we need a set of principles to start with and we need a set of directions that we can agree upon.

So this was the starting point that we came up with in a conference a few months ago. It is an initial starting point, and I hope in the discussion we can delve into what we might have missed here, because I don’t think it is sufficiently, I don’t think it’s comprehensive yet. I don’t think it’s global necessarily. And I think the AI considerations, you know, in authoritarian states may be different than in, you know, different kind of states.

So, you know, it’s worth thinking about different political contexts, different educational contexts when it comes to this. Begin with equity means that we need to be invested in AI that raises the floor, not just focuses on the ceiling, that we need to be thinking like Rachel Lotan does in her research about what is the bias in a classroom for who gets how much air time, and how do we use the AI to rebalance class time and classroom participation.

And that is not going to happen if we don’t design for it, if we don’t design to mitigate some of the biases that Apoorv [Bamba] was talking about this morning. That, to me, is a critical educational goal that is not going to come from commercial AI. That has to be driven by the educators. We have to demand it, and edtech companies need to build for a feature, meaning to set a feature and set of goals around educational equity. The floor, not just the ceiling. 

Centering teacher needs. I’ve already talked a little bit about that. To me, it goes beyond just professional development, though. Like there’s all sorts of busywork that teachers have to do that could effectively be supported by AI. And I am excited about the potential, for example, you know, of AI’s to do machine grading when it’s possible.

There’s a company called Gradescope, it was a student project at the University of California at Berkeley, that has been really effective in kind of a human AI combination that makes grading not just of multiple choice assessments, but also of open-ended items much more effective. And that to me is a model of what’s possible in this space. Can we relieve the burden on teachers so they can spend more time doing the higher value work with their students, spending more time caring for their students, listening to their students, helping their students form an educational identity? 

Promoting high-quality instruction. I mean, that’s basically what I was talking about before. We know a lot about what high-quality instruction looks like and it is incredibly rare to find. And this to me feels like the professional development mandate that we all have, to build the tools that scale high-quality instruction. Most of us get to experience it, if you’re lucky, once in your lifetime, close your eyes and picture the great teacher that you had, picture of the great classmate that you had who contributed to your learning.

How do we get that available to more people? That to me seems to be the goal. John, you were talking about going to the Stuyvesant School and learning with a great classmate, Eric Holder — 50 years later, that person still in your mind, the impact that they had on you. How do we give experiences like Stuyvesant to more for students? That is an absolutely critical question. I think for us, if we want to democratize high-quality education, make teaching like that and classmates like that available to more people. 

And build and inform educational theory. So this is the role of a research university, I think, and we have many research universities represented here. And I think what’s so exciting about a project like Code in Place is that it was simultaneously an educational outreach project. I lead the Digital Education office, right? So we’re supporting this to democratize learning. But it was a research project on the back end and the students didn’t know that. They didn’t need to know that. You know, they opted in, actually, to a research program. The IRB required it. It was supervised research. But to me, finding those synergies that both advance opportunities to learn but also advance the research on the back end.

There was a massive data collection that happened in that project that could not otherwise have been done. So it was advantageous both for advancing the research agenda for Dora and her team in the education school at Stanford and for democratizing learning. And I hope we can develop a model for new kinds of research-practice partnerships that bring in edtech, where a lot of that data now lies.

And think about instead of a researcher and a practitioner, it’s actually more of a triangle in which there’s an edtech company that has the data. There’s a school that’s implementing some intervention, and then there’s the researcher, and the three of them have this this new kind of trilateral collaboration. RPIP is a new acronym that I see coming in to form research-practice-industry partnerships, and the three of them each might have a role to play.

So we might end up with something that looks like this virtuous cycle of improvement. It looks like we lost the text on the top right. But cognitive science like Rachel Lotan’s research getting translated into teaching and learning test-beds like Code in Place, implemented in that, data-driven tools like the Empowered and TeachFX tools that give feedback to educators to try to improve their performance.

And then that informs the data science research, which, you know, faculty like Dora and Chris Piech, they depend on high-quality datasets in order to get these analyses, in order to publish their papers and advance what we know about how people learn at all. So I’ll close there. Thank you so much for your engagement and I look forward to the questions that come in the next session.