Embracing the DARPA Model for EdTech Innovation: Charting the Course with GPT-4 and Beyond

By: Russell Shilling, Ph.D.

“GPT-4 provides an excellent example of the need to employ additional research and development

The recent discussions about GPT-4 and related artificial intelligence technologies in recent months have been interesting, with some advocating bans in the classroom while others see opportunities to improve education. However, both positions rely primarily on opinions and anecdotes about a software platform not developed to educate. What is needed are concentrated evidence-based research efforts to explore whether and how educators can effectively use these new applications. GPT-4 and its rapidly evolving relatives are potentially disruptive technologies. GPT-4 provides an excellent example of the need to employ additional research and development models for education technologies, especially where artificial intelligence is concerned. We must test best practices for this technology in the classroom and study algorithmic effectiveness, potential bias, and other limitations and quirks. Fundamental research models typically used in education will not adequately keep up with the pace of development, innovation, and adoption.  We must adopt research models that increase the likelihood of effective technologies reaching scale and providing the equitable solutions sought by educators.

What is the DARPA Model?

During my military career, I was a program officer at the Defense Advanced Research Projects Agency (DARPA), managing programs in education, training, and psychological health. DARPA is a military research agency tasked with rapidly creating innovative new technologies by investing in high-risk, high-reward programs that push the boundaries of current scientific understanding. DARPA identifies ambitious goals or “moonshots” and creates programs to reach them in 3-4 years. In addition to major military innovations for stealth and hypersonics, DARPA programs have revolutionized our lives by developing critical technologies for the internet, mRNA vaccines, GPS, speech recognition systems, artificial intelligence, and much more. Many in the education community have been advocating for an organization that uses these same approaches to improve education outcomes for all students while radically improving equity and inclusion.

Applying the DARPA Model to Education

I’ve helped philanthropies think about how to adapt DARPA approaches for creating higher-risk, high-impact programs with an eye on equity, inclusion, and scaling. I was an early advisor for the Advanced Education Research and Development Fund (AERDF) and continue to advise on other programs currently in development. We are currently applying this model to a Gates Foundation program to improve measurement in PreK-3. I’ve also advocated for this approach in government programs. I am pleased with the recent announcement that the Institute for Education Science (IES) at the U.S. Department of Education will launch a pilot ARPA-ED program.

Let’s briefly explore how this can work in education. First, the DARPA model is not a panacea or a replacement for traditional research programs but a powerful alternative. In Pasteur’s Quadrant (1997), Donald Stokes proposed a four-quadrant research model that balances the pursuit of fundamental understanding and practical application. The four quadrants are:

1. Bohr’s quadrant – Named for Niels Bohr, research focuses on pursuing fundamental knowledge and theory with little regard for practical applications.
2. Edison’s quadrant – Named for Thomas Edison, creates practical applications with little concern for advancing fundamental knowledge.
3. Pasteur’s quadrant – Named for Louis Pasteur, this quadrant conducts fundamental research towards a specific application, often resulting in significant societal impact.
4. Sine Nomine – Stokes did not name this quadrant, but it includes routine data collection activities such as standardized testing, replication studies, field guides, etc.

First, DARPA programs uniquely live in Pasteur’s Quadrant since we often do not entirely understand how to create the proposed application. The other three quadrants, especially fundamental research, are all required for a healthy research ecosystem that feeds directly into Pasteur’s Quadrant. Fundamental research uncovers new insights into pedagogy, equity, and technology that identify and refine educational requirements. Second, no single research design or structure defines a DARPA program; instead, the program design is created to maximize success. A well-designed program often succeeds by producing highly impactful related discoveries, even when the primary goal is not achieved.

There are best practices that help maximize impact, especially in education and the social sciences. In short, funding agencies and developers must clearly define the problem they are trying to solve using a prescribed process. Once the problem is thoroughly defined, there are specific requirements for recruiting program officers, employing multidisciplinary teams, iterative development, and flexible program management. Let’s look at these processes in greater detail.

Recruiting Expert Program Officers 

Recruiting program officers who are experts in the problems to be solved is essential. Unlike many government agencies, DARPA Program officers are unique in that they are term-limited and have relatively small portfolios to allow them to give each program their focused attention. DARPA recruits top experts and innovators as program officers. In many ways, the program officer is a co-principal investigator, frequently interacting with their team as another subject matter expert. In education, program officers should be domain experts with a good understanding of the classroom, pedagogy, equity, inclusion, and technology. For a program looking at the impact of GPT-style applications, funders should recruit program officers with highly specialized knowledge about education and artificial intelligence technologies coupled with the ability to recruit world-class talent to innovate and create practical applications.

Identifying the Challenge to be Solved

At DARPA, before any program is approved, the program officer must answer seven questions, aka Heilmeier’s catechism, to define the potential program. When submitting a proposal to research programs, I often encourage researchers and developers to follow this outline.

1. Using no jargon, describe what problem you are trying to solve.
2. How is it done today, and what are the limitations of the current approach?
3. What is new in your approach, and why do you think it will be successful?
4. If you are successful, what difference will it make?
5. What are the risks, and how will you mitigate them?
6. How much will it cost, and how long will it take?
7. What are the midterm and final “exams” to check for success?
   

I would also add two additional questions to the catechism for education programs:

1. How does this solution scale while meeting the diversity and inclusion needs of a broad range of students?
2. What are the ethical risks, including equity, privacy, and other sensitive issues?

Especially for education, this process assures that the program is rooted in practical solutions, is as clearly understood as possible, and demonstrates a clear understanding of the plan and associated risks.

Creating Multidisciplinary Teams 

To solve the defined problem, an excellent way to spur innovation in education is to create multidisciplinary teams, including expertise outside the traditional education community. Multidisciplinary teams are often an eclectic mix of researchers, developers, and practitioners. Education programs should always include educators and other stakeholders to help ground the program in developing practical and valuable solutions for the classroom. For my DARPA programs, I recruited a mix of academics and experts, including those who were world experts on a specific technology but had not yet applied it to the defined problem. The result is a broadening of perspectives from all involved.

Iterative Development & Program Flexibility

As stated earlier, DARPA-like programs in education live in Pasteur’s Quadrant, a process that involves conducting fundamental research toward an applied goal. At the program’s start, there may not be a clear pathway to solving the defined problem because the program goals are pushing the boundaries of what is known. The only successful way for this process to succeed is through an iterative or spiral development process of discovery, integration, and rigorous testing. At each iteration, the project’s effectiveness and scalability are evaluated. Program officers eliminate elements shown not to be effective or scalable while adding new capabilities and approaches as they are identified. In this process, it is not uncommon for the program’s goals to shift as ongoing research results clarify both the problem and the solution; the focus should be maximizing impact. P This program flexibility includes the ability to add or remove components of the program as new knowledge is gleaned instead of maintaining a fixed plan. In fact, programs are terminated early if they are shown to be nonviable.

Challenges for Implementation

This discussion is only an outline of the framework but a good starting point for applying a DARPA-like process in education, especially to develop potentially disruptive applications of technologies like GPT-4. However, there are challenges to this model in practice. For example, DARPA is successful, but it is also incredibly well-funded. This level of funding allows the agency to take considerable risks while still demonstrating high impact. DARPA programs are said to fail to reach all their primary objectives 85%-90% of the time. Although DARPA-like programs can be designed to reduce this level of risk, there is an understandable reluctance to apply this model widely in areas with less funding. The advantage of the DARPA approach is that programs that “fail fast” allow budgets to be allocated more efficiently with reduced waste.

Some educators will also argue that many reforms in the classroom do not require disruptive innovation but require more pragmatic, straightforward policy and education reforms. They are correct. Using the DARPA model to improve education outcomes is only one of many approaches we need as we move forward. 

Finally, pushing boundaries requires constant attention to the ethical implications and unintended consequences of what we are developing regarding privacy, security, equity, and impact.

These inherent risk factors, along with philosophical differences between traditional research models and a DARPA model, challenge organizations trying to implement both. When making funding decisions, it’s challenging to make a 1:1 comparison between lower-risk fundamental research and these higher-risk, potentially higher-complexity programs. 

Conclusion

The challenges to adopting a DARPA model in education R&D are not insurmountable but will require planning and constant attention. Confronting these challenges will be worth the results. By investing in rapid-cycle high-risk, high-reward programs that push the boundaries of what is possible, we can develop new, highly effective education technologies and approaches that scale. These results will potentially transform how we teach and learn, improve equity and inclusion, and prepare students for the ever-changing requirements of the 21st century.