Reviewing 44 Courses Against a Deadline, and Deciding Where AI Belonged

I started with a work problem.

I had 44 technical training courses to review before the end of the fiscal year, and each one needed a written review document. I had part of my time available for the assignment. There was no template or existing workflow, and no one who had done it before could hand me a process to follow.

Before I could do much with the courses, I had to figure out what I was supposed to produce.

Understanding what the review needed to do

I started by working out what a finished review needed to accomplish. Once I understood the deliverable, I could work backward through the steps needed to create it.

The reviews would become part of the organization’s records. I wouldn’t necessarily know who read them or what happened after I turned them in, so the documents needed to make sense to someone who might never have the opportunity to ask me a follow-up question. That affected both what I included and how much of my reasoning I needed to explain.

Building the process

I reviewed each course from three perspectives, taking them one at a time:

• As a learner, could I follow the material and come away able to do what the course was supposed to teach?
• As a training specialist, did the objectives, course structure, and assessments support the intended learning? Did they align with one another?
• As an auditor, could I find enough in the record to understand why the course was designed the way it was?

I also recorded myself talking through each review. Explaining what I was seeing helped me notice things I might have passed over while reading silently. It also gave me a transcript of my reasoning for each course, which became useful in a way I hadn’t anticipated.

I built the template and the workflow as I went, refining both as I learned what the reviews required. After a handful of courses, I had a process I could repeat. But each review document was taking roughly two to two and a half days, and I had 44 to complete with only part of my time available.

At that pace, I wasn’t going to meet the deadline.

Figuring out where AI could help

That was when I began looking at AI as something I needed to make the assignment workable. I already understood the review process because I’d been doing it myself. I could look at the individual steps and consider which ones AI might help with, along with what I would have to do to check its work.

The order mattered. Building the process by hand gave me something concrete to evaluate. It also gave me a basis for recognizing when an AI draft wasn’t doing what the review needed to do.

AI use was encouraged, but the guidance available to me was mostly about restrictions: which tools were approved and what information could go into them. Those were necessary boundaries. I still needed to work out how to apply them to this particular assignment.

The rules drew a clear line around the course content itself: that stayed with the in-house model. What they didn’t say was how to treat everything around it, including my own notes, my narrated reviews, and my reasoning about each course. So I had to work out, task by task, which parts of the job actually touched the content and which didn’t.

The in-house model handled the work that did. For most of the drafting, I used a frontier model, working from my review notes and transcripts rather than from the courses themselves.

I didn’t have someone positioned to review and approve that judgment. I had to make the decision based on the guidance available to me, and I understood my approach to be within what the rules allowed.

That exposed a gap in how the guidance reached the work. The people setting policy had established broad boundaries, but I still had to decide what those boundaries meant for a particular set of notes and a particular drafting task. There wasn’t a route for getting that decision resolved, or for sending what I was learning back to the people responsible for the guidance.

I don’t see that as a failure of any one person. I see it as something the governance process needed to account for. If the people writing the rules don’t hear about the decisions workers are having to make, they have little opportunity to clarify the guidance where it’s needed. The restrictions can be clear while the person trying to apply them still has unanswered questions.

Drafting, checking, and keeping a record

I gave the AI my notes and transcripts and directed it to draft from those sources. I wanted the findings in each review to come from the review I’d actually performed. A finding that sounded reasonable but wasn’t supported by my observations could mislead the person reading the document, especially if nothing in the wording gave them a reason to question it.

At the time, I wasn’t choosing a named technique. I was trying to solve that problem: keep the model from inventing findings by grounding it in my own source material. I later learned that the idea already had a name and a standard form — retrieval-augmented generation, or RAG — but what mattered to me during the assignment was having source material I could check the draft against.

I checked every claim in every document against its source, corrected what needed correcting, and approved the finished review. That checking took a substantial part of the time. It was also necessary for me to take responsibility for what I delivered. Having a person review an AI draft only means something if that person does enough review to determine whether the claims are supported.

I kept the supporting record outside the final documents. My notes and the drafting record show which parts came from my review, which came from the AI draft, and what I changed. Keeping that material separate let me preserve the evidence without making the finished reviews harder to read.

Work I kept for myself

I did the source-document searches by hand. For each governing document, I needed the most current and relevant version. Having the model search would still have left me checking whether it had found the right document and the right version. In this workflow, I could do the search myself in less time than it would take to verify its results.

Some courses also included quotations without identifying the source document. Tracing those passages required me to decide where they might have originated and then investigate. I kept that work because a plausible answer wasn’t enough. I would still have needed to establish that the proposed source was the actual source, and I wasn’t comfortable relying on a model’s guess to make that judgment.

What I can say about the results

I delivered all 44 reviews about three weeks before the project deadline.

My observed pace went from roughly two to two and a half days per review document at the start to two or three documents per day by the end. That’s a description of my own throughput, with an important limitation: I was also getting better at the review process as I went. I can’t separate how much of the change came from AI and how much came from practice. It wasn’t a validated organizational measure.

Finishing early tells me that I completed the assignment within the time available. It doesn’t establish the quality of the documents, and I don’t have feedback on how they were used after delivery.

The framework I developed through this work later became a conference session, Use AI Effectively & Safely in High-Risk Environments, at The Learning Guild’s AI & Learning Design Online Conference. That gave me an opportunity to share the reasoning with other practitioners. It doesn’t provide an independent assessment of the course reviews themselves.

What I’d carry into the next assignment

I’d start by understanding the work and building enough of the process myself to know what a useful result looks like. Doing that made it possible to identify where AI could help and what I needed to verify before using its output.

I’d also include the checking time when deciding whether a task was worth giving to AI. Providing source material gave me a basis for verification, but a draft could still misread that material or rely on something outdated. For the document searches, doing the work myself was the better use of my time.

The assignment also changed the questions I was asking about governance. At first, I wanted to understand the boundaries so I could finish the work within them. As I went, I began thinking about what the guidance needed to address for someone doing this kind of work. I’d had to make decisions that the available rules didn’t resolve for me.

Those decisions are useful feedback. I’d want the people responsible for the guidance to see where I had questions, how I arrived at an interpretation, and what I had to check. The next person working through a similar assignment should have more to go on than I did.