Every course on Artemis, the learning platform many TUM courses run on, can describe what students should learn as a set of competencies. Exercises and lecture units get linked to those competencies, and from those links Artemis calculates each student's progress and mastery and suggests what to study next. The trouble is that course content keeps changing and the links don't. Nobody gets an error when a link goes stale. The progress bar is just quietly wrong.
My bachelor thesis, Agentic Orchestration for Autonomous Competency Management in Artemis, builds a system of LLM agents that notices those content changes, reconsiders the affected links, and applies its changes without asking an instructor first. That last part is the uncomfortable one. Getting an LLM to suggest a link is easy. Letting it write to course data that students see is a different problem.
The split I landed on is simple to state. The model decides what should change. Artemis decides how much the model may do, where it may do it, and when a run counts as done. Every safety property lives in Java rather than in a prompt. Whether that holds up, and what it costs, is the rest of the story.
Why stale links matter
Artemis's competency features live in a module called Atlas. A competency is an intended learning outcome with a Bloom level, and a competency link connects it to an exercise or a lecture unit. Links are the raw material of the analytics. An exercise without a link contributes nothing to the competency it actually trains. A link that survived a rewrite sends student activity to a competency the exercise no longer covers.
The stakes are real. In a study of a course with 1,278 students, Anzinger et al. (my supervisor is the lead author) found that students who used Atlas voluntarily showed higher mastery progress and passing rates than those who didn't. The authors are careful to note that the observational design doesn't prove causation. But if students rely on those numbers, the numbers have to follow the course content.
Before my thesis, Artemis already had three kinds of AI help for competency links: a conversational Atlas Chat, AtlasML suggestions in the exercise editor, and a checklist in the Hyperion exercise-authoring module. All three waited for an instructor to ask. The editor suggestions and the checklist looked at one exercise at a time, none of them handled lecture units, and none reconsidered links when content changed.
How I ended up with this topic
My supervisor, Maximilian Anzinger, had been my team lead in the iPraktikum. When I asked him about a thesis topic, he had this idea and left the design almost entirely to me. My examiner was Prof. Dr. Stephan Krusche, whose research group builds Artemis. I got a real production codebase, a real problem, and no one telling me which architecture to pick.
I was already working in the Atlas code before the thesis officially started on 27 May 2026. One of my first contributions replaced the Atlas Chat's string-based routing, where the main agent literally wrote %%ARTEMIS_DELEGATE_TO_COMPETENCY_EXPERT%% into its reply to hand off work, with proper Spring AI tool calls (#12373). The marker approach is a small example of an application trusting free text from a model. Avoiding that became the thread through the whole design.
The scary part is the autonomy
An approval dialog hides many problems, because a human catches the weird cases. Once changes go straight to the database, every failure mode that dialog used to absorb becomes your responsibility:
- Risky tool use. Research like ToolEmu shows that tool-using agents take risky actions when instructions are underspecified, including the safest agent the authors evaluated. A prompt that says "only touch this course" is a hope, not a guarantee.
- Concurrency. Artemis runs on several servers. Two content changes in the same course can start two runs that write conflicting changes.
- Cost loops. Instructors save often. If every save triggers a model call, a busy afternoon becomes an invoice. If the agent's own writes trigger new runs, it becomes a loop.
- Provenance. If an instructor can't tell which links an AI created, they can't review them later.
So I set one rule for the whole design: anything that has to hold goes into application code. Prompts shape the quality of decisions. They never carry safety.
The model decides what, Artemis decides everything else
A maintenance run starts from one of two places. An instructor can trigger it for an exercise or lecture unit. Or, in courses where an administrator has switched it on, Artemis collects content changes and starts a batch once the course has been quiet for a configurable window. Either way, the run goes through the same pipeline.
Inside the run, an orchestrator works in a ReAct loop. It starts with the changed content and a compact index of the course: every competency, its linked exercises and lecture units, its relations, and a separate list of exercises that have no link at all. When it needs more, it asks for full details on specific items. This progressive disclosure keeps the prompt small while still letting the orchestrator ask for anything in the course.
The model decides what
Artemis enforces how much, where, and when it counts as done
Architecturally, this is an orchestrator-worker multi-agent setup built on Spring AI tool calling: every tool is a Java method that Artemis executes, while the model only names the tool and its arguments. The orchestrator never writes anything itself. It delegates to three workers, each with its own prompt and a deliberately small tool set. The Assigner links, unlinks, or reweights exercises and lecture units. The Creator creates competencies. The Editor changes a competency's title, description, or Bloom level, and may delete one only when no links remain. Workers don't see the orchestrator's conversation, keep no memory between tasks, and can't delegate further. Every extra delegation costs another model call, but that isolation makes each worker's scope easy to reason about.
Three guardrails I'm proud of
Flavor stripping that can only delete
Exercise statements love a story. At TUM, a surprising number of exercises involve penguins who need help. That narrative is good teaching, but it's noise when you're deciding which competency an exercise trains, and it costs tokens on every run. So a pre-pass removes this flavor text before the orchestrator sees the content.
The obvious approach is to ask a model to rewrite the statement without the story. I didn't, because a rewrite can quietly change a method name or a required output format, and nobody would notice. Instead, the model returns a list of search-and-replace edits, and Artemis applies them itself. Below are an illustrative example of that format and the loop that applies it (trimmed, comments mine):
1// Input statement (illustrative)2The zoo penguins have lost track of their fish supply. Help them out!3Implement sortByWeight(List<Fish>): List<Fish> so that ...45// What the flavor-strip model returns6{7 "edits": [8 {9 "reasoning": "Story framing; the next sentence is the spec.",10 "search": "The zoo penguins have lost track of their fish supply. Help them out!",11 "replace": ""12 }13 ]14}1for (EditDTO edit : edits) {2 String replacement = edit.replace() == null ? "" : edit.replace();3 if (replacement.length() > 1) {4 continue; // a rewrite, not a deletion: never trusted5 }6 int[] span = findSpan(working, edit.search()); // exact, then whitespace-tolerant7 if (span == null) {8 continue; // off-target span: skip, don't guess9 }10 working = working.substring(0, span[0]) + replacement + working.substring(span[1]);11}An edit whose replacement is longer than one character is thrown away. An edit whose span can't be found (ignoring whitespace differences only) is skipped. If the call fails or the result is empty, Artemis falls back to the original text. The model can delete narrative, but it has no way to rewrite the specification: every word it keeps stays exactly as written, and Artemis only tidies whitespace afterwards. Even the field order is deliberate. reasoning comes before search, so the model has to write down why a passage is narrative before it commits to deleting it.
A budget the model can't talk its way around
An agent that loops until it feels done is a cost risk with no upper bound. Each run gets one shared budget of 256 tool calls across the orchestrator and all its workers. Reads count. Failed calls count. The slot is reserved before the tool runs, so concurrent calls can't overspend it, and delegating to a worker doesn't reset the counter. Separately, a run can delegate at most 16 times.
After 224 calls the run switches into wrap-up mode: writes and delegations are rejected, and only reads and completion are allowed. That reserves the last 32 calls for the agents to look at what they did and report honestly. The limit is plain Java:
1/** Maximum number of autonomous tool callbacks in one top-level run. */2public static final int LIMIT = 256;34/** Work stops here; the remaining callbacks are reserved for verification and completion. */5public static final int WRAP_UP_AT = LIMIT - 32;"Done" has to be earned
A subtler failure than a wrong write is an agent that says it finished when it didn't. So the orchestrator can only report a verified success after a fresh read of the course index that happened after its last change. The index in the original prompt doesn't count. Here's the check that enforces it:
1public synchronized void complete(boolean verified, String message) {2 if (completion != null) {3 throw new IllegalStateException("completeOrchestration was already called.");4 }5 if (activeWork != 0) {6 throw new IllegalStateException("Wait for active work before completing orchestration.");7 }8 if (verified && (verifiedVersion != mutationVersion || workBlocked() || exhausted())) {9 throw new IllegalStateException(10 "verified=true requires a successful index refresh after the latest work, "11 + "with no budget-blocked work.");12 }13 completion = new OrchestrationCompletionDTO(verified, message);14}If that check fails, or the orchestrator never completes, the run ends as Failed, or as Partial if some changes already landed. Artemis doesn't pretend a partial run was clean: each result lists every change applied before the failure. Around these three guardrails sit the more conventional ones:
- Scope and concurrency. The active course comes from trusted server context, never from model output, and a read for an object outside the course returns nothing. A per-course claim in Artemis's shared state (Hazelcast or Redis) turns a concurrent request into an In progress answer instead of a second run.
- Instructors and administrators stay in charge. Automatic mode is off by default. Administrators enable it per course and set the inactivity window and a daily run limit. Changes to competency links never trigger a run, so the agents can't set themselves off.
- Provenance and reporting. Every competency and link an AI feature creates is flagged as AI-generated. The flag is recorded, not yet shown in the UI. Instructors get a notification for every automatic run that changes something or fails, and can opt into an email that lists up to 50 applied changes with the model's reasoning for each.
Shipping into a codebase that isn't mine
A thesis prototype on a fork would have been easier. I wanted the work in upstream Artemis instead. Since March 2026 I've opened 22 pull requests against it. 17 are merged, including flavor stripping, cost tracking, the automatic trigger, per-course configuration, the orchestrator itself, and the shared tool budget. Five are still in review: worker delegation, the AI-origin flags, lecture-unit support, a hardening pass, and the email reports. The complete autonomous path isn't fully upstream yet.
Contributing to a codebase this size shapes your work in ways that don't come up on a solo project. Review takes time: splitting the tool services into per-role tool surfaces was a roughly 5,000-line diff that sat in review from late June to late August. The test infrastructure has budgets of its own. Artemis limits how many Spring server contexts its integration tests may start, and one of my orchestrator tests pushed it past ten. The fix (#13051) was mostly a 27-line deletion. That kind of constraint makes you design for someone else's CI, not just your own laptop.
Building agents with agents
I built this with AI coding agents, and the thesis says so in its AI statement. Claude Code and Codex generated most of the code and most of the first manuscript drafts under my direction. Agents also generated the synthetic course I benchmarked against. I made the design decisions, reviewed and tested every change, and checked every citation against its source.
Working this way moves the bottleneck from typing to judgment: knowing what to ask for, noticing output that looks plausible but is wrong, and building checks instead of trusting myself. I ran structured audit passes over every chapter and over the citations, and treated their findings like review comments to accept or reject. It's the same philosophy as the system itself: let the model produce, and make the verification deterministic.
The benchmark, or: one more thing
My supervisor and I agreed on a cost reference. We assumed that manually creating three or four competencies takes a tutor at least an hour. At Germany's 2026 minimum wage of €13.90, rounded up, that sets a ceiling of €15 per run. That was the bar an automatic run had to clear.
The fixture is a synthetic course with 60 exercises across all five exercise types, 12 lecture units, and 12 reference competencies over 12 introductory computer science topics. For each primary run, a benchmark runner created a fresh course through Artemis's normal APIs, set up one of six conditions, applied the content changes, and recorded the result. The conditions cover a single exercise, a single lecture unit, a fully maintained course where nothing should change, assignment to existing competencies, a half-missing competency set (mixed), and a bootstrap that starts with no competencies at all. Six conditions times ten repetitions, in randomized order, gave 60 runs. Each bootstrap course then continued for three more batches, adding 30 runs, for 90 in total, all on gpt-5.6-luna. Every provider request went through a local meter, so every cost figure can be recomputed from the published data.
Provider cost per maintenance run, in euro cents
90 runs, one dot each. Lines span the cheapest to the most expensive run, diamonds mark the median.
Stages 2 to 4 continue the stage 1 course
Tap a dot for its exact cost.
Target: 1,500 ct (€15, one tutor hour). Most expensive run: 5.36 ct.
Show the numbers
| Group | Min | Median | Max |
|---|---|---|---|
| Single exercise | 0.20 | 0.37 | 0.59 |
| Lecture unit | 0.22 | 0.45 | 0.69 |
| Maintained course | 0.63 | 1.16 | 1.78 |
| Assignment | 1.52 | 1.91 | 2.56 |
| Mixed | 1.74 | 2.06 | 2.40 |
| Bootstrap, stage 1 | 2.04 | 2.84 | 2.95 |
| Bootstrap, stage 2 | 3.10 | 4.00 | 5.36 |
| Bootstrap, stage 3 | 3.27 | 3.73 | 4.05 |
| Bootstrap, stage 4 | 2.11 | 2.28 | 3.02 |
All 90 runs completed: 68 made changes and 22 decided nothing needed to change. The most expensive single run cost about €0.054, or 0.36 percent of the target. Building a complete competency structure for all 72 learning objects of the course from scratch, across all four bootstrap stages, cost at most €0.14. The whole campaign cost €1.88, and every one of its 2,475 provider requests was accounted for.
The breakdown was more interesting than the total. Orchestration accounts for 49 percent of the cost, while the workers issue the most requests. Cache reads make up 79.8 percent of all input tokens but only 12.7 percent of the cost, so repeated context is cheap and newly written context and reasoning output are what you pay for. Looking isn't free either: runs on an already correct course made about three quarters as many read calls as runs that had to assign links, and cost about 60 percent as much without writing anything. No run used more than 127 of its 256 tool calls.
What I did not prove
The benchmark measures cost and execution behavior. It does not measure whether the agents' decisions are good. Some outcomes were very consistent, for example twelve new competencies in every first bootstrap stage and six in nine of ten mixed runs. Others weren't: finished bootstrap courses ended with 16 to 28 competencies. And consistent wouldn't mean correct anyway. Judging that needs instructors reviewing real course material, and it's the first thing I'd do next. Other known gaps:
- Flavor stripping is structurally safe, but not validated for meaning. The model can't rewrite the specification, but nobody has checked yet whether it ever deletes something a competency decision needed.
- The course claim can be replaced after 30 minutes without stopping the original run. I found this near the end of the implementation period in my own audit. In that case, two runs could write to the same course at once. The longest benchmark run took 341 seconds, so an overlap needs a run more than five times longer than anything I measured. It's in the thesis with the exact failure mode and is first on the fix list.
- Provenance still trusts the client in places, and instructors can't see it yet. Some endpoints set the AI flag based on which endpoint the client chose, not on server-side evidence, and the flag isn't displayed in the UI.
- Prompt-injection protection is partial. Course text is sanitized in the main prompt paths, but not everywhere it reaches a model.
- Some fences were barely exercised. No run reached wrap-up mode or the budget limit, and only two runs delegated to the Editor, so those paths are tested but not measured. The benchmark course was also written with the same model (
gpt-5.6-luna) that was benchmarked, which may favor content it handles well.
A system that writes without approval should be judged on its weak spots, so I'd rather name them myself.
Where it stands
During my defense, I ran a live maintenance run. Early in the talk, I edited a prepared exercise to start it, explained the architecture while it worked, and came back near the end to inspect the result. It had finished. A live agent in a thesis talk is a gamble, so I had planned what to say for every outcome, from still running to partial.
The thesis is submitted and I'm waiting for the grade. The merged parts are in upstream Artemis, and the remaining pull requests are in review. The main lesson carries over to anything with a model in the loop: the useful engineering in autonomous agents is the fence, not the agent. The model decides what should change. The application decides how much, where, and when it counts as done. I took the same approach, at a much smaller scale, for the self-maintaining Apple Wallet support tracker.
Resources
- Artemis on GitHub, the open-source learning platform
- My Artemis pull requests, merged and in review
- Published benchmark evidence and the offline analyzer for reproducing every cost figure
- Anzinger et al., Investigating Student Interaction with Competency-Based CS Education

