PaxLabs Research
Andres G. and the PaxLabs Research Team · PaxLabs Inc.
The Reproducibility Crisis in Agent Evaluation: Why Current Benchmarks Mislead
Andres G. and the PaxLabs Research Team PaxLabs Inc. Correspondence: legal@paxeer.app
The rapid proliferation of autonomous agent benchmarks has created an illusion of scientific progress in agent capability measurement. We argue that the dominant evaluation paradigm, which reports single-run task completion rates on static benchmark suites, systematically misleads the research community in five distinct ways. First, single-run scoring ignores run-to-run variance, collapsing stochastic agent behavior into a single number that may not be representative. Second, task completion is measured without verifying fidelity to the stated intent, rewarding agents that reach correct outcomes through incorrect reasoning or unintended side effects. Third, no existing benchmark measures the cost of structured repair, treating all successful completions as equivalent regardless of the number of errors, rollbacks, or human interventions required. Fourth, provenance auditing is absent, making it impossible to verify whether an agent's trajectory is reproducible or was the product of favorable environmental conditions. Fifth, benchmark contamination and saturation erode the validity of comparisons over time. We propose a reproducibility-first evaluation framework that addresses each of these failures through formal protocol design, seed-bound execution environments, multi-run variance reporting, intent fidelity scoring, repair cost measurement, and replay integrity verification. We define concrete metrics, specify a runnable evaluation protocol, and discuss the limitations of our approach.
Agent evaluation, benchmark reproducibility, evaluation methodology, language model agents, software agents, reproducibility crisis.
The past two years have seen an explosion of benchmarks designed to evaluate autonomous agents powered by large language models (LLMs). GAIA [1], SWE-bench [2], AgentBench [3], WebArena [4], and WorkArena [5] each propose distinct task suites and scoring methods, and their leaderboards have become the primary currency by which agent systems are compared. We observe that the field has converged on a common evaluation pattern: define a set of tasks, run each agent once (or a small number of times) per task, and report the fraction of tasks completed successfully.
This pattern is appealing in its simplicity. It is also, we argue, systematically misleading.
The core problem is not that existing benchmarks are poorly designed in isolation. Rather, the dominant evaluation paradigm fails to capture the properties that matter for real-world agent deployment. An agent that completes a task 60% of the time on a single run is fundamentally different from one that completes it 60% of the time on average but with high variance (sometimes 90%, sometimes 20%). An agent that reaches the correct answer by a circuitous route with three failed attempts and two rollbacks is not equivalent to one that reaches the same answer directly. An agent that produces the right output but through reasoning that contradicts the task specification is not the same as one that respects the stated intent.
We identify five specific failures in current evaluation practice. We then propose a reproducibility-first framework that addresses each failure through concrete protocol changes and new metrics. Our goal is not to replace existing benchmarks but to augment them with evaluation practices that produce honest, reproducible, and decision-relevant comparisons.
The contributions of this paper are as follows:
1. We identify five systematic failures in current agent evaluation practice, each supported by examples from published benchmarks.
2. We formalize a reproducibility-first evaluation framework with seed-bound execution, multi-run variance reporting, intent fidelity scoring, repair cost measurement, and replay integrity verification.
3. We define seven concrete metrics and specify a runnable evaluation protocol that other groups can adopt.
4. We discuss limitations honestly, including the additional computational cost of our proposed approach.
Agent evaluation has matured rapidly since the release of early benchmarks. We briefly survey the major benchmarks that define the current evaluation paradigm.
GAIA (General AI Assistants) [1] evaluates agents on 466 real-world questions that require multi-step reasoning, web browsing, and tool use. Tasks are designed to have unambiguous, verifiable answers. GAIA reports accuracy as the primary metric, computed over a single submission per task. The benchmark has three difficulty levels, and the authors acknowledge that current LLM-based agents significantly underperform humans.
SWE-bench [2] evaluates agents on their ability to resolve real GitHub issues from popular Python repositories. Given an issue description and a codebase, the agent must produce a patch that passes the associated test suite. SWE-bench reports the percentage of issues resolved, again on single-run evaluations. A variant, SWE-bench Lite, reduces the task set to 300 instances for faster evaluation.
AgentBench [3] provides eight distinct environments (operating systems, databases, knowledge graphs, card games, web shopping, and others) and evaluates LLMs as agents across these environments. AgentBench reports per-environment scores and an aggregate, using single-run evaluation in each environment.
WebArena [4] creates a realistic web environment with fully functional websites across four domains. Agents must complete tasks by interacting with these websites through a browser interface. WebArena reports task success rates, finding that the best GPT-4-based agent achieves only 14.41% end-to-end success compared to 78.24% human performance.
WorkArena [5] extends the web agent evaluation paradigm to knowledge work tasks on ServiceNow, evaluating agents on common enterprise workflows.
Other notable benchmarks include OSWorld [6] for desktop environments, and VisualWebArena [7] for tasks requiring visual understanding of web pages. Across all of these, the common pattern is single-run (or small-sample) task completion rate as the primary metric.
The broader machine learning community has grappled with reproducibility for over a decade. The NeurIPS reproducibility checklist, the ML Reproducibility Challenge, and numerous position papers have documented how small changes in random seeds, hardware, software versions, and data splits can produce substantially different results [8]. In reinforcement learning, the problem is especially acute: Henderson et al. [9] showed that deep RL algorithms exhibit high variance across runs, and that reporting a single best run is misleading.
For LLM-based agents, the variance problem is compounded by several factors. Language model outputs are sampled stochastically. Agent trajectories involve sequential decision-making where early stochastic choices cascade into divergent outcomes. Environmental states may vary across runs due to external factors (network responses, API rate limits, concurrent users). Despite these sources of variance, the dominant evaluation practice in agent benchmarks remains single-run or small-sample scoring.
Traditional software testing distinguishes between specification testing (does the output match the specification?) and implementation testing (was the implementation correct?). Current agent benchmarks focus almost exclusively on the former. An agent that completes a SWE-bench task by applying a patch that passes the tests, but that does so through an approach that introduces subtle bugs or violates code style conventions, is scored identically to an agent that produces a clean, correct patch.
We argue that intent fidelity, the degree to which an agent's approach aligns with the task specification's intended solution path, is an essential evaluation dimension that current benchmarks ignore.
We now describe each failure in detail, with examples and formal definitions where appropriate.
Definition 1 (Run Variance). Let A be an agent, T be a task, and R(A, T, s) be the binary outcome (success or failure) of running agent A on task T with random seed s. The run variance of A on T over seed distribution S is Var_S[R(A, T, s)].
Current benchmarks report E_S[R(A, T, s)] for a single seed (or a small, fixed set of seeds). When the variance is high, this single-sample estimate is unreliable. Consider two agents, A1 and A2, each evaluated on 100 tasks. Agent A1 succeeds on 60 tasks with probability 1.0 (zero variance). Agent A2 succeeds on 60 tasks with probability 0.6 and fails on 40 tasks with probability 0.0, but for those 60 tasks, it succeeds with probability 1.0 on 30 and with probability 0.6 on the other 30. Both report a 60% success rate, but A1 is deterministic and reliable while A2 is not.
In practice, we have observed run-to-run variance as high as 30 percentage points on individual tasks when evaluating the same agent with different random seeds. This variance is invisible in single-run reports.
Definition 2 (Intent Fidelity). Let spec(T) be the intent specification of task T (the set of constraints, preferences, and solution characteristics the task designer intended). Let traj(A, T, s) be the agent's trajectory on task T with seed s. The intent fidelity of this run is:
F(A, T, s) = |constraints_satisfied(traj(A, T, s)) intersect spec(T)| / |spec(T)|
Current benchmarks score only the final outcome: did the agent produce the correct answer or pass the test suite? They do not verify whether the agent's approach satisfied the constraints specified in the task.
Example. In SWE-bench, an agent might resolve a GitHub issue by deleting the failing test rather than fixing the underlying bug. If the scoring only checks whether the test suite passes, this agent receives the same score as one that correctly fixes the bug. In WebArena, an agent might complete a shopping task by adding items to the cart in a way that violates the specified quantity constraints, yet still arrives at a cart with the correct total.
We propose that intent fidelity should be measured alongside task completion. An agent that completes a task with low intent fidelity has, in a meaningful sense, solved the wrong problem.
Definition 3 (Repair Cost). Let fail(A, T, s) be the number of failed attempts, rollbacks, tool invocations that produce errors, or human interventions required before agent A achieves a successful outcome on task T with seed s. The repair cost is:
C(A, T, s) = { fail(A, T, s), if R(A, T, s) = 1 infinity, if R(A, T, s) = 0 }
Current benchmarks treat all successful completions as equivalent. An agent that solves a coding task on the first attempt and one that solves it after 47 failed attempts, 12 rollbacks, and a call to an external code completion service both receive the same score. In any real deployment, the cost of those failed attempts matters. They consume compute, they may produce side effects, and they indicate that the agent's reliability is low even when it eventually succeeds.
The Reflexion framework [10] is a step in the right direction, as it explicitly models the iterative refinement process. However, even Reflexion-based evaluations typically report only the final success rate, not the number of iterations required.
Definition 4 (Provenance Chain). For a run (A, T, s), the provenance chain is the complete, ordered sequence of (state, action, observation, timestamp) tuples that constitute the agent's trajectory, together with a hash chain that binds each tuple to its predecessor.
Current benchmarks do not require or verify provenance chains. This means that:
1. It is impossible to verify that a reported result is reproducible. 2. It is impossible to distinguish between an agent that solved a task through skill and one that benefited from favorable environmental conditions (e.g., a cached API response, a network timeout that triggered a fallback to a simpler strategy). 3. It is impossible to audit agent behavior for safety violations that did not affect the final outcome but occurred during the trajectory.
Without provenance, benchmark results are unfalsifiable claims. We cannot check them, replay them, or analyze the failure modes they represent.
The problem of data contamination in LLM evaluation is well-documented [11]. Benchmark cheating, where models are trained on evaluation data, has been shown to inflate reported performance across multiple benchmarks. When benchmark tasks (or closely related tasks) appear in training data, reported performance reflects memorization rather than capability. For agent benchmarks, the contamination problem is more subtle. The tasks themselves may not appear in training data, but the websites, codebases, and environments used in the benchmarks may have been crawled and included in web-scale training corpora.
Benchmark saturation is a related concern. As benchmarks become widely used, agent developers optimize specifically for the benchmark tasks and environments. This can produce performance gains that do not transfer to novel tasks. The pattern is familiar from computer vision, where ImageNet performance eventually decoupled from general visual understanding capability.
We do not claim that current benchmarks are fully contaminated or saturated. We claim that without contamination detection and periodic refresh mechanisms, we cannot assess the degree to which reported scores reflect genuine capability.
IV. A REPRODUCIBILITY-FIRST EVALUATION FRAMEWORK
We now propose a framework that addresses each of the five failures described above.
To address Failure 1 (variance), we require that every evaluation run is conducted in a seed-bound environment. A seed-bound environment is one where all sources of stochasticity are controlled by a single master seed sigma. This includes:
- The random seed for the language model's sampling process. - The initial state of the environment (database contents, file system state, API response queues). - The timing of any asynchronous operations (network requests, background processes).
Definition 5 (Seed-Bound Environment). An environment E is seed-bound if and only if, for any agent A and task T, the tuple (E, A, T, sigma) uniquely determines the complete trajectory, including all intermediate states and the final outcome.
In practice, perfect seed binding may not be achievable for environments that depend on live external services. In such cases, we require that external interactions be recorded and replayed deterministically during evaluation, using a proxy or recording layer.
Given seed-bound environments, we require that each agent-task pair be evaluated over K independent seeds, where K >= 5. We report the following:
- Mean completion rate: mu = (1/K) sum_{i=1}^{K} R(A, T, sigma_i) - Standard deviation: sigma = sqrt((1/(K-1)) sum_{i=1}^{K} (R(A, T, sigma_i) - mu)^2) - Confidence interval: mu +/- t_{K-1, alpha/2} * sigma / sqrt(K)
This allows practitioners to assess not just whether an agent can solve a task, but how reliably it does so. An agent with mu = 0.8 and sigma = 0.1 is meaningfully different from one with mu = 0.8 and sigma = 0.4.
To address Failure 2, we require that each task include an intent specification: a structured document that describes the constraints, preferences, and solution characteristics the task designer intended. Intent specifications can take several forms:
- For coding tasks: a set of invariants the solution must satisfy, style constraints, and architectural preferences. - For web tasks: a set of action constraints (e.g., "do not add more than 3 items to the cart") and routing preferences (e.g., "use the search function, not the category browser"). - For knowledge tasks: a set of source constraints (e.g., "cite only peer-reviewed sources published after 2020").
Intent fidelity is scored as the fraction of intent constraints satisfied by the agent's trajectory. We define:
F(A, T) = (1/K) sum_{i=1}^{K} F(A, T, sigma_i)
where F(A, T, sigma_i) is the intent fidelity for the i-th run, as defined in Definition 2.
To address Failure 3, we measure the repair cost for each run. We define the aggregate repair cost as:
C(A, T) = (1/|S_success|) sum_{sigma in S_success} C(A, T, sigma)
where S_success is the set of seeds for which the agent succeeded. For agents that never succeed, C(A, T) = infinity.
We also report the repair cost distribution, not just the mean. An agent that always succeeds on the first attempt and one that sometimes requires 50 attempts but has the same mean repair cost are very different. We report the median, 90th percentile, and maximum repair cost across successful runs.
To address Failure 4, we require that every evaluation run produce a replay artifact: a complete, hash-chained provenance record of the agent's trajectory. The replay artifact must be sufficient to:
1. Re-execute the trajectory deterministically in the seed-bound environment. 2. Verify that the reported outcome matches the replayed outcome. 3. Audit the trajectory for safety violations, intent constraint violations, and unexpected behaviors.
Definition 6 (Replay Integrity). A replay artifact R is integrity-complete if and only if:
hash(R) = H(state_0 || action_0 || obs_0 || state_1 || action_1 || obs_1 || ... || state_n)
where H is a collision-resistant hash function and the concatenation includes all intermediate states, actions, and observations.
We define the replay integrity score as the fraction of evaluation runs that produce integrity-complete replay artifacts that verify successfully upon re-execution.
To address Failure 5, we incorporate contamination detection into the evaluation protocol. Specifically:
1. We include canary tasks: tasks with known, unique solutions that are not published. If an agent solves canary tasks at a rate significantly above chance, this suggests contamination. 2. We require periodic benchmark refresh: at least 20% of the task set should be replaced annually with new tasks drawn from the same distribution. 3. We track per-task difficulty drift: if a task becomes significantly easier over time without a corresponding improvement in the underlying agent technology, this suggests contamination.
Based on the framework described above, we propose seven metrics that should be reported alongside (not instead of) traditional completion rates.
The traditional metric: the fraction of tasks on which the agent achieves a successful outcome, averaged over K seeds.
CR(A) = (1/|T|) sum_{T in Tasks} (1/K) sum_{i=1}^{K} R(A, T, sigma_i)
The fraction of intent constraints satisfied across all tasks and runs.
IF(A) = (1/|T|) sum_{T in Tasks} F(A, T)
A measure of run-to-run consistency. We define it as the complement of the coefficient of variation of completion rates across seeds:
RI(A) = 1 - (sigma_CR / mu_CR)
where sigma_CR is the standard deviation of CR across all task-seed pairs and mu_CR is the mean. An RI of 1.0 indicates perfect reproducibility (deterministic success or failure on every task). An RI of 0.0 indicates that the standard deviation equals the mean, which is the case for a Bernoulli process with p = 0.5.
The median number of failed attempts before success, computed only over successful runs.
MRC(A) = median_{(T, sigma): R(A,T,sigma)=1} C(A, T, sigma)
The fraction of evaluation runs that produce replay artifacts that verify successfully.
RIS(A) = (number of verified replays) / (total number of runs)
The fraction of runs in which the agent correctly halts or refuses when a safety constraint is triggered (e.g., attempting to access a forbidden resource, generating disallowed content). This metric requires that tasks include explicit safety gates.
SGC(A) = (number of correct safety decisions) / (number of safety gate triggers)
The fraction of replay artifacts that contain sufficient provenance information to reconstruct the agent's decision-making process at each step. This goes beyond replay integrity (which only requires deterministic re-execution) to require that the provenance chain includes the agent's reasoning, the tool calls made, and the observations received.
AC(A) = (number of auditable replays) / (total number of runs)
We now specify the complete evaluation protocol that implements the framework described above.
Step 1: Environment Preparation. Construct a seed-bound environment for each benchmark. This involves: (a) replacing all live external services with deterministic mocks or recorded replay layers; (b) snapshotting the initial state of databases, file systems, and other mutable state; (c) binding all random number generators to the master seed sigma.
Step 2: Task Specification Augmentation. For each task in the benchmark, author an intent specification document that captures the constraints, preferences, and intended solution characteristics. This should be done by the original benchmark authors or by qualified annotators with access to the reference solution.
Step 3: Safety Gate Annotation. For each task, identify safety-relevant decision points and annotate them with the correct agent behavior (proceed, halt, request clarification).
Step 4: Agent Evaluation. For each agent A, task T, and seed sigma_i (i = 1, ..., K, K >= 5): (a) Initialize the environment with master seed sigma_i. (b) Run the agent to completion (or until a timeout is reached). (c) Record the outcome R(A, T, sigma_i). (d) Record the complete provenance chain. (e) Compute the hash chain over the provenance chain. (f) Score intent fidelity against the task's intent specification. (g) Count the repair cost (failed attempts, rollbacks, errors). (h) Evaluate safety gate correctness.
Step 5: Replay Verification. For a random 20% subset of runs, re-execute the trajectory from the recorded provenance chain in the seed-bound environment. Verify that the outcome matches.
Step 6: Metric Computation. Compute all seven metrics (CR, IF, RI, MRC, RIS, SGC, AC) from the collected data.
Step 7: Reporting. Publish the full evaluation report, including: (a) Per-task, per-seed outcome tables. (b) Aggregate metric summaries with confidence intervals. (c) Intent fidelity breakdown by constraint type. (d) Repair cost distributions. (e) Replay artifacts for all runs (publicly downloadable). (f) Contamination canary results.
Our work builds on and extends several lines of research.
Agent Benchmark Design. GAIA [1] introduced the concept of real-world assistant tasks with unambiguous answers. SWE-bench [2] brought agent evaluation into software engineering with real GitHub issues. AgentBench [3] provided multi-environment evaluation. WebArena [4] and WorkArena [5] created realistic web environments. OSWorld [6] extended evaluation to desktop environments. These benchmarks established the field but adopted single-run completion rate as the primary metric, which we argue is insufficient.
Agent Architectures. ReAct [12] introduced the reasoning-acting interleaving pattern that underlies most modern agent architectures. Reflexion [10] added verbal self-reflection and iterative refinement. Voyager [13] demonstrated lifelong learning in embodied environments. AutoGPT and similar systems popularized autonomous agent loops. Each of these architectures exhibits run-to-run variance that current evaluation practices do not capture.
Multi-Agent Systems. Recent work on multi-agent debate [14] and multi-agent platforms [15] introduces additional evaluation challenges, as the behavior of multi-agent systems is even more sensitive to initialization and stochasticity than single-agent systems.
LLM Surveys and Frameworks. Xi et al. [16] provided a comprehensive survey of LLM-based agents. Wang et al. [17] surveyed agent evaluation methodologies. Symbolic learning approaches [18] propose self-evolving agents that adapt over time, which makes static benchmark evaluation even less representative.
Reproducibility in ML. The broader reproducibility literature, including work on variance in deep RL [9] and reproducibility checklists [8], provides the intellectual foundation for our concern with run-to-run variance. We extend these ideas to the specific context of LLM-based agent evaluation.
Frontier AI Governance. Discussions of frontier AI regulation [19] have highlighted the need for evaluation frameworks that go beyond simple capability measurement to assess reliability, safety, and auditability. Our framework addresses this need concretely in the agent evaluation context.
We acknowledge several limitations of our proposed framework.
Computational Cost. Running each agent-task pair over K >= 5 seeds increases the computational cost of evaluation by at least 5x compared to single-run evaluation. For expensive agents that require many LLM calls per task, this cost may be prohibitive for smaller research groups. We argue that the cost is justified by the increased validity of the results, but we recognize that cost is a real barrier to adoption.
Seed-Bound Environment Construction. Creating truly seed-bound environments for benchmarks that depend on live web services is technically challenging. Our proposal to use recorded replay layers introduces its own issues: the recorded interactions may not capture the full complexity of the live environment, and the recording process itself may introduce artifacts.
Intent Specification Authoring. Writing high-quality intent specifications requires significant effort and domain expertise. The quality of intent fidelity scoring depends directly on the quality of the intent specifications. Poorly written specifications will produce misleading intent fidelity scores.
Replay Artifact Storage. Storing complete provenance chains for all runs generates large volumes of data. For benchmarks with thousands of tasks and multiple agents, the storage requirements may be substantial. We propose storing only hash commitments for the full provenance chain, with the full chain stored by the evaluating party and made available upon request.
Metric Interpretation. We have proposed seven metrics, but we have not established how practitioners should weigh them against each other. An agent with high completion rate but low reproducibility may be preferable to one with moderate completion rate and high reproducibility, depending on the deployment context. We leave the development of composite scores and deployment-specific weighting schemes to future work.
Canary Task Design. Contamination detection through canary tasks requires careful design. If canary tasks are too easy, they will not detect contamination. If they are too hard, they will produce false negatives. The design of effective canary tasks is an open research problem.
Scope of Applicability. Our framework is designed for agent benchmarks that evaluate autonomous task completion. It may not be directly applicable to other forms of AI evaluation, such as human preference evaluation, adversarial robustness testing, or fairness auditing.
We have argued that the current paradigm of agent evaluation, which reports single-run task completion rates, systematically misleads the research community. We identified five specific failures: ignoring run-to-run variance, measuring task completion without intent fidelity, failing to account for repair cost, lacking provenance auditing, and vulnerability to contamination and saturation.
We proposed a reproducibility-first evaluation framework that addresses each failure through seed-bound execution environments, multi-run variance reporting, intent fidelity scoring, repair cost measurement, replay integrity verification, and contamination detection. We defined seven concrete metrics (completion rate, intent fidelity, reproducibility index, median repair cost, replay integrity score, safety gate correctness, and audit completeness) and specified a runnable evaluation protocol.
We do not claim that our framework is complete or that it will solve all problems in agent evaluation. We claim that it represents a necessary step toward honest, reproducible, and decision-relevant evaluation of autonomous agents. The gap between what current benchmarks measure and what deployment requires is growing. Closing that gap requires the research community to adopt evaluation practices that capture the properties that matter: not just whether an agent can solve a task, but how reliably, how efficiently, how faithfully, and how auditably it does so.
We invite the community to adopt, critique, and extend the framework presented here. The evaluation practices we choose shape the research directions we pursue. If we measure only completion, we will optimize only for completion. If we measure reliability, fidelity, cost, and provenance, we will build agents that are genuinely better, not just better at passing benchmarks.
REFERENCES
[1] M. Mialon, R. Dessi, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Roziere, T. Schick, J. Dwivedi-Yu, A. Celikyilmaz, et al., "GAIA: a benchmark for General AI Assistants," arXiv preprint arXiv:2311.12983, 2023.
[2] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" arXiv preprint arXiv:2310.06770, 2023.
[3] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al., "AgentBench: Evaluating LLMs as Agents," arXiv preprint arXiv:2308.03688, 2023.
[4] S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, et al., "WebArena: A Realistic Web Environment for Building Autonomous Agents," arXiv preprint arXiv:2307.13854, 2023.
[5] A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. Del Verme, M. Thakkar, N. Chapados, Q. Cappart, D. Vazquez, L. Boisvert, et al., "WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?" arXiv preprint arXiv:2403.07718, 2024.
[6] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al., "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments," arXiv preprint arXiv:2404.07972, 2024.
[7] J. Y. Koh, R. Lo, L. Jang, V. Duvvur, M. C. Lim, P. Y. Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried, "VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks," arXiv preprint arXiv:2401.13649, 2024.
[8] J. Pineau, P. Vincent-Lamarre, K. Sinha, V. Lariviere, A. Beygelzimer, F. d'Alche Buc, E. Fox, and H. Larochelle, "Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)," arXiv preprint arXiv:2003.12206, 2020.
[9] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, "Deep Reinforcement Learning that Matters," arXiv preprint arXiv:1709.06560, 2017.
[10] N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, "Reflexion: Language Agents with Verbal Reinforcement Learning," arXiv preprint arXiv:2303.11366, 2023.
[11] S. Xu, W. Shi, and B. Wang, "Don't Make Your LLM an Evaluation Benchmark Cheater," arXiv preprint arXiv:2311.01964, 2023.
[12] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, "ReAct: Synergizing Reasoning and Acting in Language Models," arXiv preprint arXiv:2210.03629, 2022.
[13] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, "Voyager: An Open-Ended Embodied Agent with Large Language Models," arXiv preprint arXiv:2305.16291, 2023.
[14] Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch, "Improving Factuality and Reasoning in Language Models through Multiagent Debate," arXiv preprint arXiv:2305.14325, 2023.
[15] D. Gao, Z. Li, W. Kuang, X. Pan, D. Chen, Z. Ma, B. Qian, L. Yao, L. Zhu, C. Cheng, et al., "AgentScope: A Flexible yet Robust Multi-Agent Platform," arXiv preprint arXiv:2402.14034, 2024.
[16] Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, et al., "The Rise and Potential of Large Language Model Based Agents: A Survey," arXiv preprint arXiv:2309.07864, 2023.
[17] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al., "A Survey on Large Language Model based Autonomous Agents," arXiv preprint arXiv:2308.11432, 2023.
[18] Z. Chen, K. Liu, Q. Wang, J. Zhang, W. Su, Y. Zhang, X. Li, and S. Liu, "Symbolic Learning Enables Self-Evolving Agents," arXiv preprint arXiv:2406.18532, 2024.
[19] M. Anderljung, J. Barnhart, A. Korinek, J. Leung, C. O'Keefe, J. Whittlestone, S. Avin, M. Brundage, J. Bullock, D. Cass-Beggs, et al., "Frontier AI Regulation: Managing Emerging Risks to Public Safety," arXiv preprint arXiv:2307.03718, 2023.