LRMoE is a memory-aware Mixture-of-Experts (MoE) training system designed for memory-constrained nodes. It jointly recalibrates backward demand and balances the retained workload during training. During forward execution, it plans retention quotas and expert placement under resource constraints. During backpropagation, it estimates token-expert contributions and distributes retained pairs across experts. Experiments demonstrate that LRMoE reduces cumulative MoE-layer training time by 21.2-44.8% compared to SYMI.
Model discrepancy necessitates accounting for uncertainties in observations and contexts when calibrating a single model parameter. The method constructs localized posteriors by weighting observation likelihoods based on their proximity to a target context, incorporating context uncertainty via covariance inflation. A conditional normalizing flow approximates these localized posteriors using amortized variational inference. Applying this to an ocean turbulence closure reduces predictive errors for temperature and zonal-velocity relative to global calibration.
ThinkJev addresses errors in typed decisions by allowing a single open hybrid-thinking model to generate answers either directly or after a thought process. It employs two systems, System 1 for direct answers and System 2 for answers after the model thinks, ensuring both operate on the same options. The system calibrates the thinking budget based on statistical tests to avoid unnecessary computation, achieving good accuracy without full deliberation. For instance, one system achieved 3.9% error on 8359 test decisions with a 5% target error rate. This approach allows for efficient, calibrated decision-making in language model outputs.
OpenFrontiers is a cross-disciplinary resource combining open research problems with a benchmark of recently resolved mathematical problems. It contains a corpus of 12,151 problems with associated information and a benchmark of 200 problems with resolution status. The evaluation protocol grades formal proofs using independent checkers in various conditions. Testing LLM agents on a subset reveals that closed-book rediscovery is rare, with success often correlating to retrieving published solutions. Ultimately, OpenFrontiers establishes a foundation for measuring research-level scientific rediscovery.
WebAnything is a framework designed to reconstruct source-grounded, reusable web environments from URLs without manual site implementation. It utilizes an Explorer-Builder to collect observable structures and interaction outcomes, along with an independent verifier for local repair or source recapture. The system successfully constructs 244 website artifacts across various domains and supports numerous executed tasks. Compared to human-assisted methods, WebAnything achieves superior scores on fidelity metrics. Fine-tuning a large language model on interaction trajectories further improves overall success in related agent tasks.
LLM agents querying knowledge graphs are increasingly checked by ontologies, but these checkers may introduce errors or false alarms. The study investigates the performance of curated, LLM-generated, and data-derived ontologies used as checks over agent answers. The research compares the performance of these different ontology sources when checking agents across public and private knowledge graphs. Findings show that a curated ontology on UniProt flagged more correct answers than data-derived or LLM-derived ones. The study analyzes the trade-offs between catching errors and false alarms when using these ontologies as feedback for LLM agents.
AI agents writing smart contracts often fail to capture the intended behavior, which is critical due to the immutable nature of blockchain code. The research proposes an evaluation method based on execution, where transactions are replayed on both AI-written and original contracts to measure agreement. The study finds that common evaluation methods miss significant differences, as scoring choices and function-level checks hide much of the behavioral disparity. An elicitation-first architecture, where agents ask clients clarifying questions before coding, is suggested to recover missing requirements. This approach aims to improve agentic synthesis by gathering missing intent before code generation.
The research proposes a behavioral test called Partner-Outsider Positive Listening (PoPLi) to detect covert collusion between AI agents. This test compares how an agent reacts to a message when a partner uses a secret code versus an uninvolved agent, focusing on the difference in reaction when the message is reworded while keeping its meaning. PoPLi yields values between 0.48 and 0.59 when a code is present, indicating the presence of a private channel. Message-only monitoring methods failed to distinguish between different collusion conditions. Consequently, PoPLi detects the use of a private channel rather than collusion based on incentives or simple codes.
A document-grounded benchmark called LibStructBench was introduced to evaluate AI agents' ability to reconstruct sequencing-library structures and molecular workflows. Experiments revealed that frontier agents often achieve approximate, but not exact, reconstructions of protocol details like molecular states and workflow connectivity. Switching the underlying language model had a significant impact on performance, especially for molecular-state reconstruction. Autonomous accumulation of procedural rules and cross-protocol examples did not yield mean improvements in the tested trajectory. These results emphasize the need for more verifiable approaches in AI reasoning about complex molecular experimental protocols.
Seq@K is a new evaluation protocol designed to measure how language models use sequential feedback during problem-solving attempts. Unlike existing metrics like Pass@K, Seq@K specifically evaluates the model's ability to refine an answer across K sequential attempts with feedback after each failure. Evaluation on 7 benchmarks and 16 models showed that Seq@K reveals a different model-selection landscape than Pass@K. Analyzing model trajectories under Seq@K helps characterize iterative refinement behaviors, such as how quickly models address feedback. These results suggest that practitioners need to consider Seq@K for faithfully benchmarking language models operating in feedback-driven agentic workflows.