RealReplicaBench Offers New Standard for Testing Long-Horizon AI Agents

·1 views

A team of researchers has released RealReplicaBench, a benchmark designed to evaluate AI agents across extended interactions in stateful, high-fidelity replicas of real-world online environments. The project, hosted on GitHub, has attracted over 1,050 stars, reflecting strong interest from the AI development community. Unlike traditional benchmarking tools, RealReplicaBench captures how an agent's past actions influence future outcomes, better reflecting the complexity of live online services. Built primarily in HTML for browser-based accessibility, the tool does present trade-offs around computational performance and integration with frameworks like TensorFlow or PyTorch. The benchmark aims to fill a critical gap by pushing developers to optimize agents for sustained, long-horizon decision-making rather than short-term performance alone.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

China Requires Two-Step Compliance Process Before Public Generative AI Launch

China's Cyberspace Administration, along with six other central departments, issued the Interim Measures for the Management of Generative AI Services in July 2023, which took effect in August 2023. The rules apply to any generative AI service offering text, image, audio, or video content to the public within mainland China, with exemptions for internal tools and services not targeting domestic users. Providers whose services carry public opinion attributes or social mobilisation capability face an additional two-stage obligation: a security self-assessment submitted to cyberspace and public security authorities, followed by a separate algorithm filing through the CAC's online system. Regulators treat most public-facing generative AI platforms as falling within the scope of these attributes, making the dual requirement broadly applicable in practice. Experts caution that the two steps are sequential and interdependent, and that providers should seek dedicated Chinese legal counsel before launching any service in the market.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Estimate LLM Costs Before Running Batch Content Moderation at Scale

Moderating high-volume user content in e-commerce hiring workflows requires cost forecasting before batch LLM jobs begin, not just after. A two-pass pipeline is recommended: a narrow classifier first labels submissions as allow, review, or block, while uncertain or high-impact cases are routed to human reviewers. Token counting before submission and asynchronous batch processing for backlog content help control costs, especially since risk profiles vary significantly across tenants by submission volume. A practical evaluation approach involves running 1,000 representative candidate submissions across tenants, comparing synchronous and batch processing on metrics like actual token usage, retry counts, and false negatives. A lightweight Python estimator can make cost assumptions explicit and tenant-specific, though it must be calibrated against the production tokenizer for accurate accounting.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

China's AI Content Labelling Rules Take Effect September 2025

China's Cyberspace Administration, alongside three other ministries, issued the Measures for Labelling of AI-Generated Synthetic Content on 14 March 2025, with the rules coming into force on 1 September 2025. The regulations require two distinct labels for AI-generated content: a visible mark for human viewers and an embedded metadata mark readable by machines. Obligations are shared across content generators, distribution platforms, app stores, and users, meaning no single party bears sole responsibility. A companion national technical standard, GB 45438-2025, specifies the required metadata fields, file formats, and encoding methods. The new measures build on earlier Chinese regulations covering deep synthesis, generative AI services, and algorithmic recommendations, adding greater specificity to labelling duties previously stated in broader terms.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Models Learn to Predict Chemical Reaction Products Using SMILES Sequences

Researchers have developed AI approaches to forward reaction prediction, which determines what products emerge from given reactants and reagents. Two main methods are used: sequence-to-sequence models that treat reaction SMILES strings as a translation problem, and graph-based models that predict which chemical bonds change during a reaction. Philippe Schwaller and colleagues introduced the Molecular Transformer in 2019, demonstrating that token probability scores can help prioritize predictions requiring expert review. A key challenge is atom-mapping — correctly tracking where each atom ends up — since models scored only on product strings may learn pattern matching rather than true chemical transformations. Inconsistent handling of reactants versus agents across datasets also makes published benchmark accuracies difficult to compare directly.

0 comments Read more at DEV Community