Two-Phase Memory Probe Tests Whether AI Code Reviewers Trust Stale Context

·2 views

A new evaluation framework targets a growing blind spot in AI code-reviewer assessments: how persistent memory across pull requests can introduce errors rather than prevent them. Unlike single-shot tests that treat reviewers as stateless, this two-phase probe uses a synthetic fixture repository with a deliberate naming-convention conflict to measure whether a reviewer correctly prioritizes current documentation over cached history. The setup involves two sequential PRs — the first establishes repository context, while the second introduces a real bug alongside a legitimate refactor, testing whether the reviewer catches the bug and cites the correct decision file. The author argues that most hiring evaluations optimize for one-off prompt compliance and miss the risk of a bot anchoring on outdated conventions weeks into production use. The probe is designed to be reproducible and scorable, offering a structured alternative to snapshot tests that cannot detect memory-related failures at all.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

RamaLama Lets You Run Local LLMs as OCI Containers with a Single Command

RamaLama is an open-source CLI tool that packages large language models as OCI containers, allowing users to pull and run models with a single terminal command and no manual Python environment setup. Developed by the container-tooling community, it uses llama.cpp under the hood and supports Docker or Podman as the container engine. A hands-on test on an Apple Silicon Mac with RamaLama 0.24.0 and Docker via OrbStack showed that a first run of the 135M-parameter SmolLM model took under three minutes, mostly due to image and model downloads. Smaller models like the 135M variant are best used only to verify a working setup, as they can produce inaccurate outputs, while a 1B-parameter model like Llama 3.2 performs more reliably. RamaLama also supports an OpenAI-compatible server mode and applies security hardening by default, dropping all Linux capabilities and disabling privilege escalation during container runs.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Salting Technique Fixed a 40-Minute PySpark Data Skew Bottleneck

A PySpark job processing a large transactions-to-customer join was completing 95% of its tasks in under 4 minutes, but one or two straggler tasks were taking over 40 minutes to finish. The Spark UI revealed the cause: data skew, where a handful of high-volume customer IDs held disproportionately large row counts, forcing all their data into a single partition processed by one executor. A pre-join aggregation on the join key confirmed that a small number of customer IDs had row counts orders of magnitude higher than the rest. The fix applied was a technique called salting, which appends a random value to the skewed join key to split one large partition into several smaller ones distributed across multiple executors. After implementing salting with 10 salt buckets, the workload was rebalanced across the cluster and the job's overall runtime returned to acceptable levels.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Security Audit Flags High-Risk Vulnerabilities in $3.2B DeFi Yield Platform USDT0

A smart-contract security audit of USDT0, a cross-chain USDT yield platform with $3.2 billion in total value locked, identified eight vulnerabilities across its core architecture as of August 31, 2026. Two issues were rated high severity: inadequate slippage protection tied to oracle reliance that could trigger forced liquidations, and upgrade functions controlled by a single wallet without multi-signature or time-lock safeguards. Additional medium-severity findings included missing replay protection on cross-chain bridge messages, flawed ERC-20 approval logic, LP token rounding errors, and reliance on a single Chainlink price feed with no fallback. The overall risk score was assessed at 7 out of 10, with auditors noting the protocol is structurally sound but its high asset value, centralised upgrade control, and cross-chain exposure create a significant combined attack surface. Auditors recommended these issues be remediated before the protocol scales further.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

GitHub Copilot Switches to Token-Based Billing, Sparking Developer Backlash

GitHub overhauled its Copilot pricing on June 1, 2026, replacing flat-rate request limits with a token-metering system built around 'AI Credits,' where one credit equals $0.01 of model usage. The change drew sharp criticism from developers, with a community announcement receiving 958 thumbs down, as some users reported their costs rising 10x to 50x depending on which AI model they had chosen. Plan tiers now bundle base and flex credits — for example, Copilot Pro at $10/month offers 1,500 total credits — but model choice heavily influences actual spend, with premium models like Claude Opus costing significantly more per interaction. Unlike Copilot, Anthropic's Claude Code retains a flat-rate subscription model but manages demand through throttling, including a five-hour rolling usage window and reduced limits during weekday peak hours. A key concern for teams is that from September 1, 2026, code review tasks will also consume GitHub Actions minutes in addition to AI Credits, effectively creating a dual billing impact on the same invoice.

0 comments Read more at DEV Community