Why CI Data Beats Surveys for Measuring Developer Experience

·1 views

Most teams default to surveys when assessing developer experience, but the data needed to understand friction already exists in CI systems, deploy logs, and support channels. Metrics like push-to-merge time, check re-run rates, and rollback frequency can reveal real pain points without requiring new tools or vendor contracts. Tracking these numbers per team or per service — rather than as org-wide averages — exposes the outliers that aggregate figures hide. A high re-run rate on CI checks, for instance, signals eroding trust in the pipeline long before visible failures occur. This approach turns vague complaints into specific, actionable lists, such as identifying exactly which jobs developers no longer believe.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How Defenders Can Use ZoomEye to Map Their Own Internet-Exposed Assets

Internet-wide scanning tools like ZoomEye, commonly used by attackers to locate vulnerable systems, can also be used by organizations to audit their own externally visible attack surface. A four-step self-assessment workflow involves building technology fingerprints, running scoped queries, comparing results against internal asset inventories, and remediating unexpected exposures. The approach gained urgency after an August 2026 joint advisory highlighted that attackers used ZoomEye and Censys to identify exposed Siemens S7 PLCs before launching exploits. ZoomEye queries run on September 19, 2026, revealed hundreds of millions of publicly reachable instances of common platforms including Apache, nginx, WordPress, and Fortinet. Security experts stress that the gap between what an organization believes is internet-facing and what is actually reachable is a primary entry point for breaches.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer finds stricter eval rule discards more data as test repetitions grow

A software developer running a frozen 159-task benchmark suite to compare two AI models discovered that a conservative measurement rule increasingly discarded tasks with each additional test repetition. The stricter rule, designed to filter unstable results, threw away 13 tasks across three repetitions and 17 across six, shrinking the informative dataset and eventually masking a statistically significant difference between the models. A second, less restrictive rule was developed and pre-registered before new test runs began, allowing an independent replication on unseen data. The looser rule reached statistical significance on the fresh data, while pooling all six repetitions under the strict rule returned a non-significant result. The developer concludes that conservative stability filters are not cost-free choices, as they systematically discard genuinely stochastic tasks the more data is collected.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Finds His LLM Eval Harness Was Grading Format, Not Factual Accuracy

A developer building an LLM drift-detection suite discovered that his exact-match grader was penalizing correct answers due to formatting differences, not factual errors. Tasks asking for the chemical symbol for gold and the smallest planet recorded near-100% failure rates across GPT-4o mini, Claude Sonnet 5, and Llama 3.1 8B, even though all models knew the correct answers. The models were responding with full sentences or markdown styling such as '**Au**' rather than the bare two-letter string the grader expected. A near-identical task asking for a country capital passed cleanly because its instructions specified an exact output shape, exposing the inconsistency in how constraints were worded across the suite. The developer notes the measurement is reliable but the precise explanation is still uncertain, and warns others maintaining exact-match eval suites to audit which tasks have failed consistently without anyone checking the raw model output.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

API Model Beat Custom-Trained AI at Same Cost, Prompting Team to Ditch Distillation

A development team initially estimated they could save roughly $86,000 by training a smaller Qwen2.5-7B model instead of using Claude Sonnet to process 6.5 million financial text units. Early benchmarking showed DeepSeek V3.2 underperforming Sonnet on hard test cases, which initially supported the case for building a distilled student model. However, after refining their extraction rubric to allow 'Unknown' as a valid output, a newer DeepSeek model passed all 46 cases across three held-out test sets. Optimized API costs then converged with the estimated $1,000 GPU cost of training and running their own model. With correction cycles far faster via API than through retraining and redeployment, the team decided renting a model was the more practical choice for this workload.

0 comments Read more at DEV Community