ClickHouse system logs silently consumed 12 GB and triggered millions of needless merges

·1 views

A developer running a self-hosted observability stack on a budget 2-core VPS noticed ClickHouse consuming 880 MB of RAM and heavy disk activity despite near-zero application traffic. Investigation revealed ClickHouse's own internal system logs — including trace_log, asynchronous_metric_log, and text_log — had grown to 579 MB and 46 million rows, dwarfing the 543 KB of actual application data. These logs, which carry no TTL by default, were responsible for 98.8% of all database inserts and triggered a 1:678 merge amplification ratio, meaning 678 existing rows were rewritten for every new row added. Disabling the heavy system logs reduced merged rows from 11 million to just 5,727 per 30-second window and freed 2.7 GB of disk space. The developer also flagged a common debugging pitfall: Docker stats reports CPU as a percentage of one core, causing apparent resource usage to appear twice as high as actual host-level consumption.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI Writes Tests Cheaply, But Your Team Bears the Long-Term Maintenance Cost

AI tools can rapidly generate test automation code — including page objects, fixtures, and assertions — dramatically reducing the initial effort required. However, the real cost emerges after tests enter the repository, as teams must validate selector durability, assertion quality, and data safety over time. AI-generated tests also frequently pass locally but fail in CI environments due to missing context around Node versions, feature flags, browser builds, and resource limits. Reviewers of AI-generated tests must prioritize intent — asking what product risk is reduced and whether a test genuinely catches failures — rather than focusing only on syntax. As test generation becomes nearly free, teams risk accumulating hundreds of low-quality tests, making restraint and architectural discipline more critical than ever.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How MCP Agent Teams Help AI Scale Beyond Single-Agent Limits

Most AI products today rely on a single conversational agent, but this architecture has a natural ceiling as task complexity grows. When one agent is burdened with planning, execution, validation, and error recovery, the hidden coordination costs accumulate inside an increasingly unwieldy prompt and context window. The Model Context Protocol (MCP) offers a way to distribute that complexity across specialized agent teams, where individual agents can be exposed as callable tools by a coordinating agent. However, this multi-agent approach carries real trade-offs, including higher token consumption, more failure points, and the need to manage shared state. Developers are advised to start with a single agent and scale out to a team only when implicit coordination within that agent becomes the primary bottleneck.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Your Browser Test Suite Gets Slower and Flakier Over Time

Browser test suites rarely fail all at once; instead, they degrade gradually through small technical decisions like added retries, mismatched environments, and poorly scoped assertions. Many failures labeled as 'flakiness' actually stem from differences between local and CI environments, such as CSS motion preferences, locale, timezone, or viewport settings that alter how components render. Feature flags compound the problem by creating multiple versions of the same UI, and most teams do not track which flag combinations were active when a test failed. Experts recommend making all environment settings explicit in test configurations rather than relying on defaults, and capturing flag state, user segment, and DOM snapshots alongside test logs. Treating browser configuration as test data, not background noise, is key to building a reliable and maintainable test suite.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Run Pytest Effectively in CI with Logging, Markers, and Coverage

A technical guide outlines best practices for running Pytest in production and CI environments, focusing on logging, test categorization, and code coverage. Developers can use flags like --log-cli-level and --color=yes for clearer output, and the caplog fixture to assert on specific log messages within tests. Pytest markers allow teams to separate fast unit tests from slower integration tests that require external resources like AWS, preventing CI failures when those environments are unavailable. An autouse fixture in conftest.py can automatically skip integration tests unless a specific environment variable is set, adding a safety net beyond manual filtering. The guide also recommends using pytest-cov to measure code coverage, while cautioning that high coverage percentages do not guarantee correctness if test assertions are weak.

0 comments Read more at DEV Community