Self-Hosting SigNoz in 2025: Key Pitfalls With ClickHouse v25 and OpenTelemetry

·3 views

A developer at Verne Software documented the real-world challenges of self-hosting SigNoz, an open-source observability platform, using Docker Compose in a modern stack. What was expected to be a quick setup turned into an in-depth troubleshooting exercise involving ClickHouse configuration changes, OpenTelemetry bugs, and Docker networking issues. ClickHouse v25.5.6 introduced a stricter configuration paradigm, requiring overrides to be placed in config.d/ and users.d/ directories rather than replacing the main config file directly. The modern SigNoz stack also demands a strict service boot order, with containers like the telemetry store migrator needing to complete successfully before the collector and main app can start. The article serves as a practical guide for teams seeking full control over their telemetry data through isolated Docker networks and deterministic deployment sequences.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

OpenAI's macOS Computer History Feature Logs Keystrokes in Unencrypted Plaintext

OpenAI has introduced a 'Computer History' feature for its ChatGPT macOS app that tracks user clicks, keystrokes, and application switches to build a searchable activity timeline. The feature uses macOS accessibility tools to give ChatGPT persistent context about a user's work, enabling the agent to act more informatively on their behalf. However, the activity logs are stored locally as unencrypted plaintext Markdown files, meaning any process running under the same user account could potentially read them. While the feature is opt-in, supports app blacklisting, and skips private browsing activity, security researchers warn the unencrypted format poses real risks given the sensitive data — such as API keys, credentials, and private messages — that may pass through a user's screen. Experts also flag an expanded prompt injection risk, as malicious content in tracked apps like Slack could theoretically influence future AI-assisted actions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How a Daylight Saving Time Bug Caused Biweekly Production Failures

A software developer encountered a recurring production bug where a report generation job failed every other Thursday at 2 AM, but worked flawlessly in local testing. Investigation revealed the root cause was Python's naive datetime object being shifted across a daylight saving time boundary on a US/Eastern-timezone server. Because the datetime lacked timezone awareness, adding 14 days during a DST transition caused the resulting due date to be off by one hour, breaking compatibility with downstream services. Using a structured debugging framework of Observe, Hypothesize, Test, and Learn, the developer isolated the issue by forcing the job to run with a known DST transition date and logging raw datetime values. The fix involved treating all datetime objects as timezone-aware from the point they entered the system, which eliminated the intermittent failures entirely.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

GrapheneOS Rumored to Expand to High-End Motorola Phones by 2027

GrapheneOS, a privacy-focused and security-hardened Android-based operating system, is rumored to become available on high-end Motorola smartphones by 2027. Currently, the OS is primarily supported on Google Pixel devices, where it is known for features such as app sandboxing and granular permission controls. Motorola's flagship hardware is noted for its durability and strong battery life, which proponents say could complement GrapheneOS's performance-oriented design. The potential pairing has generated interest among privacy-conscious users who seek a balance between security and everyday usability. No official confirmation from either GrapheneOS developers or Motorola has been reported at this time.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Engineer Replaces kube-proxy with eBPF in Homelab Cluster, Loses Monitoring for 6 Hours

A software engineer upgraded Cilium from version 1.15 to 1.16 on a four-node bare-metal homelab cluster running Talos Linux, enabling full eBPF-based kube-proxy replacement by setting a single configuration flag. The change appeared successful at first — all pods remained running and the cluster showed healthy status — but at 2:47 AM an alert revealed that the SIEM had received zero network flow or Kubernetes audit logs since the upgrade. The root cause was that the eBPF datapath bypasses the iptables and netfilter hooks that the security monitoring stack relied on to capture packet-level data. The engineer had migrated away from kube-proxy's iptables rules, which, while less scalable, fed the observability pipeline that the SIEM depended on. The incident highlighted a critical gap in Cilium's documentation: replacing kube-proxy with eBPF silently removes the datapath layer that many security and monitoring tools assume will always be present.

0 comments Read more at DEV Community