How to Build a Fast Local Search Index from Your X (Twitter) Archive

·1 views

A developer has outlined a method to convert a large X (formerly Twitter) data archive — typically 150 to 400 MB — into a lightweight, queryable local search index. The core challenge is that parsing the archive's main file, tweets.js, as a single JSON object causes Node.js to exceed its default memory heap limit. The solution involves streaming the file and processing one record at a time, so memory usage stays proportional to a single post rather than the entire dataset. The resulting index, stored as NDJSON, can be searched instantly using grep or loaded into DuckDB for ranked, full-text queries. Because post IDs increase monotonically, the index can be updated incrementally when a new archive export is downloaded, avoiding a full rebuild.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Dev Builds Theatre Prop Continuity Tracker to Catch Staging Errors

A developer created Prop Continuity Desk, a rehearsal planning web app designed to help small theatre productions track the physical movement of props across scenes. The app is built around a fictional six-scene play with twelve props and detects two key mistakes: placing the same prop in two locations within one scene, and moving a prop between scenes without a recorded transfer cue. It uses Next.js, React, TypeScript, and Sanity as its content backend, with scene reviews stored in separate documents to avoid overwriting the data being reviewed. A practice-copy feature lets visitors explore the rules without altering the live production plan, and any exported cue sheet from a practice session is clearly labelled as such. The project was built using Codex starting 26 September 2026, with the AI proposing the scenario, writing the code, and running the tests under the account owner's authorization.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Sekura JS Explores Sharing Software Verification Proofs Without Exposing Source Code

Sekura JS is developing an approach that separates a software module's source code from its verification contract (SJV) and proof artifact (SJP), allowing developers to share evidence of formal verification without disclosing proprietary implementations. The SJV document specifies the properties a module claims to satisfy, while the SJP carries the mathematical evidence produced during verification. A third-party verifier can replay the stored SMT obligations using tools like Z3 or CVC5 to confirm their validity without ever seeing the underlying source code. However, the project acknowledges a key limitation: replaying an SJP can confirm that the mathematical proofs are valid, but cannot independently verify that those proofs were actually generated from the specific private codebase the developer claims. This creates an explicit trust boundary between the private source code and the proof artifacts exchanged between parties.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Where Game Development Budgets Actually Go, Explained by Category

Game development costs are typically split across five core categories: engineering, art and audio, design, QA and testing, and project management. Engineering tends to consume the largest share, usually between 40 and 50 percent, covering gameplay systems, networking, platform work, and performance optimisation. Art and audio follow at 20 to 30 percent, while design, QA, and project management each account for roughly 5 to 15 percent. These ranges are based on agency-scoped projects and shift depending on genre, platform targets, team structure, and scope. When budgets are limited, understanding which category drives the most complexity for a given project can help teams prioritise spending more effectively.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Organisations Must Modernise Ageing Educational Games Before They Vanish

Thousands of educational games built between 2000 and 2015 are becoming inaccessible due to obsolete technologies such as Adobe Flash, discontinued in December 2020, and legacy platforms like Java applets and early Unity builds. The shift in classroom devices toward Chromebooks, iPads, and smartphones has further rendered older games unusable, as they were designed for desktop monitors with mouse-and-keyboard input. Accessibility gaps are another pressing concern, with many legacy games failing to meet WCAG 2.2 standards published in 2023, leaving learners with disabilities unable to engage with the content. Institutions in the UK, US, and elsewhere also face legal obligations under laws such as the Equality Act 2010 and the ADA, making non-compliant content a potential liability. Modernising these tools is presented not as a discretionary upgrade but as a necessary step to preserve past educational investment and meet current technical, legal, and pedagogical expectations.

0 comments Read more at DEV Community