SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Relatório anual no Trino travava com 144 milhões de linhas; divisão por mês resolveu

0
·3 views

A autora relata que um script Python de relatório mensal, que processava cerca de 12 milhões de linhas, deixou de funcionar ao ser adaptado para o ano inteiro, o que resultaria em aproximadamente 144 milhões de linhas. A consulta única excedia o limite de memória do cluster Trino, causava falhas demoradas e disputava recursos com outros usuários. A principal correção foi dividir o processamento em doze consultas mensais, cada uma com volume semelhante ao que o script já tratava bem. Além disso, a agregação passou a ser feita no próprio banco, evitando trazer colunas desnecessárias e milhões de linhas para o Python. Por fim, cada mês passou a ser salvo como checkpoint, para que uma falha não exija reprocessar todo o período.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Handling Slow Third-Party APIs in Laravel with Timeouts, Retries and a Circuit Breaker

A developer describes how a slow shipping-rate provider exhausted PHP-FPM workers and took down unrelated pages in a Laravel app. The article explains that Laravel's HTTP client ships with a 30-second timeout, no retries, and no automatic failure on error status codes, which can tie up workers and treat errors as data. It recommends short explicit timeouts, retrying only transient failures like connection errors, 429 and 5xx responses, and using backoff with jitter that honours Retry-After headers. The author proposes a shared HTTP macro in AppServiceProvider so every partner call follows the same rules, and adds safe retries for writes via idempotency keys and a circuit breaker to fail fast when a partner is down.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Builder Creates Private Mail App on Grok Bot's Inbox Without Using Any Public API

A developer gave the Grok Bot assistant its own email address and built a single-user mail app called Grok Mail on top of it. The app, built with Next.js on Vercel and storing data in Neon Postgres behind a Clerk login, never connects to the mail provider directly because no public inbox API exists. Instead, an hourly Grok Bot routine syncs incoming mail into the database, and every outgoing message becomes a draft card that the user must approve before sending. The author highlights risks such as untrusted inbound email and prompt injection, and notes that Origin repositories require a higher Vercel plan than the free Hobby tier.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Converting Video to MP4 Can Shrink or Grow a File, Depending on Bitrate

A test of five video conversions by CompressVid, an online video toolkit, found that changing format to MP4 could cut file size by up to 77% or increase it by as much as 79%. The company explains that the container format itself adds very little to file size, so the real change comes from the codec and the bitrate chosen during re-encoding. When a tool transcodes to H.264 for compatibility, the new bitrate can be much lower or higher than the original's. Results also depend on the source, since phone recordings with high real-time bitrates may still produce larger files than an H.264 encode. The findings suggest that the file extension alone does not determine size, and that the encoding settings matter far more.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Police seize LockBit ransomware infrastructure and expose its leader

In February 2024, the UK's National Crime Agency, the FBI, Europol and partners across about ten countries took control of LockBit's leak site and administration systems. The operation seized servers and cryptocurrency accounts and gave investigators data on affiliates, while releasing around a thousand decryption keys to help victims recover files without paying. Police also rebuilt the leak site to mock the gang and taunt its operators, undermining the trust that ransomware networks depend on. In May 2024, the group's leader was named as Dmitry Khoroshev, a Russian national, who was sanctioned and charged in the US.

0 comments Read more at DEV Community