J’s Digest

 The Principle and Advantages of Topic Clustering When facing a vast news corpus, topic clustering is like first sorting scattered puzzle pieces into thematic groups, then sampling representative pieces from each group. This “cluster‐then‐sample” approach avoids wasted computation on near‐duplicate articles and ensures every distinct topic receives proper attention. Topic clustering isn’t a strict statistical method but a prefiltering step based on semantic similarity. It makes sampling more purposeful, reducing redundant model work on similar content while boosting both diversity and coverage among the chosen samples. By clustering, we can flexibly allocate sampling quotas—for example, oversampling key issues or diving deeper into hot topics within specific time frames. Together, these advantages pave a far more efficient path for downstream NLP analysis. Using MinHash and TF–IDF for Topic Clustering MinHash creates compact hash signatures of texts to rapidly estimate pairw...

 The Core Concept of Stratified Sampling Stratified sampling is like sorting a basket of fruit by color into layers, then picking representative fruits from each layer to ensure all types are covered. When news articles vary unevenly by crime category or media source, stratification lets us sample within each important subgroup to maintain balance in the overall analysis. This method is especially fitting whenever the population shows clear differences and we want sufficient representation from every subgroup. In Taiwanese crime-news reporting, different years, regions, or outlets may emphasize different issues. Without stratification, simple random sampling could overconcentrate on the most popular reports and miss other key clues. Researchers can choose any stratification criterion that fits their goals—time period, geography, crime type, or other factors. By incorporating this step before sampling, we account for data diversity up front, making downstream NLP analysis more re...

  The Core of Sample-Size Calculation: From Theory to Practice In any sampling design, determining the sample size is crucial to balancing result confidence with resource constraints. Theoretically, we seek the best estimate of the true population characteristics—yet without wasting unnecessary time and electricity. It’s like choosing which movies to watch: we don’t spend all day watching every recommended film, but instead use reviews or trailers to pick the few most worthwhile titles. From a statistical standpoint, the sample size not only governs estimate stability but also reflects our confidence in the conclusions. The parameters in the sampling formula interlock—adjust one, and you change the amount of work needed downstream. When a simple yet powerful formula yields about a thousand required samples, we can save resources while retaining sufficient precision. The beauty of theory is that it gives us a grounded yet flexible framework for real-world application. Once you gr...

Introduction When I began performing NLP semantic analysis on 20,000 Taiwanese news articles, I was struck by the sheer data volume and the immense pressure it would place on energy consumption and hardware. Processing the entire dataset would indeed give the most complete perspective—but it would also demand hours (or days) of computation on high-end servers, exacting a heavy toll in both time and cost. To achieve efficient analysis under limited resources, I started thinking about how to be smarter: How could I preserve depth of insight while drastically cutting energy use? At first, I considered distributed computing with GPU acceleration, model pruning, and knowledge distillation—techniques designed to boost throughput without sacrificing accuracy. But these engineering measures proved expensive to deploy, offered less energy savings than hoped, and increased system maintenance complexity. Next, I explored active learning and incremental-update strategies so the model could iterat...

 Thought for a couple of seconds 全量處理的悖論與資源天秤 當我們面對二萬篇新聞時，直覺告訴我們「全量跑過一次就最完備」，彷彿不漏掉任何文章才是最安全的選擇。然而，真正動手執行後才發現，巨量資料的推論不僅耗費數倍的計算時間，更讓硬體、記憶體與雲端運算成本如暴漲般攀升。那種「一篇篇慢慢看過去」的耐力賽，既考驗系統承載，也容易因為中途中斷而前功盡棄。 此時，資源與效能之間的天秤開始傾斜。我們不禁要問：花費這些時間與金錢，換來的增量價值究竟在哪裡？如果只為了多跑出幾篇文章的結果，就耗盡所有資源，似乎得不償失。於是，對大數據的熱情，必須與務實的資源管理思維相互融合，才能在有限的硬體與預算中，找到最能成就目標的平衡點。 在推論時間與結果準確度的賽跑中，全量處理宛如一頭猛獸，一旦啟動，難以停下。這場悖論提醒我們，面對大資料，不見得專注在「多」上，而要思考「精不精準」以及「快不快速」。唯有另闢蹊徑，才能跳脫直線思維，找到突破困境的金鑰匙。 抽樣思維：以小搏大的策略 抽樣思維，就像在海洋裡撈魚，不必滿網捕撈，而是用一個代表性的樣本去推估整體情況。當我們計算出在95%信心水準、3%誤差範圍內，只需約一千篇文章，就能準確反映二萬篇新聞的分布。這瞬間點亮了我們的思路：用有限的樣本，換取無限的洞見。 不再是與時間的賽跑，而是與統計學的智慧相互攜手。透過隨機抽樣，我們就像是讓資料自己告訴我們「重點在哪裡」，而非盲目地全部推論。如此一來，原本龐大的推論任務瞬間變得可行，也讓資源分配更具效率，讓分析流程更具彈性。 從實際效益來看，抽樣不僅節省了近九成以上的推論時間，更在模型訓練與調參上，提供了更快的迭代速度。反覆實驗後，我們驚喜發現：少量抽樣得到的結論，與全量推論的結果竟然高度吻合，為專案帶來了質與量的雙重突破。 信心水準與誤差邊界的祕密 在抽樣的世界裡，信心水準與誤差邊界是我們的導航儀。信心水準代表了我們對抽樣結果的信心有多大，而誤差邊界則限定了結果可能偏離真實值的範圍。當我們設定95%的信心水準與3%的誤差時，就等於告訴自己：九十九次裡，至少九十五次的推論誤差不會超過三個百分點。 學習這些概念後，我們再也不會對「一千筆就能代表整體」心生疑慮。相反地，每次抽樣前，都會先核對計算公式，確保水準與邊界都符合需求。這種用統計量化風險的做法，不僅讓我們對結果更有...

破解 Yahoo News 重導與資源管理的挑戰 在爬 Google News 文章時，我們一度卡在「重導向」的迷宮裡：即使開了  allow_redirects=True ，最終仍拿到 Google 的中繼頁面。直到 5 月 3 日那天，我和書慶大哥花了好幾天，才終於「客服」了這個難題。我們回頭觀察 Yahoo News 的 HTML，發現每篇文章頂端都會載入原始撰寫媒體的 Logo 圖片。於是，我們改為解析  <img>  標籤裡的 Logo 檔名，從中提取出對應媒體名稱，再透過這個線索獲得真實的文章來源 URL。 解決了重導向問題後，接著面臨的是記憶體飆升的痛苦。原本的程式在載入每個新分頁後，往往忘了適時關閉，導致 ChromeDriver 的進程一個接一個堆積，就像無限增長的恐怖小說怪物，最終讓整台機器陷入瀕死狀態。我們嘗試過在每次  driver.get()  後立刻呼叫  driver.quit() ，卻又因為重啟成本太高，把執行效率拉到地板。最後，我們找到了一個折衷：在「載入完畢、抓到 Logo 並擷取連結」的訊號之後，再給瀏覽器一個短暫的緩衝（約 1–2 秒），才關閉分頁。這樣既能保留穩定的執行上下文，也不會長時間占用系統資源。 除了這些硬體層面的優化，我們還在瀏覽器層面做了兩項改良。第一是切換到「無痕模式」，讓瀏覽器不會持久化過多歷史紀錄；第二是在每次關閉分頁前，主動執行一段 JavaScript 清除所有 Cookies。這樣就不會因為歷史與 Cookie 爆炸，又觸發瀏覽器的記憶體或磁片緩衝上限，整個抓取流程才能在一天之內，耗費三小時，持續跑完三千則 Yahoo News，並將最終的 28,000 多筆資料下載下來而不當機。 網路封鎖與多 IP 輪換的實踐 那段時間，我不知刷了多少次 Google，IP 很快就被封鎖。想起 5 月 2 日下午，我們還在聊「12 月為何突然暴增」，於是馬上用不同 IP 重跑相同關鍵字，才發現那波高峰其實是某起社會矚目新聞引爆了大量二次報導。這次經驗讓我們明白：要破解「同一 IP 被臨時限流」的問題，最直接就是輪換多個代理伺服器，並搭配隨機延遲，讓每一次請求都像來自不同的「真人」。 我們先組了一個小型代理池，每次發出請求就從中隨機抽一個 IP；若...