← Logs

run8-restart.log

2026-05-17 23:02:16,283 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-05-17.log
2026-05-17 23:02:16,284 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-05-17.log
2026-05-17 23:02:16,284 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-05-17.log
2026-05-17 23:02:16,284 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-05-17 23:02:16,309 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-05-17 23:02:16,309 - __main__ - INFO - Configuration loaded and validated successfully
2026-05-17 23:02:16,312 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-05-17 23:02:16,312 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-05-17 23:02:16,327 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-05-17 23:02:16,327 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-05-17 23:02:16,328 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-05-17 23:02:16,329 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=1, exclude_terms=0
2026-05-17 23:02:16,331 - Yiddish24Scraper - INFO - File downloader initialized with 1 terms, prefix 'Y2', and file type organization enabled
2026-05-17 23:02:16,332 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=1, exclude_terms=0
2026-05-17 23:02:16,347 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=1, exclude_terms=0
2026-05-17 23:02:16,351 - TorahAnytimeScraper - INFO - File downloader initialized with 1 terms, prefix 'TA', and file type organization enabled
2026-05-17 23:02:16,351 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=1, exclude_terms=0
2026-05-17 23:02:16,353 - BatorahScraper - INFO - File downloader initialized with 1 terms, prefix 'BT', and file type organization enabled
2026-05-17 23:02:16,354 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=1, exclude_terms=0
2026-05-17 23:02:16,354 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-05-17 23:02:16,354 - __main__ - INFO - Kol Halashon scraper initialized and enabled
2026-05-17 23:02:16,354 - __main__ - INFO - Running 1 scrapers concurrently...
2026-05-17 23:02:16,354 - KolHalashonScraper - INFO - Starting Kol Halashon scraping
2026-05-17 23:02:16,383 - KolHalashonScraper - WARNING - Could not match parasha 'none' to database parsha_list
2026-05-17 23:02:16,383 - KolHalashonScraper - ERROR - Could not determine current parsha, continuing with other search URLs
2026-05-17 23:02:16,383 - KolHalashonScraper - INFO - Processing parsha URL (id=1)
2026-05-17 23:02:16,383 - KolHalashonScraper - INFO - Processing 8 total URLs (1 parsha + 7 search)
2026-05-17 23:02:16,383 - KolHalashonScraper - WARNING - Could not match parasha 'none' to database parsha_list
2026-05-17 23:02:16,384 - KolHalashonScraper - WARNING - Could not resolve parsha_info for URL placeholder replacement
2026-05-17 23:02:16,384 - KolHalashonScraper - WARNING - No daf schedule found for today - cannot replace daf placeholders
2026-05-17 23:02:16,384 - KolHalashonScraper - INFO - Scraping all 5 URLs with unified method...
2026-05-17 23:02:16,384 - KolHalashonScraper - INFO - Starting multi-URL browser scrape with 5 URLs, max 10 pages per URL (download_inline=False)
2026-05-17 23:02:16,384 - KolHalashonScraper - INFO - Running browser in non-headless mode (Cloudflare bypass)
2026-05-17 23:02:16,384 - KolHalashonScraper - INFO - Using undetected-chromedriver for Cloudflare bypass
2026-05-17 23:02:16,586 - undetected_chromedriver.patcher - INFO - patching driver executable /home/openclaw/.local/share/undetected_chromedriver/undetected_chromedriver
2026-05-17 23:02:17,139 - KolHalashonScraper - INFO - Browser created for multi-URL scrape
2026-05-17 23:02:17,139 - KolHalashonScraper - INFO - Processing URL (id=3): https://www.kolhalashon.com/en/regularSite/searchResults/100/49/Meilah?urlFilters=3:3%7C20:13%7C|Harav Avraham Loifer
2026-05-17 23:02:18,988 - KolHalashonScraper - INFO - Page loaded with 49 shiur containers after 0s
2026-05-17 23:02:18,988 - KolHalashonScraper - INFO - [DELAY] Waiting 60.0s for page content to stabilize (shiur_found_wait)
2026-05-17 23:03:25,027 - KolHalashonScraper - INFO - Processing page 1 of URL (pagination shows: page 1 of 3, has_next=True)
2026-05-17 23:03:25,028 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:03:31,043 - KolHalashonScraper - INFO - Extracted 24 shiurim from DOM
2026-05-17 23:03:31,043 - KolHalashonScraper - INFO - Page 1: Browser extracted 24 shiurim
2026-05-17 23:03:31,051 - KolHalashonScraper - INFO - Navigating to page 2...
2026-05-17 23:03:31,095 - KolHalashonScraper - INFO - [DELAY] Waiting 10s for page to reload after clicking next button
2026-05-17 23:03:48,143 - KolHalashonScraper - INFO - Processing page 2 of URL (pagination shows: page 2 of 3, has_next=True)
2026-05-17 23:03:48,143 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:03:54,154 - KolHalashonScraper - INFO - Extracted 24 shiurim from DOM
2026-05-17 23:03:54,154 - KolHalashonScraper - INFO - Page 2: Browser extracted 24 shiurim
2026-05-17 23:03:54,163 - KolHalashonScraper - INFO - Navigating to page 3...
2026-05-17 23:03:54,196 - KolHalashonScraper - INFO - [DELAY] Waiting 10s for page to reload after clicking next button
2026-05-17 23:04:11,239 - KolHalashonScraper - INFO - Processing page 3 of URL (pagination shows: page 3 of 3, has_next=False)
2026-05-17 23:04:11,239 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:04:17,249 - KolHalashonScraper - INFO - Extracted 4 shiurim from DOM
2026-05-17 23:04:17,249 - KolHalashonScraper - INFO - Page 3: Browser extracted 4 shiurim
2026-05-17 23:04:17,256 - KolHalashonScraper - INFO - No more pages available after page 3
2026-05-17 23:04:17,257 - KolHalashonScraper - INFO - URL complete: 52 shiurim extracted from 3 page(s), 52 new total so far
2026-05-17 23:04:17,344 - KolHalashonScraper - INFO - [DELAY] Waiting 0.7s for filter sections to expand (filter_expand_wait)
2026-05-17 23:04:18,077 - KolHalashonScraper - INFO - [DELAY] Waiting 0.7s after clicking 'display more' buttons (filter_expand_wait)
2026-05-17 23:04:18,788 - KolHalashonScraper - INFO - Expanded filter sections: clicked 2 'display more' buttons
2026-05-17 23:04:19,798 - KolHalashonScraper - INFO - DOM filter debug: {'displayAllSections': 2, 'filterContents': 5, 'filterOptions': 45, 'totalCheckboxes': 43, 'type2Checkboxes': 33}
2026-05-17 23:04:19,809 - KolHalashonScraper - INFO - Extracted 31 speakers from DOM filter checkboxes
2026-05-17 23:04:19,809 - KolHalashonScraper - INFO - Extracted 31 speakers from URL filter dropdown
2026-05-17 23:04:19,818 - social_media_monitor.scrapers.kolhalashon_scraper - INFO - Speaker sync complete: 0 new, 31 updated, 517 total
2026-05-17 23:04:19,818 - KolHalashonScraper - INFO - [DELAY] Waiting 1.0s between URLs (shiur_url_delay)
2026-05-17 23:04:20,819 - KolHalashonScraper - INFO - Processing URL (id=4): https://www.kolhalashon.com/en/regularSite/searchResults/100/49/Meilah?urlFilters=3:3%7C20:13%7C|Harav Avraham Shlomo Yavo Zt"l
2026-05-17 23:04:21,223 - KolHalashonScraper - INFO - Login required – attempting to log in
2026-05-17 23:04:21,372 - KolHalashonScraper - INFO - [DELAY] Waiting 1.0s after login button click
2026-05-17 23:04:24,420 - KolHalashonScraper - INFO - Submitted login form with stored credentials
2026-05-17 23:04:26,512 - KolHalashonScraper - INFO - Login confirmed
2026-05-17 23:04:27,450 - KolHalashonScraper - INFO - Page loaded with 49 shiur containers after 0s
2026-05-17 23:04:27,451 - KolHalashonScraper - INFO - [DELAY] Waiting 60.0s for page content to stabilize (shiur_found_wait)
2026-05-17 23:05:33,493 - KolHalashonScraper - INFO - Processing page 1 of URL (pagination shows: page 1 of 3, has_next=True)
2026-05-17 23:05:33,493 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:05:39,503 - KolHalashonScraper - INFO - Extracted 24 shiurim from DOM
2026-05-17 23:05:39,503 - KolHalashonScraper - INFO - Page 1: Browser extracted 24 shiurim
2026-05-17 23:05:39,510 - KolHalashonScraper - INFO - Navigating to page 2...
2026-05-17 23:05:39,546 - KolHalashonScraper - INFO - [DELAY] Waiting 10s for page to reload after clicking next button
2026-05-17 23:05:56,595 - KolHalashonScraper - INFO - Processing page 2 of URL (pagination shows: page 2 of 3, has_next=True)
2026-05-17 23:05:56,595 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:06:02,607 - KolHalashonScraper - INFO - Extracted 24 shiurim from DOM
2026-05-17 23:06:02,607 - KolHalashonScraper - INFO - Page 2: Browser extracted 24 shiurim
2026-05-17 23:06:02,615 - KolHalashonScraper - INFO - Navigating to page 3...
2026-05-17 23:06:02,644 - KolHalashonScraper - INFO - [DELAY] Waiting 10s for page to reload after clicking next button
2026-05-17 23:06:19,685 - KolHalashonScraper - INFO - Processing page 3 of URL (pagination shows: page 3 of 3, has_next=False)
2026-05-17 23:06:19,685 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:06:25,695 - KolHalashonScraper - INFO - Extracted 4 shiurim from DOM
2026-05-17 23:06:25,695 - KolHalashonScraper - INFO - Page 3: Browser extracted 4 shiurim
2026-05-17 23:06:25,702 - KolHalashonScraper - INFO - No more pages available after page 3
2026-05-17 23:06:25,703 - KolHalashonScraper - INFO - URL complete: 52 shiurim extracted from 3 page(s), 104 new total so far
2026-05-17 23:06:25,783 - KolHalashonScraper - INFO - [DELAY] Waiting 0.7s for filter sections to expand (filter_expand_wait)
2026-05-17 23:06:26,518 - KolHalashonScraper - INFO - [DELAY] Waiting 0.7s after clicking 'display more' buttons (filter_expand_wait)
2026-05-17 23:06:27,231 - KolHalashonScraper - INFO - Expanded filter sections: clicked 2 'display more' buttons
2026-05-17 23:06:28,243 - KolHalashonScraper - INFO - DOM filter debug: {'displayAllSections': 2, 'filterContents': 5, 'filterOptions': 45, 'totalCheckboxes': 43, 'type2Checkboxes': 33}
2026-05-17 23:06:28,256 - KolHalashonScraper - INFO - Extracted 31 speakers from DOM filter checkboxes
2026-05-17 23:06:28,256 - KolHalashonScraper - INFO - Extracted 31 speakers from URL filter dropdown
2026-05-17 23:06:28,265 - social_media_monitor.scrapers.kolhalashon_scraper - INFO - Speaker sync complete: 0 new, 31 updated, 517 total
2026-05-17 23:06:28,265 - KolHalashonScraper - INFO - [DELAY] Waiting 1.0s between URLs (shiur_url_delay)
2026-05-17 23:06:29,265 - KolHalashonScraper - INFO - Processing URL (id=5): https://www.kolhalashon.com/en/regularSite/searchResults/100/49/Meilah?urlFilters=3:3%7C20:13%7C|Harav Elazar Nissan Rubin
2026-05-17 23:06:30,738 - KolHalashonScraper - INFO - Page loaded with 49 shiur containers after 0s
2026-05-17 23:06:30,738 - KolHalashonScraper - INFO - [DELAY] Waiting 60.0s for page content to stabilize (shiur_found_wait)
2026-05-17 23:07:36,774 - KolHalashonScraper - INFO - Processing page 1 of URL (pagination shows: page 1 of 3, has_next=True)
2026-05-17 23:07:36,774 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:07:42,788 - KolHalashonScraper - INFO - Extracted 24 shiurim from DOM
2026-05-17 23:07:42,788 - KolHalashonScraper - INFO - Page 1: Browser extracted 24 shiurim
2026-05-17 23:07:42,797 - KolHalashonScraper - INFO - Navigating to page 2...
2026-05-17 23:07:42,838 - KolHalashonScraper - INFO - [DELAY] Waiting 10s for page to reload after clicking next button
2026-05-17 23:07:59,885 - KolHalashonScraper - INFO - Processing page 2 of URL (pagination shows: page 2 of 3, has_next=True)
2026-05-17 23:07:59,885 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:08:05,898 - KolHalashonScraper - INFO - Extracted 24 shiurim from DOM
2026-05-17 23:08:05,898 - KolHalashonScraper - INFO - Page 2: Browser extracted 24 shiurim
2026-05-17 23:08:05,908 - KolHalashonScraper - INFO - Navigating to page 3...
2026-05-17 23:08:05,938 - KolHalashonScraper - INFO - [DELAY] Waiting 10s for page to reload after clicking next button
2026-05-17 23:08:22,982 - KolHalashonScraper - INFO - Processing page 3 of URL (pagination shows: page 3 of 3, has_next=False)
2026-05-17 23:08:22,982 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:08:28,992 - KolHalashonScraper - INFO - Extracted 4 shiurim from DOM
2026-05-17 23:08:28,992 - KolHalashonScraper - INFO - Page 3: Browser extracted 4 shiurim
2026-05-17 23:08:28,999 - KolHalashonScraper - INFO - No more pages available after page 3
2026-05-17 23:08:29,000 - KolHalashonScraper - INFO - URL complete: 52 shiurim extracted from 3 page(s), 156 new total so far
2026-05-17 23:08:29,078 - KolHalashonScraper - INFO - [DELAY] Waiting 0.7s for filter sections to expand (filter_expand_wait)
2026-05-17 23:08:29,812 - KolHalashonScraper - INFO - [DELAY] Waiting 0.7s after clicking 'display more' buttons (filter_expand_wait)
2026-05-17 23:08:30,525 - KolHalashonScraper - INFO - Expanded filter sections: clicked 2 'display more' buttons
2026-05-17 23:08:31,535 - KolHalashonScraper - INFO - DOM filter debug: {'displayAllSections': 2, 'filterContents': 5, 'filterOptions': 45, 'totalCheckboxes': 43, 'type2Checkboxes': 33}
2026-05-17 23:08:31,544 - KolHalashonScraper - INFO - Extracted 31 speakers from DOM filter checkboxes
2026-05-17 23:08:31,544 - KolHalashonScraper - INFO - Extracted 31 speakers from URL filter dropdown
2026-05-17 23:08:31,554 - social_media_monitor.scrapers.kolhalashon_scraper - INFO - Speaker sync complete: 0 new, 31 updated, 517 total
2026-05-17 23:08:31,554 - KolHalashonScraper - INFO - [DELAY] Waiting 1.0s between URLs (shiur_url_delay)
2026-05-17 23:08:32,555 - KolHalashonScraper - INFO - Processing URL (id=6): https://www.kolhalashon.com/en/regularSite/searchResults/100/49/Meilah?urlFilters=3:3%7C20:13%7C|Harav Shimon Spitzer
2026-05-17 23:08:33,963 - KolHalashonScraper - INFO - Page loaded with 49 shiur containers after 0s
2026-05-17 23:08:33,964 - KolHalashonScraper - INFO - [DELAY] Waiting 60.0s for page content to stabilize (shiur_found_wait)
2026-05-17 23:09:39,999 - KolHalashonScraper - INFO - Processing page 1 of URL (pagination shows: page 1 of 3, has_next=True)
2026-05-17 23:09:39,999 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:09:46,008 - KolHalashonScraper - INFO - Extracted 24 shiurim from DOM
2026-05-17 23:09:46,008 - KolHalashonScraper - INFO - Page 1: Browser extracted 24 shiurim
2026-05-17 23:09:46,015 - KolHalashonScraper - INFO - Navigating to page 2...
2026-05-17 23:09:46,034 - KolHalashonScraper - INFO - [DELAY] Waiting 10s for page to reload after clicking next button
2026-05-17 23:10:03,077 - KolHalashonScraper - INFO - Processing page 2 of URL (pagination shows: page 2 of 3, has_next=True)
2026-05-17 23:10:03,077 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:10:09,091 - KolHalashonScraper - INFO - Extracted 24 shiurim from DOM
2026-05-17 23:10:09,091 - KolHalashonScraper - INFO - Page 2: Browser extracted 24 shiurim
2026-05-17 23:10:09,101 - KolHalashonScraper - INFO - Navigating to page 3...
2026-05-17 23:10:09,131 - KolHalashonScraper - INFO - [DELAY] Waiting 10s for page to reload after clicking next button
2026-05-17 23:10:26,174 - KolHalashonScraper - INFO - Processing page 3 of URL (pagination shows: page 3 of 3, has_next=False)
2026-05-17 23:10:26,174 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:10:32,184 - KolHalashonScraper - INFO - Extracted 4 shiurim from DOM
2026-05-17 23:10:32,184 - KolHalashonScraper - INFO - Page 3: Browser extracted 4 shiurim
2026-05-17 23:10:32,191 - KolHalashonScraper - INFO - No more pages available after page 3
2026-05-17 23:10:32,192 - KolHalashonScraper - INFO - URL complete: 52 shiurim extracted from 3 page(s), 208 new total so far
2026-05-17 23:10:32,273 - KolHalashonScraper - INFO - [DELAY] Waiting 0.7s for filter sections to expand (filter_expand_wait)
2026-05-17 23:10:33,006 - KolHalashonScraper - INFO - [DELAY] Waiting 0.7s after clicking 'display more' buttons (filter_expand_wait)
2026-05-17 23:10:33,717 - KolHalashonScraper - INFO - Expanded filter sections: clicked 2 'display more' buttons
2026-05-17 23:10:34,728 - KolHalashonScraper - INFO - DOM filter debug: {'displayAllSections': 2, 'filterContents': 5, 'filterOptions': 45, 'totalCheckboxes': 43, 'type2Checkboxes': 33}
2026-05-17 23:10:34,740 - KolHalashonScraper - INFO - Extracted 31 speakers from DOM filter checkboxes
2026-05-17 23:10:34,740 - KolHalashonScraper - INFO - Extracted 31 speakers from URL filter dropdown
2026-05-17 23:10:34,751 - social_media_monitor.scrapers.kolhalashon_scraper - INFO - Speaker sync complete: 0 new, 31 updated, 517 total
2026-05-17 23:10:34,751 - KolHalashonScraper - INFO - [DELAY] Waiting 1.0s between URLs (shiur_url_delay)
2026-05-17 23:10:35,752 - KolHalashonScraper - INFO - Processing URL (id=7): https://www.kolhalashon.com/en/regularSite/searchResults/100/49/Meilah?urlFilters=3:3%7C20:13%7C|Harav Shmuel Dovid Friedman
2026-05-17 23:10:37,133 - KolHalashonScraper - INFO - Page loaded with 49 shiur containers after 0s
2026-05-17 23:10:37,133 - KolHalashonScraper - INFO - [DELAY] Waiting 60.0s for page content to stabilize (shiur_found_wait)
2026-05-17 23:11:43,169 - KolHalashonScraper - INFO - Processing page 1 of URL (pagination shows: page 1 of 3, has_next=True)
2026-05-17 23:11:43,169 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-05-17 23:11:49,182 - KolHalashonScraper - INFO - Extracted 24 shiurim from DOM
2026-05-17 23:11:49,182 - KolHalashonScraper - INFO - Page 1: Browser extracted 24 shiurim
2026-05-17 23:11:49,191 - KolHalashonScraper - INFO - Navigating to page 2...
2026-05-17 23:11:49,217 - KolHalashonScraper - INFO - [DELAY] Waiting 10s for page to reload after clicking next button
2026-05-17 23:12:06,263 - KolHalashonScraper - INFO - Processing page 2 of URL (pagination shows: page 2 of 3, has_next=True)
2026-05-17 23:12:06,264 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
/home/linuxbrew/.linuxbrew/opt/python@3.14/lib/python3.14/multiprocessing/resource_tracker.py:396: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {'/mp-ri3ux3qg'}
warnings.warn(