โ† Logs

social_media_monitor_2026-10-05.log

2026-10-05 01:00:07,466 - Yiddish24Scraper - INFO - Page 1: 13/100 new items
2026-10-05 01:00:07,466 - Yiddish24Scraper - INFO - Low new item ratio (13.0%), stopping pagination
2026-10-05 01:00:07,466 - Yiddish24Scraper - INFO - Total Yiddish24 results found: 13
2026-10-05 01:00:14,436 - IveltScraper - INFO - Found 50 active topics on ivelt
2026-10-05 01:00:20,607 - IveltScraper - INFO - Filtering post containing exclude term: '*ื‘ื™ื˜ืข ืžืื›ื˜ ื–ื™ื›ืขืจ ื‘ืœื•ื™ื– ืฆื• ื‘ืืจื™ื›ื˜ืŸ ืงืจืื ื˜ืข ื ื™ื™ืขืก*'
2026-10-05 01:00:34,038 - IveltScraper - INFO - ivelt scrape complete. Checked 5 topics.
2026-10-05 01:00:34,039 - IveltScraper - INFO - All forums scrape complete. Found 8 new posts.
2026-10-05 01:00:34,039 - __main__ - INFO - Scraper rss completed: 0 results
2026-10-05 01:00:34,039 - __main__ - INFO - Scraper yiddish24 completed: 13 results
2026-10-05 01:00:34,040 - __main__ - INFO - Scraper ivelt completed: 8 results
2026-10-05 01:00:35,793 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-10-05 01:00:35.793738
2026-10-05 01:00:35,794 - __main__ - INFO - Email sent successfully
2026-10-05 01:00:35,794 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-10-05 01:00:35,794 - __main__ - INFO - Monitor complete. Found 21 total results: 13 new yiddish24 results, 8 new ivelt results.
2026-10-05 01:00:35,794 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-10-05 01:00:35,794 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-10-05 01:00:35,944 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-10-05 01:00:35,944 - __main__ - INFO - Closed all database connections
2026-10-05 01:00:35,946 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-10-05 01:00:35,946 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-10-05 02:00:01,975 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-10-05.log
2026-10-05 02:00:01,975 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-10-05.log
2026-10-05 02:00:01,976 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-10-05.log
2026-10-05 02:00:01,976 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-10-05 02:00:02,005 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-10-05 02:00:02,005 - __main__ - INFO - Configuration loaded and validated successfully
2026-10-05 02:00:02,007 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-10-05 02:00:02,008 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-10-05 02:00:02,022 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-10-05 02:00:02,022 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-10-05 02:00:02,074 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully (UA='SocialMediaMonitor/1.0')
2026-10-05 02:00:02,076 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=1, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-10-05 02:00:02,076 - __main__ - INFO - RSS scraper initialized and enabled
2026-10-05 02:00:02,082 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-10-05 02:00:02,082 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-10-05 02:00:02,082 - __main__ - INFO - Yiddish24 scraper initialized and enabled
2026-10-05 02:00:02,087 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-10-05 02:00:02,092 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-10-05 02:00:02,092 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=13
2026-10-05 02:00:02,095 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-10-05 02:00:02,095 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=2
2026-10-05 02:00:02,095 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=True, exclude_filters=1, include_filters=0
2026-10-05 02:00:02,097 - AllDafScraper - INFO - All Daf scraper initialized: enabled=True sections=['Daf Yomi Shiurim', 'Supplemental Clips', 'Reference Materials'] skip_terms=0 include_terms=0
2026-10-05 02:00:02,097 - __main__ - INFO - Ivelt scraper initialized and enabled
2026-10-05 02:00:02,097 - __main__ - INFO - Running 3 scrapers concurrently...
2026-10-05 02:00:02,097 - Yiddish24Scraper - INFO - Starting Yiddish24 scraping (max 10 pages, limit 100 items)
2026-10-05 02:00:02,183 - Yiddish24Scraper - INFO - Fetching Yiddish24 page 1 with data_id='latest', page_limit=100
2026-10-05 02:00:02,188 - IveltScraper - INFO - Loaded 182 include, 4 exclude filters from DB topic_filters
2026-10-05 02:00:02,188 - IveltScraper - INFO - Fetching ivelt active topics from https://www.ivelt.com/forum/search.php?search_id=active_topics...
2026-10-05 02:00:02,189 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.theyeshivaworld.com/feed
2026-10-05 02:00:02,434 - RSSFeedScraper - INFO - Total RSS/HTML results: 0
2026-10-05 02:00:03,801 - httpx - INFO - HTTP Request: POST https://www.yiddish24.com/ajax/cat_pagination.php "HTTP/1.1 200 OK"
2026-10-05 02:00:04,041 - Yiddish24Scraper - INFO - Received response with 541688 characters
2026-10-05 02:00:04,044 - Yiddish24Scraper - INFO - Extracted HTML from JSON result field (317010 chars)
2026-10-05 02:00:04,046 - Yiddish24Scraper - INFO - Saved HTML debug file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/debug/yiddish24_page1_response.html
2026-10-05 02:00:04,116 - Yiddish24Scraper - INFO - Found 100 potential article elements on page 1
2026-10-05 02:00:04,156 - Yiddish24Scraper - INFO - Page 1 extraction: 100 successful, 0 failed out of 100 elements
2026-10-05 02:00:04,156 - Yiddish24Scraper - INFO - Parsed 100 articles from HTML on page 1
2026-10-05 02:00:04,220 - Yiddish24Scraper - INFO - Term 'ืื•ื™ืกืฆื•ื’ืŸ ืื•ืŸ ืงืจื™ืฉืงืขืœืขืš' matched in DESCRIPTION: 'ืื•ื™ืกืฆื•ื’ืŸ ืื•ืŸ ืงืจื™ืฉืงืขืœืขืš - ืื ื”ื™ื™ื‘ืŸ ื—ืง ืžื™ื˜ ื ืคืจื™ืฉืงื™ื™ื˜'
2026-10-05 02:00:04,220 - Yiddish24Scraper - INFO - Downloading 1 media file(s) for article: ื”ื›ืœ ื‘ื›ืœ
2026-10-05 02:00:04,721 - social_media_monitor.utils.file_downloader - INFO - Starting download from: https://cloudfront.yiddish24.com/__________________________________________699645496.mp3
2026-10-05 02:00:05,907 - social_media_monitor.utils.file_downloader - INFO - Successfully downloaded: Y2___________________________________________699645496.mp3 (7.6 MB)
2026-10-05 02:00:05,907 - social_media_monitor.utils.file_downloader - INFO - Organized audio file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio (audio/mpeg)
2026-10-05 02:00:05,919 - social_media_monitor.utils.media_tagger - INFO - Tagged Y2___________________________________________699645496.mp3: title, artist, album, comment, source_url, genre, publisher
2026-10-05 02:00:05,919 - Yiddish24Scraper - INFO - Renamed downloaded file to include match term: Y2___________________________________________699645496_ืื•ื™ืกืฆื•ื’ืŸ_ืื•ืŸ_ืงืจื™ืฉืงืขืœืขืš.mp3
2026-10-05 02:00:05,930 - Yiddish24Scraper - INFO - Term 'ืžืขืจืงื•ื•ื™ืจื“ื™ื’ -' matched in DESCRIPTION: 'ืžืขืจืงื•ื•ื™ืจื“ื™ื’ - ืืจื™ื™ื ื‘ืจืขื ื’ืขืŸ ื ื™ื’ื•ื ื™ื ืคื•ื ื“ืจื•ื™ืกืŸ'
2026-10-05 02:00:05,931 - Yiddish24Scraper - INFO - Downloading 1 media file(s) for article: ื”ื›ืœ ื‘ื›ืœ
2026-10-05 02:00:06,431 - social_media_monitor.utils.file_downloader - INFO - Starting download from: https://cloudfront.yiddish24.com/__________________________________________263948292.mp3
2026-10-05 02:00:08,709 - social_media_monitor.utils.file_downloader - INFO - Successfully downloaded: Y2___________________________________________263948292.mp3 (57.4 MB)
2026-10-05 02:00:08,710 - social_media_monitor.utils.file_downloader - INFO - Organized audio file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio (audio/mpeg)
2026-10-05 02:00:08,730 - social_media_monitor.utils.media_tagger - INFO - Tagged Y2___________________________________________263948292.mp3: title, artist, album, comment, source_url, genre, publisher
2026-10-05 02:00:08,731 - Yiddish24Scraper - INFO - Renamed downloaded file to include match term: Y2___________________________________________263948292_ืžืขืจืงื•ื•ื™ืจื“ื™ื’_-.mp3
2026-10-05 02:00:09,409 - Yiddish24Scraper - INFO - Page 1 summary: 8 new out of 100 total articles
2026-10-05 02:00:09,410 - Yiddish24Scraper - INFO - Page 1: 8/100 new items
2026-10-05 02:00:09,410 - Yiddish24Scraper - INFO - Low new item ratio (8.0%), stopping pagination
2026-10-05 02:00:09,410 - Yiddish24Scraper - INFO - Total Yiddish24 results found: 8
2026-10-05 02:00:14,003 - IveltScraper - INFO - Found 50 active topics on ivelt
2026-10-05 02:00:31,724 - IveltScraper - INFO - Filtering post containing exclude term: '*ื‘ื™ื˜ืข ืžืื›ื˜ ื–ื™ื›ืขืจ ื‘ืœื•ื™ื– ืฆื• ื‘ืืจื™ื›ื˜ืŸ ืงืจืื ื˜ืข ื ื™ื™ืขืก*'
2026-10-05 02:00:35,068 - IveltScraper - INFO - Filtering post containing exclude term: '*ื‘ื™ื˜ืข ืžืื›ื˜ ื–ื™ื›ืขืจ ื‘ืœื•ื™ื– ืฆื• ื‘ืืจื™ื›ื˜ืŸ ืงืจืื ื˜ืข ื ื™ื™ืขืก*'
2026-10-05 02:00:46,210 - IveltScraper - INFO - ivelt scrape complete. Checked 6 topics.
2026-10-05 02:00:46,210 - IveltScraper - INFO - All forums scrape complete. Found 28 new posts.
2026-10-05 02:00:46,210 - __main__ - INFO - Scraper rss completed: 0 results
2026-10-05 02:00:46,211 - __main__ - INFO - Scraper yiddish24 completed: 8 results
2026-10-05 02:00:46,211 - __main__ - INFO - Scraper ivelt completed: 28 results
2026-10-05 02:00:47,149 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-10-05 02:00:47.149340
2026-10-05 02:00:47,149 - __main__ - INFO - Email sent successfully
2026-10-05 02:00:47,149 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-10-05 02:00:47,150 - __main__ - INFO - Monitor complete. Found 36 total results: 8 new yiddish24 results, 28 new ivelt results.
2026-10-05 02:00:47,150 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-10-05 02:00:47,150 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-10-05 02:00:47,219 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-10-05 02:00:47,220 - __main__ - INFO - Closed all database connections
2026-10-05 02:00:47,222 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-10-05 02:00:47,222 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-10-05 02:30:02,129 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-10-05.log
2026-10-05 02:30:02,129 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-10-05.log
2026-10-05 02:30:02,130 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-10-05.log
2026-10-05 02:30:02,130 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-10-05 02:30:02,164 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-10-05 02:30:02,164 - __main__ - INFO - Configuration loaded and validated successfully
2026-10-05 02:30:02,167 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-10-05 02:30:02,168 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-10-05 02:30:02,181 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-10-05 02:30:02,181 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-10-05 02:30:02,183 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully (UA='SocialMediaMonitor/1.0')
2026-10-05 02:30:02,184 - __main__ - INFO - GitHub scraper initialized and enabled
2026-10-05 02:30:02,184 - __main__ - INFO - GitHub feed scraper initialized and enabled
2026-10-05 02:30:02,184 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=1, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-10-05 02:30:02,188 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-10-05 02:30:02,188 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-10-05 02:30:02,193 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-10-05 02:30:02,197 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-10-05 02:30:02,197 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=13
2026-10-05 02:30:02,200 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-10-05 02:30:02,200 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=2
2026-10-05 02:30:02,200 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=True, exclude_filters=1, include_filters=0
2026-10-05 02:30:02,202 - AllDafScraper - INFO - All Daf scraper initialized: enabled=True sections=['Daf Yomi Shiurim', 'Supplemental Clips', 'Reference Materials'] skip_terms=0 include_terms=0
2026-10-05 02:30:02,202 - __main__ - INFO - YouTube Intel scraper initialized and enabled
2026-10-05 02:30:02,202 - __main__ - INFO - Running 3 scrapers concurrently...
2026-10-05 02:30:02,202 - GitHubScraper - INFO - Searching GitHub for terms: ['"Hermes Agent"', '"grok bot OR grokbot"', '"muse NOT music NOT museum"', '"prime agent"', 'paperclip']
2026-10-05 02:30:02,812 - httpx - INFO - HTTP Request: GET https://api.github.com/user "HTTP/1.1 200 OK"
2026-10-05 02:30:02,813 - GitHubFeedScraper - INFO - GitHub auth preflight passed for abeperl; token scopes: not reported (fine-grained token)
2026-10-05 02:30:03,233 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22Hermes+Agent%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-10-05 02:30:03,260 - GitHubScraper - INFO - Found 668 GitHub repositories for term: "Hermes Agent"
2026-10-05 02:30:03,851 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22grok+bot+OR+grokbot%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-10-05 02:30:03,852 - GitHubScraper - INFO - Found 0 GitHub repositories for term: "grok bot OR grokbot"
2026-10-05 02:30:03,854 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=1 "HTTP/1.1 200 OK"
2026-10-05 02:30:05,036 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=2 "HTTP/1.1 200 OK"
2026-10-05 02:30:05,895 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22muse+NOT+music+NOT+museum%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-10-05 02:30:05,896 - GitHubScraper - INFO - Found 0 GitHub repositories for term: "muse NOT music NOT museum"
2026-10-05 02:30:06,134 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=3 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:06,135 - GitHubFeedScraper - INFO - GitHub account feed page 3 unchanged (ETag)
2026-10-05 02:30:06,700 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22prime+agent%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-10-05 02:30:06,704 - GitHubScraper - INFO - Found 44 GitHub repositories for term: "prime agent"
2026-10-05 02:30:07,882 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=paperclip+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-10-05 02:30:07,886 - GitHubScraper - INFO - Found 271 GitHub repositories for term: paperclip
2026-10-05 02:30:08,025 - GitHubScraper - INFO - Total GitHub results found: 0
2026-10-05 02:30:09,753 - httpx - INFO - HTTP Request: POST https://api.github.com/graphql "HTTP/1.1 200 OK"
2026-10-05 02:30:10,147 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Biswaspr1022/KodeshMode/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:10,190 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/efiles-star/ShasTracker/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:10,263 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/akivayevdayev-debug/Shelah_app/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:10,281 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/yehoshuayaakov/mishnayos-signup/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:10,555 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/nnanach/torah-bot/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:10,615 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Oranburg/havruta/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:10,644 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/azernjj/talmudai/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:10,659 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/hitchin999/YidCal/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:10,895 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/ethanpil/daily-zmanim/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:10,966 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Chabadrt/chabad-cms-backend/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:10,997 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/rabbishimon613-lang/chabad-tracker/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:11,001 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/adunnmobilenotary-gif/halacha-search/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:11,182 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/IsraelCIL/SiddurAmIsrael/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:11,298 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/ElikoGrins/Halacha-bot/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:11,349 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/ronrefael/kav-siddur-site/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:11,576 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/EzraBrand/talmud-nlp-indexer/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:11,617 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/kdroidFilter/Zayit/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:11,687 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/brainheart/tanakh-contabulate/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:11,871 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Hbleiberg/Hebrew_Blender/events?per_page=10 "HTTP/1.1 200 OK"
2026-10-05 02:30:11,903 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Esoterics611/maayan/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:11,975 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/stenagmois-byte/SimHebrew-Concordance/events?per_page=10 "HTTP/1.1 200 OK"
2026-10-05 02:30:12,073 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/naorbrown/daf-yomi-history-bot/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:12,258 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/jepick/talmud-map/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:12,325 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/wuisabel-gif/Aleph-Bet/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:12,540 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/paperlesspaper/integrations/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:12,573 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/shloimie-beep/bnei-neviim-academy/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:12,753 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Asifah-Analytics-C-A-V-E/asifah-analytics/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:12,818 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/complytime/complytime-collector-components/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:12,937 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/DavidPatlas-AI/tehillim-russian-site/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:13,028 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/dniasoff/learning-tracker/events?per_page=10 "HTTP/1.1 200 OK"
2026-10-05 02:30:13,304 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/hytric/NLP_term_project/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:13,322 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/dmo18/MySugya/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:13,364 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/yossi-weinberger/ten10/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:13,385 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/RonTurrentine/KJV-Strongs-EBook/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:13,585 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/zarbinco/zarbin-seo/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:13,816 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/v97969181-commits/master-pdf-editor-legacy-release/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:13,854 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/AUX440/liquidtext-ultimate-patch-kit/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:13,878 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/duvarakesh06/FAB-Subtitler-12.1-Offline-Release/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:13,933 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/elmadanimohammed78-del/daily-flow-landing/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:14,270 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/apusaha0011/MenuSideKick/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:14,293 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Roger0129/bitwarden-2024-8-0-patchless-install/events?per_page=10 "HTTP/1.1 200 OK"
2026-10-05 02:30:14,300 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/rafistrauss/commons-announcements/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:14,344 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Aljademaratas/readiris-product-code-generator/events?per_page=10 "HTTP/1.1 200 OK"
2026-10-05 02:30:14,646 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/semiagenticRob/100Reps/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:14,666 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/benemanuel/status/events?per_page=10 "HTTP/1.1 200 OK"
2026-10-05 02:30:14,695 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/chfle/chfle/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:14,794 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/revanite-io/pvtr-gcp-cloud-storage/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:14,933 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Jossifresben/translation-alignment/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:15,005 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/egkristi/ToponymiaEuropaea/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:15,022 - YouTubeIntelScraper - INFO - YouTube Intel found 34 candidates and 6 new videos
2026-10-05 02:30:15,200 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/hebcal/hebcal/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:15,279 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/KosherJava/zmanim/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:15,304 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Israel-Kli/shabbat-timetable/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:15,461 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/woodpeckerbros/watch-reminder/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:15,597 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/hebcal/learning/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:15,734 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/svenalot-ui/sidur-native/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:15,829 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/hebcal/hebcal-go/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:15,941 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Sefaria/ai-chatbot/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:16,157 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/levidao-eth/Torah/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:16,181 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/joelcw/intsayt/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:16,334 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/AdamYairLee/torah-korean-translation/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:16,421 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/complytime/complytime-policies/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:16,511 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/zakdev26/sefaria-links/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:16,571 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/ainihonuragawa-lang/TalmudFinance/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:16,648 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/vaisisrael/gemara-benichuta/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:16,763 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Yiddish-Dialect-Dictionary/-brukhe-machine/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:16,792 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/SnirRadomsky/daf-yomi/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:16,955 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/elconde/kosher-races/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,052 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/jacopovirga/kindle-zmanim/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,062 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Selecr-Surplus/minyan-finder/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,069 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/robmoffat/FOSS-Contribution-Gemara/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,270 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/aharonium/opensiddur.org/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,365 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/theOnionsAreWatching/Shtigletz/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,441 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/mig2/mishnah-style/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,461 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/naorbrown/peninei-halacha-yomi-bot/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,623 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/vistaTeam/satmar/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,630 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/skilead770/stoliskikosher/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,766 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/mak-sandboxuser/chabad-react-and-supabase/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,836 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/a0556770172-png/koshermat/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,978 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/bnq9705-creator/siddur/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:17,982 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Kosher-T/fannexus/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,106 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/yosefmalul52-lang/costa-rica-kosher-retreat/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,130 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/RavBogard/shirei-tfilah/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,247 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/shliachflow/chabad-southern-delaware-camp/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,283 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/ReuvenB-S/kosher-browser-data/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,402 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/AviMish10/tanach-map/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,407 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/TanachatGKRB/TanachatGKRB.github.io/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,558 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/TamamTamar/chabad-yafo/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,571 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/DeepSynchronicity/chabadofhurlingham/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,710 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/TG-CONSULTANT/tanachok-stock-FG1/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,761 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/semiagenticRob/digital-siddur/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,927 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/TG-CONSULTANT/tanachok-stock-mat/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,983 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/orelzion/daily-tanach/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:18,988 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Frish-Gebakn/kol-satmar-website/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:19,081 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/satmarthubby-blip/Satmarthubby/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:19,259 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/satmaribianca05-eng/AFSD-Satmari-Bianca/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:19,275 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/akshatasatmar/WeatherServiceApi/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:19,337 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/elchunenfreund/satmar-matzos-app/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:19,509 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/Sefaria/Sefaria-Export/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:19,607 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/mritunjaysharma2004/SATMARG-An-Online-Pilgrimage-planner-/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:19,898 - httpx - INFO - HTTP Request: GET https://api.github.com/repos/hebcal/hebcal-es6/events?per_page=10 "HTTP/1.1 304 Not Modified"
2026-10-05 02:30:19,941 - GitHubFeedScraper - INFO - GitHub feed healthy: login=abeperl fetched=200 account_new=0 list_repos=100 list_new=0 private_new=0 total_new=0
2026-10-05 02:30:19,943 - __main__ - INFO - Scraper github completed: 0 results
2026-10-05 02:30:19,943 - __main__ - INFO - Scraper github_feed completed: 0 results
2026-10-05 02:30:19,943 - __main__ - INFO - Scraper youtube_intel completed: 6 results
2026-10-05 02:30:25,624 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-10-05 02:30:25.624583
2026-10-05 02:30:25,624 - __main__ - INFO - Email sent successfully
2026-10-05 02:30:25,627 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-10-05 02:30:25,628 - __main__ - INFO - Monitor complete. Found 6 total results: 6 new youtube intel results.
2026-10-05 02:30:25,628 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-10-05 02:30:25,628 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-10-05 02:30:25,660 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-10-05 02:30:25,660 - __main__ - INFO - Closed all database connections
2026-10-05 02:30:25,663 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-10-05 02:30:25,663 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-10-05 03:00:02,552 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-10-05.log
2026-10-05 03:00:02,553 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-10-05.log
2026-10-05 03:00:02,553 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-10-05.log
2026-10-05 03:00:02,553 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-10-05 03:00:02,555 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-10-05.log
2026-10-05 03:00:02,556 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-10-05.log
2026-10-05 03:00:02,556 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-10-05.log
2026-10-05 03:00:02,556 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-10-05 03:00:02,581 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-10-05 03:00:02,581 - __main__ - INFO - Configuration loaded and validated successfully
2026-10-05 03:00:02,584 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-10-05 03:00:02,584 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-10-05 03:00:02,589 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-10-05 03:00:02,591 - __main__ - INFO - Configuration loaded and validated successfully
2026-10-05 03:00:02,594 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-10-05 03:00:02,606 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-10-05 03:00:02,607 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-10-05 03:00:02,627 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-10-05 03:00:02,640 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-10-05 03:00:02,641 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-10-05 03:00:02,668 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully (UA='SocialMediaMonitor/1.0')
2026-10-05 03:00:02,670 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=1, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-10-05 03:00:02,678 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-10-05 03:00:02,678 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-10-05 03:00:02,684 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-10-05 03:00:02,693 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-10-05 03:00:02,693 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=13
2026-10-05 03:00:02,696 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-10-05 03:00:02,696 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=2
2026-10-05 03:00:02,696 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=True, exclude_filters=1, include_filters=0
2026-10-05 03:00:02,696 - __main__ - INFO - Kol Halashon scraper initialized and enabled
2026-10-05 03:00:02,699 - AllDafScraper - INFO - All Daf scraper initialized: enabled=True sections=['Daf Yomi Shiurim', 'Supplemental Clips', 'Reference Materials'] skip_terms=0 include_terms=0
2026-10-05 03:00:02,699 - __main__ - INFO - All Daf scraper initialized and enabled
2026-10-05 03:00:02,699 - __main__ - INFO - Running 2 scrapers concurrently...
2026-10-05 03:00:02,699 - KolHalashonScraper - INFO - Starting Kol Halashon scraping
2026-10-05 03:00:02,704 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully (UA='SocialMediaMonitor/1.0')
2026-10-05 03:00:02,706 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=1, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-10-05 03:00:02,706 - __main__ - INFO - RSS scraper initialized and enabled
2026-10-05 03:00:02,712 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-10-05 03:00:02,713 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-10-05 03:00:02,713 - __main__ - INFO - Yiddish24 scraper initialized and enabled
2026-10-05 03:00:02,716 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-10-05 03:00:02,721 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-10-05 03:00:02,721 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=13
2026-10-05 03:00:02,723 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-10-05 03:00:02,723 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=2
2026-10-05 03:00:02,723 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=True, exclude_filters=1, include_filters=0
2026-10-05 03:00:02,726 - AllDafScraper - INFO - All Daf scraper initialized: enabled=True sections=['Daf Yomi Shiurim', 'Supplemental Clips', 'Reference Materials'] skip_terms=0 include_terms=0
2026-10-05 03:00:02,726 - __main__ - INFO - Ivelt scraper initialized and enabled
2026-10-05 03:00:02,726 - __main__ - INFO - Running 3 scrapers concurrently...
2026-10-05 03:00:02,726 - Yiddish24Scraper - INFO - Starting Yiddish24 scraping (max 10 pages, limit 100 items)
2026-10-05 03:00:02,753 - KolHalashonScraper - INFO - Found current parsha: Bereshis (ื‘ืจืืฉื™ืช)
2026-10-05 03:00:02,754 - KolHalashonScraper - INFO - Using parsha URL template (id=1): https://www2.kolhalashon.com/regularSite/searchResults/1/{parshaid}/{parshaen}?u...
2026-10-05 03:00:02,754 - KolHalashonScraper - INFO - Parsha raw URL template: https://www2.kolhalashon.com/regularSite/searchResults/1/{parshaid}/{parshaen}?urlFilters=3:3%7C{speakersfilter}
2026-10-05 03:00:02,754 - KolHalashonScraper - INFO - Speaker IDs for URL id=1: [7098, 999121, 952143, 7651, 985996, 7643, 7124, 934613, 2393, 7031]
2026-10-05 03:00:02,754 - KolHalashonScraper - INFO - Parsha speaker filter string: [7098, 999121, 952143, 7651, 985996, 7643, 7124, 934613, 2393, 7031]
2026-10-05 03:00:02,754 - KolHalashonScraper - INFO - Parsha encoded speaker filter: 2:7098%7C2:999121%7C2:952143%7C2:7651%7C2:985996%7C2:7643%7C2:7124%7C2:934613%7C2:2393%7C2:7031%7C
2026-10-05 03:00:02,754 - KolHalashonScraper - INFO - Parsha processed URL: https://www2.kolhalashon.com/regularSite/searchResults/1/1/Bereshit?urlFilters=3:3%7C2:7098%7C2:999121%7C2:952143%7C2:7651%7C2:985996%7C2:7643%7C2:7124%7C2:934613%7C2:2393%7C2:7031%7C
2026-10-05 03:00:02,754 - KolHalashonScraper - INFO - Current parsha: Bereshis (ื‘ืจืืฉื™ืช)
2026-10-05 03:00:02,755 - KolHalashonScraper - INFO - Processing parsha URL (id=1)
2026-10-05 03:00:02,755 - KolHalashonScraper - INFO - Processing 3 total URLs (1 parsha + 2 search)
2026-10-05 03:00:02,755 - KolHalashonScraper - INFO - Replacing parsha placeholders with id=1, name=Bereshit, in_humash=1
2026-10-05 03:00:02,757 - KolHalashonScraper - INFO - Found 1 scheduled daf(s) for 1-day range: [('2026-10-05', 59, 28)]
2026-10-05 03:00:02,757 - KolHalashonScraper - INFO - Generated daf URL for date=2026-10-05, masechet_id=59, name=Nida, daf_no=28
2026-10-05 03:00:02,757 - KolHalashonScraper - INFO - Scraping all 3 URLs with unified method...
2026-10-05 03:00:02,757 - KolHalashonScraper - INFO - Starting multi-URL browser scrape with 3 URLs, max 10 pages per URL (download_inline=True)
2026-10-05 03:00:02,808 - KolHalashonScraper - INFO - Detected installed Chrome major version 154 from '/usr/bin/google-chrome' (output: 'Google Chrome 154.0.8037.57')
2026-10-05 03:00:02,808 - KolHalashonScraper - INFO - Pinning undetected-chromedriver version_main=154 to detected installed Chrome major version
2026-10-05 03:00:02,808 - KolHalashonScraper - INFO - Running browser in headless mode
2026-10-05 03:00:02,809 - KolHalashonScraper - INFO - Browser download directory set to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio
2026-10-05 03:00:02,809 - KolHalashonScraper - INFO - Using undetected-chromedriver for Cloudflare bypass (attempt 1/3)
2026-10-05 03:00:02,822 - Yiddish24Scraper - INFO - Fetching Yiddish24 page 1 with data_id='latest', page_limit=100
2026-10-05 03:00:02,826 - IveltScraper - INFO - Loaded 182 include, 4 exclude filters from DB topic_filters
2026-10-05 03:00:02,826 - IveltScraper - INFO - Fetching ivelt active topics from https://www.ivelt.com/forum/search.php?search_id=active_topics...
2026-10-05 03:00:02,829 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.theyeshivaworld.com/feed
2026-10-05 03:00:03,175 - RSSFeedScraper - INFO - Total RSS/HTML results: 0
2026-10-05 03:00:03,677 - httpx - INFO - HTTP Request: POST https://www.yiddish24.com/ajax/cat_pagination.php "HTTP/1.1 200 OK"
2026-10-05 03:00:03,924 - Yiddish24Scraper - INFO - Received response with 541688 characters
2026-10-05 03:00:03,928 - Yiddish24Scraper - INFO - Extracted HTML from JSON result field (317010 chars)
2026-10-05 03:00:03,931 - Yiddish24Scraper - INFO - Saved HTML debug file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/debug/yiddish24_page1_response.html
2026-10-05 03:00:04,015 - Yiddish24Scraper - INFO - Found 100 potential article elements on page 1
2026-10-05 03:00:04,057 - Yiddish24Scraper - INFO - Page 1 extraction: 100 successful, 0 failed out of 100 elements
2026-10-05 03:00:04,057 - Yiddish24Scraper - INFO - Parsed 100 articles from HTML on page 1
2026-10-05 03:00:04,343 - undetected_chromedriver.patcher - INFO - patching driver executable /home/openclaw/.local/share/undetected_chromedriver/undetected_chromedriver
2026-10-05 03:00:04,732 - KolHalashonScraper - INFO - Browser created for multi-URL scrape
2026-10-05 03:00:04,846 - KolHalashonScraper - INFO - Configured download behavior via Browser.setDownloadBehavior CDP for /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio
2026-10-05 03:00:04,884 - KolHalashonScraper - INFO - Configured download behavior via Page.setDownloadBehavior CDP for /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio
2026-10-05 03:00:04,884 - KolHalashonScraper - INFO - STEP 1 login: opening base URL https://www2.kolhalashon.com before processing target URLs
2026-10-05 03:00:04,998 - Yiddish24Scraper - INFO - Page 1 summary: 0 new out of 100 total articles
2026-10-05 03:00:04,998 - Yiddish24Scraper - INFO - Page 1: 0/100 new items
2026-10-05 03:00:04,999 - Yiddish24Scraper - INFO - Low new item ratio (0.0%), stopping pagination
2026-10-05 03:00:04,999 - Yiddish24Scraper - INFO - Total Yiddish24 results found: 0
2026-10-05 03:00:05,852 - KolHalashonScraper - INFO - Processing URL (id=1): https://www2.kolhalashon.com/regularSite/searchResults/1/1/Bereshit?urlFilters=3:3%7C{speakersfilter}
2026-10-05 03:00:05,852 - KolHalashonScraper - INFO - Replaced speakersfilter placeholder with 10 speaker filter(s)
2026-10-05 03:00:05,852 - KolHalashonScraper - INFO - Final processed URL: https://www2.kolhalashon.com/regularSite/searchResults/1/1/Bereshit?urlFilters=3:3%7C2:7098%7C2:999121%7C2:952143%7C2:7651%7C2:985996%7C2:7643%7C2:7124%7C2:934613%7C2:2393%7C2:7031%7C
2026-10-05 03:00:05,852 - KolHalashonScraper - INFO - STEP 2 visit processed URL: https://www2.kolhalashon.com/regularSite/searchResults/1/1/Bereshit?urlFilters=3:3%7C2:7098%7C2:999121%7C2:952143%7C2:7651%7C2:985996%7C2:7643%7C2:7124%7C2:934613%7C2:2393%7C2:7031%7C
2026-10-05 03:00:06,311 - KolHalashonScraper - INFO - Login required โ€“ attempting to log in
2026-10-05 03:00:06,356 - KolHalashonScraper - INFO - [DELAY] Waiting 1.0s after login button click
2026-10-05 03:00:09,393 - KolHalashonScraper - INFO - Submitted login form with stored credentials
2026-10-05 03:00:11,413 - KolHalashonScraper - INFO - Login confirmed
2026-10-05 03:00:15,274 - IveltScraper - INFO - Found 50 active topics on ivelt
2026-10-05 03:00:25,171 - KolHalashonScraper - INFO - Waiting for JS filtering (10s): 0 shiurs, 1138 filter options, reloading=True
2026-10-05 03:00:29,730 - KolHalashonScraper - INFO - JS filtering complete after 15s: 9 shiurs, 1131 filter options
2026-10-05 03:00:29,802 - KolHalashonScraper - INFO - [0s] Found 9 shiur containers (reloading=False), page state: {'bodyLength': 751239, 'hasAppRoot': True, 'hasShiurimDisplay': True, 'isReloading': False, 'readyState': 'complete'}
2026-10-05 03:00:29,803 - KolHalashonScraper - INFO - Page loaded with 9 shiur containers after 0s
2026-10-05 03:00:29,803 - KolHalashonScraper - INFO - [DELAY] Waiting 70.0s for page content to stabilize (shiur_found_wait)
2026-10-05 03:00:36,325 - IveltScraper - INFO - ivelt scrape complete. Checked 4 topics.
2026-10-05 03:00:36,325 - IveltScraper - INFO - All forums scrape complete. Found 12 new posts.
2026-10-05 03:00:36,326 - __main__ - INFO - Scraper rss completed: 0 results
2026-10-05 03:00:36,326 - __main__ - INFO - Scraper yiddish24 completed: 0 results
2026-10-05 03:00:36,326 - __main__ - INFO - Scraper ivelt completed: 12 results
2026-10-05 03:00:38,634 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-10-05 03:00:38.634897
2026-10-05 03:00:38,635 - __main__ - INFO - Email sent successfully
2026-10-05 03:00:38,635 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-10-05 03:00:38,635 - __main__ - INFO - Monitor complete. Found 12 total results: 12 new ivelt results.
2026-10-05 03:00:38,635 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-10-05 03:00:38,635 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-10-05 03:00:38,636 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-10-05 03:00:38,637 - __main__ - INFO - Closed all database connections
2026-10-05 03:00:38,638 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-10-05 03:00:38,638 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-10-05 03:01:45,843 - KolHalashonScraper - INFO - Processing page 1 of URL (pagination shows: page 1 of 1, has_next=False)
2026-10-05 03:01:45,844 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-10-05 03:01:51,857 - KolHalashonScraper - INFO - Extracted 9 shiurim from DOM
2026-10-05 03:01:51,858 - KolHalashonScraper - INFO - Page 1: Browser extracted 9 shiurim
2026-10-05 03:01:51,860 - KolHalashonScraper - INFO - Starting inline downloads for page 1 (reusing current page)
2026-10-05 03:01:51,860 - KolHalashonScraper - INFO - Found current parsha: Bereshis (ื‘ืจืืฉื™ืช)
2026-10-05 03:01:51,860 - KolHalashonScraper - INFO - Using parsha URL template (id=1): https://www2.kolhalashon.com/regularSite/searchResults/1/{parshaid}/{parshaen}?u...
2026-10-05 03:01:51,860 - KolHalashonScraper - INFO - Parsha raw URL template: https://www2.kolhalashon.com/regularSite/searchResults/1/{parshaid}/{parshaen}?urlFilters=3:3%7C{speakersfilter}
2026-10-05 03:01:51,861 - KolHalashonScraper - INFO - Speaker IDs for URL id=1: [7098, 999121, 952143, 7651, 985996, 7643, 7124, 934613, 2393, 7031]
2026-10-05 03:01:51,861 - KolHalashonScraper - INFO - Parsha speaker filter string: [7098, 999121, 952143, 7651, 985996, 7643, 7124, 934613, 2393, 7031]
2026-10-05 03:01:51,861 - KolHalashonScraper - INFO - Parsha encoded speaker filter: 2:7098%7C2:999121%7C2:952143%7C2:7651%7C2:985996%7C2:7643%7C2:7124%7C2:934613%7C2:2393%7C2:7031%7C
2026-10-05 03:01:51,861 - KolHalashonScraper - INFO - Parsha processed URL: https://www2.kolhalashon.com/regularSite/searchResults/1/1/Bereshit?urlFilters=3:3%7C2:7098%7C2:999121%7C2:952143%7C2:7651%7C2:985996%7C2:7643%7C2:7124%7C2:934613%7C2:2393%7C2:7031%7C
2026-10-05 03:01:51,861 - KolHalashonScraper - INFO - Download directory: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio
2026-10-05 03:01:51,861 - KolHalashonScraper - INFO - Using 1 provided shiur(s) for download (page-derived, in-memory)
2026-10-05 03:01:51,861 - KolHalashonScraper - INFO - Starting download of 1 shiurim (one_per_speaker=True)
2026-10-05 03:01:51,866 - KolHalashonScraper - INFO - Configured download behavior via Browser.setDownloadBehavior CDP for /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio
2026-10-05 03:01:51,869 - KolHalashonScraper - INFO - Configured download behavior via Page.setDownloadBehavior CDP for /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio
2026-10-05 03:01:51,870 - KolHalashonScraper - INFO - Reusing current page for downloads (no navigation)
2026-10-05 03:01:57,948 - KolHalashonScraper - INFO - Processing page 1 (pagination shows: page 1 of 1, has_next=False)
2026-10-05 03:01:57,960 - KolHalashonScraper - INFO - Page 1 has 9 shiur items
2026-10-05 03:01:57,960 - KolHalashonScraper - INFO - [p1:0] Skip (not_requested_or_already_downloaded): 40:05 Harav Mordechai Yehuda Friedman ืฉื™ืขื•ืจ ืขืœ ืคืจืฉืช ื”ืฉื‘ื•ืข ืคืจ by Unknown (ID: 42060870)
2026-10-05 03:01:57,960 - KolHalashonScraper - INFO - [p1:1] Skip (not_requested_or_already_downloaded): 1:04:49 Harav Elazar Nissan Rubin Parashat Bereshis, Part 1 by Unknown (ID: 39146933)
2026-10-05 03:01:57,961 - KolHalashonScraper - INFO - [p1:2] Skip (not_requested_or_already_downloaded): 12:22 Harav Yitzchak Aharon Goldhirsh ืคืจืฉืช ื‘ืจืืฉื™ืช, Parashat by Unknown (ID: 39014625)
2026-10-05 03:01:57,961 - KolHalashonScraper - INFO - [p1:3] Attempting download: 12:00:00 Harav Yitzchak Aharon Goldhirsh ื‘ืจืืฉื™ืช ืชืฉืคื’, Parash by Unknown (ID: 38929592)
2026-10-05 03:02:01,202 - KolHalashonScraper - INFO - Extracted media URL for shiur 38929592: https://www.kolhalashon.com/api/files/GetFileToPlay/38929592/1/1791298918.OCorM8I8fTsuseJdiDHUYpuiahzyAYO29CPTEvcqzX8
2026-10-05 03:02:01,446 - KolHalashonScraper - INFO - Starting HTTP download: https://www.kolhalashon.com/api/files/GetFileToPlay/38929592/1/1791298918.OCorM8I8fTsuseJdiDHUYpuiahzyAYO29CPTEvcqzX8
2026-10-05 03:02:01,449 - AllDafScraper - INFO - All Daf processing 1 scheduled daf(s)
2026-10-05 03:02:01,449 - AllDafScraper - INFO - All Daf scraping Nidah 28 (https://alldaf.org/todays-daf?cycle=dafyomi&masechta=Nidah&daf=28)
2026-10-05 03:02:01,685 - KolHalashonScraper - ERROR - HTTP 404 for https://www.kolhalashon.com/api/files/GetFileToPlay/38929592/1/1791298918.OCorM8I8fTsuseJdiDHUYpuiahzyAYO29CPTEvcqzX8
2026-10-05 03:02:01,686 - KolHalashonScraper - INFO - Direct HTTP download failed for 38929592, falling back to Selenium click
2026-10-05 03:02:01,707 - KolHalashonScraper - WARNING - Container not found for shiur 38929592
2026-10-05 03:02:01,707 - KolHalashonScraper - WARNING - [p1:3] Failed to download: 12:00:00 Harav Yitzchak Aharon Goldhirsh ื‘ืจืืฉื™ืช ืชืฉืคื’, Parash by Unknown (ID: 38929592) reason=not_found
2026-10-05 03:02:02,707 - KolHalashonScraper - INFO - [p1:4] Skip (not_requested_or_already_downloaded): 38:49 Harav Yitzchak Aharon Goldhirsh ืฉื™ืขื•ืจ 2-ืืžื•ื ื” ื•ื‘ื˜ื—ื•ืŸ ืœ by Unknown (ID: 36020138)
2026-10-05 03:02:02,707 - KolHalashonScraper - INFO - [p1:5] Skip (not_requested_or_already_downloaded): 41:01 Harav Mordechai Yehuda Friedman ืฉื™ืขื•ืจ ืขืœ ืคืจืฉืช ื”ืฉื‘ื•ืข ืคืจ by Unknown (ID: 36004621)
2026-10-05 03:02:02,707 - KolHalashonScraper - INFO - [p1:6] Skip (not_requested_or_already_downloaded): 34:09 Harav Yitzchak Aharon Goldhirsh ืืžื•ื ื” ื•ื‘ื˜ื—ื•ืŸ ืœืจื’ืœ ื”ืžืฆื‘ by Unknown (ID: 36015396)
2026-10-05 03:02:02,707 - KolHalashonScraper - INFO - [p1:7] Skip (not_requested_or_already_downloaded): 39:52 Harav Mordechai Yehuda Friedman ืขื ื™ื ื ืขืœ ืคืจืฉืช ื”ืฉื‘ื•ืข ืคืจ by Unknown (ID: 34260036)
2026-10-05 03:02:02,707 - KolHalashonScraper - INFO - [p1:8] Skip (not_requested_or_already_downloaded): 40:38 Harav Mordechai Yehuda Friedman ืขื ื™ื ื™ื ืขืœ ืคืจืฉืช ื”ืฉื‘ื•ืข ืค by Unknown (ID: 30486279)
2026-10-05 03:02:02,708 - KolHalashonScraper - INFO - reuse_current_page=True -> stopping downloads after the current page
2026-10-05 03:02:02,708 - KolHalashonScraper - INFO - Processed 1 page(s), saw 9 unique shiur IDs
2026-10-05 03:02:02,708 - KolHalashonScraper - INFO - Download complete: 0 downloaded, 1 failed, 8 skipped
2026-10-05 03:02:02,710 - KolHalashonScraper - INFO - Page 1 downloads: 0 downloaded, 1 failed
2026-10-05 03:02:02,719 - KolHalashonScraper - INFO - No more pages available after page 1
2026-10-05 03:02:02,719 - KolHalashonScraper - INFO - URL complete: 9 shiurim extracted from 1 page(s), 9 new total so far
2026-10-05 03:02:14,375 - KolHalashonScraper - INFO - DOM filter debug: {'displayAllSections': 0, 'filterContents': 2, 'filterOptions': 1138, 'totalCheckboxes': 1134, 'type2Checkboxes': 1127}
2026-10-05 03:02:14,474 - KolHalashonScraper - INFO - Extracted 1127 speakers from DOM filter checkboxes
2026-10-05 03:02:14,476 - KolHalashonScraper - INFO - Extracted 1126 speakers from URL filter dropdown
2026-10-05 03:02:14,494 - social_media_monitor.scrapers.kolhalashon_scraper - INFO - New speaker discovered: Harav Pinchas Bogats
2026-10-05 03:02:14,505 - social_media_monitor.scrapers.kolhalashon_scraper - INFO - New speaker discovered: Rebbetzin Matana Hakim
2026-10-05 03:02:14,505 - social_media_monitor.scrapers.kolhalashon_scraper - INFO - New speaker discovered: ื”ืจื‘ ืืœื—ื ืŸ ืื”ืจืžืŸ
2026-10-05 03:02:14,517 - social_media_monitor.scrapers.kolhalashon_scraper - INFO - Speaker sync complete: 3 new, 1123 updated, 1810 total
2026-10-05 03:02:14,517 - KolHalashonScraper - INFO - Found 3 new speakers for this URL
2026-10-05 03:02:14,517 - KolHalashonScraper - INFO - [DELAY] Waiting 1.0s between URLs (shiur_url_delay)
2026-10-05 03:02:15,518 - KolHalashonScraper - INFO - Processing URL (id=2): https://www2.kolhalashon.com/regularSite/searchResults/6/-1/-1/-1/-1/New%20Shiurei%20Torah?urlFilters=3:3%7C14:1%7C
2026-10-05 03:02:15,518 - KolHalashonScraper - INFO - Final processed URL: https://www2.kolhalashon.com/regularSite/searchResults/6/-1/-1/-1/-1/New%20Shiurei%20Torah?urlFilters=3:3%7C14:1%7C
2026-10-05 03:02:15,518 - KolHalashonScraper - INFO - STEP 2 visit processed URL: https://www2.kolhalashon.com/regularSite/searchResults/6/-1/-1/-1/-1/New%20Shiurei%20Torah?urlFilters=3:3%7C14:1%7C
2026-10-05 03:02:15,820 - KolHalashonScraper - INFO - Waiting for JS filtering (0s): 0 shiurs, 0 filter options, reloading=True
2026-10-05 03:02:16,575 - KolHalashonScraper - INFO - Waiting for JS filtering (0s): 0 shiurs, 0 filter options, reloading=True
2026-10-05 03:02:21,083 - KolHalashonScraper - INFO - JS filtering complete after 5s: 8 shiurs, 2 filter options
2026-10-05 03:02:21,132 - KolHalashonScraper - INFO - [0s] Found 8 shiur containers (reloading=False), page state: {'bodyLength': 126724, 'hasAppRoot': True, 'hasShiurimDisplay': True, 'isReloading': False, 'readyState': 'complete'}
2026-10-05 03:02:21,132 - KolHalashonScraper - INFO - Page loaded with 8 shiur containers after 0s
2026-10-05 03:02:21,132 - KolHalashonScraper - INFO - [DELAY] Waiting 70.0s for page content to stabilize (shiur_found_wait)
2026-10-05 03:03:37,169 - KolHalashonScraper - INFO - Processing page 1 of URL (pagination shows: page 1 of 1, has_next=False)
2026-10-05 03:03:37,169 - KolHalashonScraper - INFO - [DELAY] Waiting 6.0s for DOM extraction (dom_extraction_wait)
2026-10-05 03:03:43,178 - KolHalashonScraper - INFO - Extracted 8 shiurim from DOM
2026-10-05 03:03:43,178 - KolHalashonScraper - INFO - Page 1: Browser extracted 8 shiurim
2026-10-05 03:03:43,178 - KolHalashonScraper - INFO - Starting inline downloads for page 1 (reusing current page)
2026-10-05 03:03:43,178 - KolHalashonScraper - INFO - Found current parsha: Bereshis (ื‘ืจืืฉื™ืช)
2026-10-05 03:03:43,179 - KolHalashonScraper - INFO - Using parsha URL template (id=1): https://www2.kolhalashon.com/regularSite/searchResults/1/{parshaid}/{parshaen}?u...
2026-10-05 03:03:43,179 - KolHalashonScraper - INFO - Parsha raw URL template: https://www2.kolhalashon.com/regularSite/searchResults/1/{parshaid}/{parshaen}?urlFilters=3:3%7C{speakersfilter}
2026-10-05 03:03:43,179 - KolHalashonScraper - INFO - Speaker IDs for URL id=1: [7098, 999121, 952143, 7651, 985996, 7643, 7124, 934613, 2393, 7031]
2026-10-05 03:03:43,179 - KolHalashonScraper - INFO - Parsha speaker filter string: [7098, 999121, 952143, 7651, 985996, 7643, 7124, 934613, 2393, 7031]
2026-10-05 03:03:43,179 - KolHalashonScraper - INFO - Parsha encoded speaker filter: 2:7098%7C2:999121%7C2:952143%7C2:7651%7C2:985996%7C2:7643%7C2:7124%7C2:934613%7C2:2393%7C2:7031%7C
2026-10-05 03:03:43,179 - KolHalashonScraper - INFO - Parsha processed URL: https://www2.kolhalashon.com/regularSite/searchResults/1/1/Bereshit?urlFilters=3:3%7C2:7098%7C2:999121%7C2:952143%7C2:7651%7C2:985996%7C2:7643%7C2:7124%7C2:934613%7C2:2393%7C2:7031%7C
2026-10-05 03:03:43,179 - KolHalashonScraper - INFO - Download directory: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio
2026-10-05 03:03:43,179 - KolHalashonScraper - INFO - Using 1 provided shiur(s) for download (page-derived, in-memory)
2026-10-05 03:03:43,179 - KolHalashonScraper - INFO - Starting download of 1 shiurim (one_per_speaker=True)
2026-10-05 03:03:43,182 - KolHalashonScraper - INFO - Configured download behavior via Browser.setDownloadBehavior CDP for /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio
2026-10-05 03:03:43,184 - KolHalashonScraper - INFO - Configured download behavior via Page.setDownloadBehavior CDP for /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio
2026-10-05 03:03:43,184 - KolHalashonScraper - INFO - Reusing current page for downloads (no navigation)
2026-10-05 03:03:49,234 - KolHalashonScraper - INFO - Processing page 1 (pagination shows: page 1 of 1, has_next=False)
2026-10-05 03:03:49,243 - KolHalashonScraper - INFO - Page 1 has 8 shiur items
2026-10-05 03:03:49,243 - KolHalashonScraper - INFO - [p1:0] Skip (not_requested_or_already_downloaded): 4:44 Harav Sholom Ahron Ehrenfeld ืœืฉื•ืŸ ื–ื•ื”ืจ ื”ืงื“ื•ืฉ ืื ืœื ืœื•ืžื“ by Unknown (ID: 33324135)
2026-10-05 03:03:49,243 - KolHalashonScraper - INFO - [p1:1] Skip (not_requested_or_already_downloaded): 9:16 Harav Naftali Shovkes ื”ืคื™ื›ืช ื™ืžื™ ื”ืฉื’ืจื” ืœื—ื™ื‘ื•ืจ ื—ื™ ื•ืคื ื™ืžื™ by Unknown (ID: 42769888)
2026-10-05 03:03:49,243 - KolHalashonScraper - INFO - [p1:2] Skip (not_requested_or_already_downloaded): 1:05:03 ื”ืจื‘ ืจืคืืœ ื‘ืจืœื™ืŸ ื' ื‘ืจืืฉื™ืช ื›"ื–, Parashat Bereshis, Cha by Unknown (ID: 42840929)
2026-10-05 03:03:49,243 - KolHalashonScraper - INFO - [p1:3] Skip (not_requested_or_already_downloaded): 1:44:54 Harav Shimon Spitzer ื™ื•ื ื ืคืจืฉืช ื‘ืจืืฉื™ืช ืชืฉืค"ื– - ืกืขื•ื“ืช by Unknown (ID: 42769253)
2026-10-05 03:03:49,243 - KolHalashonScraper - INFO - [p1:4] Skip (not_requested_or_already_downloaded): 14:48 Harav Chaim Mordechai Pollack ื‘ืขื–ืจืช ื”ืฉื ื”ื™ื™ื‘ ืžื™ืจ ืืŸ ื“ื™ by Unknown (ID: 42840639)
2026-10-05 03:03:49,243 - KolHalashonScraper - INFO - [p1:5] Attempting download: 0:35 ื”ืจื‘ ื™ืฉืจืืœ ื“ื•ื“ ื›ื”ืŸ ื”ื•ื•ืืจื˜ ื”ื™ื•ืžื™ -ื›ื’ ืชืฉืจื™ ืชืฉืค''ื–, Parasha by Unknown (ID: 42769119)
2026-10-05 03:03:52,892 - KolHalashonScraper - INFO - Extracted media URL for shiur 42769119: https://www.kolhalashon.com/api/files/GetFileToPlay/42769119/3/1791176630.LG_QiHE400aBRnrJ1jNbsU8eoO-4t0SRK1tFIPf49fs
2026-10-05 03:03:53,212 - KolHalashonScraper - INFO - Starting HTTP download: https://www.kolhalashon.com/api/files/GetFileToPlay/42769119/3/1791176630.LG_QiHE400aBRnrJ1jNbsU8eoO-4t0SRK1tFIPf49fs
2026-10-05 03:03:53,214 - httpx - INFO - HTTP Request: GET https://alldaf.org/api/trpc/auth.me,popups.fetchByPlatform,posts.fetchList,posts.fetchList,posts.fetchList?batch=1&input=%7B%220%22%3A%7B%22platform%22%3A%22AllDaf%22%7D%2C%221%22%3A%7B%22platform%22%3A%22AllDaf%22%7D%2C%222%22%3A%7B%22limit%22%3A20%2C%22take%22%3A20%2C%22type%22%3A%22Daf+Yomi+Shiurim%22%2C%22allDafSeriesType%22%3A%22Daf+Yomi+Shiurim%22%2C%22metadataValues%22%3A%5B%22Nidah+28%22%5D%2C%22filter%22%3A%7B%7D%2C%22platform%22%3A%22AllDaf%22%7D%2C%223%22%3A%7B%22limit%22%3A20%2C%22take%22%3A20%2C%22type%22%3A%22Supplemental+Clips%22%2C%22allDafSeriesType%22%3A%22Supplemental+Clips%22%2C%22metadataValues%22%3A%5B%22Nidah+28%22%5D%2C%22filter%22%3A%7B%7D%2C%22platform%22%3A%22AllDaf%22%7D%2C%224%22%3A%7B%22limit%22%3A20%2C%22take%22%3A20%2C%22type%22%3A%22Reference+Materials%22%2C%22allDafSeriesType%22%3A%22Reference+Materials%22%2C%22metadataValues%22%3A%5B%22Nidah+28%22%5D%2C%22filter%22%3A%7B%7D%2C%22platform%22%3A%22AllDaf%22%7D%7D "HTTP/1.1 200 OK"
2026-10-05 03:03:53,216 - AllDafScraper - INFO - All Daf fetched 13 record(s) for Nidah 28
2026-10-05 03:03:53,316 - AllDafScraper - INFO - All Daf Nidah 28 complete: 0 new item(s)
2026-10-05 03:03:53,317 - AllDafScraper - INFO - All Daf complete: 0 new item(s), 0 downloaded across 1 daf(s)
2026-10-05 03:03:54,666 - KolHalashonScraper - INFO - Download complete: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio/42769119_temp.mp3 (18544301 bytes)
2026-10-05 03:03:54,667 - KolHalashonScraper - INFO - Renamed: 42769119_temp.mp3 -> KH_ื”ื•ื•ืืจื˜ ื”ื™ื•ืžื™ -ื›ื’ ืชืฉืจื™ ืชืฉืค''ื–, Parashat Bereshis - ื”ืจื‘ ื™ืฉืจืืœ ื“ื•ื“ ื›ื”ืŸ (42769119).mp3
2026-10-05 03:03:54,682 - KolHalashonScraper - INFO - Tagged: KH_ื”ื•ื•ืืจื˜ ื”ื™ื•ืžื™ -ื›ื’ ืชืฉืจื™ ืชืฉืค''ื–, Parashat Bereshis - ื”ืจื‘ ื™ืฉืจืืœ ื“ื•ื“ ื›ื”ืŸ (42769119).mp3 (ื”ืจื‘ ื™ืฉืจืืœ ื“ื•ื“ ื›ื”ืŸ โ€” ื”ื•ื•ืืจื˜ ื”ื™ื•ืžื™ -ื›ื’ ืชืฉืจื™ ืชืฉืค''ื–, Parashat Bereshis)
2026-10-05 03:03:54,683 - KolHalashonScraper - INFO - [p1:5] Direct HTTP download completed: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio/KH_ื”ื•ื•ืืจื˜ ื”ื™ื•ืžื™ -ื›ื’ ืชืฉืจื™ ืชืฉืค''ื–, Parashat Bereshis - ื”ืจื‘ ื™ืฉืจืืœ ื“ื•ื“ ื›ื”ืŸ (42769119).mp3
2026-10-05 03:03:54,683 - KolHalashonScraper - INFO - [p1:5] Downloaded: 0:35 ื”ืจื‘ ื™ืฉืจืืœ ื“ื•ื“ ื›ื”ืŸ ื”ื•ื•ืืจื˜ ื”ื™ื•ืžื™ -ื›ื’ ืชืฉืจื™ ืชืฉืค''ื–, Parasha by Unknown
2026-10-05 03:03:55,683 - KolHalashonScraper - INFO - [p1:6] Skip (not_requested_or_already_downloaded): 1:17:05 Harav Menashe Yisrael Reizman ื™ื•ื ืโ€™ ืคืจืฉืช ื‘ืจืืฉื™ืช ืชืฉืค by Unknown (ID: 42840529)
2026-10-05 03:03:55,684 - KolHalashonScraper - INFO - [p1:7] Skip (not_requested_or_already_downloaded): 23:41 Harav Daniel Chaim Alter ืจืืฉื•ืŸ, Parashat Bereshis, Sef by Unknown (ID: 42769091)
2026-10-05 03:03:55,684 - KolHalashonScraper - INFO - reuse_current_page=True -> stopping downloads after the current page
2026-10-05 03:03:55,684 - KolHalashonScraper - INFO - Processed 1 page(s), saw 8 unique shiur IDs
2026-10-05 03:03:55,684 - KolHalashonScraper - INFO - Download complete: 1 downloaded, 0 failed, 7 skipped
2026-10-05 03:03:55,684 - KolHalashonScraper - INFO - Page 1 downloads: 1 downloaded, 0 failed
2026-10-05 03:03:55,704 - KolHalashonScraper - INFO - No more pages available after page 1
2026-10-05 03:03:55,704 - KolHalashonScraper - INFO - URL complete: 8 shiurim extracted from 1 page(s), 17 new total so far
2026-10-05 03:03:55,795 - KolHalashonScraper - INFO - [DELAY] Waiting 0.7s for filter sections to expand (filter_expand_wait)
2026-10-05 03:03:56,530 - KolHalashonScraper - INFO - [DELAY] Waiting 0.7s after clicking 'display more' buttons (filter_expand_wait)
2026-10-05 03:03:57,243 - KolHalashonScraper - INFO - Expanded filter sections: clicked 2 'display more' buttons
2026-10-05 03:03:58,251 - KolHalashonScraper - INFO - DOM filter debug: {'displayAllSections': 2, 'filterContents': 7, 'filterOptions': 51, 'totalCheckboxes': 49, 'type2Checkboxes': 8}
2026-10-05 03:03:58,258 - KolHalashonScraper - INFO - Extracted 8 speakers from DOM filter checkboxes
2026-10-05 03:03:58,259 - KolHalashonScraper - INFO - Extracted 8 speakers from URL filter dropdown
2026-10-05 03:03:58,267 - social_media_monitor.scrapers.kolhalashon_scraper - INFO - Speaker sync complete: 0 new, 8 updated, 1810 total
2026-10-05 03:03:58,267 - KolHalashonScraper - INFO - [DELAY] Waiting 1.0s between URLs (shiur_url_delay)
2026-10-05 03:03:59,267 - KolHalashonScraper - INFO - Processing URL (id=8): https://www.kolhalashon.com/en/regularSite/searchResults/100/59/Nida?urlFilters=3:3%7C20:28%7C{speakersfilter}
2026-10-05 03:03:59,268 - KolHalashonScraper - INFO - Replaced speakersfilter placeholder with 5 speaker filter(s)
2026-10-05 03:03:59,268 - KolHalashonScraper - INFO - Final processed URL: https://www.kolhalashon.com/en/regularSite/searchResults/100/59/Nida?urlFilters=3:3%7C20:28%7C2:7651%7C2:1608%7C2:7011%7C2:1762%7C2:7163%7C
2026-10-05 03:03:59,268 - KolHalashonScraper - INFO - STEP 2 visit processed URL: https://www.kolhalashon.com/en/regularSite/searchResults/100/59/Nida?urlFilters=3:3%7C20:28%7C2:7651%7C2:1608%7C2:7011%7C2:1762%7C2:7163%7C
2026-10-05 03:03:59,537 - KolHalashonScraper - INFO - Waiting for JS filtering (0s): 0 shiurs, 0 filter options, reloading=True
2026-10-05 03:04:00,286 - KolHalashonScraper - INFO - Waiting for JS filtering (0s): 0 shiurs, 0 filter options, reloading=True
2026-10-05 03:04:04,804 - KolHalashonScraper - INFO - Waiting for JS filtering (5s): 0 shiurs, 97 filter options, reloading=True
2026-10-05 03:04:10,063 - KolHalashonScraper - INFO - Waiting for JS filtering (10s): 13 shiurs, 6 filter options, reloading=False
2026-10-05 03:04:11,566 - KolHalashonScraper - INFO - JS filtering complete after 12s: 13 shiurs, 6 filter options
2026-10-05 03:04:11,628 - KolHalashonScraper - INFO - [0s] Found 13 shiur containers (reloading=False), page state: {'bodyLength': 175426, 'hasAppRoot': True, 'hasShiurimDisplay': True, 'isReloading': False, 'readyState': 'complete'}
2026-10-05 03:04:11,628 - KolHalashonScraper - INFO - Page loaded with 13 shiur containers after 0s
2026-10-05 03:04:11,628 - KolHalashonScraper - INFO - [DELAY] Waiting 70.0s for page content to stabilize (shiur_found_wait)