โ Logs
social_media_monitor_2026-07-02.log
2026-07-02 19:30:06,952 - __main__ - INFO - Scraper github_feed completed: 0 results
2026-07-02 19:30:06,952 - __main__ - INFO - Scraper linkedin_feed completed: 0 results
2026-07-02 19:30:06,953 - __main__ - WARNING - LinkedIn Feed: 32 consecutive failures (0 results)
2026-07-02 19:30:06,953 - __main__ - INFO - No new results found across all enabled sources
2026-07-02 19:30:06,953 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-07-02 19:30:06,953 - __main__ - INFO - Monitor complete. No new results found across all sources.
2026-07-02 19:30:06,953 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-07-02 19:30:06,954 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 19:30:06,984 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-07-02 19:30:06,985 - __main__ - INFO - Closed all database connections
2026-07-02 19:30:06,987 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 19:30:06,987 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-07-02 20:04:39,300 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-07-02.log
2026-07-02 20:04:39,300 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-07-02.log
2026-07-02 20:04:39,300 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-07-02.log
2026-07-02 20:04:39,300 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-07-02 20:04:39,330 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-07-02 20:04:39,330 - __main__ - INFO - Configuration loaded and validated successfully
2026-07-02 20:04:39,332 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-07-02 20:04:39,333 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-07-02 20:04:39,348 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-07-02 20:04:39,348 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-07-02 20:04:39,351 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-07-02 20:04:39,351 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-07-02 20:04:39,352 - __main__ - INFO - RSS scraper initialized and enabled
2026-07-02 20:04:39,355 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-07-02 20:04:39,355 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-07-02 20:04:39,355 - __main__ - INFO - Yiddish24 scraper initialized and enabled
2026-07-02 20:04:39,358 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-07-02 20:04:39,363 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-07-02 20:04:39,363 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-07-02 20:04:39,365 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-07-02 20:04:39,365 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-07-02 20:04:39,365 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-07-02 20:04:39,365 - __main__ - INFO - Ivelt scraper initialized and enabled
2026-07-02 20:04:39,365 - __main__ - INFO - Running 3 scrapers concurrently...
2026-07-02 20:04:39,365 - Yiddish24Scraper - INFO - Starting Yiddish24 scraping (max 10 pages, limit 100 items)
2026-07-02 20:04:39,451 - Yiddish24Scraper - INFO - Fetching Yiddish24 page 1 with data_id='latest', page_limit=100
2026-07-02 20:04:39,456 - IveltScraper - INFO - Loaded 181 include, 0 exclude filters from DB topic_filters
2026-07-02 20:04:39,456 - IveltScraper - INFO - Fetching ivelt active topics from https://www.ivelt.com/forum/search.php?search_id=active_topics...
2026-07-02 20:04:39,457 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.theyeshivaworld.com/feed
2026-07-02 20:04:39,742 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.thegatewaypundit.com/feed/
2026-07-02 20:04:40,139 - RSSFeedScraper - INFO - Total RSS/HTML results: 4
2026-07-02 20:04:43,575 - httpx - INFO - HTTP Request: POST https://www.yiddish24.com/ajax/cat_pagination.php "HTTP/1.1 200 OK"
2026-07-02 20:04:43,805 - Yiddish24Scraper - INFO - Received response with 467357 characters
2026-07-02 20:04:43,808 - Yiddish24Scraper - INFO - Extracted HTML from JSON result field (301725 chars)
2026-07-02 20:04:43,812 - Yiddish24Scraper - INFO - Saved HTML debug file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/debug/yiddish24_page1_response.html
2026-07-02 20:04:43,904 - Yiddish24Scraper - INFO - Found 100 potential article elements on page 1
2026-07-02 20:04:43,943 - Yiddish24Scraper - INFO - Page 1 extraction: 100 successful, 0 failed out of 100 elements
2026-07-02 20:04:43,943 - Yiddish24Scraper - INFO - Parsed 100 articles from HTML on page 1
2026-07-02 20:04:43,954 - Yiddish24Scraper - INFO - Term 'ืขื ืื ื ืืืืื' matched in DESCRIPTION: 'ืขื ืื ื ืืืืื - ืืืื ืข ืคืื ืื ืืขืฉืขืขื ืืฉื ืคืืจืืื ืื ืืื ืฉืืขื ืขืฉืจ ืืชืืื'
2026-07-02 20:04:43,954 - Yiddish24Scraper - INFO - Downloading 1 media file(s) for article: ืืื ืืื
2026-07-02 20:04:44,517 - social_media_monitor.utils.file_downloader - INFO - Starting download from: https://cloudfront.yiddish24.com/pinches_260_260702034240.mp3
2026-07-02 20:04:46,404 - social_media_monitor.utils.file_downloader - INFO - Successfully downloaded: Y2_pinches_260_260702034240.mp3 (24.8 MB)
2026-07-02 20:04:46,405 - social_media_monitor.utils.file_downloader - INFO - Organized audio file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio (audio/mpeg)
2026-07-02 20:04:46,430 - social_media_monitor.utils.media_tagger - INFO - Tagged Y2_pinches_260_260702034240.mp3: title, artist, album, comment, source_url, genre, publisher
2026-07-02 20:04:46,431 - Yiddish24Scraper - INFO - Renamed downloaded file to include match term: Y2_pinches_260_260702034240_ืขื ืื ื_ืืืืื.mp3
2026-07-02 20:04:46,505 - Yiddish24Scraper - INFO - Term 'ื ืืขืืื ืง' matched in DESCRIPTION: 'ื ืคืจืืฉืข ืืืจืืื ืขืจืฉืื ืขื ืืขื ืื, ืื ืืืขืจืืงื ืคืจืืืืจื ืืืืขืจืฆืืจืขืื ืืืจืื ืืื ืคืืจืฉืืืขื ืข ืงืื ืฆืขืกืืขืก ื ืืืฆืืืขืื ืืืจ ืคืืืขืจืื ื ืคืืจ ืงืื ืืจืื ืืืืฃ ืื ืืืจืืื ืืืืกืขืจ ืืืขื, ืืืขืจ ืืขืืจืื ืืื ื ืืฉื ืืจืืื ืืื ืฆื ืืจืขืื. ืืืื...'
2026-07-02 20:04:46,506 - Yiddish24Scraper - INFO - Downloading 1 media file(s) for article: ืืืขืจืืงื ืคืจืืืืจื ืืืืขืจืฆืืจืขืื ืืืจืื ืืื ืคืืจืฉืืืขื ืข ืงื
2026-07-02 20:04:47,006 - social_media_monitor.utils.file_downloader - INFO - Starting download from: https://cloudfront.yiddish24.com/KM_N113_57_1783020563.mp3
2026-07-02 20:04:47,039 - social_media_monitor.utils.file_downloader - INFO - Successfully downloaded: Y2_KM_N113_57_1783020563.mp3 (155.3 KB)
2026-07-02 20:04:47,040 - social_media_monitor.utils.file_downloader - INFO - Organized audio file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio (audio/mpeg)
2026-07-02 20:04:47,042 - social_media_monitor.utils.media_tagger - INFO - Tagged Y2_KM_N113_57_1783020563.mp3: title, artist, album, comment, source_url, genre, publisher
2026-07-02 20:04:47,042 - Yiddish24Scraper - INFO - Renamed downloaded file to include match term: Y2_KM_N113_57_1783020563_ื_ืืขืืื ืง.mp3
2026-07-02 20:04:47,701 - Yiddish24Scraper - INFO - Page 1 summary: 8 new out of 100 total articles
2026-07-02 20:04:47,702 - Yiddish24Scraper - INFO - Page 1: 8/100 new items
2026-07-02 20:04:47,702 - Yiddish24Scraper - INFO - Low new item ratio (8.0%), stopping pagination
2026-07-02 20:04:47,702 - Yiddish24Scraper - INFO - Total Yiddish24 results found: 8
2026-07-02 20:04:51,651 - IveltScraper - INFO - Found 50 active topics on ivelt
2026-07-02 20:04:58,932 - IveltScraper - INFO - Filtering post containing exclude term: '*ืืืืข ืืืื ืืืืขืจ ืืืืื ืฆื ืืืจืืืื ืงืจืื ืืข ื ืืืขืก*'
2026-07-02 20:05:02,944 - IveltScraper - INFO - Filtering post containing exclude term: '*ืืืืข ืืืื ืืืืขืจ ืืืืื ืฆื ืืืจืืืื ืงืจืื ืืข ื ืืืขืก*'
2026-07-02 20:09:39,462 - __main__ - ERROR - Scraper ivelt timed out after 300s; continuing with other results
2026-07-02 20:09:39,463 - __main__ - INFO - Scraper rss completed: 4 results
2026-07-02 20:09:39,463 - __main__ - INFO - Scraper yiddish24 completed: 8 results
2026-07-02 20:09:39,463 - __main__ - INFO - Scraper ivelt completed: 0 results
2026-07-02 20:09:40,384 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-07-02 20:09:40.384338
2026-07-02 20:09:40,384 - __main__ - INFO - Email sent successfully
2026-07-02 20:09:40,385 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-07-02 20:09:40,385 - __main__ - INFO - Monitor complete. Found 12 total results: 4 new rss results, 8 new yiddish24 results.
2026-07-02 20:09:40,385 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-07-02 20:09:40,385 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 20:09:40,474 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-07-02 20:09:40,474 - __main__ - INFO - Closed all database connections
2026-07-02 20:10:13,070 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 20:10:13,070 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-07-02 20:30:00,643 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-07-02.log
2026-07-02 20:30:00,643 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-07-02.log
2026-07-02 20:30:00,643 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-07-02.log
2026-07-02 20:30:00,643 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-07-02 20:30:00,673 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-07-02 20:30:00,674 - __main__ - INFO - Configuration loaded and validated successfully
2026-07-02 20:30:00,676 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-07-02 20:30:00,676 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-07-02 20:30:00,690 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-07-02 20:30:00,691 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-07-02 20:30:00,693 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-07-02 20:30:00,693 - __main__ - INFO - GitHub scraper initialized and enabled
2026-07-02 20:30:00,693 - __main__ - INFO - GitHub feed scraper initialized and enabled
2026-07-02 20:30:00,694 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-07-02 20:30:00,694 - __main__ - INFO - LinkedIn Feed scraper initialized and enabled
2026-07-02 20:30:00,696 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-07-02 20:30:00,696 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-07-02 20:30:00,699 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-07-02 20:30:00,703 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-07-02 20:30:00,703 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-07-02 20:30:00,706 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-07-02 20:30:00,706 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-07-02 20:30:00,706 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-07-02 20:30:00,706 - __main__ - INFO - Running 3 scrapers concurrently...
2026-07-02 20:30:00,706 - GitHubScraper - INFO - Searching GitHub for terms: ['paperclip', '"Hermes Agent"', 'multica']
2026-07-02 20:30:00,806 - LinkedInFeedScraper - INFO - Fetching LinkedIn feed (25 posts)
2026-07-02 20:30:05,484 - LinkedInFeedScraper - ERROR - LinkedIn feed fetch failed: Unauthorized: LinkedIn session cookies are expired or invalid.
2026-07-02 20:30:05,492 - LinkedInFeedScraper - INFO - LinkedIn Feed scrape complete. Found 0 new posts.
2026-07-02 20:30:06,194 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=1 "HTTP/1.1 200 OK"
2026-07-02 20:30:06,513 - GitHubScraper - INFO - GitHub feed: 48 fetched across pages, 0 new events for abeperl
2026-07-02 20:30:06,541 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=paperclip+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 20:30:06,635 - GitHubScraper - INFO - Found 242 GitHub repositories for term: paperclip
2026-07-02 20:30:07,956 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22Hermes+Agent%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 20:30:07,960 - GitHubScraper - INFO - Found 415 GitHub repositories for term: "Hermes Agent"
2026-07-02 20:30:07,985 - GitHubScraper - INFO - Added new GitHub repository: minions
2026-07-02 20:30:09,072 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=multica+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 20:30:09,166 - GitHubScraper - INFO - Found 54 GitHub repositories for term: multica
2026-07-02 20:30:09,291 - GitHubScraper - INFO - Total GitHub results found: 1
2026-07-02 20:30:09,291 - __main__ - INFO - Scraper github completed: 1 results
2026-07-02 20:30:09,291 - __main__ - INFO - Scraper github_feed completed: 0 results
2026-07-02 20:30:09,291 - __main__ - INFO - Scraper linkedin_feed completed: 0 results
2026-07-02 20:30:09,292 - __main__ - WARNING - LinkedIn Feed: 33 consecutive failures (0 results)
2026-07-02 20:30:10,043 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-07-02 20:30:10.043314
2026-07-02 20:30:10,043 - __main__ - INFO - Email sent successfully
2026-07-02 20:30:10,044 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-07-02 20:30:10,044 - __main__ - INFO - Monitor complete. Found 1 total results: 1 new github results.
2026-07-02 20:30:10,044 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-07-02 20:30:10,044 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 20:30:10,072 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-07-02 20:30:10,072 - __main__ - INFO - Closed all database connections
2026-07-02 20:30:10,074 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 20:30:10,075 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-07-02 21:04:39,278 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-07-02.log
2026-07-02 21:04:39,279 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-07-02.log
2026-07-02 21:04:39,279 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-07-02.log
2026-07-02 21:04:39,279 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-07-02 21:04:39,310 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-07-02 21:04:39,310 - __main__ - INFO - Configuration loaded and validated successfully
2026-07-02 21:04:39,312 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-07-02 21:04:39,312 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-07-02 21:04:39,328 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-07-02 21:04:39,328 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-07-02 21:04:39,373 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-07-02 21:04:39,373 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-07-02 21:04:39,373 - __main__ - INFO - RSS scraper initialized and enabled
2026-07-02 21:04:39,377 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-07-02 21:04:39,377 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-07-02 21:04:39,377 - __main__ - INFO - Yiddish24 scraper initialized and enabled
2026-07-02 21:04:39,380 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-07-02 21:04:39,384 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-07-02 21:04:39,384 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-07-02 21:04:39,386 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-07-02 21:04:39,386 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-07-02 21:04:39,386 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-07-02 21:04:39,386 - __main__ - INFO - Ivelt scraper initialized and enabled
2026-07-02 21:04:39,386 - __main__ - INFO - Running 3 scrapers concurrently...
2026-07-02 21:04:39,386 - Yiddish24Scraper - INFO - Starting Yiddish24 scraping (max 10 pages, limit 100 items)
2026-07-02 21:04:39,474 - Yiddish24Scraper - INFO - Fetching Yiddish24 page 1 with data_id='latest', page_limit=100
2026-07-02 21:04:39,478 - IveltScraper - INFO - Loaded 181 include, 0 exclude filters from DB topic_filters
2026-07-02 21:04:39,478 - IveltScraper - INFO - Fetching ivelt active topics from https://www.ivelt.com/forum/search.php?search_id=active_topics...
2026-07-02 21:04:39,480 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.theyeshivaworld.com/feed
2026-07-02 21:04:39,777 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.thegatewaypundit.com/feed/
2026-07-02 21:04:40,146 - RSSFeedScraper - INFO - Total RSS/HTML results: 4
2026-07-02 21:04:43,432 - httpx - INFO - HTTP Request: POST https://www.yiddish24.com/ajax/cat_pagination.php "HTTP/1.1 200 OK"
2026-07-02 21:04:43,656 - Yiddish24Scraper - INFO - Received response with 444470 characters
2026-07-02 21:04:43,659 - Yiddish24Scraper - INFO - Extracted HTML from JSON result field (291912 chars)
2026-07-02 21:04:43,662 - Yiddish24Scraper - INFO - Saved HTML debug file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/debug/yiddish24_page1_response.html
2026-07-02 21:04:43,747 - Yiddish24Scraper - INFO - Found 100 potential article elements on page 1
2026-07-02 21:04:43,787 - Yiddish24Scraper - INFO - Page 1 extraction: 100 successful, 0 failed out of 100 elements
2026-07-02 21:04:43,787 - Yiddish24Scraper - INFO - Parsed 100 articles from HTML on page 1
2026-07-02 21:04:43,865 - Yiddish24Scraper - INFO - Term 'ืฉืืื' matched in TITLE: 'ืืจื ืืืจื ืืจืืกืืื ืฉืืื"ื'
2026-07-02 21:04:43,865 - Yiddish24Scraper - INFO - Downloading 1 media file(s) for article: ืืจื ืืืจื ืืจืืกืืื ืฉืืื"ื
2026-07-02 21:04:44,366 - social_media_monitor.utils.file_downloader - INFO - Starting download from: https://cloudfront.yiddish24.com/__80__________237_1783022225351_577_0_260_442953174.mp3
2026-07-02 21:04:44,580 - social_media_monitor.utils.file_downloader - INFO - Successfully downloaded: Y2___80__________237_1783022225351_577_0_260_442953174.mp3 (8.4 MB)
2026-07-02 21:04:44,582 - social_media_monitor.utils.file_downloader - INFO - Organized audio file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio (audio/mpeg)
2026-07-02 21:04:44,588 - social_media_monitor.utils.media_tagger - INFO - Tagged Y2___80__________237_1783022225351_577_0_260_442953174.mp3: title, artist, album, comment, source_url, genre, publisher
2026-07-02 21:04:44,589 - Yiddish24Scraper - INFO - Renamed downloaded file to include match term: Y2___80__________237_1783022225351_577_0_260_442953174_ืฉืืื.mp3
2026-07-02 21:04:45,196 - Yiddish24Scraper - INFO - Page 1 summary: 10 new out of 100 total articles
2026-07-02 21:04:45,197 - Yiddish24Scraper - INFO - Page 1: 10/100 new items
2026-07-02 21:04:45,197 - Yiddish24Scraper - INFO - Low new item ratio (10.0%), stopping pagination
2026-07-02 21:04:45,197 - Yiddish24Scraper - INFO - Total Yiddish24 results found: 10
2026-07-02 21:04:51,748 - IveltScraper - INFO - Found 50 active topics on ivelt
2026-07-02 21:05:05,939 - IveltScraper - INFO - Filtering post containing exclude term: '*ืืืืข ืืืื ืืืืขืจ ืืืืื ืฆื ืืืจืืืื ืงืจืื ืืข ื ืืืขืก*'
2026-07-02 21:05:06,591 - IveltScraper - INFO - ivelt scrape complete. Checked 4 topics.
2026-07-02 21:05:06,592 - IveltScraper - INFO - All forums scrape complete. Found 14 new posts.
2026-07-02 21:05:06,592 - __main__ - INFO - Scraper rss completed: 4 results
2026-07-02 21:05:06,592 - __main__ - INFO - Scraper yiddish24 completed: 10 results
2026-07-02 21:05:06,592 - __main__ - INFO - Scraper ivelt completed: 14 results
2026-07-02 21:05:07,371 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-07-02 21:05:07.371624
2026-07-02 21:05:07,371 - __main__ - INFO - Email sent successfully
2026-07-02 21:05:07,372 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-07-02 21:05:07,372 - __main__ - INFO - Monitor complete. Found 28 total results: 4 new rss results, 10 new yiddish24 results, 14 new ivelt results.
2026-07-02 21:05:07,372 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-07-02 21:05:07,372 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 21:05:07,466 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-07-02 21:05:07,466 - __main__ - INFO - Closed all database connections
2026-07-02 21:05:07,468 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 21:05:07,469 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-07-02 21:30:00,661 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-07-02.log
2026-07-02 21:30:00,661 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-07-02.log
2026-07-02 21:30:00,661 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-07-02.log
2026-07-02 21:30:00,661 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-07-02 21:30:00,691 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-07-02 21:30:00,691 - __main__ - INFO - Configuration loaded and validated successfully
2026-07-02 21:30:00,693 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-07-02 21:30:00,694 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-07-02 21:30:00,709 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-07-02 21:30:00,709 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-07-02 21:30:00,711 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-07-02 21:30:00,712 - __main__ - INFO - GitHub scraper initialized and enabled
2026-07-02 21:30:00,712 - __main__ - INFO - GitHub feed scraper initialized and enabled
2026-07-02 21:30:00,712 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-07-02 21:30:00,712 - __main__ - INFO - LinkedIn Feed scraper initialized and enabled
2026-07-02 21:30:00,715 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-07-02 21:30:00,715 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-07-02 21:30:00,717 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-07-02 21:30:00,722 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-07-02 21:30:00,722 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-07-02 21:30:00,724 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-07-02 21:30:00,724 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-07-02 21:30:00,724 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-07-02 21:30:00,724 - __main__ - INFO - Running 3 scrapers concurrently...
2026-07-02 21:30:00,724 - GitHubScraper - INFO - Searching GitHub for terms: ['paperclip', '"Hermes Agent"', 'multica']
2026-07-02 21:30:00,818 - LinkedInFeedScraper - INFO - Fetching LinkedIn feed (25 posts)
2026-07-02 21:30:06,526 - LinkedInFeedScraper - ERROR - LinkedIn feed fetch failed: Unauthorized: LinkedIn session cookies are expired or invalid.
2026-07-02 21:30:06,534 - LinkedInFeedScraper - INFO - LinkedIn Feed scrape complete. Found 0 new posts.
2026-07-02 21:30:07,343 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=1 "HTTP/1.1 200 OK"
2026-07-02 21:30:07,893 - GitHubScraper - INFO - GitHub feed: 59 fetched across pages, 7 new events for abeperl
2026-07-02 21:30:07,895 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=paperclip+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 21:30:07,899 - GitHubScraper - INFO - Found 242 GitHub repositories for term: paperclip
2026-07-02 21:30:08,959 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22Hermes+Agent%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 21:30:08,963 - GitHubScraper - INFO - Found 415 GitHub repositories for term: "Hermes Agent"
2026-07-02 21:30:09,832 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=multica+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 21:30:09,836 - GitHubScraper - INFO - Found 54 GitHub repositories for term: multica
2026-07-02 21:30:09,964 - GitHubScraper - INFO - Total GitHub results found: 0
2026-07-02 21:30:09,964 - __main__ - INFO - Scraper github completed: 0 results
2026-07-02 21:30:09,964 - __main__ - INFO - Scraper github_feed completed: 7 results
2026-07-02 21:30:09,964 - __main__ - INFO - Scraper linkedin_feed completed: 0 results
2026-07-02 21:30:09,965 - __main__ - WARNING - LinkedIn Feed: 34 consecutive failures (0 results)
2026-07-02 21:30:10,734 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-07-02 21:30:10.733998
2026-07-02 21:30:10,734 - __main__ - INFO - Email sent successfully
2026-07-02 21:30:10,734 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-07-02 21:30:10,734 - __main__ - INFO - Monitor complete. Found 7 total results: 7 new github feed results.
2026-07-02 21:30:10,734 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-07-02 21:30:10,734 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 21:30:10,764 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-07-02 21:30:10,764 - __main__ - INFO - Closed all database connections
2026-07-02 21:30:10,765 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 21:30:10,765 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-07-02 22:04:39,236 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-07-02.log
2026-07-02 22:04:39,237 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-07-02.log
2026-07-02 22:04:39,237 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-07-02.log
2026-07-02 22:04:39,237 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-07-02 22:04:39,268 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-07-02 22:04:39,268 - __main__ - INFO - Configuration loaded and validated successfully
2026-07-02 22:04:39,270 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-07-02 22:04:39,271 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-07-02 22:04:39,285 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-07-02 22:04:39,286 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-07-02 22:04:39,291 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-07-02 22:04:39,292 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-07-02 22:04:39,292 - __main__ - INFO - RSS scraper initialized and enabled
2026-07-02 22:04:39,299 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-07-02 22:04:39,299 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-07-02 22:04:39,299 - __main__ - INFO - Yiddish24 scraper initialized and enabled
2026-07-02 22:04:39,306 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-07-02 22:04:39,314 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-07-02 22:04:39,314 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-07-02 22:04:39,317 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-07-02 22:04:39,317 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-07-02 22:04:39,317 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-07-02 22:04:39,317 - __main__ - INFO - Ivelt scraper initialized and enabled
2026-07-02 22:04:39,317 - __main__ - INFO - Running 3 scrapers concurrently...
2026-07-02 22:04:39,318 - Yiddish24Scraper - INFO - Starting Yiddish24 scraping (max 10 pages, limit 100 items)
2026-07-02 22:04:39,407 - Yiddish24Scraper - INFO - Fetching Yiddish24 page 1 with data_id='latest', page_limit=100
2026-07-02 22:04:39,411 - IveltScraper - INFO - Loaded 181 include, 0 exclude filters from DB topic_filters
2026-07-02 22:04:39,411 - IveltScraper - INFO - Fetching ivelt active topics from https://www.ivelt.com/forum/search.php?search_id=active_topics...
2026-07-02 22:04:39,412 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.theyeshivaworld.com/feed
2026-07-02 22:04:39,710 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.thegatewaypundit.com/feed/
2026-07-02 22:04:40,051 - RSSFeedScraper - INFO - Total RSS/HTML results: 4
2026-07-02 22:04:43,440 - httpx - INFO - HTTP Request: POST https://www.yiddish24.com/ajax/cat_pagination.php "HTTP/1.1 200 OK"
2026-07-02 22:04:43,657 - Yiddish24Scraper - INFO - Received response with 441845 characters
2026-07-02 22:04:43,660 - Yiddish24Scraper - INFO - Extracted HTML from JSON result field (291402 chars)
2026-07-02 22:04:43,662 - Yiddish24Scraper - INFO - Saved HTML debug file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/debug/yiddish24_page1_response.html
2026-07-02 22:04:43,751 - Yiddish24Scraper - INFO - Found 100 potential article elements on page 1
2026-07-02 22:04:43,790 - Yiddish24Scraper - INFO - Page 1 extraction: 100 successful, 0 failed out of 100 elements
2026-07-02 22:04:43,790 - Yiddish24Scraper - INFO - Parsed 100 articles from HTML on page 1
2026-07-02 22:04:43,808 - Yiddish24Scraper - INFO - Term 'ืขืจืฉืืืื ืืื' matched in DESCRIPTION: 'ืขืจืฉืืืื ืืื - ืืืืืจ ืืื ืืืืขืจืืขืื, ืืืืืขืจ ืก'ืืื ืฆื ืฉืคืขื'
2026-07-02 22:04:43,808 - Yiddish24Scraper - INFO - Downloading 1 media file(s) for article: ืืื ืืื
2026-07-02 22:04:44,331 - social_media_monitor.utils.file_downloader - INFO - Starting download from: https://cloudfront.yiddish24.com/KM311935D_260_260702054713.mp3
2026-07-02 22:04:45,454 - social_media_monitor.utils.file_downloader - INFO - Successfully downloaded: Y2_KM311935D_260_260702054713.mp3 (5.6 MB)
2026-07-02 22:04:45,455 - social_media_monitor.utils.file_downloader - INFO - Organized audio file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio (audio/mpeg)
2026-07-02 22:04:45,461 - social_media_monitor.utils.media_tagger - INFO - Tagged Y2_KM311935D_260_260702054713.mp3: title, artist, album, comment, source_url, genre, publisher
2026-07-02 22:04:45,461 - Yiddish24Scraper - INFO - Renamed downloaded file to include match term: Y2_KM311935D_260_260702054713_ืขืจืฉืืืื ืืื.mp3
2026-07-02 22:04:45,490 - Yiddish24Scraper - INFO - Term 'ืืง ืืืฉืจืื' matched in TITLE: 'ืืง ืืืฉืจืื - ืืจื ืืืื ืืขืืืืืืืฉ'
2026-07-02 22:04:45,490 - Yiddish24Scraper - INFO - Downloading 1 media file(s) for article: ืืง ืืืฉืจืื - ืืจื ืืืื ืืขืืืืืืืฉ
2026-07-02 22:04:45,991 - social_media_monitor.utils.file_downloader - INFO - Starting download from: https://cloudfront.yiddish24.com/20260702053009-160364.mp3
2026-07-02 22:04:47,258 - social_media_monitor.utils.file_downloader - INFO - Successfully downloaded: Y2_20260702053009-160364.mp3 (12.2 MB)
2026-07-02 22:04:47,259 - social_media_monitor.utils.file_downloader - INFO - Organized audio file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio (audio/mpeg)
2026-07-02 22:04:47,271 - social_media_monitor.utils.media_tagger - INFO - Tagged Y2_20260702053009-160364.mp3: title, artist, album, comment, source_url, genre, publisher
2026-07-02 22:04:47,271 - Yiddish24Scraper - INFO - Renamed downloaded file to include match term: Y2_20260702053009-160364_ืืง_ืืืฉืจืื.mp3
2026-07-02 22:04:47,949 - Yiddish24Scraper - INFO - Page 1 summary: 14 new out of 100 total articles
2026-07-02 22:04:47,949 - Yiddish24Scraper - INFO - Page 1: 14/100 new items
2026-07-02 22:04:47,949 - Yiddish24Scraper - INFO - Low new item ratio (14.0%), stopping pagination
2026-07-02 22:04:47,950 - Yiddish24Scraper - INFO - Total Yiddish24 results found: 14
2026-07-02 22:04:51,691 - IveltScraper - INFO - Found 50 active topics on ivelt
2026-07-02 22:04:57,938 - IveltScraper - INFO - ivelt scrape complete. Checked 1 topics.
2026-07-02 22:04:57,939 - IveltScraper - INFO - All forums scrape complete. Found 3 new posts.
2026-07-02 22:04:57,939 - __main__ - INFO - Scraper rss completed: 4 results
2026-07-02 22:04:57,939 - __main__ - INFO - Scraper yiddish24 completed: 14 results
2026-07-02 22:04:57,940 - __main__ - INFO - Scraper ivelt completed: 3 results
2026-07-02 22:04:58,880 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-07-02 22:04:58.880814
2026-07-02 22:04:58,881 - __main__ - INFO - Email sent successfully
2026-07-02 22:04:58,881 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-07-02 22:04:58,881 - __main__ - INFO - Monitor complete. Found 21 total results: 4 new rss results, 14 new yiddish24 results, 3 new ivelt results.
2026-07-02 22:04:58,881 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-07-02 22:04:58,881 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 22:04:58,938 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-07-02 22:04:58,938 - __main__ - INFO - Closed all database connections
2026-07-02 22:04:58,941 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 22:04:58,941 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-07-02 22:30:00,633 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-07-02.log
2026-07-02 22:30:00,633 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-07-02.log
2026-07-02 22:30:00,633 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-07-02.log
2026-07-02 22:30:00,633 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-07-02 22:30:00,663 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-07-02 22:30:00,663 - __main__ - INFO - Configuration loaded and validated successfully
2026-07-02 22:30:00,665 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-07-02 22:30:00,666 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-07-02 22:30:00,679 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-07-02 22:30:00,679 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-07-02 22:30:00,727 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-07-02 22:30:00,727 - __main__ - INFO - GitHub scraper initialized and enabled
2026-07-02 22:30:00,727 - __main__ - INFO - GitHub feed scraper initialized and enabled
2026-07-02 22:30:00,728 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-07-02 22:30:00,728 - __main__ - INFO - LinkedIn Feed scraper initialized and enabled
2026-07-02 22:30:00,731 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-07-02 22:30:00,731 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-07-02 22:30:00,734 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-07-02 22:30:00,738 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-07-02 22:30:00,738 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-07-02 22:30:00,740 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-07-02 22:30:00,740 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-07-02 22:30:00,740 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-07-02 22:30:00,741 - __main__ - INFO - Running 3 scrapers concurrently...
2026-07-02 22:30:00,741 - GitHubScraper - INFO - Searching GitHub for terms: ['paperclip', '"Hermes Agent"', 'multica']
2026-07-02 22:30:00,835 - LinkedInFeedScraper - INFO - Fetching LinkedIn feed (25 posts)
2026-07-02 22:30:04,569 - LinkedInFeedScraper - ERROR - LinkedIn feed fetch failed: Unauthorized: LinkedIn session cookies are expired or invalid.
2026-07-02 22:30:04,582 - LinkedInFeedScraper - INFO - LinkedIn Feed scrape complete. Found 0 new posts.
2026-07-02 22:30:05,543 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=1 "HTTP/1.1 200 OK"
2026-07-02 22:30:06,439 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=paperclip+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 22:30:06,442 - GitHubScraper - INFO - Found 242 GitHub repositories for term: paperclip
2026-07-02 22:30:07,357 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=2 "HTTP/1.1 200 OK"
2026-07-02 22:30:08,327 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22Hermes+Agent%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 22:30:08,329 - GitHubScraper - INFO - Found 415 GitHub repositories for term: "Hermes Agent"
2026-07-02 22:30:09,336 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=multica+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 22:30:09,339 - GitHubScraper - INFO - Found 54 GitHub repositories for term: multica
2026-07-02 22:30:09,546 - GitHubScraper - INFO - Total GitHub results found: 0
2026-07-02 22:30:10,546 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=3 "HTTP/1.1 200 OK"
2026-07-02 22:30:11,351 - GitHubScraper - INFO - GitHub feed: 300 fetched across pages, 299 new events for abeperl
2026-07-02 22:30:11,352 - __main__ - INFO - Scraper github completed: 0 results
2026-07-02 22:30:11,353 - __main__ - INFO - Scraper github_feed completed: 299 results
2026-07-02 22:30:11,353 - __main__ - INFO - Scraper linkedin_feed completed: 0 results
2026-07-02 22:30:11,353 - __main__ - WARNING - LinkedIn Feed: 1 consecutive failures (0 results)
2026-07-02 22:30:12,408 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-07-02 22:30:12.408375
2026-07-02 22:30:12,408 - __main__ - INFO - Email sent successfully
2026-07-02 22:30:12,409 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-07-02 22:30:12,409 - __main__ - INFO - Monitor complete. Found 299 total results: 299 new github feed results.
2026-07-02 22:30:12,409 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-07-02 22:30:12,409 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 22:30:12,448 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-07-02 22:30:12,448 - __main__ - INFO - Closed all database connections
2026-07-02 22:30:12,450 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 22:30:12,450 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-07-02 23:04:39,285 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-07-02.log
2026-07-02 23:04:39,285 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-07-02.log
2026-07-02 23:04:39,285 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-07-02.log
2026-07-02 23:04:39,285 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-07-02 23:04:39,316 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-07-02 23:04:39,316 - __main__ - INFO - Configuration loaded and validated successfully
2026-07-02 23:04:39,318 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-07-02 23:04:39,318 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-07-02 23:04:39,332 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-07-02 23:04:39,332 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-07-02 23:04:39,335 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-07-02 23:04:39,335 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-07-02 23:04:39,336 - __main__ - INFO - RSS scraper initialized and enabled
2026-07-02 23:04:39,338 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-07-02 23:04:39,338 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-07-02 23:04:39,338 - __main__ - INFO - Yiddish24 scraper initialized and enabled
2026-07-02 23:04:39,341 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-07-02 23:04:39,345 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-07-02 23:04:39,345 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-07-02 23:04:39,347 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-07-02 23:04:39,347 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-07-02 23:04:39,348 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-07-02 23:04:39,348 - __main__ - INFO - Ivelt scraper initialized and enabled
2026-07-02 23:04:39,348 - __main__ - INFO - Running 3 scrapers concurrently...
2026-07-02 23:04:39,348 - Yiddish24Scraper - INFO - Starting Yiddish24 scraping (max 10 pages, limit 100 items)
2026-07-02 23:04:39,435 - Yiddish24Scraper - INFO - Fetching Yiddish24 page 1 with data_id='latest', page_limit=100
2026-07-02 23:04:39,440 - IveltScraper - INFO - Loaded 181 include, 0 exclude filters from DB topic_filters
2026-07-02 23:04:39,440 - IveltScraper - INFO - Fetching ivelt active topics from https://www.ivelt.com/forum/search.php?search_id=active_topics...
2026-07-02 23:04:39,441 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.theyeshivaworld.com/feed
2026-07-02 23:04:39,627 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.thegatewaypundit.com/feed/
2026-07-02 23:04:39,960 - RSSFeedScraper - INFO - Total RSS/HTML results: 3
2026-07-02 23:04:43,223 - httpx - INFO - HTTP Request: POST https://www.yiddish24.com/ajax/cat_pagination.php "HTTP/1.1 200 OK"
2026-07-02 23:04:43,449 - Yiddish24Scraper - INFO - Received response with 441650 characters
2026-07-02 23:04:43,452 - Yiddish24Scraper - INFO - Extracted HTML from JSON result field (292915 chars)
2026-07-02 23:04:43,455 - Yiddish24Scraper - INFO - Saved HTML debug file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/debug/yiddish24_page1_response.html
2026-07-02 23:04:43,537 - Yiddish24Scraper - INFO - Found 100 potential article elements on page 1
2026-07-02 23:04:43,578 - Yiddish24Scraper - INFO - Page 1 extraction: 100 successful, 0 failed out of 100 elements
2026-07-02 23:04:43,579 - Yiddish24Scraper - INFO - Parsed 100 articles from HTML on page 1
2026-07-02 23:04:43,632 - Yiddish24Scraper - INFO - Term 'ืืืขืืืื ืคืขืืืืื' matched in TITLE: 'ืฉืืช ืคืขืงื - ืืืขืืืื ืคืขืืืืื'
2026-07-02 23:04:43,632 - Yiddish24Scraper - INFO - Downloading 1 media file(s) for article: ืฉืืช ืคืขืงื - ืืืขืืืื ืคืขืืืืื
2026-07-02 23:04:44,132 - social_media_monitor.utils.file_downloader - INFO - Starting download from: https://cloudfront.yiddish24.com/New_Recording_86111_162_260702063803.mp3
2026-07-02 23:04:44,402 - social_media_monitor.utils.file_downloader - INFO - Successfully downloaded: Y2_New_Recording_86111_162_260702063803.mp3 (40.3 MB)
2026-07-02 23:04:44,403 - social_media_monitor.utils.file_downloader - INFO - Organized audio file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio (audio/mpeg)
2026-07-02 23:04:44,419 - social_media_monitor.utils.media_tagger - INFO - Tagged Y2_New_Recording_86111_162_260702063803.mp3: title, artist, album, source_url, genre, publisher
2026-07-02 23:04:44,420 - Yiddish24Scraper - INFO - Renamed downloaded file to include match term: Y2_New_Recording_86111_162_260702063803_ืืืขืืืื_ืคืขืืืืื.mp3
2026-07-02 23:04:45,113 - Yiddish24Scraper - INFO - Page 1 summary: 8 new out of 100 total articles
2026-07-02 23:04:45,113 - Yiddish24Scraper - INFO - Page 1: 8/100 new items
2026-07-02 23:04:45,113 - Yiddish24Scraper - INFO - Low new item ratio (8.0%), stopping pagination
2026-07-02 23:04:45,114 - Yiddish24Scraper - INFO - Total Yiddish24 results found: 8
2026-07-02 23:04:51,522 - IveltScraper - INFO - Found 50 active topics on ivelt
2026-07-02 23:05:17,293 - IveltScraper - INFO - ivelt scrape complete. Checked 4 topics.
2026-07-02 23:05:17,293 - IveltScraper - INFO - All forums scrape complete. Found 13 new posts.
2026-07-02 23:05:17,293 - __main__ - INFO - Scraper rss completed: 3 results
2026-07-02 23:05:17,293 - __main__ - INFO - Scraper yiddish24 completed: 8 results
2026-07-02 23:05:17,293 - __main__ - INFO - Scraper ivelt completed: 13 results
2026-07-02 23:05:18,167 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-07-02 23:05:18.167272
2026-07-02 23:05:18,167 - __main__ - INFO - Email sent successfully
2026-07-02 23:05:18,167 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-07-02 23:05:18,167 - __main__ - INFO - Monitor complete. Found 24 total results: 3 new rss results, 8 new yiddish24 results, 13 new ivelt results.
2026-07-02 23:05:18,167 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-07-02 23:05:18,167 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 23:05:18,245 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-07-02 23:05:18,246 - __main__ - INFO - Closed all database connections
2026-07-02 23:05:18,248 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 23:05:18,248 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-07-02 23:30:00,658 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-07-02.log
2026-07-02 23:30:00,658 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-07-02.log
2026-07-02 23:30:00,658 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-07-02.log
2026-07-02 23:30:00,658 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-07-02 23:30:00,692 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-07-02 23:30:00,692 - __main__ - INFO - Configuration loaded and validated successfully
2026-07-02 23:30:00,694 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-07-02 23:30:00,695 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-07-02 23:30:00,710 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-07-02 23:30:00,710 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-07-02 23:30:00,713 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-07-02 23:30:00,713 - __main__ - INFO - GitHub scraper initialized and enabled
2026-07-02 23:30:00,713 - __main__ - INFO - GitHub feed scraper initialized and enabled
2026-07-02 23:30:00,713 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-07-02 23:30:00,714 - __main__ - INFO - LinkedIn Feed scraper initialized and enabled
2026-07-02 23:30:00,716 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-07-02 23:30:00,716 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-07-02 23:30:00,719 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-07-02 23:30:00,723 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-07-02 23:30:00,724 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-07-02 23:30:00,726 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-07-02 23:30:00,726 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-07-02 23:30:00,727 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-07-02 23:30:00,727 - __main__ - INFO - Running 3 scrapers concurrently...
2026-07-02 23:30:00,727 - GitHubScraper - INFO - Searching GitHub for terms: ['paperclip', '"Hermes Agent"', 'multica']
2026-07-02 23:30:00,837 - LinkedInFeedScraper - INFO - Fetching LinkedIn feed (25 posts)
2026-07-02 23:30:03,553 - LinkedInFeedScraper - ERROR - LinkedIn feed fetch failed: Unauthorized: LinkedIn session cookies are expired or invalid.
2026-07-02 23:30:03,566 - LinkedInFeedScraper - INFO - LinkedIn Feed scrape complete. Found 0 new posts.
2026-07-02 23:30:04,301 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=1 "HTTP/1.1 200 OK"
2026-07-02 23:30:05,150 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=paperclip+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 23:30:05,156 - GitHubScraper - INFO - Found 242 GitHub repositories for term: paperclip
2026-07-02 23:30:05,937 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=2 "HTTP/1.1 200 OK"
2026-07-02 23:30:06,694 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22Hermes+Agent%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 23:30:06,696 - GitHubScraper - INFO - Found 416 GitHub repositories for term: "Hermes Agent"
2026-07-02 23:30:07,523 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=multica+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-07-02 23:30:07,611 - GitHubScraper - INFO - Found 54 GitHub repositories for term: multica
2026-07-02 23:30:07,749 - GitHubScraper - INFO - Total GitHub results found: 0
2026-07-02 23:30:08,081 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=3 "HTTP/1.1 200 OK"
2026-07-02 23:30:08,839 - GitHubScraper - INFO - GitHub feed: 300 fetched across pages, 0 new events for abeperl
2026-07-02 23:30:08,840 - __main__ - INFO - Scraper github completed: 0 results
2026-07-02 23:30:08,841 - __main__ - INFO - Scraper github_feed completed: 0 results
2026-07-02 23:30:08,841 - __main__ - INFO - Scraper linkedin_feed completed: 0 results
2026-07-02 23:30:08,841 - __main__ - WARNING - LinkedIn Feed: 2 consecutive failures (0 results)
2026-07-02 23:30:08,842 - __main__ - INFO - No new results found across all enabled sources
2026-07-02 23:30:08,842 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-07-02 23:30:08,842 - __main__ - INFO - Monitor complete. No new results found across all sources.
2026-07-02 23:30:08,842 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-07-02 23:30:08,842 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 23:30:08,895 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-07-02 23:30:08,896 - __main__ - INFO - Closed all database connections
2026-07-02 23:30:08,898 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-07-02 23:30:08,898 - social_media_monitor.database.connection - INFO - Closed 0 database connections
Live tail active โ waiting for new lines...