โ Logs
social_media_monitor_2026-06-23.log
2026-06-23 20:00:02,045 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-06-23 20:00:02,047 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-06-23 20:00:02,047 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-06-23 20:00:02,047 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-06-23 20:00:02,047 - __main__ - INFO - Ivelt scraper initialized and enabled
2026-06-23 20:00:02,047 - __main__ - INFO - Running 3 scrapers concurrently...
2026-06-23 20:00:02,047 - Yiddish24Scraper - INFO - Starting Yiddish24 scraping (max 10 pages, limit 100 items)
2026-06-23 20:00:02,052 - Yiddish24Scraper - INFO - Fetching Yiddish24 page 1 with data_id='latest', page_limit=100
2026-06-23 20:00:02,060 - IveltScraper - INFO - Loaded 181 include, 0 exclude filters from DB topic_filters
2026-06-23 20:00:02,060 - IveltScraper - INFO - Fetching ivelt active topics from https://www.ivelt.com/forum/search.php?search_id=active_topics...
2026-06-23 20:00:02,061 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.theyeshivaworld.com/feed
2026-06-23 20:00:02,364 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.thegatewaypundit.com/feed/
2026-06-23 20:00:02,735 - RSSFeedScraper - INFO - Total RSS/HTML results: 3
2026-06-23 20:00:07,949 - httpx - INFO - HTTP Request: POST https://www.yiddish24.com/ajax/cat_pagination.php "HTTP/1.1 200 OK"
2026-06-23 20:00:08,166 - Yiddish24Scraper - INFO - Received response with 451936 characters
2026-06-23 20:00:08,169 - Yiddish24Scraper - INFO - Extracted HTML from JSON result field (294793 chars)
2026-06-23 20:00:08,172 - Yiddish24Scraper - INFO - Saved HTML debug file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/debug/yiddish24_page1_response.html
2026-06-23 20:00:08,256 - Yiddish24Scraper - INFO - Found 100 potential article elements on page 1
2026-06-23 20:00:08,299 - Yiddish24Scraper - INFO - Page 1 extraction: 100 successful, 0 failed out of 100 elements
2026-06-23 20:00:08,299 - Yiddish24Scraper - INFO - Parsed 100 articles from HTML on page 1
2026-06-23 20:00:09,033 - Yiddish24Scraper - INFO - Page 1 summary: 2 new out of 100 total articles
2026-06-23 20:00:09,034 - Yiddish24Scraper - INFO - Page 1: 2/100 new items
2026-06-23 20:00:09,034 - Yiddish24Scraper - INFO - Low new item ratio (2.0%), stopping pagination
2026-06-23 20:00:09,034 - Yiddish24Scraper - INFO - Total Yiddish24 results found: 2
2026-06-23 20:00:13,916 - IveltScraper - INFO - Found 50 active topics on ivelt
2026-06-23 20:00:27,925 - IveltScraper - INFO - Filtering post containing exclude term: '*ืืืืข ืืืื ืืืืขืจ ืืืืื ืฆื ืืืจืืืื ืงืจืื ืืข ื ืืืขืก*'
2026-06-23 20:00:34,263 - IveltScraper - INFO - Filtering post containing exclude term: '*ืืืืข ืืืื ืืืืขืจ ืืืืื ืฆื ืืืจืืืื ืงืจืื ืืข ื ืืืขืก*'
2026-06-23 20:00:39,693 - IveltScraper - INFO - ivelt scrape complete. Checked 4 topics.
2026-06-23 20:00:39,694 - IveltScraper - INFO - All forums scrape complete. Found 68 new posts.
2026-06-23 20:00:39,694 - __main__ - INFO - Scraper rss completed: 3 results
2026-06-23 20:00:39,694 - __main__ - INFO - Scraper yiddish24 completed: 2 results
2026-06-23 20:00:39,694 - __main__ - INFO - Scraper ivelt completed: 68 results
2026-06-23 20:00:40,712 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-06-23 20:00:40.712132
2026-06-23 20:00:40,712 - __main__ - INFO - Email sent successfully
2026-06-23 20:00:40,712 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-06-23 20:00:40,712 - __main__ - INFO - Monitor complete. Found 73 total results: 3 new rss results, 2 new yiddish24 results, 68 new ivelt results.
2026-06-23 20:00:40,713 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-06-23 20:00:40,713 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 20:00:40,839 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-06-23 20:00:40,839 - __main__ - INFO - Closed all database connections
2026-06-23 20:00:40,842 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 20:00:40,842 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-06-23 20:30:01,969 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-06-23.log
2026-06-23 20:30:01,970 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-06-23.log
2026-06-23 20:30:01,970 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-06-23.log
2026-06-23 20:30:01,970 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-06-23 20:30:01,997 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-06-23 20:30:01,997 - social_media_monitor.config.settings - WARNING - Twitter monitoring enabled but credentials validation not implemented
2026-06-23 20:30:01,997 - __main__ - INFO - Configuration loaded and validated successfully
2026-06-23 20:30:02,000 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-06-23 20:30:02,001 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-06-23 20:30:02,017 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-06-23 20:30:02,017 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-06-23 20:30:02,077 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-06-23 20:30:02,078 - __main__ - INFO - GitHub scraper initialized and enabled
2026-06-23 20:30:02,078 - __main__ - INFO - GitHub feed scraper initialized and enabled
2026-06-23 20:30:02,079 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-06-23 20:30:02,091 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-06-23 20:30:02,091 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-06-23 20:30:02,095 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-06-23 20:30:02,100 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-06-23 20:30:02,100 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-06-23 20:30:02,102 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-06-23 20:30:02,102 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-06-23 20:30:02,102 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-06-23 20:30:02,102 - __main__ - INFO - Running 2 scrapers concurrently...
2026-06-23 20:30:02,102 - GitHubScraper - INFO - Searching GitHub for terms: ['paperclip', '"Hermes Agent"', 'multica']
2026-06-23 20:30:03,155 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=paperclip+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 20:30:03,240 - GitHubScraper - INFO - Found 240 GitHub repositories for term: paperclip
2026-06-23 20:30:03,529 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=1 "HTTP/1.1 200 OK"
2026-06-23 20:30:04,558 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22Hermes+Agent%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 20:30:04,562 - GitHubScraper - INFO - Found 394 GitHub repositories for term: "Hermes Agent"
2026-06-23 20:30:04,683 - GitHubScraper - INFO - Added new GitHub repository: Awesome-Ollama
2026-06-23 20:30:05,627 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=2 "HTTP/1.1 200 OK"
2026-06-23 20:30:06,340 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=multica+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 20:30:06,344 - GitHubScraper - INFO - Found 53 GitHub repositories for term: multica
2026-06-23 20:30:06,478 - GitHubScraper - INFO - Total GitHub results found: 1
2026-06-23 20:30:07,770 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=3 "HTTP/1.1 200 OK"
2026-06-23 20:30:08,465 - GitHubScraper - INFO - GitHub feed: 298 fetched across pages, 1 new events for abeperl
2026-06-23 20:30:08,466 - __main__ - INFO - Scraper github completed: 1 results
2026-06-23 20:30:08,467 - __main__ - INFO - Scraper github_feed completed: 1 results
2026-06-23 20:30:09,136 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-06-23 20:30:09.136210
2026-06-23 20:30:09,136 - __main__ - INFO - Email sent successfully
2026-06-23 20:30:09,136 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-06-23 20:30:09,136 - __main__ - INFO - Monitor complete. Found 2 total results: 1 new github results, 1 new github feed results.
2026-06-23 20:30:09,137 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-06-23 20:30:09,137 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 20:30:09,217 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-06-23 20:30:09,217 - __main__ - INFO - Closed all database connections
2026-06-23 20:30:09,219 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 20:30:09,219 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-06-23 21:00:02,216 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-06-23.log
2026-06-23 21:00:02,216 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-06-23.log
2026-06-23 21:00:02,216 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-06-23.log
2026-06-23 21:00:02,216 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-06-23 21:00:02,242 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-06-23 21:00:02,242 - __main__ - INFO - Configuration loaded and validated successfully
2026-06-23 21:00:02,244 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-06-23 21:00:02,245 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-06-23 21:00:02,258 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-06-23 21:00:02,258 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-06-23 21:00:02,259 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-06-23 21:00:02,260 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-06-23 21:00:02,260 - __main__ - INFO - RSS scraper initialized and enabled
2026-06-23 21:00:02,264 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-06-23 21:00:02,264 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-06-23 21:00:02,264 - __main__ - INFO - Yiddish24 scraper initialized and enabled
2026-06-23 21:00:02,271 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-06-23 21:00:02,277 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-06-23 21:00:02,278 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-06-23 21:00:02,281 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-06-23 21:00:02,281 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-06-23 21:00:02,281 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-06-23 21:00:02,281 - __main__ - INFO - Ivelt scraper initialized and enabled
2026-06-23 21:00:02,281 - __main__ - INFO - Running 3 scrapers concurrently...
2026-06-23 21:00:02,282 - Yiddish24Scraper - INFO - Starting Yiddish24 scraping (max 10 pages, limit 100 items)
2026-06-23 21:00:02,288 - Yiddish24Scraper - INFO - Fetching Yiddish24 page 1 with data_id='latest', page_limit=100
2026-06-23 21:00:02,294 - IveltScraper - INFO - Loaded 181 include, 0 exclude filters from DB topic_filters
2026-06-23 21:00:02,294 - IveltScraper - INFO - Fetching ivelt active topics from https://www.ivelt.com/forum/search.php?search_id=active_topics...
2026-06-23 21:00:02,295 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.theyeshivaworld.com/feed
2026-06-23 21:00:02,563 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.thegatewaypundit.com/feed/
2026-06-23 21:00:03,036 - RSSFeedScraper - INFO - Total RSS/HTML results: 3
2026-06-23 21:00:14,263 - IveltScraper - INFO - Found 50 active topics on ivelt
2026-06-23 21:00:36,229 - Yiddish24Scraper - ERROR - Error fetching Yiddish24 page 1:
2026-06-23 21:00:36,244 - Yiddish24Scraper - ERROR - Traceback (most recent call last):
File "/home/openclaw/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
yield
File "/home/openclaw/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 377, in handle_async_request
resp = await self._pool.handle_async_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
raise exc from None
File "/home/openclaw/.local/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
response = await connection.handle_async_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pool_request.request
^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
return await self._connection.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
raise exc
File "/home/openclaw/.local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
) = await self._receive_response_headers(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
event = await self._receive_event(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
data = await self._network_stream.read(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
self.READ_NUM_BYTES, timeout=timeout
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 32, in read
with map_exceptions(exc_map):
~~~~~~~~~~~~~~^^^^^^^^^
File "/home/linuxbrew/.linuxbrew/opt/python@3.14/lib/python3.14/contextlib.py", line 162, in __exit__
self.gen.throw(value)
~~~~~~~~~~~~~~^^^^^^^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ReadTimeout
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/openclaw/Social_Media_Monitor/src/social_media_monitor/scrapers/yiddish24_scraper.py", line 207, in _fetch_page
response = await self.make_request(
^^^^^^^^^^^^^^^^^^^^^^^^
...<5 lines>...
)
^
File "/home/openclaw/Social_Media_Monitor/src/social_media_monitor/scrapers/base_scraper.py", line 231, in make_request
response = await session.post(url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpx/_client.py", line 1905, in post
return await self.request(
^^^^^^^^^^^^^^^^^^^
...<13 lines>...
)
^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpx/_client.py", line 1585, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpx/_client.py", line 1674, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
)
^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpx/_client.py", line 1702, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<3 lines>...
)
^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpx/_client.py", line 1739, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpx/_client.py", line 1776, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 376, in handle_async_request
with map_httpcore_exceptions():
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/home/linuxbrew/.linuxbrew/opt/python@3.14/lib/python3.14/contextlib.py", line 162, in __exit__
self.gen.throw(value)
~~~~~~~~~~~~~~^^^^^^^
File "/home/openclaw/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ReadTimeout
2026-06-23 21:00:36,244 - Yiddish24Scraper - INFO - Page 1: 0/0 new items
2026-06-23 21:00:36,244 - Yiddish24Scraper - INFO - No more results found at page 1, stopping pagination
2026-06-23 21:00:36,245 - Yiddish24Scraper - INFO - Total Yiddish24 results found: 0
2026-06-23 21:00:39,789 - IveltScraper - INFO - Filtering post containing exclude term: '*ืืืืข ืืืื ืืืืขืจ ืืืืื ืฆื ืืืจืืืื ืงืจืื ืืข ื ืืืขืก*'
2026-06-23 21:00:42,135 - IveltScraper - INFO - ivelt scrape complete. Checked 4 topics.
2026-06-23 21:00:42,135 - IveltScraper - INFO - All forums scrape complete. Found 40 new posts.
2026-06-23 21:00:42,135 - __main__ - INFO - Scraper rss completed: 3 results
2026-06-23 21:00:42,136 - __main__ - INFO - Scraper yiddish24 completed: 0 results
2026-06-23 21:00:42,136 - __main__ - INFO - Scraper ivelt completed: 40 results
2026-06-23 21:00:42,943 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-06-23 21:00:42.943786
2026-06-23 21:00:42,944 - __main__ - INFO - Email sent successfully
2026-06-23 21:00:42,944 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-06-23 21:00:42,944 - __main__ - INFO - Monitor complete. Found 43 total results: 3 new rss results, 40 new ivelt results.
2026-06-23 21:00:42,944 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-06-23 21:00:42,944 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 21:00:42,980 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-06-23 21:00:42,981 - __main__ - INFO - Closed all database connections
2026-06-23 21:00:42,982 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 21:00:42,983 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-06-23 21:30:02,176 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-06-23.log
2026-06-23 21:30:02,177 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-06-23.log
2026-06-23 21:30:02,177 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-06-23.log
2026-06-23 21:30:02,177 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-06-23 21:30:02,202 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-06-23 21:30:02,203 - social_media_monitor.config.settings - WARNING - Twitter monitoring enabled but credentials validation not implemented
2026-06-23 21:30:02,203 - __main__ - INFO - Configuration loaded and validated successfully
2026-06-23 21:30:02,205 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-06-23 21:30:02,205 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-06-23 21:30:02,221 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-06-23 21:30:02,221 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-06-23 21:30:02,265 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-06-23 21:30:02,265 - __main__ - INFO - GitHub scraper initialized and enabled
2026-06-23 21:30:02,266 - __main__ - INFO - GitHub feed scraper initialized and enabled
2026-06-23 21:30:02,266 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-06-23 21:30:02,274 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-06-23 21:30:02,274 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-06-23 21:30:02,277 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-06-23 21:30:02,281 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-06-23 21:30:02,281 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-06-23 21:30:02,284 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-06-23 21:30:02,284 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-06-23 21:30:02,284 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-06-23 21:30:02,284 - __main__ - INFO - Running 2 scrapers concurrently...
2026-06-23 21:30:02,284 - GitHubScraper - INFO - Searching GitHub for terms: ['paperclip', '"Hermes Agent"', 'multica']
2026-06-23 21:30:03,433 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=paperclip+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 21:30:03,458 - GitHubScraper - INFO - Found 240 GitHub repositories for term: paperclip
2026-06-23 21:30:03,597 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=1 "HTTP/1.1 200 OK"
2026-06-23 21:30:04,730 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22Hermes+Agent%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 21:30:04,732 - GitHubScraper - INFO - Found 394 GitHub repositories for term: "Hermes Agent"
2026-06-23 21:30:05,600 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=2 "HTTP/1.1 200 OK"
2026-06-23 21:30:06,345 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=multica+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 21:30:06,346 - GitHubScraper - INFO - Found 53 GitHub repositories for term: multica
2026-06-23 21:30:06,475 - GitHubScraper - INFO - Total GitHub results found: 0
2026-06-23 21:30:07,460 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=3 "HTTP/1.1 200 OK"
2026-06-23 21:30:08,213 - GitHubScraper - INFO - GitHub feed: 298 fetched across pages, 0 new events for abeperl
2026-06-23 21:30:08,214 - __main__ - INFO - Scraper github completed: 0 results
2026-06-23 21:30:08,214 - __main__ - INFO - Scraper github_feed completed: 0 results
2026-06-23 21:30:08,214 - __main__ - INFO - No new results found across all enabled sources
2026-06-23 21:30:08,215 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-06-23 21:30:08,215 - __main__ - INFO - Monitor complete. No new results found across all sources.
2026-06-23 21:30:08,215 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-06-23 21:30:08,215 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 21:30:08,339 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-06-23 21:30:08,339 - __main__ - INFO - Closed all database connections
2026-06-23 21:30:08,341 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 21:30:08,341 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-06-23 22:00:02,418 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-06-23.log
2026-06-23 22:00:02,418 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-06-23.log
2026-06-23 22:00:02,418 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-06-23.log
2026-06-23 22:00:02,418 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-06-23 22:00:02,448 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-06-23 22:00:02,448 - __main__ - INFO - Configuration loaded and validated successfully
2026-06-23 22:00:02,450 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-06-23 22:00:02,451 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-06-23 22:00:02,470 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-06-23 22:00:02,470 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-06-23 22:00:02,471 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-06-23 22:00:02,472 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-06-23 22:00:02,472 - __main__ - INFO - RSS scraper initialized and enabled
2026-06-23 22:00:02,475 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-06-23 22:00:02,475 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-06-23 22:00:02,475 - __main__ - INFO - Yiddish24 scraper initialized and enabled
2026-06-23 22:00:02,479 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-06-23 22:00:02,483 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-06-23 22:00:02,483 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-06-23 22:00:02,485 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-06-23 22:00:02,485 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-06-23 22:00:02,485 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-06-23 22:00:02,486 - __main__ - INFO - Ivelt scraper initialized and enabled
2026-06-23 22:00:02,486 - __main__ - INFO - Running 3 scrapers concurrently...
2026-06-23 22:00:02,486 - Yiddish24Scraper - INFO - Starting Yiddish24 scraping (max 10 pages, limit 100 items)
2026-06-23 22:00:02,569 - Yiddish24Scraper - INFO - Fetching Yiddish24 page 1 with data_id='latest', page_limit=100
2026-06-23 22:00:02,577 - IveltScraper - INFO - Loaded 181 include, 0 exclude filters from DB topic_filters
2026-06-23 22:00:02,577 - IveltScraper - INFO - Fetching ivelt active topics from https://www.ivelt.com/forum/search.php?search_id=active_topics...
2026-06-23 22:00:02,578 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.theyeshivaworld.com/feed
2026-06-23 22:00:02,828 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.thegatewaypundit.com/feed/
2026-06-23 22:00:03,251 - RSSFeedScraper - INFO - Total RSS/HTML results: 3
2026-06-23 22:00:08,676 - httpx - INFO - HTTP Request: POST https://www.yiddish24.com/ajax/cat_pagination.php "HTTP/1.1 200 OK"
2026-06-23 22:00:08,887 - Yiddish24Scraper - INFO - Received response with 469484 characters
2026-06-23 22:00:08,889 - Yiddish24Scraper - INFO - Extracted HTML from JSON result field (303251 chars)
2026-06-23 22:00:08,890 - Yiddish24Scraper - INFO - Saved HTML debug file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/debug/yiddish24_page1_response.html
2026-06-23 22:00:08,963 - Yiddish24Scraper - INFO - Found 100 potential article elements on page 1
2026-06-23 22:00:09,003 - Yiddish24Scraper - INFO - Page 1 extraction: 100 successful, 0 failed out of 100 elements
2026-06-23 22:00:09,003 - Yiddish24Scraper - INFO - Parsed 100 articles from HTML on page 1
2026-06-23 22:00:09,741 - Yiddish24Scraper - INFO - Page 1 summary: 16 new out of 100 total articles
2026-06-23 22:00:09,741 - Yiddish24Scraper - INFO - Page 1: 16/100 new items
2026-06-23 22:00:09,741 - Yiddish24Scraper - INFO - Low new item ratio (16.0%), stopping pagination
2026-06-23 22:00:09,742 - Yiddish24Scraper - INFO - Total Yiddish24 results found: 16
2026-06-23 22:00:14,311 - IveltScraper - INFO - Found 50 active topics on ivelt
2026-06-23 22:00:33,626 - IveltScraper - INFO - ivelt scrape complete. Checked 4 topics.
2026-06-23 22:00:33,626 - IveltScraper - INFO - All forums scrape complete. Found 57 new posts.
2026-06-23 22:00:33,626 - __main__ - INFO - Scraper rss completed: 3 results
2026-06-23 22:00:33,626 - __main__ - INFO - Scraper yiddish24 completed: 16 results
2026-06-23 22:00:33,626 - __main__ - INFO - Scraper ivelt completed: 57 results
2026-06-23 22:00:34,639 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-06-23 22:00:34.639481
2026-06-23 22:00:34,639 - __main__ - INFO - Email sent successfully
2026-06-23 22:00:34,640 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-06-23 22:00:34,640 - __main__ - INFO - Monitor complete. Found 76 total results: 3 new rss results, 16 new yiddish24 results, 57 new ivelt results.
2026-06-23 22:00:34,640 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-06-23 22:00:34,640 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 22:00:34,766 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-06-23 22:00:34,766 - __main__ - INFO - Closed all database connections
2026-06-23 22:00:34,770 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 22:00:34,770 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-06-23 22:30:01,816 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-06-23.log
2026-06-23 22:30:01,817 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-06-23.log
2026-06-23 22:30:01,817 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-06-23.log
2026-06-23 22:30:01,817 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-06-23 22:30:01,844 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-06-23 22:30:01,844 - social_media_monitor.config.settings - WARNING - Twitter monitoring enabled but credentials validation not implemented
2026-06-23 22:30:01,844 - __main__ - INFO - Configuration loaded and validated successfully
2026-06-23 22:30:01,847 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-06-23 22:30:01,847 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-06-23 22:30:01,871 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-06-23 22:30:01,871 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-06-23 22:30:01,872 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-06-23 22:30:01,872 - __main__ - INFO - GitHub scraper initialized and enabled
2026-06-23 22:30:01,872 - __main__ - INFO - GitHub feed scraper initialized and enabled
2026-06-23 22:30:01,874 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-06-23 22:30:01,876 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-06-23 22:30:01,876 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-06-23 22:30:01,879 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-06-23 22:30:01,883 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-06-23 22:30:01,883 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-06-23 22:30:01,885 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-06-23 22:30:01,885 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-06-23 22:30:01,886 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-06-23 22:30:01,886 - __main__ - INFO - Running 2 scrapers concurrently...
2026-06-23 22:30:01,886 - GitHubScraper - INFO - Searching GitHub for terms: ['paperclip', '"Hermes Agent"', 'multica']
2026-06-23 22:30:03,099 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=paperclip+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 22:30:03,154 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=1 "HTTP/1.1 200 OK"
2026-06-23 22:30:03,191 - GitHubScraper - INFO - Found 240 GitHub repositories for term: paperclip
2026-06-23 22:30:04,462 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22Hermes+Agent%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 22:30:04,467 - GitHubScraper - INFO - Found 394 GitHub repositories for term: "Hermes Agent"
2026-06-23 22:30:05,237 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=2 "HTTP/1.1 200 OK"
2026-06-23 22:30:06,000 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=multica+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 22:30:06,093 - GitHubScraper - INFO - Found 53 GitHub repositories for term: multica
2026-06-23 22:30:06,216 - GitHubScraper - INFO - Total GitHub results found: 0
2026-06-23 22:30:07,343 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=3 "HTTP/1.1 200 OK"
2026-06-23 22:30:08,263 - GitHubScraper - INFO - GitHub feed: 298 fetched across pages, 2 new events for abeperl
2026-06-23 22:30:08,264 - __main__ - INFO - Scraper github completed: 0 results
2026-06-23 22:30:08,264 - __main__ - INFO - Scraper github_feed completed: 2 results
2026-06-23 22:30:09,027 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-06-23 22:30:09.026935
2026-06-23 22:30:09,027 - __main__ - INFO - Email sent successfully
2026-06-23 22:30:09,027 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-06-23 22:30:09,027 - __main__ - INFO - Monitor complete. Found 2 total results: 2 new github feed results.
2026-06-23 22:30:09,027 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-06-23 22:30:09,027 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 22:30:09,120 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-06-23 22:30:09,120 - __main__ - INFO - Closed all database connections
2026-06-23 22:30:09,122 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 22:30:09,122 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-06-23 23:00:02,509 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-06-23.log
2026-06-23 23:00:02,509 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-06-23.log
2026-06-23 23:00:02,510 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-06-23.log
2026-06-23 23:00:02,510 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-06-23 23:00:02,537 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-06-23 23:00:02,537 - __main__ - INFO - Configuration loaded and validated successfully
2026-06-23 23:00:02,539 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-06-23 23:00:02,539 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-06-23 23:00:02,592 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-06-23 23:00:02,593 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-06-23 23:00:02,640 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-06-23 23:00:02,642 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-06-23 23:00:02,642 - __main__ - INFO - RSS scraper initialized and enabled
2026-06-23 23:00:02,648 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-06-23 23:00:02,648 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-06-23 23:00:02,648 - __main__ - INFO - Yiddish24 scraper initialized and enabled
2026-06-23 23:00:02,653 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-06-23 23:00:02,658 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-06-23 23:00:02,658 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-06-23 23:00:02,660 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-06-23 23:00:02,660 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-06-23 23:00:02,660 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-06-23 23:00:02,661 - __main__ - INFO - Ivelt scraper initialized and enabled
2026-06-23 23:00:02,661 - __main__ - INFO - Running 3 scrapers concurrently...
2026-06-23 23:00:02,661 - Yiddish24Scraper - INFO - Starting Yiddish24 scraping (max 10 pages, limit 100 items)
2026-06-23 23:00:02,749 - Yiddish24Scraper - INFO - Fetching Yiddish24 page 1 with data_id='latest', page_limit=100
2026-06-23 23:00:02,760 - IveltScraper - INFO - Loaded 181 include, 0 exclude filters from DB topic_filters
2026-06-23 23:00:02,760 - IveltScraper - INFO - Fetching ivelt active topics from https://www.ivelt.com/forum/search.php?search_id=active_topics...
2026-06-23 23:00:02,761 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.theyeshivaworld.com/feed
2026-06-23 23:00:03,092 - RSSFeedScraper - INFO - Fetching RSS feed: https://www.thegatewaypundit.com/feed/
2026-06-23 23:00:05,034 - RSSFeedScraper - INFO - Total RSS/HTML results: 3
2026-06-23 23:00:08,222 - httpx - INFO - HTTP Request: POST https://www.yiddish24.com/ajax/cat_pagination.php "HTTP/1.1 200 OK"
2026-06-23 23:00:08,438 - Yiddish24Scraper - INFO - Received response with 465521 characters
2026-06-23 23:00:08,440 - Yiddish24Scraper - INFO - Extracted HTML from JSON result field (306944 chars)
2026-06-23 23:00:08,441 - Yiddish24Scraper - INFO - Saved HTML debug file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/debug/yiddish24_page1_response.html
2026-06-23 23:00:08,512 - Yiddish24Scraper - INFO - Found 100 potential article elements on page 1
2026-06-23 23:00:08,558 - Yiddish24Scraper - INFO - Page 1 extraction: 100 successful, 0 failed out of 100 elements
2026-06-23 23:00:08,558 - Yiddish24Scraper - INFO - Parsed 100 articles from HTML on page 1
2026-06-23 23:00:08,570 - Yiddish24Scraper - INFO - Term 'ืฆืื ืคืื ืื -' matched in DESCRIPTION: 'ืฆืื ืคืื ืื - ืืืกื ืืขืืขืฆื ืืื ืืื ืืืกืื ืืืื ืขืจ ืคืื ืืื'
2026-06-23 23:00:08,570 - Yiddish24Scraper - INFO - Downloading 1 media file(s) for article: ืืื ืืื
2026-06-23 23:00:09,370 - social_media_monitor.utils.file_downloader - INFO - Starting download from: https://cloudfront.yiddish24.com/_____3_260_260623125513.mp3
2026-06-23 23:00:09,421 - social_media_monitor.utils.file_downloader - INFO - Successfully downloaded: Y2______3_260_260623125513.mp3 (2.4 MB)
2026-06-23 23:00:09,421 - social_media_monitor.utils.file_downloader - INFO - Organized audio file to: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/downloads/Audio (audio/mpeg)
2026-06-23 23:00:09,427 - social_media_monitor.utils.media_tagger - INFO - Tagged Y2______3_260_260623125513.mp3: title, artist, album, comment, source_url, genre, publisher
2026-06-23 23:00:09,427 - Yiddish24Scraper - INFO - Renamed downloaded file to include match term: Y2______3_260_260623125513_ืฆืื_ืคืื ืื_-.mp3
2026-06-23 23:00:10,289 - Yiddish24Scraper - INFO - Page 1 summary: 8 new out of 100 total articles
2026-06-23 23:00:10,290 - Yiddish24Scraper - INFO - Page 1: 8/100 new items
2026-06-23 23:00:10,290 - Yiddish24Scraper - INFO - Low new item ratio (8.0%), stopping pagination
2026-06-23 23:00:10,290 - Yiddish24Scraper - INFO - Total Yiddish24 results found: 8
2026-06-23 23:00:14,562 - IveltScraper - INFO - Found 50 active topics on ivelt
2026-06-23 23:01:01,774 - IveltScraper - INFO - ivelt scrape complete. Checked 5 topics.
2026-06-23 23:01:01,775 - IveltScraper - INFO - All forums scrape complete. Found 80 new posts.
2026-06-23 23:01:01,775 - __main__ - INFO - Scraper rss completed: 3 results
2026-06-23 23:01:01,775 - __main__ - INFO - Scraper yiddish24 completed: 8 results
2026-06-23 23:01:01,775 - __main__ - INFO - Scraper ivelt completed: 80 results
2026-06-23 23:01:02,663 - social_media_monitor.email.sender - INFO - Email sent successfully at 2026-06-23 23:01:02.663274
2026-06-23 23:01:02,663 - __main__ - INFO - Email sent successfully
2026-06-23 23:01:02,663 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-06-23 23:01:02,663 - __main__ - INFO - Monitor complete. Found 91 total results: 3 new rss results, 8 new yiddish24 results, 80 new ivelt results.
2026-06-23 23:01:02,663 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-06-23 23:01:02,663 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 23:01:02,829 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-06-23 23:01:02,829 - __main__ - INFO - Closed all database connections
2026-06-23 23:01:02,830 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 23:01:02,830 - social_media_monitor.database.connection - INFO - Closed 0 database connections
2026-06-23 23:30:01,957 - social_media_monitor.utils.logging_config - INFO - Posts log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_posts_2026-06-23.log
2026-06-23 23:30:01,957 - social_media_monitor.utils.logging_config - INFO - Downloads log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_downloads_2026-06-23.log
2026-06-23 23:30:01,958 - social_media_monitor.utils.logging_config - INFO - Logging configured with level INFO, log file: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/logs/social_media_monitor_2026-06-23.log
2026-06-23 23:30:01,958 - __main__ - INFO - === Social Media Monitor v2.0 (Async) Started ===
2026-06-23 23:30:01,987 - social_media_monitor.config.settings - INFO - Configuration loaded from /home/openclaw/Social_Media_Monitor/config.yaml
2026-06-23 23:30:01,987 - social_media_monitor.config.settings - WARNING - Twitter monitoring enabled but credentials validation not implemented
2026-06-23 23:30:01,988 - __main__ - INFO - Configuration loaded and validated successfully
2026-06-23 23:30:01,990 - social_media_monitor.database.connection - INFO - Database connection pool initialized with 5 connections
2026-06-23 23:30:01,990 - social_media_monitor.database.connection - INFO - add_speaker_filters column already exists
2026-06-23 23:30:02,016 - social_media_monitor.database.connection - INFO - Database setup completed successfully
2026-06-23 23:30:02,016 - __main__ - INFO - Database initialized at: /mnt/data/openclaw-shared/home/shared/Social_Media_Monitor/social_media_monitor.db
2026-06-23 23:30:02,017 - social_media_monitor.scrapers.reddit_scraper - INFO - Reddit API initialized successfully
2026-06-23 23:30:02,017 - __main__ - INFO - GitHub scraper initialized and enabled
2026-06-23 23:30:02,017 - __main__ - INFO - GitHub feed scraper initialized and enabled
2026-06-23 23:30:02,018 - RSSFeedScraper - INFO - RSS scraper initialized: feeds=2, html_urls=0 (0 with custom selectors), include_terms=0, exclude_terms=0
2026-06-23 23:30:02,021 - Yiddish24Scraper - INFO - File downloader initialized with 55 terms, prefix 'Y2', and file type organization enabled
2026-06-23 23:30:02,021 - Yiddish24Scraper - INFO - Yiddish24 scraper initialized: data_id='latest', page_limit=100, total_pages=10, debug_html=True, include_terms=55, exclude_terms=0
2026-06-23 23:30:02,024 - AhBlickLiveScraper - INFO - AhBlickLive scraper initialized: urls=1, max_items=100, include_terms=49, exclude_terms=0
2026-06-23 23:30:02,028 - TorahAnytimeScraper - INFO - File downloader initialized with 53 terms, prefix 'TA', and file type organization enabled
2026-06-23 23:30:02,028 - TorahAnytimeScraper - INFO - TorahAnytime scraper initialized: limit=400, offset=0, project_id=1, include_terms=53, exclude_terms=10
2026-06-23 23:30:02,031 - BatorahScraper - INFO - File downloader initialized with 19 terms, prefix 'BT', and file type organization enabled
2026-06-23 23:30:02,031 - BatorahScraper - INFO - Batorah scraper initialized: api_urls=2, page_size=100, include_terms=19, exclude_terms=1
2026-06-23 23:30:02,031 - KolHalashonScraper - INFO - Kol Halashon scraper initialized: base_url=https://www2.kolhalashon.com, max_shiurim=50, fetch_all_speakers=True, headless=False, exclude_filters=1, include_filters=0
2026-06-23 23:30:02,031 - __main__ - INFO - Running 2 scrapers concurrently...
2026-06-23 23:30:02,031 - GitHubScraper - INFO - Searching GitHub for terms: ['paperclip', '"Hermes Agent"', 'multica']
2026-06-23 23:30:03,107 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=paperclip+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 23:30:03,133 - GitHubScraper - INFO - Found 240 GitHub repositories for term: paperclip
2026-06-23 23:30:03,350 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=1 "HTTP/1.1 200 OK"
2026-06-23 23:30:04,364 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=%22Hermes+Agent%22+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 23:30:04,368 - GitHubScraper - INFO - Found 393 GitHub repositories for term: "Hermes Agent"
2026-06-23 23:30:05,381 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=2 "HTTP/1.1 200 OK"
2026-06-23 23:30:06,064 - httpx - INFO - HTTP Request: GET https://api.github.com/search/repositories?q=multica+in%3Aname%2Cdescription%2Creadme+stars%3A%3E150&sort=updated&order=desc&per_page=20 "HTTP/1.1 200 OK"
2026-06-23 23:30:06,069 - GitHubScraper - INFO - Found 53 GitHub repositories for term: multica
2026-06-23 23:30:06,190 - GitHubScraper - INFO - Total GitHub results found: 0
2026-06-23 23:30:07,704 - httpx - INFO - HTTP Request: GET https://api.github.com/users/abeperl/received_events?per_page=100&page=3 "HTTP/1.1 200 OK"
2026-06-23 23:30:08,693 - GitHubScraper - INFO - GitHub feed: 298 fetched across pages, 0 new events for abeperl
2026-06-23 23:30:08,694 - __main__ - INFO - Scraper github completed: 0 results
2026-06-23 23:30:08,694 - __main__ - INFO - Scraper github_feed completed: 0 results
2026-06-23 23:30:08,694 - __main__ - INFO - No new results found across all enabled sources
2026-06-23 23:30:08,694 - __main__ - INFO - Cache stats: 0 items in memory, 0 files on disk
2026-06-23 23:30:08,694 - __main__ - INFO - Monitor complete. No new results found across all sources.
2026-06-23 23:30:08,695 - __main__ - INFO - === Social Media Monitor v2.0 Complete ===
2026-06-23 23:30:08,695 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 23:30:08,759 - social_media_monitor.database.connection - INFO - Closed 5 database connections
2026-06-23 23:30:08,760 - __main__ - INFO - Closed all database connections
2026-06-23 23:30:08,761 - social_media_monitor.database.connection - INFO - Closing all database connections
2026-06-23 23:30:08,761 - social_media_monitor.database.connection - INFO - Closed 0 database connections
Live tail active โ waiting for new lines...