can you check the repo in question hooks ? the post commit hook, how does it look ?
It looks very… normal.
On the server side, it also looks quite good.
>> rhodecode-docker-5.1.0.beta3 hooks $ cat post-commit
#!/home/rhodecode/venv/bin/python
import os
import sys
path_adjust = []
if path_adjust:
sys.path = path_adjust
try:
from vcsserver import hooks
except ImportError:
if os.environ.get('RC_DEBUG_SVN_HOOK'):
import traceback
print(traceback.format_exc())
hooks = None
# TIMESTAMP: 2024-03-04T12:51:11.767387
RC_HOOK_VER = '5.1.0'
# special trick to pass in some information from rc to hooks
# mod_dav strips ALL env vars and we can't even access things like PATH
for env_k, env_v in [('RC_CORE_BINARY_DIR', '/usr/local/bin/rhodecode_bin/vcs_bin'), ('RC_GIT_EXECUTABLE', '/usr/local/bin/rhodecode_bin/vcs_bin/git'), ('RC_SVN_EXECUTABLE', '/usr/local/bin/rhodecode_bin/vcs_bin/svn'), ('RC_SVNLOOK_EXECUTABLE', '/usr/local/bin/rhodecode_bin/vcs_bin/svnlook')]:
os.environ[env_k] = env_v
def main():
if hooks is None:
# exit with success if we cannot import vcsserver.hooks !!
# this allows simply push to this repo even without rhodecode
sys.exit(0)
if os.environ.get('RC_SKIP_HOOKS') or os.environ.get('RC_SKIP_SVN_HOOKS'):
sys.exit(0)
repo_path = os.getcwd()
push_data = sys.argv[1:]
os.environ['RC_HOOK_VER'] = RC_HOOK_VER
try:
result = hooks.svn_post_commit(repo_path, push_data, os.environ)
sys.exit(result)
except Exception as error:
# TODO: johbo: Improve handling of this special case
if not getattr(error, '_vcs_kind', None) == 'repo_locked':
raise
print(f'ERROR: {error}')
sys.exit(1)
sys.exit(0)
if __name__ == '__main__':
main()
I’ve just added an email integration also at the repo level: doesn’t change anything, unfortunately.
Can you try latest beta build ?
Not better with 5.1.0.11.03.2024.
No noticeable errors in the system logs. Not more in the rhodecode logs.
I forgot to mention that I don’t have a mail sender on my server, but emails do work when tested from the Settings > Email page.
are there any mentions in celery logs about running any integrations ?
Where should I look to get those logs?
I searched by myself but didn’t find anything relevant.
Could you please point out where are the Celery logs stored? Should I enable any kind of debug mode?
you can run ./rcstack stack rhodecode logs celery
No, I don’t see any error regarding integrations. But still no entries in the journal, and no e-mail received.
celery-1 | ENTRYPOINT: Running rhodecode_celery with cmd '/usr/local/bin/rhodecode_bin/bin/celery'
celery-1 | {"timestamp": "2024-03-28T13:51:59.905742+00:00", "levelname": "INFO", "name": "rhodecode.config.middleware", "message": "Pyramid app config starting", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-03-28T13:51:59.907844+00:00", "levelname": "INFO", "name": "rhodecode.config.settings_maker", "message": "logging configuration based on main .ini file", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-03-28T13:51:59.919101+00:00", "levelname": "ERROR", "name": "rhodecode.statsd", "message": "StatsD is enabled, but failed to connect to statsd server, fallback: disable statsd", "req_id": "00000000-0000-0000-0000-000000000000", "exc_info": "Traceback (most recent call last):\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/_vendor/statsd/__init__.py\", line 43, in client_from_config\n client = StatsClient(\n ^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/_vendor/statsd/udp.py\", line 34, in __init__\n family, _, _, _, addr = socket.getaddrinfo(\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.11/socket.py\", line 962, in getaddrinfo\n for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name or service not known"}
celery-1 | {"timestamp": "2024-03-28T13:51:59.976900+00:00", "levelname": "INFO", "name": "rhodecode.model", "message": "RhodeCode 5.1.0 initializing db for postgresql://rhodecode:***@database/rhodecode", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-03-28T13:51:59.977649+00:00", "levelname": "INFO", "name": "rhodecode.config.environment", "message": "Enabled VCS backends: odict_keys(['hg', 'git', 'svn'])", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-03-28T13:52:02.747086+00:00", "levelname": "INFO", "name": "celery.rhodecode.loader", "message": "Configuring celery based on `{'broker_url': 'redis://redis:6379/8', 'max_tasks_per_child': 20, 'task_always_eager': False}` settings", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-03-28T13:52:02.748135+00:00", "levelname": "INFO", "name": "rhodecode.config.middleware", "message": "Pyramid app created and configured in 2.84s", "req_id": "00000000-0000-0000-0000-000000000000"}
celery-1 | {"timestamp": "2024-03-28T13:52:02.753903+00:00", "levelname": "ERROR", "name": "rhodecode.statsd", "message": "StatsD is enabled, but failed to connect to statsd server, fallback: disable statsd", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc", "exc_info": "Traceback (most recent call last):\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/_vendor/statsd/__init__.py\", line 43, in client_from_config\n client = StatsClient(\n ^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode/lib/_vendor/statsd/udp.py\", line 34, in __init__\n family, _, _, _, addr = socket.getaddrinfo(\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.11/socket.py\", line 962, in getaddrinfo\n for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name or service not known"}
celery-1 |
celery-1 | -------------- celery@be52c9877da3 v5.3.6 (emerald-rush)
celery-1 | --- ***** -----
celery-1 | -- ******* ---- Linux-5.15.0-101-generic-x86_64-with-glibc2.35 2024-03-28 13:52:03
celery-1 | - *** --- * ---
celery-1 | - ** ---------- [config]
celery-1 | - ** ---------- .> app: __main__:0x7f4c75238190
celery-1 | - ** ---------- .> transport: redis://redis:6379/8
celery-1 | - ** ---------- .> results: rpc://
celery-1 | - *** --- * --- .> concurrency: {min=2, max=20} (prefork)
celery-1 | -- ******* ---- .> task events: ON
celery-1 | --- ***** -----
celery-1 | -------------- [queues]
celery-1 | .> celery exchange=celery(direct) key=celery
celery-1 |
celery-1 |
celery-1 | [tasks]
celery-1 | . celery.accumulate
celery-1 | . celery.backend_cleanup
celery-1 | . celery.chain
celery-1 | . celery.chord
celery-1 | . celery.chord_unlock
celery-1 | . celery.chunks
celery-1 | . celery.group
celery-1 | . celery.map
celery-1 | . celery.starmap
celery-1 | . rhodecode.integrations.types.slack.post_text_to_slack
celery-1 | . rhodecode.integrations.types.webhook.post_to_webhook
celery-1 | . rhodecode.lib.celerylib.tasks.beat_check
celery-1 | . rhodecode.lib.celerylib.tasks.check_for_update
celery-1 | . rhodecode.lib.celerylib.tasks.create_repo
celery-1 | . rhodecode.lib.celerylib.tasks.create_repo_fork
celery-1 | . rhodecode.lib.celerylib.tasks.post_pull
celery-1 | . rhodecode.lib.celerylib.tasks.post_push
celery-1 | . rhodecode.lib.celerylib.tasks.pre_pull
celery-1 | . rhodecode.lib.celerylib.tasks.pre_push
celery-1 | . rhodecode.lib.celerylib.tasks.repo_maintenance
celery-1 | . rhodecode.lib.celerylib.tasks.repo_size
celery-1 | . rhodecode.lib.celerylib.tasks.send_email
celery-1 | . rhodecode.lib.celerylib.tasks.sync_last_update
celery-1 |
celery-1 | {"timestamp": "2024-03-28T13:52:03.312020+00:00", "levelname": "INFO", "name": "celery.worker.consumer.connection", "message": "Connected to redis://redis:6379/8", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
celery-1 | {"timestamp": "2024-03-28T13:52:03.321784+00:00", "levelname": "INFO", "name": "celery.worker.consumer.mingle", "message": "mingle: searching for neighbors", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
celery-1 | {"timestamp": "2024-03-28T13:52:04.331454+00:00", "levelname": "INFO", "name": "celery.worker.consumer.mingle", "message": "mingle: all alone", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
celery-1 | {"timestamp": "2024-03-28T13:52:04.346362+00:00", "levelname": "INFO", "name": "celery.apps.worker", "message": "celery@be52c9877da3 ready.", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
celery-1 | {"timestamp": "2024-04-09T14:10:36.560753+00:00", "levelname": "WARNING", "name": "celery.worker.consumer.consumer", "message": "consumer: Connection to broker lost. Trying to re-establish the connection...", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc", "exc_info": "Traceback (most recent call last):\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py\", line 340, in start\n blueprint.start(self)\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/celery/bootsteps.py\", line 116, in start\n step.start(parent)\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py\", line 742, in start\n c.loop(*c.loop_args())\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/celery/worker/loops.py\", line 97, in asynloop\n next(loop)\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/kombu/asynchronous/hub.py\", line 373, in create_loop\n cb(*cbargs)\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/kombu/transport/redis.py\", line 1344, in on_readable\n self.cycle.on_readable(fileno)\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/kombu/transport/redis.py\", line 569, in on_readable\n chan.handlers[type]()\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/kombu/transport/redis.py\", line 916, in _receive\n while c.connection is not None and c.connection.can_read(timeout=0):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/redis/connection.py\", line 478, in can_read\n return self._parser.can_read(timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/redis/_parsers/base.py\", line 128, in can_read\n return self._buffer and self._buffer.can_read(timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/redis/_parsers/socket.py\", line 95, in can_read\n return bool(self.unread_bytes()) or self._read_from_socket(\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/rhodecode/venv/lib/python3.11/site-packages/redis/_parsers/socket.py\", line 68, in _read_from_socket\n raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)\nredis.exceptions.ConnectionError: Connection closed by server."}
celery-1 | {"timestamp": "2024-04-09T14:10:36.770079+00:00", "levelname": "WARNING", "name": "py.warnings", "message": "/home/rhodecode/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:391: CPendingDeprecationWarning: \nIn Celery 5.1 we introduced an optional breaking change which\non connection loss cancels all currently executed tasks with late acknowledgement enabled.\nThese tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered\nback to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss\nsetting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.\n\n warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)\n", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
celery-1 | {"timestamp": "2024-04-09T14:10:36.778352+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to redis://redis:6379/8: Error 111 connecting to redis:6379. Connection refused..\nTrying again in 2.00 seconds... (1/100)\n", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
celery-1 | {"timestamp": "2024-04-09T14:10:38.839451+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to redis://redis:6379/8: Redis is loading the dataset in memory.\nTrying again in 4.00 seconds... (2/100)\n", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
celery-1 | {"timestamp": "2024-04-09T14:10:45.482046+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to redis://redis:6379/8: Error 111 connecting to redis:6379. Connection refused..\nTrying again in 6.00 seconds... (3/100)\n", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
celery-1 | {"timestamp": "2024-04-09T14:10:59.631879+00:00", "levelname": "ERROR", "name": "celery.worker.consumer.consumer", "message": "consumer: Cannot connect to redis://redis:6379/8: Redis is loading the dataset in memory.\nTrying again in 8.00 seconds... (4/100)\n", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
celery-1 | {"timestamp": "2024-04-09T14:11:07.680506+00:00", "levelname": "INFO", "name": "celery.worker.consumer.connection", "message": "Connected to redis://redis:6379/8", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
celery-1 | {"timestamp": "2024-04-09T14:11:07.695293+00:00", "levelname": "INFO", "name": "celery.worker.consumer.mingle", "message": "mingle: searching for neighbors", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
celery-1 | {"timestamp": "2024-04-09T14:11:08.715829+00:00", "levelname": "INFO", "name": "celery.worker.consumer.mingle", "message": "mingle: all alone", "req_id": "d527e2d9-7562-474d-a80c-a7c2673bd5bc"}
