Hello,
I have the Community edition of RhodeCode 5.2.1 installed (rcstack) on a VPS. As because Elasticsearch is not a community feature, I enabled WHOOSH instead.
I have imported some SVN repositories from an other server. The “Full Text Search” page of the Settings still shows “Indexed documents: 0”.
From the documentation, I find no equivalent with rcstack to what we had with the ‘rhodecode-index’ command.
Any clues to force re-indexation?
Thanks.
Hi,
you can access rhodecode-tools CLI by running this:
./rcstack cli attach
This has tools installed, we’re working on documenting this and making it more explicit how to build it using rcstack
Understood, thanks.
But unfortunately this doesn’t solve my problem because rhodecode-list-instances still returns:
No instances present.
And if I don’t have an instance name, I can’t use rhodecode-index of course.
There’s no need to write a permanent instance key/host into the config file. You can simply call it like that:
rhodecode-index --api-key=xxx --api-host=http://rc-host --config=/vol/confvolume/.rhoderc
I’m quite new to RhodeCode, so I’m not sure where am I supposed to find/create the API credentials. If I use a personal Auth Token, I get:
ERROR Calling method `get_server_info`. Server reported: API KEY invalid or, has bad role for an API call
ERROR Calling method `store_exception`. Server reported: API KEY invalid or, has bad role for an API call
Can you help me on this?
Please see more here: Full-text Search — RhodeCode Enterprise 4.27.1 4.27.1 documentation
best option is to create an API role token for super-admin account so it would have access to all repos in order to index them.
FYI: We also shiped rcstack 5.3.0 now that allows a easier indexing using a simple command. For exmaple:
./rcstack cli cmd allows to run any of the rhodecode-tools commands
./rcstack cli cmd rhodecode-index --api-key=xxx --api-host=http://rc-host --config=/vol/confvolume/.rhoderc
OK it works now. I thought that my existing Auth Token was good for API calls, but it turned that it wasn’t true, so I created a new one.
For future reference, here is what I finally do:
cd ~/docker-rhodecode/ && sudo ./rcstack cli attach
> touch /root/.rhoderc
> rhodecode-index --api-host=http://my.domain.com --api-key=<API_KEY>
Still, during the index process, I can notice two things:
FETCH [TH1/th1_scripts_keithley] commit info [done in: 0.507s]
ERROR Calling method `get_repo_fts_tree`. Server reported: failed to get repo: `TH1/th1_scripts_keithley` nodes
Stored exception as ID: 140719630028784
SystemExit, SystemExit Explanation:
ERROR Calling method `get_repo_fts_tree`. Server reported: failed to get repo: `TH1/th1_scripts_keithley` nodes
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/commands/index.py", line 199, in cmd
main(ctx, instance, engine_location, es_version, repo_dir, mapping, source,
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/commands/index.py", line 494, in main
cli.run_with_lock(_build_index, ctx, 'index_build')
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/lib/cli.py", line 368, in run_with_lock
func(ctx)
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/commands/index.py", line 483, in _build_index
func()
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/commands/index.py", line 464, in __files
api_data['nodes'] = get_nodes_data(
^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/lib/repo_data_fetch.py", line 121, in get_nodes_data
response = api_call(
^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/lib/api.py", line 261, in api_call
raise_sys_exit(1, error_message=msg)
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/lib/utils.py", line 393, in raise_sys_exit
raise exc # pragma: no cover
^^^^^^^^^
SystemExit: 1
You can exlcude certain files or patterns by making the index config file (please see indexing docs).
But can you look into logs for rcstack:
./rcstack stack rhodecode logs
To look for an exception that is causing this file to be fetch for full-text-search parsing ?
Now that it worked the first time, I’m not sure to understand: is the indexing process supposed to run out-of-the-box at each commit? Or should we plan to run it frequently?
Yes,
The indexer is smart and it will onle re-index things that changed or are new.
It’s up to end users to choose cadence of how often you should re-index to get the latest updated version
I have planned a cron task from the host based on what you proposed:
sudo ./rcstack cli cmd rhodecode-index --index-types=commits --config=/etc/rhodecode/conf/rhoderc_indexer.conf --api-key=xxx --api-host=http://rc-host
I have excluded the indexing of file contents because of my low performance server.
It should work now, thank you!
Well, it doesn’t work actually. Not sure if it has already worked or not (has it been broken recently?).
At first I thought that it was an issue related to access rights, so I switched the cron job to be attached to the root user. Not better.
The error is: “the input device is not a TTY”
Jan 8 08:00:01 work CRON[880283]: (root) CMD ((cd /home/ubuntu/docker-rhodecode/ ; sudo ./rcstack cli cmd rhodecode-index --index-types=commits --config=/etc/rhodecode/conf/rhoderc_indexer.conf --api-host=<DOMAIN> --api-key=<API_KEY>) 2>&1 | logger -t rhodecode-index)
Jan 8 08:00:01 work rhodecode-index: attaching /home/ubuntu/docker-rhodecode/config/_shared under: /vol/confvolume
Jan 8 08:00:01 work rhodecode-index: the input device is not a TTY
Is the indexer really designed to be run outside of an interactive command line?
So the docker run command runs witn --tty --interactive flags, i think if it’s under cron then it breaks, because there’s no TTY to attach.
I believe we need to add an explicit flag for running without a TTY, or remove those flags…
5.6.0 is out, and you can now add --no-tty for automated calls
Thank you.
Unfortunately 5.6.0 has broken the rhodecode-index command:
sudo ./rcstack cli cmd rhodecode-index --no-tty --index-types=commits --config=/etc/rhodecode/conf/rhoderc_indexer.conf --api-host=https://my.domain.com --api-key=<API_KEY>
attaching /home/ubuntu/docker-rhodecode/config/_shared under: /vol/confvolume
ENTRYPOINT: Running rhodecode_vcsserver with cmd 'rhodecode-index'
checking if config files needs bootstrapping
ERROR: Failed to connect to backend at:`https://my.domain.com/_admin/api`, make sure it is running and accepting connections.
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f384572b250>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='my.domain.com', port=443): Max retries exceeded with url: /_admin/api (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f384572b250>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/lib/api.py", line 166, in _api_call
response = requests.post(
^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='my.domain.com', port=443): Max retries exceeded with url: /_admin/api (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f384572b250>: Failed to establish a new connection: [Errno 111] Connection refused'))
ERROR: Failed to connect to backend at:`https://my.domain.com/_admin/api`, make sure it is running and accepting connections.
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f384572b250>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='my.domain.com', port=443): Max retries exceeded with url: /_admin/api (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f384572b250>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/lib/api.py", line 166, in _api_call
response = requests.post(
^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='my.domain.com', port=443): Max retries exceeded with url: /_admin/api (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f384572b250>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/commands/index.py", line 199, in cmd
main(ctx, instance, engine_location, es_version, repo_dir, mapping, source,
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/commands/index.py", line 267, in main
server_info = api_call(instance, 'get_server_info')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/lib/api.py", line 235, in api_call
response = _wrapped_api_call(
^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/lib/api.py", line 181, in _api_call
raise_sys_exit(1, error_message=msg)
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/lib/utils.py", line 393, in raise_sys_exit
raise exc # pragma: no cover
^^^^^^^^^
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f38457a9810>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='my.domain.com', port=443): Max retries exceeded with url: /_admin/api (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f38457a9810>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rhodecode/venv/lib/python3.11/site-packages/rhodecode_tools/lib/api.py", line 166, in _api_call
response = requests.post(
^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rhodecode/venv/lib/python3.11/site-packages/requests/adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='my.domain.com', port=443): Max retries exceeded with url: /_admin/api (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f38457a9810>: Failed to establish a new connection: [Errno 111] Connection refused'))
You’re right, I forgot to say that it’s a replacement value of what I put in the actual call.
We cannot confirm that, 5.6.0 is running on our instance and indexing fine.
./rcstack cli cmd rhodecode-index --no-tty --index-types=commits --config=/etc/rhodecode/conf/rhoderc_indexer.conf --api-host=https://code.rhodecode.com --api-key=REDACTED
attaching /root/rhodecode_docker/config/_shared under: /vol/confvolume
ENTRYPOINT: Running rhodecode_vcsserver with cmd 'rhodecode-index'
checking if config files needs bootstrapping
Using `<ElasticSearch2IndexBuilder(http://elasticsearch:9200)>` for building full text search
FETCH instance <RhodeCodeInstance(CLI-INSTANCE-https://code.rhodecode.com)> all repositories [done in: 0.806s]
889 Repositories will be indexed with following data types ['commits']
>>> STARTING BUILDING INDEX <<<
FETCH [u/att9030/lw61k-u1ry4] repository configuration
FETCH [u/att9030/lw61k-u1ry4] repository configuration [done in: 0.214s]
...
are you sure indexer can connect to your domain name properly ?
To my mind, nothing has changed on my server since my latest successful (manual) indexing, made 10 days ago.
How can I check the connection further?