Auto-generation of a Let's Encrypt certificate for HTTPS

justinmassiot · November 22, 2023, 8:48am

Hello,

On a Virtual Private Server, I have installed RhodeCode (beta version 5.2.1) via rcstack and docker, as advised per the documentation.

Now I’m trying to allow SSL for both the web interface and VCS operations. For this I would rely on a Let’s Encrypt certificate.
I followed the documentation about LE and the one about enabling SSL, but still the website is linked to the internal self-signed certificate.

What is the procedure to force the certificate request from let’s Encrypt?
Is there a way to check LE has effectively provided a valid certificate?
Or should I look at the logs? Which one?

Thank you.
Justin

rhodecode-support · November 22, 2023, 9:18am

Hi Justin,

In both cases when there are some issues with SSL certificates, especially with the Letsencrypt ones.
All logs and errors related to the ssl certificates would be printed to the traefik logs.

here’s how to view traefik logs with tail options and trim the results to just one last hour

./rcstack stack router logs --follow --since=1h

In case of letsencrypt the typical problems that happen are:

acme directory permissions (installer should handle that, but still it’s important to make sure the acme storage has the right permissions)
acme certificates credentials missing. In many cases using DNS acme resolver it’s important to make sure all credentials are correct in order to generate a proper certificate

justinmassiot · November 22, 2023, 9:31am

msg=“Unable to obtain ACME certificate for domains "my.domain.tld"”
error=“cannot get ACME client ovh: some credentials information are missing: OVH_ENDPOINT,OVH_APPLICATION_KEY,OVH_APPLICATION_SECRET,OVH_CONSUMER_KEY” ACME CA=“https://acme-v02.api.letsencrypt.org/directory”

Here it is!
Thank you, I will investigate further.

justinmassiot · November 22, 2023, 9:49pm

Certificate validated!

From the docs, it was unclear to me that the example with “route53” provider should be replaced according to the user infrastructure. My hosting and DNS provider is OVHcloud, so I have followed this nice tutorial: https://medium.com/nephely/configure-traefik-for-the-dns-01-challenge-with-ovh-as-dns-provider-c737670c0434

By the way, if I enable “Require SSL for vcs operations” in the Admin settings, I get an error 406 while trying to checkout through HTTPS. Is this expected with a Let’s Encrypt certificate?

justinmassiot · November 25, 2023, 10:01pm

If I enable “Require SSL for vcs operations” in the Admin settings, I get an error 406 while trying to checkout through HTTPS. Is this expected with a Let’s Encrypt certificate?

rhodecode-support · November 27, 2023, 7:41pm

It should be, we need to check that case…

justinmassiot · November 28, 2023, 1:23pm

I also notice that the documentation states that SSH port 9443 port is used, whereas .custom/traefik_custom/traefik.yaml seems to use 9022 by default. My firewall is configured to allow incoming traffic on port 9022 (and not 9443).

I suppose there is no relation between this and the certificate, but I prefer to mention it just in case…

rhodecode-support · November 28, 2023, 1:48pm

Thanks, this is a mistake in docs. We’ll get that fixed today.

rhodecode-support · November 29, 2023, 10:41am

the 406 error should be reported if RhodeCode cannot determine that SSL is used.

Can you show the actuall call and the output of the command when called ?

justinmassiot · December 1, 2023, 8:18pm

OK so now I reinstalled rcstack on 2 different machines.
Despite the notes I took related to my first install, I’m unable to obtain a Let’s Encrypt certificate again.

From what I can see in the logs, the certification authority is not contacted at all, even at restart. Something weird though: the TLS challenge is mentioned whereas it is configured to use the DNS challenge.
Here is what I have at router restart:

rc_cluster_router-traefik-1  | time="2023-12-01T20:12:19Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yaml"
rc_cluster_router-traefik-1  | time="2023-12-01T20:12:19Z" level=info msg="Traefik version 2.10.5 built on 2023-10-11T13:54:02Z"
rc_cluster_router-traefik-1  | time="2023-12-01T20:12:19Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
rc_cluster_router-traefik-1  | time="2023-12-01T20:12:19Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
rc_cluster_router-traefik-1  | time="2023-12-01T20:12:19Z" level=info msg="Starting provider *file.Provider"
rc_cluster_router-traefik-1  | time="2023-12-01T20:12:19Z" level=info msg="Starting provider *traefik.Provider"
rc_cluster_router-traefik-1  | time="2023-12-01T20:12:19Z" level=info msg="Starting provider *docker.Provider"
rc_cluster_router-traefik-1  | time="2023-12-01T20:12:19Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"

The firewall is not enabled on those installations.
I don’t know if I’m missing something or what can go wrong. May the renew limit be reached?

In order to go further, is there a way to force traefik to reach Let’s Encrypt asking for a fresh new certificate?

Thanks,
Justin

justinmassiot · December 4, 2023, 2:19pm

RC version:

RhodeCode Version:  RhodeCode Community Edition, version 5.0.0
Latest version:     4.27.1

Things are getting clearer for me now.
For a reason I can’t explain, my configuration in ~/docker-rhodecode/.custom/traefik_custom/traefik.yaml is not propagated to ~/docker-rhodecode/config/traefik/traefik.yaml. A soon as I edit the latter manually, Traefik tries to contact the certificate authority as expected.

Any idea of why is this happening?

rhodecode-support · December 4, 2023, 2:31pm

We’re sorry about that. It’s an obvious mistake we missed in the docs.

In the file: .custom/docker-compose-router.override.yaml

There’s a section to bind those mounts (which you should uncomment)

  traefik:

    volumes:
      # Bind to docker.sock that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock

#      # custom traefik config, enable SSL etc...
#      # put your traefik setup in this directory structure, and uncomment to override
#      # default traefik static/dynamic configs
      - $PWD/.custom/traefik_custom:/etc/traefik:ro
      - $PWD/.custom/traefik_custom/dynamic:/etc/traefik_dynamic:ro

We’re adding this to docs now

justinmassiot · December 4, 2023, 2:53pm

Thanks, I think it should work now… but I did too many certificate requests these days so I need to wait a little.

I’ll post back regarding the HTTP 406 error when things got stabilized on my side.

justinmassiot · December 4, 2023, 8:35pm

the 406 error should be reported if RhodeCode cannot determine that SSL is used.

Problem found: this was because the option Proxy subversion HTTP requests was disabled. Re-enabling it solves the issue.
Now I’m waiting for a new Let’s Encrypt certificate to confirm everything is working

justinmassiot · December 5, 2023, 2:25pm

It’s working now: my web browsers are happy!
But TortoiseSVN still warns for an Unknown certificate issuer, don’t know why…

Well, I think we can consider this issue to be closed.

rhodecode-support · December 5, 2023, 2:30pm

looks like this: TortoiseSVN failing validation after DST expiry - Help - Let's Encrypt Community Support

justinmassiot · December 5, 2023, 2:44pm

Nice, thank you! Works on Windows 10 with TortoiseSVN 1.14.5 (build 29465).

For future readers:

Download the Let’s Encrypt R3 certificate here: https://letsencrypt.org/certs/lets-encrypt-r3.der (the currently Active certificate as of 2023-12-05, check the architecture here)
Right click > Install Certificate

Source: https://groups.google.com/g/tortoisesvn/c/cVUXqh8VMh0/m/I_EDM65yAQAJ

justinmassiot · January 19, 2024, 1:53pm

Hello,

Just to be sure: did you know that restarting traefik forces the Let’s Encrypt certificate to renew? If you do this many times, then Let’s Encrypt claims that’s too many requests in a short amount of time:

Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: <DOMAIN>, retry after <DATA_IN_FUTURE>: see https://letsencrypt.org/docs/duplicate-certificate-limit/" ACME CA="https://acme-v02.api.letsencrypt.org/directory" providerName=letsEncryptCertResolver.acme

Would be create if it could transmit a renewal request based on the validity date of the existing certificate, not at each restart.

rhodecode-support · January 19, 2024, 3:40pm

The default behaviour is to re-use certificate instead of creating a new one. Do you have this mount exposed in your setup to ensure traefik acme certificates are persisted instead of re-created each time when container starts ?

inside docker-compose-router.override.yaml

volumes:
    # Mount ACME shared json config for lets-encrypt
    - $PWD/.custom/traefik_custom/acme:/acme

rhodecode-support · January 19, 2024, 3:47pm

Then just make sure this is used in certificate storage

certificatesResolvers:
  letsEncryptCertResolver:
    acme:

      # The storage option sets the location where your ACME certificates are saved to.
      # Make sure this is permanently stored to prevent certificateRevocation
      storage: /acme/acme.json