Edited by @aaronmdjones:
- Correct some data types and casts
- Minor style fixups (e.g. we put * on the variable name not the type)
- librb/src/openssl.c:
- Defer call of BIO_free(3ssl) to the end of the conditional block
to avoid having calls to it in multiple paths
- Check the return value of SSL_CTX_set0_tmp_dh_pkey(3ssl) because if
it fails then we must use EVP_PKEY_free(3ssl) to avoid a memory leak
This could fail if, for example, the user supplied DSA parameters
in the DH parameters file instead.
- ircd/newconf.c:
- Check whether OSSL_DECODER_CTX_new_for_pkey(3ssl) was able to parse
the given CHALLANGE public key as a valid RSA public key, and then
check whether OSSL_DECODER_from_bio(3ssl) actually loads it
successfully
- ircd/s_newconf.c:
- Use EVP_PKEY_free(3ssl) instead of OPENSSL_free(3ssl) on EVP_PKEY
pointers; this will avoid inadvertent memory leaks if the EVP_PKEY
structure contains any dynamically-allocated child members
- modules/m_challenge.c:
- Unconditionally use EVP(3ssl) to generate the SHA-1 digest of the
random challenge; this API has been around for a very long time and
is available in all supported versions of OpenSSL
- Add lots of error checking to all steps of the process
Tested against 1.1.1 and 3.0; both with missing and provided DH parameters
(which works as you'd expect; the server will not negotiate a DHE cipher
without them), and CHALLENGE, including missing keys or keys of the wrong
type (e.g. when you supply an EdDSA key instead of an RSA key).
This does break compatibility with OpenSSL 1.1.0 and below, which are now
all end-of-life and unsupported anyway.
Closes#357
8e9a741832