Security Using libssh2 with CryptoNG and ECDSA

Libssh2 lets us choose between multiple different crypto backends. On Windows, an attractive option is to use the WinCNG backend. CNG is part of the operating system, it’s maintained by Microsoft and serviced by Windows update, and reusing OS functionality helps us keep our binaries small.

However, the WinCNG backend used to have a significant limitation: it lacked support for ECDSA. Given the widespread use of ECDSA for host and user authentication, this limitation often made OpenSSL a more attractive choice, even on Windows systems.

Not being a huge fan of OpenSSL, I decided to do something about that, got my hands dirty, and added ECDSA support to libssh2’s CNG backend. The feature is already available in master and should be part of the upcoming 1.12 release.

Building libssh2 with WinCNG and ECDSA support

To build libssh2 with WinCNG and ECDSA support, we have to use two build options:

  • CRYPTO_BACKEND=WinCNG: This (existing) option enables the WinCNG backend.
  • ENABLE_ECDSA_WINCNG=ON: This new option enables ECDSA support in the WinCNG backend.

The implementation uses a feature of BCryptDeriveKey that’s only available in Windows 10 and later versions of Windows. Any attempt in using ECDSA on older Windows versions is bound to fail in a Unable to exchange encryption keys error. It’s for that reason that ECDSA support needs to be enabled explicitly using the ENABLE_ECDSA_WINCNG option.

Building with cmake

Freed of the OpenSSL dependency, it’s now rather straightforward to build libssh2 without vcpkg.

The following example makefile, taken from the IAP Desktop source code shows how we can build libssh2.dll with ECDSA support, statically-linked C runtime, and debug symbols:


!if ("$(PLATFORM)" == "x86")

!if ("$(CONFIGURATION)" == "Debug")

CMAKE = cmake
CMAKE_GENERATOR = "Visual Studio 17 2022"

	$(CMAKE) \
	$(CMAKE) --build $(CMAKE_BUILDDIR) \
Any opinions expressed on this blog are Johannes' own. Refer to the respective vendor’s product documentation for authoritative information.
« Back to home