Building libssh2 on Windows: lessons learnt

Considering how popular and widely used SSH is, it is surprising that there are not too many good open source SSH libraries.

To add SSH support to IAP Desktop, I was looking for an SSH library that works on Windows, that can be used from .NET, and uses a liberal (non-GPL based) open source license. Given these constraints, the list of suitable libraries quickly narrowed down to SSH.NET and libssh2.

SSH.NET is MIT-licensed and quite popular, but the library does not seem to be particularly well-maintained anymore: At the time (early 2021), the last commit on Github was 5 months old and the last release happened 3 years ago. For a security-sensitive function like SSH, I prefer to rely on a library for which I can be reasonably sure that security issues are going to be fixed in time – and SSH.NET did not seem like that project to me.

Libssh2 is written in plain C and runs on many platforms. The library is a bit less feature-rich than its LGPL-based cousin libssh, but it is well-maintained and relied on by countless open source tools such as curl. Compared to SSH.NET, libssh2 therefore seemed like the superior choice.

But my journey with libssh2 was not all smooth.

The nuget package is outdated

The libssh2 project does not provide any ‘official’ binaries, so I was somewhat surprised to see a libssh2 nuget package. This package comes from a third party however and more importantly, it is from 2013 and hopelessly outdated. The DLLs in the package also are not Authenticode-signed, so it’s not clear how trustworthy the build is. So the nuget package clearly was not helpful.

Building libssh2

Building libssh2 by yourself is not very difficult. If you have Visual C++ installed, then all it takes is:

git clone https://github.com/libssh2/libssh2.git
cd libssh2
git checkout libssh2-1.9.0

mkdir dll
cmake -DCRYPTO_BACKEND=WinCNG -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=./dll --build .
cmake --build . --target install

Although these steps produced a proper libssh2.dll for me, I encountered a few issues with that DLL:

• The DLL dynamically links to the C runtime library. That’s not necessarily a bad thing – but redistributing the VC CRT is a bit of a pain, so I much prefer statically linking to the CRT.
• The WinCNG backend in version 1.9 of libssh2 does not work on modern versions of Windows 10. This is a real bummer because its support for Microsoft Cryptography API Next Generation (CNG) is actually one of libssh2’s nicest features.

Switching libssh2 to use a static CRT proved much harder than it ought to be – but broken CNG support was actually the more severe problem.

As an alternative to CNG, libssh2 supports three other crypto backends – OpenSSL, libgcrypt, and mbedTLS. Libgcrypt is LGPL-licensed and hence was not an option for me. OpenSSL is very popular and Apache 2-licensed – but it’s also rather heavyweight and difficult to build. So I turned my attention to libmTLS first.

Building libmTLS itself was rather straightforward – but getting libssh2 to use libmTLS clearly was not: FindmbedTLS.cmake evidently was not written with VC++ in mind and passes invalid parameters to the linker. I was able to fix some of these issues, but eventually concluded that this avenue was not worth pursuing any further.

Using vcpkg

What turned out to be the easiest and most reliable way to build libssh2 on Windows is to use vcpkg and OpenSSL as the crypto backend. Vcpkg comes with a libssh2 port and makes it fairly easy to “fork” this port by creating an overlay. In the overlay’s portfile.cmake, I was then able to tweak the build flags according to my requirements.

The second piece to the puzzle was to create a custom triplet that causes vcpkg to dynamically link libssh2 (i.e., to create a libssh2.dll), but to statically link to the CRT and OpenSSL – I called this triplet libssh2-x86-windows-mixed:

set(VCPKG_TARGET_ARCHITECTURE x86)
if(${PORT} MATCHES "libssh2")
	set(VCPKG_CRT_LINKAGE dynamic)
	set(VCPKG_LIBRARY_LINKAGE dynamic)
else()
	set(VCPKG_CRT_LINKAGE static)
	set(VCPKG_LIBRARY_LINKAGE static)
endif()

With this configuration in place, I was able to build libssh2 and its dependency in one go:

vcpkg.exe install libssh2 `
	--triplet libssh2-x86-windows-mixed `
	--overlay-ports=path\to\overlays `
	--overlay-triplets=path\to\vcpkg-triplets

With the build working, I was able to turn my attention to using libssh2 – and I will talk a bit more about that experience in the next post.

• Share this article: