To set the SSH key under windows

• use the OpenSSH format of the key
• copy the key to ~/.ssh/gitlab.ppk
• Under ~/.ssh/config refer to the key

~/.ssh/config

# Private GitLab instance
Host YOURGITLABSUBDOMAIN.YOURDOMAIN.com
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/gitlab.ppk

make the TortoiseGit use the same client, as your CLI, which understands the ~/.ssh/config file

Figure out where the client is

# make which command work
Set-Alias which where.exe

which ssh.exe
C:\Windows\System32\OpenSSH\ssh.exe

Now set the ssh.exe as in https://stackoverflow.com/questions/304319/is-there-an-equivalent-of-which-on-the-windows-command-line