Yubikey ssh and signing failed for RSA "PIV AUTH pubkey" from agent: agent refused operation

You may face this issue if you use ssh and Yubikey like covered in this guide. 

Previously I had to reboot machine to address this issue but I found nice trick to get it work.

Originally error looks like:

ssh server

sign_and_send_pubkey: signing failed for RSA "PIV AUTH pubkey" from agent: agent refused operation

root@xxx: Permission denied (publickey).

It may happen when you forgot to tap confirmation on Yubikey and it was just slow.

First attempt to fix it was to kill all ssh agent processes which may be run on system:

ps aux|grep ssh

odintsov   16493  0.0  0.0   7972  5820 ?        S    12:36   0:00 /usr/bin/ssh-agent -D -a /run/user/1000/keyring/.ssh

odintsov   16494  0.0  0.0 164340 11584 ?        SLl  12:36   0:00 /usr/lib/openssh/ssh-pkcs11-helper

odintsov   27798  0.0  0.0   7972  3848 ?        Ss   13:20   0:00 ssh-agent -s

odintsov   27801  0.0  0.0 164340 11572 ?        SLl  13:20   0:00 /usr/lib/openssh/ssh-pkcs11-helper

The best way to kill them is:

pkill ssh

After that we need to start ssh agent again:

eval `ssh-agent -s`

After that load Yubikey key to agent:

ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

For last command I use fancy alias in ~/.bashrc:

alias ssh_add='ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so'

It's not clear why Yubikey fails that way. I think it has something to do with ssh-pkcs11-helper being stuck in operation to Yubikey hardware.