If you created your ssh keys with default rsa settings, their format is not compatible in openssl. To encrypt and decrpt messages in openssl, you need your public key and private key in pem format, and start like this:

-----BEGIN PUBLIC KEY-----
or
-----BEGIN RSA PRIVATE KEY-----

Examples:

# convert your ssh public key to openssl pem format
% ssh-keygen -f ~/.ssh/id_rsa.pub -e -m pem | openssl rsa -RSAPublicKey_in -pubout > id_rsa.pub.pem

# convert your ssh private key to openssl pem format
# this will replace your private key in-place, MAKE A COPY first.
% ssh-keygen -p -N "" -m pem -f tmp.private

# encrypt and decrypt your messages
% echo "test" | openssl rsautl -encrypt -pubin -inkey id_rsa.pub.pem > tmp.txt
% openssl rsautl -decrypt -inkey tmp.private -in tmp.txt