How to use OAEP padding with OpenSSL "rsautl" command? I was told to encrypt a password using an RSA public key with OAEP padding.

OpenSSL "rsautl" uses PKCS#1 v1.5 padding as the default padding schema. So if you want to use OAEP padding, you have to using the "-oaep" option as shown below:

C:\Users\fyicenter>type test.txt
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ

C:\Users\fyicenter>\local\openssl\openssl.exe

OpenSSL> rsautl -encrypt -pubin -inkey my_rsa_pub.key -in test.txt 
   -out cipher.txt -oaep

OpenSSL> rsautl -decrypt -inkey my_rsa.key -in cipher.txt 
   -out decipher.txt -oaep

OpenSSL> exit

C:\Users\fyicenter>type decipher.txt
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ

Note that the "-oaep" padding option is needed to decrypt the cipher text, if the clear text was padded using the OAEP padding schema. This is to undo what the OAEP padding did to the clear text.

If you want to see what the OAEP padding did to the clear text, you can use the "rsautl -decrypt -raw -hexdump" command to keep whatever OAEP padding did to the clear text:

C:\Users\fyicenter>\local\openssl\openssl.exe
   
OpenSSL> rsautl -decrypt -inkey my_rsa.key -in cipher.txt -raw -hexdump
0000 - 00 4c 6b e6 bb a0 9e 89-22 e0 a2 53 f9 a1 bb 7d   .Lk....."..S...}
0010 - bb 95 4a 77 e3 5b 95 c9-a1 74 7e 4d ed 8c 1a 56   ..Jw.[...t~M...V
0020 - 9d 61 10 0f 23 02 5a 97-5a 60 af 4f 0f 13 5c 7b   .a..#.Z.Z`.O..\{
0030 - 22 77 49 45 b0 bd af 0a-d1 71 03 be 44 ea 4e c3   "wIE.....q..D.N.
0040 - 6d 75 84 d0 24 e3 ff 6a-73 a1 fc 84 52 0a b1 34   mu..$..js...R..4
0050 - 4b 94 44 bf 5c 07 1b 79-e8 82 8d 74 59 1e 4a 0d   K.D.\..y...tY.J.
0060 - af 3a 65 a2 2e 15 ea e1-2b b1 d4 55 31 7a bb 7f   .:e.....+..U1z..
0070 - fe a6 8b 2b 65 17 ff 19-ac e4 76 67 16 aa 4c 43   ...+e.....vg..LC

The output confirms that OAEP padding is not a simple padding schema of just inserting bytes to the clear text.

⇒ OpenSSL Signing Documents with RSA Keys

⇐ OpenSSL "rsautl" Using OAEP Padding

⇑ OpenSSL "rsautl" Command for RSA Keys

⇑⇑ OpenSSL Tutorials