How to encrypt a large file with an RSA public key using OpenSSL "rsautl" command?

If you are trying to use an RSA public key to encrypt a file larger than the key size directly, you will get the "data too large for key size" error.

One option to resolve the problem is to use the RSA-AES hybrid encryption process as described blow:

1. Generate a one-time random AES (Advanced Encryption Standard) symmetric encryption password shorter than the RSA public key. This can be done using the OpenSSL "rand n" command.

2. Encrypt the large input data with the AES algorithm using the short password. This can be done using the OpenSSL "enc -e -aes*" command.

3. Encrypt the short password with the RSA public key. This can be done using the OpenSSL "rsautl -encrypt" command.

4. Send the AES encrypted data and the RSA encrypted password to the owner of the public key.

For example:

C:\Users\fyicenter>type clear.txt
The quick brown fox jumped over the lazy dog.
The quick brown fox jumped over the lazy dog.
The quick brown fox jumped over the lazy dog.

C:\Users\fyicenter>dir clear.txt
   138 clear.txt

C:\Users\fyicenter>\local\openssl\openssl.exe

OpenSSL> pkey -pubin -in my_rsa_pub.key -text -noout
Public-Key: (1024 bit)
...

OpenSSL> rand 32 -out aes256_pass.txt

OpenSSL> enc -e -aes256 -pass file:./aes256_pass.txt -in clear.txt 
   -out cipher.txt

OpenSSL> rsautl -encrypt -pubin -inkey my_rsa_pub.key -in aes256_pass.txt 
   -out aes256_pass_cipher.txt

Commands used in this test:

• "pkey -pubin -in my_rsa_pub.key -text -noout" - OpenSSL command confirming that the RSA public key, my_rsa_pub.key, is 1024-bit long.
• "dir clear.txt" - Windows command confirming that the input data, clear.txt, is 138-byte (1104-bit) long, longer than the RSA public key.
• "rand 32 -out aes256_pass.txt" - OpenSSL command generating a random password, aes256_pass.txt, of 32-byte (256-bit) long, shorter than the RSA public key.
• "enc -e -aes256 -pass file:./aes256_pass.txt -in clear.txt -out cipher.txt" - OpenSSL command encrypting the large input data, clear.txt, with AES 256-bit algorithm using the random password. The AES encrypted data is stored in the output file, cipher.txt.
• "rsautl -encrypt -pubin -inkey my_rsa_pub.key -in aes256_pass.txt -out aes256_pass_cipher.txt" - OpenSSL command encrypting the random password, aes256_pass.txt, with ARS algorithm using the RSA public key. The RSA encrypted password is stored in the output file, aes256_pass_cipher.txt.

You can publicly the AES encrypted data and the RSA encrypted password to the owner of the RSA public key. He/she can decrypt the AES password with his/her RSA private key, then decrypt the AES encrypted data with AES password.

⇒ OpenSSL "rsautl" - Decrypt Large File with RSA Key

⇐ OpenSSL rsautl "data too large for key size" Error

⇑ OpenSSL "rsautl" Command for RSA Keys

⇑⇑ OpenSSL Tutorials