Why am I getting the "data too large for key size" error, when using OpenSSL "rsautl" command to encrypt a large file?

Because of the nature of the RSA algorithm, a single encryption process can only encrypt input data that is smaller than the modulus value of the RSA key. In other words, the size (number of bytes) of the input data should be smaller than the size (number bytes) of the modulus, which is also the RSA key size.

If you try to use an RSA public key to encrypt a file larger than the key size, you will get the "data too large for key size" error. For example:

C:\Users\fyicenter>type clear.txt
The quick brown fox jumped over the lazy dog.
The quick brown fox jumped over the lazy dog.
The quick brown fox jumped over the lazy dog.

C:\Users\fyicenter>dir *.txt
   138 clear.txt

C:\Users\fyicenter>\local\openssl\openssl.exe
OpenSSL> rsautl -encrypt -pubin -inkey my_rsa_pub.key -in clear.txt 
   -out cipher.txt
   
RSA operation error
18472:error:0406D06E:rsa routines:RSA_padding_add_PKCS1_type_2:
   data too large for key size:.\crypto\rsa\rsa_pk1.c:153:
error in rsautl

OpenSSL> OpenSSL> pkey -pubin -in my_rsa_pub.key -text -noout
Public-Key: (1024 bit)
...

The test output tells us that:

• The RSA public key size is 1024-bit long.
• The input data, clear.txt, has 138 bytes = 1104 bits, which is larger than the RSA key size.
• "rsautl" will not encrypt any input data that is larger (longer) than the RSA key size.

Actually, OpenSSL could be improved to encrypt larger input files by dividing the input into multiple 128-byte blocks and perform encryption one block at a time.

⇒OpenSSL "rsautl" Command for RSA Keys

⇒⇒OpenSSL Tutorials