Can I using MD5 digest algorithm when generating a self-signed certificate using the OpenSSL "req -x509" command?

Yes, you can use MD5 digest algorithm when generating a self-signed certificate using the OpenSSL "req -x509 -md5" command Without the "-md5" option, the default SHA256 digest algorithm will be used in the signing process. See the example below:

C:\Users\fyicenter>\local\openssl\openssl.exe

OpenSSL> req -x509 -in rsa_test.csr -key rsa_test.key -out rsa_test.crt -md5
Enter pass phrase for rsa_test.key:fyicenter

OpenSSL> x509 -in rsa_test.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            96:28:6c:2e:b9:4f:84:59
    Signature Algorithm: md5WithRSAEncryption
        Issuer: C=us, ST=NY, L=New York, O=Donald Inc., OU=IT, 
                CN=www.donald.inc/emailAddress=john@donald.inc
        Validity
            Not Before: Aug 23 02:11:42 2016 GMT
            Not After : Sep 22 02:11:42 2016 GMT
        Subject: C=us, ST=NY, L=New York, O=Donald Inc., OU=IT, 
                CN=www.donald.inc/emailAddress=john@donald.inc
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (512 bit)
                Modulus:
                    00:f6:d5:d3:79:87:8d:9d:83:49:6f:fb:08:67:08:
                    fb:0f:ab:b4:7f:51:55:7b:49:fa:e3:47:8e:6e:22:
                    d7:ba:ad:dc:10:56:e9:b3:42:f7:25:20:9d:a5:e3:
                    5f:5e:7c:95:cb:5a:22:f3:8f:3d:e1:b2:0a:fa:15:
                    c5:16:64:17:03
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                99:FB:5B:B6:BE:B4:E2:2B:4D:46:75:3F:0E:5E:52:36:F1:0E:A4:DB
            X509v3 Authority Key Identifier:
                keyid:99:FB:5B:B6:BE:B4:E2:2B:4D:46:75:3F:0E:5E:52:36:F1:0E:A4:DB

            X509v3 Basic Constraints:
                CA:TRUE
    Signature Algorithm: md5WithRSAEncryption
         f6:97:36:2e:01:9a:3b:11:3e:49:c7:c9:5e:bd:91:28:6e:c3:
         9d:f4:7a:e9:57:07:f2:cf:44:04:92:6a:c3:6e:31:05:c3:30:
         5d:36:cb:32:ff:20:ac:06:c5:5c:d9:2b:f1:06:71:90:23:1d:
         2d:5d:ce:9c:7d:0a:6c:39:e2:09

The output from the "x509 -text -noout" command confirms that MD5 algorithm was used as digest algorithm in the digital signature.

⇒ OpenSSL "req -x509" - Sign CSR with Different Key

⇐ OpenSSL "req -x509 -set_serial" - Certificate Serial Number

⇑ OpenSSL "req" Command

⇑⇑ OpenSSL Tutorials