If you’re setting up a certificate authority for your organization, so that you can build and use PKI certificates in house, you need to make sure that your Macs are set to recognize that CA (and the certificates it uses) are trusted. One way to do that is to install new trusted root certificates as needed into /Library/Keychains/System.keychain, so that certificates issued by your CA are recognized and trusted by your Mac.
You can use the security command to install the trusted root into your Mac’s System.keychain (the trusted root .cer file is in this case stored in /private/tmp/certs.)
sudo security add-trusted-cert -d -r trustRoot -k "/Library/Keychains/System.keychain" "/private/tmp/certs/certname.cer"
add-trusted-cert = Add certificate (in DER or PEM format) from certFile to per-user or local Admin Trust Settings.
-d = Add this certificate to admin certificate store; default is to store it in the user’s keychain.
-r = Specifies the result you want, in this case you want to use trustRoot (see the security man page for the other options.)
-k = Specifies the keychain to use, in this case the specified keychain is /Library/Keychains/System.keychain
You can also build this into an installer package, where you’re installing the certificate you want to use to a specified directory, then setting the following in a postflight script to automatically install the trusted certificate into System.keychain, then remove the certificate from the directory
#!/bin/bash security add-trusted-cert -d -r trustRoot -k "/Library/Keychains/System.keychain" "/private/tmp/certs/certname.cer" srm "/private/tmp/certs/certname.cer"