A surprisingly simple bug in a well-fuzzed cryptographic library from Mozilla leading to an easy stack overflow in RSA-PSS code (vulnerability exists elsewhere also).

Problem Code: https://searchfox.org/mozilla-central/rev/f8576fec48d866c5f988baaf1fa8d2f8cce2a82f/security/nss/lib/cryptohi/secvfy.c#489

The VFYContext object contains a fixed size buffer to hold a the bytes of a signature to be validated. These bytes come from the certificate from validated and the size of the signature in the certificate are not validated before attempting to copy them into the buffer, leading to, in the case of RSA-PSS at least (the vulnerability exists in multiple algorithms) a stack-based overflow with attacker controlled data. Though stack-based is not too important in this case because within the same VFYContext object is a pointer to a SECHashObject which contains functions pointers. Overwritting that can immediately provide a control flow hijack.

Must of this post is actually dedicated to a discussion about why this vulnerability was missed for so long with the original code being checked-in to the repository in 2003, being exposed and exploitable since 2012. Its neither a subtle to detect bug, not a hard to hit bug. Additionally, its not as though Mozilla has an immature security process and simply is not looking but still the bug was missed.

Ormandy puts forward a few issues that might have led to this that are worth considering for your own security process or fuzzing campaigns:

• Missing end-to-end testing, in the development world there is unit testing, and there is integration testing. Both are useful, and neither replaces the other. As NSS was a modular library it was easy to fuzz various components in isolation. The fuzzer would test the certificate decoder, but those decoded certificates would not be passed into other areas of the application so integration issues couldn’t easily be detected.
• Arbitrary size limits, there is a 10,000 byte limit on fuzzed input which does not reflect the limits of the code itself.
• Misleading metrics, the combined code-coverage information was exposed rather than the coverage of any individual fuzzer leading to code that was well covered by fuzzers that couldn’t possibly generate the relevant input for this bug.