Data-centric Security: The Importance and Advantages of Format Preserving Data Protection

Micro Focus Contributor
Micro Focus Contributor
1 0 3,794

The emergence of strict new data privacy regulations, such as the GDPR and CCPA, is driving the need for CISOs to more effectively address data protection and governance in complex and geographically diverse hybrid IT ecosystems. The terms pseudonymization and anonymization are now common in the context of these privacy regulations when it comes to data privacy and protection. While pseudonymization of data still allows for some form of re-identification (even indirect and remote), anonymization of data means it cannot be re-identified. CISOs look to the vendor community for data security solutions to address these privacy requirements but may struggle with the confusing array of security models and services.

Data-centric Security The Importance and Advantages of Format Preserving Data Protection.pngEnterprises must choose a solution that offers a variety of data protection formats that not only allow pseudonymization and anonymization of sensitive data, but also enable business processes, applications, and analytics workloads to operate on the data in its protected state. The ability to protect high value data, with data utility and usability, is critical to achieving cyber resilience, the ability to evolve and adapt to rapidly changing threats and regulatory mandates. Format-Preserving Encryption (FPE) is a powerful data protection technology, and is currently becoming the de facto standard across the industry. FPE warrants a deeper examination, and the following section expands on FPE and its importance.

Format-Preserving Encryption

Format-Preserving Encryption refers to encrypting data in such a way that the output (the ciphertext) is in the same format as the input (the plaintext). “Format-preserving” implies that encrypting a 16-digit credit card number produces a ciphertext which is another 16-digit number; encrypting an English word produces a ciphertext comprising the same number of English characters; and so forth. These properties have several benefits and simplify data protection, especially for legacy applications, where it avoids major redesign and refactoring of applications and business processes:

  1. Minimal or zero database schema impact – FPE facilitates retrofitting encryption technology to existing devices or software where conventional encryption modes would not be feasible. In particular, database applications may not support changes to data length or format.
  2. Minimal or zero data storage impact – Since length preservation is an inherent property of FPE, enterprises do not have to worry about additional storage usage, unlike conventional (non‑format-preserving) encryption methods, which typically expand data.

Note: Some exceptions do apply where the length of the output with some variants of FPE can be slightly longer than that of the input data.

  1. Analytics on protected data – Format-preserved protected data elements such as credit card numbers, SSNs, etc., can still be used as index keys to facilitate statistical research, even across databases. With FPE, the same inputs to the algorithm will create the same ciphertext. This deterministic encryption preserves the referential integrity of the data and thereby the ability to glean valuable information from the protected dataset. Other crucial benefits of Secure Analytics enabled by the use of FPE is expanding the access to data across a broader set of analysts, and potential monetization of data sets, without compromising on security and privacy.
  2. Cross-application dataflow preservation – FPE lets protected data flow across applications without requiring changes to those applications to accept the protected data, an infeasible approach with conventional encryption methods, since applications require data of specific lengths and formats.
  3. Using protected data without requiring decryption – FPE can allow protection of only specified key portions of data elements, enabling use of the data in its protected state. For example, the “first six” digits of credit card numbers are used for charge routing, and the “last four” of SSNs is used for customer verification. If these are left in the clear, many applications in the data flow will not need access to the entire data element, and can perform required business functions without requiring any change to the applications, and not requiring to perform any decryption. Such partial encryption can facilitate functions such as sort and certain search use cases, such as “Starts with”, “Ends with”, etc., without requiring any decryption of the encrypted data.
  4. Test data management – FPE can also be used, especially in the form of its irreversible variant, to obfuscate/scrub production data to populate test databases, enabling realistic test conditions based on production volume, variability, etc.

NIST Special Publication 800-38G, Recommendation for Block Cipher Modes of Operation: Methods for Format-Preserving Encryption, specifies two AES modes, FF1 and FF3, for format-preserving encryption. However, NIST has concluded that FF3 is no longer suitable as a general-purpose FPE method based on findings of cryptanalytic attacks on the FF3 algorithm. Few vendors have implemented FPE within their data security solutions other than Voltage Security (now part of Micro Focus), who hold the patent for FF1-mode AES. Customers need to be cautious about other vendor implementations of FPE, ensuring that they are not using the vulnerable FF3 mode or a non-standard form of FPE not validated by NIST, or that, if using FF1, their vendor has obtained a license from Voltage for FF1 mode.

Voltage SecureData is the only enterprise-grade solution that fits the criteria for implementing crypto services for hybrid and multi-cloud (or rather cloud-agnostic) implementations.

Why Voltage SecureData?

Voltage SecureData is an industry leader in the data security space, where hundreds of enterprises rely on it to secure sensitive data at the application layer and establish the trust of their customers. Voltage SecureData has been implemented across a wide variety of customer on-premises infrastructure and cloud hosting locations, providing ubiquitous, stateless, scalable, highly performant and highly resilient data security. Voltage SecureData supports hybrid (mixed on-premises and cloud) and multi-cloud implementations, as well as multiple enterprise platforms such as midrange, IBM Z, Hadoop, and HPE NonStop. When it comes to implementing cloud data security, Voltage SecureData is the optimal choice, since it best addresses the issues and challenges of CSP native crypto services.

Voltage SecureData offers a stateless, identity-based key management solution, even supporting few industry standard and popular FIPS 140‑2‑compliant, general-purpose HSM products for key derivation. It does not store keys, and hence is less susceptible to attack. It offers client libraries for local crypto operations, as well as a REST interface for remote operations. It supports a range of crypto methods:

  • Format-Preserving Encryption
  • Embedded Format-Preserving Encryption (eFPE) for seamless key rotation (see below)
  • Secure Stateless Tokenization (SST) for PCI-DSS use cases
  • Obviously protected output preventing false positives during scans, DLP and audits.
  • Format-Preserving Hash (FPH) for data anonymization and enabling searches for data encrypted using probabilistic encryption modes, or searches on old encrypted data post key rotation
  • Standard AES encryption with 256-bit keys, for both structured and unstructured data.

Voltage Crypto-methods.PNG

Figure 1: Voltage Crypto-methods: Input-Output examples

As illustrated, it also supports partial encryption to expose certain portions of data elements to enable business functions to operate on persistently encrypted data. It provides the right level of abstraction for developers, making it easy to add encryption without requiring extensive crypto knowledge.

Voltage SecureData also introduces a variant of FPE called Embedded Format-Preserving Encryption (eFPE), which embeds key metadata within the ciphertext as part of the crypto operation. This provides significant benefits:

  • Zero Key Rotation Impact – When a key is rotated, the data encrypted with the previous key need not be re-encrypted, as the solution detects the embedded key metadata from the ciphertext and determines which key to use for decryption.
  • Atomic Key Identification – If a data element is re-encrypted after a key rotation, the new key metadata is automatically updated as part of the encryption operation.

Note: If the metadata is stored elsewhere, a separate operation is required to update that metadata, which introduces the risk that this second operation might fail, or be omitted.

  • Obviously Protected Output – The embedded key metadata introduces additional characters into the ciphertext (beyond the input alphabet), and hence an encrypted credit card or Social Security number can be differentiated from actual values, preventing false positives during scans, DLP and audits.

SecureData and Unicode - Normalization is normal in Unicode processing, and is a critical issue for format-preserving data protection, because if a protected value contains any normalizable characters, any normalization after encryption will destroy the ciphertext. Other solutions offering format-preserving data protection for Unicode generally try to avoid the issue by simply saying “Never normalize ciphertext”. Since one of the benefits of format preservation is the ability to pass data through other applications in its protected state, and normalization may occur at many points in the life of a data element, this is not a good solution. Voltage SecureData is also the only solution in the industry to solve the Unicode normalization problem by providing Safe Unicode FPE. This support adds a new built-in format named PREDEFINED::UNICODE_BASE32K, or “Base32K” for short. Safe Unicode FPE provides a robust and elegant solution to the normalization problem, allowing Format-Preserving Encryption of Unicode data without risk.

Voltage Safe-Unicode FPE.png

 Figure 2: Voltage Safe-Unicode FPE

The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.