In our last post on choosing a database protection technology, we discussed three different protection methods: transparent data encryption (TDE), in-cell encryption (ICE), and format-preserving encryption (FPE). Of these three, we mentioned FPE is best for organizations transferring data among various storage locations: a data item protected (encrypted) with FPE can be transferred, for example, to Hadoop and used directly in its protected form.
FPE is effective on data in all three states (motion, storage, and use), avoids schema changes across structured data repositories, and maintains referential integrity. This makes it possible to execute queries directly on protected data and re-identify (decrypt) only after return of the final result set. This substantially reduces the number of re-identification operations otherwise required by TDE and ICE.
We should note FPE has another feature eliminating the need for decryption in ninety percent of all use cases: partial protection. One can choose an FPE format that leaves some characters in the clear even after applying protection. Examples include leaving the last four digits of a national identity or passport number in the clear, as well as the first six and last four digits of a credit card or payment account number (PAN).
So how does partial protection eliminate the need for decryption in so many use cases? Well, let’s imagine a customer inquiring about an account transaction. Often an agent will ask a question similar to, “What are the last four digits of your social security number?” for validation purposes.
In this case, partial protection offers two obvious benefits: first, the application need not re-identify the source as the last four digits are already in clear text. Second, and perhaps more important, the agent never sees the real social security number and remains compliant with the organization’s security policy.
The core idea behind partial protection is to keep a portion of the data in the clear for verification or validation purposes and thus eliminate the need for re-identification with such applications. This way, FPE offers substantially better performance than either TDE or ICE in these and similar applications.
Let’s consider another use case. Suppose a credit card brand chooses to offer a discount or promotion to a particular group of stores, say small businesses. If the store protects the first six digits of the PAN, it can then retrieve the transactions eligible for this promotion directly and share these records with the fulfillment bureau without fear of PAN abuse.
Want another example? Consider a street address; protect the letters yet not the digits. We can now validate with the house or building number along with other information. For more real-world examples, note next time when calling customer service if the validation process can be implemented with partial protection or not!
One might wonder if partial protection has less security than full protection. In an abstract sense, it does: if we only protect the middle six digits of a PAN and leave the first six and last four in the clear, there is a one in a million chance of guessing the entire PAN, is there not? Sure, there is! However how good of a chance is that to an attacker? Guess wrong in a few cases and the payment gateway will block any further authorization attempts.
Partial protection works well for validation use cases. But what about searching records, for example name matching? For that use case, we can introduce alternative column encodings that do not contain sensitive data. Which is the subject of our next post!
Meanwhile what are your thoughts on partial protection? Have you been part of a customer service application team? How did your team handle authentication and verification? We would love to hear your thoughts on these matters, so please leave a note in the comments section below.
Data security and encryption