This is the more elaborate version of the Introduction to Cloud-Native Secrets Management: Part II series post. Click here to view the concise version.
In the first part of the Cloud-Native blog series we introduced the various secrets types, the technical challenges and the business implications of misusing secrets in a Cloud-Native environment. In this, second part of the blog series we’ll deep-dive into the usage of secrets.
Which Services Use Secrets
Secrets are used by a broad range of applications ranging from legacy client-server to modern cloud native applications, and everything in between.
How Secrets are Used by Applications
When a development team is building an application, they typically have to work with a variety of secrets such as database credentials and session keys. Applications access and use secrets in several instances, yet maintaining the security of those secrets remains problematic. For example:
- Hard-coded into the source code – this obvious malpractice comes with two major disadvantages. From a security point of view, the secret is written in plain-text in the source code, allowing everybody that has access to it (and if the code is checked From an operational point of view, if there is no separation between the code and the configuration, when the underlying secret needs to be changed, the source code needs to change as well.
- Entered via the command-line – requires either manually typing the secrets (either from the user’s memory or worse, from a note with the secret plainly written) or creating a script for the sake of automation, that now has the secret in clear-text.
- Obtained from a configuration file – in case the secret is stored within a configuration file which is external to the code, it’s imperative not to store the configuration file in the SCM, because otherwise the secret will still be available to everyone who has access to the SCM. And even if the configuration file is not stored in the SCM, a greater issue remains ofand how to prevent unauthorized access because once the SW is deployed, users on the system will have access to this configuration file.
- Received from environment variables – in case of delegating the secret to a script, one should keep in mind that the script will be visible to all those having access to the script.
- Via user-provided services – Cloud PaaS like Cloud Foundry and Pivotal allow secret retrieval from a user-provided service that is bound to the application in order to obtain access. This scheme allows a service to be bound to multiple applications which is a very effective way to store secrets that need to be shared by micro-services. Once created, these values are persistent until you delete the service or update them. These values are only visible to users who have the appropriate access credentials to the service. Yet, the secret within a user-provided service isn’t encrypted, allowing the attacker to easily read it, once obtained.
- By mounted volumes – different operating systems (i.e. Windows and Unix) offer the possibility to mount a logical drive, and later to assign it to specific application’s use. Such possibility exits also with the modern Cloud-Native containers such as Docker and Kubernetes, allowing with the use of different Oss to be shared between different containers within a pod, and to define an explicit lifetime for the volume. The main deficiency of this option, is that although it has the possibility to encrypt the volume at rest, it’s impossible to do so (within the limitations of the existing tools) for data in transit from the volume to the containers and elsewhere.
Different Secret Types in Use
Now, let’s review different types of secrets according to the dichotomy of the legacy IT and the emerging Cloud-Native arena. One might argue that the distinction between the two isn’t clear cut, as for instance API keys were also used before the virtualization and then the cloud revolution, but we sense that their proliferation grew significantly with the Cloud-Native App Explosion. This is by no means a full list, but describes some of the most popular secrets used by applications.
Username / password – Users often need to authenticate themselves in some way to the application (either native or cloud) in order to obtain access to information and services. This process may involve handing over some set of credentials which the application can directly verify (i.e. a username and password), or the application may rely on a third-party identity provider (like Active Directory Federation Services or AWS Identity Federation) to authenticate the user on their behalf.
DB connection strings – When the application connects to a database it needs to have a connections string for the provider. The connection string contains the information that the provider needs to know to be able to establish a connection to the database or the data file. The connection string contains credentials with a clear text password and is not using integrated security.
Cryptographic keys – applications use cryptographic keys for various purposes such as data encryption, authentication, identity verification, integrity assurance etc. A common example is DB encryption, where two potential approaches exist:
- The standard approach: if you are using TDE (Transparent Data Encryption) the database will generate an encryption key and encrypt this with another or master key. It also can be a pass-phrase or a longer key stored in a .pem file. This is usually stored in a restricted directory that only root/administrator and the database service account can access. On database initiation, the key is read and loaded into memory. This is then used to decrypt the encryption keys.
- that prevents tampering. For instance, an HSM can be used to store the master encryption key used for TDE. The key is secure from unauthorized access attempts as the HSM is a physical device and not an operating system file. All encryption and decryption operations that use the master encryption key are performed inside the HSM. This means that the master encryption key is never exposed in insecure memory. Typically, the HSM is located in a highly secure location that is separate from the database for greater security.
Cloud Service Access Credentials – each of the Cloud Service Providers (CSPs) provides slightly different set of authentication options, within their authentication modules (e.g. AWS IAM, Azure AD). The details of authentication may vary according to the various options provided by the CSP (like availability of MFA) and on how the CSP is accessed, but fall into two general types:
- A server-centric flow allows an application to directly hold the credentials of a service account to complete authentication, typically to be used by applications that work with its own data rather than user data.
- A user-centric flow allows an application to obtain credentials from an end user, enabling the user to sign in in order to complete the authentication, usually this flow will be used if the application needs to access user data.
API Keys – a code passed between programs calling an API to identify and authenticate the legitimate clients (the calling program, its developer, or its user) to the resource (i.e.
OAuth 2.0 – an authorization framework specified in RFC 6749 by the IETF, for granting clients limited access to a protected web service or API. This is done by an authorization server which issues the clients with access tokens. Access token is issued to the client once the client successfully authenticates itself. The authentication can be achieved by sending the user’s browser to the authorization server to be authenticated and allow the client request; password credentials grant; client credentials grant; JWT or SAML assertion. This access token defines the privileges of the client (what data the client can and cannot access). Now every time the client wants to access the end-user’s data, the access token secret is sent with the access token as a password. For instance, Google Cloud Storage uses OAuth 2.0 for API authentication and authorization.
The Caveats of Cloud-Native Secret Management
As many start to realize the damaging potential of a major security breach, different set of vault-like tools begin to emerge in the Cloud-Native eco-system. They encrypt the data while at rest within the vault as well as using TLS for encrypting the secret while being transferred from the vault to the application.
Although utilization of such tools is far better than simply coding the secret into the source code, severe security implications arise from the use, as we’ll exemplify below.
When an application needs to use a certain secret stored within the vault (e.g. connection strings to a database containing sensitive pricing and revenue information for a financial analysis), it first has to authenticate itself with the vault by providing proper credentials (typically just username and password).
Once the access request is authenticated by the vault it reads the secret data from its storage and decrypts it with a key stored on the same storage, or rarely in external HSM. For the experienced security professional, this already raises a red flag, as this opens a possibility to obtain the key to the entire vault and obtain an access for its whole content. Obfuscation methods that are often used to protect this key, can only slow down an attacker, but cannot prevent the breach.
After decrypting the secret, the vault provides it the application in a secure manner via TLS as plain text. Therefore, once out of the vault, the secret that we worked so hard to protect within the vault and in the transit process to the requesting application, suddenly available as clear text, and can be harvested, opening an easy and lucrative attack surface for the potential adversary.
This is a major security issue, with a broad effect on the Cloud-Native industry, as cryptographic keys, credentials and secrets are not strongly secured and exposed when in-use. It strongly contrasts with private keys and credentials in traditional environments that are typically secured in hardware like HSM/TPM, especially in medium to high trust .
This broadly practiced modus operandi opens a possibility for a skilled hacker working for a crime syndicate or rogue state to intercept all secrets being used by the Cloud-Native applications. The security implication of revealing such secrets ranges from the relatively low impact of an attacker revealing a username and password for a low importance system, rising with gaining access to admin credentials, API tokens, leaking personally identifiable information, harvesting private encryption keys, and lastly obtaining biometric identification information (i.e. fingerprint or iris scan), that is impossible to revoke, and the impact of revealing it is life-long.
In addition, direct passing of the secrets to the applications creates an auditing nightmare, as there no possibility to trace the usage of a certain secret by the applications in a centralized manner.
In this part of the Cloud-Native blog series, we have demonstrated the imperative of protecting the secrets while in-use. In the next posts of this series, we’ll dive deeper into the issue of containers’ authentication and shed light on potential approaches for resolving them.