Deploying UrbanCode Deploy Blueprint Designer Cloud Applications Part 7 – Using an Amazon Relational Database Service (RDS)

In Part 6 of this blog series we demonstrated how to provision a two-instance environment where we provisioned our own MySQL database. At the risk of losing some cloud portability, there is BPD support for the Amazon Relational Database Service (RDS). In this blog we are going to demonstrate how to change our database from a deployed version of MySQL to a RDS service provided by Amazon. Two use cases are provided – first, instantiating the RDS as part of the provisioning and second, how to connect to a pre-existing RDS.

Where the Support Comes From

BPD’s support for Amazon uses Boto, which is the Amazon SDK for Python. In UCD BPD Version 6.2.4.1, the supported version of Boto is 2.47 and Boto 3 is now available. So this version of UCD BPD is a little behind and advances with Aurora are not yet supported. The databases in RDS supported in version Boto 2.47 are MySQL, Oracle, SQL Server, and Postgres and there are quite a number of server classes on which they can run. (In UCD 6.2.6.0 and later, support for Aurora RDS, application load balancers, and Elastic File System was added. Also, the AWS API was upgraded to Boto 3. In later blogs of this series, we will be upgrading the server. The latest version at the time of this writing is 6.2.6.1 and 6.2.7.0 is nearing release.)

You can find out more about Boto RDS support here:

http://boto.cloudhackers.com/en/latest/ref/rds.html

You can find out more about UCD 6.2.6.1 here:

https://developer.ibm.com/urbancode/products/urbancode-deploy/whats-new/whats-new-urbancode-deploy-6-2-6-0/

Provisioning an Amazon RDS

In order to provision to RDS we are going to need to modify our jke.db and jke.war components. For jke.db we have a design decision to make. We can either modify our existing component or we can create a new jke.db component. From a design and best practices point of view, we really don’t want multiple jke.db components in UCD so I am opting for modifying our existing component. The modifications will include the addition of component environment properties, a new component version containing a tokenized create_database.sql file, and the addition of a new component process.

We are going to add the following component environment properties to the jke.db component:

Figure 1 – jke.db Component Environment Properties

There are a couple of things to point out. First, even though properties are under the jke.db component, I have purposefully left out any mention of JKE in the parameter names. This is because they can be later used as the basis for creating a component template. This might also support the argument for making this a new component governed by a component template but for now we will stick to out original design decision. Secondly, I have intentionally given default values that I know will fail if not provided during execution. This forces the values to be set in the blueprint.

Now let’s take a look at the tokenized create-database.sql. As you can see, the file is boilerplate code for creating a database and I have changed it to be completely normalized through parameters or “tokens”. We will use this to create version 2.1 of the jke.db component and zip it up with its component version name consistent with that of version 2.0. Typically, you would not put the version number in the name of the artifact but we will do so to follow the naming pattern of the original jke.db component.

Figure 2 – Tokenized create_database.sql File

We need to create a new component process that adds a step to replace the tokens with the properties. The new component process, called deploy_rds, simply unzips jke.db_2.1_artifacts.zip, replaces the tokens inside using the Replace Tokens step, and then executes MySQL to create the database on the remote RDS instance.

Figure 3 – jke.db deploy_rds Component Process

The Replace Tokens step is configured to act on properties between the delimiters “#”.

Figure 4 – jke.db deploy_rds Replace Tokens Step

This process assumes a MySQL client is available. Assumptions can sometimes be wrong so we could extend this process to check that a MySQL client is installed and available and gracefully fail if it is not. (We are satisfying this dependency by installing the MYSQL server since we already have that component).

The create_database step executes a MySQL command that is configured using the component environment properties. The properties are replaced with values set in the blueprint to override the default component environment properties in jke.db. By using db_endpoint_address and db_port, the command will use these values to connect to the remote RDS instance and create the database remotely. Security groups must be added to the RDS instance in the blueprint so it can be reached on the port we set for the database server.

Figure 5 – Execute MySQL Database Creation

We now have two versions of jke.db and we have to modify the Unzip steps in our processes to normalize the component name. Originally, the configuration for that step named the zip file explicitly. I have changed the zip filename to jke.db_${p:version.name}_artifacts.zip. ${p:version.name} is a built in property that holds the component version number. Now I can deploy any version without changing the processes. I have done the same for the Unzip step in the jke.war processes.

For the jke.war component we are going to add JKE_DB_HOST and JKE_DB_PORT as component environment properties and set default values. If you followed Part 6, these properties already exist.

Figure 6 – New jke.war Component Environment Properties

Make a duplicate copy of the jke.war deploy process and call it something like deploy_rds. Add the following to the Update Property File step in the Add/Update properties box.

Figure 7 – JKE_DB_HOST and JKE_DB_PORT Component Environment Properties

Figure 8 – Updating Database and Host Properties in JKEDB.properties

Creating a Blueprint with RDS

There are instructions for how to integrate RDS into a blueprint in the UCD Knowledge Center but I will do so here to be explicitly clear. On the design palette to the right you will find the RDS in the Services drawer.

From there, drag it onto the canvas. Select the RDS instance. For the properties on the left, set the Key and Name to rds-dev-instance, the Db Instance Class to db.t2.micro, the Engine to mysql, and the Engine Version to 5.6.35. For the Vpc Security Group, you need to modify the code with the proper subnet. The properties will look like:

Figure 10 – rds-dev-instance Properties

At this point our blueprint looks like the following:

Figure 11- Blueprint With Detached RDS

The natural thing to do at this point is to wire the rds-dev-instance database to the jke.db and jke.war components. However when you do so, a dialog pops up that asks for values but doesn’t allow for any selection.

Figure 12 – Wire rds-dev-instance to jke.db

When you release the mouse, a dialog will appear.

Figure 13 – Attach to RDS Instance – jke.db

Click Cancel and remove the connection that was created. In UCD 6.2.4.1, wiring a RDS visually does not work. Although there is a security group defined for the RDS, it is not shown in the GUI. These are visual capabilities/errors that are perhaps fixed in later versions of UCD. For now, we need to connect the RDS via the code. This will be discussed in a following section called Configuring the Database Connection URL.

There are several requirements in order for this blueprint to work.

  1. The RDS needs a database administrator user and password to use when the RDS is instantiated. We will investigate a way to supply this as a parameter.
  2. A MySQL client must be installed for db to create the JKE database JKEDB. We are satisfying this requirement by still installing the MySQL Server component since we already have it. This is a little wasteful and redundant but it gets the job done for now.
  3. The db component must know the database administrator username and password as well as the database endpoint address and port in order to create the JKE database JKEDB. The endpoint address and endpoint port essentially build the database connection URL.
  4. The war component must also know the RDS endpoint address and port upon which the database is communicating.
  5. The RDS must have a security group defined so war can connect to it.

 Configuring the RDS

When the RDS is dragged on the canvas, the code for it is added to the blueprint. There is no capability for wiring it in the GUI with UCD 6.2.4.1, so the connection must be made in the code.

Figure 14 – RDS Source Code

The RDS has a number of property settings that need to be set in order to create the database and connect to it. The parameters master_username and master_password are the database administrative credentials. These are required and could be set directly but we will also have to set them again for jke.db so it is better to create a common parameter only setting them once and have the ability to change them later, if desired, at the time we provision.

To add a parameter in the GUI, select the RDS then select the arrow after the parameter you want to set. Type the name of the parameter to create and the button on the bottom will change from Set to Create Property.

Figure 15 – Create New Parameter

In the window that follows, configure the parameter. Here we are also adding it to a group. When looking at the source, Parameter Group appears as a drawer in the palette on the left. When we provision and step through the configuration, the Database Credentials group will appear in the provisioning dialog allowing you to set or modify the parameter values in that group. Do the same for the master_password and check it as hidden.

Figure 16 – Configuring Parameters and Parameter Groups

In the code, you can find the parameters added to the blueprint source at the bottom of the parameter list. The list of groups is directly below that.

Figure 17 – Modified Code After Adding Parameters and Parameter Groups

And the RDS code will change as follows.

Figure 18 – RDS Source Code after Database Credentials

Configuring the Database Connection URL

The database component jke.db needs to know the database hostname, the port and the database credentials. The application component jke.war will also need to know the database hostname and port. The RDS instance surfaces these values through Heat in parameters named endpoint_address and endpoint_port. We will add them in the source code. The name of out RDS instance is rds-dev-instance so the parameters are retrieved as:

{ get_attr: [rds-dev-instance, endpoint_address] }

{ get_attr: [rds-dev-instance, endpoint_port] }

Place these values for db_endpoint_address and dp_port in jke.db. Note, the inputs in the code stem from the component environment variables we added to the jke.db component.

Figure 19 – jke.db Database Connection Parameters

If you recall from Part 6, after selecting the jke.war component in the Diagram, we set the jke.war JKE_DB_HOST parameter from the Environment Inputs at the bottom of the palette to the left. There we selected “IP Address for jke_database”. But we did not do so for the JKE_DB_PORT. In fact, there is no equivalent parameter for “IP Port for jke-database”. Therefore, production blueprint in Part 6 will only work with the default port of 3306 or it matches the port specified in JKEDB.properties in jke.war. Further, the “IP Address for jke_database” approach we took in Part 6 will not work here because we are using an RDS and BPD does not recognize endpoint parameters for a RDS (possibly a bug). As discussed above, the way to get the endpoint address and port is to use:

{ get_attr: [database-key-name, endpoint_address] }

Set these values for JKE_DB_HOST and JKE_DB_PORT in jke.war.

Figure 20 – jke.war Database Connection Parameters

We still need to set the Environment Inputs for the jke.db component. If you notice in Figure 19 above, database_admin_user, database_admin_password, db_name, db_password, and db_user are all set to defaults. Ignoring database_admin_user and database_admin_password for now, go to the Diagram view of the blueprint and select the jke.db component. Open the Environment Inputs and using the pencil set Db Name to JKEDB, Db Password to password, and Db User to jke_user.

Figure 21 – jke.db Environment Input Values

Lastly, set the Db Admin User and Db Admin Password using the arrow next to those parameters. We don’t need to create the properties this time – we just need to use them. The final code for the jke.db component is now fully configured.

Figure 22 – Final jke.db Code Configuration

The final blueprint looks as follows. Note, the jke.war and jke.db are each using a different process – deploy_rds and the version is indicated as latest for jke.db and jke.war, versions 2.1 an 2.0 respectively. Also note that the rds-dev-instance is now wired to jke.db and jke.war. The visualization is derived from the code.

Figure 23 – Final RDS Development Blueprint

Provisioning the Blueprint with RDS

Now it is time to provision your blueprint. Click Provision at the top left of the blueprint. Create a new configuration called JKE-RDS-Development_amazon1. Among the usual groups of parameters you will see to more sections – Database Credentials and Ungrouped Parameters. Clicking on the Database Credentials will expose the two properties we added as new parameters – DB Master Username and DB Master Password.

Figure 24 – Provision Database Credentials Parameters

The Availability Zone parameter under Ungrouped Parameters was added when the RDS was added to the blueprint and represents the availability zone of the RDS. Match it to the value in Availability Zone 1 under Network Parameters.

Figure 25 – Ungrouped Parameters – Availability Zone

Figure 26 – Network Parameters – Availability Zone 1

When all the parameters are set, click Provision.

Using an Existing RDS

It is also possible to leverage an existing RDS database. Create an RDS database using the AWS console or a UCD blueprint. Then in the Service drawer in the palette on the right, click the ŸŸŸ tab, and the database will appear.

Figure 27 – Existing RDS Service

Simply drag in on the canvas and configure the endpoint_address and endpoint_port in the jke.war and jke.db components in the blueprint to reference the name and port of that RDS – jke-mysql-db in this case. Provision the blueprint.

Figure 28 – JKE Development Attached to Existing RDS Service

Closing Thoughts and Gotcha’s

When you dragged the RDS on the canvas and added security groups, they do not appear on the RDS instance even though they are correctly added to the code. Also, if you later change the name of the RDS instance using a parameter, the name of the RDS does not change even though the default name appears in the code. In additional you cannot visually wire an RDS. These are functional visual errors that should be fixed in later versions of UCD.

Since the RDS has a name, if you are going to deploy multiple instances of the blueprint change the key and the name of the RDS instance so they do not collide in AWS. If Using BPD 6.2.4.1, when you drag the RDS onto the palette, the default DB Instance Class of the instance is db.m1.small and the Engine Version is 5.6.19a. These are no longer supported in AWS. Use db.t2.micro (free version) and 5.6.35 or 5.6.37 respectively. AWS sometimes changes what they support so if your blueprints start failing, it could be because of supported combinations in Amazon Web Services being deprecated.

 My next blog entry will build upon this and introduce application processes, snapshots, and environment promotion in UrbanCode Deploy.

David J. Arnone

is an ALM and DevOps Architect specializing in IBM CLM and UrbanCode Deploy solutions and services. As a certified Rational CLM Architect, David joined Zilker Technology in January, 2017 and is part of the growing Zilker DevOps practice team.

David Arnone. David Arnone

TAGS

NEWSLETTER

Don't miss out! Get updates on new webcasts, events, and blogs.