Skip to content

The Mobile Cloud Project, Part 1.2: AWS Data

By Ray Acayan
November 3, 2009


I. Overview

The Mobile Cloud Project is my proof-of-concept for developing and hosting a scalable mobile application platform on the Amazon Web Services cloud.

Part 1.2 of this project explores the use of Amazon S3 and EBS for cloud-based storage and Amazon SimpleDB and RDS for database applications.

This project builds on the environment created in my previous blog post, “The Mobile Cloud Project, Part 1.1: Basic AWS”, where I presented an overview of Amazon Web Services and a procedure for creating virtual servers (Reserved and On-Demand instances) on Amazon EC2.

Please see the main project page for more information on the other parts of this project.


II. Consulting

I do my best to explain the concepts and techniques behind my projects. If you like my work and can use my expertise in your projects, I am available for consulting at a competitive rate.


III. Amazon S3

Amazon Simple Storage Service (S3) is an unlimited virtual data storage area hosted on the Amazon Web Services (AWS) cloud. You can create as much storage as you need in S3, and pricing is based on storage used ($0.15/GB/month) and amount of data transferred. You can mount an Amazon S3 bucket as a local filesystem on your computer using third-party applications such as Jungle Disk and S3Fox.

The following procedures assume that you have already created an Amazon AWS account, which includes access to Amazon EC2 and S3.

1. Using Jungle Disk to store files securely on Amazon S3

  • Jungle Disk is a popular service for integrated online storage and backup that also allows you to synchronize Amazon S3 files and folders between your Windows, Mac, and Linux machines. Jungle Disk adds a layer of file encryption on top of Amazon S3 security, so you need to install and configure the Jungle Disk client software on every computer that requires access to these encrypted files. Pricing is only $2/month in addition to regular Amazon S3 storage fees and data transfer fees.
  1. Go to http://www.jungledisk.com and create a new Jungle Disk Desktop Account.
  2. Go to http://www.jungledisk.com/desktop/download.aspx and download the Jungle Disk Desktop Edition for your operating system.
  3. Start the “Jungle Disk Desktop” application and create an online disk with standard or high security.
  4. Select “Jungle Disk for both Automatic Backup and as a Network Drive” and select a drive letter (Windows) or name.
  5. Start using your Jungle Disk drive and configure online backup as required.

2. Using S3Fox to share files on Amazon S3

  • S3Fox Organizer (S3Fox) is a free add-on for Firefox which enables you to transfer files between your local hard disk and Amazon S3 and share them on the web. S3Fox does not perform native encryption, so it is more suitable for sharing files rather than storing confidential data.
  1. Go to: https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key and make a note of your “Access Key ID” and “Secret Access Key” (click “Show). You will need these in the next step.
  2. Install Firefox from http://www.firefox.com if you don’t have it already.
  3. Install the Firefox add-in “Amazon S3 FireFox Organizer (S3Fox)” at https://addons.mozilla.org/en-US/firefox/addon/3247
  4. In Firefox, select “Tools”, then “S3 Organizer” to launch S3Fox.
  5. In the “S3 Account Manager” window, enter a description in the “Account Name” text box, then enter your Access Key ID and Secret Access Key.
  6. Y ou can now create an S3 bucket by right-clicking on the right pane and selecting “Create Directory”. Note that the directory name should be globally unique across all Amazon S3 buckets, such as a domain name.
  7. You can also transfer files between your local filesystem (left pane) and S3 (right pane) by clicking the arrows, similar to FTP.
  8. You can share a file or folder by right-clicking on it in S3 (right pane), selecting “Edit ACL”, and changing Read/Write/FullControl permissions.
  9. Right-click again on the file or folder, select “Copy URL to Clipboard”, and paste it on an email message or text editor. The file/folder can now be accessed over the web via that URL, e.g. http://<filename>.s3.amazonaws.com

3. Bundling an AMI

  • “Bundling an AMI” essentially creates an image snapshot of the current EC2 instance. When an EC2 instance is terminated, all the files and directories in the root folder are deleted, including the operating system, installed software, and all data. Bundling the currently running instance into an AMI allows you to restore it later or run multiple instances with the same AMI configuration.
  • The following procedure assumes that you have already installed PuTTY and FileZilla on your Windows client and configured them to connect to the EC2 instance, as described in Steps IV.9 and IV.10 in my previous blog post.
  1. The Amazon EC2 and AMI tools first need to be installed on the EC2 instance. Then, the private key and cert must be copied to the instance before the AMI tools can be executed on that instance.

    Connect to the EC2 instance using PuTTY and enter the following commands at the Linux prompt:

    2. $ sudo mkdir /.aws
    $ sudo mkdir /.aws/.ec2
  2. On your Windows client, use FileZilla to copy your private key and cert files, “C:\aws\ec2\ec2-pk.pem” and “C:\aws\ec2\ec2-cert.pem” to the “/.aws/.ec2″ directory on the EC2 instance.
  3. Connect to the EC2 instance using PuTTY.
  4. Enter the following at the Linux prompt to add the “multiverse” to your apt sources.list:
    5. $ sudo nano /etc/apt/sources.list
  5. Add the text “multiverse” as in the line below, then save the file:
    6. deb http://us.ec2.archive.ubuntu.com/ubuntu/ karmic main universe multiverse
  6. Enter the following commands to install the Amazon EC2 and AMI tools:
    7. $ sudo apt-get update
    $ sudo apt-get install ec2-api-tools
    $ sudo apt-get install ec2-ami-tools
  7. You can now execute EC2 AMI commands from the Linux command line.
  8. Enter the following commands to bundle an AMI for this instance using the “ec2-bundle-vol” command, then store the AMI in the EBS volume created previously.
    9. $ sudo rm -Rf /mnt/ami_snapshot
    $ sudo mkdir /mnt/ami_snapshot
    $ ec2-bundle-vol -d /mnt/ami_snapshot -k /.aws/.ec2/ec2-pk.pem -c /.aws/.ec2/ec2-cert.pem -u <username> -r i386 -p ami_snapshot
    (Note that <username> is your 12-digit AWS Account ID without the hyphens.)

    (Note: You may want to put these commands into a script as part of a regular backup procedure.)

  9. Wait a few minutes while the AMI image is being created.
  10. Run the following command to upload your AMI to an S3 bucket. This will allow you to later attach this AMI to a new or existing EC2 instance.
    11. $ ec2-upload-bundle -b <s3_bucket_name> -m /mnt/ami_snapshot/<ami_filename>.manifest.xml -a <access_key_id> -s <secret_access_key>
    $ ec2-register <s3_bucket_name>/ami_snapshot.manifest.xml -K /ebs/.aws/.ec2/ec2-pk.pem -C /ebs/.aws/.ec2/ec2-cert.pem

    The <s3_bucket_name> is a unique name you create to identify the S3 bucket where this AMI will be stored. You can find the <access_key_id> and <secret_access_key> from your AWS credentials:
    https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key

 

IV. Amazon EBS

Amazon Elastic Block Storage (EBS) is a type of storage that acts like an unformatted external hard disk that can be mounted on an Amazon EC2 instance and formatted with a filesystems such as ext3 (Linux) or NTFS (Windows). EBS costs 10 cents per GB provisioned, while S3 costs 15 cents per GB used.

The procedures below assume an Amazon EC2 instance running Ubuntu Linux, an Elastic IP address associated with this instance, a local Windows client with Putty and Filezilla configured to connect to it, and the EC2 API tools installed on the EC2 instance. These were all configured in my previous blog post “The Mobile Cloud Project, Part 1.1: Basic AWS”.

1. Creating an Amazon EBS Volume

  1. Sign in to the AWS Console at http://console.aws.amazon.com
  2. Go to “Volumes” on the left menu.
  3. Click “Create Volume”.
  4. Enter “10″ GiB, select the same Availability Zone as your instance, and select “No Snapshot” then click “Create”.
  5. Wait a few minutes until the Status changes from “creating” to “available”.
  6. Right-click on the volume and click “Attach Volume”.
  7. Select the Instance ID of the EC2 instance you want to attach this volume to, select a Device name from the listbox (e.g. /dev/sdf), then click “Attach”.
  8. Connect to the EC2 instance via Putty.
  9. Enable Cryptsetup-LUKS encryption on the EBS volume (e.g. /dev/sdf) you just created by entering the following commands in the instance prompt:
    10. $ sudo apt-get install cryptsetup
    $ sudo modprobe dm-crypt
    $ sudo modprobe sha256
    $ sudo modprobe aes
    $ sudo cryptsetup –verify-passphrase luksFormat /dev/sdf -c aes -s 256 -h sha256

    (Type uppercase “YES” to continue. Enter and Verify your LUKS passphrase.)

  10. Create an ext3 filesystem on the EBS volume (e.g. /dev/sdf) you just created by entering the following commands in the instance prompt:
    11. $ sudo mke2fs -j /dev/sdf
    $ sudo mkdir /ebs
    $ sudo mount /dev/sdf /ebs
    $ sudo chown ubuntu:ubuntu /ebs
  11. You can now access this EBS volume through /ebs

 

V. Amazon SimpleDB

Amazon SimpleDB is essentially a cloud-based non-relational database cluster for your applications running on EC2 and S3. SimpleDB uses REST requests and responses over HTTP, unlike relational databases which require a persistent connection. Pricing is based on storage size and data transferred, and Amazon currently offers a popular “free tier” for low-usage applications.

SimpleDB has its own terminology: domain (database or table), items (rows), attributes (columns), and value (similar to a cell on a spreadsheet). Queries use “Intersection” instead of “OR” and “Union” instead of “AND”. All data in SimpleDB are treated as strings.

1. Using SimpleDB with Ruby on Rails

  • There are several programming interfaces available between SimpleDB and the major programming languages. The example below shows how to perform simple CRUD (Create, Read, Update, Delete) operations on SimpleDB using Ruby on Rails and the AWS-SDB interface.
$ export AMAZON_ACCESS_KEY_ID=’‘
$ export AMAZON_SECRET_ACCESS_KEY=’‘
$ sudo gem install aws-sdb
$ irb –simple-prompt
>> require ‘rubygems’
>> require ‘aws_sdb’
>> require ‘pp’
>> require ‘yaml’
>> sdb = AwsSdb::Service.new
>> sdb.create_domain(’db_test’) # Create a domain called “db_test”
>> sdb.list_domains # List all domains
>> sdb.put_attributes(’db_test’, ”) # Insert a new record
>> sdb.get_attributes(’db_test’, ”) # Retrieve record with this ID
>> sdb.delete_attributes(’db_test’, ”) # Delete record with this ID
>> sdb.query(domain, ) # Query the domain
>> sdb.delete_domain(’db_test’) # Delete the “db_test” domain

More details on this example can be found in this tutorial by Craic Computing. The article “Introduction to AWS for Ruby Developers” is also a great resource.

 

VI. Amazon RDS

Amazon Relational Database Service (RDS) is essentially a dedicated special-purpose EC2 instance with a fully-managed MySQL (5.1) database accessible via API calls over the web. RDS currently runs on a single DB Instance but will support database replication in the future.

The following commands are used to create, list, and delete DB Instances in RDS using the API command line tools.

$ rds-create-db-instance <db_instance_name> -s 50 -c db.m1.small -e MySQL5.1 – u sa -p –backup-retention-period 3
$ rds-describe-db-instances
$ rds-delete-db-instance <db_instance_name> –final-snapshot-identifier myfinaldbsnapshot

From the point of view of an application, a DB Instance appears just like any other MySQL server. Applications require no modification aside from pointing to the name or Elastic IP address of RDS DB Instance, instead of referring to a physical MySQL server.


VII. Next Steps

 

VIII. References

1. Amazon Web Services Technical Documentation
http://aws.amazon.com/documentation

2. Jungle Disk
http://www.jungledisk.com

3. S3Fox Organizer
http://www.s3fox.net/

4. A Tutorial on aws-sdb: A Ruby Interface to AWS SimpleDB
http://www.craic.com/tutorials/aws/aws_sdb/aws_sdb_tutorial.html

5. Introduction to AWS for Ruby Developers
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=846&categoryID=176




Posted in Projects. Tagged with , , , , , , , .

By Ray Acayan

No comments