I. Overview

The Mobile Cloud Project is my proof-of-concept for developing and hosting a scalable mobile application platform on the Amazon Web Services cloud.

Part 1.2 of this project explores the use of Amazon S3 and EBS for cloud-based storage and Amazon SimpleDB and RDS for database applications.

This project builds on the environment created in my previous blog post, “The Mobile Cloud Project, Part 1.1: Basic AWS”, where I presented an overview of Amazon Web Services and a procedure for creating virtual servers (Reserved and On-Demand instances) on Amazon EC2.

Please see the main project page for more information on the other parts of this project.

II. Consulting

I do my best to explain the concepts and techniques behind my projects. If you like my work and can use my expertise in your projects, I am available for consulting at a competitive rate.

III. Amazon S3

Amazon Simple Storage Service (S3) is an unlimited virtual data storage area hosted on the Amazon Web Services (AWS) cloud. You can create as much storage as you need in S3, and pricing is based on storage used ($0.15/GB/month) and amount of data transferred. You can mount an Amazon S3 bucket as a local filesystem on your computer using third-party applications such as Jungle Disk and S3Fox.

The following procedures assume that you have already created an Amazon AWS account, which includes access to Amazon EC2 and S3.

1. Using Jungle Disk to store files securely on Amazon S3

• Jungle Disk is a popular service for integrated online storage and backup that also allows you to synchronize Amazon S3 files and folders between your Windows, Mac, and Linux machines. Jungle Disk adds a layer of file encryption on top of Amazon S3 security, so you need to install and configure the Jungle Disk client software on every computer that requires access to these encrypted files. Pricing is only $2/month in addition to regular Amazon S3 storage fees and data transfer fees.

1. Go to http://www.jungledisk.com and create a new Jungle Disk Desktop Account.
2. Go to http://www.jungledisk.com/desktop/download.aspx and download the Jungle Disk Desktop Edition for your operating system.
3. Start the “Jungle Disk Desktop” application and create an online disk with standard or high security.
4. Select “Jungle Disk for both Automatic Backup and as a Network Drive” and select a drive letter (Windows) or name.
5. Start using your Jungle Disk drive and configure online backup as required.

2. Using S3Fox to share files on Amazon S3

• S3Fox Organizer (S3Fox) is a free add-on for Firefox which enables you to transfer files between your local hard disk and Amazon S3 and share them on the web. S3Fox does not perform native encryption, so it is more suitable for sharing files rather than storing confidential data.
  

1. Go to: https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key and make a note of your “Access Key ID” and “Secret Access Key” (click “Show). You will need these in the next step.
2. Install Firefox from http://www.firefox.com if you don’t have it already.
3. Install the Firefox add-in “Amazon S3 FireFox Organizer (S3Fox)” at https://addons.mozilla.org/en-US/firefox/addon/3247
4. In Firefox, select “Tools”, then “S3 Organizer” to launch S3Fox.
5. In the “S3 Account Manager” window, enter a description in the “Account Name” text box, then enter your Access Key ID and Secret Access Key.
6. Y ou can now create an S3 bucket by right-clicking on the right pane and selecting “Create Directory”. Note that the directory name should be globally unique across all Amazon S3 buckets, such as a domain name.
7. You can also transfer files between your local filesystem (left pane) and S3 (right pane) by clicking the arrows, similar to FTP.
8. You can share a file or folder by right-clicking on it in S3 (right pane), selecting “Edit ACL”, and changing Read/Write/FullControl permissions.
9. Right-click again on the file or folder, select “Copy URL to Clipboard”, and paste it on an email message or text editor. The file/folder can now be accessed over the web via that URL, e.g. http://<filename>.s3.amazonaws.com

3. Bundling an AMI

• “Bundling an AMI” essentially creates an image snapshot of the current EC2 instance. When an EC2 instance is terminated, all the files and directories in the root folder are deleted, including the operating system, installed software, and all data. Bundling the currently running instance into an AMI allows you to restore it later or run multiple instances with the same AMI configuration.
• The following procedure assumes that you have already installed PuTTY and FileZilla on your Windows client and configured them to connect to the EC2 instance, as described in Steps IV.9 and IV.10 in my previous blog post.

1. The Amazon EC2 and AMI tools first need to be installed on the EC2 instance. Then, the private key and cert must be copied to the instance before the AMI tools can be executed on that instance.

   Connect to the EC2 instance using PuTTY and enter the following commands at the Linux prompt:

$ sudo mkdir /.aws
$ sudo mkdir /.aws/.ec2
2. On your Windows client, use FileZilla to copy your private key and cert files, “C:\aws\ec2\ec2-pk.pem” and “C:\aws\ec2\ec2-cert.pem” to the “/.aws/.ec2″ directory on the EC2 instance.
3. Connect to the EC2 instance using PuTTY.
4. Enter the following at the Linux prompt to add the “multiverse” to your apt sources.list:
$ sudo nano /etc/apt/sources.list
5. Add the text “multiverse” as in the line below, then save the file:
deb http://us.ec2.archive.ubuntu.com/ubuntu/ karmic main universe multiverse
6. Enter the following commands to install the Amazon EC2 and AMI tools:
$ sudo apt-get update
$ sudo apt-get install ec2-api-tools
$ sudo apt-get install ec2-ami-tools
7. You can now execute EC2 AMI commands from the Linux command line.
8. Enter the following commands to bundle an AMI for this instance using the “ec2-bundle-vol” command, then store the AMI in the EBS volume created previously.
$ sudo rm -Rf /mnt/ami_snapshot
$ sudo mkdir /mnt/ami_snapshot
$ ec2-bundle-vol -d /mnt/ami_snapshot -k /.aws/.ec2/ec2-pk.pem -c /.aws/.ec2/ec2-cert.pem -u <username> -r i386 -p ami_snapshot
(Note that <username> is your 12-digit AWS Account ID without the hyphens.)

(Note: You may want to put these commands into a script as part of a regular backup procedure.)

9. Wait a few minutes while the AMI image is being created.
10. Run the following command to upload your AMI to an S3 bucket. This will allow you to later attach this AMI to a new or existing EC2 instance.
$ ec2-upload-bundle -b <s3_bucket_name> -m /mnt/ami_snapshot/<ami_filename>.manifest.xml -a <access_key_id> -s <secret_access_key>
$ ec2-register <s3_bucket_name>/ami_snapshot.manifest.xml -K /ebs/.aws/.ec2/ec2-pk.pem -C /ebs/.aws/.ec2/ec2-cert.pem

The <s3_bucket_name> is a unique name you create to identify the S3 bucket where this AMI will be stored. You can find the <access_key_id> and <secret_access_key> from your AWS credentials:
https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key

IV. Amazon EBS

Amazon Elastic Block Storage (EBS) is a type of storage that acts like an unformatted external hard disk that can be mounted on an Amazon EC2 instance and formatted with a filesystems such as ext3 (Linux) or NTFS (Windows). EBS costs 10 cents per GB provisioned, while S3 costs 15 cents per GB used.

The procedures below assume an Amazon EC2 instance running Ubuntu Linux, an Elastic IP address associated with this instance, a local Windows client with Putty and Filezilla configured to connect to it, and the EC2 API tools installed on the EC2 instance. These were all configured in my previous blog post “The Mobile Cloud Project, Part 1.1: Basic AWS”.

1. Creating an Amazon EBS Volume

1. Sign in to the AWS Console at http://console.aws.amazon.com
2. Go to “Volumes” on the left menu.
3. Click “Create Volume”.
4. Enter “10″ GiB, select the same Availability Zone as your instance, and select “No Snapshot” then click “Create”.
5. Wait a few minutes until the Status changes from “creating” to “available”.
6. Right-click on the volume and click “Attach Volume”.
7. Select the Instance ID of the EC2 instance you want to attach this volume to, select a Device name from the listbox (e.g. /dev/sdf), then click “Attach”.
8. Connect to the EC2 instance via Putty.
9. Enable Cryptsetup-LUKS encryption on the EBS volume (e.g. /dev/sdf) you just created by entering the following commands in the instance prompt:
$ sudo apt-get install cryptsetup
$ sudo modprobe dm-crypt
$ sudo modprobe sha256
$ sudo modprobe aes
$ sudo cryptsetup –verify-passphrase luksFormat /dev/sdf -c aes -s 256 -h sha256

(Type uppercase “YES” to continue. Enter and Verify your LUKS passphrase.)

10. Create an ext3 filesystem on the EBS volume (e.g. /dev/sdf) you just created by entering the following commands in the instance prompt:
$ sudo mke2fs -j /dev/sdf
$ sudo mkdir /ebs
$ sudo mount /dev/sdf /ebs
$ sudo chown ubuntu:ubuntu /ebs
11. You can now access this EBS volume through /ebs

V. Amazon SimpleDB

Amazon SimpleDB is essentially a cloud-based non-relational database cluster for your applications running on EC2 and S3. SimpleDB uses REST requests and responses over HTTP, unlike relational databases which require a persistent connection. Pricing is based on storage size and data transferred, and Amazon currently offers a popular “free tier” for low-usage applications.

SimpleDB has its own terminology: domain (database or table), items (rows), attributes (columns), and value (similar to a cell on a spreadsheet). Queries use “Intersection” instead of “OR” and “Union” instead of “AND”. All data in SimpleDB are treated as strings.

1. Using SimpleDB with Ruby on Rails

• There are several programming interfaces available between SimpleDB and the major programming languages. The example below shows how to perform simple CRUD (Create, Read, Update, Delete) operations on SimpleDB using Ruby on Rails and the AWS-SDB interface.

More details on this example can be found in this tutorial by Craic Computing. The article “Introduction to AWS for Ruby Developers” is also a great resource.

VI. Amazon RDS

Amazon Relational Database Service (RDS) is essentially a dedicated special-purpose EC2 instance with a fully-managed MySQL (5.1) database accessible via API calls over the web. RDS currently runs on a single DB Instance but will support database replication in the future.

The following commands are used to create, list, and delete DB Instances in RDS using the API command line tools.

From the point of view of an application, a DB Instance appears just like any other MySQL server. Applications require no modification aside from pointing to the name or Elastic IP address of RDS DB Instance, instead of referring to a physical MySQL server.

VII. Next Steps

• In Part 1.3: AWS Fault Tolerance, I will explore the AutoScaling and Elastic Load Balancing features of Amazon Web Services.
• Please see The Mobile Cloud Project page for more info on the other parts of this project.

VIII. References

1. Amazon Web Services Technical Documentation
http://aws.amazon.com/documentation

2. Jungle Disk
http://www.jungledisk.com

3. S3Fox Organizer
http://www.s3fox.net/

4. A Tutorial on aws-sdb: A Ruby Interface to AWS SimpleDB
http://www.craic.com/tutorials/aws/aws_sdb/aws_sdb_tutorial.html

5. Introduction to AWS for Ruby Developers
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=846&categoryID=176

I. Overview

This side project presents my procedure for migrating my WordPress blog from a web hosting provider running cPanel to an Amazon EC2 instance.

This project builds on the environment created in my previous blog post, “The Mobile Cloud Project, Part 1.1: Basic AWS”, where I presented an overview of Amazon Web Services and a procedure for creating virtual servers (Reserved and On-Demand instances) on Amazon EC2.

II. Consulting

I do my best to explain the concepts and techniques behind my projects. If you like my work and can use my expertise in your projects, I am available for consulting at a competitive rate.

III. Procedure

Once the Amazon EC2 instance was created and configured in my previous blog post, this migration procedure is actually fairly straightforward. From the point of view of a terminal console, an Amazon EC2 instance running Ubuntu Linux is very similar to any other Ubuntu server. The only difference is that certain tasks that were simplified behind a web hosting provider’s cPanel now need to be done explicitly on an EC2 instance.

This procedure assumes an existing WordPress blog at http://www.RayAcayan.com/blog (i.e. this blog), an existing EC2 instance running Ubuntu 9.10 (Karmic), an Elastic IP address associated with that instance, and a local Windows client with Putty and FileZilla configured to connect to both the old blog’s web hosting provider and the new blog’s EC2 instance. These were all configured in my previous blog post “The Mobile Cloud Project, Part 1.1: Basic AWS”.

1. Install packages on EC2 instance

1. Connect to the EC2 instance using Putty.
2. Install Apache2, MySQL, PHP, Ruby on Rails, Python, and Django packages with the following commands.
$ sudo apt-get update
$ sudo apt-get install apache2 mysql-server libapache2-mod-auth-mysql
$ sudo apt-get install php5-mysql phpmyadmin
$ sudo apt-get install build-essential libssl-dev libreadline5-dev zlib1g-dev
$ sudo apt-get install libmysqlclient15-dev

$ sudo apt-get install ruby-full rubygems1.9.1 rails libmysql-ruby

$ sudo apt-get install python2.6 python3 python-django
$ sudo apt-get install libapache2-mod-python python-mysqldb

2. Enable mod_rewrite on Apache

1. The mod_rewrite module allows WordPress to use pretty permalinks on your blog, such as /category/postname/
$ sudo a2enmod rewrite
$ sudo nano /etc/apache2/sites-enabled/000-default
2. Modify this file to show the following changes (i.e. change “None” to “All” to allow pretty permalinks, and add # comments to disable directory listing) then save this file.
DocumentRoot /var/www/
[...]
Options FollowSymLinks
AllowOverride All
[...]
# Options Indexes FollowSymLinks MultiViews
# AllowOverride None
# Order allow,deny
# allow from all
3. Restart Apache:
$ sudo /etc/init.d/apache2 restart

3. Install WordPress on the EC2 instance

1. Connect to the EC2 instance using Putty.
2. Create the WordPress username and database on MySQL by entering the following commands:
$ sudo mysql -u root -p <password>
mysql> create database wordpress;
mysql> create user <wp_username>;
mysql> set password for <wp_username> = password(”<password>”);
mysql> grant all on wordpress.* to <wp_username>@localhost identified by `<password>’;
mysql> quit
3. Enter the following commands at the Linux prompt to download and install WordPress:
$ cd /var/www/blog
$ sudo wget http://wordpress.org/latest.tar.gz
$ sudo tar -xzvf latest.tar.gz
$ sudo mv wordpress blog
4. Go to http://<elastic ip>/blog/wp-admin/install.php to execute the WordPress install script.
5. Download and install the WordPress themes and plugins that were in the previous blog (cPanel).

4. Copy WordPress files and database from the old blog to the new blog

1. Open FileZilla client and connect to the web hosting provider’s server that is hosting the old blog. (See the provider’s instructions on how to do this.)
2. Download all files in the old blog: ~/public_html/blog or /var/www/blog
3. Open a second FileZilla client to connect to the new blog (EC2).
4. Upload your custom files and directories to the /var/www/blog directory of the new blog.
5. Make sure that you have also copied any hidden files (.htaccess) from the old blog to the new blog.
6. Open the WordPress Dashboard on the old blog (cPanel), click “Export” in the “Tools” menu on the left, and save the .xml file.
7. Open the WordPress Dashboard on the new blog (EC2), click “Import” in the “Tools” menu on the left, and upload the .xml file from the previous step.
8. Install the “WP-DB-Backup” plugin on both the old and new blogs.
9. On the old blog’s WordPress admin, click “Backup” on the “Tools” menu on the left and save the file as “oldwordpress.sql”.
10. Connect to the EC2 instance via FileZilla and upload the “oldwordpress.sql” file.
11. Connect to the EC2 instance via Putty.
12. At the Linux prompt, enter “mysql wordpress < oldwordpress.sql –u root -p” to import the old WordPress database to the new one.
13. Verify that the new blog is working at http://<elastic ip>/blog including links and styles.

5. Remap DNS to the new blog IP address on EC2

• DNS needs to be configured to remap the blog’s domain name to the Elastic IP address of the new blog on EC2.

1. Before proceeding, add an entry “<elastic ip> www.RayAcayan.com” to the local hosts file at C:\windows\system32\drivers\etc
2. Verify that the blog is working properly using the normal URL http://www.RayAcayan.com

At this point, only the local computer can access the new blog via URL because of the hosts file entry. The new IP address will need to be propagated to the rest of the Internet via an external DNS provider so that everyone else can access it.

3. Select an external DNS provider such as http://www.easyDNS.com and purchase a DNS-only service for about $20/year.
4. Add a DNS entry to map the hostname to the Elastic IP address of the new blog.
5. Go to the domain registrar’s website where the old blog domain is currently registered.
6. Change the name servers of the domain to point to the name servers of the external DNS provider.
7. Wait several hours for the new DNS settings to propagate on the Internet. Enter “nslookup” at the command prompt. If the response is the elastic IP address of the EC2 instance, then the DNS propagation has completed.
8. Remove the hosts file entry created in step 5.i. and run “ipconfig /dnsflush” at the command prompt. Verify that the blog is still reachable via the normal URL: http://www.RayAcayan.com/blog

6. Install Additional Applications

1. Install Webalizer


Webalizer is a web stats application offered by your web hosting provider’s cPanel. You will need to install it manually on your EC2 instance if you want to continue using it.

1. Connect to the EC2 instance using Putty.
2. Enter the following command to install and configure Webalizer on the EC2 instance:
$ sudo apt-get install webalizer
$ sudo nano /etc/webalizer/webalizer.conf
Change the text “LogFile /var/log/apache2/access.log.1″ to “LogFile /var/log/apache2/access.log” then save the webalizer config file.
3. Create a cron job to run webalizer every 30 minutes:
$ sudo crontab -e
(Add the following line to the crontab:)
30 * * * * /usr/bin/webalizer

7. Configure Basic Security

1. Secure the Apache web server with mod_security and Core Rule Set

The “mod_security” Apache module is an open source intrusion detection and prevention engine for web applications. It protects the Apache web server from common attacks such as SQL injection, cross-site scripting, etc.

1. Connect to the EC2 instance using Putty.
2. Enter the following commands at the Linux prompt:
$ sudo mkdir ~/install
$ cd ~/install
$ sudo wget http://www.modsecurity.org/download/modsecurity-apache_2.5.10.tar.gz
$ sudo tar –xzvf modsecurity-apache_2.5.10.tar.gz
$ cd modsecurity-apache_2.5.10/rules
$ sudo cp *.conf /etc/apache2/conf.d
$ sudo cp base_rules /etc/apache2/conf.d/modsecurity
$ sudo cp optional_rules /etc/apache2/conf.d
$ sudo mkdir /etc/apache2/logs
$ sudo apt-get install libapache-mod-security
$ sudo a2enmod mod-security
3. Restart Apache:
$ sudo /etc/init.d/apache2 restart
2. Configure password-protection on the PHPMyAdmin directory
1. Connect to the EC2 instance using Putty.
2. Enter the following commands at the Linux prompt:
$ sudo mkdir /usr/local/apache2
$ sudo htdigest -c /usr/local/apache2/.htpasswd_phpmyadmin “PHPMyAdmin” phpmyadmin
(Enter password)
$ sudo cp /etc/apache2/mods-available/auth_digest.load /etc/apache2/mods-enabled
$ sudo nano /etc/apache2/apache2.conf
3. Add the following text to the end of the apache2.conf file then save it:
<Directory /usr/share/phpmyadmin/>
  Authtype digest
  AuthName “PHPMyAdmin”
  AuthUserFile /usr/local/apache2/.htpasswd_phpmyadmin
  require user phpmyadmin
</Directory>
4. For added security, insert the following lines inside the <Directory> tag above to only allow connections from certain IP addresses:
  Order deny,allow
  Deny from all
  Allow from <static IP address>
5. Restart Apache:
$ sudo /etc/init.d/apache2 restart
3. Configure password-protection on the WordPress Admin directory
1. Connect to the EC2 instance using Putty.
2. Enter the following commands at the Linux prompt:
$ sudo mkdir /usr/local/apache2
$ sudo htdigest -c /usr/local/apache2/.htpasswd_wp-admin “WordPress Admin” wp-admin
(Enter password)
$ sudo cp /etc/apache2/mods-available/auth_digest.load /etc/apache2/mods-enabled
$ sudo nano /etc/apache2/apache2.conf
3. Add the following text to the end of the apache2.conf file then save it:
<Directory /var/www/blog/wp-admin/>
  Authtype digest
  AuthName “WordPress Admin”
  AuthUserFile /usr/local/apache2/.htpasswd_wp-admin
  require user wp-admin
</Directory>
4. For added security, insert the following lines inside the <Directory> tag above to only allow connections from certain IP addresses:
  Order deny,allow
  Deny from all
  Allow from <static IP address>
5. Restart Apache:
$ sudo /etc/init.d/apache2 restart
4. Configure password-protection on the Webalizer directory
1. Connect to the EC2 instance using Putty.
2. Enter the following commands at the Linux prompt:
$ sudo mkdir /usr/local/apache2
$ sudo htdigest -c /usr/local/apache2/.htpasswd_webalizer “Webalizer Stats” webalizer
(Enter password)
$ sudo cp /etc/apache2/mods-available/auth_digest.load /etc/apache2/mods-enabled
$ sudo nano /etc/apache2/apache2.conf
3. Add the following text to the end of the apache2.conf file then save it:
<Directory /var/www/webalizer/>
  Authtype digest
  AuthName “Webalizer Stats”
  AuthUserFile /usr/local/apache2/.htpasswd_webalizer
  require user webalizer
</Directory>
4. For added security, insert the following lines inside the <Directory> tag above to only allow connections from certain IP addresses:
  Order deny,allow
  Deny from all
  Allow from <static IP address>
5. Restart Apache:
$ sudo /etc/init.d/apache2 restart

IV. References

1. Amazon Web Services Technical Documentation
http://aws.amazon.com/documentation

2. Official Ubuntu Documentation
https://help.ubuntu.com

3. RubyOnRails.org
http://rubyonrails.org/download/

4. Python Programming Language — Official Website
http://www.python.org

5. The Django Project
http://www.djangoproject.com/

6. Moving WordPress
http://codex.wordpress.org/Moving_WordPress

7. easyDNS
http://www.easydns.com

8. OWASP ModSecurity Core Rule Set Project
http://www.owasp.org/index.php/Category:OWASP_ModSecurity_Core_Rule_Set_Project

I. Overview

The Mobile Cloud Project is my proof-of-concept for developing and hosting a scalable mobile application platform on the Amazon Web Services cloud..

Part 1.1 of this project provides a basic overview of Amazon Web Services and describes my procedure for building Ubuntu Linux virtual servers (Reserved and On-Demand instances) on Amazon EC2.

Please see the main project page for more information on the other parts of this project.

II. Consulting

I do my best to explain the concepts and techniques behind my projects. If you like my work and can use my expertise in your projects, I am available for consulting at a competitive rate.

III. Overview of Amazon Web Services

Amazon Web Services (AWS) is a collection of technologies that enable us to host our applications on Amazon’s server and network infrastructure (the “cloud”). AWS can provide an individual or startup with the potential computing power of thousands of networked servers, petabytes of storage space, and fault-tolerant capabilities, which rival the infrastructure of large corporations that possess multiple data centres and IT departments. You require no physical hardware, network infrastructure, electricity, nor legions of IT staff, in order to run your powerful virtual enterprise in the cloud. The scalability of AWS makes it an ideal solution for hosting the back-end platform for popular mobile apps used by millions of users.

1. Amazon EC2

An “instance”, or virtual server, is the basic unit of computing resource in the Amazon Elastic Compute Cloud (EC2). You can create as many instances as you need, and pricing is based on instance type, “size”, and hourly usage. A Reserved Instance is a dedicated virtual server which is often kept running all the time, while an On-Demand instance is usually turned on only as required.

2. Amazon S3

Amazon Simple Storage Service (S3) is a virtual data storage area. A “bucket” is analogous to a filesystem that can store objects or files. You can create as much data storage as you need in S3, and pricing is based on storage size and amount of data transferred.

3. Elastic IP Addresses

An Elastic IP address is an Internet-routable static IP address that is allocated to your AWS EC2 account. You can map an Elastic IP address to any running instance (which has a private IP address), and quickly remap to a different instance if the other instance fails. Network address translation is enabled automatically via 1:1 NAT. There is no charge for elastic IP addresses while in use.

IV. Procedure

In this procedure, I will demonstrate how to create two EC2 instances: a Reserved Instance called “Galactica” (which will eventually host this blog and related web apps) and an On-Demand instance called “Pegasus” for future use.

1. Sign up for a free AWS account

1. Go to http://aws.amazon.com and click “Sign Up Now”.
2. Fill out the requested information.

2. Sign up for Amazon EC2 and S3

1. Go to http://aws.amazon.com/ec2 and click “Sign Up for Amazon EC2”.
2. Review the pricing details and enter your credit card and billing information.
3. Complete the identity verification by telephone.
4. You are now signed up for both EC2 and S3.

3. Sign in to the AWS Management Console

1. Go to http://aws.amazon.com/console and click “Sign in to the AWS Console” (EC2).

4. Create Key Pairs

You will need to create a key pair for each instance you plan to launch. This allows you to access your instance securely via SSH (Linux) or RDP (Windows).

1. In Windows Explorer, create the folders “C:\aws” and “C:\aws\ec2″
2. Click “Key Pairs” in the left menu, click “Create Key Pair”, and enter a name for this key pair (e.g. “Galactica”).
3. Save the “Galactica.pem” private key file in the C:\aws\ec2 folder.
4. Create another key pair for the second instance (e.g. “Pegasus”).

5. Create a Security Group

A Security Group is essentially a set of firewall rules that can be applied to your instances. A cloud-based virtual server is still vulnerable to hackers just like any other server connected to the Internet. You need to remain vigilant and follow cloud security best practices. (See the Cloud Security Alliance website for more information.)

1. Click “Security Groups” from the left menu, then click “Create Security Group”.
2. Enter a name for this security group (e.g. “Stargate”) then click “Create”.
3. In the “Allowed Connections” section, select “SSH” then click “Save”.
4. In the same section, select “HTTP” then click “Save”.
5. Add more firewall rules as required. All other network traffic will be blocked.

6. Purchase a Reserved Instance

A Reserved Instance is a prepaid instance that offers “significantly” discounted usage fees compared to an On-Demand instance. In this example, a “Small” Linux/Unix reserved instance will cost $350 for a 3-yr term and $0.03/hr (roughly $22/month) usage fees, while the same On-Demand instance will cost $0.085/hr in usage fees (as of Nov 1, 2009), or about $62/month if used 24/7. Usage fees and discounts are calculated automatically. For example, if you have X reserved instances, then the first X instances currently running will be charged at the discounted usage fee, while the remaining instances are charged at the on-demand usage fee.

1. Click “Instances” on the left menu.
2. Click on the “Reserved Instances” dropdown list and select “Purchase Reserved Instances”.
3. Select “Linux/UNIX” platform, “m1.small” instance type, “Best Available” zone, a term of “3 years”, and “1” instance.
4. Click “Continue” then click “Place Order” to confirm.
5. You may need to wait a few minutes until the payment is processed and the reserved instance is activated. Click “Close”.

7. Launch AMI Instances

An Amazon Machine Image (AMI) is essentially a disk image containing a pre-configured operating system and software. You have the ability to create your own AMI disk image from an existing server or select a pre-built AMI from Amazon’s list.

1. Click “Instances” on the left menu, then click “Launch Instance”.
2. Click “Community AMIs” tab.
3. In the textbox to the right of “All Images” and “All Platforms”, enter “ubuntu-images-us” for a list of official Ubuntu releases (currently Ubuntu 9.10 Karmic as of Oct 29, 2009).
4. Select an AMI from the list. Amazon charges less for 32-bit instances, so select the cheaper 32-bit “i386” AMI rather than the 64-bit “amd64” AMI. In this example, select the “ami-1515f67c” AMI ID and click “Select”.
5. On the next page, enter “1” instance, “Small” (m1.small) instance type, “Galactica” key pair, and “Stargate” security group, then click “Launch”.
6. Repeat these steps to launch another instance for the “Pegasus” key pair.

8. Create Elastic IP address

As mentioned previously, an Elastic IP address is simply an Internet-routable static IP address that is allocated to your AWS EC2 account and can be mapped (associated) with any running instance.

1. Click “Elastic IPs” on the left menu.
2. Click “Allocate New Address”, then click “Yes, Allocate” to confirm.
3. Check the box next to the IP address, then click “Associate”.
4. Select the appropriate Instance ID, then click “Associate”.
5. You can now configure your DNS to map your existing domain name to this Elastic IP address and access this instance over the Internet.

9. Install and configure PuTTY

PuTTY is a free SSH client that enables you to configure and install packages on your Amazon EC2 instance from Windows.

1. Download PuTTY, PuTTYgen, and PSCP from:http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
2. Create a folder “C:\putty” and move these files to that folder.
3. Go to Control Panel, System, Advanced system settings, Environment Variables, Add “C:\putty” to the PATH system variable.
4. Run the “puttygen.exe” application from Windows.
5. Click the “File” menu, select “Load private key”, then select C:\aws\ec2\Galactica.pem (from Step 4.iii.)
6. The application will prompt you to save the file in PuTTY’s own format and click “OK”.
8. Click the “File” menu, click “Save private key”. Click “Yes” to save this key without a passphrase to protect it.
9. Save the “Galactica.ppk” file in the C:\aws\ec2 folder.
10. Run the “putty.exe” application from Windows.
11. Enter the Elastic IP address in both the “Host Name” and “Saved Sessions” text boxes.
12. On the left pane, click “SSH” then click “Auth”.
13. Click “Browse” then select the file C:\aws\ec2\Galactica.ppk.
14. On the left pane, click “Connection” then click “Data”.
15. Enter “ubuntu” in the “Auto-login username” text box.
16. Click “Session” at the top of the left pane, then click “Save”.
17. Click “Open” to verify that you can connect to this EC2 instance. Click “Yes” if the PuTTY Security Alert pops up.
18. You are authenticated via username and private key, so you do not need to enter a login password.

10. Install and configure FileZilla

FileZilla is a free SFTP client that allows you to transfer files to/from your Amazon EC2 instance.

1. Download and install FileZilla Client from http://filezilla-project.org/
2. Run the FileZilla application in Windows.
3. Select Edit, Settings from the menu.
4. On the left pane, click “SFTP” then click the “Add keyfile” button.
5. Select the C:\aws\ec2\Galactica.ppk private key file then click “Open”.
6. Select File, Site Manager from the menu.
7. Click “New Site” and enter your elastic IP address in the “Host” text box.
8. In Servertype, select “SFTP – SSH File Transfer Protocol”.
9. Leave the Logontype as “Normal” and enter “root” in the User text box.
10. Click “Connect” to verify SFTP connectivity to your EC2 instance.

11. Install the Amazon EC2 API tools on your Windows client

You will need to download your Amazon private key and cert in order to install the Amazon EC2 API tools, which are used to access additional EC2 functions (via command line or programmaticaly via the EC2 API) that are not available on the AWS Management Console web interface.

1. On your Windows machine, create the folders “C:\aws” and “C:\aws\ec2″.
2. Go to https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key
3. Click the “X.509 Certificates” tab.
4. Click “Create a New Certificate”.
5. Click “Download Private Key File” and save the pk*.pem file to C:\aws\ec2.
6. Click “Download X.509 Certificate” and save the cert*.pem file to C:\aws\ec2.
7. In Windows Explorer, make a copy of the pk*.pem file and name it “ec2-pk.pem”.
8. Also make a copy of the cert*.pem file and name it “ec2-cert.pem”.
9. Download the EC2 API Tools from http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351&categoryID=88
10. Unzip the files into the “C:\aws\ec2″ folder.
11. In Windows, click the “Start” menu, “Settings”, “Control Panel”, “System”, click “Advanced system settings”, then click “Environment Variables…”.
12. In the bottom window called “System variables”, click “New…”.
13. Enter “EC2_PRIVATE_KEY” as the Variable name, enter “c:\aws\ec2\ec2-pk.pem” as the Variable value, then click “OK”.
14. Click “New…” again in the bottom window called “System variables”.
15. Enter “EC2_CERT” as the Variable name, enter “c:\aws\ec2\ec2-cert.pem” as the Variable value, then click “OK”.
17. In the bottom window “System variables”, find and select the variable named “Path”.
    Click “Edit…”, insert “%EC2_HOME%\bin;” at the beginning of the Variable value, and click “OK”.
18. In Windows, click the “Start” menu, click “Run”, enter “cmd”, and click “OK” to open a command prompt.
19. Enter the following command to verify that the EC2 API command-line tools are working properly:
C:\ec2-describe-regions

12. Install software on your Amazon EC2 instance

• You can now install software packages on your Amazon EC2 instance. Keep in mind that although this is a virtual server, you still need to follow the license agreements that come with your software.

Optional: Terminate the Instances

• If you are just following along and no longer need these instances, you can terminate them to avoid further usage charges. However, note that all your changes will be lost if you have not yet bundled an AMI snapshot of this instance (described in Part 1.2: AWS Data).

1. Click “Instances” on the left menu.
2. Click the checkbox of the instance you want to terminate.
3. Click “Instance Actions” dropdown list and select “Terminate” to shutdown this instance.

V. Next Steps

• In Part 1.2: AWS Data, I will explore the use of Amazon S3 and EBS for cloud-based storage and Amazon SimpleDB and RDS for database applications.
• In Part 1.3: AWS Fault Tolerance, I will explore the AutoScaling and Elastic Load Balancing features of Amazon Web Services.
• Please see The Mobile Cloud Project page for more info on the other parts of this project.

VI. References

1. Amazon Web Services Technical Documentation
http://aws.amazon.com/documentation

2. Cloud Security Alliance
http://www.cloudsecurityalliance.org

3. Ubuntu Server Edition on Amazon EC2
http://www.ubuntu.com/products/whatisubuntu/serveredition/features/ec2

The Mobile Cloud Project is my proof-of-concept for developing and hosting a scalable mobile application platform on the Amazon Web Services cloud.

Part 1: Amazon Web Services

• Part 1.1: Basic AWS provides a basic overview of Amazon Web Services and describes my procedure for building Ubuntu Linux virtual servers (Reserved and On-Demand instances) on Amazon EC2.
• Part 1.2 AWS Data explores the use of Amazon S3 and EBS for cloud-based storage and Amazon SimpleDB and RDS for database applications.
• Part 1.3: AWS Fault Tolerance explores the AutoScaling and Elastic Load Balancing features of Amazon Web Services.

Part 2: Back-End Platform

• Part 2.1: Ruby on Rails Platform describes my development of a simplified Ruby on Rails platform for processing a JSON or RSS/XML input data stream and storing the contents in Amazon SimpleDB.
• Part 2.2: Python-Django Platform explores the development of the same simplified platform using Python and the Django web framework.

Part 3: Mobile Apps

• Part 3.1: iPhone Mobile App describes my development of a simple iPhone 3.0 mobile app for retrieving and displaying the JSON or RSS/XML data stream using push notification.
• Part 3.2: Android Mobile App explores the development of the same mobile app for the Android open source platform.
• Part 3.3: BlackBerry Mobile App explores the development of the same mobile app for the BlackBerry platform.
• Part 3.4 Windows Mobile App explores the development of the same mobile app for the Windows Mobile platform.

I. Overview

This article illustrates my general approach toward generating and analyzing trade ideas. I use the GLD gold ETF daily chart as an example, but this analysis can be applied to any security in any timeframe.

Many investors follow a passive long-term investing style by placing their money in mutual funds or index funds for diversification. Needless to say, these investors suffered some losses last year as stock markets plunged across North America: TSX: -40%, DJIA: -38%, S&P500: -42%.

A basic knowledge of technical analysis gives investors another tool to protect their portfolio from extreme market volatility and to take advantage of investment opportunities. The ideas of technical analysis are not difficult to learn, although their implementation is more of an art than an exact science. It does not require much time, and long-term investors can analyze charts on a daily or even weekly basis depending on their risk tolerance and time horizon. Chart analysis does not have to be expensive either; the basic daily GLD chart below was obtained for free at http://www.stockcharts.com.

In my experience, there are only three basic requirements that are essential to become successful in trading and investing: discipline, discipline, and discipline.

Finally, I am only presenting a part of my trade analysis methodology, not a comprehensive market analysis, and I am certainly not making any stock recommendations. There is a wealth of investment books, blogs, newspapers, and research journals that you can learn from, and it is up to you to apply this knowledge to your own investment portfolio.

II. Technical Analysis

There are many reasons why technical analysis works. Market psychology and the human emotions of fear and greed are the basic elements that drive all markets. The collective opinions and perspectives of market participants are reflected in stock charts, and well-known chart patterns have a tendency to repeat themselves in a self-fulfilling prophecy.

This daily chart of the GLD gold ETF was obtained for free at http://www.stockcharts.com using the default parameters. The following numbered points below correspond to the numbers on the chart above. Although this analysis pertains to a single chart, I usually look at several charts simultaneously, such as the index futures and currency charts, in order to determine relative strength or weakness.

1. Trend

As the saying goes, “The trend is your friend… until the end.” Trading with the trend gives you a higher probability of profitable trades than trading against the trend. In this chart, I look at the price and the 50 day moving average (50dma) and their relation to the 200 day moving average (200dma). Here, both the price and 50dma are above the 200dma, so the stock is in a general uptrend. However, this does not mean that I should buy now. This just means that I only look for potential buy trades and ignore any potential short trades.

I also prefer the moving averages to be pointing up in an uptrend. In this case, both the 50dma and 200dma are flat, suggesting a pause in the uptrend, a neutral outlook, or a possible reversal in the future.

If either the price or the 50dma fall below the 200dma, then I would consider this the end of the uptrend and the possible start of a new downtrend or a neutral sideways trend.

Elliott wave counts can also be useful in identifying trends and counter-trends. In my opinion, we may have just completed the end of a wave C correction, which suggests that a new bull market primary trend could resume shortly.

2. Support and Resistance

Next, I look for areas of support and resistance on the chart. When a support level gets broken, it tends to become a new resistance level. Similarly, a broken resistance becomes a new support level.

The 50dma and 200dma are very important, and I also draw trend lines between major peaks and major troughs in the chart. For intraday trades, I also look at the previous day’s high, low, and close. Fibonacci ratios are other useful levels to watch.

At point #2 on the chart, the price bounced off the 200dma in a double-bottom pattern, which is a bullish signal.

The stock price also recently broke up through the down trendline near point #4, which a bullish signal as well. This previous resistance line becomes a new support line for GLD.

3. Divergence (Trend Reversal)

Momentum indicators normally follow price movements, so the indicator should go up when the price goes up, and go down when the price goes down.

Divergence occurs when there is a conflict in direction between price versus momentum indicators. Positive divergence occurs when the stock price is falling while the momentum indicator is trending upward. This suggests that the downward price move is unsustainable due to the upward momentum, resulting in a bullish signal. Negative divergence is a bearish signal that occurs when the stock price is rising while the momentum indicator is in a downtrend.

The MACD histogram is a good momentum indicator to watch. In fact, in his book “Come into My Trading Room”, Dr. Alexander Elder regards MACD histogram divergence as the strongest signal in technical analysis.

Point #3 on the chart shows a positive divergence due to a rising trendline on the MACD histogram, while the price made a new low at Point #2. This was another bullish signal for GLD.

4. Trend Continuation

Given the signals above, Point #2 was a good buying opportunity for GLD on Apr 19. If you missed it, you can still look for other support and resistance levels to confirm the uptrend or to signal a possible trend reversal.

Point #4 shows a previous resistance line that is now a support line. If this support holds, then GLD should continue on its uptrend; if it breaks, then GLD may head back down again toward the 200dma.

Point #4 also shows a 50dma resistance level. If GLD can break though the 50dma on large volume, then it will likely head upward toward the next resistance level. If not, then GLD may head into a sideways pattern between the 50dma and 200dma range.

In this case, I would place a buy order just above the 50dma in order to participate in the trend continuation, if the price can break through the 50dma.

5. Uncertainty

If GLD can continue its uptrend, the next hurdle (Point #5) is a major psychological resistance level at $100, which corresponds to a gold price of $1,000 per ounce. GLD came very close to the $100 barrier on Jul 15, 2008 and again on Feb 29, 2009, but was unable to surpass it. If GLD can break though this resistance level, then it will achieve an all-time high with no more resistance levels above. Otherwise, GLD could bounce off the $100 resistance again and head back down.

At these points of uncertainty, it is often a good idea to take some profits or cut your losses by closing out some or all of your position. In this case, I would think about selling GLD just below $100 and buying it back after it finally breaks decisively above the $100 resistance.

III. Fundamental Analysis

Fundamental analysis is always important regardless of your trading methods or time horizon. Sharp fundamental analysts can anticipate the direction of a stock’s price before it happens (i.e. before it appears on a chart for technical analysts to analyze). You simply cannot trade in a vacuum and ignore the market and industry context surrounding your investments.

In gold’s case, there are many macroeconomic reasons to own gold: diversification, rising demand, falling supply, as a hedge against rising inflation (after the current deflationary environment), as a hard-asset hedge against financial (i.e. paper) instruments, as a hedge against falling currencies, etc. For similar reasons, China has reportedly been hoarding gold at a fast pace (“China bolsters its gold reserves” – Reuters) which suggests rising gold demand in the future.

In the case of equities, fundamental analysis includes industry analysis, financial statement analysis, ratio analysis, due diligence, etc, and is a wide-ranging subject that is clearly outside the scope of this blog. The CFA Institute and the Schweser CFA study guides present a comprehensive and concentrated amount of material that should be valuable to all investors.

IV. Risk Management and Discipline

Institutions use a variety of risk management tools for analyzing portfolio risk, pricing derivatives, and hedging risk. On the other hand, individual traders and investors can benefit from a few simple rules of thumb such as setting a maximum loss limit of 2% per trade, immediately closing a trade if the market proves that your analysis was wrong, setting stop losses, and diversification by limiting each trade to no more than 5% of your total investment capital. Although trading rules are important, having the discipline to follow them is vital to success in both trading and investing.

Finally, risks are unpredictable and they are everywhere in black swan formation. A flu pandemic, bank failure, or nuclear crisis can easily destroy the trade ideas you derived from technical and fundamental analysis. Having the proper risk management controls in place should protect your investment portfolio over the long term.

V. Conclusion

You can be the best developer / doctor / lawyer / accountant / etc in your field and still lose 40% of your hard-earned money after investing it in a reputable equity mutual fund last year. You could spend 6 months developing an excellent iPhone app that doesn’t sell because it is too complex and doesn’t generate enough popularity among the general public. You can share your awesome ideas with thousands of followers on social networks and not make a dime in online advertising. You can develop insanely great software that never sees the light of day due to economic circumstances beyond your control.

Trading and investing are some of the very few activities where people can make money directly from their ideas. Trade analysis is not difficult, time-consuming, or expensive, but it requires complete discipline when implementing your trading plan.

A basic knowledge of trade analysis techniques, which span the range of technical analysis, fundamental analysis, and risk management, is essential to everyone who earns an income and/or has money to invest. I encourage you to continue learning about these subjects to help improve your investment performance and recover any losses you may have suffered from last year’s disaster.

I. Overview

This project presents a procedure for migrating my email from a local Microsoft Outlook PST/POP3 configuration to a web-based IMAP solution using Mozilla Thunderbird and Enigmail for PGP encryption. Although this is my personal project, it can be extended to companies interested in reducing their IT costs by outsourcing their enterprise email administration and spam / malware protection to a cloud-based provider like Google Gmail.
I hope that you will benefit from this resource in your own projects.

Security

Email security has always been an important concern of mine. Over the past several years, I have stored my email locally in password-protected Outlook PST files, which are then encrypted within a PGPdisk on top of an encrypting file system in a portable USB key. I found this to be a reasonable precaution short of stopping all email use altogether (“Knuth versus Email” – Don Knuth).

Like most web mail providers, Gmail provides security in terms of TLS/SSL, which encrypts network communication between my computer and Gmail’s servers. However, the email messages themselves are not encrypted, so anyone who can crack my password can simply log into gmail.com and view all my mail. To prevent this, I configured Gmail to allow POP3 access so that my local Outlook could retrieve my email, store it in a local password-protected PST file, and delete Gmail’s copy from the server. This approach is far from foolproof, since my computer itself can be hacked by malware, botnets, and GhostNet crackers recently discovered by UofT’s Citizen Lab (“Vast Spy System Loots Computers in 103 Countries” – The New York Times).

Security is also a top concern for businesses and individuals considering cloud-based services such as Amazon EC2 or web-based storage like Microsoft SkyDrive. The network traffic may be encrypted, but the data itself is usually not, or is easily accessible after getting past a single username and password. It is often a good idea to encrypt all private data using PGP before storing it anywhere on the web. PGP adds another layer of security so that a cracker who obtains your username and password will not be able to read your encrypted data without your private key, or other employees’ encrypted data without their private keys, even when they are stored together in a common WebDAV disk. In my case, I am comfortable storing my PGP-encrypted email on Google’s Gmail servers.

Multi-Access Capability

The proliferation of netbooks and wireless devices also complicates access to locally secure email. My three servers (Vista x64, Mac OS X, Ubuntu Linux), laptop (XP), Blackberry, and iPhone cannot all access my local PST file at the same time. Even worse, each device had its own email storage containing unsecure copies of my email. A recent USB-related data corruption event also highlighted the vulnerability of my PST file itself as a single point of failure. Therefore, the Outlook PST/POP3 solution was no longer viable with regard to security and multi-access capability.

Several technologies are ideal replacements for Microsoft Outlook and POP3. Internet Message Access Protocol, or IMAP, allows synchronized access to a web mailbox from multiple devices simultaneously. Mozilla Thunderbird and Lightning are excellent open-source email and calendars clients for Windows, Mac OS X, and Linux. GnuPG is a free implementation of the OpenPGP encryption standard, and Enigmail is an OpenPGP extension for Thunderbird.

II. Consulting

I do my best to explain the concepts and techniques behind my projects. If you like my work and can use my expertise in your projects, I am available for consulting at a competitive rate.

III. Procedure

< Click here to view the detailed procedure for this project >

IV. Future Enhancements

1. PGP Encryption in Social Networks

Social networking sites are increasingly used as alternatives to traditional email, but are notorious for lax or non-existent security measures. If a cracker gets into your Facebook or LinkedIn account, they will have full access to all of your private messages. With PGP encryption, the messages in those Inbox and InMail accounts will appear as gibberish to crackers since only you possess the private key/s to decrypt them.

2. PGP Encryption in the Enterprise

Some firms have corporate policies against the use of email encryption, with the rationale that managers and executives need access to their employees’ email in certain situations. However, this policy also leaves employees’ email vulnerable to malicious administrators or crackers with the ability to intercept these messages. A better approach would be to enforce encryption on all intercompany email, with the employees’ private keys accessible to managers and executives who are authorized to decrypt and read their email under strict corporate guidelines.

3. Gmail in the Enterprise

Due to Google’s high-performance cloud network, Gmail can provide a more robust and reliable email service than many in-house corporate environments. However, Gmail still lacks some features required by enterprise users, such as LDAP authentication, private global address lists, auto-encryption via GPG/PGP keys, DNS redirection, and cloud-based antivirus. Google can consider providing its own LDAP servers or integrate into a firm’s existing LDAP servers in order to securely process email authentication for corporate users. Gmail could also partition corporate email from general-public email to create global address lists that are private to each company. Auto-encryption is a highly desirable feature because it ensures email encryption not only between intercompany users within Gmail, but also between all Gmail users. DNS redirection will enable Gmail to process incoming and outgoing messages transparently as if they were from/to user@mycompany.com instead of user@gmail.com. Finally, Google can consider bundling a cloud-based antivirus solution such as Symantec to provide enhanced virus and malware protection for email attachments at all endpoints of the Gmail cloud network.

V. References

1. PGP – Pretty Good Privacy
http://en.wikipedia.org/wiki/Pretty_Good_Privacy

2. TLS/SSL – Transport Layer Security / Secure Sockets Layer
http://en.wikipedia.org/wiki/Secure_Sockets_Layer

3. POP3 – Post Office Protocol
http://en.wikipedia.org/wiki/Post_Office_Protocol

4. IMAP – Internet Message Access Protocol
http://en.wikipedia.org/wiki/Internet_Message_Access_Protocol

5. Mozilla Thunderbird
http://www.mozillamessaging.com/en-US/thunderbird/

6. Lightning
https://addons.mozilla.org/en-US/thunderbird/addon/2313

7. Provider for Google Calendar
https://addons.mozilla.org/en-US/thunderbird/addon/4631

8. GPG – GNU Privacy Guard
http://www.gnupg.org/

9. Enigmail
http://enigmail.mozdev.org/home/index.php

10. Gmail Help
http://mail.google.com/support/

11. Microsoft SkyDrive
http://en.wikipedia.org/wiki/Windows_Live_SkyDrive

12. Amazon EC2
http://aws.amazon.com/ec2/

13. Microsoft Windows Live SkyDrive
http://skydrive.live.com/

14. Google App Engine
http://code.google.com/appengine/

15. Gmail Labs
http://gmailblog.blogspot.com/2008/06/introducing-gmail-labs.html

Over the years, I have had the privilege of meeting and working with many smart and creative individuals. From tech startups to community open-source projects to member-driven projects at HackLab.TO, it appears to me that the common thread of genius is cultivated by providing an open environment for tinkering and experimentation.

Although pure research certainly has its place in academia and R&D, it sometimes creates an air of universal legitimacy where none was intended. The Gaussian copula is an example of a financial theory from academia which was quickly adopted by risk managers due to its simplicity and research credentials, but is now claimed to be responsible for the greatest financial crisis in recent history (Wired – “The Formula That Killed Wall Street”).

On the other hand, experimentation makes no pretense of viability or legitimacy, and thus provides tinkerers with the freedom to start small, fail quickly, learn from their mistakes, and create better ideas without causing a financial disaster. Leonardo da Vinci, Nikola Tesla, Thomas Edison, Mark Twain, Mozart, Picasso, and Richard Feynman are some of the more famous genius experimentalists of our age.

Author and journalist Malcom Gladwell provides some fascinating observations about youthful prodigies versus late-blooming genius (The New Yorker – “Late Bloomers”). I would argue that this is not necessarily a function of a person’s age, but rather the number of failures, or learning experiences, they have made before finally making tremendous progress. Young prodigies found their niche early, failed quickly, and achieved early success. Late bloomers, on the other hand, experimented with different fields or careers before eventually finding their niche, then continued to fail and learn quickly in that niche before achieving success. Failure, experimentation, focus, and discipline are vital to uncovering our hidden genius.

The role of organizations in cultivating or destroying genius is not, in my opinion, the result of the organizations themselves or their bureaucracy. Rather, it is a function of how much freedom their employees are given to experiment with their ideas. In the current recessionary environment, the cost-cutting mandates and tight project-management schedules are highly detrimental to the creative employee’s ability to try new ideas that are not included in the budget or project schedule.

I conclude with the following comment I had recently written in response to a blog post (“The best of intentions”) by Unspace founder Pete Forde:
_____________________________________________________________________

Pete,

Great advice! A web startup is an excellent environment for cultivating rock star talents and nurturing genius in all of its forms.

Unfortunately, there are many smart individuals who do not have the connections, business knowledge, or financial capacity to leave a full-time job in order to start a new web development company. Instead, they continue to languish in heavily bureaucratic organizations, many of which are controlled by head offices in other countries. The opposing business and technical agendas at these firms can be highly restrictive and quite damaging to the natural talents and creativity of young prodigies. At some point, the light of genius inevitably dims and dies out completely.

I shudder to think what might have happened if one of your partners (mef) had continued working for me at one of these bureaucratic financial firms. Nevertheless, not everyone can work for a startup, and large organizations continue to hire bright people and turn them into mindless droids. What you have accomplished at Unspace should be a shining example not only for startups, but for these larger firms as well. In my opinion, if companies can provide an open environment for tinkering and experimentation by their creative employees, and the proper channels to hear and implement their ideas, we will find that geniuses are truly everywhere.

Regards,
Ray Acayan

_____________________________________________________________________

I. Overview

Part 2 of this project presents a Ruby on Rails implementation of a QuantLib web application for pricing American options. This builds on the QuantLib Ruby server created in Part 1. Click anywhere on the image below to view the web application.

II. Consulting

I do my best to explain the concepts and techniques behind my projects. If you like my work and can use my expertise in your projects, I am available for consulting at a competitive rate.

III. Procedure

1. Create the Rails Project and MVC

a.) Create the Rails Project
Using a terminal window on the QuantLib Ruby server, create the Rails project as follows:

A Rails application uses the Model-View-Controller (MVC) framework, which separates an application’s data (model), user interface (view), and business/control logic (controller) into three distinct components.

b.) Create the MySQL Database

Create a new username and 3 databases for development, test, and production, as required by the Rails framework.

Modify the database.yml file to use the following database configuration for development: test: and production:
  RayQuantLib/config/database.yml

c.) Create the Model

Create a Model named “AmericanOption” and perform the initial database migration as follows:

d.) Create the Controller and View

Create a Controller and View named “AmericanOption” as follows:

Declare this resource by adding the following line to:
  RayQuantLib/config/routes.rb

2. Create the Application Layout View

Create the following file for the XHTML application layout. This layout is displayed at the beginning of every web page in the application, and the “yield” function passes control to the appropriate Controller.

  RayQuantLib/app/views/layouts/application.html.erb

Create the CSS file referenced in application layout:
  RayQuantLib/public/stylesheets/style.css

3. Create the Input Form View

Create the input form for “Pricing American Options” in the following file. The sample code below shows some design patterns for creating a form, listbox, date, and textbox in Ruby on Rails:

  RayQuantLib/app/views/american_option/index.html.erb

Rather than building the form directly with XHTML tags, the Rails form_tag helper is used here to defend against Cross-Site Request Forgery (CSRF) attacks via the unique authenticity_token which Rails generates for each form session.

4. Process Form Input in the Controller

In a Rails application, the Controller is responsible for handling business logic and passing variables between the Model and the View. The sample Ruby code below, which is entered within the index function (def index), shows some design patterns for retrieving the values entered in the Input Form.

  RayQuantLib/app/controllers/american_option_controller.rb

The sample Ruby code below is used for input form validation:

5. Process QuantLib Calculations in the Controller

To enable QuantLib, the following lines must be entered at the top of the Controller file:
  RayQuantLib/app/controllers/american_option_controller.rb

The sample code below shows how to call QuantLib functions within Ruby on Rails. The QuantLib-SWIG interface library (installed in Part 1) allows us to call QuantLib C++ functions as if they were Ruby functions.

6. Display Controller Output to the View

The sample Ruby code below displays the output by populating the @output variable, which is passed back to the View.

The output code above makes use of the following helper function in the Controller:
  RayQuantLib/app/controllers/american_option_controller.rb

Finally, the following code displays the output in our View:

  RayQuantLib/app/views/american_option/index.html.erb

IV. Future Enhancements

1. QuantLib Functionality

Additional QuantLib features can be developed in the future, including dynamic hedging, swap valuation, bond pricing, and Monte Carlo simulation.

2. QuantLib Documentation

A more detailed explanation of the QuantLib functions and design patterns used in this project can be added in the future. In the meantime, the official QuantLib documentation can be found in the references below.

V. References

1. QuantLib Reference Manual
http://quantlib.org/reference/modules.html

2. Rails Framework Documentation
http://api.rubyonrails.org/

I. Overview

Part 1 of this project describes my procedure for installing a dedicated QuantLib web application server based on Linux and Ruby on Rails.

QuantLib is a free open-source software library for quantitative finance, financial modeling, trading, and risk management. Written in C++, this library is typically used in standalone applications at financial institutions. This project shows how the powerful features of QuantLib can be implemented in web applications. This capability allows firms to outsource their internal financial software development in favor of modern QuantLib web applications hosted by external application service providers (ASPs).

Ruby on Rails is a popular open source framework for developing web applications, and Linux is a robust and secure platform for hosting them.

II. Consulting

I do my best to explain the concepts and techniques behind my projects. If you like my work and can use my expertise in your projects, I am available for consulting at a competitive rate.

III. Procedure

1. Install LAMP Server

A basic LAMP (Linux, Apache, MySQL, PHP) server can be installed as follows:
a.) Download the latest version of Ubuntu Server (currently 9.10 Karmic as of Oct 29, 2009) from ubuntu.com.
b.) Burn the iso image onto a CD and boot the server with that CD. Install with default options.
c.) Install packages with the following commands:

2. Install Ruby on Rails

3. Install QuantLib

a.) The Boost C++ libraries must be downloaded and installed first:

b.) Download and install the latest version of QuantLib:

IV. Future Enhancements

In Part 2 of this project, I will use this QuantLib Ruby platform to explore some simple design patterns used in developing QuantLib web applications for quantitative finance.

V. References

1. Ubuntu.com
http://www.ubuntu.com/

2. RubyOnRails.org
http://rubyonrails.org/download/

3. QuantLib.org
http://quantlib.org/extensions.shtml

4. SWIG
http://swig.sourceforge.net/

I. Overview

This project simulates a real-time GPS map of several TTC streetcars in downtown Toronto. I created this simulation to explore the Google Maps API, animation techniques, geocoding, and external custom maps. It can be extended to other applications such as real-time mapping of certain GPS-enabled mobile devices. Click anywhere on the map to view the simulation. Feel free to zoom in/out, drag the map, and select Hybrid mode.

II. Consulting

I do my best to explain the concepts and techniques behind my projects. If you like my work and can use my expertise in your projects, I am available for consulting at a competitive rate.

III. Components

1. The TTC Transit Map Component

This external component is a cool custom map created by Ian Stevens which shows the entire TTC transit routes overlayed onto Google Maps. The original version can be found here. I use this map to display the routes taken by the animated TTC streetcars.

2. Streetcar Initialization Component

This component creates an array of streetcar objects for each streetcar route. The number of streetcars currently in the route is a function of the frequency of arrival at each station.

2. Route Initialization Component

This component creates a route for each streetcar line. The GDirections() function is used to calculate the route between the start and end points, and the resulting line is retrieved using the getPolyline() function.

The loadFromWaypoints() function calls the “load” event which is handled by the following listener:

3. Position Update Component

This component updates the current position of each streetcar and displays it on the map. The animate() function is initially called from the “load” listener, then calls itself recursively using setTimeout() after each polling interval (tick).

The GetPointAtDistance() function is provided by a Google Maps extension called “epoly.js” which was developed by Mike Williams. More info can be found here. This function returns point “p” on the route which is a distance “d” meters away from the starting point. This point is then used to update the streetcar’s current position (marker) using the setPoint() function.

III. Future Enhancements

1. Real-time GPS Coordinates

In this simulation, the streetcar’s position is updated by 2 meters every 500 milliseconds. A real application would update the current position by retrieving real-time GPS coordinates and use a more infrequent polling interval such as 60 seconds. At this time, it is unknown if the TTC will provide real-time GPS coordinates to the public.

2. Mobile Devices

It would be cool if someone could develop a real-time TTC streetcar map for the iPhone and other mobile devices. Route schedules like Red Rocket are nice, but a real-time map can reveal better routes based on current streetcar positions and can significantly reduce waiting times at the streetcar stop.

3. Other Route Algorithms

The GDirections route calculator provided by the Google Maps API is based on least-cost-routing using street maps. A different algorithm could provide a complete polyLine specifying all the waypoints in the route. For example, we could display a plane’s flight path, which does not follow the directions on a road map. This enables new applications such as creating a real-time air traffic control map showing the GPS positions and routes of all planes in the sky. The possibilities are endless.

IV. References

1. Google Maps API Developer’s Guide
http://code.google.com/apis/maps/documentation/index.html

2. Google Mapis API Reference
http://code.google.com/apis/maps/documentation/reference.html

I. Overview

This is a simple project for displaying an image overlay whenever the user moves the mouse over an element on the web page. This effect is demonstrated on my website in the menu items above, the scrolling stock ticker, and the icons to the right of my photo.

This feature is useful for displaying value-added information quickly on the same web page without requiring the user to click the mouse, find the information on a different page, then click back to the original page. It is also helpful for mobile devices with limited screen space, where small icons representing web sites can be enlarged as the user moves the mouse over them.

II. Source Code

I do my best to explain the concepts and techniques behind my projects, but I cannot provide the complete source code. If you like my work and can use my expertise in your projects, I am available for consulting at a very competitive rate.

III. Components

1. Mouse Position Component

This component captures the mouse cursor’s current position, which is later used to determine the screen position for displaying the image overlay. It uses <div< elements for the X and Y position of the mouse, then updates those variables as follows:

The getCursorXY() function uses the value of document.body.scrollLeft and document.bodyscrollTop to determine the mouse cursor’s X and Y position on the web page.

2. Event Assignment Component

This component is a set of JavaScript functions for assigning events to trigger elements such as icons or text, which are assumed to be in basic form, such as an <img> or <a>. It finds a trigger element and encapsulates it in a <div> or <span> tag containing mouseover, mousemove, and mouseout events pointing to the corresponding JavaScript functions. For example:

3. Image Display Component

This component is a set of JavaScript functions which respond to the mouse events and display the image overlay. The onmouseover() event calls a function which dynamically creates an <img> element from a file as follows:

The mousemove() function moves the chart based on the mouse cursor’s current position. The onmouseout() function hides the chart by setting the imgbox visibility to “hidden”

IV. Future Enhancements

1. Dynamic Image Caching

I do not currently implement caching on this website. Consequently, there is an intermittent delay when retrieving images as you move the mouse over certain items such as the scrolling stock ticker. A cache can be developed to automatically create image snapshots of this information and display the cached image instead.

2. Other Types of Overlay

The overlay effect can enhanced to display dynamic content types such as dynamic stock charts, video surveillance cameras, and real-time GPS maps, rather than just static images. The possibilities are endless.

I. Overview

The scrolling stock ticker above is a simple example of a lightweight component for retrieving real-time data and displaying it dynamically in a web browser. Many stock tickers today are based on heavy Java applets, which tend to slow down my computer considerably. Other tickers based on JavaScript and Ajax do not have a scrolling feature. The basic marquee scrollers do not retrieve real-time data. So, I decided to develop my own simple scrolling stock ticker which combines all of these functions.

II. Consulting

I do my best to explain the concepts and techniques behind my projects. If you like my work and can use my expertise in your projects, I am available for consulting at a competitive rate.

III. Components

1. Stock Data Retrieval Component

This component is a group of PHP functions for querying a web data source, retrieving the resulting data, and storing the result in a DOM document object. These functions are located in a standalone PHP file.

First, the stock quotes are retrieved using the @fopen() file pointer to the following url:

The Yahoo Finance site will return a comma-delimited CSV file based on the query parameters. More info on Yahoo’s data protocol can be found here. Other web services return data in RSS, JSON, or XML formats, which will require different query and parsing methods.

Next, the CSV-formatted results are parsed by the @fgetcsv() function, which stores the comma-delimited values into an array. The array values are then placed into <span> elements and returned in the following format:

2. CSS Display Component

Not exactly a “component”, this is simply a CSS file which contains formatting for the <span> data above. For example:

3. Ajax Stock Update Component

This component is a group of JavaScript functions for dynamically updating the stock quotes using Ajax. Ajax is a group of web techniques for updating a web page without the user having to press the Refresh button.

The function uses the http_request.onreadystatechange property to call the Stock Data Retrieval Component (i.e. the standalone PHP file) to retrieve a new set of <span> elements containing the newly-updated stock quotes. The document object is then updated dynamically by replacing the <span> contents of the <div id=”RayStockTickerID”> node as follows:

4. Scrolling Marquee Component

This component is a group of JavaScript functions which retrieve the stock quotes from the DOM document object, display the css-formatted stock quotes in a line of text, then scroll the text from right to left one pixel at a time with the setTimeout() function.

As each stock quote falls off the left of the screen, its <span> node is removed and appended to the end of the chain to create a continuous loop. At the end of each loop, the stock quotes are updated dynamically by the Ajax Stock Update Component.

5. Chart Display Component

When you move the mouse to a particular stock in the ticker, this component will display an overlay of the corresponding daily stock chart. This is accomplished using onmouseover() event handlers within each stock’s <span> tag. The onmouseover() event calls a function which dynamically creates an <img> element and retrieves the chart from Yahoo Finance as follows:

The mousemove() function moves the chart based on the mouse cursor’s current position. The onmouseout() function hides the chart by setting the imgbox visibility to ‘hidden’.

6. Main Component

The main component is a a section of PHP code in the main website which creates all the components above and initiates the program.

IV. Future Enhancements

A. Different Input Formats

As mentioned above, the input data streams may be formatted in RSS, JSON, XML, etc. Functions can be developed for parsing and querying these other data protocols.

B. Caching

This simple project does not implement caching since it does not use previous stock data. A cache can be developed to store data into an external database or text file before displaying it on the screen.

C. Data Processing

The incoming data can be processed and enriched before displaying to the web page. For example, the stock ticker could display only those stocks which have crossed their 50-day moving average, or those that have exceeded their daily VaR limits within a portfolio.

D. Different Output Formats

Instead of a simple scrolling ticker display, this app can be modified to create an output data stream in other formats like RSS, JSON, XML, database, e-mail, etc, or other platforms such as Facebook, Twitter, and mobile devices.

Our network is our most valuable asset. Unlike other assets such as stocks and real estate, which tend to grow at a constant rate over a very long period of time, the value of our network grows exponentially as its size increases. Not only does it provide value to ourselves, but we in turn create value for each individual in our network. This value can be unlocked in monetary terms through business development, customer acquisition, or career development, as well as non-monetary terms through friendships, referrals, mentoring, and free advice.

To encourage myself and fellow Torontonians to participate in increasing the value of our networks, I have compiled the following list of technology and business networking events in Toronto. It is by no means comprehensive, but let me know if you find other interesting events and I will add them to this list.

Technology Networking Events

Weekly/Monthly Events:

Ad-hoc Events:

Annual Events:

Other Event Calendars:

Business Networking Events

Other Groups:

Fixing the Resume

Setting aside my valid concerns about privacy, spam, and identity theft, I am posting my resume online for the whole world to see. It’s a social experiment to determine the value of social networks in improving the lives of its individual members.

Is it possible to create the perfect resume using the collective wisdom of social networks? Can Twitterers (Tweeps) produce a more concise but slightly unreadable resume, FTW?
Do Facebook friendships provide more networking potential than professional networks like LinkedIn? Is it really all about who you know, rather than what you know?

Feel free to post a (moderated) comment about my resume, and I’ll modify it according to your suggestions. You can find it here: www.RayAcayan.com/blog/Resume

Landing the Job

Are social networks more effective at connecting employers and job-seekers through the hidden job market, than traditional sites like Monster and Workopolis? (More about my latest venture, Social Job Market Inc., in a future blog.) If you like my resume and know someone who needs a senior professional in technology management or quantitative finance in Toronto, please contact me at RayAcayan@gmail.com.

This is my first blog.

After many grueling years of hard work and study, I finally have some free time to reflect on what has gone before and to prepare for what lies ahead. Actually, I’m just extremely bored.

First, a little bit about myself. I’m a recent Rotman MBA grad with a triple-major in Investment Banking, Funds Management, and Risk Management & Financial Engineering. I have worked in Information Technology for my entire career, and I dabble in hedge funds, proprietary trading, real estate, political analysis, web development, social media, private equity, and network security. I’m also studying for the CFA and PMP certifications. You could say that I am “A Geek of All Trades”.

In June 2008, I was laid off from my full-time job after 11 years in the financial industry. Shortly afterward, I graduated with an MBA from the Rotman School of Management (University of Toronto) and proceeded straight to semi-retirement. There were no jobs available as the financial industry collapsed, and we are now headed toward the worst economic conditions since the Great Depression.

So what happens now?

I have started looking for a full-time job, preferably in technology management or quantitative finance. If you have any leads, please e-mail me at RayAcayan@gmail.com. Thanks in advance for all your help. In the meantime, I continue to support my wife and 3-year-old son with some gold investments, left-over severance pay, and limited government assistance.

The job-hunting experience can be extremely boring, and I would rather do something more productive and somewhat entertaining. Hence, the blog. I’ll write about some interesting news, events, and insights that I come across in my adventures as a geek of all trades.