Build and Manage EC2 API Clouds

Oracle on Amazon EC2

2012-02-15T02:11:00.015-08:00

Oracle have pre-installed images on Amazon EC2. The following are steps to get a database up and running using the pre-installed image.

1. Bring up server using ami-8d97bcf9 - Oracle 11.2 Standard Edition EBS backed.
(See Reference 1 below)
a. attach 2 EBS disks, one for the database and one for the flash recovery. I
attach at /dev/sdf and /dev/sdg respectively.
b. logon to server via ssh and reply to meesage up to where it asks to create a database.
c. logon to server via another ssh session and initialise the EBS disks:


       mkfs -t ext3 /dev/sdf
       tune2fs -m0 /dev/sdf
       mkdir /u02
       mount /dev/sdf /u02
       mkfs -t ext3 /dev/sdg
       tune2fs -m0 /dev/sdg
       mkdir /u03
       mount /dev/sdg /u03

d. Back to the first orginal ssh session and continue to create the database
use /dev/sdf for database and mount on /u02
and /dev/sdg for flash recovery and mount on /u03
2. Enable SCP so we can transfer in files:

vi /etc/ssh/sshd_config

uncomment line:

Subsystem sftp /usr/libexe/openssh/sftp-server

restart ssh:

/etc/init.d/sshd restart

3. To build the automation scripts need Ruby 1.8.7 (required for Amazon interface
right_aws) need to upgrade from the supplied Ruby 1.8.5.
a. Setup Oracle public repository as none defined. (See Reference 2 below)


       cd  /etc/yum.repos.d/
       wget http://public-yum.oracle.com/public-yum-el5.repo
       vi /etc/yum.repos.d/public-yum-el5.repo and set 'enabled=0' parameter to
       'enabled=1' for the Oracle Linux Version 5.4 repository

b. Setup epel repository (See Reference 3 below)

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

c. Install pre-requsites for Ruby 1.8.7 install (See Reference 4 below)


       yum install zlib-devel
       yum install openssl-devel

d. Install Ruby 1.8.7 from source.


       cd /usr/local/src
       wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p357.tar.gz
       tar xvzf ruby-1.8.7*.tar.gz
       cd ruby-1.8.7*
       ./configure --enable-shared --enable-pthread
       make
       make install
       cd ext/zlib
       ruby extconf.rb --with-zlib-include=/usr/include --with-zlib-lib=/usr/lib64
       cd ../../
       make
       make install
       cd ext/openssl
       ruby extconf.rb
       make
       make install
       cd ../../
       ruby -v

e. Make Ruby 1.8.7 default ruby implementation so can be accessed at startup


       mv /usr/bin/ruby /usr/bin/ruby_old 
       ln -s /usr/local/src/ruby-1.8.7-p357/ruby /usr/bin/ruby

f. Install Ruby Gems


       cd /usr/local/src
       wget http://rubyforge.org/frs/download.php/75711/rubygems-1.8.15.tgz
       tar xvzf rubygem*.tgz
       cd rubygem*
       ruby setup.rb
       gem install rubygems-update
       update_rubygems

g. Install right_aws the Ruby Amazon AWS Interface

gem install right_aws

4. Setup Automation Scripts
a. Use scp to copy
cloud_init.rb
settings.rb
config_listener.rb
to /root
NOTE: Verify in cloud_init.rb the lines for ubuntu are commented out and the
lines for centos/redhat are uncommented
b. Go to directory /etc/rc.d/init.d and add a file called cloud_init containing:


       #!/bin/sh
       #
       # chkconfig: - 98 01
       # description: initialize cloud requirements for application
       # NOTE: parameter on config_listener.rb is ORACLE_HOME
       #
       ruby /root/cloud_init.rb
       ruby /root/config_listener.rb /u01/app/oracle/product/11.2.0/db_1

c. Don't forget to make it executable

chmod a+x cloud_init

d. Setup the cloud_in to run at startup. It is level 98 because Oracle starts at
level 99.

chkconfig --level 345 cloud_init on

5. Create the oracle image:
a. shutdown the database and listener (under user oracle)


        su - oracle        
        /u01/app/oracle/product/11.2.0/db_1/bin/dbshut
        lsnrctl stop

b. Update the listener config file so the config_listener.ora script works at startup

vi /u01/app/oracle/product/11.2.0/db_1/network/admin/listener.ora

change 'HOST = ip.999.999.999.999' to 'HOST = '
NOTE: ip.999.999.999.999 will be actual ip address of server
c. Update the oracle user bash profile so that it can also be updated by the
startup script to the correct address

 vi /home/oracle/.bash_profile

change

export ORACLE_HOSTNAME=ec2-999-999-999-999.eu-west-1.compute.amazonaws.com

export ORACLE_HOSTNAME=

NOTE: ec2-999-999-999-999 will be actual ip address of server
d. Go back to the root user and umount and detach the EBS disks.


        umount /u02
        umount /u03
        detach using the AWS console or other tools.
        rm -r /u02
        rm -r /u03

e. Create an image called oracle-YYMMDD where YYYMMDD is current date

References
1. http://databasesincloud.wordpress.com/2012/01/25/desaster-recovery-in-the-cloud-part-2/
2. http://www.oracle.com/technetwork/topics/linux/yum-repository-setup-085606.html
3. http://gembuls.wordpress.com/2011/02/12/how-to-install-epel-repository-on-centos/
4. http://www.xelacomp.com/blog/02-09-2011/install-ruby-187-rubygems-and-passenger-centos5-64-bit

Scripts can be found at
settings.rb
Cloud_init.rb
config_listener.rb

Hybrid Cloud Management

2012-02-05T05:44:00.001-08:00

The future is definitely hybrid clouds.

Just like when shopping, you want to be able to buy from the best store.

With rental cars, you want to use you own car sometimes and at other times use a rental car (like when you are interstate or your car is being repaired).

Similarly companies will want to use different public clouds for redundancy or to take advantage of different features or pricing. Also they may want to use one-site servers, i.e. a private cloud, as well as public clouds.

This has implication for cloud management. You need an approach that spans multiple public clouds and can work with your private cloud internal servers. So the tools that come with the cloud provider will over time give way to tools that span multiple clouds.

EC2Dream is the only free tool that can work with any EC2 API cloud. It works with the Amazon and OpenStack Cloud.com public clouds as well as Eucalyptus and OpenStack Cloud.com private clouds.

You also see the commercial cloud management companies adopting the same approaches:
RightScale with the myCloud initiative.
enStratus provides on-premise or SaaS self-service management

You can also look at:
Cloudkick
Scalr
Kaavo
Tapin Systems

References
Cloud Management
Multi-Language, Multi-Framework, what about Multi-Cloud?

Projects

2012-02-05T05:43:00.001-08:00

CLOUD PROJECTS

Future Route
Migrated the CreditPal application to Amazon AWS. This saved money compared to previous hosted solution. This is a multi-tiered Java and MySql application.

My1Login
Migrated the my1Logon website to Amazon EC2 so that they could respond to growth quickly. This is a PHP and MySQL application.

British Medical Journals
Setup Disaster Recovery on Amazon AWS for the BMJ website and applications. This is a multi-tier Java and Oracle application.

GRID PROJECTS

Credit Suisse
Migrated their datasynapse applications to version 5.1. Further project to migrate to Platform Symphony.

DynamoDB - Data as a Service

2012-01-21T04:01:00.000-08:00

In Data as a Service we now have a solution to the cloud database scalability problem and we could be ushering in the era of cloud applications!!

Within the next few years a Data as a Service (DaaS) capability will become central to Cloud Computing offerings. DynamoDB data service can be configured to scale up to very large transaction rates and database sizes with consistent performance. Amazon competitors will need to have an offering (maybe using Cassandra?).

The future of cloud applications could be:
1. Define and store data in Data as a Service like DynamoDB.
2. Run the the application in a PaaS like cloud foundry.
3. Use analytics applications like Hadoop (like Amazon Elastic Map Reduce) for Analytics directly on the Data as a service.

The could mean:
1. Theoretically infinite scalability of applications
2. Just paying for the amount of infrastructure you need
3. Radically reduced setup time for applications
4. Reduced number of Operations staff to manage the database....

Currently many applications are build use Object relational mapping through software like Hibernate. Hibernate is developing NoSQL support so these applications could be ported onto Data as a Service platforms. Although I am sure developers will need to make coding changes to be able to use the power of these Data as a Service offerings.

Will we see packaged products that use Relation Databases (e.g. Content Management Systems) move to NoSQL implementations? Alfresco seems to looking into it.

When we start to see major packages like e-commerce packages like Magento, Content Management Systems like Alfresco support NoSQL Data as a service like DynamoDB we will know we are on the way to the new era of cloud applications....

It is no coincidence that Oracle have announced a NoSQL capability too!!!

References
Amazon DynamoDB internet scale data storage the nosql way
PaaS Data
Next Iteration of PaaS - will amazon join that race
Should Nosql startups be afraid of DynamoDB?
First look: Oracle NoSQL Database

PaaS will become Custom SaaS

2011-10-17T02:29:00.000-07:00

The SaaS marketplace is following the same path as the Enterprise Software market of 1980s and 1990s. First applications were sold are "turnkey" solutions and vendors kept the source code to themselves as the "secret sauce". Ofcourse customers wanted to customize them so application were developed so they can be configured and the more successful applications became platforms that the application was built on so that they could be changed more easily.

Salesforce.com is trying to create a customization platform via Force.com etc.

Other (smaller) SaaS vendors need to move their applications to standard stacks:
- Java
- PHP/Perl
- Ruby
-.NET
and get them running under a PaaS product like Cloud Foundry.

Then they will have a basis for re-architecting their product to make it easier to customize by:
- integrating with RAD tools like Eclipse and Source Control like GIT.
- Perhaps developing a core part of the application that are services that are not customized and an outer core that can be customized.
- Writing the documentation that developers need to customize the product.
- Perhaps running it on there own-labeled Cloud Foundry implementation on Amazon AWS.

Surely, Enterprise software vendors will also move their products to standard stacks in a PaaS product to more effectively compete with the Saas product vendors.

Products like SaaSgrid that are proprietary tools for building SaaS application are repositioning a PaaS. SaaGrid is now Apprenda Platform as a Serice for the .NET platform.

We will know this is starting to happen when PaaS vendors announce support for applications rather that just system software products and start adding more features for developing SaaS applications.

The PaaS market is getting crowded so this is an indication of a market starting to develop. For Java alone there a dozen participants:

   Amazon Elastic Beanstalk
   CloudBees Run@Cloud
   Cumulogic
   Google App Engine
   IBM SmartCloud Application Services
   Microsoft Azure
   Oracle Public Cloud
   OutSystems Agile Platform
   Red Hat OpenShift
   Salesforce.com Heroku for Java
   VMWare CloudFoundry
   WSO2 Stratos/StratosLive

References
Custom SaaS Calls for RAD PaaS

Free Trial of Open Source Applications

2011-06-14T05:26:00.000-07:00

Do you want a free way to trial or teach yourself an application without the hassle of installation and setup? Well, combining:
AWS Free Tier and
BitNami Free Developer Plan
you can run a micro instance on Amazon EC2 for a year and try out dozens of well known open source applications like SugarCRM, Magento etc etc.

Just sign up for a AWS Free Tier and the BitNami Free Developer Plan

In the future more "free" micro instances of applications will appear on Amazon EC2, even proprietary products. It is an easy way for vendors to offer trials of their applications and a good way to teach yourself an application. Ofcourse, once trialled the application could be installed inside the company firewall, on any hosting or cloud site, or even Amazon EC2.

Every University Student of Computer Science, Business Studies etc should be using it. You could run the first year of an online e-commerce startup for free with Magento.

The Cloud Market is a good place for looking for Amazon Images. Maybe in the future they can list trial "micro instance" applications and allow you to automatically run them. Perhaps get some sponsorship from companies or provide feedback from trials of applications.

In the future we will see a tremendous number of free tools and offers around cloud computing. For example there is now a free VM to Cloud Calculator

CloudFoundry now have a Micro Instance so you can run a complete instance of Cloud Foundry on your own computer. This is a great way to try out Cloud Foundry for Free!!! In the future there will be apps available for cloud Foundry (Cloud Foundry App Store anyone?) and this this will be a great way to try out apps as well....

Other Free Offers
Windows Azure - 30 days free
Force.com - 60 days free
Cloud Foundry - Free Signup
Cloud Foundry + RightScale = Fast Multi-Cloud Onramp - Free Signup
Free cloud computing applications

Network limitations in Amazon EC2

2011-06-10T07:13:00.000-07:00

The limiting factor in the Amazon EC2 cloud seems to currently be the internal network. James Hamilton let the cat out of the bag in his latest presentation where he states that networking costs are 8% of a datacentre and rising while servers are 57% and falling. He saids:
  1. Current networks over-subscribed
  2. Forces workload placement restrictions
  3. Goal: all points in datacenter equidistant

When Amazon can push the whole of the Amazon EC2 internal network to 10 Gbit/s and make it more consistent by having a flatter network or add a proper SAN capability then Amazon EC2 will really start to hum for enterprises.

Currently Elastic Block Store Dervices (EBS) response can be variable and you need to be careful doing snapshots and backups to make sure they are consistent. Currently the Block Storage devices don't give the same performance that enterprises have been getting from (more expensive) SANs.

There are some new development to provide network virtualization using OpenFlow that could be promising in the long term but it will be a number of years before these bear fruit.

References
Today’s EC2/EBS Outage: Lessons learned
Honesty Box: EBS Performance Revisited
Improved EBS snapshots on Amazon EC2
Amazon EC2 I/O Performance: Local Ephemeral Disks vs. RAID 0 Striped EBS Volumes
Gluster Virtual Storage Appliance for Amazon Web Services
Playing with NFS & GlusterFS on Amazon cc1.4xlarge EC2 instance types
OpenFlow - Learn More

Cloud Foundry - Open PaaS

2011-04-13T02:28:00.000-07:00

VMware's Cloud Foundry is enormously significant. Like other PaaS products it removes the integration issues for middleware and automates deployments giving developers and admin significant productivity improvements.

If IaaS improved the productivity of the Data Centre "Platform as a Service" is going to improve developer and admin productivity.

But most other PaaS Products are proprietary and a lock-in to the vendor's platform. Hardly an attractive prospect no matter what the productivity improvement. Cloud Foundry works with multiple clouds, frameworks and services with the source code available to be understood, enhanced, modified or extended.
Look out for the eco-system of products and partners that will be build on this product. When the major open source application products like SugarCRM, Alfresco, Magnento etal supply versions that run on Cloud Fountry then this could really take-off and all the public clouds will have to offer the Cloud Foundry service.

If you look at the source you will see that alot of it is Ruby confirming my belief that Ruby will be important in building Cloud Infrastructure. EC2Dream is written 100% in Ruby. CloudFoundry has a command line client written in Ruby!! Also internally the client uses an http and JSON API to communicate with the cloud. Amazon Cloud Formation also uses http and JSON...

Cloud Foundry will be supplying a free Cloud Foundry Micro Cloud. Basically a downloaded VMware image for Cloud Foundry that can be run under VMware Player or Fusion (on Mac). This means you can build and unit test your application on your Desktop and then deploy to a Private or Public Cloud.

Interesting Redhat have released OpenShift which is similar to Cloud Foundry. It uses a Ruby-based Client Comand Line Interface. They have not open sourced the code (yet) and it will be initially available on Amazon EC2. Also they use git for source control while Cloud Foundry did not want to force users to use git.

In summary Proprietary PaaS, that is language specific (e.g. Ruby Heroku) or SaaS specific (e.g. VMForce), is not the future of PaaS. Open PaaS that supports a broad range of technologies and can also be deployed by Enterprises inside their firewalls is the future.....VMWare is definitely after enterprises with this technology. Within a few years enterprises will all be adopting this "deploy and scaling" approach to application execution.

Now there is a .NET implementation called Iron Foundry. Cloud Foundry now supports the most environments of any PaaS product!!

References
VMware Cloud Foundry: A Cloud Primer
PaaS Is The Future Of Cloud Services: Battle Lines Are Clearly Drawn
CloudFoundry Blogs, Reviews, Articles
VMware CloudFoundry: Ruby powered PaaS
VMware’s CloudFoundry and Red Hat’s OpenShift – Compare and Contrast
Try CloudFoundry on an AWS Micro instance
CloudFoundry Architecture - Part1
CloudFoundry Architecture - Part2
CloudFoundry DeepDive
From Righscale's blog
Slideshare
Cloud Foundry Live Webinar
Monitor your Rails app on VMWare’s Cloud Foundry
Monitoring VMware’s Cloud Foundry
DotCloud

Scaling Cloud Databases

2011-03-18T08:42:00.000-07:00

To scale applications it has become accepted practice to introduce a load balancer in front application servers and have clients access the applications via the load balancer. With Amazon EC2 it is very simple to hook up your applications to the Load Balancer and Auto Scaling Services. Traditionally scaling SQL databases is expensive and calls for "scaling up" to bigger servers and clusters of servers. Then there are the redundancy and failover requirements to consider!

Currently there are a number of new approaches to scaling databases and in the next few years we will see which approaches become accepted in cloud computing. My guess is that Amazon RDS will introduces improved scaling of the database. Currently, dbShards and Xeround both have solutions running on EC2. Scalebase even works with Amazon RDS so it would be pretty easy to make Scalebase part of the Amazon RDS offering.

Scale-Out SQL Databases
For SQL databases there are a large number of new products to scale out databases horizontally. Many of these are shared nothing databases that use sharding to partition a database.

    ScaleBase - A Database Load Balancer for your existing server (MySQL, Oracle Database, Microsoft SQL Server or IBM DB2).
    Xeround - MySQL Storage Engine Architecture to implement in-memory database utilizing NoSQL concepts.
    dbShards - Application-Aware Sharding.
    TransLattice - more of an distributed application platform, but it contains a distributed relational SQL database that supports eventual consistency.
    NimbusDB - Scalable SQL transactional database.
    VoltDB - in-memory shared-nothing system. Stored procedures become the unit of transaction.
    ScaleDB - Engine for MySQL using shared-disk architecture rather than shared nothing.
    Drizzle - Leaner, faster version of MySQL removing a host of non-core features (including views, triggers, prepared statements, stored procedures, query cache, ACL, and a number of data types).

Sharding is like a load balancer for the data. A big part of sharding is picking a good partitioning scheme.
1. Vertical partitioning. (data related to a specific feature is stored on the same machines)
2. Key-based partitioning.
3. Directory-based partitioning.

NoSQL databases
Lightweight, open source relational databases that do not expose a SQL interface. These are diverse products and can be key value, Document-oriented, or Column-family stores.
    CouchDB
    Amazon SimpleDB
    Cassandra
    MongoDB
and more....
Designed for high availabiliy and eventual consistency rather than guaranteed ACID (atomicity, consistency, isolation, durability) database transactions like SQL databases.

Cache/In Memory Data Grids
A number of commercial and open source products cache Key/Object Stores. Often fronting SQL databases and using the JCache API.
    Coherence
    Gemfire
    Gigaspaces
    Ehcache/Terracotta
    MemCached. This was often used with MySQL

Summary
Given the prevalence of SQL databases it is hard to imagine non-SQL databases becoming the common database approach. More likely "scale out" SQL databases with front-end caches will become the next architecture. Although as application code moves more to Object Relational Mapping (using products like hibernate) NoSQL databases become more viable.

References
Scaling Databases on the cloud
Are cloud-based memory architectures the next big thing
Distributed Caching Using the JCACHE API and ehcache
The Workday architecture — a new kind of OLTP software stack

Amazon CloudFormation

2011-02-25T05:00:00.000-08:00

Amazon CloudFormation is a new way of provisioning software stacks in Amazon EC2. Instead of issuing commands (or running a script) against the different Amazon AWS services you just specify a template JSON file with all your requirements. For example the sample WordPress Template.

Simple CloudFormation API
Once a Template "Stack" is defined you can use the CloudFormation API command CreateStack, passing runtime parameters to the template, to start the Software Stack. Similarly there is a DeleteStack command plus several other commands to display events etc. This makes it a very simple API to implement and will be very useful for tool builders and customers to run Software Stacks in Amazon.

Software Stack = Machine Image(s) + Template + Runtime Parameters
Running Software Stacks becomes:
1. Create Amazon Machine Image(s) with the base application software.
2. Create a Template defining the software stack
3. Run the stack passing configuration parameters at runtime and dynamic parts of the application (like code) via elastic beanstalk.
NOTE: This means Amazon Machine Images should not contain configuration parameters or application code (WAR files etc)

In the future this could become a simple standard for various cloud vendors to provide some interoperability and portability between clouds.

Chef
Ofcourse for bigger and complex systems there is still a role for Chef to build servers. CloudFormation just controls the Amazon AWS resources. A close integration of Chef and CloudFormation would be very desirable. Because they share the same JSON format it could be very consistent. For now it is easy to pass the chef information to a bootstrap script running on the server.

References
Create Your AWS Stack From a Recipe
AWS CloudFormation in the AWS Management Console
Integrating AWS CloudFormation and Chef
The Impact of Amazon’s new CloudFormation service

Amazon Elastic Beanstalk

2011-01-19T02:04:00.000-08:00

Amazon have finally annouced their Platform as a Service offering - Elastic Beanstalk so it looks like 2011 will be the year of PaaS.

The first version provides a Tomcat platform that can deploy and run your WAR file on a automated scalable architecture that includes all the load balancing, auto scaling, DNS, monitoring and notification. This is an ideal companion to Database as a Service Amazon RDS.

Containers and Custom Amazon Machine Images
What is really important is Amazon have the concept of an Elastic Beanstalk Container. When you create your application you choose your Solution Stack. Currently there are only 2 stacks:
            64bit Amazon Linux running Tomcat 6
            32bit Amazon Linux running Tomcat 6

What is powerful is you can use Elastic BeanStalk with a custom Amazom Machine Image. All you need to do is first bring up the Elastic BeanStalk AMI as a regular EC2 server and customize it before using with Elastic Beanstalk. This means you can use Memcached, Terracotta, Nginx, Jetty etc. We will see public custom AMIs appear for the various software stacks just like for EC2 Servers.

Flexibility and Portability
Early PaaS products lacked flexibility to tailor the components - often applications have all sorts of custom requirements (caching, third-party tools, version requirements etc). Being able to tailor Elastic BeanStalk AMIs makes it very flexible!! However to really take off PaaS needs to be Portable, so if some company could create an Elastic Beanstalk for Eucalyptus and OpenStack.

Production Application = Infrastructure + Product + Configuration + Data
Installation of a Production Application will become little more than:
    1. Running the product install script on the Elastic Beanastalk AMI and doing a little configuration and saving the new Application AMI.
    2. Setting up the data requirements for the application (for example creating a database).
    3. Starting the Application AMI in Elastic Beanstalk and deploying any application code.
This means it is incredibly easy to setup production java-based applications like Alfresco. When a PHP version of Elastic Beanstalk is available then it will really open the floodgates to applications like SugarCRM, Magento etc etc.

Elastic Beanstalk looks like an ideal platform for building Restful or SOA (Service Orientated Architecture) Web Services. See also Soap Vs REST

Many larger enterprises have already reduced costs by consolidating their data centers and virtualizing their servers. But I think we will see adoption of public and internal clouds increase when mature, flexible, and portable PaaS products are available. A number of major vendors are developing PaaS platforms:
    VMware vFabric
    Redhat PaaS Makara
    Windows Azure
Amazon have set the bar for the cost of PaaS. It is a free part of a Cloud Offering! Free always helps adoption and there is tremendous potential here for the future!!!!

References
Amazon Elastic Beanstalk
Amazon Elastic Beanstalk Documentation
How to customize an Amazon Elastic Beanstalk instance
AWS Elastic Beanstalk
PaaS Strategy: Sell the Condiments, Not the Sandwiches
How SteamCannon Compares to AWS Elastic Beanstalk
Elastic Beanstalk a service of services
JRuby on Rails on Amazon Elastic Beanstalk
PaaS shouldn’t be built in Silos
Platform as a Service: The Next Generation Application Server?
Is PaaS Just Outsourced Application Server Platforms?

Developing Chef Cookbooks

2010-12-22T04:57:00.001-08:00

Chef is an open source framework for configuration management that is rapidly gainng popularity in Cloud Computing. Chef consists of:
    Chef Server - Contains all the Configurations called     cookbooks.
    Chef Client - Runs on your server and performs the     configuration.

The architecture is that the chef client "pulls" its configuration from the chef server and runs the cookbook to configure an application. This is good for production but for testing it is better to "push" out the configuration directly to your server, run it and see the result.

Fortunately Chef also have an option called Chef Solo that is a chef client that can be installed on your server and does not require the Chef Server. This allows the configuration to be "pushed" directly to your server and run for testing.

Developing chef cookbooks can be done with either:
    EC2 Cloud Servers
    Local Virtual Machines.

Using Local Virtual Machines

The two most popular virtual machines for local use are VMware and Virtual Box.
For VMware depending on the client operating system install
    Windows - either VMware Player or VMware Server
    Linux - VMware Server
    Mac OSX - VMware Fusion.

For Virtual Box Download and install Virtual Box for your operating system.

Virtual Box will run VMware images so decide on the image and download from sites like VMPlanet or VMWare Appliances. Ubuntu is a very popular linux version used in the cloud. Ubuntu 10.10 can be downloaded from VMPlanet.

You can also install Ubuntu from ISO machine images:
    Install Ubuntu 10.10 on VMware Player

Finally you need to make sure ssh is installed on your local virtual machine
    sudo apt-get install openssh-server openssh-client

Chef Cookbook development

    1. Install Chef Solo on your EC2 Server or Local Virtual Machine.
    2. Copy the cookbook to the server.
    3. Run chef solo on the server to test the cookbook.
    4. Repeat modifying the cookbook, copying and testing until the cookbook is complete.
LittleChef is a good tool for doing "push" development of chef cookbooks. The next version of EC2Dream will come with some similar features for developing Chef cookbooks.

Using Chef for Production Servers

For production the server can automatically "pull" and run the configuration as it starts up. Typically you do this using a Chef Server. Either:
    1. Install and run your own chef server. (A Micro Instance costs $15 a month).
    2. Use a service like the Opscode Platform. Signup and get a 5 servers for free but it is cost effective for more.
    3. Continue to use Chef Solo and "pull" the configuration from a source control system like git and run it.

Source Control

For serious development of configuration you need source control. Git is recommended because it is fast fully distributed source control system.
    Windows - download the latest full installer for msysgit
    Mac - Can be installed using Macports or the Git OSX Installer
    Linux - Git can installed using the package management system
For an introduction to git read the first two chapters of Pro Git.

Git References
    Installing Git
    Setup and Initialization
    Git For Windows Developers
    Getting Started with Git and GitHub on Windows
     An Illustrated Guide to Git on Windows

Chef References
    Our First Chef Cookbook
    Automated Deployments with LittleChef
    Info on ssh
    Chef installation and minimal configuration
    Automated Configuration Management with opscode
    Using Opscode Chef to start up a node on AWS EC2 - A Simple Example

Tags and Management Features

2010-09-28T08:17:00.000-07:00

Amazon is gradually adding features that enterprises need to manage their infrastructure. Amazon now allows you to tag the majority of Amazon EC2 resources:
    EC2 instances
    Amazon Machine Images (AMIs)
    EBS volumes
    EBS snapshots
    Spot Requests
    Amazon VPC resources such as VPCs, subnets, connections, and gateways.
You can tag existing resources and tag new resources when you create them. As well as identifying a resource with a meaningful name resources can be identified as, for example, production and test resources.

EC2Dream GUI supports:
  Creating tags for new or existing resources
  Filtering based on tags
  Specifying a user-defined tag as a nickname for a resource.

See Using Resource Tags in EC2Dream for more details.

Combine EC2Dream with the Chef configuration system and Git source control management for a complete open source management environment for EC2 compatible clouds.

Amazon are targeting the enterprise market so they are adding management features.
They have already annouced User managment. The ability to specify multiple users and the access each user has to the Amazon features all billed to the same billing address.

Other area of management that I predict are:
Version Management - S3 Data service already supports versioning.
Deployment - Automatic deployment of Java, PHP code.
Script Management - Maybe the ability to run Chef scripts.
Improved Monitoring - Maybe Nagios support to extend the current Cloudwatch monitoring.
AutoScaling - Extra Autoscaling features like scaling by time of day.

They are also beefing the API up so it can be used by orchestration tools that big enterprises already use in their data centres. For example Idempotent instance creation

References:
Using Resource Tags in EC2Dream
Resource Tagging
Filtering

Micro Server Instance Type

2010-09-09T09:45:00.000-07:00

Amazon has an increasing number of different
instance types:
  High Memory
  High Cpu
  High Performance Computing
  Reserved Instances
  Spot Pricing of Instances.

Now it has Micro Instances. These are incredibly cheap
at 2 cents an hour($14 a month) for linux servers and they can be autoscaled to create a really cheap scalable infrastructure. I hope Amazon bought more servers!!!

This is going to make it tough for other hosting companies and make it irresistibly cheap for companies instead of buying servers. It will also put pressure on per seat prices of SaaS providers like salesforce.com etc. The latest version of EC2Dream supports the new Micro instance type.

Amazingly it is now possible to run your own ERP, CRM, E-Commerce or Content Management System from $14 a month. Install SugarCRM, Open Bravo, Magento or Alfresco using EC2Dream.

References
Amazon Instance Types
Creating Your Online Business: Step by Step Instructions to Buy a Domain, Setup Amazon EC2, and Install Drupal

Cloud Software Stacks

2010-07-01T05:34:00.001-07:00

The whole cloud software area is starting to take shape and looks like the future of the data centre. Data Centers will gradually evolve from physical machines to virtualized servers through to cloud architectures.
Potentially Cloud software Stacks could become a one-stop software-bundle for the data centre, including hypervisers, virtualized operating systems and applications, management tools and APIs, monitoring, backup etc etc.
So instead of sourcing different products from multiple vendors to build a data centre customers could "like microsoft office" get a complete data centre software bundle. No wonder vendors are keen to get into this product segment! But with so many products you have to try and pick a vendor that will survive.

Many cloud management products are still works in progress but common features are taking shape:
- Support the major virtualization technologies (Xen, KVM, VMware)
- Support or planning to support Public Cloud APIs (Amazon EC2/S3 vCloud)
- Support for major linux distributions and some support Windows Servers.
- Support SANs. (EMC etc)
- Monitoring, metering & billing
- Authentication (with support for LDAP and Active Directory).
- Often have open source and enterprise editions.
- Support for extending into a public cloud ("Cloudbursting")
- One management tool for private and public clouds.
- Migration of Machine Images between clouds.

It appears the Cloud Software Stacks are forming into three camps:

1. Amazon AWS Compatible clouds

Eucalyptus - Provides an Amazon AWS compatible cloud.
- Consists of Open Source and Enterprise Editions.
- The Open Source version targets Linux systems that use Xen (versions 3.*) and KVM for virtualization.
- The Enterprise version adds support for Windows servers, VMware virtualization technologies, SAN Adapter, User and Grioup Management, Accounting anmd Quota Management.
- Currently it is being downloaded 10,000 times a month so it is popular.

Nimbula - Cloud Operating System is an automated cloud management system delivering Amazon EC2-like services behind the firewall.
- Commercial version only.
- Currently only supports Xen and KVM but VMware support is planned.
- Supports Microsoft Windows, Linux, Solaris, BSD and AIX servers.
- Offers an API, a command line and a web control panel but does not appear to offer public cloud API support.
- is able to deploy a virtual infrastructure on bare metal so good for large data centers.
- This is a new product by from the original developers of Amazon EC2 so very good pedigree.

Amazon - Virtual Private Cloud

2. VMWare vCloud Compatible clouds

VMware vCloud Director has the following features:
* Supports multi tenancy/organizational isolation
* Allows for the creation of central application catalogs and personalization of templates
* Enables creation and deployment of vApps from catalogs/templates
* Controls user resource usage through roles/rights, quotas and leases
* Enables programmatic control through the RESTful vCloud API
* Provides an additional level of abstraction from underlying hardware

- Launched at VMworld 2010.

3. Others

OpenStack based on NASA Nebula and backed by others like Rackspace is a promising open source initiative and maybe the future for other Cloud stacks to coalesce around.

Abiquo - the provision of physical infrastructure is completely separated from the virtual application infrastructure by a “Resource Cloud”.
-Community and Enterprise Editions.
-Supports Xen, KVM or Virtual Box, VMware and Microsoft’s Hyper-V all in the community edition.

Cloud.com (formerly VMOps) - Massively Scalable Infrastructure Management.
- Has Community, Enterprise Edition and Service Provider Editions .
- Works with Xen and KVM hypervisors as well as VMware vSphere or Citrix XenServer.
- Own API but plan to implement Amazon EC2/S3 and vCloud on top.

OpenQRM - Next generation, open-source Data-center management platform
- supports VMware, Xen, KVM and Linux-VServer vms.
- supports major linux distribution like Debian, Ubuntu, CentOS and openSuse.
- Nagios integration for monitoring.

OpenNebula - Open-Source toolkit for building private, public and hybrid clouds.
- Apache Open Source Licence. Enterprise Edition coming out.
- supports Xen, KVM and VMware virtualization platforms
- on-demand access to Amazon EC2 and ElasticHosts Cloud providers.
- Local Unix-like command line interface and Cloud interfaces, such as libvirt, EC2 Query API and OGF OCCI API
- Support for multiple users, image transferring and cloning, virtual network management and service contextualization.
- Currently no support for windows servers but is planned.

Elastra
Enomaly
NewScale
Aserver - typically for service providers
CloudSwitch - Hybrid Cloud
OpSource - Virtual Private Cloud

References:
Why private clouds are a catch-22 for buyers
The Red Ocean of Cloud Infrastructure Stacks
Eleven Open Source Cloud Computing Projects to Watch
PrivateCloud - news, resources, and conversation on enterprise cloud computing
Virtualization Industry Radar
Nimbula Technical White Paper
cloud.com Edition Comparison
Does OpenStack Change the Cloud Game

Cloud Status and Real-Time Information

2010-06-25T07:00:00.000-07:00

There are a proliferation of websites providing status and real-time information about Public Cloud Providers.

Many of the Public Cloud Providers have an official site that provides the current status of their services and often the status over the previous week or month:

    Amazon AWS
    Google App Engine
    Microsoft Azure
    Rackspace
    Salesforce.com
    GoGrid System Status Blog

    CloudStatus from Hyperic shows both Amazon App Engine and Amazon EC2 statuses.

Some sites show the response time and/or performance of a sample application running on the cloud:

    CloudClimate shows EC2, S3, GoGrid, NewServers, Opsource, Content Delivery Networks.
    CloudSleuth shows Amazon, GoGrid, Opsource, Rackspace.

There are also specialized information about particular cloud features of services:

    CloudExchange show real-time prices of Amazon EC2 Spot Instances.
    If your a google account holder Google dashboard shows all your usage of the various google services (Gmail, Blogger etc).

Load Balancing and Auto Scaling

2010-05-19T07:24:00.000-07:00

Amazon ELB and AutoScaling

The Amazon Elastic Load Balancer Service supports HTTP and TCP requests. It now supports sticky sessions via cookies so it should be much more usable as many applications require sessions to be routed to the same server.
HTTPS is supported indirectly via using TCP and port 443. ELB directly forwards the TCP packets to the instance. Because support is at the TCP level, not at the HTTPS level, the ELB will not:
- perform SSL termination
- add an X-Forwarded-For header (which the HTTP listener does)
- optimize connections from your back-end instances using keepalive.
This means, for example, with HTTPS you cannot get the client's IP address from the X-Forwarded-For header as with HTTP

The Amazon Auto Scaling service uses CloudWatch as the monitoring mechanism for determining to scale up or down.

Costs of Elastic Load Balancers are from $0.025 hr ($18.25 mth) and a small cost for GB processed plus Cloudwatch cost per instance $0.015 hr ($10.95 mth). So this is typically cheaper than running load balancing on your instance.

Summary: Cheap, reliable, scalable bare bones scaling solution.

Scalr

Scalr is another popular load balancing/scaling solution. It can be run from the Scalr website for $99 a month (it is creeping up in price from $50 a month) or you can run your own copy of Scalr as it is open source. The website also provides some management features like scripting and SSH access.

The Scalr architecture is built around several Amazon Machine Images (Small or Large) and targets multi-tier applications
1. Load Balancer runnning Nginx for the load balancer. They also support HTTPS. It is possible to use other load balancers like Pound.
2. Application Server Image - LAMP/Java Tomcat/RubyOnRails.
3. MySQL Master/slave for the database.
You also need to do administration of Amazon EC2 through the Scalr platform.

Scalr has some nice features like time based scaling, auto deployment of updated Amazon Machine Images. They have some pre-defined machine images so if your application matches these it will be faster to setup. (NOTE: Currently Amazon's RDS does not support MySQL Master/Slave while Scalr does )

Summary: More features that Amazon Elastic Load Balancer/Auto Scaling. Still relatively cheap and open source so you can always run it yourself although easier to use through their website.

Cloud Foundry

Cloud Foundry is a great platform for deploying and auto scaling java applications. It is specify to Java applications supporting:
    Java 1.6
    Apache 2.2.3
    SpringSource tc Server (Tomcat 6)
    MySQL 5.0.45
    CentOS 5.2

The Apache HTTP Server load balances over the Tomcat servers and it uses Hyperic to monitor the servers and auto scale the Tomcat servers. It support MySql Master/slave configuration for scaling the database. The database is stored on an Amazon EBS and automatically backed up to S3. (Similar to Amazon RDS). Currently it is in extended beta and free.

Summary: The fastest way to deploy and run Java your applications. Restricted to their particular java software stack but a great platform if your application matches this stack.

Rightscale

RightScale was the first company to provide load balancing and Scaling of Amazon EC2. They are also the most expensive from $2500 up-front and $500 a month.
They have a powerful scripting and templating system and their platform is more comprehensive than Scalr. The scripting however is also complex and proprietary. They support other clouds like Rackspace etc.
RightScale provide an Apache+HAProxy setup for load balancing and also will automatically setup a MySql Master/Slave database as does Scalr.

Summary: If you have complex/large scalability requirements and Rightscale have pre-built scripts and templates for your application then they are a good option.

Others

Makara provide scaling, monitoring and deployment of Java and PHP applications. Similar to Cloud Foundry in some ways. Now they are owned by Redhat good for Redhat and CentOS users.

Cloudkick provide management, auto scaling, monitoring and alerting. Now they have been acquired by Rackspace good for using with Rackspace cloud.

References

Autoscaling in 10 mins
Amazon Elastic Load Balancer Setup
Elastic Load Balancing with Sticky Sessions
Autoscaling Web Services on Amazon EC2

Auto-Scaling Web Sites Using Amazon EC2 and Scalr
7 Secrets to Successfully Scaling with Scalr (on Amazon)
The Official Scalr blog

Any thoughts on Rightscale and Scalr for dynamic ec2 instance managment
RightScale - How do I set up Autoscaling?
Thoughts on RightScale

Amazon EBS Images

2010-02-18T07:34:00.001-08:00

Amazon server instances can now be booted from EBS Snapshots as well as instance stores bundled onto S3.

This means changes in your server instance are persisted on an EBS volume so you don't lose them when the server shuts down. It should become the default way most people boot up their server images on Amazon EC2.

This takes Amazon much closer to the VMware approach of running virtual machines from Storage Area Networks. In addition to terminating an instance you can stop and start them which is like a pause. Also you can snapshot them just like on VMware.

Another benefit is the size of your root partition can be as large as 1 TB overcoming the old instance store limitation of 10GB.

EC2Dream fully supports booting amazon instances from EBS images. At the press of a button
    - Instances can be stopped and started.
    - Images can be created.
    - Instance attributes can be modified.
    - Additional block storage devices can be specified in the launch profile and automatically attached and mounted.
    - Snapshots can be taken of additional block storage devices.

You can now even move a running EBS Boot Image to another server. This is equivalent to VMware VMotion. Eric Hammond has once again come up with the clever information.

It is important to backup your EBS via snapshots and to S3. Auto Snappy offers clever scheduling of snapshots and rotating the old ones out.

Finally, performance of EBS can be quite variable. So for an application do alot of disk I/O you may need to load balance more instances or stripe EBS volumes together to create a RAID device.

It is interesting to see Amazon add more enterprise features to its cloud solution while VMware adds more cloud features to its strong Virtualization base vCloud. In the future both Amazon EC2 and vCloud will offer strong cloud solutions.

References:
Creating Consistent Snapshots of a Live Instance with XFS on a Boot-from-EBS AMI
Increasing Root Disk Size of an EBS Boot AMI on EC2
Resizing the Root Disk on a Running EBS Boot EC2 Instance
Amazon EC2 – Boot from EBS and AMI conversion
Understanding and using Amazon EBS
Amazon’s Elastic Block Store explained

Chef - Automated Infrastructure in the Cloud

2010-01-22T06:07:00.000-08:00

Chef is an open source configuration management tool. It is particularly popular for cloud computing. Chef is written in ruby and uses a Ruby DSL (Domain specific language). It consists of the following components:

Recipes and Cookbooks - you write recipes that describe the configuration of your server. Collections of recipes are cookbooks. There are some recipes already available

Chef Client - To operate you need a chef client installed on your server.

Chef Server - The chef client can communicate with the Chef Server which stores recipes in a source control system like git.

Chef-Solo - The simplest way to use chef is to use chef-solo to run local recipes without needing the Chef Server.

Getting Started with Chef-Solo
In this example we will use chef-solo to configure an ubuntu ec2 server to use apache2. Go into EC2Dream and ....

1. Create a security group called say App with port 22 and port 80 open and launch a server from a ubuntu ebs image:
  063491364108/ubuntu-9.10-karmic-server-i386-20100121 (ami-6743ae0e)
See here for more details
NOTE: Add 2 Block Devices to the launch specification of size 1 GB as devices /dev/sdq2 and /dev/sdq1.

3. Copy the files from EC2Dream\chef folder to the root user home directory.
To do this click on the App in the EC2Dream tree and in the server click on the WinSCP icom and a WinSCP session will open (or on Linux or Mac OS X a scp command will display).
Then go to the EC2Dream\chef folder and drag and drop the files.

4. Log on to the App server by clicking on the terminal icon and a PuTTY session will open (or on Linux or Mac OS X a ssh session).

  a. Install chef by running chef-bootstrap.sh
  b. Create directories
     /etc/chef
    /etc/chef/recipes
    /etc/chef/recipes/cookbooks

  c. Copy solo.rb and node.json to /etc/chef

  e. Untar apache2.tar.gz to /etc/chef/recipes/cookbooks

5. Run chef-solo -c /etc/chef/solo.rb -j /etc/chef/node.json

6. Use a browser and go to the public IP address eg http://ec2-99-99-99-99.compute-1.amazonaws.com and you should see the following:

7. If the chef script is put in S3 then it is possible to build a generic process to configure servers using chef. S3 now supports versioning so version can be kept.

NOTE: Opscode now have some pre-build images for ubuntu so you don't have to install your chef client.

Multiple Server Architectures
So what about complex systems that consist of architectures requiring multiple servers. Amazon have a feature CloudFormation that make it easy to provision AWS resources and then deploy them the same way every time. This way you can describe the various AWS resources that make up a system and bring it up in a repeatable way as well as deleting resources when shutting down.

Then Chef can be responsible for application software deployment, configuration and execution on the Amazon EC2 instances. See Integrating AWS Cloud Formation With Opscode Chef

Using Hosted Chef Server on Opscode Platform
An alternative to using chef solo is to keep the chef scripts in Chef Server. Opscode Platform is a Chef Servers supplied as a service. Go to Opscode and register. You get the first 5 nodes free

The steps are:
1. Verifying your system requirements and dependencies are all in order
2. Setting up your user environment
3. Setting up and configuring a new Chef Client on the Opscode Platform
4. Creating a cookbooks repository and uploading cookbooks
5. Cooking with Chef

References
Amazon EC2 + Chef = Mmmmm
Opscode
Apache2 Cookbook
Cooking with Chef 101
Using Chef for Automated Infrastructure in the Cloud
Configuration Management with Chef on Debian, Part 1
Chef Solo
Creating an Amazon EC2 AMI for Opscode Chef 0.8
integrating-aws-cloudformation-and-chef

Spot Instances

2010-01-20T03:54:00.000-08:00

Amazon Spot Instances enable you to pay a discount price to use spare servers on Amazon EC2. You need to be willing to relinquish them when the spot price exceeds the price you are willing to pay. Currently spot prices are around 40% of on demand price see cloudexchange.org.
As well as the obvious computational tasks spot instances are really useful for test and prototype systems. When setting up systems run them as spot instances and only as on-demand when setup and testing is complete.
EC2Dream lets you link to cloudexchange.org to see current spot prices, enter a price in the launch parameters and launch a server.

References
Listing Recent Prices for EC2 Spot Instances
Cloud Exchange
The Cloud Market

Amazon RDS

2009-11-30T06:16:00.001-08:00

EC2Dream provides the first GUI graphical administration for Amazon Relational Database Service (RDS)

Amazon RDS provides MySQL and Oracle as a service:
1. It automatically creates an EC2 MySQL 5.x or Oracle 11g instance with attached EBS storage.
2. It automates the backups using the snapshot mechanism.
3. All configuration is done via web service calls.
4. Existing DBA system administration tools can be used.

One of the problems of system administration is that all software applications have different configuration files, configuration databases, registries etc.

If all configuration could be reduced to web services calls then that would be highly significant for system admin. You then don't need to know how the system configuration is stored in the application just the web service call. Then it wouild be possible to automate configuration and the vision of Autonomic Computing may start to be realised. If all software applications had web services apis for configuration then intelligent clients could be written to automate configuration etc etc.

Also alot of the work Amazon have done for snapshotting and configuration could be reused to create other services. When will we see JAS (Java Application Service) for Tomcat????. Jan 2011 - Amazon have annouced this and it is called Amazon Elastic Beanstalk.

References
Top 10 things to know about Amazon RDS
Amazon RDS FAQs
MySQL Reference Manual 5.1
MySQL Configuration and Tuning
Cloud Databases and Oracle

Install and Run Openbravo on EC2

2009-11-04T03:22:00.000-08:00

Openbravo is is the leading professional web-based enterprise resource planning solution.

The EC2Dream graphical environment now has scripts to make setup and management of OpenBravo on Amazon EC2 simple and fast - 30 mins!!! The scripts built a complete system that persists on an EBS-based image.

See the Documentation for more info.

References
Openbravo Documentation

Install and Run SugarCRM on EC2

2009-10-07T08:04:00.000-07:00

SugarCRM is the leading Open Source Customer Relationship Management (CRM) System. A complete CRM system for businesses of all sizes. Core CRM functionality includes sales automation, marketing campaigns, support cases, project mgmt, calendaring and more.

The EC2Dream graphical environment now has scripts to make setup and management of SugarCRM on Amazon EC2 simple and fast - 30 mins!!! The scripts built a complete system from a Ubuntu image. Installing PHP5, Apache, MySql and SugarCRM and persisting data on an EBS.

See the Documentation for more info.

References
SugarCRM Downloads and docs
SugarCRM Community Edition v5.2.0f AMI provided by SugarCRM, Inc.

Install and Run Magento on EC2

2009-10-06T05:52:00.000-07:00

Magento is a Popular and Growing Open Source eCommerce Platform. Magento provides a huge list of features as standard.

The EC2Dream graphical environment now has scripts to make setup and management of Magento on Amazon EC2 simple and fast!!! The scripts built a complete system from an EBS-based Ubuntu image. Installing PHP5, Apache, MySql and Magento and persisting data on an EBS.

See the Documentation for more info.
References
Using Magento on Amazon EC2
Magento User Guide

Install and Run Alfresco on Amazon EC2

2009-09-29T08:09:00.000-07:00

Alfresco is a Popular Open Source Enterprise Content Management System. They have an official Amazon Machine Image based on Ubuntu, Tomcat and MySql although curently not EBS-based images.

The EC2Dream graphical environment now has scripts to make setup and management of Alfresco on Amazon EC2 simple and fast. 15 minutes to create an Alfresco EBS-based Image. If you build in on the Amazon Elastic Beanstalk image then you will automatically have a auto-scaling, production-ready environment!!!

See the Documentation for more info.

IT as a Service

2009-09-26T03:12:00.001-07:00

It is significant that cloud computing is categorizing into
  IaaS (Infrastructure as a Service),
  PaaS (Platform as a Service) and
  SaaS (Software as a Service but really Application as a Service).

Infrastructure as a Service
IaaS Vendors are now well defined with half a dozen big players (Amazon EC2, VMWare VSphere, GoGrid, Rackspace, AServer). The time for new entrants is closing. The APIs are defined. We will now see incremental improvement, competition on price and services being offered on top on of IaaS.
  When running your systems on IaaS you still have to manage the platform (patching, configuring etc) and write the code for your application but it will be most easiest to migrate existing system to cloud computing.
Software as a Service
This area has been established for some time by the likes of SalesForce.com, NetSuite etc. If you just want to run a well-defined application (ie Accounting, CRM etc) then this is the easiest way to go.
Platform as a Service
  Typically these platforms are multi-tenanted so they will be cheaper to run than dedicated servers on IaaS, but existing systems will not be able to be easily migrated, they will have to rewritten.
  The first wave of PaaS products came from the SaaS vendors who realised that users wanted to customize the SaaS applications. Typically they are very propretiary so if you are willing to write (lockin) your application to the standard and restrictions of the SaaS vendor platform it will be quicker to develop and you will only have to maintain your code not the platform.
  The second wave is more language dependent and not related to a particular SaaS application. Heroku and Engine Yard for Ruby applications, Azure for .NET applications, SpringSource and Wavemaker for java, Google App Engine for python and java. Potential these could develop into more open standards.
Summary
1. Large systems will migrate to IaaS because of the ease of migration and the ability to administer and control the platform.
2. Smaller applications will migrate to PaaS, particularly as more language-specific standards appear.
3. Commodity applications will be best supplied on a SaaS basis.

References
Opening Up Platform as a Service - What is Open PaaS?
WaveMaker Cloud Edition

PowerShell EC2 Interface

2009-09-09T08:06:00.000-07:00

Power-EC2Dream is a C# interface to Amazon EC2 and S3 (and Eucalyptus) that can be used from Windows PowerShell to write scripts against Amazon EC2 (and Eucalyptus).

The PowerShell scripting shell and language is the strategic direction for Microsoft Scripting. There are an increasing number of scripts available so it should be the best tool for configurating Windows Servers in Amazon EC2.

API documentation?	powerec2dream
Dowload?	Version 2009-07-15-Beta2

Examples
Show the running instances:

$ec2 = new-object Amazon.EC2.PowerEC2Dream($env:KEY,$env:SECRET_KEY) $res = $ec2.DescribeInstances()

For ease of scripting parameters are passed as strings and return values are typically hashtables. Show the attributes of a particular image:

$ec2 = new-object Amazon.EC2.PowerEC2Dream($env:KEY,$env:SECRET_KEY) $res = $ec2.DescribeImages(@("ami-7ac02713")) write-host $res[0].Keys foreach ($r in $res) { write-host $r.Values }

Also see Scripting with Powershell

Cloud APIs

2009-09-07T01:19:00.001-07:00

Three main Cloud APis are emerging:

Amazon EC2 - The most likely de facto standard. Based on a modified Xen hypervisor with Management tools and an API that is the most extensive for cloud computing. Amazon have first mover advantage and it is getting easier to setup servers and amazon EC2 will provide services like monitoring, auto-scaling, VPN etc as part of the service. Eucalyptus offer a private cloud compatible with EC2. Other clouds such as Openstack and CloudStack offer the EC2 API as well as their own proprietary API. All these products also support the VMware hypervisor so the EC2 API can be used to manage all the major hypervsors except Microsoft's Hyper-V.

VMware - VCloud. Based on the VMware hypervisor with Virtual Infrastructure management tools which allow you to build a private cloud. VMWare are offering their product to companies to use to build public clouds and be a big supporter of hyrid clouds. See vCloud API. VCloud API has been open sourced by VMware.

Azure - Based on HyperV. Microsoft products only. See Windows Azure SDK and Windows Azure Details

A common API for clouds is not emerging. Attempts like Jclouds - a lightweight open source multi-cloud framework have been useful for admin tools developers. With backing from VMware it is emerging as the best product for accessing multiple clouds. It is also used by other cloud management products to access multiple clouds as it is a good way to access the smaller cloud vendors like Terremark, GoGrid and Rackspace.

It looks like Amazon EC2/S3 is becoming the most common API and more cloud products are supporting it in addition to there own APIs.

References
Comparison_of_AWS_and_OpenStack
VMware vs. Amazon … ROUND ONE … FIGHT!
Amazon Web Services: The De Facto Cloud API?
Premium vs. Commodity Cloud Computing Providers
Cloud APIs
DMTF Cloud Standards.
Do We Need ODBC for the Cloud?
Apache libCloud (Python)
IT utilities, the biggest game in town - A good summary.

Graphical System Admin for Eucalyptus

2009-07-17T07:44:00.000-07:00

EC2Dream now supplies Graphical System Administration for Eucalyptus!. It replaces the command line amazon EC2 tools with a graphical environment that can be installed on your Windows, Linux, or Mac OS X client machines.
* Can operate completely behind your corporate firewall.
* You can even try it out at no cost using the Eucalyptus Community Cloud.

In EC2Dream you can create mutiple environments and switch between them:
1. Private cloud for Eucalyptus
2. Public Cloud for Amazon EC2
3. Hybrid Cloud - Coming soon will be the abililty to move images between the private and public clouds to create the ultimate flexibility.

For more info see the documentation. All the benefits of cloud computing but in your own private cloud.

References
Is Eucalyptus ready to be your private cloud?
Private Cloud Computing is real get over it
Microsoft Cloud Computing Infrastructure
vSphere in a Box: A Virtual Private Cloud Blueprint

EC2Dream - Graphical System Admin Tool

2009-06-05T03:50:00.000-07:00

Support for Amazon EC2 and Eucalyptus.
   Multiple environments based on access key, region and keypair.
   A tree view of your servers.
   One click SSH, SCP and Remote Desktop access.
   Script for one click bundling of an instance.
   One click deregister and delete Image.
   Launch and terminate EC2 and Eucalyptus Servers
   View Server's System log
   Save Launch profile
   Ability to build and run remote scripts.
   Server scripts to provide auto mount of EBS, and synchronisation of dynamic IP address.
   Support for Ruby for Linux Servers.
   Support for Powershell for Windows 2003 Servers
   Create, Delete, Display, Authorize, Revoke Security Groups
   Create, Delete, Display, Attach, Detach EBS Volumes
   Create, Delete, Display EBS Snapshots
   Create, Delete, Display, Associate, Disassociate Elastic IPs
   Create, Delete Keypairs
   Register, DeRegister Images.
   Monitoring Graphs
   Scripts for applications
   Nicknames for EBS and snapshots
   EBS Image creation and selection.
   Start, Stop EBS based servers.
   Support for Spot Requests
   Links to cloudexchange.org and cloudmarket.com.

Support for Amazon RDS:
   Launch and terminate RDS Servers
   Create, Delete, Display DB Snapshots
   Create, Delete, Modify DB Parameters
   Modify DB Instance
   View DB Events
   Restore DB from Snapshot and to point in time
   Once click access to MySQL Administrator GUI

For more info on using EC2Dream see the Documentation.

EC2Dream is simple to install and replaces multiple tools currently required to administer Amazon EC2. For Eucalyptus it provides a Graphical system admin environment instead of the Amazon EC2 Command line tools!!! It is released under the Apache 2.0 Open Source license.

Architecture of EC2Dream System Administration:

EC2Dream Installation

2009-06-05T03:49:00.002-07:00

EC2Dream is written in Ruby and uses the FXRuby graphics library so the installation process is mostly about getting Ruby installed with FXRuby.

NOTE: It will continue to use the existing version of Ruby installed with previous version of EC2Dream.

NOTE: To use EC2Dream with CloudStack you need to update the api version in the right_aws ruby gem after installing EC2Dream.
To do this look for file right_ec2.rb (look in the ruby folder that contains the gems and then under folder right_aws-3.0.0/lib/ec2) and change
    API_VERSION = "2011-02-28"
to
    API_VERSION = "2010-11-15"

WINDOWS INSTALL

1. Download either the latest Ruby 1.8.7 or Ruby 1.9.2 from http://rubyinstaller.org/downloads/
At time of writing latest Ruby 1.9.2 was:
http://rubyforge.org/frs/download.php/75127/rubyinstaller-1.9.2-p290.exe
an latest Ruby 1.8.7 was:
http://rubyforge.org/frs/download.php/75851/rubyinstaller-1.8.7-p358.exe

2. Run the Ruby installer and install at the default location like C:\Ruby and check add ruby executables to your PATH. Don't use spaces in folder name like "C:\Program File\Ruby"

3. Download FXRuby from http://rubyforge.org/frs/download.php/66849/fxruby-1.6.20-x86-mingw32.gem

4. In a Command window install FXRuby by running
    gem install FXRuby-1.6.20-x86-mingw32.gem

5. Download and unzip the latest EC2Dream to a folder on say C: drive and it will create an EC2Dream directory. (No spaces in directory name)

6. Edit the run.bat file and set the EC2DREAM_HOME to the folder of the EC2Dream install (i.e. C:\ec2dream)

7. Execute run.bat to start EC2Dream GUI tool
NOTE: During startup the run.bat will first time automatically install the additional Ruby gems (components). If this fails they can be manually installed via these commands
    gem install <EC2DREAM_HOME folder>\ruby\right_aws-3.0.0.gem
    gem install capistrano
    gem install tzinfo
    gem install rubyzip
    gem install gchartrb

8. Create a shortcut to the run.bat using the ec2dream.ico icon file.

If you are using Amazon RDS and would like to also use the MySQL GUI Tools

1. Download MySQL GUI Tools from http://dev.mysql.com/downloads/gui-tools/5.0.html

2. Install in the default location

3. It will then be accessible from the Server tab in EC2Dream when using RDS.

Note: If installed at non-default location set the EC2DREAM_MYSQL_ADMIN environment variable in run.bat using forward slashes.
e.g. SET EC2DREAM_MYSQL_ADMIN=C:/Program Files/MySQL/MySQL Tools for 5.0/MySQLAdministrator.exe

4. Amazon RDS needs to be set in the Settings tab of EC2Dream.

MAC OS X INSTALL
Ruby should be pre-installed on recent MacOS Tiger and Leopard machines.
However FOX toolkit and some additional Ruby gems need to be installed.

Follow documentation on http://www.fox-toolkit.net/documentation/building-fox-on-mac-os-x

1. Install X11 from http://www.apple.com/downloads/macosx/apple/macosx_updates/x11formacosx.html or the MacOS X install DVD

2. Install XCode from install DVD.

3. Install MacPorts as per http://www.macports.org.

4. Install fox:
  sudo port install fox

5. Install gems (ruby components):
  sudo gem install fxruby
  sudo gem install <EC2DREAM_HOME directory>/ruby/right_aws-3.0.0.gem
  sudo gem install capistrano (should already be installed)
  sudo gem install tzinfo
  sudo gem install rubyzip
  sudo gem install gchartrb

6. Download and unzip the latest EC2Dream to a directory say /home/myuser and it will create an EC2Dream directory.

7. Edit the run.sh file and set the EC2DREAM_HOME variable to the directory of the EC2Dream install (i.e. /home/myuser/ec2dream)

8. Give run.sh execute permissions and execute ./run.sh to start EC2DREAM GUI tool

9. Create a shortcut to the run.sh using the ec2dream.ico icon file

FEDORA (11) INSTALL

1. Install Ruby using the package manager of your linux machine
  su -
  yum install -y ruby gcc gcc-c++ ruby-devel ruby-docs ruby-ri ruby-irb ruby-rdoc rubygems

  NOTE: If no rubygems package found then download and install from the rubygem.org site. See http://bparanj.blogspot.com/2008/04/rubygems-110-on-centos-5.html

2. Find a version of the FOX toolkit for your linux machine from http://dag.wieers.com/rpm/packages/fox/ download and install.

In my case i was using fedora 11 on an AMD64 machine so I chose
http://ftp.belnet.be/packages/dries.ulyssis.org/fedora/fc10/x86_64/RPMS.dries/
  fox-1.6.33-1.fc10.rf.x86_64.rpm
  fox-devel-1.6.33-1.fc10.rf.x86_64.rpm
and then install both using
  rpm -ivh fox-1.6.33-1.fc10.rf.x86_64.rpm
  rpm -ivh fox-devel-1.6.33-1.fc10.rf.x86_64.rpm

An alternatively is to download the fox files from http://sourceforge.net/projects/mingw-cross/files/

  NOTE: The Fox Tool Kit software version must match the FXRuby Version (i.e. currently FXRuby is is 1.6.19 so
Fox must be 1.6 and 1.6.19 or greater but I would recommend the 1.6.3x versions).

3. Install gems (ruby components):
  sudo gem install fxruby
  sudo gem install capistrano
  sudo gem install <EC2DREAM_HOME directory>/ruby/right_aws-3.0.0.gem
  sudo gem install tzinfo
  sudo gem install rubyzip
  sudo gem install gchartrb

4. Download and unzip the latest EC2Dream to a directory say /home/myuser and it will create an EC2Dream directory.

5. Edit the run.sh file and set the EC2DREAM_HOME to the directory of the EC2Dream install (i.e. /home/myuser/ec2dream)

6. Give run.sh execute permissions and execute ./run.sh to start EC2DREAM GUI tool

7. Create a shortcut to the run.sh using the ec2dream.ico icon file

LINUX UBUNTU (JAUNTY) INSTALL

1. Install Ruby using the package manager of your linux machine

  sudo apt-get install -y ruby gcc g++ libxrandr-dev libopenssl-ruby ruby1.8-dev

2. Install Rubygems
  sudo apt-get install ruby-rubygems

3. Install fox toolkit.
  sudo apt-get install -y libfox-1.6-dev libfox-1.6-0

4. Install gems (ruby components):
  sudo gem install fxruby
  sudo gem install capistrano
  sudo gem install <EC2DREAM_HOME directory>/ruby/right_aws-3.0..0.gem
  sudo gem install tzinfo
  sudo gem install rubyzip
  sudo gem install gchartrb

5. Download and unzip the latest EC2Dream to a directory say /home/myuser and it will create an EC2Dream directory.

6. Edit the run.sh file and set the EC2DREAM_HOME to the directory of the EC2Dream install (i.e. /home/myuser/ec2dream)

7. Give run.sh execute permissions and execute ./run.sh to start EC2DREAM GUI tool

8. Create a shortcut to the run.sh using the ec2dream.ico icon file.

UPGRADE
1. Rename current ec2dream folder to a different name

2. Unzip the ec2dream zip installation file.

3. Copy the env directory from old version of EC2Dream to new version EC2Dream.

4. Edit the run.sh or run.bat and set the EC2DREAM_HOME variable.

5. Run run.bat or give run.sh execute permissions and execute ./run.sh to start EC2DREAM GUI tool

6. Create a shortcut to the run.sh using the ec2dream.ico icon file

7. Uninstall the right_aws software as a new customised version is supplied.
  gem uninstall right_aws

8. If upgrading make sure the following packages are installed:
  gem install EC2DREAM_HOME/ruby/right_aws-3.0.0.gem
  gem install tzinfo
  gem install rubyzip
  gem install gchartrb

EC2Dream User Guide

2009-06-05T03:49:00.001-07:00

Methodology

Installation

Setup an Amazon EC2 or Eucalyptus Account

Create an EC2Dream Environment

  Run a Linux Instance

  Build a Linux Base Image

  Build a Linux Application Image

Using Resource Tags

Using Amazon RDS

Using Windows Servers

Power-EC2Dream - Powershell EC2 Interface API Documentation

Scripting with PowerShell

2009-05-07T02:23:00.000-07:00

The PowerShell scripting shell and language is the strategic direction for Microsoft Scripting. There are an increasing number of scripts available so it should be the best tool for configurating Windows Servers in Amazon EC2. PowerShell is a .NET 2.0 application and it uses .NET to achieve virtually everything so the power of .NET is leverage from within PowerShell. You can access EC2 and S3 with the .NET interfaces.

PowerShell 1.0 supports Windows XP and Windows Server 2003 but has no remote scripting capability. To do configuration of Amazon EC2 Windows instances install the PowerShell 1.0 EC2Dream environment(see below) and either:
1. Copy configuration scripts to the Windows 2003 Server instance via the Remote Desktop (need to enable drive access in remote desktop to do this) and run them in Remote Desktop.
2. Use sshd that comes with cygwin (see below)

PowerShell 2.0 supports remote scripting but is only available with Vista and Windows Server 2008(currently unsupported by Amazon EC2) and is still a preview release. The latest preview being CTP3.

There are an increasing number of powershell scripts available at sites like the Script Center Script Repository. If you want a GUI for Powershell download PowerGUI.

Architecture of EC2Dream System Administration with Powershell:

References - PowerShell
MSDN Windows Powershell
Windows PowerShell Getting Started Guide
Windows PowerShell Blog
Scripting with Windows PowerShell
PowerShell basics .NET
The Windows PowerShell Toolbox
Powershell script to create Remote Desktop connection files
Windows PowerShell Snap-In for EC2
How to create a PowerShell batch file
Running Windows PowerShell Scripts
Microsoft TechNet Script Center
Windows PowerShell Quick Reference
How to Download Windows PowerShell 1.0
Download Windows PowerShell
The Script Center Script Repository

References - SSH and PowerShell Addons
Installing an SSH Server on Windows 2003
Using PowerShell through ssh
How to Download Windows PowerShell 1.0
Using PowerShell through ssh
copSSH
How to setup Cygwin to run *nix commands in Windows
cygwin sshd
Shell Tools
PowerShell Plus
NetCmdlets

Provisioning

2009-04-22T07:49:00.000-07:00

Many software companies are creating Virtual Machines pre-built for various environments but this seems to me to be a partial solution. I think it is better to have scripts to automatically build applications from a base image. The script would be customizable as well. You also need to script the management of the image, deploying new builds etc.

Most provisioning tools (Puppet, Chef etc) are very complex (well for me, anyway -)). The best tool I have found so far is Capistrano to do remote execution of ruby scripts. Also Sprinkle looks interesting as it uses simple declarative ruby scripts but it is a very young development.

So the Methodology to build Application Image should be:
1. Build Base Image manually.
2. Create a script for Application.
3. Build Application Image with script to Instance and EBS.
4. Bundle Application Image and snapshot EBS.

To Scale:
1. Create EBS from snapshot.
2. Start Application Image with new EBS.
3. Minimal script to change runtime parameters as required.

If there were a library of pre-built scripts then it would be really easy to build lots of different servers. This approach has alot of potential for cloud environments.

References
Capistrano
Capistrano and EC2
Sprinkle Intro
Using the EC2 environment for fewer moving parts

Methodology

2009-04-17T07:24:00.000-07:00

Simple Systems
For running a system of a dozen or so servers that do not have large performance requirements adopt a simple approach:

1. For Each Server(or group of scaled servers) create a unique security group for identification.

2. Use images are already tailored for Amazon and can be trusted.
For Linux use RightScale CentOS, Alestic Ubuntu or Amazon Linux Centos images. For Windows use Rightscale Windows images.

3. Use scripts to configure the servers for an application for networking multi-tier applications, backup, initialisation using either Ruby and Bash for linux or PowerShell for Windows.

4. It is better to use EBS-based images.
(For the the older instance-store images create an Elastic Block Store (EBS) for each server to persist application changes which can be auto-mounted at startup).

5. Create a standard build script to build a base image to be the basis of all application images. Optionally create, or bundle, an image.

6. Build Applications images from the Base image or with the build script.
(For the older instance-store images install as much of the application as possible on the EBS to minimize change to the Base image and maximize sharing of an Application image).

7. Create an image for EBS-based images using a naming convention containing the date. i.e. securityGroup-YYMMDD
(For the older instance-store images use a script to bundle the application image following the conventions for storing images and backups on S3 ).

8. EC2Dream System Admin has been designed to support this methodology.

Complex Systems
For running larger, more complex, high performing systems then it is worth investing in developing Chef Cookbooks to config the Servers. In this approach:
1. You start the server telling it what sort of server it is.
2. It configures itself automatically from scripts from the Chef Server.
3. Use all the disciplines of development (version control, automatic deployment of code etc) in the configuring of servers.

I would recommend using a combination of CloudFormation to provision the AWS resources (Instances, Disk, Users etc) and Chef for application software deployment and configuration of the instances. See Integrating AWS Cloud Formation With Opscode Chef

Also look to Amazon BeanStalk, Cloud Formation and Cloud Foundry for further automated infrastructure.

References
Practical Cloud Computing Patterns

Snapshots

2009-04-09T05:37:00.000-07:00

Snapshots are a way of backing up an Elastic Block Store (EBS). An EBS can also be created from a snapshot so this is a way of restoring an EBS or cloning an EBS. Snapshots are stored on S3 although they are not currently accessible via a bucket so you cannot make a backup copy of the snapshot (for example an off-site backup).
Snapshots are not useful for backing up volatile data that needs consistency (e.g SQL databases). In this case you need to either:
  - Use a backup utility of the product that can do a backup in a consistent fashion.
  - Have a a slave copy of that can be frozen to do a backup.
Once a backup is taken it can be either:
  - Copied to S3
  - If the backup is on an EBS then a snapshot could be taken of the backup.

Ruby scripts can be used to manage snapshots
First a script to create a snapshot of a volume. The script gets the volume id from the user data or you can hard code the volume. It can be run regularly via cron on linux or similar on Windows (ie AT command).

#!/usr/bin/ruby # script to create a snapshot require 'rubygems' require 'right_aws' require 'net/http' AMAZON_PUBLIC_KEY=<public key> AMAZON_PRIVATE_KEY=<private key> url_user_data = 'http://169.254.169.254/latest/user-data' # switch between the next 2 lines to either use user data or # a hard coded volume ec2_vol = Net::HTTP.get_response(URI.parse(url_user_data)).body # ec2_vol ='vol-xxxxx' print "Create Snapshot ",ec2_vol," ",DateTime.now,"\n" # remove last parameter if not in the EU zone ec2 = RightAws::Ec2.new(AMAZON_PUBLIC_KEY, AMAZON_PRIVATE_KEY,:endpoint_url => 'https://eu-west-1.ec2.amazonaws.com/') print "snapshot ",vol,"\n" vol = ec2.create_snapshot(ec2_vol) print "result ",vol,"\n" print "Snapshot Complete ",ec2_vol," ",DateTime.now,"\n"

Snapshots - How they work
When you take a series of snapshots for a volume, each unique piece of information is stored once, regardless of how many snapshots need that piece of data. (So each snapshot just has the differences between the other snapshots).
For example:
a. Say you have a 3GB volume that you put 1GB on and take a snapshot, S1.
b. Then you put an additional 1GB on it (so that you're using 2GB) and take another snapshot, S2.
c. Internally Amazon is storing 2GB (less after compression) of total unique data.
d. If you deleted S1, it will still be storing 2GB, because all that data was still needed by S2.
e. If instead you deleted S2, the amount of data stored goes down to 1GB, the amount that was needed for S1.

Charges for Snapshots
Charges aren't tracked per snapshot. Amazon looks at the total amount of unique data for all the snapshots you have.
So if you have a volume, V1, take a snapshot of it, S1, create a new volume, V2, from the snapshot and then create a snapshot, S2, of V2, you will not be charged twice for the data shared between S1 and S2.
Charging is for the total amount of unique data needed by all your undeleted snapshots.

Delete Snapshots
The following script can also be run to delete snapshots when they reach a certain age. I tested restoring from a snapshot after deleting an old one so the snapshot are not dependent on one another.

#!/usr/bin/ruby require 'rubygems' require 'right_aws' require 'net/http' require 'date' AMAZON_PUBLIC_KEY=<public key> AMAZON_PRIVATE_KEY=<private key> # switch between the next 2 lines to either use user data or # a hard coded volume url_user_data = 'http://169.254.169.254/2008-02-01/user-data' ec2_vol = Net::HTTP.get_response(URI.parse(url_user_data)).body # ec2_vol ='vol-xxxxx' # remove last parameter if not in the EU zone ec2 = RightAws::Ec2.new(AMAZON_PUBLIC_KEY, AMAZON_PRIVATE_KEY,:endpoint_url => 'https://eu-west-1.ec2.amazonaws.com/') # delete snapshots 30 days old age = 30 print "delete snapshots for vol ",ec2_vol,"\n" sa = ec2.describe_snapshots() sa.each do |s| aws_volume_id = s[:aws_volume_id] if aws_volume_id == ec2_vol then days_old = Integer((Time.now - s[:aws_started_at]) / (60*60*24)) if days_old > age then ec2.delete_snapshot(s[:aws_id]) print "snapshot ",s[:aws_id]," ",days_old," days old for vol ",s[:aws_volume_id]," deleted \n" else print "snapshot ",s[:aws_id]," ",days_old," days old for vol ",s[:aws_volume_id],"\n" end end end

EC2 snapshot performance
A volume created from a snapshot will be available for use immediately, but will be populated lazily from S3 in the background. However, if your application is hitting regions of the volume that are not yet pulled from S3, then performance can be impacted waiting for the data from S3 (it will be fetched immediately, but the path to S3 is slower than the path to the EBS volume).

Sharing a Snapshot
By default only your account can access a snapshot, however you can share it with a list of EC2 accounts, or share it publicly. You do this via the Modify Snapshot Attribute function. This is useful to allow data to be access by multiple accounts and to provide a public pre-build snapshot image for an EBS that could complement a public Amazon Machine Image.

In linux to see performance of disk use
iostat -x

Security

2009-04-08T07:06:00.000-07:00

Security on Amazon EC2 is basically at 2 levels:
1. Amazon provides various security features.
2. Various security features need to be configured in the instance you are running.

First, the Amazon features.

Amazon have an AWS Security WhitePaper that documents EC2s security environment.

a. Amazon promise that the Xen-based hypervisor provides instance isolation.
b. Amazon force you to use certificates and keys to access your instance via ssh.
c. The Amazon firewall (via security groups) by default denys all inbound traffic.

Secondly, configuring for security in the instance

a. Select an Amazon Machine Image - only use Amazon AMIs or Community AMIs from well know suppliers e.g. Rightscale or RPath. Also follow the guidelines on Sharing AMIs Safely

b. The primary means of securing access to an Machine is security groups.
- Currently I create a security for each instance as this helps identify them in AWS Management console.
- Create a ssh connection but restrict it to the IP address of your client.
- As well as specifying open ports you can specify other security groups that have access but currently cannot limit this by port number.
- Configure the minimum number of open ports

c. Configure security in your instance.
- Consider using the firewall of your instance as an additional firewall.
- Some application like databases allow you to configure IP addresses for clients.

For other topics such as VPN, Encryption, Securing Resources see:

Securing your data in the Cloud
Securing your data in the Cloud Slides
20 Rules for Amazon Cloud Security
Three tools to help you configure iptables
Amazon Web Services: Overview of Security Processes
Tips for Securing Your EC2 Instance
Securing n-tier and distributed applications on EC2
Sharing AMIs Safely

Networking Multi-Tier Applications

2009-04-02T03:05:00.000-07:00

Consider a multi-tier application that consists of
  1. Web First Tier e.g. Apache
  2. Clustered Middle Tier e.g. JBoss cluster
  3. Data Tier e.g. MySQL, Postgres SQL etc

Amazon uses dynamic IP addresses so running multi-tier applcations has some challenges.
Amazon instances have 2 IP address:
  1. A Public Address accessible from the internet.
  2. A Private Address accessible within Amazon that is used when instances communicate with each other.
An instance can have an Elastic IP Address so the address is static however there is a limit of 5 IP addresses available. These can be increase via this form

  Typically the web first tier has a static IP address and the DNS provider maps this static IP address to the domain name of the application.
  To handle the dynamic IP Addresses it is best to use hostnames and build a local hosts file from EC2 meta-data. Amazon hostnames are IP-<private IP Address> in the Rightscale Centos Images.
  The following ruby script called syncHost.rb builds a local hosts file from EC2 meta-data. It takes a prefix parameter so you can have test and production setup. (ie name all test servers test-xxx, and production say live-xxx and the script only builds a local host files for hosts that match the prefix and it uses the name without the prefix in the host file!).
The script also creates a file called local_hostname.sh that sets env variable LOCAL_HOSTNAME to the machine name that can be used by other scripts. (ie . ./local_hostname.sh)

#!/usr/bin/ruby
#
# fetch hosts from amazon EC2 meta-data
# Use the security group name as host name until tagging available
# Optional first parameter with hostname prefix
# To cope with dynamic scaling do this every x mins add to crontab
# Based on http://dysinger.net/2008/10/13/using-amazon-ec2-metadata-as-a-simple-dns
#

require 'rubygems'
require 'right_aws'
require 'net/http'
require 'resolv'
require '/root/settings'

# set if hosts must have a particular prefix this is an optional first argument
hostPrefix = ""
if ARGV[0] != nil
  hostPrefix = ARGV[0]
  puts "Matching hosts with prefix #{hostPrefix}"
end

url = 'http://169.254.169.254/latest/meta-data/instance-id'
instance_id = Net::HTTP.get_response(URI.parse(url)).body

# switch between the next 2 lines to either use vol-id passed as user data or
# a hard coded volume
url_user_data = 'http://169.254.169.254/latest/user-data'
ec2_vol = Net::HTTP.get_response(URI.parse(url_user_data)).body
# ec2_vol ='vol-xxxxx'

print "sync Hosts file\n"

ec2 = RightAws::Ec2.new(Settings.AMAZON_PUBLIC_KEY, Settings.AMAZON_PRIVATE_KEY,:endpoint_url => Settings.REGION)

original_hosts = File.open("/etc/hosts")
File.open("/etc/hosts.bak", "w") {|f| f.write original_hosts }

hosts = "127.0.0.1 localhost\n"
ec2.describe_instances.each do |r|
hostname = r[:aws_groups]
h = hostname[0]
if r[:aws_state] == 'running'
  # if host prefix then match the prefix and host is after prefix
   if hostPrefix.length > 0
    if hostPrefix.length < h.length and h[0,hostPrefix.length]==hostPrefix
     ip_address = Resolv::DNS.new.getaddress(r[:private_dns_name]).to_s
     hosts << ip_address+" #{h[hostPrefix.length,h.length]}\n"
    end
   else
    ip_address = Resolv::DNS.new.getaddress(r[:private_dns_name]).to_s
    hosts << ip_address+" #{h}\n"
   end
  end
end
File.open("/etc/hosts", "w") {|f| f.write hosts }

# create local_hostname shell script
ip_address = %x(hostname -i)
print "ip address " + ip_address
ip_address[ip_address.length-1] = ""
hostFile = "/etc/hosts"
hostFound = false
File.open(hostFile).each_line{ |s|
a = s.split(' ')
  if a[0] == ip_address
   hostFound = true
   print "hostname is " + a[1] + "\n"
   File.open('/mnt/local_hostname.sh', 'w') {|f| f.write("export LOCAL_HOSTNAME="+a[1]) }
   system('chmod 700 /mnt/local_hostname.sh')
  end
}
if hostFound == false
if hostPrefix.length > 0
  puts "ERROR: No hostname found for this server's IP address with hostname prefix #{ hostPrefix}"
else
  puts "ERROR: No hostname found for this server's IP address"
end
end

Then to start your multi-tier application:
1. Start all instances of the multi-tier application either via the AWS Management Console or a script that uses the RunInstances API call. Pass the EBS vol id as user data.
2. Logon to each machine and run
ruby syncHost.rb
and then start the application service (e.g the database )

Security: There is a good article on Securing n-tier and distributed applications on EC2. I have a similar setup for security as mentioned in this article.

Multi-Casting: Amazon does not support multi-casting (I guess to preserve bandwidth). If your application uses multi-casting (e.g JBoss Cache, monitoring tools) they will need to be reconfigured to use unicasting.

S3 for Backing up log files

2009-03-30T06:25:00.000-07:00

In situations where you cannot logon via ssh to a machine to see what is happening it is very useful to be able to view the application logs. Also if your application is not installed on an EBS then the log files will be lost when the machine is shut down.

There is a good utility s3sync that can rsync the log directory of you application. There are some other ways of doing this documented in the Userful Links.

Prerequisite is: Ruby AWS.

1. Download and untar the latest s3sync from s3sync

$ wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
$ tar xvzf s3sync.tar.gz

2. Make directory for ssl certificates and download some (important, read README for info about these SSL certs)

$ mkdir certs
$ cd certs
$ wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar

run this shell archive

$ sh ssl.certs.shar

get back into main s3sync dir

$ cd ..

3. create backupLogs.sh

#!/bin/bash # # Backup application logs # change /logDirectory/ to application log folder # NOTE: This script assumes that the environment variable LOCAL_HOSTNAME has been # set with the hostname of the machine # export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export SSL_CERT_DIR=/root/s3sync/certs # If using the EU domain uncomment next 2 lines #export AWS_CALLING_FORMAT=SUBDOMAIN #export EC2_URL=https://eu-west-1.ec2.amazonaws.com/ if [ -z "$LOCAL_HOSTNAME" ] then echo "ERROR: env variable LOCAL_HOSTNAME is not set" else echo "host is" $LOCAL_HOSTNAME echo date echo "backup log folder to S3 bucket :"$LOCAL_HOSTNAME ruby /root/s3sync/s3sync.rb -r --delete --ssl /logDirectory/ :/$LOCAL_HOSTNAME fi

4. In CloudBerry (which I recommend) , S3browser or S3fox, create the bucket.

5.Set up cronjob to run backup script every hour.

To backup at 10 past every hour

$ crontab -e

add the following line.

10 * * * * /path/to/backupLogs.sh

Note
to get SSL to work with latest Right AWS (version 1.10) I had to modify
s3sync_mod.rb s3sync script

# added the following line to bypass ssl problem

http.verify_mode = OpenSSL::SSL::VERIFY_NONE

before

http.start

Useful Links
s3sync readme
S3Sync
s3fs
Duplicity has s3 support
A List of Amazon S3 Backup Tools
Rsync to Amazon S3
SubCloud

Networking in EC2

2009-03-26T05:58:00.000-07:00

Instances running in EC2 have both a Public and Private DNS. Amazon allocates addresses dynamically when an instance starts.

Access from outside EC2 (eg HTTP and ssh) is via the Public DNS IP address. This can be set to a static address by using an Elastic IP address.

Security Groups provide a system wide firewall to restrict access to the machines.
Second level protection can be provided by a firewall on the instance or configuration of IP addresses in some applications.

Access between machines is via the Private DSN. As IP addresses are dynamic the best way is to create a hosts file and then do configuration via hostname. Then when an IP address changes the host file just needs to be updated.

A script can be written to use Amazon EC2 Metadata as a simple dns (see below).
Currently this examples uses the security group as the machine name. When Amazon introduces tagging (Coming Soon!) then the tag name can be used as the machine name. This script can be run every couple of minutes via a cron job for no cost.

In my experience of running multi-tiered applications you need to:
1. Start all Amazon instances for an application.
2. Update the local hosts file.
3. Start the applications in order (ie database first)

Amazon say that "bandwidth is at least 250Mbps, and gets as high as 1000Mb depending on your instance size and that this should be more than enough for most servers". Bandwidth is higher on the large machine instances compared to the small machine instances.

Use Amazon EC2 Metadata as a simple dns
I use the amazon metadata for creating /etc/hosts and do this on a cron schedule. This does everything I need. Instead of fancy DynDNS tricks or having to run and manage an internal DNS server I just have a ruby script that looks at the metadata ec2 to build /etc/hosts. It's easy. To set it up yourself and try it all you need are 3 easy steps.

Step 1 - Start each of your instances with unique security group that matches what you want their internal hostname to be. Such as "onion" or "potato" or whatever you want to call them.

Step 2 - Make sure you have ruby, rubygems and amazon-ec2 (rubygem) installed. Then create a ruby script in /usr/local/sbin/hosts that has the following:
#!/usr/bin/env ruby
%w(optparse rubygems EC2 resolv pp).each { l require l }
options = {}
parser = OptionParser.new do p
  p.banner = "Usage: hosts [options]"
  p.on("-a", "--access-key USER", "The user's AWS access key ID.") do aki
    options[:access_key_id] = aki
  end
  p.on("-s",
       "--secret-key PASSWORD",
       "The user's AWS secret access key.") do sak
    options[:secret_access_key] = sak
  end
  p.on_tail("-h", "--help", "Show this message") {
    puts(p)
    exit
  }
  p.parse!(ARGV) rescue puts(p)
end
if options.key?(:access_key_id) and options.key?(:secret_access_key)
  puts "127.0.0.1 localhost"
  EC2::Base.new(options).describe_instances.each do |r|
    hostname = r[:aws_groups]
    if r[:aws_state] == 'running'
      ip_address = Resolv::DNS.new.getaddress(r[:private_dns_name]).to_s 
      puts(ip_address+"     #{hostname}\n")
    end
  end
else
  puts(parser)
  exit(1)
end
Step 3 - Setup a cron job to update /etc/hosts as often as you like. I do it once per hour on all my machines

0 * * * * /usr/local/sbin/hosts -a myaccess -s mysecret >/etc/hosts

All my machines have this ec2 security key + script + cron approach. I do not have to run dyndns or any private dns servers to keep track of all my internal server ip addresses. My /etc/hosts looks like the following on the three machines in the test cluster:
127.0.0.1 localhost
10.252.202.221 oahu.ec2 oahu
10.253.115.175 maui.ec2 maui
10.253.114.190 hawaii.ec2 hawaii
(from http://tim.dysinger.net/2008/10/13/using-amazon-ec2-metadata-as-a-simple-dns)

Cloud Management Suppliers

2009-03-26T02:13:00.001-07:00

RightScale - The Big Daddy of Cloud Management. Limited free developer account. Expensive $500 a month but powerful scripting functionality if they support the systems you use - Centos/Ubuntu, MySQL, Tomcat, PHP.

Scalr - a fully redundant, self-curing and self-scaling hosting environment. Cheap at $50 a month but not as much functionality as rightscale. There is also an Open Source version you can run yourself if you want to.

enStratus

Kaavo - application-centric management solution. New to market.

Cacti - Open source free complete network graphing solution. Have a prebuilt image. Simple PHP/MySQL solution.

Hyperic HQ for Amazon Web Services. Free support for only 4 EC2 instances.

Zenoss - Commercial Open Source Application, Systems & Network Monitoring. Core free product has limited functionality.

Tapinsystems - system management in the cloud. New to market.

Cloudkick - free (currently) monitoring with email alerts. Over time the Amazon AWS Management Console should have more of the monitoring functions.

Scripting with Ruby

2009-03-20T06:50:00.000-07:00

It is possible to write scripts with Amazon EC2 in many scripting languages: bash, python, perl.

However Ruby seems to be a good choice:
- Ruby code is much more readable than perl or python and fully object orientated.
- Rightscale provide a good and frequently updated Amazon AWS interface called Right_AWS.
- Ruby runs efficiently on all Operating Systems.

Ruby AWS Gems
Two interfaces to AWS from Ruby:
a. Amazon EC2
b. Right AWS
I am using the Right AWS gem from RightScale

1. Ruby Language Documentation

The Pragmatic Programmer's Guide
Ruby Programming for the Absolute Beginner
The Ruby Course Slides
Quick links for Ruby Programmers
Ruby Tutorial
Ruby API Doc
Techotopia Ruby_Essentials
Ruby Tutorial

Ruby Forum

Right AWS Documentation
RightScale AWS Forums

2. Installing Ruby and AWS.

If you use a rightscale image then java and ruby will already be installed. I have found you just need to update existing gems and install the rightscale AWS interface.
   gem update
   gem install right_aws

Otherwise you need to install
a. Java 1.5 or higher (For CentOS see http://wiki.centos.org/HowTos/JavaOnCentOS)
b. The Amazon Tools
c. Ruby
   yum install -y ruby
   yum install -y ruby-devel ruby-docs ruby-ri ruby-irb ruby-rdoc
d. AWS Interface for Ruby
   gem install right_aws

To install the very latest version of RightAWS
a. download from http://rubyforge.org/projects/rightscale
b. install from the file:
   gem install right_aws-1.10.0.gem

3. Coding ruby scripts for AWS

Two parts to coding ruby for Amazon EC2:

a. Acessing metadata
The metadata is documented in Instance Metadata
and an example of accessing user-data is:

url_user_data = 'http://169.254.169.254/2008-02-01/user-data'
parameter = Net::HTTP.get_response(URI.parse(url_user_data)).body

b. Using the API
Ruby AWS follows the API so you can read the standard API docs see Amazon EC2 Query API
Typical first connect to EC2:

AMAZON_PUBLIC_KEY=<public key>
AMAZON_PRIVATE_KEY=<private key>
ec2 = RightAws::Ec2.new(AMAZON_PUBLIC_KEY, AMAZON_PRIVATE_KEY)

then issue commands (For example attach a volume):

vol = ec2.attach_volume(ec2_vol, instance_id, '/dev/sdf')

Some comands return values in hash objects that can be accessed as follows:

sa = ec2.describe_instances()
sa.each do |s|
  aws_image_id = s[:aws_image_id]
  aws_instance_id = s[:aws_instance_id]
  aws_state = s[:aws_state]
  print "image ",aws_image_id," instance ",aws_instance_id," State ",aws_state,"\n"
end

4. Scripts to do backup to S3
See s3sync.net

Monitoring EC2 Instances

2009-03-18T06:47:00.000-07:00

EC2Dream will create graphs based on Amazon CloudWatch. Amazon has a monitoring facility that can easily be switch on and off without any configuration on your server The service costs 1.5 cents per hour per EC2 instance (of any size) which comes out to $10.95 per month for an instance running 24x7.

Monitis and similar fee-based monitoring services.

Munin open source monitoring.

Nagios open source monitoring. A popular alerting system where you can monitor things like diskspace and set thresholds. An agent must be installed on the server and a Nagios server run.

Cacti is a popular open-source network monitoring tool. In Amazon EC2 there are pre-built images so it is easy to run. The following are the steps required to setup a simple test server and use cacti to monitor it.

1. Go to the AWS Management console and setup a cacti security group with SSH port 22, HTTP port 80.

2. Launch a prebuild cacti ami. ( In the AWS management console go to AMIs in navigation menu and search for cacti.)
Instance: ami-5e638737 Manifest: rbuilder-online/cacti-appliance-1.1.0-x86_19912.img.manifest.xml

3. In Amazon EC2 create a security group called testServer with SSH port 22, HTTP port 80 and custom udp port 161.

4. Find an image to use as a test server and launch it. In my case I am starting a CentOS image:
Instance: ami-cb52b6a2 Manifest: rightscale-us/CentOS5_2V4_1_10.manifest.xml

5. Login with to the test server instance with ssh (or putty) and setup SNMP
a. Install SNMP
  yum install net-snmp net-snmp-utils net-snmp-devel -y
b. Save original snmpd config.
  Go to /etc/snmp and cp snmpd.conf snmpd.conf.orig
c. edit snmpd.conf and set to
  com2sec local 127.0.0.1 private
  com2sec mynetwork x.x.x.x testServer

  where x.x.x.x is internal IP address of cacti instance.
d. Start snmp service
  service snmpd start
  chkconfig snmpd on

6. Use the public DNS of the cacti instance and go to that address in your browser.
ie http://ec2-72-44-36-84.compute-1.amazonaws.com/

7. Login as user admin, password admin and cacti will force a change of password.

8. Create device testServer in Cacti
a. Select Devices.
b. Click Add on the far right of the screen.
c. Enter:

Description:     testServer
Hostname:        enter Private IP address of testServer i.e. 10.250.138.211
Host template: ucd/net SNMP host.
SNMP Version:   Version 2
SNMP Comunity:  testServer

Press Create.
e. It will should a screen with a number of ncd/net graphs listed. Press Save

9. Create the Graphs for testServer.

a. Select New Graphs in the Menu.
b. Select host testServer.
c. Check all the graph names.
d. Press create.

10. Setup Graph Reporting Tree.

a. Click on Graph Trees in the Menu.
b. Click on Default Tree.
c. Click on Add in the far right.
select tree item type host
select testServer and press create.
d. testserver will appear on a page in a list. Press Save.

Note: Whenever graphs are changed for a device, the tree host must be saved
to refresh the Graph Tree.

11. Display Graphs.
a. Click on the "graphs" tab on the top of the screen.
b. Click on testServer in the tree on the left hand side of the page.
c. Also under localhost are graphs for the Cacti server itself.

12. Add Interface Traffic Graphs.

a. Click Device from menu and then click testServer and the settings for the testServer device are shown.
b. In the Associated Data Queries section, in the Add Data Query pulldown select "SNMP - Interface Statistics" and click "add".
c. Click "save" to save the device.
d. In the Menu select "New Graphs" and then select "testServer" from the pulldown.
e. You should now see a "Data Query [SNMP - Interface Statistics]" section.
f. Check all ther checkboxes and press "create".
g. From the Menu select "Graph Trees" and click on "Default Tree".
h. Click on ther host "testServer" and press "save". This updates the graph page to add the new "Interface Statistics" graphs.
i. Finally select "graphs" from the tab on the top right and after a few minutes you should see Interface Statistcs if there is network activty on testServer. (You will see an empty graph if cacti have successful connected to testServer and got some statistics).

Improving security of Cacti and SNMP
1. Change security group for the testServer for custom port 161 so it only supports the IP Address of the Cacti machine.
2. Change security group for cacti so http port 80 only comes from an IP address.
3. Set up SNMP V3 security

More Info
Cacti
Cacti Documentation
Cacti User Group
How to install on CentOS SNMP
SNMPv3: A Security Enhancement for SNMP

TroubleShooting
1. On the cacti instance take a look at the cacti log at /srv/cacti/log/cacti.log
2. Run snmpwalk. For example is the testServer is IP address 10.250.59.80 to see all the entries for cpu run the following:
snmpwalk -v 2c -c testServer 10.250.59.80 | grep -i cpu
Check if the values are empty or if there are any errors.
3. Check the system log of the testServer machine for SNMP messages. If CentOS look at /var/log/messages.
4. Force a poll for snmp data. Go to /usr/share/cacti/ and enter php pooler.php

Useful Links
Monitor your servers and ec2 instances with cacti
Monitoring Linux Server Performance
snmpwalk
SNMP Ready, Set, Patch!
Cacti Basic Howto

Contact

2009-03-18T04:28:00.001-07:00

I'm a consultant and system admin in the area of Cloud and Grid Computing as well as a Certified Java and Ruby Developer.

In the last few years I been self-employed and have setup systems for a number of clients on Amazon AWS as well used my expertise in Grid Computing using TIBCO GridServer or Platform Symphony at investment banks in the City of London.

I have a wealth of experience in the full technology stack including SQL Databases, Application Servers, Development tools and frameworks.

I have worked in many roles at software companies including technical consultant, software engineer, support, training and pre-sales consulting.

If I can help you please contact me at
neillwturner-at-gmail.com.

http://cardcloud.com/neillturner

Benefits

2009-03-12T02:36:00.000-07:00

"I’ll argue that non-differentiated services -- services that need to be good but aren’t the company’s competitive advantage -- are much more economically hosted in very high-scale cloud computing environments." James Hamilton, Amazon Web Services Vice President and Distinguished Engineer.

* The Pay-As-You-Go Model. There are significant advantages to this model for efficiency as one only pays for those resources one has actually consumed. If the application scales along the right revenue generating dimensions these costs will be in line with the revenue being generated.

* Managing Peak Capacity. Many IT organizations need to maintain extra capacity for anticipated peak loads, capacity that sits idle for most of the time.

* Higher reliability at lower cost. Negotiating several contracts with different datacenter and network providers to make sure the IT tasks can survive complex failure scenarios is a difficult task and many organizations find it hard to achieve this in a cost efficient manner. Amazon EC2 with its Regions and Availability Zones gives its customers access to several high-end datacenters with highly redundant networking capabilities at a single pricing model, without any negotiations.

References
The Economics of the Cloud

Amazon EC2 Overview

2009-03-11T09:08:00.000-07:00

* Above the Clouds: A Berkeley View of Cloud Computing - A very good overview of Cloud Computing.
* What is Amazon Web Services?
* Using Amazon Web Services
* Amazon Elastic Compute Cloud Introduction
* Web Scale Computing
* Cloud computing with Amazon Web Services
* Autoscaling Web Services on Amazon EC2
* Amazon AWS (EC2,S3) environment Overview
* Cloud Computing from a Developer’s Perspective
* IBM developerWorks Cloud Computing

See Amazon EC2 Command Line Tools Reference for technical info.

Elastic Block Storage

2009-03-11T08:59:00.000-07:00

The best detail explanation on EBS is at
http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/

The important feature of an EBS is that it allow changes to be persisted between
instance starts.

Main difference between EBS and S3:
1. EBS cannot be shared between instances
2. EBS is much faster than S3

See EBS Tutorial #3 in Getting Started with AWS Management Console

To automatically mount an EBS at Linux machine startup
1. Add the following mountebs.rb script to /root inserting your private and public key and make sure it has execute privilege.

#!/usr/bin/ruby # script to auto mount ebs at startup require 'rubygems' require 'right_aws' require 'net/http' url = 'http://169.254.169.254/latest/meta-data/instance-id' instance_id = Net::HTTP.get_response(URI.parse(url)).body # set the following keys AMAZON_PUBLIC_KEY='<public key>' AMAZON_PRIVATE_KEY='<private key>' # switch between the next 2 lines to either use vol-id passed as user data or # a hard coded volume url_user_data = 'http://169.254.169.254/latest/user-data' ec2_vol = Net::HTTP.get_response(URI.parse(url_user_data)).body # ec2_vol ='vol-xxxxx' print "auto mounting ",ec2_vol,"\n" # remove the last parameter if not in EU region ec2 = RightAws::Ec2.new(AMAZON_PUBLIC_KEY, AMAZON_PRIVATE_KEY,:endpoint_url => 'https://eu-west-1.ec2.amazonaws.com/') print "attach ",ec2_vol,"\n" vol = ec2.attach_volume(ec2_vol, instance_id, '/dev/sdf') print "result ",vol,"\n" # It can take a few seconds for the volume to become ready. # This is just to make sure it is ready before mounting it. # sleep for up to 5 minutes waiting for it to be attached aws_attachment_status = "" max_sleep = 300 current_sleep = 0 while (aws_attachment_status != "attached") and (current_sleep<max_sleep) sleep 5 current_sleep = current_sleep + 5 sa = ec2.describe_volumes(ec2_vol) sa.each do |s| aws_attachment_status = s[:aws_attachment_status] end end if aws_attachment_status == "attached" message = "vol "+ec2_vol+" attached\n" print message system('logger '+message) print "mount /dev/sdf /mnt/data01\n" system('mkdir /mnt/data01') system('mount /dev/sdf /mnt/data01') else message = "vol "+ec2_vol+" did not attach\n" print message system('logger ',message) end

2. In /etc/rc.local add 'ruby /root/mountebs.rb'
3. In AWS Management Console when launching an instance show the advanced options
and enter in the user data the volume id

Note: for the script to work ruby, ruby_gems, and right_aws must be installed.
(i.e. gem install right_aws)
These are available in the RightScale public images.

Also the script will not mount an unformated EBS. The very first time you need to:

mkfs -t ext3 /dev/sdf
mkdir /mnt/data01
mount /dev/sdf /mnt/data01

Tip: To share an EBS put on and instance with an NFS server and mount the NFS on the other instances.

Also see Running MySQL on Amazon EC2 with Elastic Block Store
Experiences with Amazon EC2 and EBS

Performance: Amazon say that disk performance of EBS should be similar to instance data performance. Some people report performance similar to instance data performance except for small images where performance is 13% less because of the lower bandwidth of small images. Other people report performance 5% less that instance data for all image sizes.
For more info see IO Performance on EBS.
Disk IO and throughput benchmarks on Amazon EC2.

Increasing the size of an EBS
reszing cannot be done "on the fly" however you can create a larger EBs with minimum down time see Amazon EBS - How to Grow the Storage. So it is best to create an EBS with a good amount of storage for your application.
Rightscale show how to use their scripts to increase the size of the EBS on a MySQL Slave and then promote it to Master. See Grow the Size of an EBS Volume

Finally, you are limited to 20 open EBS but can incease this with this form

Tools

2009-03-11T08:33:00.000-07:00

EC2Dream Graphical System Admin is a single install for a number of tools necessary for Amazon EC2 System admin. This saves installing the tools separately and the tools are all configured to work together.

The EC2Dream System Admin installs on Windows, Linux or MacOS X with the following tools:

Ruby - scripting tool. Also Right AWS, Capistrano and other Ruby Packages. EC2Dream is itself written in Ruby.

PuTTY - Windows SSH client for terminal access to linux servers.

WinSCP - Windows SCP client to copy files to linux servers.

LaunchRDP - Launch Windows Remote Desktop from commandline.

S3Sync - command line and rsync for Amazon S3.

PowerShell - Windows scripting shell and language. PowerShell 1.0, used by Windows XP and Windows Server 2003, does not support remote scripting so use Remote Desktop to run the scripts on the windows server. PowerShell 2.0 supports remote scripting but is only available with Vista, Windows 7 and Windows Server 2008.

PowerShell Community Extensions provides very useful extra commands

Power-EC2Dream - a C# interface to Amazon EC2 and S3 (and Eucalyptus) that can be used from PowerShell to write scripts against Amazon EC2 (and Eucalyptus). It includes the ThreeSharp - S3 .Net Interface.

For a Windows Client It is also recommended to install the following additional tools:
CloudBerry, S3browser or S3Fox - console for S3. Cloudberry seems to have the most functionality and I would recommend it. I could not get S3Fox to recognise folders so I didn't find it that useful.

Amazon S3

2009-03-11T08:22:00.000-07:00

Amazon Simple Storage Service (S3) is a way to store unlimited amounts of data that can be shared by your amazon instances or accessed from your client machine.
It is important because all images of your amazon ec2 instance need to be stored in S3.
NOTE: S3 has no concept of folders. It is purely an object store with a key. You can simulate folders by using a / in the key and Cloudberry and S3Browser supports this convention.

Console for S3
Currently the AWS Management console does not support S3 so you need to install:
a. CloudBerry - Windows Only (requires .NET 2.0) but better support for simulating folders with slashes. This has the more functionality than S3Browser and this is what I would recommend. The user interface is similar to the winscp browser and even supports moving data between one amazon user and another without incuring charges plus using Cloudfront.
b. S3Browser - Windows Only (requires .NET 2.0) but better support for simulating folders with slashes.
c. S3Fox - Firefox Plugin to provide access to S3 from a browser.
d. S3Hub - MacOS Only.

S3CMD
To provide a command line access to S3 install the S3cmd tool from http://s3tools.org/s3cmd
Note: This requires python to be installed and for windows don't use the PGP security option.

Best Practices
There is a good best practices on using Amazon S3 here

Also see the amazon tutorial

Convention for storing AMIs
In the EC2Dream Sys Admin tool by default we use a bucket to store your AMIs in called -ami.
When an ami is bundled I would store in a folder called machine-YYMMDD.
ie So the database machine bundled on the 10 March 2009 would go to
-ami/DB-090310/

Getting Started

2009-03-11T06:41:00.000-07:00

Four possible ways to get started with Amazon EC2 depending on what admin tool you use:

1. EC2Dream System Admin A client app for admin. Follow the Documentation. It has the following sections:
  1. Setting Up
  2. Creating an Environment
  3: Running an Instance
  4: Building a Base Image
  5: Building an Application Image

Also see the Slides on Slideshare

2. AWS Management Console - a web console from any browser. Follow the AWS Management Console Tutorial document or the Amazon Getting Started Guide

Amazon also have a video presentation of the console here.

3. Elasticfox - a plug in for Firefox.
Install and follow Elasticfox Getting Started Guide

4. Amazon EC2 Command Line Tools however this uses line commands so it is not user friendly.