Agile DevOps - Test-Kitchen and EC2Dream

I have a dream... that we have collaborative open source tools and use Cloud and DevOps to rapidly build infrastructure just like developers use agile development.....EC2Dream is a graphic user interface that provides a 'single pane of glass' to do agile devops primarily on cloud servers by using:
-Rapid agile iterative development of chef cookbooks, puppet modules or ansibe playbooks to build servers.
-Test Kitchen to deploy and test them.
-Cloud Formation or OpenStack Heat using cfndsl Ruby DSL for easy orchestration.
-Amazon AWS, Google Compute Engine, Openstack, Rackspace, HP Cloud, Local and Hosted servers.

Lightweight DevOps

"One of my mantras - focus and simplicity. Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains." - Steve Jobs

Lightweight DevOps is NOT using Chef Server or Puppet Master to pull running updates regularly on your servers. Instead use:
  • Puppet apply, chef solo/zero or ansible to config servers by pushing changes to servers using tools like test-kitchen or pulling changes from AWS S3.

  • Puppet/chef/ansible in development to create generic images that deploy applications in live from say S3 or build docker containers.

  • Immutable infrastructure where ever possible - replace servers instead of updating them.

  • Only one git repository for all environments.

  • Puppet forge, chef community or ansible galaxy to minimise the amount of config management to write. Contribute if there is no generic cookbook/module/playbook for your software component.

  • Tools like test-kitchen to run verification tests.

It works best with a cloud where it can replace some of the functions of Chef Server or Puppet Master:
  • Know where your servers are and what their roles are instead of chef server or puppermaster. (eg AWS Console).

  • Orchestrate the servers with a dsl on top (eg cloudformation with cfndsl).

  • Monitor the servers, manage the log files and generate alarms (e.g. AWS CloudWatch).

  • Run your databases as services (e.g. AWS RDS).

Try and keep it simple and build as little as possible yourself as this will make you as efficient as possible.


References
DevOps is Not a Feature!
De-Centralise and Conquer: Masterless Puppet in a Dynamic Environment
Why You Should Be Using Nodeless-Masterless Puppet
Testing infrastructure with Test Kitchen
Scaling Puppet in EC2
Keep Chef out of your Docker containers
Chef Cookbook Continuous Integration