Richard's Diary

Saturday, January 24, 2009

SSH Keys with Capistrano With help from
  1. Install capistrano and run "capify ." in your app's root directory
  2. As root on target machine, "useradd admin" and give it with proper sudo priveleges. In keeping with best practices, note that this user does not have a password for login so we will create a set of keys for this purpose
  3. On local machine, preferably .ssh directory, "ssh-keygen -t rsa" to generate a keypair:
    1. giving it not the default name but a modified name so it is clear what user/host it is for.
    2. Do not enter a passphrase since you will have to enter it each time you use the key
    3. remove write permissions from the pair
    4. Push it to remote machine
      scp -i id_rsa-gsg-keypair .ssh/
  4. As root on remote machine, su to admin:
    1. create ~/.ssh directory. The owner of the directory is admin, not root, but we will need to make sure this directory is only RWX by admin
    2. touch .ssh/authorized_keys so that this file is owned by admin. Again, we will need to ensure that this file is only RWX by admin
    3. exit from the admin user
  5. As root again, cat /home/admin/ >> .ssh/authorized_keys since this file is only readable by root. Admin user can now read the public key
  6. rm /home/admin/ to cleanup

  1. you should now be able to
    ssh -i .ssh/id_rsa-admin
  2. In order for Capistrano to ssh, add to config/deploy.rb
    ssh_options[:keys] = [File.join(ENV["HOME"], ".ssh", "id_rsa-admin")]
    1. Whatever local machine runs capistrano tasks needs to have the private key "~/.ssh/id_rsa-admin"

Friday, January 23, 2009

Nine Month Mark Coming up on 3 months left to my original deadline of having something up, here is a summary of what I've accomplished in the last 3 months...

In a word - ADMIN.

It's the guts of getting things done in Linux. Networking, running services, databases/file systems. Sprinkled between all these dry subjects, mainly because it needed to be done but it also helped to maintain my sanity, was a couple of trips, taxes, business startup admin and reading.

I feel good about what I've accomplished and can definitely taste the light at the end of the tunnel now, whereas I wasn't so sure 3 months ago. What needs to happen now is:
  1. Deployment
  2. Monitoring
  3. Imports
  5. Endless features/optimization
More Subversion Tested using a remote SVN Host so that Capistrano can work
  1. Created a local repository under /path/to/repo/trunk/projectname so that there could be branches and tags directories under /path/to/repo/
  2. Ran svnsync init --username myname file:///path/to/repo (it asked for password)
  3. Then ran svnsync sync --username myname
    SVN reads the latest version number at remote repo and then transmits all revisions since then to remote repo (this time it didn't ask for password)
Upgrading to Rails 2.2. Managing vendor libraries in
file:///usr/share/doc/packages/subversion/html/book/svn-book.html#svn.advanced.vendorbr is helpful for understanding the steps involved. Unfortunately, rake rails:freeze:gems deletes the existing vendor/rails directory and good to know. Some sample svn commands

Sunday, January 18, 2009

Rails Deployment with Passenger
  1. Install Apache2
    1. /etc/sysconfig/apache2
      1. configure modules loaded
      2. add APACHE_CONF_INCLUDE_FILES="httpd.conf.local"
    2. /etc/apache2/httpd.conf.local
      1. specify ip that apache listens on
      2. log rotation file:///usr/share/apache2/manual/logs.html.en
        apache2 continues to write to same log file unless the process is restarted. while one can do "apache2ctl graceful", it still causes unnecessary load when we can pipe logs.
      3. ErrorLog "|/usr/sbin/rotatelogs2 /var/log/apache2/error_log.%Y-%m-%d-%H_%M_%S 10M"
      4. CustomLog "|/usr/sbin/rotatelogs2 /var/log/apache2/access_log.%Y-%m-%d-%H_%M_%S 10M" combined
    3. vhosts.d/domain.conf
      1. specify ip of VirtualHost so that server is not dependent on DNS lookup
      2. specify domain of VirtualHost so people can't just assign domain to my ip and create bogus traffic that app can't respond to or if it can, will end up creating duplicate content
      3. RailsEnv development|production
      4. PassengerUseGlobalQueue on
      5. Override the default root Deny All for static assets by carving out the app/public folder excluding cgi and rb files and unwanted directories
    4. /etc/apache2/httpd.conf shows Options None, AllowOverride None as a Directory / (root) default so we don't need to "Options -Indexes"
    5. Default is "HostnameLookups off"
    6. configure access logs fields in /etc/apache2/mod_log_config.conf
      LogFormat "%a %t \"%r\" %>s %b %D %v \
      \"%{Referer}i\" \"%{User-Agent}i\"" combined
  2. Install Passenger
    1. add lines recommended by Passenger to httpd.conf.local
  3. Log naming and definition
    1. System /var/log/messages (iptables, ssh, sudo, command history) via logrotate /etc/logrotate.conf
    2. Rolling over Apache access and error logs (done)
    3. Rails log is DocumentRoot/log/#{environment}.log. I was able to change path to /var/log (done)
      1. Rails console prints to application server. It used to be mongrel console but now it's passenger console. Passenger prints everything to Apache error log and because it lives above any virtual host, it logs to the default Apache server error log. While initially this seemed like a problem, it's good practice to route the console log to the application log as development matures.
    4. Postgresql log by configuring postgresql.conf (done)
  4. Log management
    1. Run rotatelogs on system logs
    2. Run cron to rsync
    3. Run cron to delete the files edited over a week ago?



Tuesday, January 13, 2009

More AMI Bundling
  1. disable password based login for root and all users via ssh (already works if you leverage Amazon AMI)
  2. make sure all accounts are unable to login by looking at shadow file
  3. for any machine, make sure you remove unneeded public keys from ~/.ssh/known_hosts or authorized_keys because anyone who possesses the private key could connect to local machine
  4. make sure private keys are readable only by owner and all keys are not writeable so they aren't accidentally modified, otherwise you need to need to resend public key but the most annoying thing about it is being unable to login when you want to
  5. create PostgreSQL and LOG volumes, attach them, format them and mount them per
  6. yum install postgresql-server
  7. configure pg_hba.conf per
  8. configure postgresql.conf per
  9. configure /etc/sysconfig/postgresql for paths
    POSTGRES_OPTIONS=">>/usr/local/pgsql/data/logfile 2>&1"
  10. change /etc/init.d/postgresql to smart shutdown
  11. chkconfig --level 345 postgresql on

Monday, January 12, 2009

AMI Bundling I created a bundle to basically save my admin work since it gets lost when an instance is terminated. It was a good practice experience as well.
  1. added keepcache=1 to /etc/yum.conf to cache rpm packages
  2. yum update
  3. Follow instructions from to install postgresql
  4. Follow instructions from Rails Deployment with Passenger to install Apache, Passenger, Gems
  5. installed s3sync
  6. s3synced /var/cache/yum/update-newkeys/packages to S3
  7. shutdown -r now
  8. bundle volume (all you need)
    1. you will get an error that the 2.6.21 kernel doesn't support rsync lutimes. need to wait for .22
    2. you will get an error that meta-data product codes could not be found, which i had already noted on previous occasions
  9. upload image to S3
  10. ec2-register picky-ricky/image.manifest.xml
    IMAGE ami-d6b354bf
  11. ec2dim -o self
    IMAGE ami-d6b354bf picky-ricky/image.manifest.xml 122430389923 available private i386 machine aki-a71cf9ce ari-a51cf9cc

Sunday, January 11, 2009

Amazon EBS Every 10G is $1/month
  1. ec2-create-volume -s 10 -z us-east-1a
    VOLUME vol-a9cf2bc0 10 us-east-1a creating 2009-01-12T02:01:55+0000
  2. ec2-describe-volumes vol-a9cf2bc0
    VOLUME vol-a9cf2bc0 10 us-east-1a available 2009-01-12T02:01:55+0000
  3. ec2-attach-volume vol-a9cf2bc0 -i i-f1cc4a98 -d /dev/sdh (this is like physically connecting device)
    ATTACHMENT vol-a9cf2bc0 i-f1cc4a98 /dev/sdh attaching 2009-01-12T02:08:01+0000
  4. yes | mkfs -t ext3 /dev/sdh (Create file system)
  5. adduser postgres;mkdir /usr/local/pgsql/data
  6. mount /dev/sdh /usr/local/pgsql/data/
  7. df -h to see storage
    /dev/sdh 10G 2M 10G 1% /usr/local/pgsql/data
  8. umount /usr/local/pgsql/data/
    1. A volume must be unmounted inside the instance before being detached
  9. ec2-detach-volume vol-a9cf2bc0
    ATTACHMENT vol-a9cf2bc0 i-f1cc4a98 /dev/sdh detaching 2009-01-12T02:08:01+0000
  10. ec2-delete-volume vol-a9cf2bc0
    VOLUME vol-a9cf2bc0
  11. ec2-describe-volumes to confirm deleted
Amazon S3 Note the security precautions in (Mounting proc is no longer necessary in
  1. clear history of root commands (in particular note when S3 password appears on command line)
  2. ssh keys deleted in /root/.ssh and /etc/ssh
  3. root cannot login (already done if you leverage an existing Amazon AMI)
  4. root password is randomized (already done if you leverage an existing Amazon AMI)

Ruby s3 utils
  1. amazon library which doesn't stream
    conn ="user","pass")
  2. right_aws is from nov 08 and streams
  3. s3sync/s3cmd is from june 08 and streams
  4. aws-s3 is from july 08
  6. rsh3ll is from 2006 and doesn't let you specify location
  7. ruby-aws is a library for affiliate api

Thursday, January 8, 2009

Amazon Hosting Setup
  1. Create AWS account and enable S3 and EC2 services
  2. Create ec2 directory locally
    1. create X.509 certificate and place files with keys in ec2 directory. Make sure private key is only readable by owner
    2. install ec2 API scripts in directory and set environment variables for keys' path, ec2 home and add ec2 binary to PATH
  3. create keypair (ec2-add-keypair gsg-keypair) so that my instance can only be launched by me. copy private key (22 lines) to a owner readonly file in directory where one usually runs ec2-run-instances
  4. ec2dim -o amazon to find all Amazon owned images as a starting point
  5. ec2-run-instances ami-5647a33f -k gsg-keypair
    RESERVATION r-141eb67d 122430389923 default
    INSTANCE i-34f4735d ami-b44bafdd pending gsg-keypair 0 m1.small 2009-01-09T04:54:59+0000 us-east-1a aki-a71cf9ce ari-a51cf9cc
  6. ec2din i-34f4735d
    RESERVATION r-141eb67d 122430389923 default
    INSTANCE i-34f4735d ami-b44bafdd domU-12-31-39-02-5D-93.compute-1.internal running gsg-keypair 0 m1.small 2009-01-09T04:54:59+0000 us-east-1a aki-a71cf9ce ari-a51cf9cc
  7. Configure firewall by setting up groups
    ec2-authorize default -p 22
    ec2-authorize default -p 80
  8. ssh -i id_rsa-gsg-keypair or use
  9. Made an edit to file and saw it show up immediately in
  10. shutdown -h now terminates the instance
Amazon Web Services TC 12/8/2008

  1. Refresh Amazon content every month
  1. One sided indemnification, with no control
  2. Accidental termination for cause could result in the loss of data
  3. Modifications to terms at any time and through website posting

Saturday, January 3, 2009

Created Subnet With Internet Connectivity Using NAT I am very happy
  1. To statically setup ip and subnet of interface on (this creates the network of 2 computers)
    1. Linux host #1 (which will function as a Linux router since it has a second interface connected to another network - the internet)
      ifconfig eth0 netmask up
    2. Linux host #2
      ifconfig eth0 netmask up
    3. when i pinged each machine they recognized and responded to each other (because iptables -L --verbose showed that each machine accepted ICMP echo-request input and accepted all output. iptables also shows the number and size of packets that matched for each rule)
  2. to forward all packets in and out on internet interface (this establishes internet connectivity for the newly created network, namely host #2 since the router already has internet)
    1. echo 1 > /proc/sys/net/ipv4/ip_forward
    2. iptables -A POSTROUTING -t nat -o ath0 -s -d 0/0 -j MASQUERADE
    3. iptables -A FORWARD -o ath0 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
    4. iptables -A FORWARD -i ath0 -m state --state ESTABLISHED,RELATED -j ACCEPT
    5. added the following to /etc/resolv.conf so that host #2 could resolve domains, leveraging the name server assigned to router
      search (this could be anything)

© 2010 Picky Ricky, Inc. originalblog