Singularity, an Alternative Openstack Guest Agent

Introduction

Recently, the openstack guest agent (nova-agent) hasn't been working up to par on Gentoo systems. As an excercise and for a bit of fun, I've re-written the agent functionality from scratch. The attempt is located on GitHub and currently works correctly on Gentoo systems. Others have volunteered to create appropriate configurators for other distributions as well as test the framework on these distributions.

Design Considerations

Singularity was designed to meet the following considerations:

  • maintainable and extensible code
  • use system libraries but leverage virtual environments for segregation
  • framework for pluggably configuring a system based on information from the hypervisor
  • configurators are focused on one task and kept small
  • configurable; particular functions can be disabled individually

Conclusion

Singularity should be stable enough to use in production but without appropriate testing this claim feels hollow. The plan is to mark it 1.0 sometime within the next week unless major issues are reported or noticed.

New Overlay Member: httpry

I've spent a few moments and added a live ebuild of httpry (an HTTP traffic analyzer) to my overlay. This utility prints out HTTP requests in real time coming through the server in the same format one would get from apache2 fullstatus. The homepage for this project is http://dumpsterventures.com/jason/httpry and a stable package can be expected in the overlay quite soon.

New Overlay Members: openstack and holland

I've recently made a push to clean the packages provided by my overlay. I've purged older packages and re-visited repoman's guidance on package design as well as updated all python builds to properly utilize the distutils eclass.

Some of the more notable changes to the overlay include the following:

  • Live Holland Ebuilds
    • Added live ebuilds for all Holland Backup Framework ebuilds
    • There are currently known collisions between the base holland ebuild and the holland-lib-common ebuild
  • Live Openstack Ebuilds
    • Added a live ebuild for the Openstack Nova Project
    • Work needs to be done to setup proper configuration items
    • A configure function needs to be added to the ebuild to setup the base items outlined in various Openstack guides
    • An init script should be crafted that allows control over the system

New Project: alunduil-overlay

Introduction

A Gentoo overlay for personal projects as well as interesting things that aren't found in eix-remote when I sit down to use them. Typically, this includes minor patches, use flags, or other minor items.

Installing alunduil-overlay

  1. install layman
  2. add http://www.alunduil.com/svn/portage/trunk/alunduil-overlay.xml to the overlay variable in /etc/layman/layman.cfg
  3. add the overlay using layman: layman -f -a alunduil-overlay

Issue Reporting

Please, report any issues with the ebuilds provided to bugzilla.alunduil.com under the alunduil-overlay product.

Cheat Sheet: Apache's mod_rewrite

Introduction

I was recently requested to document common Apache rewrite pitfalls and examples. As a result, I have crafted the following document. It is intended as a two page document where the first page is a reference guide of commonly used rewrite variables and flags; the second page is a short list of examples, gotchas, and troubleshooting advice.

Rewrite Cheat Sheet

Common Variables

HTTP Headers

  • HTTP_USER_AGENT
  • HTTP_REFERER
  • HTTP_COOKIE
  • HTTP_FORWARDED
  • HTTP_HOST
  • HTTP_PROXY_CONNECTION
  • HTTP_ACCEPT

Connection & Request

  • REMOTE_ADDR
  • REMOTE_HOST
  • REMOTE_PORT
  • REMOTE_USER
  • REMOTE_IDENT
  • REQUEST_METHOD
  • SCRIPT_FILENAME
  • PATH_INFO
  • QUERY_STRING
  • AUTH_TYPE

Server Internals

  • DOCUMENT_ROOT
  • SERVER_ADMIN
  • SERVER_NAME
  • SERVER_ADDR
  • SERVER_PORT
  • SERVER_PROTOCOL
  • SERVER_SOFTWARE

Date and Time

  • TIME_YEAR
  • TIME_MON
  • TIME_DAY
  • TIME_HOUR
  • TIME_MIN
  • TIME_SEC
  • TIME_WDAY
  • TIME

Specials

  • API_VERSION
  • THE_REQUEST
  • REQUEST_URI
  • REQUEST_FILENAME
  • IS_SUBREQ
  • HTTPS

Variable Descriptions

REQUEST_FILENAME:
 The full local filesystem path to the file or script matching the request, if this has already been determined by the server at the time REQUEST_FILENAME is referenced. Otherwise, such as wehn used in virtual host context, the same value as REQUEST_URI.
HTTPS:Will contain the text "on" if the connection is using SSL/TLS, or "off" otherwise. (This variable can be safely used regardless of whether or not mod_ssl is loaded.)

Flag Descriptions

nocase|NC:This makes the test case-insensitive — differences between uppercase (i.e. A, B, C, &c) and lowercase (i.e. a, b, c, %c) are ignored, both in the expanded TestString and the CondPattern.
last|L:Stop the rewriting process immediately and don't apply any more rules.
proxy|P:Force the substitution URL to be internally sent as a proxy request.
qsappend|QSA:Appends any query string created in the rewrite target to any query string that was in the original request URL.
redirect|R[=code]:
 Forces an external redirect, optionally with the specified HTTP status code.
forbidden|F:Returns a 403 FORBIDDEN response to the client browser.

Good Examples

Add www to the domain of all requests:

RewriteCond %{HTTP_HOST} !^www [NC]
RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]

Force all requests to HTTPS:

RewriteCond %{HTTPS} off
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]

Redirect a specific subweb to another domain:

RewriteRule ^/?subweb/(.*) https://other.example.com/$1 [R,L,QSA]

Block specific IPs from access:

RewriteCond %{REMOTE_ADDR} ^127.0.0
RewriteRule ^ - [F,L]

Creating a filesystem alias with mod_rewrite:

RewriteRule ~/?alias/(.*) /var/www/vhosts/$1/httpdocs/$1 [L,R]

A condition that stops CMS software from over-riding fullstatus (added before the offending rewriterule):

RewriteCond %{REQUEST_URI} !server-status [NC]

Bad Examples

Recursive rewrites:

RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]

Gotchas

Some rewrites may conflict with existing rewrites provided by CMS packages (i.e. wordpress, drupal, &c). Check for any existing rewrites in an .htaccess file.

RewriteRule and RewriteCond can only be used in the following contexts:

  • server config
  • virtual host
  • directory
  • .htaccess

Common Troubleshooting

  • Enable logging with RewriteLog and RewriteLogLevel; example:

    RewriteLog <file path>
    RewriteLogLevel 3 # range: 0 — 9
    
  • Check the regular expressions with a PCRE checker; many can be found on the Internet

  • Utilize curl to test redirects, R, curl -I example.com

Extending Android Battery Life

Recently, I've stumbled across an application in the android market, Juice Devender, that does exactly what it advertises. The application extends battery life through automated micromanaging of peripherals and software. This combined with another application, Juice Plotter, allows one to actually experiment with different battery preservation techniques (or different ROMs for rooted phones) and compare the results.

More specifically, the application behaves much like any other power management software in Linux installations. It controls the frequency scaling of the processor (if the phone is rooted), it controls networking interfaces and screens to help conserve power. All of this functionality can easily be configured and modified for your desired power settings, but the defaults will provide a significant boost to battery life. This is one more useful utility in a long list that allows us to get the most out of our phones.

Automating Android Sound Profiles

Recently, I've found the wonderful android application, Profile Valet. This little application handles location and time based profiles for various settings on your Android device. It can control things like sound, wifi, and bluetooth based on time schedules or locations (GPS backed). Using a utility like this can help prevent the common, "I forgot to unsilence my phone," situation; resulting in missed calls, texts, or emails. It can also be used to control silencing your phone for certain events. Simply setup a profile for various movie theaters to silence your phone automatically when you enter that area.

By automating the little things in life we can open up our minds with less stress and turn our attention to more fun or important matters. I know this concept has been introduced by the **Getting Things Done** paradigm but it can be expanded in the digital age of smartphones even further than simply organizing one's life. It can be expanded to allow our devices to be the tool they were designed to be.

An Explanation of LVM Snapshots

Introduction

It seems that disk snapshots have become a hot topic and a confusing topic. I intend to simply outline what snapshots look like in terms of the lower layers of abstraction and nothing more. Snapshots are built into things like LVM, SAN, &c but I will not be covering those technologies. Instead, what I intend to clarify is how snapshots work in LVM.

Lower Layers

Layer 1: The Hard Disk

A hard disk is historically split into several pieces called partitions. These logically separate the blocks on a disk and allow us to make sure that disks are not over-allocated for one particular purpose or lose access to a system if a critical partition (anything with run in the name) is filled to capacity.

Layer 2: LVM Physical Volume

The partitions that disks are carved into typically get turned into LVM physical volumes. There is a one to one mapping between partitions on hard drives (or whole disks as they don't need to be partitioned) and physical volumes. LVM physical volumes connect the underlying disk structure to the organization that LVM is going to provide.

Layer 3: LVM Volume Groups

The physical volumes we create can be grouped together into volume groups, groups of extents (the smallest unit of disk in LVM) that can then be re-allocated to logical volumes.

Layer 4: LVM Logical Volumes

By carving our pools, volume groups, up into particular chunks we have come full circle and are using our disk (now a few layers removed) as a group of extents that get partitioned again. At this layer the partitions are logical volumes and can be allocated to particular purposes just like the partitions we started with.

It should be evident that we have created small units of contiguous extents (not contiguous physical blocks) that map through the aforementioned layers back to physical blocks on a disk somewhere. This abstraction allows us to perform the magic involved with LVM snapshots.

LVM Snapshots

For certain purposes (database backups, &c), it is necessary to have the disk being backed up in a consistent state. If we want these conditions it usually involves stopping any services that write to that region of the disk and ensuring the changes have been flushed and that nothing further will write to that region. Stopping production services for backup purposes is a possibility if you've got the cash to create a redundant environment but this becomes a much simpler problem when using LVM snapshots.

An LVM snapshot is a temporary region of disk set aside to preserve the consistency of the region it's a snapshot of. When an LVM snapshot is created the following happens:

  • LVM reserves space for the original blocks that get changed by writes; this is the snapshot space (name and size in the LVM command)
  • LVM creates a new device (e.g. /dev/mapper/vg-snapshot) to access the disk contents at the snapshot's creation time
  • When a write occurs on the original disk the original extent gets written into the snapshot area

The reason that LVM snapshots don't use a differencing disk is speed. With this setup for snapshots we have instant creation and instant deletion:

creation:Since we only create a new device node that passes through to the original logical volume for anything not contained in the snapshot, access continues normally until things start getting written to the logical volume. Once writes have happened we can short circuit this pass-through with the extents in the snapshot that map to locations in the logical volume.
deletion:Since only the original, un-interesting extents are ever stored in the snapshot it doesn't contain any long-lived information that needs to be applied back to the original disk and can simply be dropped without concern.

Of course, this means that when we are reading this data or writing to the logical volume, we have extra operations to perform that didn't occur before the snapshot existed:

read:Check that the extent requested hasn't been added to the snapshot location and read from the logical volume if this is the case.
write:Write the new extent to the logical volume but also write the original contents and location of that extent into the snapshot region.

Conclusion

LVM snapshots are a convenient way of getting a consistent view of a particular logical volume while doing backups or other sensitive-to-write operations. One caveat to keep in mind (besides the ever so slight write performance drop), is the size of the snapshot dictates how much information can be written to the logical volume before the snapshot is useless and is no longer valid. Keep the amount of writing you expect for the logical volume in mind when determining the size of the snapshot you're creating.

Optimizing Gentoo: CFLAGS II

Introduction

As I mentioned in Optimizing Gentoo: CFLAGS, you want to be comfortable with the existing documentation on CFLAGS before going crazy trying to optimize your CFLAGS on your system. With the wrong CFLAGS things can and will randomly stop working or possibly building. Thus, it's a good idea to have a good understanding of what you're doing to the code when you modify these sacred parameters.

Let's build on what we accomplished last time and get any ebuilds that are machine instruction set aware building those sections as well as the changes you made to the CFLAGS last time.

Finding USE Flag Names

How can we find the flags that we want to add to our /etc/portage/package.use or /etc/make.conf files? As always BASH is our friend and can be used to find this answer for us:

. /etc/make.conf && gcc -Q -c -v ${CFLAGS} --help=target | grep enabled

This displays the currently enabled flags based on your CFLAGS parameter from /etc/make.conf and allows us to find which flags have USE flags with the following:

gawk '/-m.*/ { print $1 }' | cut -d 'm' --complement -f 1 | xargs -i{} equery h "{}"

The resulting USE flags can be added to /etc/portage/package.use or /etc/make.conf as your needs dictate.

Conclusion

Using a little scripting we can extract the USE flags that correspond with the instruction sets we enabled last time.

Using Memcached with MediaWiki and Wordpress

Introduction

Memcached is a simple key/value memory store that uses keys to perform fast lookups of values stored in memory. Most languages have simple bindings that make this very easy from a programming perspective. This also means that it's not quite as simple to configure as APC.

Installation and Configuration

I recommend using your choice of installation mechanism (or your distribution's choice; mine's emerge) to install Memcached.

The configuration usually consists of an RC configuration parameter file that specifies how much memory to allocate and a port to listen on. In Gentoo this file is located in /etc/conf.d/ as usual but look for it in /etc/sysconfig on RHEL -like distributions.

Update the configuration to listen on the interface of your choice, use the amount of memory that's appropriate for your application and system, &c. That's all there is to Memcached and you're now ready to start the service.

Configuring Wordpress to Use Memcached

Note

If you don't already have the PHP bindings for Memcached installed, install them with your package manager or with PECL.

I've previously mentioned the W3 Total Cache Wordpress plugin but unfortunately this plugin doesn't allow you to easily configure Memcached usage through the web interface. It defaults to using the Memcached server located at 127.0.0.1:11211 which isn't appropriate for a remote installation of Memcached. We need to modify DOCROOT/wp-content/w3-total-cache-config.php so that the Memcached servers point to our remote server's location before enabling it in the interface. The following sed command can make this change for us:

sed -i -e 's/127.0.0.1:11211/HOST:PORT/g' DOCROOT/wp-content/w3-total-cache-config.php

Configuring MediaWiki to Use Memcached

Note

Just like Wordpress, MediaWiki needs the PHP bindings for Memcached pre-installed.

Setting up MediaWiki with Memcached is a simple matter and only requires an edit of its LocalSettings.php file. We need to add the following section to enable Memcached:

## Shared memory settings
$wgMainCacheType = CACHE_MEMCACHED;
$wgParserCacheType = CACHE_MEMCACHED;
$wgMessageCacheType = CACHE_MEMCACHED;
$wgMemCachedServers = array("giskard.alunduil.com:11211");

$wgSessionsInMemcached = true;

These settings not only enable Memcached but also specifies the set of Memcached servers to use and whether or not to put sessions into Memcached.

If you do use multiple Memcached servers and want to apply a weighting factor you can do this with nested PHP arrays:

$wgMemCachedServers = array(array("host:port", weight), array("host2:port2", weight));

Conclusion

Using Memcached can greatly reduce the load placed on databases behind webapps or other data storage and retrieval systems. This usually improves the perceived load on the web application itself but not necessarily.

Contents © 2014 Alex Brandt - Powered by Nikola