status page that many people use to pull stats into tools such as Cacti and other RRDTool based stats packages. This works well but does not always provide enough details, questions such as these remain unanswered:
To answer this I wrote a script that keeps a running track of your Apache process, it has many fine grained controls that let you fine tune exactly what to keep stats on. I got the initial idea from an old ONLamp article titled Profiling LAMP Applications with Apache’s Blackbox Logs.
The article proposes a custom log format that provides the equivelant to an airplanes blackbox, a flight recorder that records more detail per request than the usual common log formats do. I suggest you read the article for background information. The article though stops short of a full data parser so I wrote one for a client who kindly agreed that I can opensource it.
Using this and some glue in my Cacti I now have graphs showing a profile of the requests I receive for the whole site, but as you are able to apply fine grained controls to select what exactly you’ll see, you could get per server overview stats and details for just a specific scripts performance and statuses:
The script creates on a regular interval a file that contains the performance data, the data is presented in variable=value data pairs, I will soon provide a Cacti and Nagios plugin to parse this output to ease integration into these tools.
The performance data includes values such as:
See the Sample Stats for a good example, variables are pretty self explanatory. To keep the data set small and manageable 2 selectors exist, one to choose which requests to keep details for and which to keep stats for. These can be combined with standard Apache directives such as Location to provide very fine grained stats for all or a subset of your site.
You would need some glue to plug this into Cacti and Nagios, I will provide a script for this soon as I have time to write up some docs for it.
Install guide etc can be found on my Wiki there is also extensive Perdoc Docs in the script, the Wiki also have links to downloading the script, the latest is always available here
Source: www.devco.net
Easy per-machine custom facts for Puppet
As this is the first time I am posting about Puppet, maybe a few words about it first. Puppet is a configuration management system that helps you manage large infrastructures. There are ofcourse many similar things, cfengine and lcfg to name just two. I like Puppet as it has a big and active community but it is also a modern approach to the problem being that it has some properties of Object Orientation and so forth.
I’ve used cfengine - actually chose it over Puppet when it was still young - but that is over a year ago now and I if you have many or even 2 of the same nodes to maintain I urge you to check out Puppet.
So the rest is only really useful if you already know Puppet, sorry if this will bore many people 
On a site I worked we used to have a file on each machine that defines a few Facter facts, this was very handy and I wanted to do something similar because I define variables like $country=”uk” and so forth in my nodes which then helps my manifests builds correct ntp configs for example. Now doing it with a global variable in the node is ok and all, but not nearly as sexy as letting each node know where they are in via facts.
In the old days hacking in new facts was a right pain and you possibly even needed to maintain your own package of facter, not so anymore, it’s in fact really nice now.
Puppet now lets you drop bits of ruby code into a directory and then sends the code off to clients on demand for integration into Facter, I won’t go into setting it up the Puppet Wiki goes into it at length. Instead I’ll show a simple bit of code to read in /etc/facts.txt on a node and present the values in it as facts.
var = nil
value = nil
facts = {}if File.exists?(”/etc/facts.txt”)
File.open(”/etc/facts.txt”).each do |line|
var = $1 and value = $2 if line =~ /^(.+)=(.+)$/
if var != nil && value != nil
facts[var] = value
var = nil
value = nil
end
endfacts.each{|var,val|
Facter.add(var) do
setcode do
val
end
end
}
end
This code essentially just reads stuff out of key=val pairs in /etc/facts.txt and adds them to facter. Drop that into your facts directory on the master in a file ending in .rb and that should be it, with facts files on your nodes the values will now be available for use in your manifests.
Maybe not the best Ruby code as it’s my first ever Ruby code and mostly based on a sample I found in the Puppet book, I would have liked to do the Facter.add() call right in the loop that parses the file instead of using a array, but I couldn’t get that going.
Anyway, now I just distribute /etc/facts.txt to all my machines, and they know where they are and I have a simple no-hassle method for teaching machines about new things.
Source: www.devco.net
Online Regex Testing
Back in 2004 I posted about The Regex Coach, its a great app that I still use today, however it only really works on Windows so I have been looking for some alternatives.
There is a really great resources called Regular-Expressions.info it even has a cheap tool that you can use to do something similar to The Regex Coach. Today Lifehacker mentioned RegExr, its a great web app but also has standalone versions for Windows, Linux and OS X, full of sample regular expressions, good explanations of what a regular expression parses as etc, it is a perfect replacement for The Regex Coach, worth checking it out!
Source: www.devco.net
British Citizenship
I previously mentioned that got a letter confirming it all went well with my application for naturalisation, the whole process is now more or less done.
I had the ceremony last Thursday and around 11:24 in the morning the Mayor of Greenwich handed me my certificate so I am now all done with that and a full Citizen of the United Kingdom. I arrived here on the 2nd of Feb 2002 and became a citizen on the 7th of Feb 2008. I could have applied last year in March already and probably would have been done with it all around September but I was procrastinating and eventually the noise about the reforms in the immigration laws gave me the kick I needed to complete it.
The biggest advantage I’ll see immediately is of course the passport, traveling as a South African - or in fact being a South African out of South Africa - is such a liability your whole life is just tough, massive headache of visas, immigration time wasting etcetc, endless hassle. In tourist visas alone I spent about GBP500 in the last few years never mind all the time wasted in getting those and even just in queuing in the non EU citizen lines at airports, all gone now! I’ve also had to struggle quite a lot with tenancy agreements for flats that I rent etc as I was never sure if I’ll even be in the country for the year they want you to sign, so had to always get 6 month break clauses put in etc.
This is a part of the certificate I received during the ceremony:
Today I’ll apply for my first UK passport, it should come through in about 2 weeks unfortunately just too late to attend FOSDEM.
The process for applying for citizenship is all hyped up to be this fantastic experience for applicants, a great introduction to the country and its people. This is done through the test you need to pass and a formal ceremony that even includes singing God Save The Queen.
Overall I’d say the whole thing just left me cold, personally I see little point to most of the hoops I had to jump through. I have to say though that the test has some value - it tests that you have a grasp of English and in that function its a success so I’d keep it for that reason. The ceremonies though? waste of time and money in my eye.
Source: www.devco.net
Easy transparent PHP input filtering
I have been working on a site that will have potentially quite a few random third parties accessing it and inserting data into a MySQL database. I am thus quite keen on a good solid input filtering method for PHP to prevent things like XSS and SQL Injection.
There are several options out there, of the ones I found Inspekt is about the closest match to my way of working, it essentially imports $_GET, $_POST etc and wraps them in an object which you then use to access variables in a filtered method. It by default then NULLs the original variables so you cannot access them anymore, if backward compatibility is desired it can leave the originals untouched. Not optimal as this gives an unsafe by default result if you want to maintain backwards compatibility.
Another problem with this approach is that it is a lot of work to change existing code, which you might thing is just par for the course but I was convinced I need to find a way to do so more transparently.
I could for example at program start just walk through the $_GET etc arrays and apply some filtering to them using addslashes() and such but this is very restrictive, what if you need to get it unfiltered, especially if you perform destructive filtering? How would you go about filtering some variables for phone numbers, some for email addresses etc?
The answer lies in PHP’s new Standard Programming Library, specifically in its ArrayAccess interface, which if you don’t care for older versions of PHP is the way to go.
The basic advantage of this is that you can expose properties of your objects by using array notation rather than object notation:
$result = $foo->getBar();
compared to:
$result = $foo[’bar’];
Both statements give access to the private variable $bar just using different syntax. So using this technique we can write a transparent filter for input variables, the basic usage of the final library would be something along these lines:
$_GET = new ArrayArmor($_GET);
print (”Filtered Variable:$_GET[test]
\n”);
print (”Unfiltered Variable: ” . $_GET->getRaw(”test”));
A possible output from this script can be seen below:
Filtered Variable: 1234\’;delete from accounts;–
Unfiltered Variable: 1234′;delete from accounts;–
You can see that the default behavior is to protect the input but even for destructive filtering methods the raw unfiltered data would be available if the programmer needed it. You can provide all sorts of extra methods to validate emails, post codes and such.
A quick and dirty example of a class that provides this kind of filtering can be seen below:
class ArrayArmor Implements ArrayAccess {
private $original;
function __construct (&$variable) {
$this->original = $variable;
}
function offsetExists($offset) {
return isset($this->original[$offset]);
}
function offsetGet($offset) {
return addslashes($this->original[$offset]);
}
function offsetSet($offset, $value) {
}
function offsetUnset($offset) {
}
function getRaw($offset) {
return($this->original[$offset]);
}
}
?>
So that’s it, a simple method that is very easy to put into existing code. This is clearly not a full example as addslashes() is hardly the be-all and end-all of input protection, but if you build on this you can get a very easy to use and flexible input filter that is safe by default.
Source: www.devco.net
RSS feed for comments on this post · TrackBack URI
Leave a reply