Compiling Python

Python is great in that you can extend the language using pip to download libraries, but if you want to run your code on a number of servers, you have to ensure that they all have the required libraries.

One way around this is to compile your code so that it runs native.  The example below reads data from a Mongo database and prints it in a pretty format.

jinkersont@testdockerhost:~/python$ cat Mongo.py
import pymongo
import pprint
from pymongo import MongoClient
muser = "dba"
mpassword = "haha"
mhost = "mongodb1.hehe.hoho"
mport = 27017
uri = "mongodb://%s:%s@%s:%s" % ( muser, mpassword, mhost, mport )
client = MongoClient(uri)
db = client['evergreen']
print db.list_collection_names()
collection = db['customers']
pprint.pprint(collection.find_one())
jinkersont@testdockerhost:~/python$

To run this we require the pymongo and pprint libraries to be installed.

apt install python-pip
pip install pymongo pprint

The output looks like this:

jinkersont@testdockerhost:~/python$ python Mongo.py
[u'customers']
{u'_id': ObjectId('5df8ed88d819a6f218b58bc9'),
 u'age': 25.0,
 u'cars': [u'Audi R8'],
 u'name': u'Honey'}
jinkersont@testdockerhost:~/python$

You can compile this code using pyinstaller

pip install pyinstaller
pyinstaller --onefile Mongo.py

Then you can run the code using:

jinkersont@testdockerhost:~/python$ file dist/Mongo
dist/Mongo: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, for GNU/Linux 2.6.32, BuildID[sha1]=294d1f19a085a730da19a6c55788ec08c2187039, stripped
jinkersont@testdockerhost:~/python$ dist/Mongo
[u'customers']
{u'_id': ObjectId('5df8ed88d819a6f218b58bc9'),
 u'age': 25.0,
 u'cars': [u'Audi R8'],
 u'name': u'Honey'}
jinkersont@testdockerhost:~/python$

Am I running on a VM?

This is a regular question - have I logged in to a physical linux server or is this a virtual machine?

One simple way of finding this out (if you have root access) is to check the Chassis in dmidecode. Here are a few examples:

myuser@myopennebulahost:~$ sudo dmidecode -s chassis-manufacturer
Dell Inc.
myuser@myopennebulahost:~$

myuser@buildserver:~$ sudo dmidecode -s chassis-manufacturer
HP
myuser@buildserver:~$

myuser@apacheserver:~$ sudo dmidecode -s chassis-manufacturer

QEMU

myuser@apacheserver:~$

We can see that the OpenNebula server is a Dell, and the build server is an HP. But the Apache server is QEMU, which is a common virtualization hardware emulator. Clearly the Apache server is a VM.

Edit:
I've noticed that some Red hat VMs return Red Hat, not QEMU
[root@freepbx ~]# dmidecode -s chassis-manufacturer
Red Hat

[root@freepbx ~]#

SELinux

Install the following packages:
apt-get install selinux-basics selinux-policy-default auditd

vi /etc/selinux/config and set to enforcing

My bread recipe

Just leaving this here so that I can find it again.

Ingredients

• 2 Co-op  yeast sachets
• 450ml (3/4 pint) warm water
• 700g (1.5lb) Co-op Strong white bread flour
• 1 heaped teaspoon of salt
• 3 heaped teaspoons of caster sugar
• 40 grams of butter or equiv
• Milk to glaze

Requires

• Mixing bowl
• Knife 
• Two baking tins
• Baking parchment

Method:

1. Mix yeast, flour, salt and sugar in a large mixing bowl.
2. Rub in the butter
3. Add water and mix in using a knife until it is workable
4. Tip out on to a lightly floured work surface and need for 15 minutes until it is stretchy
5. Return to bowl, cover and leave in a warm room for 1 hour to rise.
6. Grease your baking tins and line with baking parchment. 
7. Heat oven to 220C
8. Tip out the risen dough on to the work surface and need for a further 5 minutes. 
9. Divide the dough in two and form rough loaf shapes, place in the lined baking tins.
10. Cover and return to warm room for 15 minutes to prove.
11. Cut a slit length-ways in each loaf and glaze with milk
12. Bake for 35 minutes until golden brown. Check that the bread is cooked by tapping the bottom of the loaf, it should sound hollow.
13. Allow to cool on a baking rack.

The full Coin Flip Porgram

I got the Coin Flip program to work! Note that there are a few methods in here that I have implemented but not used, this is fairly normal. So for instance when writing the Coin class, it seemed obvious to be able to turn a coin over, not just flip it, but I don't need that right now.

require 'optparse'
class Coin
  attr_reader :face, :value
  @@FaceRange = [0, 1]
  @@ValidValues = [1, 2, 5, 10, 20, 50, 100, 200]
  @@prng = Random.new # Pseudo-random Number Generator
  # Default Face to an illegal value so that it will flip the coin
  def initialize (value=10, face=-1) # CONSTRUCTOR
    self.value = value
    self.face = face
  end
  def face= (new_face)        # Set a valid face
    @face ||= 0 # default
    if @@FaceRange.include? new_face
      @face = new_face
    else
      self.flip
    end
  end
  def value= (new_value)      # Set a valid value
    @value ||= 1 # default
    @value = new_value if @@ValidValues.include? new_value
  end
  def set_face (new_face)     # Set face to a chosen value
     @face = new_face if [0, 1].include? new_face
  end
  def turn                    # Turn the coin over
     @face = (@face + 1).modulo(2)
  end
  def flip                    # Flip the coin
    @face = @@prng.rand(0..1)
  end
  def to_s # Over-ride the to_s method inherited from the Object class
    "Value is #{@value} Face is #{@face}"
  end
end

class CList
  attr_reader :size
  def initialize (size=10)
    @size = size
    @coins = []
    (1..size).each do |i| 
      @coins.push(Coin.new)
    end
  end
  def flip
    @coins.each do |coin|
      coin.flip
    end
  end
  def countheads
    heads = 0
    @coins.each do |coin|
      heads += 1 if coin.face == 1
    end
    heads
  end
  def to_s
    @coins.each do |coin|
      puts coin
    end
    "Total #{@coins.length}"
  end
end

options = {}
OptionParser.new do |opts|
  opts.banner = "Usage: clist.rb [options]"

  opts.on("-v", "--[no-]verbose", "Run verbosely") do |v|
    options[:verbose] = v
  end
  opts.on("-c", "--coins [CCOUNT]", "Number of coins to flip") do |c|
    options[:count] = c.to_i
  end
  opts.on("-f", "--flips [FCOUNT]", "Number of flips") do |f|
    options[:flips] = f.to_i
  end
end.parse!

ccount = options[:count] # Number of coins
fcount = options[:flips] # Number of flips

puts "Flipping a row of #{ccount} coins #{fcount} times"

my_list = CList.new(ccount)
hoccurs = Array.new(ccount)
(0..hoccurs.length).each { |i| hoccurs[i] = 0 } # Zero Array
(1..fcount).each do 
  my_list.flip
  hoccurs[my_list.countheads] += 1
end

(0..hoccurs.length-1).each do |i|
  times="times"
  times="time" if hoccurs[i] == 1
  puts "Flipped #{i} heads #{hoccurs[i]} #{times}" if hoccurs[i] > 0
end

Here is how to run it: If we want 10 coins and 40 flips:

$ ruby clist.rb -c 10 -f 40
Flipping a row of 10 coins 40 times
Flipped 2 heads 2 times
Flipped 3 heads 4 times
Flipped 4 heads 10 times
Flipped 5 heads 10 times
Flipped 6 heads 9 times
Flipped 7 heads 5 times

tjinkers@TJINKERS-LAP ~/ruby
$

But it's much more fun with bigger numbers than that

Learning Ruby - First Object

Learning how to construct an object in Ruby, this is what I have so far.

class Coin
  attr_reader :face, :value
  @@FaceRange = [0, 1]
  @@ValidValues = [1, 2, 5, 10, 20, 50, 100, 200]
  @@prng = Random.new # Pseudo-random Number Generator
  # Default Face to an illegal value so that it will flip the coin
  def initialize (value=10, face=-1) # CONSTRUCTOR
    self.value = value
    self.face = face
  end
  def face= (new_face)        # Set a valid face
    @face ||= 0 # default
    if @@FaceRange.include? new_face
      @face = new_face
    else
      self.flip
    end
  end
  def value= (new_value)      # Set a valid value
    @value ||= 1 # default
    @value = new_value if @@ValidValues.include? new_value
  end
  def set_face (new_face)     # Set face to a chosen value
     @face = new_face if [0, 1].include? new_face
  end
  def turn                    # Turn the coin over
     @face = (@face + 1).modulo(2)
  end
  def flip                    # Flip the coin
    @face = @@prng.rand(0..1)
  end
  def to_s # Over-ride the to_s method inherited from the Object class
    "Value is #{@value} Face is #{@face}"
  end
end

coin1 = Coin.new(20, 1)
puts coin1.face, coin1.value
puts coin1
coin1.set_face(0)
puts coin1

coin2 = Coin.new(50, 0)
puts "Coin 2 is #{coin2}"
coin2.set_face(0)
puts "Coin 2 is #{coin2}"
puts "Turn it"
coin2.turn
puts "Coin 2 is #{coin2}"
puts "Turn it"
coin2.turn
puts "Coin 2 is #{coin2}"

puts "Flip it"
coin2.flip
puts "Coin 2 is #{coin2}"
puts "Flip it"
coin2.flip
puts "Coin 2 is #{coin2}"
puts "Flip it"
coin2.flip
puts "Coin 2 is #{coin2}"
puts "Turn it"
coin2.turn
puts "Coin 2 is #{coin2}"

coin3 = Coin.new(5)
puts "Coin 3 is #{coin3}"

coin4 = Coin.new
puts "Coin 4 is #{coin4}"
puts "Flip it"
coin4.flip
puts "Coin 4 is #{coin4}"


$ ruby  coins2.rb
1
20
Value is 20 Face is 1
Value is 20 Face is 0
Coin 2 is Value is 50 Face is 0
Coin 2 is Value is 50 Face is 0
Turn it
Coin 2 is Value is 50 Face is 1
Turn it
Coin 2 is Value is 50 Face is 0
Flip it
Coin 2 is Value is 50 Face is 1
Flip it
Coin 2 is Value is 50 Face is 1
Flip it
Coin 2 is Value is 50 Face is 1
Turn it
Coin 2 is Value is 50 Face is 0
Coin 3 is Value is 5 Face is 1
Coin 4 is Value is 10 Face is 1
Flip it
Coin 4 is Value is 10 Face is 0

tjinkers@TJINKERS-LAP ~/ruby
$

Perl: Am I Root?

Testing to see if you are running with root privileges is straight-forward in Perl. Here is some sample code:

[tjinkers@ebl-oel6vm1 perl]$ cat ./testroot.pl
#!/usr/bin/perl

if ( $< eq 0 ){
   print "I'm root\n";
} else {
   print "Not root\n";
}
[tjinkers@ebl-oel6vm1 perl]$ ./testroot.pl
Not root
[tjinkers@ebl-oel6vm1 perl]$ sudo ./testroot.pl
I'm root
[tjinkers@ebl-oel6vm1 perl]$

As the saying goes, "That was easy!"


And the follow-up question - how do you check when using Bash

# Make sure only root can run our script

if [[ $EUID -ne 0 ]]; then

  echo "This script must be run as root" 1>&2

  exit 1

fi

Jar file 101

Here is a very simple example. I have a very simple single class application called Frame2.
The manifest file tells jar which class file has the "Main-Class" directive to run. Note that this file needs to end with a new line character.

PS C:\Users\tjinkers\java\swing> ls Frame2.*
Directory: C:\Users\tjinkers\java\swing

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 22/01/2014 15:37 693 Frame2.class
-a--- 22/01/2014 10:20 377 Frame2.java

PS C:\Users\tjinkers\java\swing> cat .\manifest.txt
Main-Class: Frame2
PS C:\Users\tjinkers\java\swing> jar cvfm Frame2.jar manifest.txt Frame2.class
added manifest
adding: Frame2.class(in = 693) (out= 471)(deflated 32%)
PS C:\Users\tjinkers\java\swing> ls Frame2.*

Directory: C:\Users\tjinkers\java\swing

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 22/01/2014 15:37 693 Frame2.class
-a--- 22/01/2014 17:04 948 Frame2.jar
-a--- 22/01/2014 10:20 377 Frame2.java

PS C:\Users\tjinkers\java\swing> ./Frame2.jar
PS C:\Users\tjinkers\java\swing>

iptables - the Linux firewall

I needed to test to see how a package would behave if a certain server was not available. I decided that the easiest way to do this was to use a firewall. As I haven't done much with firewalls previously, I though I should write it up.

To see what you have configures, use:

[root@ebl-oel57vm1 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  144.20.15.203        0.0.0.0/0

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
[root@ebl-oel57vm1 ~]#
[root@ebl-oel57vm1 ~]# iptables -A INPUT -s 138.1.187.71 -j DROP
[root@ebl-oel57vm1 ~]# ping 138.1.187.71 | head -3
^C
[root@ebl-oel57vm1 ~]#

This tells me that I have 3 different "chains" configured (a chain of rules), but only the INPUT chain has anything configured. It tells me that I will not accept requests from 144.20.15.203

This wasn't exactly what I wanted, I wanted to stop my server contacting this IP address, so I needed to do the following:

[root@ebl-oel57vm1 ~]# ## Revert back to the start
[root@ebl-oel57vm1 ~]# iptables -F INPUT
[root@ebl-oel57vm1 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
[root@ebl-oel57vm1 ~]# ## Define an outbound firewall rule
[root@ebl-oel57vm1 ~]# iptables -A OUTPUT -d 138.1.187.71  -j REJECT
[root@ebl-oel57vm1 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  0.0.0.0/0            138.1.187.71        reject-with icmp-port-unreachable
[root@ebl-oel57vm1 ~]#
[root@ebl-oel57vm1 ~]# ping 138.1.187.71 | head -3
PING 138.1.187.71 (138.1.187.71) 56(84) bytes of data.
From 10.79.243.156 icmp_seq=1 Destination Port Unreachable
From 10.79.243.156 icmp_seq=1 Destination Port Unreachable
[root@ebl-oel57vm1 ~]#

Notice how when I blocked the IP Address on the inbound side my requests out didn't receive a reply, whilst when I blocked on the outbound side I got the message that the port was unreachable.

Pings
To stop your server responding to a Ping:
iptables -A INPUT -p icmp --icmp-type echo-request -j DROP
To stop your server from pinging another server
iptables -A OUTPUT -p icmp --icmp-type echo-request -j DROP

Editing IPTables
iptables-save > myfile
edit myfile
iptables-restore < myfile
rm myfile

 

Displaying files in date order

Here is an example of a program that reads all the files directory and displays them in date order.

#!/usr/bin/perl
use strict;

my ($MYDIR, $mtime, $filename,%filearray, $key);
my $LOGDIR="/mytest/logs";

sub numerically { $filearray{$a} <=> $filearray{$b}; }

# Read all the file names in to an array
opendir $MYDIR, $LOGDIR  or die "Unable to open file: $!\n";
my @allfiles = readdir $MYDIR;
closedir $MYDIR;
# For each file, get its modification time and store
print "Unsorted\n";
for (@allfiles) {
   next if $_ eq "." || $_ eq "..";
   $filename = $_;
   $mtime = (stat("/mytest/logs/" . $filename))[9];
   printf "\t$mtime\t\t$filename\n";
   # populate a hash called %filearray with a
   # key of the file name and a value of the mtime
   $filearray{$filename} = $mtime;
}

print "Sorted\n";
foreach $key (sort numerically ( keys(%filearray))) {
  print "\t$filearray{$key}\t\t$key\n";
}

Simple Perl

Perl is a highly complicated language, but it is possible to do some useful things using some pretty simple code. Here are a few examples.

Example 1
Here is a simple example of a program that reads a file and prints it out. I'm sure that it could be written with fewer lines of code, but this has the advantage of being a framework for any program you might need to write.

$ cat testfile
micky mouse
mini mouse
daffy duck

tjinkers@JINKERSON-PC ~/example
$ cat ./filelister.pl
#!/usr/bin/perl
use File::stat;
my $IFILE = $ARGV[0];
die "Unable to open input file\n" unless (open $inputfile, $IFILE);
while (<$inputfile>) {
        chomp;
        print $_ . "\n";
}
close  $inputfile;
tjinkers@JINKERSON-PC ~/example
$  ./filelister.pl testfile
micky mouse
mini mouse
daffy duck

tjinkers@JINKERSON-PC ~/example

Things to note:

• $ARGV[0] is the first command line argument
• 'die ... unless' - most programming languages put the condition before the action, but Perl is not fussy, and so here we tell the program to 'die' (end with an error condition) unless it is able to open its input file.
• 'chomp' - if the final character of a string is a carriage return / line feed, remove it. Like so many Perl instructions, if you don't specify the variable, it uses the result of the previous operation, stored in $_
• 'print $_ . "\n"' - print the result of the previous operation to stdout, concatenated with a new line character.

Example 2
Here is a  re-write of the above example, this time using a subroutine.

#!/usr/bin/perl
use strict;
use warnings;
sub processfile {
   die "Unable to open input file\n" unless (open my $inputfile, $_[0]);
   while (<$inputfile>) {
        chomp;
        print $_ . "\n";
   }
   close  $inputfile;
}

main {
   my $IFILE = $ARGV[0];
   processfile($IFILE);
   exit 0;
}

Things to note:

• use strict; forces you to get the syntax write - always good practice!
• We either need to declare the subroutine ahead of the main section, or include a dummy declaration above it.

Example 3
Here is another example, this time using a subroutine that returns a value, otherwise known as a function.

#!/usr/bin/perl
use strict;
use warnings;
use Scalar::Util qw(looks_like_number);
sub addthem {
   my $var=0;
   my $sum=0;
   foreach $var ( @_ ) {
     print $var . "\n";
     $sum=$sum+$var if (looks_like_number($var));
   }
   return $sum;
}
main {
   print addthem(@ARGV) . "\n";
   exit 0;
}

 Things to note:

• The subroutine is called within the print statement.
• The variable passed to the subroutine is an array of values which we rattle through, adding them up.
• We are using the Scalar::Util utility to determine if the value passed is numeric. This utility contains a number of useful functions that are not quite popular enough to include in the main language definition.

Example 4
Here is a more complex example. The request was to write a piece of code that would read through a file. If it found 2 predetermined lines next to each other, the code had to insert the contents of a file beneath the second line. If the lines were not next to each other in sequence, they were to be ignored.

#!/usr/bin/perl
use File::stat;
use IO::File;
use warnings;
use strict;

die "Could not open stdin\n" unless ( open my $stdin, "< &STDIN");
my $trigger = "safety";
while (<$stdin>) {
        chomp;
        my $line = $_;
        print $line . "\n";
        if ( $line eq "line 2" ) {
                $trigger = "primed";
        } else {
                if ( $line eq "line 3" && $trigger eq "primed" ) {
                        ## print "Fire!\n";
                        die "Could not open include file\n" unless ( open my $ifile, "./insertfile.txt");
                        while (<$ifile>) {
                                chomp;
                                print $_ . "\n";
                        }
                        close $ifile;
                }
                $trigger = "safety";
        }

} 

File System Manipulation

Extending a 'normal' fdisk partition

The Task
We have a server with ext3 files systems. The file systems for root (/) and usr (/usr) are 4GB and need to be increased to 8GB.

1. use fdisk to create a new partition of the correct size
2. dd if=/dev/sda5 of=/dev/sda10
3. e2fsck -f /dev/sda10
4. resize2fs /dev/sda10
5. e2label /dev/sda5 "oldroot"
6. Reboot the system

Note: The above example assumes that you are using labels in /etc/fstab. If you are using UUID you will have to edit /etc/fstab. Actually - UUIDs are copied across, so we need to work a little harder on these.

Solaris 11 Package Updates

Using a Solaris 11 Repository
Solaris 11 has moved to a package management system that is similar to yum, in that is pulls from a network and manages dependencies.
The command used is pkg - enter pkg --help for a full list of options.

root@solaris:~# pkg publisher
PUBLISHER                             TYPE     STATUS   URI
solaris                               origin   online   http://pkg.oracle.com/solaris/release/
root@solaris:~# pkgrepo info -s http://pkg.oracle.com/solaris/release/
PUBLISHER PACKAGES STATUS           UPDATED
solaris   4292     online           2011-11-09T15:42:00.183118Z
root@solaris:~#
root@solaris:~# pkg update
No updates available for this image.
root@solaris:~#

Before you install a package you can dry-run it:

root@solaris:~# pkg install -nv gcc-3
           Packages to install:         2
     Estimated space available:   9.67 GB
Estimated space to be consumed: 219.43 MB
       Create boot environment:        No
Create backup boot environment:        No
            Services to change:         1
          Rebuild boot archive:        No

Changed packages:
solaris
  developer/gcc-3
    None -> 3.4.3,5.11-0.175.0.0.0.2.537:20111019T093736Z
  developer/gnu-binutils
    None -> 2.19,5.11-0.175.0.0.0.2.537:20111019T095434Z
Services:
  restart_fmri:
    svc:/application/texinfo-update:default
root@solaris:~#

The you go ahead and install the package:

 root@solaris:~# pkg install -nv gcc-3
           Packages to install:         2
     Estimated space available:   9.67 GB
Estimated space to be consumed: 219.43 MB
       Create boot environment:        No
Create backup boot environment:        No
            Services to change:         1
          Rebuild boot archive:        No

Changed packages:
solaris
  developer/gcc-3
    None -> 3.4.3,5.11-0.175.0.0.0.2.537:20111019T093736Z
  developer/gnu-binutils
    None -> 2.19,5.11-0.175.0.0.0.2.537:20111019T095434Z
Services:
  restart_fmri:
    svc:/application/texinfo-update:default
root@solaris:~#

 To list the packages that you have installed:

timj@solaris11vm2:~$ pkg list | head
NAME (PUBLISHER)                                  VERSION                    IFO
SUNWcs                                            0.5.11-0.170               i-r
archiver/gnu-tar                                  1.26-0.175.0.0.0.2.537     i--
audio/audio-utilities                             0.5.11-0.175.0.0.0.2.1     i--
codec/flac                                        1.2.1-0.175.0.0.0.0.0      i--
codec/libtheora                                   1.1.1-0.175.0.0.0.0.0      i--
codec/ogg-vorbis                                  2.30.0-0.175.0.0.0.0.0     i--
codec/speex                                       1.2-0.175.0.0.0.0.0        i--
communication/im/pidgin                           2.10.0-0.175.0.0.0.0.0     i--
compress/bzip2                                    1.0.6-0.175.0.0.0.2.537    i--
timj@solaris11vm2:~$

To list the details of a package:

timj@solaris11vm2:~$ pkg info xcalc
          Name: x11/xcalc
       Summary: xcalc - scientific calculator for X
   Description: xcalc is a scientific calculator desktop accessory that can
                emulate a TI-30 or an HP-10C.
      Category: Applications/Accessories
         State: Installed
     Publisher: solaris
       Version: 1.0.4.1
 Build Release: 5.11
        Branch: 0.175.0.0.0.0.1215
Packaging Date: September 27, 2011 12:58:35 PM
          Size: 104.32 kB
          FMRI: pkg://solaris/x11/xcalc@1.0.4.1,5.11-0.175.0.0.0.0.1215:20110927T125835Z
timj@solaris11vm2:~$

Want to know where a file comes from?:

timj@solaris11vm1:~$ pkg search libresolv.so
INDEX      ACTION VALUE                        PACKAGE
basename   link   lib/sparcv9/libresolv.so     pkg:/system/library@0.5.11-0.175.0.0.0.2.1
basename   link   usr/lib/sparcv9/libresolv.so pkg:/system/library@0.5.11-0.175.0.0.0.2.1
basename   link   lib/libresolv.so             pkg:/system/library@0.5.11-0.175.0.0.0.2.1
basename   link   usr/lib/amd64/libresolv.so   pkg:/system/library@0.5.11-0.175.0.0.0.2.1
basename   link   lib/amd64/libresolv.so       pkg:/system/library@0.5.11-0.175.0.0.0.2.1
basename   link   usr/lib/libresolv.so         pkg:/system/library@0.5.11-0.175.0.0.0.2.1
timj@solaris11vm1:~$

To update all packages:

root@solaris11vm1:~# pkg update
No updates available for this image.
root@solaris11vm1:~# 

What to do when things go wrong:

root@solaris11vm1:~# pkg verify -v   gcc-3
PACKAGE                                                                 STATUS
pkg://solaris/developer/gcc-3                                               OK
root@solaris11vm1:~# mkfile 1024 /usr/sfw/bin/gcc
root@solaris11vm1:~# pkg verify -v   gcc-3
PACKAGE                                                                 STATUS
pkg://solaris/developer/gcc-3                                            ERROR
        file: usr/sfw/bin/i386-pc-solaris2.11-gcc-3.4.3
                Mode: 01600 should be 0555
                Unexpected Exception: Request error: class file/memory mismatch
root@solaris11vm1:~# pkg fix   gcc-3
Verifying: pkg://solaris/developer/gcc-3                        ERROR        
        file: usr/sfw/bin/i386-pc-solaris2.11-gcc-3.4.3
                Mode: 01600 should be 0555
                Unexpected Exception: Request error: class file/memory mismatch
Created ZFS snapshot: 2011-12-07-15:27:53
Repairing: pkg://solaris/developer/gcc-3                   
                                                                             

DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1         1/1      0.1/0.1

PHASE                                        ACTIONS
Update Phase                                     3/3

PHASE                                          ITEMS
Image State Update Phase                         2/2
root@solaris11vm1:~# pkg verify -v   gcc-3
PACKAGE                                                                 STATUS
pkg://solaris/developer/gcc-3                                               OK
root@solaris11vm1:~#

Mail merge in PHP

Just for a laugh

The question was, how do you do a mail merge in PHP. OK, it was a question that no-one was asking, but I never let the deter me before.

Input Data $ cat inputdata Tim,Jinkerson, Ben,De Mora, Jes,Ferrier, tjinkers@TJINKERS-GB ~/merge $ cat template FAO: .f1 .f2 Dear .f1 Now is the winter of our discontent made glorious summer by this sun of York Many thanks Tim tjinkers@TJINKERS-GB ~/merge $

The Program

$ cat mymerge.php $filename = "template"; $mergef = "inputdata"; $fdm = fopen($mergef,"r"); $fnum=1; $mdata = fgets($fdm, 512); while (!feof($fdm)) {   list($f1, $f2, $f3, $f4, $f5) = explode(",", $mdata);   $fd = fopen($filename,"r");   $contents = fread ($fd, filesize($filename));   fclose ($fd);   $ofilename = "file.".$f1.".".$f2.".".$fnum;   $contents=str_replace(".f1", $f1, $contents);   $contents=str_replace(".f2", $f2, $contents);   $contents=str_replace(".f3", $f3, $contents);   $contents=str_replace(".f4", $f4, $contents);   $contents=str_replace(".f5", $f5, $contents);   $fdo = fopen($ofilename,"w");   fwrite ($fdo,$contents);   fclose ($fdo);   $fnum++;   $mdata = fgets($fdm, 512); } fclose ($mergef); ?>

tjinkers@TJINKERS-GB ~/merge
$

For each line of the input data, the program reads the template file as a single string, and then does a replace on the markers for the input date. It writes the results out to a file who's name is based on the input data.

$ php mymerge.php $ ls file.Ben.De Mora.2  file.Jes.Ferrier.3  file.Tim.Jinkerson.1  inputdata  mymerge.php  template tjinkers@TJINKERS-GB ~/merge $ cat file.Tim.Jinkerson.1 FAO: Tim Jinkerson Dear Tim Now is the winter of our discontent made glorious summer by this sun of York Many thanks Tim tjinkers@TJINKERS-GB ~/merge

$

Creating a Solaris 10 Sparse Zone

 First, create a directory for your zone to be built in.

mkdir -p /zones/myzone

 Create a zone config file.

create -b
set zonepath=/zones/myzone
set autoboot=true
add inherit-pkg-dir
set dir=/lib
end
add inherit-pkg-dir
set dir=/platform
end
add inherit-pkg-dir
set dir=/sbin
end
add inherit-pkg-dir
set dir=/usr
end
add net
set address=10.0.2.17/24
set physical=hme0
end

 zonecfg -z myzone -f sparsezone.cfg
 zoneadm -z myzone install

 zoneadm list -v

zlogin -C myzone 
zlogin myzone shutdown -y -i0 -g0

Creating a second database from the linux command line

The objective  here was to make a copy of an existing database and bring it up under a new instance. I could in theory have used a different version of the database but at the time I only had one available to me.

Create a new database:

su - root
mkdir /var/lib/postgresql/testinstance
chown postgres:postgres /var/lib/postgresql/testinstance
su - postgres
initdb -D /var/lib/postgresql/testinstance

Starting the Instance

pg_ctl start -D /var/lib/postgresql/testinstance -l /var/tmp/testinstance.log -o "-p 5434"

Check the status:

pg_ctl status -D /var/lib/postgresql/testinstance

Stopping the Instance

pg_ctl stop -D /var/lib/postgresql/testinstance

Create user on new database:

create role timj login password 'xxxxxx';
create database cddb;
alter role timj superuser;
alter role timj nosuperuser;
grant all on database cddb to timj;

Connecting to the database:

psql -d postgres -h localhost -p 5434 -U postgres

Dump the old database:

/usr/bin/pg_dump -h localhost -p 5432 -U timj -F p -v -f "/tmp/mybackup.pgsql" cddb

Restore to new database:

psql -h localhost -p 5434 -U timj -d cddb < /tmp/mybackup.pgsql

Dumping database in custom mode:

/usr/bin/pg_dump -h localhost -p 5432 -U timj -F c -v -f "/tmp/mybackup.pgsqlc" cddb

Use pg_restore if you have a non-ascii backup:

create role timj login password 'xxxxxx';
create database cddb;
grant all on database cddb to timj;
pg_restore -h localhost -p 5434 -U timj -d cddb /tmp/mybackup.pgsqlc

Point-In-Time Recover in Postgresql

The way you do point in time recovery in Postgresql is to restore a known backup, and then reapply WAL logs until you reach the time you wish to stop at. The database must be in Archive Log Mode (just like Oracle), and you must have all the logs to complete the recovery.

Putting the database in Archive Log Mode

Step 1:
Back up your database
pg_dump -h localhost -p 5432 -U timj -F c -v -f "D:\pgbackup\cddb_21Dec2009_2.backup" cddb

Now let's put the database in archive log mode:

Edit postgresql.conf:

fsync = on
synchronous_commit = on
wal_sync_method = fsync
full_page_writes = on
archive_mode = on
archive_command = 'copy "%p" "D:\\pglogs\\%f"'
archive_timeout = 3600 # switch logs every hour

And remember to bounce the database.

Taking a base backup

I now have to take a base backup of my system.
To do this I logged in as postgres and issued the following:
SELECT pg_start_backup('label2');

I backed the database up using a normal file system backup with the datbase up, and then issued:
SELECT pg_stop_backup();

Next - put in a transaction that we will want recovering:

insert into artist (name) values ('Vicky Beeching');

As this is my first time doing this, I forced a log switch to help with the recovery!

SELECT pg_switch_xlog();

The Disaster!

After my backup, I created a record in the artist table that should be there post recovery. The table has a time trigger on it, so I knew exactky the time I wanted to recover to. I then deleted Nick Drake's entry from the artist table - could there be a bigger mistake than that!

The Recovery

Step 1: Take the database down. This may seem like the obvious first step, but there is a gotchya. The default checkpoint time-out in Postgresql is 5 minutes. This means that if you are relying on recoving your database up to the last good transaction, and if that transaction was less than 5 minutes before hand, it may not have been written to a log file in the pg_xlog directory. Therefore, it may be worth while logging on as an admin user and issuing the command checkpoint; before you shut down the instance.

Step 2: Take a backup!
With the database down, copy off the data directory and any other tablespaces you have in use.

Step 3: Delete your database!
We are going to rebuild the database using the good copy we created in our initial backup, the archived WAL logs and the WAL logs from pg_xlog, so we need to get rid of everything in the existing database.

Step 4: Copy across your backed up database
Copy your good backup into the live data directory.

Step 5: Delete everything from pg_xlog
The files in the restored pg_xlog directory will be from before the backup, and so of no use to you.

Step 6: Copy across the content of your pg_xlog directory
From the backup you took after you discovered you needed to recover, copy the files to your live pg_xlog directory.

Step 7: Create a recovery.conf file
Something like this:

restore_command = 'copy "D:\\pglogs\\%f" "%p"' # Windows
recovery_target_time ('2009-12-22 15:34:56')

The copy command should be in the syntax of whatever OS you are using, very much like the entry in postgresql.conf

Step 8: Restart the database
You might want to restrict who can access while you are checking the recovery.
At the end of the recovery recovery.conf is renamed recovery.done

Step 9: Take a fresh baseline baclup!
It is really good practice to do this before your users start posting transactions, if at all possible.

Solaris 10 CPU Usage

psrinfo shows how many CPUs you have:

bash-3.00# psrinfo 0       on-line   since 08/24/2009 13:40:13 1       on-line   since 08/24/2009 13:40:18 2       on-line   since 08/24/2009 13:40:18 3       on-line   since 08/24/2009 13:40:18 4       on-line   since 08/24/2009 13:40:18 5       on-line   since 08/24/2009 13:40:18 6       on-line   since 08/24/2009 13:40:18 7       on-line   since 08/24/2009 13:40:18 8       on-line   since 08/24/2009 13:40:18

adding the –v option gives you details about those CPUs

bash-3.00# psrinfo -v | head Status of virtual processor 0 as of: 11/19/2009 03:50:06   on-line since 08/24/2009 13:40:13.   The sparcv9 processor operates at 1165 MHz,         and has a sparcv9 floating point processor. Status of virtual processor 1 as of: 11/19/2009 03:50:06   on-line since 08/24/2009 13:40:18.   The sparcv9 processor operates at 1165 MHz,         and has a sparcv9 floating point processor. Status of virtual processor 2 as of: 11/19/2009 03:50:06   on-line since 08/24/2009 13:40:18. bash-3.00#

“prstat -s cpu -n 10” is a bit like top, but it tells you what cpu you are running against

bash-3.00# prstat -s cpu -n 10 PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 3138 900233 895M 588M cpu16 1 0 141:24:49 1.7% mysqld/26 1005 root 19M 18M sleep 59 0 38:34:21 0.0% esd/1 944 daemon 3192K 2208K sleep 29 0 0:00:00 0.0% statd/1 542 root 1768K 256K sleep 29 0 0:00:00 0.0% efdaemon/1 8633 root 1712K 1168K sleep 1 0 0:00:16 0.0% utmpd/1 165 root 7568K 4920K sleep 29 0 0:00:00 0.0% devfsadm/8 940 root 2992K 2136K sleep 29 0 0:00:00 0.0% keyserv/3 147 root 7464K 4472K sleep 29 0 0:00:00 0.0% syseventd/15 9 root 11M 11M sleep 29 0 1:00:07 0.0% svc.configd/12 7 root 32M 31M sleep 29 0 0:04:45 0.0% svc.startd/13 Total: 129 processes, 541 lwps, load averages: 1.12, 1.14, 1.18 bash-3.00#

You can the use the mpstat command to see how each processor is doing:

bash-3.00# mpstat CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 80 0 566 264 53 50 0 2 9 0 203 2 2 0 96 1 1 0 193 15 2 25 0 0 1 0 57 0 0 0 99 2 0 0 48 8 0 15 0 0 0 0 12 0 0 0 100 3 0 0 8 3 0 5 0 0 0 0 4 0 0 0 100 4 15 0 72 8 0 15 0 0 2 0 33 0 0 0 99 5 0 0 4 3 0 4 0 0 0 0 1 0 0 0 100 6 0 0 5 7 1 10 0 0 0 0 0 0 0 0 100 7 0 0 5 6 0 9 0 0 0 0 0 0 0 0 100 8 64 0 369 22 0 47 0 2 7 0 185 3 1 0 96 9 3 0 73 3 0 6 0 0 1 0 22 0 0 0 100

“mpstat –p 10 10”  will refresh the screen every 10 seconds

Perl: Chose your Database Vendor on the command line!

I have created two identical databases. Well, when I say 'identical', one is MySQL and the other is PostgreSQL, but other than that they are pretty similar! They have the same schema and pretty much the same data. It occurred to me that the Perl DBI package hides the database details away from you so successfully that it would be possible to write a program that allowed you to specify the type of database you are accessing on the command line!

Firstly, my new version of the command line routine:

sub dbinfo() {
        use Term::ReadKey;
        my $helpinstructions = "Valid parameters are\n-h {host}\n" .
                               "-d {database}\n" . 
                               "-u {user}\n" .
                               "-p [password]\n" .
                               "-dms [mysql/pgsql]\n";
        my %mydbinfo = (        db => "",
                                host => "",
                                user => "",
                                passwd => "",
                                dms => ""
        );


        die $helpinstructions unless ($#ARGV > 5);

        for (my $i=0; $i <= $#ARGV; $i++) {
                $ARGV[$i]= "-?" if ($ARGV[$i] ne '-h' and
                                    $ARGV[$i] ne '-d' and
                                    $ARGV[$i] ne '-u' and
                                    $ARGV[$i] ne '-dms' and
                                    $ARGV[$i] ne '-p' );
                die $helpinstructions if ( $ARGV[$i] eq "-?" ) ;
                $mydbinfo{db} = $ARGV[++$i] if ( $ARGV[$i] eq "-d" ) ;
                $mydbinfo{host} = $ARGV[++$i] if ( $ARGV[$i] eq "-h" ) ;
                $mydbinfo{user} = $ARGV[++$i] if ( $ARGV[$i] eq "-u" ) ;
                $mydbinfo{dms} = $ARGV[++$i] if ( $ARGV[$i] eq "-dms" ) ;
                if ( $ARGV[$i] eq "-p" ) {
                        if ( $i ne $#ARGV) {
                                $mydbinfo{passwd} = $ARGV[++$i];
                        }
                        else {
                                print "\nPlease Enter passwd: ";
                                ReadMode 'noecho';
                                chomp($mydbinfo{passwd} = ReadLine);
                                ReadMode 'normal';
                                print "\n\n";
                        }
                }
        
        }
        die $helpinstructions unless ( $mydbinfo{dms} eq 'mysql' or
                                       $mydbinfo{dms} eq 'pgsql');
        $mydbinfo{dms} = 'PgPP' if ($mydbinfo{dms} eq 'pgsql');
        return %mydbinfo;
}
return true;

And then the program that calls it:

#!/usr/bin/perl
use strict;
use DBI();

require "D:\\Documents and Settings\\timj\\perl\\choosedbcommanddbsettings.pl" or
die "Can't Open DB Settings File";


my %mydbinfo = dbinfo();
my $dbconnect = "DBI:" . $mydbinfo{dms} . ":database=" . $mydbinfo{db} . ";host=" . $mydbinfo{host} ;

my $dbh1 = DBI->connect($dbconnect,$mydbinfo{user}, $mydbinfo{passwd}, { raiseError => 1, AutoCommit => 0 })
|| die "Database connection not made: $DBI::errstr";
my $sql = "SELECT artist.name, cd.title FROM artist, cdtable AS cd WHERE artist.id = cd.artid";
my $sth = $dbh1->prepare($sql);

$sth->execute();

my( $name, $title );
$sth->bind_columns( \$name, \$title );

print "Name\t\tTitle\n";
while( $sth->fetch() ) {
        print "$name\t$title\n";
}
$dbh1->disconnect() if($dbh1);

and here's the output:

D:\Documents and Settings\timj\perl>choosedbreport.pl -dms mysql -h localhost -u timj -d cddb -p

Please Enter passwd:

Name            Title
Phatfish        Guaranteed
Phatfish        15
Bethany Dillon  Waking Up
Bethany Dillon  Bethany Dillon
Bethany Dillon  Imagination
Nick Drake      Made To Love Magic
Nick Drake      Five Leaves Left
Nick Drake      Bryter Layter

D:\Documents and Settings\timj\perl>choosedbreport.pl -dms pgsql -h localhost -u timj -d cddb -p

Please Enter passwd:

Name            Title
Nick Drake      Made To Love Magic
Nick Drake      Five Leaves Left
Nick Drake      Bryter Layter
Bethany Dillon  Waking Up
Bethany Dillon  Bethany Dillon
Bethany Dillon  Imagination
Phatfish        Guaranteed
Phatfish        15
Portland        These Broken Hands

D:\Documents and Settings\timj\perl>

Maybe not very useful, but it's a good demonstration of the power of DBI();

Perl: DB Connection details on the command line: Part 2

The previous post showed a simple routine for extracting out the database connection variables from the command line. It worked, but it had two flaws:

1. It doesn't validate your parameters
2. It requires you to put your password on the command line, visible to a PS command in Unix

The code below seeks to address both these issues. If you enter a paramter that I don't understand, it kicks you out, and it you end your input with a -p, it accepts the password without echoing it back to the screen, just like you hoped it would!

sub dbinfo() {
 use Term::ReadKey;
 my $helpinstructions = "Valid parameters are\n-h {host}\n" .
                        "-d {database}\n" . 
          "-u {user}\n" .
          "-p [password]\n";
 my %mydbinfo = ( db => "",
    host => "",
    user => "",
    passwd => ""
 );


 die $helpinstructions unless ($#ARGV > 5);

 for (my $i=0; $i <= $#ARGV; $i++) {
  $ARGV[$i]= "-?" if  ($ARGV[$i] ne '-h' and
     $ARGV[$i] ne '-d' and
     $ARGV[$i] ne '-u' and
     $ARGV[$i] ne '-p' );
  die $helpinstructions if ( $ARGV[$i] eq "-?" ) ;
  $mydbinfo{db} = $ARGV[++$i] if ( $ARGV[$i] eq "-d" ) ;
  $mydbinfo{host} = $ARGV[++$i] if ( $ARGV[$i] eq "-h" ) ;
  $mydbinfo{user} = $ARGV[++$i] if ( $ARGV[$i] eq "-u" ) ;
  if ( $ARGV[$i] eq "-p" ) {
   if ( $i ne $#ARGV) {
    $mydbinfo{passwd} = $ARGV[++$i];
   }
   else {
    print "\nPlease Enter passwd: ";
    ReadMode 'noecho';
    chomp($mydbinfo{passwd} = ReadLine);
    ReadMode 'normal';
    print "\n\n";
   }
  }
 
 }
 return %mydbinfo;
}
return true;

And the output:

D:\Documents and Settings\timj\perl>testcommandline.pl -h localhost -d cddb -u timj -p

Please Enter passwd:

Name            Title
Phatfish        Guaranteed
Phatfish        15
Bethany Dillon  Waking Up
Bethany Dillon  Bethany Dillon
Bethany Dillon  Imagination
Nick Drake      Made To Love Magic
Nick Drake      Five Leaves Left
Nick Drake      Bryter Layter

D:\Documents and Settings\timj\perl>

The great thing is that you can directly replace your previous version of the dbsettings routine with this one, and there is no need to alter your existing scripts!