When building systems in VirtualBox that have more than a few servers, or for special requirements like modelling server clusters, it is useful to have a separate virtual machine to provide the internet services one would typically find in a network. The most common services required are DNS, some kind of internet gateway or proxy server, an NTP server, and a HTTP server.
In this document I go through building a utility server to be used in a number of larger infrastructure models. Although generally I prefer to build my own installations, in this case I will use a pre-built appliance, just to provide an example of how quick and easy it can be.
Tuesday, 8 March 2016
Openfiler is dead. Long live openmediavault.
Just a quick update.
My efforts to update Openfiler to run on recent *EL versions of Linux died fairly quickly. The more I delved into it the more it became like trying to untie spaghetti with a spoon. However, in the meantime, I discovered openmediavault, which does the job admirably and has the benefit of being actively developed.
My efforts to update Openfiler to run on recent *EL versions of Linux died fairly quickly. The more I delved into it the more it became like trying to untie spaghetti with a spoon. However, in the meantime, I discovered openmediavault, which does the job admirably and has the benefit of being actively developed.
Wednesday, 13 August 2014
Openfiler progress - pump up the volumes
I have been making some progress with the update to Openfiler. So far I have the basic authentication happening, but haven't really touched anything to do with LDAP or AD/Samba integration. System date/time needed a bit of work to overcome the fact that 'timeconfig' cli no longer exists. Networking didn't need much and seems to be reasonably solid. Similarly with the rest of the basic system administration.
Most of my time has been taken up with getting the volume management working. First thing was to get it to even see the disks. 'list_disks.pl' relies on 'parted print-fdisk' to get a list of the disks and their partitions, but the 'print-fdisk' option is another Openfiler-ism. It was a simple case to replace this with a call to 'fdisk -l', which gives the same information.
By far the most effort has gone into getting the RAID configuration working. Once again, Openfiler utilised a patched version of a standard utility - in this case mdadm - to provide output in an Openfiler-specific format. Since I am trying to avoid this, I had to do some digging into where the information is used within Openfiler, which included digging into the original source of mdadm to understand exactly what standard mdadm outputs vs what the Openfiler patch does, and then working out how to do the same without a patch. I started down the same path as I had previously with 'authconfig', but now realise that was a bit of a hack. In the end I have put the code where it should be, in 'md.inc', which I find a far more satisfying solution. In the process, my understanding of PHP has grown measurably. This has given me the impetus to go back and do the same for 'authconfig' but I will probably leave that to a later time.
Having gotten RAID and LVM workable, at least for simple 'volumes with file-systems', I now need to tackle iSCSI. The original Openfiler uses the iSCSI Enterprise Target implementation, with it's command set scattered throughout the product. The standard iSCSI that comes with CentOS 6 is the Linux SCSI Target Framework with a completely different command set and configuration (which has already been superceded in the standard kernel by the LIO framework). SCST is also another viable contender with a lot of support. I could take the easy way and simply build and install IET, but that wont necessarily be a long term option (and, once again, means it will be one more thing 'out-of-band'). So it looks like I will be doing a lot of 'find-and-replace' but at the same time, I will think about re-architecting to maybe make iSCSI a 'plugin' where I will come up with a generic API, and the implementation will be done in include files that are chosen at install time.
After that, I think the next major task is sorting out services, particularly managing both 'init.d' based and xinetd-based services.
Most of my time has been taken up with getting the volume management working. First thing was to get it to even see the disks. 'list_disks.pl' relies on 'parted print-fdisk' to get a list of the disks and their partitions, but the 'print-fdisk' option is another Openfiler-ism. It was a simple case to replace this with a call to 'fdisk -l', which gives the same information.
By far the most effort has gone into getting the RAID configuration working. Once again, Openfiler utilised a patched version of a standard utility - in this case mdadm - to provide output in an Openfiler-specific format. Since I am trying to avoid this, I had to do some digging into where the information is used within Openfiler, which included digging into the original source of mdadm to understand exactly what standard mdadm outputs vs what the Openfiler patch does, and then working out how to do the same without a patch. I started down the same path as I had previously with 'authconfig', but now realise that was a bit of a hack. In the end I have put the code where it should be, in 'md.inc', which I find a far more satisfying solution. In the process, my understanding of PHP has grown measurably. This has given me the impetus to go back and do the same for 'authconfig' but I will probably leave that to a later time.
Having gotten RAID and LVM workable, at least for simple 'volumes with file-systems', I now need to tackle iSCSI. The original Openfiler uses the iSCSI Enterprise Target implementation, with it's command set scattered throughout the product. The standard iSCSI that comes with CentOS 6 is the Linux SCSI Target Framework with a completely different command set and configuration (which has already been superceded in the standard kernel by the LIO framework). SCST is also another viable contender with a lot of support. I could take the easy way and simply build and install IET, but that wont necessarily be a long term option (and, once again, means it will be one more thing 'out-of-band'). So it looks like I will be doing a lot of 'find-and-replace' but at the same time, I will think about re-architecting to maybe make iSCSI a 'plugin' where I will come up with a generic API, and the implementation will be done in include files that are chosen at install time.
After that, I think the next major task is sorting out services, particularly managing both 'init.d' based and xinetd-based services.
Wednesday, 6 August 2014
Printing authconfig output as XML
Just a quick followup to my post about bringing Openfiler up-to-date. Here is a script which takes the output of 'authconfig --test' and formats it as XML.
#!/bin/bash
authconfig $* --test | sed 's/ is \| are /=/
s/ = /=/
s/ by default\|always \| (.*)//
y/ +/__/
s/^_/ /
s/_or_.*=/=/
/="/! s/=/="/
/"$/! s/$/"/
/.=/! d
s/=/\#/' | awk -- '
BEGIN {
FS="#";
printf("<?xml version=\"1.0\"?>\n");
printf("<authconfig>\n");
printf(" <globals>\n");
SETTING="false";
}
!/^ / {
if (NR>1) {
if (SETTING=="true")
{ printf("/>\n");
printf(" </key>\n") }
else
{ printf("/>\n") }
}
SETTING="false";
printf(" <key name=\"%s\" value=%s",$1,$2)
}
/^ / {
if (SETTING=="false")
{ printf(">\n");
printf(" <settings")
}
printf("%s=%s",$1,$2);
SETTING="true";
}
END {
if (SETTING=="true")
{ printf("/>\n");
printf(" </key>\n") }
else
{ printf("/>\n") }
printf(" </globals>\n");
printf("</authconfig>\n");
}'
Tuesday, 5 August 2014
Now for my next trick....
In looking for my next challenge, I have decided to step away from the Oracle stuff for now and am going to try my hand at something down the stack.
I have used Openfiler for a while whenever I need a 'quick-and-dirty' SAN for any of my projects. It does the job I need it to do, but is getting a bit long in the tooth - the last official release was over 2 1/2 years ago. A significant problem is that it is based on the now-defunct rPath Linux distribution. About this time last year, Openfiler announced they would be moving back to a CentOS based distribution, releasing a preliminary download, but nothing has been heard since.
So I have decided to take up the baton.
Based on what I have seen so far, I am going to have to learn a lot more about the ins-and-outs of Linux storage, authentication, kernel building, rpm packaging and PHP than I ever thought I would, but maybe that is a positive thing.
My initial goal is to get the current Openfiler functionality on a vanilla CentOS 6 install, using only currently available repositories (i.e. CentOS, EPEL, ATrpms et al). I want to avoid having to maintain my own versions of code that already exists. As an example I have found so far, Openfiler relies on getting an XML representation of the output of 'authconfig'. The current version of 'authconfig' does not produce this output. Given 'authconfig' is a Python script, it is reasonably easy to put the functionality back in, but then I would be left having to keep track of developments in 'authconfig' and maintaining my own branch. In the tradition of UNIX, I would rather develop something that can take input from the standard 'authconfig' and output what Openfiler needs.
Once I get the basics of presenting the core storage services going well on stock CentOS code, then I might start looking into things like SCST and DBRD/clustering.
I think my biggest problem is going to be stopping from jumping all over the place and just focusing on one piece at a time :-)
Anyway, expect semi-regular updates on my progress. And help or advice is always welcome.
I have used Openfiler for a while whenever I need a 'quick-and-dirty' SAN for any of my projects. It does the job I need it to do, but is getting a bit long in the tooth - the last official release was over 2 1/2 years ago. A significant problem is that it is based on the now-defunct rPath Linux distribution. About this time last year, Openfiler announced they would be moving back to a CentOS based distribution, releasing a preliminary download, but nothing has been heard since.
So I have decided to take up the baton.
Based on what I have seen so far, I am going to have to learn a lot more about the ins-and-outs of Linux storage, authentication, kernel building, rpm packaging and PHP than I ever thought I would, but maybe that is a positive thing.
My initial goal is to get the current Openfiler functionality on a vanilla CentOS 6 install, using only currently available repositories (i.e. CentOS, EPEL, ATrpms et al). I want to avoid having to maintain my own versions of code that already exists. As an example I have found so far, Openfiler relies on getting an XML representation of the output of 'authconfig'. The current version of 'authconfig' does not produce this output. Given 'authconfig' is a Python script, it is reasonably easy to put the functionality back in, but then I would be left having to keep track of developments in 'authconfig' and maintaining my own branch. In the tradition of UNIX, I would rather develop something that can take input from the standard 'authconfig' and output what Openfiler needs.
Once I get the basics of presenting the core storage services going well on stock CentOS code, then I might start looking into things like SCST and DBRD/clustering.
I think my biggest problem is going to be stopping from jumping all over the place and just focusing on one piece at a time :-)
Anyway, expect semi-regular updates on my progress. And help or advice is always welcome.
Monday, 28 July 2014
What's in the toolbox....
Over the years I have picked up a few bits of software that form part of my personal SOE. The list below is some of what is currently included.
My preference for a Linux desktop is also driven by the fact that nearly all of my work is done on Linux or UNIX systems, and I don't have to do a 'context switch' between working on the customer systems and my desktop.
Using Thunderbird for email allows me to have all my email accounts in the one client, with the exception of my work Microsoft Exchange email account. For whatever reason, they don't have the Exchange IMAP service enabled. However, to get around that I use davmail gateway. This gives me access to everything I need from Exchange, but gives Thunderbird local standards-based services to talk to..
For diagramming, I normally use Libre Office Draw, or Dia. I have never used Visio, and to be honest have never run into anything that I couldn't do with Draw or Dia and some free clipart.
However, if I must use Windows, my preferred utility is MobaXterm. Where previously you might have needed PuTTY + XMing +Winscp, MobaXterm has all of those plus more, all in the one utility. You can get it as a standalone executable or as an installer. In addition, it gives me a 'bash' shell on Windows, so once again that is one less thing I have to adjust to.
sqluldr2 is a utility to quickly extract large amounts of data from Oracle tables to character separated text files. It is a lot quicker and easier to run than the equivalent Oracle utilities. Sadly, it seems to be unmaintained, but should still work with 10.2.0.4 client libraries.
SQLDeveloper is not bad. Pretty easy to install and run, does pretty much everything I need for poking around in a database, and has some additional functionality if you connect 'as sysdba'.
CamStudio for doing screen recordings, if you need to do that sort of thing. For example, if you need to document an installation, just start CamStudio, do the installation, then later playback the recording through VLC, and take use VLC to save screenshots at the appropriate times
PDFtk is a toolkit for working with pdf files - merging and splitting.
And finally PokerTH, which is 'solitaire' for grown-ups.
Desktop
My preferred desktop is openSUSE (currently version 13.1). I have been using openSUSE as my primary desktop O/S for almost as long as I have been using Linux - nearly 20 years now. It is not just the familiarity I have with it, although that certainly helps. Things like YaST, the openSUSE Build Service and repositories mean pretty much anything I need to do I can do with a mimimum of fuss.My preference for a Linux desktop is also driven by the fact that nearly all of my work is done on Linux or UNIX systems, and I don't have to do a 'context switch' between working on the customer systems and my desktop.
Productivity
As you would expect, along with a Linux desktop, I prefer to use Firefox for browsing, Libre Office for documents/spreadsheet/presentations, and Thunderbird for email. These have the added benefit of being cross-platform, so when I am forced to use a Windows desktop, I can use the same applications there.Using Thunderbird for email allows me to have all my email accounts in the one client, with the exception of my work Microsoft Exchange email account. For whatever reason, they don't have the Exchange IMAP service enabled. However, to get around that I use davmail gateway. This gives me access to everything I need from Exchange, but gives Thunderbird local standards-based services to talk to..
For diagramming, I normally use Libre Office Draw, or Dia. I have never used Visio, and to be honest have never run into anything that I couldn't do with Draw or Dia and some free clipart.
System Utilities
Once again, the standard Linux desktop give me everything I need by way of utilities to access customer systems. Most remote access scenarios have a Linux equivalent, and ssh with X-Forwarding and/or tunneling gives me everything else.However, if I must use Windows, my preferred utility is MobaXterm. Where previously you might have needed PuTTY + XMing +Winscp, MobaXterm has all of those plus more, all in the one utility. You can get it as a standalone executable or as an installer. In addition, it gives me a 'bash' shell on Windows, so once again that is one less thing I have to adjust to.
sqluldr2 is a utility to quickly extract large amounts of data from Oracle tables to character separated text files. It is a lot quicker and easier to run than the equivalent Oracle utilities. Sadly, it seems to be unmaintained, but should still work with 10.2.0.4 client libraries.
SQLDeveloper is not bad. Pretty easy to install and run, does pretty much everything I need for poking around in a database, and has some additional functionality if you connect 'as sysdba'.
CamStudio for doing screen recordings, if you need to do that sort of thing. For example, if you need to document an installation, just start CamStudio, do the installation, then later playback the recording through VLC, and take use VLC to save screenshots at the appropriate times
PDFtk is a toolkit for working with pdf files - merging and splitting.
Other stuff
For Windows, PortableAPPS is invaluable. Over 300 FOSS applications that can be downloaded and run from a USB, or just downloaded and installed as a 'local user' i.e. without requiring administrative permissions on the desktop. So if you want Firefox but don't have permission to install software, you can install the portable version in your user directory, and just delete it when you are finished.And finally PokerTH, which is 'solitaire' for grown-ups.
Thursday, 24 July 2014
Build an Oracle VM 3 demo environment in VirtualBox - from scratch.
Hypervisor
"A hypervisor or virtual machine monitor (VMM) is a piece of computer software, firmware or hardware that creates and runs virtual machines. " - WikipediaIn the world of server virtualisation, there are two kinds of hypervisor. 'Type 2' hypervisors are also known as 'hosted' because they rely on an underlying host operating system (i.e Linux, Windows et al) on which to run. VirtualBox is an example of a Type 2 hypervisor. 'Type 1' hypervisors on the other hand are 'bare metal' hypervisors. They are in effect specialised operating systems in their own right, running directly on the host hardware and dedicated to managing guest virtual machines. VMWare ESX/ESXi, Microsoft Hyper-V and Oracle VM Server are examples of this kind of hypervisor.
If you want to set up a demo environment of Oracle VM Server, because it takes over the whole machine, you have a couple of choices. If you have enough spare hardware lying around, you can set up an environment on physical infrastructure. Not everyone has that much spare hardware lying around. The alternative is to do it on virtual hardware.
Inception
With VirtualBox, running on a decent host, you can create virtual versions of all the infrastructure you need. As far as the guest systems are concerned, they are running on real hardware. It is just in this case, the guest system will be Oracle VM Server. And, just like in the movie 'Inception', you can go multiple layers deep in virtualisation.There are pre-built 'appliances' of Oracle VM Manager and Oracle VM Server for VirtualBox. If you just want to get it up and running that way, you can find the instructions for doing so here. Personally, I prefer to 'learn by doing'. In the real world, you would have to build this infrastructure in 'bare metal' so I wanted to replicate that.
The documents below describe in detail the process for building a complete demo of Oracle VM 3 in VirtualBox. It is based on Oracle VM 3.2. It should still work for version 3.3 which was released earlier this month, with the exception that you can no longer use an external database for the Oracle VM Manager repository - you must use the MySQL database that comes with the distribution.
Subscribe to:
Posts (Atom)