systec.blogsite.org

…and we have done a lot of work. Currently 153 containers are running and we are in the middle of the main upgrade process from Tomcat 5 to Tomcat 6. Also our central login portal is gaining awareness by our users. Last month more then 50k logins and more than 5 million requests

And we are still growing. Beside this we finished the setup for our SMS-gateway. One network based GSM device is now located in germany and in productive usage. For monitoring reason Bernhard and I wrote a small perl daemon based upon inotify (Linux Kernel Notification for Files).

Thats all for the moment…

Mario

Today we started to install Debian 5.03 on our new blades. We are installing via preseed so we have to manage that the bnx2x firmware works. It isn’t included in the netinstaller. Also there is a little issue with the cdrom dection (trough ILO2) but the workaround is just easy.

You could find the documentation about it in the wiki here and here. Here is a short exerpt:

1. cdrom could not be found
2. press no
3. choose “manual select”
4. choose “/dev/cdrom”

To integrate the bnx2x firmware in the netinstaller use the script you could find on the following page:
http://dannf.org/bloggf/tech/add-firmware-to.html
Special thanks to Dann Frazier who wrote this script!

You could find additional inforamtion here:
http://wiki.debian.org/HP/ProLiant

Today we got our new two hardware blade servers for our  OpenVZ farm. Currently we have 106 containers running and calculate that we will close this year with about 120. For your information: we started with about 75 containers this year.

I think we could have a little party at the end of this year beacause a raise by 50 containers in one year isn’t bad.

Oh I forgot to mention, we buyed HP BL460cG6 servers. Some infos about it: 2 QuadCore 2,56 Ghz; 48 GB 1300 Ram, 300GB 10k SAS disk and some aditional parameters like FC adapter…

Mario

so, some weeks went by since my last blog-post.

last week i was in london for sightseeing. great city and nice people there!

work is back and many construction-sites are open. new ssl-certificates, web-application firewall meeting, loadbalancer integration and so on.

also many articles in our wiki are open and have to be finished. i hope that now (autumn & winter) is more time for that!

Ouh, two hard weeks are gone. We had a lot of work and a lot of IT crowd to manage. Different problems and different new systems are up and running. We extend our SMS gateway with a GSM modem positioned in Germany. Now we could send the German Short Messages via a German cell phone. This reduces the cost for our customers.

We also had a night shift this week (till 2am). In this night shift we simulated the catastrophe scenario for our central application portal (loss of the hole datacenter at our site). It was interesting because we were able to come up and running within 2 until 3 hours (from scratch) in Germany.

I think with the analyse of this test we will be able to simplify some components for a shorter recover time. Some topics will be DRBD and glusterfs.

We also implemented (the programmers) a kind of “webservice”. A middle layer between our data and the software from an external software partner. We decided to secure this by using client authentication (RSA PKI certificates) and also ip address restrictions.

Well, I think to weeks to remember because a lot of new and reworked things are now up an running – but a lot of work is still on the stack…….

Mario

…. Uhhh… Yeah…. Chaos is back….

Since Monday I am back at work. Tomorrow we will travel to Vienna to conference about the future of a Terminal Server project we are serving since 1999.

I think we will get back at 8pm (I hope) and then straight into our office because we have renew some SSL certificates and we have to reconfigure a web application. This is the first step in a series till next Thursday when we will have our next night shift. We will then reconfigure some services (like our SMS Gateway) and we will do a worst case scenario recovery test…..

Mario

Currently I am having holidays ,  a week to relax. Now I have time to stay with my family and this is really great! Next week I will return to work and the “normal chaos” will get back my soul , but until then I am not intend to think much about work…. offline………….

Mario

The day before yesterday evening I shortly decided to move the Apache Tomcat loadbalancer into DRBD. Therefore I backed up the configuration file, made a tar-ball (.tar.gz) from the private directory of the container, restore that tar-ball in the DRBD partition and edited the configuration file. After these changes I installed the .deb file I made for starting this “new” type of container. It took exactly seven (7) minutes to move the machine (from stop to start including all other tasks like taring and editing).

What should I say? It works! Tomorrow I have to provide the .deb file with an additional backup script (based on cron) because the one that we provide with our openvz-common package does only cover the normal file structure (/var/lib/vz) and rsync does not sync over mount borders (and I mount the DRBD device under /var/lib/vz/drbd0).

Special thanks to Werner Fischer who leaves a comment (here)! We become acquainted with him at the LinuxTAG 09 in Berlin and have a lot of fun.

Mario

First – all systems are up and running! It is always exciting to make changes in productive environments because testing in a test environment is a bit “sterile” or “artificial”.

The resize operations on VMWare were straight forward as always but it took some time to backup the virtual harddrives before.

The really interesting thing was to shrink the ext3 partition on one of our OpenVZ servers, lvreduce the LVM logical volume, grow the ext3 to the maximum of the resized LVM logical volume and to implement DRBD for productive usage between to hot servers. Result: perfect! I decided to push up the syncer rate of DRBD (default 240K/sec.) to 20M/sec. The initial primary/secondary sync took about 30 minutes (20 G device).

Today the status is that the DRBD devices are in sync and the nighly reboot of the servers had no impact. I think I will make a little .deb package today for my management scripts of the OpenVZ containers (stratup). After that time is coming to copy our loadbalancer into the DRBD…..

Mario

Tomorrow evening I will insert a night shift because I have to change configurations of some VMWare servers (extend hard disk space) and do some stuff on two OpenVZ hardware nodes. On these nodes I will prepare the LVM configuration to suit our needs for DRBD. This means that I have to shrink a LVM volume (backup before!) to gain space for an additional logical volume (for DRBD).

I think i have to spend the whole work time tomorrow to prepare the environment so I could start immediately after returning to work place (8pm). Oh, I could not make these changes while working hours (we have to provide the services online).

Mario