Archive for the 'Linux' Category

Last night I decided to finally try and understand why some of the emails I was sending was getting caught in spam filters, particularly Yahoo and Hotmail. So it began.

#0 Read this. Get the basics.

#1 To test everything out, send yourself an email to a GMail account. When viewing the message, click the drop down and click “View Original”. From there you will be able to see the return-path and comments on the SPF.

#2

Make sure the return-path is set correctly in your code. This return-path will be used to do the reverse ptr lookup. In PHP Pear Mail that meant:

$mail =& Mail::factory('mail', array('Return-Path' => sprintf("-f %s", '[email protected]')));

#2

Read this. You need to request your hosting provider, in my case ServerBeach, to update your IPs SPF record.

Then the txt spf record needs to be updated to allow permit the server IP:

v=spf1 ip4:11.11.111.11 include:_spf.google.com ~all

#3

Setting up DKIM is hard. Read this. I’m using Exim4 and by default DKIM support is built in. To do it read this. But you’ll quickly realize that won’t work on etch! So you need to upgrade to Lenny and then make sure the lenny backports are on, because you need to go to exim 4.7+

# Main
deb http://http.us.debian.org/debian/ stable main non-free contrib
# Source
deb-src http://http.us.debian.org/debian/ stable main non-free contrib
# Security
deb http://security.debian.org/ stable/updates main contrib non-free
# backports for lenny
deb http://backports.debian.org/debian-backports lenny-backports main

So now install the backports version of exim after you’ve upgraded Debian to Lenny:

apt-get -t lenny-backports install exim4

Basically create the keys in the exim config directories /etc/exim4/:

openssl genrsa -out dkim.private.key 768
openssl rsa -in dkim.private.key -out dkim.public.key -pubout -outform PEM

Then update the file exim4.conf.template:

Make sure the dkim_private_key is to a fully qualified path

driver = smtp
dkim_domain = example.com
dkim_selector = x
dkim_private_key = /etc/exim4/dkim.private.key
dkim_canon = relaxed

Then update DNS with the txt entry and it should work. Just remember George you can’t do this in etch, you need to be in lenny and using exim > 4.70.

Setup a new server on SB etchy, true to form, the pain of setting up the server ensues..

For some reason /etc/apt/sources.list is off, by default had:

# Main
deb http://http.us.debian.org/debian/ etch main non-free contrib
# Source
deb-src http://http.us.debian.org/debian/ etch main non-free contrib
# Security
deb http://security.debian.org/ etch/updates main contrib non-free

Changed it to:

# Main
deb http://http.us.debian.org/debian/ stable main non-free contrib
# Source
deb-src http://http.us.debian.org/debian/ stable main non-free contrib
# Security
deb http://security.debian.org/ stable/updates main contrib non-free

Installed apt-get install apache2, but didn’t know how to check that worker was installed by default:

apache2 -l
Compiled in modules:
  core.c
  mod_log_config.c
  mod_logio.c
  worker.c
  http_core.c
  mod_so.c

apt-get install php5 libapache2-mod-php5

And then I realized PHP should not run with worker in a production env:

62.1. Why shouldn’t I use Apache2 with a threaded MPM in a production environment?

PHP is glue. It is the glue used to build cool web applications by sticking dozens of 3rd-party libraries together and making it all appear as one coherent entity through an intuitive and easy to learn language interface. The flexibility and power of PHP relies on the stability and robustness of the underlying platform. It needs a working OS, a working web server and working 3rd-party libraries to glue together. When any of these stop working PHP needs ways to identify the problems and fix them quickly. When you make the underlying framework more complex by not having completely separate execution threads, completely separate memory segments and a strong sandbox for each request to play in, feet of clay are introduced into PHP’s system.

If you feel you have to use a threaded MPM, look at a FastCGI configuration where PHP is running in its own memory space.

And finally, this warning against using a threaded MPM is not as strong for Windows systems because most libraries on that platform tend to be threadsafe.

Setup Memcaches: http://www.sematopia.com/2009/03/setting-up-memcached-on-ubuntudebian-and-windows-xampp/

Disable Etags:

sudo a2enmod headers
sudo /etc/init.d/apache2 restart

Header unset ETag
FileETag None

Install Subversion:

apt-get install subversion
apt-get install php-pear

Intall APC:

apt-get install apache2-dev php5-dev php-pear make
ln -s /usr/bin/apxs2 /usr/bin/apxs
pecl install apc

Install Mysql:

apt-get install mysql-client mysql-common mysql-server php5-mysql

Add to apache2.conf:

AliasMatch \.svn /non-existant-page
Options -Indexes

Header unset ETag
FileETag None

<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
Header set Expires "Thu, 15 Apr 2015 20:00:00 GMT"
</FilesMatch>

Enable mod-rewrite

a2enmod rewrite

Install SSL:

apt-get install openssl ssl-cert

Configure Exim4 (cause by default does not support external mail sending)

dpkg-reconfigure exim4-config

Change the time zone to est:

dpkg-reconfigure tzdata

Add spelling:

apt-get install php5-pspell
apt-get install aspell-en

I just finished installing the PECL filter extension on my Debian install (AMD64), and by God, it was painful. This process was so brutal with so many ways to screw up. Here are the steps:

Install php5-dev package. During the process of not figuring out what was going on, I also installed the php4-dev package. I’m pretty sure I only needed php5-dev.

Download PCRE – Perl Compatible Regular Expressions from here. Be sure you have g++ installed (not just gcc as I was getting a strange error). Extract, configure, make and make install. If your running AMD64, be sure to compile using the -fPIC flag. See here for information why. Filter will look for the library in /usr/local/, so you’ll have to setup a link or copy it from /usr/local/lib/ so the PECL install can find it.

The Filter package also looks for the header files from the php_pcre extension. The only way I found to get at these header files was to download the PHP source. So depending on what version of PHP you have, go here and download the zip. Copy the entire pcre folder from within the extension (ext) folder into your /usr/include/php5/ext/ folder.

Now it’s time to attempt the actual PECL Filter install. If your doing this from the tarball and your running on an AMD64, as above, be sure to add the -fPIC flag. I used the PECL installer, so I ran: pecl install filter. Since the package is not “stable” yet, you’ll need to point right to the channel, just read the message, in my case it was: pecl install channel://pecl.php.net/filter-0.11.0. After the compile/install is done, add extension=filter.so to your php.ini file. If you can’t find it, type find / -name php.ini from your command line.

That’s it. I may have forgotten a step, if so add a comments please.

This was surprisingly a lot harder to figure out than I initially expected. I was doing a lot of searching and kept finding either solutions for windows or packages that cost money. To add to the problem, I needed a solution that ran in a command line only version of Linux (Ubuntu Server). I guess I was naive to think the PHP gd library would support this. After a lot of digging last night, I found webthumb. Getting it work was surprisingly easy using the instructions provided, just replace mozilla for firefox and add the xwd package:

apt-get install xvfb
apt-get install firefox
apt-get install netpbm
apt-get install xwd

wget http://www.boutell.com/webthumb/webthumb.tar.gz
tar xvzf webthumb.tar.gz
cd webth*
./webthumb http://www.sematopia.com/ | pnmscale -xysize 100 100 | pnmtojpeg > thumb.jpg

Enjoy

It seems these days I spend more time figuring out annoying problems rather than doing any real work. The following gave me another big headache: Getting SVN to auto-update properly via it’s hooks. SVN provides you the ability to specify hooks after certain events occur with the repository in question. For example, say you have a live application under version control and you want SVN to auto-update the live copy (on the server) after every commit. I guess that makes sense, especially if you want to develop and test in your production environment, on the fly.

In the directory were your repository exists there should be a hooks directory. This directory includes a bunch of tmpl files that get called depending on the event that just occurred. The file that I cared about was the post-commit.tmpl file.

Assuming your running SVN using Apache, then most likely when your post-commit script gets called it will be executed using the user www-data. This is important because it means your repository needs to be checked out via the same user to ensure no permission problems occur. Assuming that OK, then add the following line to your post-commit.tmpl file:

/usr/bin/svn update <path to repo> --username xxxxxx --password xxxxxx

Hardcoding your SVN password in the file is probably not the best idea, but if your the only one who would have access to the file, then who cares. Notice how I put a full path to SVN, that’s because when the hook scripts get called all environment variables, etc. get removed. The next step, and probably most important, rename the file removing the .tmpl extension and change the ownership so the file becomes executable.

cp post-commit.tmpl post-commit
chmod +x post-commit

That should do it, so when you update your repository, SVN should auto-update your live working copy. Most likely you’ll run into problems, if so check the links below for more info.

Helpful Link #1
Helpful Link #2
Helpful Link #3

This caused me a headache tonight. I’ll explain below how to forward requests from one Apache HTTP server to another on your network. Here are the main characters:

Server0: Your primary HTTP webserver that all external requests come to initially.
Server1: A secondary server you have setup (with Apache) to handle requests for xyz.com. The (internal) server IP is 192.168.2.10.

At the heart of making this possible is the mod_rewrite module available with Apache. This module seems very powerful and I only understand a small portion of it. First off you need to enable mod_rewrite and mod_proxy on Server0:

a2enmod rewrite
a2enmod proxy
/etc/init.d/apache2 force-reload

mod_proxy needs to be enabled and configured (proxy.conf), otherwise Apache won’t allow mod_rewrite to forward the request. Here is the relevant portion of proxy.conf (found in /etc/apache2/mods-enabled):

<Proxy *>
   Order deny,allow
   Deny from all
</Proxy>

<Proxy http://xyz.com>
   Order deny,allow
   Deny from all
   Allow from all
</Proxy>

Now that you have the prerequisites ready, you can create the site definition on Server0 (/etc/apache2/sites-available/xyz.com):

<VirtualHost *>
   ServerName www.xyz.com
   # RewriteLog "/var/log/apache2/rewrite.log"
   # RewriteLogLevel 9
   RewriteEngine     On
   RewriteRule       ^(.*)$        http://xyz.com$1  [P]
   ServerAlias xyz.com
</VirtualHost>

Notice the rewrite rule, which will forward the request. Since xyz.com (on server0) will now point to server1, you need to update your /etc/hosts file accordingly:

192.168.2.10    xyz.com
192.168.2.10    www.xyz.com

Now enable the the new site and restart Apache:

a2ensite xyz.com
/etc/init.d/apache2 restart

Now your done with Server0. You’ll be glad to know there is nothing out of the ordinary for you to do on Server1. All you have to do is create the site definition and enable (like you normally would). So in /etc/apache2/sites-available create a site definition called xyz.com:

<VirtualHost *>
   ServerName www.xyz.com
   DocumentRoot /var/www/xyz
   ServerAlias xyz.com
</VirtualHost>

Then enable the site, restart Apache and everything should work. Odds have it something will go wrong and if it does, make sure to look in the Apache logs (/var/log/apache2/). Also, enable the rewrite log (see above). The logs will most likely always tell you exactly what went wrong.

This was a busy week, a lot of catching up from all the time spent at CASCON last week. Continuing on from last week, I’ve gotten more intimate (?) with Dojo and its inner workings. I wrote about Dojo a couple months ago, and I’m still really impressed with everything going on there. A lot of big name sites are built off this toolkit, including Meebo.

IBM results came out last week, the System i had a rough quarter, but IBM did well overall. Last year at this time (3Q05) the System i had a break out quarter due to an upgrade cycle. This year (3Q06) there were no new upgrades and results were the same as 3Q04. This article sums things up:

IBM revenue for the System i fell 22 percent in the third quarter of 2006, or 23 percent at constant currency when compared to the same period of 2005. At the same time, nearly every other major IBM business unit saw revenue increase. Overall, earnings rose 47 percent compared to the previous year, up to $2.22 billion on revenues of $22.6 billion, which were up 5 percent.

The supply chain issues IBM faced in the second quarter of this year for System i sales, which were connected to a 7 percent drop in revenue for System i over the second quarter of 2005, have cleared up. IBM CFO Mark Loughridge noted that IBM’s supply chain did a superb job delivering System p, i, z, and storage but faltered over System x deliveries, contributing to the low rise of 4 percent for the System x. System p revenues increased 10 percent, and System z was a breakout star with a 25 percent increase in revenues while IBM delivered a 16 percent increase in System z MIPS (millions of instructions per second).

For the System i, IBM says 3Q06 compares to a particularly strong 3Q05 quarter, which was driven by upgrade activity from a fully refreshed roadmap, which in turn caused slowing revenue as customers leveraged those previous upgrades. Overall, IBM notes that System i revenue performance remains dependent upon cyclical upgrades.

I was at a Halloween fund raiser for Diabetes last night, it was a good time for a good cause. I’m amazed that no matter what type of social event I may be at, there’s always someone thats directly involved with the System i in some way.

In other news, RedHat got murdered this week in the markets. Cisco announced they would be creating a fork the RedHat’s version of Linux and providing half price support until the new year. If any good can be seen out of this, it just re-affirms RedHat’s dominance in the Enterprise Linux market.

FireFox 2 came out this week, it generally seems more responsive, but definitely not as polished as it should be. The memory leak issue doesn’t seem to have been fixed, since as I type FireFox is using upwards of 80mb memory and climbing. The new spell checking feature is great though, any work misspelled get underlined in red, right click the and suggestions appear.

Initially I didn’t plan on writing anything up about this, since its pretty basic to setup. But I ran into a slight problem, which I imagine others might run into also. I guess most would have Apache running already, so all that is needed is SVN:

> apt-get install subversion libapache2-svn

Say you have a domain already set up, call it abc.com. You want to be able to checkout a repository from abc.com/svn.

Open the following file (/etc/apache2/mods-available/dav_svn.conf) in an editor and follow the instructions. In the most basic form (with no authentication) you’ll have something like this when your done:

< Location /abc.com/svn >
DAV svn
SVNPath /svn
< /Location>

Inside the < Location /abc.com/svn > that path must point to a directory in your web server which does NOT exist. abc.com will exist, but make sure svn doesn’t. The SVNPath will point to the actual repository which MUST be outside your web server space.

This way, if you go to a browser and type: http://www.abc.com/svn nothing will show up, but when you checkout your repository, you will be re-directed to the actual svn directory (in this case /svn).

Now create the SVN repository, in directory /svn

> svnadmin create /svn

Be sure to change the permissions on this directory, so the web server can write to the folder:

> chown -R www-data /svn

Restart Apache2:

> /etc/init.d/apache2 -k restart

Now from a remote computer, you should be able to type:

svn co http://xxx.xxx.xxx.xx/abc.com/svn < name >

(were xxx.xxx.xxx.xxx is your server ip and name is what you want to call the local folder)

This has been a rough week, 4 assignments, an MIS, etc. I made that phpPostFix script, its very simple, but I won’t be able to use it. My main server with GoDaddy is so terrible. Its running an old version of PHP (4.2). This version did not have file_get_contents as a function, so to make up for it, a while ago, I set up the PEAR Compat replacements. To make a long story short, there are limitations in PHP 4.2 that for some reason, file_get_contents (the PEAR version used in PHP 4.2) can not accept URLs with more than 1 parameter.

I know what your thinking: “Just upgrade your version of PHP”. You obviously have never used a GoDaddy product. This is not an easy task, especially since their servers are married to Plesk and all the crap that come with it. Above all that the main server is fully deployed, I can’t risk upgrading a shady setup. Anyways, here is the script that I wrote. It does the job as explained in the previous post.

Using the script is as easy as the following:

$to = "[email protected]";
$subject = "Test email";
$query_string = 'to='.urlencode($to).'&subject='.urlencode($subject);
file_get_contents("http://www.someserver.com/phpPostFix.php?".htmlentities($query_string));

I still need to be able to send email from development server. So I spent some time thinking about it earlier this week. I’m going to write the emails which need to be sent to an XML file (send.xml). The schema will be as follows:

<email>
<to></to>
<cc></cc>
<bcc></bcc>
<subject></subject>
<body></body>
</email>

Every 5 minutes the main server will read the XML file, parse it, and send the emails based on the results. Once the emails are read, it will clear the original XML file (or append it to another backup.xml file). At this point any new emails will be written to a fresh send.xml. This method makes intuitive sense (sending in batch), especially since the development server could be sending hundreds of emails a day. Lets hope this goes without a problem.

So I’ve spent the last 2 days straight trying to get a full scale mail server running on my development Ubuntu server. I had heard it wasn’t going to be easy, but it needed to happen. The plan was to go all out (Postfix + Courier IMAP + Amavisd-new + SpamAssassin + ClamAV + SASL + TLS + SquirrelMail + Postgrey). At around 7:00pm today, when I was completely defeated as to why it wasn’t working it hit me: I bet my ISP has port 25 blocked. A quick search on Google proved me right.

Depressed and annoyed I turned off my laptop, went to the library and then to the gym. While reading Data Mining slides it hit me, I have my main dedicated server at GoDaddy (not to say they haven’t done me wrong in the past). I was considering using some relay host, but I thought of something better: I’ll write a PHP script, which will reside on the main server, and will send mail the standard way (sendmail). The PHP script (lets call it phpPostFix) will be accessed as a URL, and its parameters will be simply:
- To:
- From:
- Subject:
- Body:

On the development server any time I have:

<?php

file_get_contents("http://www.xxxxxx.com/phpPostFix.php?to=[email protected]&from=.....");

?>

Then the email will be sent. Simple as that. I’ll post the script when its done.

In Ubuntu-Server everything is from the command line, and most likely your going to need to set a static ip to your server. All network interface settings are in the /etc/network/interfaces file. Here is what to do:

1. Backup /etc/network/interfaces
2. edit the file with vi. Here you can set the static setting as below. In my case I set the static router ip to be 192.168.1.100

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# This is a list of hotpluggable network interfaces.
# They will be activated automatically by the hotplug subsystem.
mapping hotplug
        script grep
        map eth0

# The primary network interface
iface eth0 inet static
address 192.168.1.100
netmask 255.255.255.0
gateway 192.168.1.1

If your going to be administering an Ubuntu server remotely, be sure to leave the install cd in the cdrom drive. While I was installing SSH, for some reason the Ubuntu-Server cd was requested. Even typing this now I’m doubting what I saw.. but anyways, no harm in leaving the cd inside.

Though the following makes sense looking back, when I first realized it I was pleasantly surprised: To turn the server on, all that has to be done is turn the computer on. Once Ubuntu-Server is at the login screen (assuming everything is setup properly) then the web server is running. Of course this makes sense, it would be the ultimate security risk leaving the web server logged in, unattended.

I’m blown away how simple it is to get Apache/PHP/MySQL up and going. In Windows I wrote a fairly extensive tutorial on getting started. Using the Debian apt-get command in Ubuntu, I literally had the server running in less than 5 minutes.

Next steps: Setup SendMail, DNS binds and configure the Apache config file to handle the beating the computer is going take.

I just finished installing Ubuntu-Server 5.10 on a server that I built from scratch. At first glance, it seems like normal Ubuntu just stripped down. The install is exactly the same as normal Ubuntu and everything is from the command line. This flavour of Ubuntu just came out in December; I expect some good things to come soon.

I’m excited to get going with this; the server is nothing crazy, the specs below:

- Intel Celeron 2.93Ghz (over clocked to 3.3Ghz) on Soyo motherboard
- 512MB DDR 400MHz (1 stick)
- 3 fans (1 CPU, 2 on the case)
- 20 gig Maxtor (for now)

Next is to get SSH and FTP set up — then I’ll administer the server from my laptop.

Just in case anyone in the future has a similar problem — The installation requirements for compiling the Cisco VPN Linux Client says to download the Kernel Source. This won’t work, download the Kernel Source Headers.

I finally made the move from Cygwin (Windows) to Linux. I’m working on getting a home-built server running, and the first step was getting a mirror image (of the future server) running on my laptop. My distro of choice is Ubuntu. The installation was easy, but I had a few problems getting the video card and my wireless card going. Everything is running smoothly now.. Next step: install Apache, PHP, MySQL, etc.. on my laptop, exactly as I would have it on the server.

My laptop is 4 years old, a Dell Inspiron 1100, if anyone has the same problem getting their video card to work, just do the following:

1. type ‘gedit /etc/X11/xorg.conf’
2. Add the following 2 lines (inside of the #)

Section "Monitor"
Identifier "Generic Monitor"
Option "DPMS"
# add
HorizSync 31.5-48.5
VertRefresh 59-75
# end add
EndSection

Once I have the server done, I’ll be installing Ubuntu-Server on the server. I’ll talk about that when it happens.