George Papayiannis » PHP/MySQL

Twitter: How To (Auto) Gain Followers & Influence Others (No One)

George A. Papayiannis — Sat, 10 Apr 2010 21:35:08 +0000

Update: Don’t do this. It’s seems obvious now, but for some reason I didn’t realize this would get you banned. Indeed the Twitter account used in the experiment was banned. Luckily we explained ourselves, promised to stop and Twitter re-instated the account :)

_________________________________________________________________

A couple months ago, I decided to spend an evening to try and figure out how I could automate the finding & following process on Twitter. First off let me be clear, this wasn’t for my personal twitter account @gapcm. God knows I don’t care about having lots of followers, nor do I care for all the self-promotion and spam that comes along with having lots of followers. That said, there are cases were a person would want to grow their follower base, even at the expense of following a ton of people – that’s were my script comes into play! First a little background..

About two years ago, my fiance and I started a site called Thursday for Dinner. Thursday for Dinner is a video blog dedicated to preserving family recipes. Our mission, as the site says, is to capture our family’s most treasured recipes and make them available to everyone. The Thursday for Dinner twitter account @tfdtv would be used as the guinea pig. Before I started this experiment, @tfdtv had about 600 followers and was listed about 8 times. After about 1.5 months of using my scripts @tfdtv is up to 2160 followers and is listed 126 times.

The idea of the script is this: If you have API credits left and your followers count is greater than your friends count, search Twitter for a specific term (you can also randomly choose from an array of terms). I always want to have more followers than friends (for this script), otherwise you’re looking a bit desperate. For each user who tweeted the given term, check to see if their already in our database as recently added. Assuming their not in the db, and that their followers to friends ratio is within a certain range, then follow them and add their user name to our database. The ratio is calculated like this:

Code:



$ratio = (($followers_count - $friends_count) / $followers_count);

It’s important to follow people within a specific ratio, cause if their follower to friends count is reasonably close, their more than likely going to follow you back. More important than just the ratio, is the term you use to search Twitter. The term you search needs to be relevant to your twitter stream/site content. For Thursday for Dinner, I search Twitter for the term foodie. It’s obvious that people who Tweet the word foodie, are going to like the premise of our site. Just before the script finishes, it checks all users in the database that are older than 3 days. If the given user is not following back, they are unfollowed. Whether their following or not, their user name is removed from the database.

To use this script, your going to need to know PHP/MySQL/Apache/REST. Your also going to need PEAR MDB2 and curl running. If you don’t know what I’m talking about, save yourself the trouble. First things first, you need to create a new MySQL database with the following table:

Code:



CREATE TABLE IF NOT EXISTS `twitter_users` (
  `id` int(11) NOT NULL auto_increment,
  `name` varchar(128) NOT NULL,
  `date` datetime NOT NULL,
  PRIMARY KEY  (`id`)
) ENGINE=MyISAM  DEFAULT CHARSET=latin1 AUTO_INCREMENT=2259 ;

I created a class called SessionControl to manage the whole MySQL process. I won’t show that here, but you can download it in the zip below. The main class is the twitter.class.php. This is were the bulk of the work is done.

Code:



<?

require_once 'sessionControl.class.php';

class Twitter {
    //
    private $username = 'twitter_username';
    private $password = 'twitter_password';
    //
    public function __construct() {

    }
    //
    public function getUsername() {
        return $this->username;
    }
    //
    private function getCURL() {
        if (!$curld = curl_init()) {
            echo "Could not initialize cURL session<br>";
            exit();
        }
        return $curld;
    }
    //
    public function friendshipDetails($u1,$u2) {
        $url = "http://twitter.com/friendships/show.json?source_screen_name=$u1&target_screen_name=$u2";
        //echo $url;
        //exit;
        $curld = $this->getCURL();
        curl_setopt($curld, CURLOPT_GET, true);
        curl_setopt($curld, CURLOPT_URL, $url);
        curl_setopt($curld, CURLOPT_HEADER, false);
        curl_setopt($curld, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curld, CURLOPT_HTTPHEADER, array('Expect:'));
        $output = curl_exec($curld);
        $pile = json_decode($output,true);
        return $pile;

    }
    //
    public function follow($u) {
        $user = SessionControl::getInstance();
        //
        $url = "http://$this->username:$this->password@twitter.com/friendships/create/$u.json";
        $curld = $this->getCURL();
        curl_setopt($curld, CURLOPT_POST, true);
        curl_setopt($curld, CURLOPT_URL, $url);
        curl_setopt($curld, CURLOPT_HEADER, false);
        curl_setopt($curld, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curld, CURLOPT_HTTPHEADER, array('Expect:'));
        $output = curl_exec($curld);
        //$this->out($output);
        $pile = json_decode($output,true);
        //$this->out($pile);
        //
        $sql = "INSERT INTO twitter_users (id, name, date) VALUES (NULL, ".$user->quote($u,"text").", NOW())";
        $user->query($sql);
        //
        return $pile;

    }
    //
    public function isUserInDB($u) {
        $user = SessionControl::getInstance();
        $result = false;
        $sql = 'SELECT * FROM twitter_users WHERE name = '.$user->quote($u,"text");
        $sqlRes = $user->query($sql)->fetchAll();
        if (!empty($sqlRes)) {
            $result = true;
        }
        return $result;
    }
    //
    private function destroy($u) {
        $user = SessionControl::getInstance();
        //
        $url = "http://$this->username:$this->password@twitter.com/friendships/destroy/$u.json";
        $curld = $this->getCURL();
        curl_setopt($curld, CURLOPT_POST, true);
        curl_setopt($curld, CURLOPT_URL, $url);
        curl_setopt($curld, CURLOPT_HEADER, false);
        curl_setopt($curld, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curld, CURLOPT_HTTPHEADER, array('Expect:'));
        $output = curl_exec($curld);
        $pile = json_decode($output,true);
        //
        return $pile;
    }
    //
    private function out($v) {
        echo '<pre>';
        print_r($v);
        echo '</pre>';
    }
    //
    public function unfollow($u = '', $days = 3) {
        $user = SessionControl::getInstance();
        if ($u == '' && $days > 1) {
            $sql = "SELECT * FROM twitter_users WHERE date < ".$user->quote($this->getSQLDate(time() - ($days * 24 * 60 * 60)),'text');
            $uf = $user->query($sql)->fetchAll();
            foreach ($uf as $u) {
                $details = $this->friendshipDetails($this->username,$u->name);
                //$this->out($details);
                //exit();
                if (isset($details['relationship']['target']['following'])) {
                    $key = $details['relationship']['target']['following'];
                } else {
                    $key = 0;
                }
                // not following, remove
                if ($key != 1) {
                    $this->destroy($u->name);
                    $this->out('Removed ' . $u->name);
                }
                //exit();
                // either way remove from db
                $sqlDel = "DELETE FROM twitter_users WHERE id = $u->id";
                $user->query($sqlDel);
            }
        }
    }
    //
    public function getSQLDate($t) {
        return date('Y-m-d H:i:s',$t);
    }
    //
    public function getUser($u) {
        $url = "http://twitter.com/users/show/$u.json";
        $curld = $this->getCURL();
        curl_setopt($curld, CURLOPT_GET, true);
        curl_setopt($curld, CURLOPT_URL, $url);
        curl_setopt($curld, CURLOPT_HEADER, false);
        curl_setopt($curld, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curld, CURLOPT_HTTPHEADER, array('Expect:'));
        $output = curl_exec($curld);
        $pile = json_decode($output,true);
        return $pile;
    }
    //
    public function getSearch($s) {
        $url = "http://search.twitter.com/search.json?q=$s";
        $curld = $this->getCURL();
        curl_setopt($curld, CURLOPT_GET, true);
        curl_setopt($curld, CURLOPT_URL, $url);
        curl_setopt($curld, CURLOPT_HEADER, false);
        curl_setopt($curld, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curld, CURLOPT_HTTPHEADER, array('Expect:'));
        $output = curl_exec($curld);
        $pile = json_decode($output,true);
        return $pile;
    }
    //
    public function getLimit() {
        $url = "http://twitter.com/account/rate_limit_status.json";
        $curld = $this->getCURL();
        curl_setopt($curld, CURLOPT_GET, true);
        curl_setopt($curld, CURLOPT_URL, $url);
        curl_setopt($curld, CURLOPT_HEADER, false);
        curl_setopt($curld, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curld, CURLOPT_HTTPHEADER, array('Expect:'));
        $output = curl_exec($curld);
        $pile = json_decode($output,true);
        return $pile;

    }
    //
    public function __sleep() {
        return array_keys(get_object_vars($this));
    }
    //
    public function __wakeup() {

    }
}

?>

From here, the main control file called go.php runs everything. This is were the steps are executed. Here it is:

Code:



<?php

require_once 'twitter.class.php';

set_time_limit(-1);

$twitter = new Twitter();
$limit = $twitter->getLimit();

function getSearchTerm() {
    //$terms = array("twitter_search_term_1",);
    //$r = mt_rand(0,2);
    //return $terms[$r];
    return 'twitter_search_term';
}

if ($limit['remaining_hits'] > 0){
    $me = $twitter->getUser($twitter->getUsername());
    if (($me['followers_count']) > $me['friends_count']) {
        $sterm = getSearchTerm();
        dump('Hits Left: '.$limit['remaining_hits'].' -- Searching: ' . $sterm);
        $search = $twitter->getSearch($sterm);
        foreach ($search['results'] as $r) {
            $user = $r['from_user'];
            if (!$twitter->isUserInDB($user)) {
                $info = $twitter->getUser($user);
                $friends_count = $info['friends_count'];
                $followers_count = $info['followers_count'];
                if ($friends_count > 0 && $followers_count > 0) {
                    $ratio = (($followers_count - $friends_count) / $followers_count);
                    if ($ratio < 0.2 && $ratio > -0.4) {
                        $twitter->follow($user);
                        dump($user . ' followed -> friends: ' . $friends_count . ' - Followers: ' . $followers_count);
                    }
                }
            } else {
                dump($user . ' in db');
            }
        }
    }
    //
    $twitter->unfollow();
} else {
    dump($limit);
}

function dump($r) {
    echo '<pre>';
    print_r($r);
    echo '</pre>';
}
?>

Key points

Twitter has a limit on the number of API calls you can execute based on a user name and IP address. So don’t run this a) too often and b) on your personal computer. You’ll need to run this on a separate box with a separate IP address.
You’ll need to automate this with a cron job. I run it every 10 minutes. If your hosting company doesn’t allow for cron jobs, you can use WebBasedCron, which I made 4-5 years ago.
The code is released under GPL. Use at your own risk. You can download the files here.

PHP: Removing All Non-Printable (Special Characters) From String

George A. Papayiannis — Fri, 12 Feb 2010 21:24:30 +0000

I was actually trying to find a way to do this in JavaScript, but eventually gave up. I basically want to remove all non-printable special characters from a string. So for a string like this:

Code:



$str = 'Characters like© and ® are not all®wed.';

I would want ‘Characters like and are not allwed.’. The following matches anything in the ASCII range of 0-31 & 128-255 and removes it.

Code:



$str = preg_replace('/[\x00-\x1F\x80-\xFF]/', '', $str);

Apache: Disabling ETags to Improve Performance

George A. Papayiannis — Sun, 31 Jan 2010 02:13:12 +0000

By removing the ETag header, you disable caches and browsers from being able to validate files, so they are forced to rely on your Cache-Control and Expires header. Basically you can remove If-Modified-Since and If-None-Match requests and their 304 Not Modified Responses.

Entity tags (ETags) are a mechanism web servers and the browser use to determine whether a component in the browser’s cache matches one on the origin server. Since ETags are typically constructed using attributes that make them unique to a specific server hosting a site, the tags will not match when a browser gets the original component from one server and later tries to validate that component on a different server.

Doing this is simple. First step make sure the Headers mod is enabled:

Code:



a2enmod headers

Then, within your apache2.conf file, add the following:

Code:



Header unset ETag
FileETag None

PHP: Remove New Lines From a String

George A. Papayiannis — Tue, 19 Jan 2010 20:06:15 +0000

You would think a function to remove new lines from a string would exist, but surprisingly it doesn’t. My first thought was to use the nl2br function. nl2br “inserts HTML line breaks before all newlines in a string” and to then strip the br’s out. So something like this:

Code:



$string = strip_tags(nl2br($string));

The key word above is “before” though. The function preserves the new lines and just adds a br after it. So I decided to write a function to do it for me:

Code:



function removeNewLines($string) {

    $string = str_replace( "\t", ' ', $string );
    $string = str_replace( "\n", ' ', $string );
    $string = str_replace( "\r", ' ', $string );
    $string = str_replace( "\0", ' ', $string );
    $string = str_replace( "\x0B", ' ', $string );

    return $string;

}

XAMPP: Setting Up Virtual Hosts Using Apache Friends

George A. Papayiannis — Thu, 29 Oct 2009 19:44:19 +0000

This is something I do often for development, but never actually knew if I was doing it correctly. I think I’ve finally gotten this right. What I want is: To be able to setup VirtualHosts in XAMPP and also define a PHP include path within that. I’m assuming you generally have an idea how to setup a VirtualHost in Apache.

The first step is to modify your Hosts file to point traffic to your local computer. My hosts file is here C:\WINDOWS\system32\drivers\etc\hosts, you’ll have to search for yours.

Code:



127.0.0.1		www.sun.com
127.0.0.1		sun.com

The next step is to define the VirtualHost within the httpd-vhosts.conf file in the conf directory of your XAMPP setup.

Code:



NameVirtualHost *
<VirtualHost *>
	DocumentRoot "C:\xampp\htdocs"
	ServerName localhost
</VirtualHost>
<VirtualHost *>
	DocumentRoot "C:\xampp\htdocs\sun"
	ServerName www.sun.com
	ServerAlias sun.com
        ErrorDocument 404 /error.php
	<Directory "C:\xampp\htdocs\sun">
		Order allow,deny
		Allow from all
	</Directory>
	<Directory "C:\xampp\htdocs\sun">
                 php_value include_path ".;C:\php\pear;C:\htdocs\sun\xx"
	</Directory>
</VirtualHost>

Using a php_value directive you can define the include path for that VirtualHost right from conf file – thank God.

MySQL: Backup Database and Upload to Amazon S3 via PHP

George A. Papayiannis — Sun, 30 Aug 2009 03:00:50 +0000

I wrote a post a few years ago about a quick way to backup a MySQL database using email. Since then I’ve updated that script to upload the file to Amazon S3 instead of emailing it. I use an Amazon S3 PHP Class you can find here to make it happen. Your also going to need cURL enabled on your server. On Ubuntu/Debian that’s as easy as:

Code:



sudo apt-get install php5-curl

Here is the script:

Code:



require_once 'S3.php';

if (!defined('awsAccessKey')) define('awsAccessKey', 'xxxxxxxxxxxxxxxxxxx');
if (!defined('awsSecretKey')) define('awsSecretKey', 'xxxxxxxxxxxxxxxxxxx');
//
// Check for CURL
if (!extension_loaded('curl') && !@dl(PHP_SHLIB_SUFFIX == 'so' ? 'curl.so' : 'php_curl.dll')) {
	exit("\nERROR: CURL extension not loaded\n\n");
}

$tmpDir = "/tmp/";
$user = "dbuser";
$password = "dbpassword";
$dbName = "dbname";
$prefix = "db_";

$sqlFile = $tmpDir.$prefix.date('Ymd_hisA').".sql";
$attachment = $tmpDir.$prefix.date('Ymd_hisA').".tgz";

$creatBackup = "mysqldump -u ".$user." --password=".$password." ".$dbName." > ".$sqlFile;
$createZip = "tar cvzf $attachment $sqlFile";
exec($creatBackup);
exec($createZip);

if (!file_exists($attachment) || !is_file($attachment)) {
	die("ERROR: No file");
}

$s3 = new S3(awsAccessKey, awsSecretKey);
$result = false;
if ($s3->putObjectFile($attachment, 'bucket-name', baseName($attachment), S3::ACL_PRIVATE)) {
	$result = true;
}

unlink($sqlFile);
unlink($attachment);

Apache: Stripping/Remove the www from URLS

George A. Papayiannis — Wed, 19 Aug 2009 23:26:23 +0000

For reasons I won’t go into, I wanted to force Apache to strip the www from the URLs. So if someone typed in http://www.abc.com, Apache would automatically send them to http://abc.com. First thing to do is make sure ModRewrite is enabled, once that’s done simply put the following in your VirtualHost configuration or in the associated .htaccess file:

Code:



RewriteEngine on
RewriteCond %{HTTP_HOST} ^www\.(.+)$
RewriteRule ^(.*)$ http://%1/$1 [R=301,L]

Setting Different PHP Include Paths per Apache Virtual Hosts

George A. Papayiannis — Thu, 16 Jul 2009 01:13:24 +0000

I was always curious how to set different PHP include paths per Virtual Host in Apache. I didn’t want to use an .htaccess file hack or the PHP function set_include_path. After some digging, I came across a post on a mailing list showing how it’s done using a Directory directive:

Code:



# PHP OPTIONS ###########################
    <Directory /path/to/site>
        php_value include_path ".:/path/to/site:/another/path"
        php_admin_value session.entropy_file "/dev/urandom"
        php_admin_value session.entropy_length "512"
        php_admin_value session.auto_start 1
        php_admin_value session.use_cookies 1
        php_admin_value session.use_only_cookies 1
        php_admin_value session.cookie_lifetime 0
        php_admin_value session.cookie_secure 1
    </Directory>
# PHP OPTIONS ###########################

Setting Up Memcached on Ubuntu/Debian and Windows XAMPP

George A. Papayiannis — Tue, 10 Mar 2009 02:53:32 +0000

I have a big problem with writing things down. This inevitably leads me to spending excessive amount of time looking all over the internet for how to solve a problem. Most recently I was trying to remember the steps to get Memcached going on my Debian server and on my laptop.

Luckily I came across this comment on php.net, which explains nicely how to get this going on Debian/Linux. The only thing I needed to do was make sure g++ was installed (i.e. apt-get install g++). Here are the steps:

Code:



-------------------------
# Prerequisite Install
-------------------------
# Download & install libevent (memcached dependency)
wget http://www.monkey.org/~provos/libevent-1.4.8-stable.tar.gz
tar xfz libevent-1.4.8-stable.tar.gz
cd libevent-1.4.8-stable
./configure && make && sudo make install

# Create a symlink to libevent
sudo ln -s /usr/local/lib/libevent-1.4.so.2 /usr/lib

# Download & install memcached
wget http://danga.com/memcached/dist/memcached-1.2.6.tar.gz
tar xfz memcached-1.2.6.tar.gz
cd memcached-1.2.6
./configure && make && sudo make install

# Run memcached as a daemon (d = daemon, m = memory, u = user, l = IP to listen to, p = port)
memcached -d -m 1024 -u root -l 127.0.0.1 -p 11211

-------------------------
# PHP5-Memcache Install
-------------------------
# Download the extension module
apt-get install php5-memcache

# Edit /etc/php5/conf.d/memcache.ini and uncomment the following line by removing the semi-colon
extension=memcache.so

# Restart apache
/etc/init.d/apache2 restart

-------------------------
# Test Install
-------------------------
# Create a file 'memcache_test.php' in your webroot and paste the following:
<?php
$memcache = new Memcache;
$memcache->connect('localhost', 11211) or die ("Could not connect");

$version = $memcache->getVersion();
echo "Server's version: ".$version."<br/>\n";

$tmp_object = new stdClass;
$tmp_object->str_attr = 'test';
$tmp_object->int_attr = 123;

$memcache->set('key', $tmp_object, false, 10) or die ("Failed to save data at the server");
echo "Store data in the cache (data will expire in 10 seconds)<br/>\n";

$get_result = $memcache->get('key');
echo "Data from the cache:<br/>\n";

var_dump($get_result);
?>

In Windows (assuming you have XAMPP installed) go into your php.ini (check phpinfo for location) file and un-comment the following:

Code:



...
extension=php_memcache.dll
...
[Memcache]
memcache.allow_failover = 1
memcache.max_failover_attempts=20
memcache.chunk_size =8192
memcache.default_port = 11211
...

Then head to Jellycan Code and download the Windows memcached port (binary). At this point you should be able to launch the .exe and run the sample code above.

RSA: Encrypting in JavaScript and Decrypting in PHP

George A. Papayiannis — Mon, 27 Oct 2008 21:07:48 +0000

I came across a really cool JavaScript library a while back that allows you to do client-side RSA encryption & decryption. If you don’t know much about RSA, read up on it here. Essentially, it’s an algorithm for public/private key encryption/decryption. You encrypt using the Public Key and Modulus and decrypt using the Private Key and Modulus.

More often than not, you would want to encrypt client-side (browser) and decrypt server-side. For my purposes, I wanted to encrypt passwords 13 characters or less in JavaScript and decrypt them in PHP. The creator of the RSA library also created a convenient Windows application that generates RSA keys for you. Using that program I created the following 128bit keys:

e = Public Key = 553799486327459813656784787218239817 (6aa86c39bbe678f7f7967a587a1149)
d = Private key = 1401845535567450041611005523755666809 (10dfc5235ef69148bdfdc6832860d79)
m = Modulus = 11570601966616835094916890432003700913 (8b46ab8a951615d07b66bdd2420f8b1)

By default the RSA library outputs encrypted strings in Hex, but because there is no built in PHP function to convert from Hex to Binary, I decided to modify the RSA.js file slightly (within the encryptedString function – line 62) to output the full Integer:

var text = key.radix == 16 ? biToDecimal(crypt)/*biToHex(crypt)*/ : biToString(crypt, key.radix);

The Key Generator also conveniently creates the JavaScript needed to create the new key pair. I’ve removed the decryption (Private) key:

setMaxDigits(19);
key = new RSAKeyPair(”6aa86c39bbe678f7f7967a587a1149″, “”, “8b46ab8a951615d07b66bdd2420f8b1″);
w.value = encryptedString(key,”1234567890123″ + “\x01″);

The trick here is adding “\x01″ to the end of the string you want to encrypt. The PEAR library used to decrypt the encrypted string will look for this, and throws an error if not found. That said, the above encrypted string is:

1276850306890326374886324100793107985

The PEAR package used to decrypt this string is called Crypt_RSA. You install this just like any other PEAR package:

PEAR install Crypt_RSA

The package depends on one of 3 Math libraries (BCMath, GMP or big_int), of which BCMath is slowest but installed by default (at least in my version of PHP 5.2.0-8+etch13 & XAMPP-win). The Crypt_RSA package works in binary, so the keys and encrypted text needs to be converted before calling the package. The following code converts the Integers to Binary and performs the decryption. Bare in mind your dealing with very large numbers, so you need to use one of the above Math libraries to work with them.

require_once ‘Crypt/RSA.php’;
//
$wrapper_name = “BCMath”;
$math_obj = &Crypt_RSA_MathLoader::loadWrapper($wrapper_name);
//
$d = $math_obj->int2bin(”1401845535567450041611005523755666809″);
$m = $math_obj->int2bin(”11570601966616835094916890432003700913″);
//
$pk = new Crypt_RSA_Key($m, $d, “private”, $wrapper_name);
$rsa_obj = new Crypt_RSA;
$rsa_obj->setParams(array(’dec_key’ => $pk));
$dec = $rsa_obj->decryptBinary($math_obj->int2bin(”1276850306890326374886324100793107985″));
echo $dec;

You can download my sample code here.