Archive for September, 2006


How to: PHP on the i5 – The i5 PHP API Toolkit and Calling RPG/CL Programs

Monday, September 25th, 2006

Before I begin to talk about PHP on the i5, a little history is in order. PHP was created by Rasmus Lerdorf in 1994 to replace a personal set of Perl scripts he used to maintain his website. He officially released version 1.0 in 1995 as a set of CGI binaries written in C, under the GNU General Public License. In 1997 two developers at the Israel Institute of Technology, Zeev Suraski and Andi Gutmans, re-wrote the parser releasing PHP 3.0 in June 1998. Zeev and Andi went on to create Zend Technologies, a company focused on pushing PHP for mission critical business applications. Rasmus currently works for Yahoo! and is actively involved in the development of PHP.

PHP and RPG have one thing very much in common, when they first came about they were meant to do pretty simple tasks and their names reflected that, Personal Home Page and Report Program Generator. As the years went on, both languages evolved, with RPG and PHP gaining various built-in functions (BIFs), new programming constructs, XML support, and more.

Zend provides two main components to the i5, the Zend Core and the Zend Platform. The Zend Core provides, among many things, the essentials needed to run PHP on the i5. This includes the PHP runtime, the i5 PHP Toolkit, native DB2 support, automatic security updates, a web-based administration console, and more. The Zend Platform provides addition features to the Core, including run-time code optimization, dynamic content caching, PHP/Java integration, and more.

In this article I will begin by giving an overview of the Zend Core installation. Next I will explain what the i5 PHP API Toolkit is and conclude by going through an example of calling an RPG program through PHP.

Installation

The Zend Core makes the installation of PHP on the i5 painless. To start off, create a SAVF in QGPL under the name ZCOREPROD. Then download the Zend Core for the i5/OS to your computer and transfer (via FTP) the included SAVF to your i5. There are numerous open source FTP applications available online, FileZilla is one example. Remember to do a binary FTP upload. Once uploaded, perform a RSTLICPGM command to start the installation, i.e.:

Code:
RSTLICPGM LICPGM(1ZCORE5) DEV(*SAVF) SAVF(QGPL/ZCOREPROD)  

Once the installation completes, you can test everything went OK by pointing your browser to: http://:8000/ZendCore/. This should take you to the web-based administration console for the Zend Core. More information on the installation can be found in the Zend Core for the i5/OS manual and i5php.net. You should now be able to put your scripts in /www/zendcore/htdocs/ and access the files through http://:8000/.

Zend also provides the Zend Studio for the i5/OS, a Java based IDE to assist in the development of PHP applications. IBM and Zend are currently working on the Eclipse PHP IDE project, with version 1.0 set to be released December 2006. For the purposes of this article, I’ll be using WDSC to work with the i5.

i5 PHP API Toolkit

The i5 PHP API Toolkit (iPAT) is an extension built into the Zend Core, giving PHP developers easy access to their i5, natively though PHP. The toolkit is broken apart into seven parts; these include Connection Management, CL Commands, Program Calls, Data Retrieval, Native File Access, System Values and Data Areas. All functions in iPAT begin with ‘i5_’ to ensure no namespace problems come about. The figure below shows the iPAT organizational layout, along with the associated functions.

i5 PHP API Toolkit

Calling RPG/CL program from PHP

The lifecycle of a PHP script calling an RPG/CL program has four main steps, these are connect, prepare, execute and close. A connection first needs to be made to the i5, this is done by using the i5_connect function, and passing in as parameters the i5 IP address, a username and password. The PHP code below shows this:

Code:
$conn = i5_connect("localhost", "gpapayia", "secret");

if (!$conn) {
    throw_error("i5_connect");
    exit();
}

If the connection returns false, then we know something went wrong. I created a function called throw_error, which takes as input a function name and outputs the error number and associated message from the last executed command. The PHP code below shows this function:

Code:
function throw_error($func) {
	echo "Error in function: ".$func." --- ";
	echo "Error Number: ".i5_errno()." --- ";
	echo "Error Message: ".i5_errormsg()."<br>";
}

Next, the program in question needs to be loaded into memory and prepared to be run. This is done by executing the i5_program_prepare function and passing in as parameters the name of the i5 program and a description of the inputs expected by the RPG code. Before I continue to explain this, I’ll introduce the RPG program I plan on using. For the purposes of this article, the example code I made is simple and serves only as illustration as to what can be done. The program takes as input a product id, a store location, and a price. The program updates the price by adding 10.99 and finishes. The RPG code below shows the stored procedure:

Code:
     C     *ENTRY        PLIST
     C                   PARM                    prod_id           7
     C                   PARM                    store_loc        10
     C                   PARM                    price             5 2
      /FREE
       price = price + 10.99;
       *INLR = *ON;
      /END-FRE

Now back to the PHP coding, once the connection is made to the i5, we need to load the given RPG program into memory and get it ready to be executed. This is done by using the i5_program_prepare function, and passing in as parameters the name of the RPG program and a description of the inputs expected. Describing the input variables may seem complicated at first glance, but it’s really is quite intuitive and straight forward. Each input variable is enclosed in an associative array by specifying the Name of the input, the IO being performed, the variable Type being passed in, and finally the Length. The PHP code below shows how this would be done. Notice how the third input, PRICE, has an IO field of I5_INOUT, since I plan on modifying this value from within the RPG code. There is no real difference between calling an RPG or CL program, just specify the correct program name in i5_program_prepare function and everything else stays the same. Again, after i5_program_prepare is executed, I check to make sure no error occurred.

Code:
$description = array(
    array(
    	"Name"=>"PROD_ID",
    	"IO"=>I5_IN,
    	"Type"=>I5_TYPE_CHAR,
    	"Length"=>"7"
    ),
    array(
    	"Name"=>"STORE_LOC",
    	"IO"=>I5_IN,
    	"Type"=>I5_TYPE_CHAR,
    	"Length"=>"10"
    ),
    array(
    	"Name"=>"PRICE",
    	"IO"=>I5_INOUT,
    	"Type"=>I5_TYPE_PACKED,
    	"Length"=>"5.2"
    ),
);

$pgm = i5_program_prepare("QGPL/GEOPGRM", $description);

if (!$pgm) {
    throw_error("i5_program_prepare");
    exit();
}

Notice the return value from the i5_program_prepare function is stored in the variable $pgm. From this point on, any time I want to refer to my RPG program GEOPGRM, I’ll use the variable $pgm. At this moment, the program is loaded into memory and is ready to be run. To execute the program, all I have to do is call the i5_program_call function and pass as parameters the prepared program ($pgm), my input parameters and where I want the output parameters to be stored. The PHP code below shows how this would be done. As above, the input and output parameters are stored in associative arrays. Notice in the $paramOut array, I specified for each given input to the RPG program, the variable names I want to refer to after the i5_program_call completes. For PRICE, I coded in AMOUNT, so after I execute my RPG program, I can get at the PRICE result by using a local PHP variable $AMOUNT. After I execute the RPG program, I check to see if an error occurred, assuming everything went OK, I output the input/output variables from my RPG program using the PHP echo function.

Code:
$parameter = array(
    "PROD_ID"=>"xyz101",
    "STORE_LOC"=>"a1001",
    "PRICE"=>0.00
);

$parmOut = array(
    "PROD_ID"=>"PROD_ID",
    "STORE_LOC"=>"STORE_LOC",
    "PRICE"=>"AMOUNT",
);

$ret = i5_program_call($pgm, $parameter, $parmOut);

if (!$ret) {
    throw_error("i5_program_call");
    exit();
}

echo "Product Id: ".$PROD_ID."<br>";
echo "Store Location: ".$STORE_LOC."<br>";
echo "Price: ".$AMOUNT."<br>";

At this point, my program executed correctly and I am ready to close my program (unload from memory) and close my connection to the i5. This is handled through the functions i5_program_close, taking as a parameter the prepared program, and i5_close, taking as a parameter the connection. The two lines of PHP code to handle this are listed below.

Code:
i5_program_close($pgm);
i5_close($conn);

One reason why PHP has become so popular is that it makes seemingly difficult tasks easy, this in turn frees the programmer to be innovative and spend more time thinking through ideas, rather than coding. Furthermore, resources to assist PHP developers are available all over the internet, including one of the best manuals for any programming language at php.net.

PHP Source Listing
RPG Source Listing

Flash Lite, Nokia and MTJ

Wednesday, September 20th, 2006

The North American market is so sheltered from the rest of the world when it comes to mobile devices. I came to this realization when I was in Greece for the 2004 Olympics, walked into a Samsung booth at the Olympic village and saw the latest phones to come on the market — 3.2 mega pixel camera, phone-to-phone video conferencing, brilliant screens & UIs — no big deal to them.

Last week during the php|works conference, I sat through a talk about Flash + PHP by Christian Wenz. During the session, he briefly mentioned Flash Lite — Flash on mobile devices? I can’t believe I had never heard of this before, its gaining lots of momentum all over the world (Japan, Korea, Europe) there are over 140 supported devices world wide. A full list can be seen here. From the list, its easy to see the largest supporter of Flash enabled devices is Nokia.

As mentioned in a previous post, the MTJ DSDP project had its first M1 release last week. This project is being sponsored mainly by Nokia, with support from other companies like IBM, Sony, Sprint, etc… There seems to be a continuous struggle about what developers should use for UIs, Flash Lite or J2ME, but it appears the answer keeps coming back to: it depends on the application and what you’re doing. The fact remains, with Nokia and so many other companies supporting Flash Lite, there is no way Nokia has been contributing so much effort to the MTJ project, without plans to make an open source Flash Lite development extension to MTJ.

The first official Flash Lite enabled phone was recently released in the US, you probably heard of saw commercials it: Verizon Wireless Chocolate. Click here to see a video of the LG VX8500 phone in action. This phone was released about a year ago in Korea.

5-second RSS parser w/ simpleXML

Tuesday, September 19th, 2006

During Rasmus’s talk last week at the php|works conference, he showed how easy it is to parse XML in PHP 5, using simpleXML. In his example, he used the Flickr public photo RSS feed, to parse and output. A lot of people don’t know about this feed, which gets updated the moment a new photo gets uploaded. Furthermore, a lot of people upload their personal photos, and make them private after their uploaded. Click here to see it in action. Here are the 2 lines of code:

Code:
<?php
    $url = 'http://www.flickr.com/services/feeds/photos_public.gne';
    foreach(simplexml_load_file($url)->entry as $it) echo $it->content;
?> 

Be sure to mark you photos as private before you upload them..

MTJ 0.7M1 released

Tuesday, September 19th, 2006

The Mobile Tools for the Java Platform (MTJ) Eclipse project had its first release this past Friday. It’s still early and there are many known issues, but it’s worth a try, especially if you have a Nokia phone. MTJ currently supports (has an extension) for Nokia PC Suite toolkits, so you’ll have to go to the Nokia website and download the toolkit for your phone. You’re also going to need a Java wrapper from the Nokia website, all the information is here.

SemCodeFix: A plugin to display preformatted code in WordPress

Saturday, September 16th, 2006

SemCodeFix is a WordPress plugin I made to display preformatted code correctly. This plugin can display any type of code, html, text, scripts, etc. Using the plugin is easy, just wrap your text in a code tag from within the WordPress editor. Here’s an example of what it would look like. The code below is Python, its a solution to the Eight queens puzzle.

Code:
def n_queens(n, width):
    if n == 0:
        return [[]] # one solution, the empty list
    else:
        return add_queen(n-1, width, n_queens(n-1, width))

def add_queen(new_row, width, previous_solutions):
    solutions = []
    for sol in previous_solutions:
        for new_col in range(width):
            if safe_queen(new_row, new_col, sol):
                solutions.append(sol + [new_col])
    return solutions

def safe_queen(new_row, new_col, sol):
    for row in range(new_row):
        if (sol[row] == new_col or  # same column clash
            sol[row] + row == new_col + new_row or  # diagonal clash
            sol[row] - row == new_col - new_row):   # other diagonal
                return 0
    return 1

for sol in n_queens(8, 8):
   print sol

To activate/use this plugin, do the following:

1. Place the plugin file in the plugin directory of your WordPress installation and activate it.
2. Copy and paste the CSS into your local.css file of your theme’s directory.
3. Depending on your theme, your might want to change the width value in the div.code_child css style.
4. Any code you have, just wrap it in a code tag from within the editor.

When installed, the plugin will override the native code styles of your theme and display output as above.
Download the plugin file
Download the CSS file

Good luck

php|works 2006: Every time Linux boots a penguin gets its wings

Friday, September 15th, 2006

The php|works conference has come to an end, and overall it was a good time.
Here are some links to read more about what happened.

Linux

php|works Toronto Conference 2006: Day 3

Friday, September 15th, 2006

Today was a light day, with only 3 talks and the day ending by 12:30pm. The highways were a mess this morning, I barely made it in time for the first talk. I’ll link to slides as they become available. Here is a quick summary of the presentations I listened in on…

9:00am – 10:00am: PHP-GTK2 by Andrei Zmievski

GTK natively is designed to work with many languages, but PHP? I guess if you really love PHP (and totally hate C), then this is for you, when making desktop application. I’ve havent used PHP-GTK extensively, but I can’t imagine it would scale well for large applications. Regardless, the talk was interesting to see whats new in PHP-GTK2, especially when it was coming from its creator.

Andrei’s Slides

10:15am – 11:15am: Handling Data Without Databases by Elliott White III

For one reason or another, it’s possible that a PHP developer would not have access or want to use a database in their application. This talk covered just that, what could a programmer do when database access is not an option. Eli covered five way to go about this, I’ll cover each briefly..

Filesystem Database

In a Filesystem database, every table and row in your database would become its own directory. This is not a requirement, but thats generally what would happen. With a good directory structure and smart intermediate PHP code to control file access, this could be a decent alternative to standard databases. Eli has examples and code in his slides.

XML

XML is surprisingly similar to a filesystem database, just in this case the directory structure becomes the tags, while the data gets stored directly in the XML. With strong XML support in PHP 5, this could be one of the easier solutions.

PHP Files

This gets into slightly more complex ways of storing data, but can be really effective, especially when using some of the built in capabilities of PHP. If you have an array you want to store in a file or XML, why not just call var_export() and store the resulting string in another PHP file? var_export([array]) takes an array, and returns a string array definition, that could be used to re-generate the array. Just write this string to a PHP file, and include() it as needed.

Serialize/Unserialize

I most commonly use serialization to store objects or data during page transitions. Behyond that, serialization is a great way to store objects in files, databases, etc. Read more about serialization here. I had an interesting problem a little while ago, on how to serialize a class using the __sleep() function.

SQLite

SQLite emulates (mimics) a relational database entirely in memory. All data is stored in a single flat file, which makes backing up and getting at your data easy. These days SQLite is very powerful and gives you a lot of the functionality you would normally expect in a standard relational database. Since this is really just faking a relational database, use it with care and only for small amounts of data.

Eli’s Slides

11:30am – 12:30pm: PHP and Mail: Best Practices by Wez Furlong

This was filled with lots of good information, but brought back bad memories of my battles with email related issues (server configurations, etc). In retrospect, I should have sat in on another talk, since I’ve had so much experience in this area. These days I use the PEAR-Mail package for most of my email related needs.

Wez’s slides

php|works Toronto Conference 2006: Day 2

Thursday, September 14th, 2006

Day 2 of php|works is a wrap, another great day of tutorials and information. Above that, the Holiday Inn is providing great service, refreshments and food. I’ll link to slides below as they become available. A summary of the day and interesting stuff learned follows…

9:00am – 10:00am: Microsoft Platforms for the PHP Developer by Joe Stagner

The day started off with Joe Stagner from Microsoft Canada giving us a one hour talk on PHP and Microsoft. There were a bunch of people there from MS giving away .NET migration books, Visual Studio Express CDs and other propaganda. I guess in the worst case, they would want us to develop/deploy on the Windows platform instead of Linux.

The talk went on about a lot of things including native COM access through PHP, .NET integration with PHP, PHP4Mono and more.

Joe’s slides

10:15am – 11:15am: Cache For Cash by Ilia Alshanetsky

I sat in on two outstanding talks today, the first was this one, and the second was the next (Flash + PHP). So what’s the motivation for caching? Simply put, the PHP engine and MySQL operations present an overhead in parsing, compiling, processing, etc… when displaying a single page. In most cases though, not everything on the page is dynamic, and doesn’t require the PHP engine to be invoked every time.

In Cache for Cash, Ilia went through six ways to cache and how one would go about implementing them. I’ll briefly go through each…

Complete Page Caching

As the subtitle says, this is the process of taking a dynamic page, and storing the generated contents in an HTML file. When the site is loaded, instead of running the PHP script, the server would load the static HTML page.

I have no way of verifying this, but I’m pretty sure WordPress, Blogger and other large scale hosted blogging engines use complete page caching. This makes sense, since most people won’t make a new post that often, and when they do, the entire page could be re-generated instantly.

Ilia quoted a greater than 40% improvement in latency when caching his pages using the blogging engine Serendipity. Caching could be taken even further, to check the browser encoding header for GZip, and if so, send the GZip’d version of the page, instead of the original. In many cases this could cut down in half the amount of data being transferred over the wire.

Content Pre-generation

Imagine you have an online store front selling 1000 products (0 through 999). Each of these products have a product description page, say by desc.php?id=xxx, the xxx would represent the numbers ranging from 0 through to 999. Since your prices wouldn’t change that often, you could write a script to generate all 1000 pages, and have the application call these pre-generated, cached files. The general theme here is simple: remove PHP from whatever you can. This is what Livedocs does; they improved their latency from 611.2 ms all the way down to 2.37 ms, because they removed both PHP and XSLT from the equation.

The secret here is not to just store the static pages on your normal HTTP server. Since the pages are static, you should use a light weight server, like lighthttpd, which would have a very fast response time. This type of caching doesn’t come without its problems; first off disk space could become an issue through so many files being generated. Furthermore, ext begins to suffer when the file system surpasses a certain number of files. Secondly, you could be easily pre-generating pages which might not ever be used. If you have thousands of pages, your wasting disk space, and before the page is ever used, it might need to be re-generated again.

On-Demand Caching

The idea here is to generate what you need, when you need it. Depending on what you’re doing, this could be a really good option. When I first heard On-Demand Caching, the first thing that came to mind, was MagpieRSS. This package grabs a given RSS feed (or XML) on demand, and gives you the ability to use the results in your application on the fly. If the headers of the given feed indicated that the XML hadn’t changed, then the cached version of the file was used locally for parsing, instead of downloading it all over again.

Ilia gave a cool idea on how to implement On-Demand Caching: Have all pages initially display a static HTML page. The short PHP code that would load this static page, would first check if the given HTML had been updated. If so, the script would throw a 404 error, instantly sending the user to the 404 page. There is nothing stopping you from creating a custom PHP 404 page, that looks at the URL it came from, and auto-generates the updated (new) content for that page. The new page would not only be displayed to the user, but also cached for future requests. The only issue here is that your logs might get cluttered with false error messages.

Partial Page Caching

The idea here is to have a light weight http server providing all static pages, in conjunction with dynamically generated content. Full caching is not possible in this case, since the pages have dynamic content, that can’t be avoided. To improve the latency of the dynamic content though, an APC style middleware should be used. I’ve gone through APC briefly in the summary of day 1, but here’s a bit more. With APC you have the ability to manually put data into shared memory. There are basically three functions you need to know:

apc_store([key],[var],[time till expiry]); — place data (var) into memory using a [key] and have it expire via the third parameter
apc_delete([key]); — remove the data associated with [key] from dynamic memory
[var] = apc_fetch([key]); — grab data from shared memory using a [key].

SQL Caching

Ilia made an interesting statement at this point, he said many application work with a database, but few do it efficiently. We learned about three types of SQL caching:

DB SQL caching: Say your blog has 10 commonly used search terms. Rather than having your site go through these expensive searches each time, you could cache them in a separate SQL table. This makes sense, since a single table read in MySQL is very fast, and the results would come back almost instantly.

On-Disk SQL Caching: Using the example above, rather than storing the search results in a single db table, you could store the results in a file. This has a similar idea to page caching.

In-Memory SQL caching: This is the same idea as above, just you would be storing SQL data in memory using APC or SHMOP. APC is meant for PHP Opcode caching, but there’s nothing stopping your from storing SQL results data instead.

Browser Caching

This type of caching virtually reduces the data sent from the server to 0. The problem here, is that browser caching is not guaranteed and every browser implements this differently. There are three ways to implement browser caching from a PHP application:

header(“Expires:”…. This tells the browser to hold this page till a certain time
header(“Last-Modified:”…. This puts all the control in the browser, as to when to re-fetch the page
header(“Etag:”…. This is the most reliable method, by sending a unique identifier check-sum to the browser, notifying it, that the page has changed

Ilia’s slides

11:30am – 12:30pm: Flash + PHP: an Alternative to AJAX by Christian Wenz

Christian is a great presenter. He has a distinct style, were he likes to talk and show examples rather than follow/read off slides. He tries to make the talks seem like he’s coming up ideas on the fly, but its easy to see he’s thought through everything he wanted to say.

This talk focused on how a PHP programmer can use Flash to easily make rich UI applications. The presentation covered an introduction to Flash, how it integrates with PHP, demo applications, and finally Macromedia Flex. I’ll just cover a few highlights.

To begin with, Flash is semi-open standard, along those lines third part tools exist for developing Flash applications, but their always behind the Adobe developed Flash development studio. The Flash scripting language is called Action Script, similar to Dojo style JavaScript, Action Script is an ECMA approved Script Standard and in recent years has really begun to gain a lot of power and functionality.

An interesting development coming up is Flash Light. It appears this is going to be a light weight version of Flash, which would be used to develop application for mobile devices. These application would run standalone on a phones OS, outside of any type of browser.

Flash and PHP

A developer has many options to connect with PHP when creating Flash applications. The original way was to use the Action Script functions loadVariable and getURL. From here loadVars came about, which was an OOP version of loadVariable, now allowing for a callback function to handle results.

Probably the most interesting way to connect with backend logic is to use Flash’s inherit ability to work with Web Services (SOA). With SOA, the developer can create their business logic, wrap it in a WSDL and have a desktop application, web application, Flash app etc. all run off the same code. The world is moving towards SOA, even MS ;)

The most complicated method of connecting with a backend, had to be Flash Remoting. This was some intense binary data transmission method, based off of AMF (Action Message Format). The Macromedia supplied version of AMF is very expensive, but luckily there are open source alternatives including AMFPHP and PHPObject.

Christian’s slides

1:30pm – 2:30pm: Derick’s Ranting Hour by Derick Rethans

This was a light-hearted, open discussion on PHP, by the people who actually develop the language. The slides include funny examples of bug reports, emails and more.

Derick’s Slides

2:45pm – 3:45pm: The State of AOP in PHP by Sebastian Bergmann

In this talk, Sebastian gave an introduction to Aspect Oriented Programming, examples of syntax, theory and how he was going about implementing this in PHP. To get a good understanding of AOP, you’ll need to do some good research.

Sebastian’s Slides

php|works Toronto Conference 2006: Day 1

Wednesday, September 13th, 2006

I just went through day 1 of the php/db works 2006 conference here in Toronto at the Holiday Inn Yorkdale. The conference continues tomorrow and Friday, with lots of interesting talks to come. I’ll link to slides below when they become available. Surprisingly, one of the main sponsors is Microsoft… I actually got wind of something I never thought I’d hear: php.net?

9:00am – 10:00am: Opening Keynote by Rasmus Lerdorf

This was by far the best talk/presentation today. If you don’t know about Rasmus, from his website:

Rasmus Lerdorf is known for having gotten the PHP project off the ground in 1995, the mod_info Apache module and he can be blamed for the ANSI92 SQL-defying LIMIT clause in mSQL 1.x which has now, at least conceptually, crept into both MySQL and PostgreSQL. Prior to joining Yahoo!…..

His talk was called Getting Rich with PHP 5, joking about how to make money with Web 2.0:

1. Build tagged, socially networked, Web 2.0ish, Ajaxy thing
2. Get 500,000 users quickly (and have it actually work)
3. Profit

All kidding aside, his talk was great — packed with good information including why the web is broken and examples of the risk it causes. PHP 5 has great XML (and WebServices) support, along those lines Rasmus talked about libxml2, SimpleXML and showed examples of using Yahoo! APIs(including Geo-tagging and Yahoo! widgets).

A large portion of his presentation was devoted to the following: He had a simple pre-made PHP app ready and wanted to tune his server, code, anything really, to maximize the number of fetches/second and to minimize the mean time to first response. For his example, he assumed:

500,000 users at an average of 100 front/backend requests daily = 578 requests/second. x 3 (because your traffic won’t be evenly distributed) = ~1700 requests/second.

Tests for throughput were done using http_load. I had never heard of this program, but from the talk I was really impressed. It runs multiple fetches on a single process of the server, displaying back to the user useful information without bogging down the server. Initial tests of his application brought back 17.2 fetches/second 284 mean millisecond to first response. 284 milliseconds might not sound like a lot, but when comparing to large scale sites these days, anything above 50-100 milliseconds is bordering unacceptable. Anyways, based on these results Rasmus would need 100 servers to makeup 1700 requests/second!

To tune his server and application he investigated how the server was handling the entire process (in % of CPU), using another program called Callgrind. This application can create output like the one seen below (taken from his slides):

pgssl.png

Eventually, using Callgrind, he noticed that 40% of his CPU time was being spent on parsing and compiling the PHP. This led him nicely into his latest development of APC or Alternative PHP Cache. Currently a PECL package, APC is an opcode cache with a lot of cool features you can use to optimize your server. After all his optimizing, using http_load he was up to 1100 fetches/second and 3.6 mean millisecond to first response! His server wasn’t even that crazy, an AMD 1.8 with 1 GB RAM.

Rasmus’s slides

10:15am – 11:15am: Scaling with AJAX by Josh Eichorn

One complaint I have about conferences that have 1 hour sessions, is that due to time constraints, content remains high-level and a lot of important details are left out. This causes potentially great ideas for talks, to focus on general recommendations or pointers on how to do something.

In all fairness to Josh, he was told extremely last minute that they needed him to present, and didn’t have a lot of time to prepare. Overall though, the talk was good and gave a lot of great ideas on how to use AJAX effectively. I do wish though there was some discussion on Comet — Then again, nothing was stopping me from asking a question.

Josh’s slides

11:30am – 12:30pm: High Volume PHP & MySQL Scaling Techniques by Elliott White III

Randomly a little while ago, I managed to make my way to the Digg.com blog and saw a post about Eli White and his new PHP book (PHP 5 in Practice). The post went on to say that Eli would be giving talks at PHP works and other PHP conferences in the coming months. As fate would have it, I sat through 2 of his presentations today, this one and Top PHP Tricks (see below).

Eli’s talk referred back often to his experiences working at the Hubble Space Telescope, but if you ask me, he really meant Digg. He was also using Macromedia Dreamweaver for his examples, and I could see in his project explorer various folders for Digg, each for a different version.

Eli started off by talking about load balancers and MySQL master/slave set-ups one could use today. He touched on static servers, pre-generating/caching content and also focused on APC (see above). Most of his time was spent talking about two specific things though, memcached and Sharding. memcached is a distributed memory caching system used to improve the performance of dynamic websites, by caching data (instead of reading from a db) into memory. It’s used by a lot of sites these days including Livejournal (which it was made originally for), Slashdot and Digg. The second focus was Sharding, which is basically methods on distributing a single database across multiple servers. He concentrated on four main type of sharding, these included Table, Range, Date and Interlaced.

Eli’s slides

1:30pm – 2:30pm: New (and Old) Trends in Web Security by Christian Wenz

I always had an affinity for security and recognized the importance of putting the time into developing secure code. I went so far at times to store copies of the users cookies and sessions in the database, to make sure no tampering occurred when logging into the system.

Christian focused on five main topics, these included:

XSS (Cross-site scripting)
SQL Injections
New types of Injection
Hacking Blogs
Avoiding Automation

The talk was packed numerous examples about what could happen if you didn’t filter your input. A creative (I thought) example of an XSS Injection I saw was:

Code:
<script>(new Image()).src = 'http://abc.com/code.php'</script>

Imagine a simple form, you don’t filter input, and you output the contents of the form back to the user. By entering the above code in the form, a user could execute external code on your server. Christian even showed new types of Injections he came up with using XPath, which followed along the same lines as an SQL Injections.

The basic lesson learned here is filter your input.

Christian’s slides

2:45pm – 3:45pm: Top PHP Tricks by Elliott White III

This was Eli’s second talk today; the first was about Scaling PHP and MySQL. In this talk, he essentially presented the useful features/abilities of PHP that we could potentially use in the future. He talked about various functions and extensions, including:

range() – creating an array of a range of elements
PHP SpellCheck – A spell checking extension available in PHP
The @ – For checking whether a value is set or not
ignore_user_abort() – If the user aborts a script, have the server finish execution
set_time_limit() – Force a limit on the amount of time a script can take

There were others I can’t remember right now.

Eli’s slides

4:00pm – 4:45pm: Organize Your Projects by Paul M. Jones

In this talk Paul focused on the all important issue of project planning and organizing development efforts. From his blog post on his talk:

The talk will consist of a one-sentence lesson for you to remember when deciding how to organize your PHP project, whether it’s a library, an application, a CMS, or a framework. I’ll then explain how to apply the One Lesson in your project, and the various follow-on effects the One Lesson will have on your project organization.

The one sentence was basically don’t pollute the namespace — use a top level identifier for your classes and functions.

Paul’s slides

J2ME: MTJ slides, Profiles & Config., Bluetooth Apps, UI, etc..

Monday, September 11th, 2006

EclipseCon Slides

I came across some great slides today, from a presentation done by Kevin Horowitz at EclipseCon 2006. Kevin is an IBM employee based out of Boca Raton in Florida. The slides are called ‘Taking it on the road – Developer tools for J2ME’
The slides answer six main questions, these are:

What is the J2ME™ environment?
What is the MTJ Project?
What are the plans?
What is out there already?
How is MTJ developing to meet the need?
Where can I get it and how can I help?


Profiles and Configurations

I know I talked about this a bit in a previous post, but I want to look into this a bit more. Profiles are at the core of J2ME – they define which types of device are supported and are built on top of configurations. The main type of configuration for J2ME (along the same lines, cell phones) is CLDC or Connected Limited Device Configuration. CLDC is comprised the small subset of Java classes needed to run the Java virtual machine. Built on top of CLDC is MIDP or Mobile Information Device Profile. There are two widely accepted types of MIDP, version 1.0 and 2.0. In general they provide an API for LCD GUI development, with 2.0 providing a two dimensional gaming API.

An interesting type of profile, designed for embedded (‘headless’) devices, is IMP or Information Module Profile. These devices have no LCD display and limited network connectivity (i.e. Vending machine). IMP is very similar to MIDP, just without the lcdui.* Java package. As all profiles, these are developed through open JSRs, for more information read them.

A feature rich version of CLDC is CDC or Connected Device Configuration. CDC includes most of Java SE, essentially all which is not UI related. The Foundation Profile, Personal Basis Profile and Personal Profile are profiles utilizing a more complete Java SE framework with a subset of AWT support.

Easiest Way to Upload Apps

Uploading games or applications to your device (phone) via USB cable is most likely disabled. This is really annoying since the other option is to download the apps and spend money on data transfer rates. There are ways to make your phone accept apps via USB cable, but these adventures would end up voiding warranty, etc. I found today a convenient and easy method of uploading apps – Bluetooth. This method assumes your laptop or computer has Bluetooth capabilities, but if it does, then do the following:

  • Turn Bluetooth on, for your device and laptop. Find your Bluetooth passwords, most likely if you don’t know them, their 1234.
  • Set your phone as ‘Discoverable’ by your mobile device. Most likely your phone will on be visible for 60 seconds.
  • From your Bluetooth console on your computer, search for available devices. Once found, right click and ‘Pair’ the device. This creates a persistent connection between the two Bluetooth devices. You will be asked for the associated Bluetooth passwords.
  • Double click the paired device in your Bluetooth console. Now, drag and drop the JAR file for your application to the ‘OBEX Object Push’ icon. Once transferred your phone will ask you to install the application.

J2ME UI Development

As shown in a previous post, J2ME MID programs must extend the MIDlet class, which controls the lifecycle of the application. This lifecycle is simple in nature, with 4 states. The figure below was taken from J2ME Application Development and shows the lifecycle:

03fig01.jpg

MIDP user interfaces are divided into 2 parts, the high-level and low-level API. The low-level API extends the Canvas class and is mainly used to develop games or any UI were you must draw pixels yourself. A good example of the low-level API in action is the WormGame, packaged in the example application of the Sun Wireless Toolkit. The high-level API is used to create standard lists, textboxes and forms. It’s important to remember when developing with MID, you have no control precisely were widgets will appear on the screen — this power is given to individual devices (i.e. the device manufacturer). The form class has numerous children items, which can be attached, these include textfield, stringitem, etc. The figure below was taken from J2ME Application Development and shows structure of MIDP UI Classes.

03fig021.jpg

All devices have a Display object, which manages the screen on the given device — most application will only have 1 instance of a display. Each new panel to be displayed to the user will be an instance of the Displayable class. Once this instance is populated with the desired UI widgets, the panel is made visible to the user, by calling the setCurrent method of the Display object.

Below is an example of a method to display a welcome message to a user:

Code:
private void displayWelcome() {
	try {
		form = new Form(applicationTitle);
		image = Image.createImage(getClass().getResourceAsStream(
				backgroundPath));
		titleImage = new ImageItem("", image, ImageItem.LAYOUT_CENTER, "");
		form.append(new Spacer(10, 10));
		form.append(titleImage);
		form.append(new Spacer(20, 20));
		form.append(welcomeText);
		form.addCommand(exitCmd);
		form.addCommand(openWebURL);
		form.addCommand(openLocalFeed);
		form.setCommandListener(this);
		display.setCurrent(form);
	} catch (Exception e) {
		System.out.println("NewsReader Error: " + e.getMessage());
	}
}


General Comments

I’ve been begun playing around with MIDP 2.0 (CLDC 1.0), making some simple and more complicated programs. I’m using the book J2ME In A Nutshell as a reference when I don’t feel like Googling. The Sun website has JavaDocs for all their standardized JSRs. The Sun Wireless Toolkit comes with example applications you can use to better understand the J2ME world. There are three types of XML parsers available, push (SAX), model (DOM) and Pull (Incremental). XML parsing on resource restricted devices is something that literature says to do with care. On the same note, most of these articles, etc. were written in 2002/03 and since then, most mobile devices have probably quadrupled in power and memory. Right now, I’m using MinML, a SAX1 supported XML parser designed for devices with 512kB of RAM.

J2ME: First App, RSE and DSDP-MTJ

Friday, September 8th, 2006

Continuing on from the day before, I started looking into specifically making an application that would run on my cell phone. At this point and time, the J2ME runtime and the associated tools seem like the best bet to get started. It’s not actually called J2ME, but rather the Sun Java Wireless Toolkit 2.5 beta, in any case some really useful development tools are packaged inside to make the mindless setup a lot easier.

To make a simple test application, start the KToolbar. I had a problem getting this to start, but that was only because KToolbar.bat was pointing to a JVM that didn’t exist anymore (I’m using the 1.5.0 8 JDK). Click ‘New Project’ and enter a project name and class name – for the sake of argument put TestProj and TestApp. Click OK, and the settings for the project are automatically loaded in a window. Check to make sure you are set up for the device you’re using, specifically the MIDP and CLDC setting are correct.

At this point KToolbar has setup the directory structure for your new application, in my case the application is inside of the apps directory of SJWT i.e. C:\WTK25\apps\TestProj. The IDE I’ll use is Eclipse, I also installed the plug-in my team here at IBM developed called RSE (Remote Systems Explorer). It can be downloaded from the Eclipse website within the DSDP Project. RSE allows you to work with remote or local devices. In this case, we can browse the file system and work with the new directory structure KToolbar made for us. (You don’t need to use Eclipse; any editor would be just fine.)

In the /src directory of TestProj create a new file called TestApp.java, the code follows:

Code:
import javax.microedition.lcdui.*;
import javax.microedition.midlet.MIDlet;

public class TestApp extends MIDlet implements CommandListener {
	public void startApp() {
		Display display = Display.getDisplay(this);
		Form mainForm = new Form("TestApp");
		mainForm.append("Your first App");
		Command exitCommand = new Command("Exit", Command.EXIT, 0);
		mainForm.addCommand(exitCommand);
		mainForm.setCommandListener(this);
		display.setCurrent(mainForm);
	}

	public void pauseApp() { }

	public void destroyApp(boolean unconditional) { }

	public void commandAction(Command c, Displayable s) {
		if (c.getCommandType() == Command.EXIT) {
			notifyDestroyed();
		}
	}
}

Click here to download the code.

This is the equivalent to a Hello World program for a cell phone. Ok, so let’s review what we have so far. In the /TestProj directory there is a project.properties file, which holds the standards you are programming with, part of mine is below:

….
JSR75: false
MMAPI: true
SATSA-APDU: false
SATSA-CRYPTO: false
SATSA-JCRMI: false
SATSA-PKI: false
WMA0: false
WMA1.1: true
WMA2.0: false
….

In the /bin directory there are 2 files. The first is the manifest file holding a few project specific properties, i.e.:

MIDlet-1: TestProj, TestProj.png, TestApp
MIDlet-Name: TestProj
MIDlet-Vendor: Unknown
MIDlet-Version: 1.0
MicroEdition-Configuration: CLDC-1.0
MicroEdition-Profile: MIDP-2.0

The second file in the /bin directory is the JAD (Java Application Descriptor) file. As defined on the Motorola Hackers Guide:

This file contains a predefined set of attributes (denoted by names that begin with “MIDlet-”) that allow application management software to identify, retrieve, and install the MIDlets. All attributes appearing in the JAD file are made available to the MIDlets. The user may define application-specific attributes and add them to the JAD file.

With the source file saved, you’re ready to build your application. From within KToolbar click the Build button and then the Run button. The source will be compiled and the toolkit will launch the Sun emulator.

On an intersting note, RSE is part of the Target Managment subproject of the Eclipse DSDP (Device Software Development Platform) project. Another subproject in DSDP is called MTJ or Mobile Tools for the Java Platform. As it turns out, Nokia is heavily involved in DSDP, specifically MTJ. Here is a quote from the press release:

Nokia and the Eclipse Foundation today announced that Nokia has joined the Eclipse Foundation as a Strategic Developer and Board member. Nokia will support the work of the Eclipse open source community by contributing software and developers to a proposed new Eclipse project.

MTJ was supposed to have had it first release by now, but its been postponed till sometime in September. I’m excited to see what their working on, I hope MTJ will support or use RSE in someway. I look forward to a day were a great open source IDE for resource constrained devices will exist. Here is a particularly interesting quote from the MTJ project page:

The purpose is to develop both frameworks that can be extended by tool vendors and tools that can be used by 3rd party mobile java application developer.

J2ME: Thoughts after first look

Friday, September 8th, 2006

For reasons I don’t feel like getting into, I’ve begun a journey to learn as much as possible about developing applications for micro devices. Before I even get started, let me say I have no prior experience with micro device programming, so a lot of what I say (in coming posts), could very well be wrong.

The cell phone I’m going to use as a test-bed is my own, a RAZR V3. Though I’m using a specific phone, a lot of what I say about micro devices will apply (I think) in the general case. The first thing needed, is to figure out the Java System settings on your device. In my case:

CLDC v1.0
MIDP v2.0
Data Space: 1545 K
Heap Size: 800 K

MIDP or Mobile Information Device Profile is a spec written for the development of Java applications on mobile devices, etc. In the grand design, MIDP sits on top of CLDC or Connected Limited Device Configuration, which is another spec outlining a framework for developing Java ME applications on resource constricted devices. Both CDLC and MIDP have been developed by the open source community, and as you can probably tell from my phone, different devices support different versions. The diagram below is from the J2ME Developers Guide for the Motorola V3 Handset. With that in mind, be sure to get your hands on the Developers Guide for your device.

cell.JPG

In my case, the heap size is 800 K, this defines the amount of memory the JVM (Java Virtual Machine) can use to create objects, story temporary variables, etc. Developing on cell phone, PDA etc. is a different monster, you need to be conscious of memory size (both disk space and heap size), processing power, screen capabilities and wireless network characteristics.

After I skimmed through the Developers Guide, read up on J2ME, and browsed a couple Wikipedia pages, I set out to find the software I needed to get started. The first thing I did was install the SDK supplied by Motorola for my phone. I’m not even sure why they call it an SDK, the entire kit came with 2 applications. The first was MIDway, which is basically a tool to upload applications to your cell phone. The second was Motorola Launchpad, which acted as an emulator, to test applications on your computer. Undoubtedly, these programs will prove useful (eventually), but for now it’s not that helpful in developing applications. The obvious next install was J2ME, which can be found from the Sun website.

I ended up downloading a couple example applications (with source) from the Motorola website, and after getting a feel for them, wanted to upload these to my phone. After about 10 minutes of frustration, I realized USB application uploads to a cell phone are restricted to ‘Developer phones’. You can check if your phone allows USB uploads by looking in the Java Settings, and seeing if a ‘Java Link’ option available. After a little bit of research, I came across the Motorola Hackers Bible. A moderator from Mobile-Review.com I guess got fed up with people asking the same questions, and wrote a masterpiece. The guide covers a lot of thing, one of which includes transferring applications to your cell phone via USB cable. Needless to say, I spent the rest of the day reading through this guide…