Team Granular Blog

Introduction

As I mentioned in my previous post on Granular Package Archive, GPA is based on two technologies/programming languages – Java and PHP. Yes, you read it right, as weird as a combination of a statically and dynamically typed languages. The Java-based backend programs are responsible for collecting all the necessary information about each RPM package in the repository, and post all that information in a database. This database acts a common entity for the backend and the frontend alike. That is, the PHP-based frontend reads information from the database, and uses that information to interface with the end-user.

Basic illustration of the working of package archive

The Java backend

There are four Java classes involved:

ReadTextFile
Fetches the entire contents of a text file, and returns it as a String.
DirectoryScanner
Scans a directory and returns the list of files contained  in it.
ExtractInfo
Extracts and returns many RPM properties for a RPM package.
PostToDB
Reads the full path to the repository on the server from the file “repository_location.txt” (using ReadTextFile), scans the repository directory for RPMs in it (using DirectoryScanner), extracts rpminfo  for each RPM in the file list (using ExtractInfo), and dumps the extracted rpminfo about each RPM into the MySQL database.

The command I’ve used to get info about a RPM is:

rpm -qp –queryformat %{PROPERTY}

where PROPERTY can be NAME, SIZE, PACKAGER, URL, VERSION, etc. Note that if you have don’t have “rpm” command installed on your repository server and you don’t have enough permissions to install it in the standard path, you might then want to install “rpm” command in the home directory of your server account. In that case, you’ll need to add the –rcfile=/location/to/rpmrc/file modifier to the above command. In that case, also make sure you have properly setup the PATH and LD_LIBRARY_PATH variables properly for the “rpm” command to work.

The PHP frontend

The frontend is actually what a user sees as a website. It provides an interface to view all the information, from database, available about the packages in the repository. And what better language could it have been to develop this interface than PHP? It’s a beautiful language to code in. To begin with, I used my existing project called MyBlog to take the design and comment/discussion system. Other than that, I wrote most of the code from scratch. The comment system provides for the option of discussion on each package by the visiting users. The developers and packagers can also use this feature to notify users about any changes made between package releases. For the comment form, I integrated the very popular open source WYSIWYG web editor FCKeditor. Overall, this PHP frontend is reasonably fast and serves its purpose well.

Package Archive for your repository

Yes, you can actually use this system for your repository too! I did the coding in a fashion to make it usable by others too. This system can be useful in case you are a part of a RPM-based Linux distro and want your users to give the option of knowing the packages contained in your repository.

The following information should help you deploy the thing for your purpose.

REQUIREMENTS:

• PHP >=4 installed on your server
• JRE >=5 installed on your server
• JDK >=1.5 installed on your server if you want to compile the .java programs on the server itself
• MySQL >= 4 installed on your server
• MySQL Connector/J
• Shell access (ssh) / crontab option on your server.

DOWNLOAD:

• pa_backend.zip – Backend Java source programs and MySQL database structure
  
• pa_frontend.zip – Frontend PHP website

USAGE:

1. Extract the contents of pa_backend.zip. Modify the file “PostToDB.java” to change the MySQL related variables and set them to reflect your MySQL database configuration – serverName, dbName, username, and password.
2. Edit the file “repository_location.txt” to put the complete server path to your own repository.
3. Compile the backend by issuing following commands:
   

   $ cd pa_backend/
   $ javac rpm/ReadTextFile.java
   $ javac rpm/DirectoryScanner.java
   $ javac rpm/ExtractInfo.java
   $ javac rpm/PostToDB.java

4. Put the whole “pa_backend” folder into a non-readable location on your server. Prefer not to include any .java file while uploading, just .class files and the lone .txt file.
5. Based on the database information you put in PostToDB.java file, create the corresponding MySQL database.
6. Next, import the file “rpmdb.sql” (present along with other extracted stuff from the pa_backend.zip archive) into the just created database. This step is required to create the necessary database tables. Using a tool like phpMyAdmin will ease this task for you.
7. The backend setup is over now. Now, download the archive pa_frontend.zip and extract it.
8. Edit the file “mb_config.php” to modify the MySQL variables to reflect your database settings – $DB_NAME, $DB_USER, $DB_PASSWORD, and $DB_HOST.
9. All the other .php files are branded according to “Granular”. So, you might want to edit these .php files to take care of branding.
10. That’s all. Now upload all the files & folders in the extracted folder “pa_frontend” to the appropriate directory in your webserver location. Say, if you upload these into a directory “packages” in the root of your webserver path, you’ll then access the frontend website using the url http://www.yoursite.com/packages/.
11. Now, the final step is to instruct your backend to extract information about each package in your repository and post it to the database. You do this by either logging into your server using ssh or setting up a cron job for the same. Either way, you’ll need to execute the following commands:
    

    $ cd /location/of/backend
    (say, “cd ~/pa_backend/”)
    $ java rpm/PostToDB section-name
    (say, “java rpm/PostToDB main”)

    where section-name is the name of a section within your repository. Repositories are generally divided into sections like RPMS.main, RPMS.kde, RPMS.non-free, etc. You’ll need to execute the second command above for each section in your repository.

12. The next time you execute these commands, perhaps to update the database with new packages, don’t forget to first truncate the table “packages” from your database. You can either do that by using phpMyAdmin, or manually by issuing commands:
    

    $ mysql -h hostname -uusername -ppassword
    mysql> use dbname;
    mysql> truncate packages;
    mysql> exit

TODO:

• Make the package scanning & info posting process fast. It is too slow right now – only 3 packages processed per second.
• Implement filter so that only .rpm files get scanned. This would be helpful in situations when a directory contains files other than RPMs as well.
• Make the scanning such that only those RPMs are processed that were added/deleted after last scan was performed.
• Make the frontend fully W3C standards compliant.
• Make the backend read the MySQL server & database information from a text file rather than hard-coding this information in the Java program; like it is currently being done in case of repository location using the “repository_location.txt” text file.

Think you can help me improve this package archive system? If yes, please let me know by leaving a comment here.