Saturday Feb 28, 2009

Information and Database Technology workshop at VJTI, Mumbai

VJTI, Mumbai organized a 5 day conference on Database research in January 2009. I presented MySQL there on the second day of the conference, 4th Jan 2009. The conference saw faculty members representing various universities/college from in and out of Maharashtra state along with some of the VJTI MTech students.

The main idea behind the conference was to initiate more research by the universities and also to connect the faculty members with the corporates.

MySQL and IBM's Tivoli were the only two products that were presented in the 5 day conference and the rest topics were all around research and development in database technologies such as distributed databases, GIS databases and such.

Working on the first sunday of the new year isnt a very good thing generally, but I think it was well worth it.

Saturday Jun 14, 2008

SMF import for IPS packages in opensolaris

Installing postgresql/mysql packages from IPS repository on OpenSolaris 2008.05 is very simple n easy and Jignesh has described the command line steps in his post in detail. I tried the same thing on a fresh install of OpenSolaris, but using the Package Manager GUI and It is as easy as it should get with GUI tools.

The idea behind this blog, is to simply clarify to users/developers that as soon as you install any PostgreSQL/MySQL version, the SMF registry is not going to be updated. This might look obvious to many of you, but I see on some reviews that people are expecting that the SMF service will be installed in the registry.

A subsequent reboot will automatically import necessary entries in the registry or it can be manually added before reboot if desired, as below,

# svccfg import /var/svc/manifest/application/database/postgresql.xml
(for Postgresql version 8.2)

This needs to be done only once and the registry is updated permanently.

Also, Note that registry entries are not cleaned up once you uninstall these packages. You can manually delete them as given below. There is no harm in keeping them in the registry, but they are not going to be useful for obvious reasons.

# svccfg delete svc:/application/database/postgresql

Saturday May 24, 2008

Day 2 @ PGCon 2008

The 2nd day of the PGCon conference also had a good list of hacker/community sessions lined up. The talks started at 10 am instead of the usual 9 am time, may be because, people getting late due to the EnterpriseDB party previous night was anticipated.

I was initially planning on attending PostGIS, but then changed my mind to attend "Deploying PostgreSQL in a Windows Enterprise" instead. Magnus covered how Postgres can be deployed with Active Directory Server connected network. I have been working on Solaris for a long time, so one of my intentions was to understand how different it is to deploy Postgres on windows platform and  possibly get an idea of how widely it is being used. I think my understanding is improved a bit after the talk and have realized that the problems are of similar nature in windoze too.

Later, I walked in "Security Enhanced PostgreSQL" session and it turned out to be a very cool project which I was not really aware of. This project provides fine grained mandatory access control even for the privileged user (such as 'postgres'). It depends heavily on the security context associated with certain tuple/cell. For PostgreSQL to be able to achieve such fine grained control, security enhanced operating system is necessary. KaiGai, the SE-PostgreSQL developer, demonstrated this on SELinux and it was really very exciting. I am now wondering if this can be implemented easily on Solaris 10 Trusted Extensions (Tx). He mentioned that for other operating systems to run SE-postgresql, there is a framework available and only certain modules need to be ported.

Post lunch, I attended "PL/Proxy, pgBouncer, pgbalancer" by Asko from Skype. I had a bit of idea about these tools and have a learnt a lot more about them from this talk. I think Skype contributing a bunch of tools is really appreciated. For a postgresql dbms user company to be able to solve their practical problems by creating tools that will help many many other users is really very creditable. pgProxy seems to be a really cool idea and the combination with pgBouncer seems to be really useful for customizing specific deployments which are exposed to heavy load. Kudos to the Skype team working in the PostgreSQL community!

For the next talk, "Performance of PostgreSQL", the speaker, I think at the last moment, decided to get a translator friend for his talk. I was disappointed by the  content of the talk. Lots of data was presented but it did not take long to realize that the tests were not performed taking all the aspects into consideration. That sort of took all the flesh out of the data presented. The translator did a good job though.

Zdenek then presented his findings and approaches on 'In-place upgrade' project. He demonstrated a working version of live upgrade from 8.1 to 8.2. He then presented various approaches to implement the same for other latest versions. Version 8.3 and later has some changes to the on-disk format which is complicating such conversion. The talk was  received very well. It was quite evident that the hacker community is really serious about this project and hence it will be interesting to see how the project shapes up in the near future.

Dan did a great job (again!) with his closing session. I think he was phenomenal in making this conference a huge success.

Most of us got together at the Royal Oak pub for the last time this year. I have never been to pubs for so many consecutive days in my life, but It was a lot of fun! Many of us are meeting up for a sightseeing tour tomorrow and then I will leave Ottawa on sunday to go to Niagara Falls and then back to India.

Friday May 23, 2008

Day 1 @ PGCon 2008

On the first day of the main conference here at PGcon, I was awake early with continued excitment of having won the Nintendo Wii and a slight hangover. I got ready early and after finishing some mails headed for breakfast. I was feeling good not only about the Wii but also because of the fun and the geeky mood here at the conference. The last 2 days had been fantastic with the tutorials and the start of the main conference today with a very interesting line up of talks was very exciting.

After the breakfast we arrived at the venue of the conference where registrations were still going on. The opening talk (keynote) by Bruce was good. The main focus for the next years for postgresql is clear. The project will work towards making effective use of multicore/multithreaded systems which are becoming a commodity now. It made total sense to me.

The first session I attended was "postgresql: from a Java Enterprise point of view". Given my long association with Java as a developer I thought this will be interesting to see how fellow java developers think of postgresql. The talk covered the J2EE aspects of entity beans, EJB/QL fairly in detail with changes in advanced versions. I was also hoping to hear how postgresql is perceived in J2EE application developer community, but that was not covered. It was more of PG user - Java developer talk than for the hackers.

I attended Logic and databases session with no expectations but simply thinking what would be the content of this one. I thought it was a very interesting talk for the users/hackers going back to the basics of relating logic to the query language and how a better understanding of this could improve the queries that one writes. Well, I learnt about some good pitfalls and that helped improve my understanding. Thanks Jeff.

then I attended Susanne's talk on "What PG could learn from MySQL". She put together the content based on her experience and with some help from her colleagues both in PG and MySQL commuity. Her analysis was based on various aspects like development model, packaging, User interface, user/customer focus and I agreed with most of the data that she presented, but not necessarily all the conclusions. One thing I liked about the pg hackers during the talk was that they were very receptive about what she had to present and I am sure that will certainly help postgres project.

"Problems with PostgreSQL with multi-core systems with Multi tera-byte data" by Jignesh Shah. To me, this was the best talk so far at the conference. He presented his study with a lot of data on the performance of postgres on multi-core high end systems. The graphs were complicated but he made some very good points about implementaion 0f postgres and also presented his thoughts on how this can be improved. It was a very detailed talk and I must say very well presented to the hacker community. I really felt excited about the performance of postgres if we resolve some of the problems with it today and I saw similar expression on everybody else' face after the talk. The points Jignesh made will surely help postgres to achieve their goal for the next few years, I believe. Boy! I love dtrace!!

I wanted to attend to see how good the new search system on postgres website is going to be and it looks really promising. I am sure it will be very helpful for all the hackers and more useful for the novice ones like me.

I was very excited about the last talk "Multi threaded query accelarator". The idea of multi threading at the application level giving high overall performance with pg backend was certainly exciting. Not only that, they have also deployed such solution in production made me walk into that room instead of going to the lightining talks which I are always more fun. The talk presented detailed design of their systems and some analysis on how multi-threaded application could improve the overall performance. I personally thought, their database design was not too complicated which allowed them to get performance even if they added some complexity at the application level. Also, I felt, multi threading at application level is going to give performance boost of limited range for OLTP and data-warehousing applications. This is in a way obvious and got confirmed from the presentation to me.

I could attend some of the lighting talks and happy that I didnt miss out on DBIx::cache and dtrace ofcourse!

EnterpriseDB hosted dinner & drinks at a local pub and I must thank them for the nice food/desert and drinks they arranged and yes, a free Tshirt. :-)

Socialising is always good !

Friday Mar 28, 2008

PostgreSQL, MySQL and Java DB at Sun Tech Days, Hyderabad 2008

Sun Tech days Hyderabad kept its promise of bringing thousands of techies to the conference even this year. Total 7000 people attended this conference that ran over three days in the last week of february. Like last year, student community dominated the number of registrations, but the number of developers from the Indian IT industry was still quite a significant percent of the total number. The conference had total 4 tracks running in parallel on the first 2 days and the last day, also called as the Open Source day, had in fact 5.

I presented 2 sessions this year. "PostgreSQL and MySQL, the databases inside Solaris" in the Solaris track on the first day and "MySQL and Java DB, the databases for Java developers" in the community track on the second day.

TalkAbout 300 people attended my session in the Solaris track and it was received very well. The recent announcement of MySQL aquisition helped here, I believe. I discussed features of both PostgreSQL and MySQL and also compared both of them specifically where they overlap. Also, I discussed how Solaris currently integrates these 2 databases and future plans with WebStack. I also presented the PostgreSQL + Zones and postgreSQL DTrace demo, which... worked.

Not so surprisingly, people seem to have been using these 2 databases already. This was evident from the questions I faced at the end. Issues such as Indic language support for PostgreSQL also came up. Unfortunately, This particular point need more attention in the PostgreSQL community and I think, if anybody has to do this, it has to be the Indian language experts. I also think solving this issue is critical for the success of OSDB with Govt of India adopting open source technologies in a big way. People also asked doubts on the comparison between these 2 databases, I was able to answer many of these. I received a comment on the talk, "I was always confused with these 2 databases and this talk has cleared a lot of that.", which is very satisfying.

The other talk was with the series of Java related talks and had much larger audience. Total, 1192 people attended this session and it is the biggest audience I have ever had. The repeat audience from the previous talk was minimal, about 1-2 %. Here, I introduced Java DB and also discussed/compared its feature set with MySQL. The later part of the presentation was focussed more on developing Java database applications. I presented the DerbyTaxdemo and also wanted to show Roumen's "Generating RESTful clients with MySQL using Netbeans" demo, but my Compiz-on-KDE crashed Netbeans and I could not show the demo.

OSDB BoothApart from the 2 sessions, we had a OSDB booth that was manned by Wei-chen and Saurabh Vyas. I spent most of my time at the booth. The booth was located right next to the conference hall and it was crowded all the time.

This year, the event was special for a lot of reasons. It is the 10th anniversary of tech days. We had Software EVP, Rich Green, MySQL Co-Founder David Axmark and Ian Murdock doing the keynotes. We also had a famous indian music band, Euphoria, performing and a fashion show for the audience. This was the first time I presented at the tech days and it was very special because it was also the first time MySQL was covered at Sun Tech Days and I feel honored for being associated with that.

Wednesday Feb 06, 2008

Perl DBI & DBD::pg inside the latest SXDE 01/08

The new released SXDE (01/08) bundles Perl DBI and Perl DBD::Pg along with many other web-stack components. This piece of integration I volunteered to do with a selfish personal goal of contributing more to the OpenSolaris codebase. It has been a great experience of following the entire Solaris integration process for the first time. With SXDE 01/08, the 2 components are out there for people to use.

DBI is a generic database Interface for Perl applications and DBD::Pg is the PostgreSQL specific database driver  which implements the DBI interface. Both these perl modules are integrated into latest Solaris. These components are ready to use once the OS is installed and no other configuration is required.

A simple usecase is as below,

First, Make sure that PG8.2 Service is running, else you need to enable the postgresql:version_82 service to start the PG postmaster.

bash-3.2$ svcs -a | grep postgres
disabled        6:03:01 svc:/application/database/postgresql:version_81
online          6:03:16 svc:/application/database/postgresql:version_82

Create a small database that we will try to access through a perl application using DBD::Pg.

bash-3.2$ /usr/postgres/8.2/bin/createdb testdb
bash-3.2$ /usr/postgres/8.2/bin/psql testdb
Welcome to psql 8.2.5, the PostgreSQL interactive terminal.

Type:  \\copyright for distribution terms
       \\h for help with SQL commands
       \\? for help with psql commands
       \\g or terminate with semicolon to execute query
       \\q to quit

testdb=# create table t (id int);
testdb=# insert into t values (1);
testdb=# insert into t values (2);
testdb=# select \* from t;
(2 rows)

testdb=# \\q

Write the perl app that will talk to this database. Here, we simply do a select on the table created above.

bash-3.2$ cat
use DBI;

$dbh = DBI->connect ( "dbi:Pg:dbname=testdb","","" ) ;
if ( !defined $dbh ) {
        die "Cannot connect to database!\\n";

print "Success connecting!\\n";

$sth = $dbh->prepare ( " SELECT from t" ) ;
if ( !defined $sth ) {
        die "Could not prepare statement\\n";
while ( @row = $sth->fetchrow() ) {
        print @row,"\\n";


Now, run the perl module to see the result. 

bash-3.2$ which perl
bash-3.2$ perl
Success connecting!

With DBI already in Solaris now, other drivers will also follow soon.

So, Go ahead and use them and Do not forget to file bugs if any, here.

Thursday Jan 31, 2008

First contribution to Apache derby from Code For Freedom contest

Code For Freedom Contest, as part of the university program at Sun has been running for a few months now. Apache Derby is one of the 5 Open source technologies participating in this contest. We recently had our first contribution to the Apache Derby project through this contest. Jazarine Jamal, a student from Amrita University has contributed Derby-2239.

Apache derby has not been very active project in this contest until very recently when a bunch of students from this University started participating. I have been working with few of the students as their mentor to take them through this fun ride. Last few weeks has been a great mentoring experience for me chatting with them to get their way through building, reproducing, debugging Apache derby code in Netbeans.  The students have shown great amount of interest and sincerity in their activities and I feel happy that we have crossed the initial hurdles now and have established some expertise in the college which other students can leverage. I really appreciate the traction generated among the students by their faculty member Mr. Vipin as much as the enthusiasm shown by the students.

There are some more contributions in the pipeline and  I am looking forward to many more from this college. It is a great success of the contest that it has built a good platform for many more contributions to Apache derby and it will hopefully continue to grow even beyond the scope of this contest.

Monday Nov 26, 2007

OSDB at Sun Tech Days, 2007, Shanghai and Beijing

OpenSource Database technologies such as Java DB and PostgreSQL got a lot of attention at Sun tech days in China. This blog is enough evidence of the busy times we had at the database booth at Shanghai tech days. The story was the same at Beijing with even more people visiting our booth.

China is very impressive! They have a strong and matured developer community. We do not get to see them, work with them in the communities much, may be because of the language barrier there is, but I was actually amazed by the kind of knowledge and expertise they have. It will be really nice to see more of them in the communities.

During the course of both the tech days, I interacted with a lot of people, mostly developers, marketing folks and students. I was doing a dual role at Shanghai evangelizing Java DB and PostgreSQL, as we had only one booth for both the technologies. At Beijing, we managed to have separate booths and I was mainly doing Java DB. Manyi's talk at Shanghai tech days was a great success for the kind of
response we got and the amount of interest that was shown in Java DB by the audience.

It was very overwhelming to see that the developers were keen to understand how they can use Java DB in the multi tier system. This was a very frequent question that we faced including few very focused discussions around how Java DB could be used in the middle tier for their specific design. I genuinely felt a need for more focused talk on 'Java DB in multi-tier architecture'.

Another frequent question that is worth a mention here is weather it is possible to use Java DB in consumer and mobile devices. The fact that there is more demand in china/japan for sophisticated gadgets, this was not so surprising. It was really useful to talk to many developers and understand their applications. In a lot of their applications like test automation systems, in-house group blogs and other web based utilities, I could help them understand how Java DB could be useful. Given the amount of good work we are doing at the Apache derby community, it was satisfying to see that people will benefit from its rich feature set.

The trip was a great experience and having Manyi Lu and Wei-chen Lu (who can speak chinese fluently) was a great help for me to be able to interact with people only speaking  Chinese.

Manyi also shared her tech days experience, that is available here.

Friday May 25, 2007

JavaDB on JavaME platform

As the consumer devices get more sophisticated, more complicated applications will run on devices and there will be greater need to manage the application data. The need for running database engine on small devices will thus be very critical.

JavaDB, a cost effective, easy to use, fully featured, pure Java database with a very small footprint (2MB), looks very promising  on small devices. I tried running it on the latest phoneME advanced (JavaME CDC) platform and could not resist the excitement of putting the details together in this blog. The procedure is very straight forward.

Prepare your setup as below, 

1. Get the latest phoneME advanced platform. Currently, it is supported only on Linux-x86 platform.
2. Get the latest JavaDB release.
3. You also need the JDBC Optional package library for CDC platform. It can be built from the source using the step by step instructions.

Now, try following on your linux box,

<CDC_HOME>/bin/cvm -Xbootclasspath/a:<JDBC_OP>/foundation_jdbc.jar 
-cp <JAVADB_HOME>/lib/derby.jar:<JAVADB_HOME>/lib/derbytools.jar

Note here that the JDBC Op package can also be built along with CDC binaries, as mentioned here. then, Xbootclasspath need not be specifed while running. By default, the CDC build does not include JDBC Op package, hence we include it in our example.

Please note that, JSR 169, The JDBC Optional package does not support java.sql.Driver. So, the 'ij' cannot be used in its typical form to connect to database via a JDBC driver. The connection in this case is established using the JDBC DataSource.

This will bring up the íj prompt. 'ij' provides Command Line Interface to interact with a database using SQL. You may create such connection directly from your ME application using DataSource objects. You can try some simple steps like below to further test this setup.

ij> create table t (id int not null, addr varchar(20));
0 rows inserted/updated/deleted
ij> insert into t values (1, 'PlanetSun');
1 row inserted/updated/deleted
ij> select \* from t;
ID         |ADDR                
1          |PlanetSun           
1 row selected
ij> exit;

Any database that may have been created by desktop Java application can be read/modified on JavaME platform and the same database can be used by the desktop application again. This can be accomplished so easily only with Java-JavaDB combination!

Wednesday May 23, 2007

Building JDBC for CDC/Foundation

JDBC Optional Package for CDC/Foundation Profile
was released back in 2004. This optional package is required on the CDC platform for any JDBC driver to work. As part of the Reference Implementation download, only source (and documentation) bundle was released. The intent here was that people who have ported CDC platform to their own platform of interest should be able to build JDBC Op Package from this source against the built binary.

The building instructions in the README file of the source bundle are some what vague and there was no clarity on this as CDC platform itself was not Generally Available. Recently, As part of the Open Source effort, mobile and embedded community was formed and CDC platform binaries (for Linux-x86 platform) are now easily available for download, as phoneME Advanced project. With this change, I believe it is more critical to have clarity around building this package.


The steps for Building JDBC Optional Package on phoneME Advanced platform are as below,

1. Download the JDBC Op Package source bundle from here.

2. Download the latest CDC Platform. Please note that currently, this platform is available only on Linux-x86. This means that you will need linux-x86 box to build JDBC Op package as per these instructions.

3. Prepare with JDK 1.4.2 binary. CDC Platform is compliant with JDK 1.4.x spec and so is the JDBC Op Package. You will need JDK 1.4.x 'javac' to build against the CDC.

4. Run following to build the JDBC Op Package.

4.1 cd <JDBC_Source>/cdcfoundation/src/share/jdbc/classes

4.2 <JDK1.4.x>/bin/javac -bootclasspath <CDC_HOME>/lib/<CDC_HOME>/lib/basis.jar -d <pathtoclasses> java/sql/\*.java

4.3 <JDK1.4.x>/bin/javac -bootclasspath <CDC_HOME>/lib/<CDC_HOME>/lib/basis.jar -d javax/sql/\*.java

Here note that, CDC platform contains 'basis.jar' file, as this set of binaries support Personal Basis profile. if you have built your own CDC from the source bundle with profile 'foundation' or 'Personal Profile', you should replace 'basis.jar' file as 'foundation.jar' or 'personal.jar' respectively.

4.4 cd <pathtoclasses>

4.5 <JDK1.4.x>/bin/jar cvf <JDBCOp>.jar java/sql/\*.class javax/sql/\*.class

I am planning to write a simple Makefile for linux-x86 platform which can be used with the JDBC Op package makefiles. Building the entire package will be just one 'make' command then. But, until I get there, this will be helpful, I believe.


added on May 25, 2007.

If you prefer building JDBC Op package along with CDC binary from its svn repository, then you need to add few lines to your makefiles as pointed out by Chris, here.

Monday Apr 09, 2007

Whats Sun doing in the database world?

This cool podcast talks about how Sun is contributing to the Database world.

While this could well be a surprise to people who associate Sun with Java, Solaris and other in-house innovations, I think it is a clear indication that the company is not only committed to open-sourcing its own products but is also comitted to open-source projects in other technology areas.

Tuesday Feb 27, 2007

JavaDB at Sun Tech Days 2007, Hyderabad

The Sun Tech Days 2007, Hyderabad saw around 10000 people this year and was probably the biggest event in its history so far. The event featured many Sun technologies over the span of 3 days. The first 2 days followed a typical Tech Days course that included talks, hands-on sessions, the pods and the last day was dedicated to Netbeans, OpenSolaris and JavaME technologies running in parallel.


Manyi Lu, presented introduction to JavaDB in the Java track on the first day. The presentation was attended by around 800 people and was received very well.

Anurag and me, along with Manyi, manned the JavaDB pod that was setup for the first 2 days. Anurag had prepared a demo showing a  catalog application using JavaDB. The classic "Derby Tax Demo" was also on display.


Total turnout at the exhibition was really huge and it was difficult for 3 of us to engage with everyone during the breaks. Students from local universities were more than the corporate representatives at the event. It was really overwhleming to see the interest shown by the corporates towards JavaDB. At the stall, we interacted with people representing government organizations (including Indian Space Research Organization), IT & ITES companies and various Training institutes apart from student community.

 JavaDB pod at Sun Tech Days 2007 

 The most frequent question we had to answer was, "How different this is from already available DBs in the market and especially Oracle?". We spent a lot of time telling people about how JavaDB is targeted for different set of DB applications (what we call them as departmental apps), explaining them about the key features of JavaDB and I believe people were satisfied as a lot of them seemed to be really keen on using JavaDB. Many DB developers were impressed to see a fully featured db engine packaged in just 2MB binary size.

Following is the list common questions asked,

      • Are there any migration tools from MySQL and Oracle to JavaDB?
      • Can we access JavaDB from PHP?
      • Are there free online courses on JavaDB?
      • Are there clustering capabilities in JavaDB?
      • Is there any in-built GUI ??

We also got an opportunity to interact with corporates who brought up their specific use-cases/apps where they can consider JavaDB. We had discussions on specific features of other DBs which they find very useful and if JavaDB supports them. I am happy to say that all such features that we talked about are either already integrated or are on our roadmap (migration tools, grant-revoke, blob/clob support). Some students and corporates were keen on being a part of the community and contributing to it.


I think the event was a great success from JavaDB perspective in terms of increasing awareness among the huge developer community in India.

Madhu posted pictures of the entire event which is definitely worth a look.

Thursday Dec 21, 2006

HADB and SOA database bottlenecks

This topic on FTPOnline talks
about current challenges in SOA development in the context of Databases. The article has described the need for Highly Available Distributed database really very well. The cluster solutions today are very sophisticated, but they are not
targeted towards databases. This disconnect brings up a lot of scalability and performance issues which could be solved with equally sophisticated database implementations which are targetted for such
clustered techonologies today.

Sun has been working on this integration for quite some time now and there exists a solution which could be very useful for the current SOA database bottlenecks. The solution is HADB, the old clustra (for the folks who know the clustra DB). HADB maintains independent copies of the fragmented data on server nodes and all this is completely transparent to the Application/user. Also, HADB provides 99.999 % (the 5 9's) availbility with a clustering distributed database solution. Due to the clustering nature, there are some hardware and network constraints to this solution, but then it is targetted only for 5 9's availability and hence, it is not for small-scale business apps.

Currently Sun Java Application Server provides HADB for session failover. Mani talks about how to use HADB in his blog. The famous Honeycomb project also uses HADB internally to satisfy their availability needs.

Thursday Jun 29, 2006

JavaDB/Postgres Support now available from SUN

Sun has recently announced Support for Open Source Databases, Postgres and JavaDB with announcement of Solaris10(Update2). Service Plans are now available for the customers with Solaris 10 OS 6/06 (S10 U2). While Postgres support is available only on Solaris, please note that JavaDB support is available on Solaris, Linux, Windows & HP-UX platforms.
Customers need to contact their local Sun representatives for pricing information.

The official page on the sun site is here.

Monday Apr 03, 2006

JavaDB is now available!

Good news for Apache Derby - Sun has gotten behind Derby and is also going to provide support for it.
Sun's Java DB is now officially available for download at here.

Check out Java DB's home

Also, the article talking about how to use Java DB in the desktop got top billing at Sun.




« July 2016

No bookmarks in folder