Showing posts with label VMware. Show all posts
Showing posts with label VMware. Show all posts

Thursday, October 31, 2013

My experience with embedding PostgreSQL - #pgconfeu

At my current company, we embed PostgreSQL based technologies in various applications shipped as shrink-wrapped software. In this session we talk about the experience of embedding PostgreSQL where it is not directly exposed to end-user and the issues encountered on how they were resolved.
We will talk about business reasons,technical architecture of deployments, upgrades, security processes on how to work with embedded PostgreSQL databases.





Tuesday, September 17, 2013

PostgreSQL replication? I need temp tables for my reports

One of the frequent things I hear that many times  PostgreSQL users avoid running PostgreSQL replication  because they want to offload reports that use temporary tables on the slaves. Since PostgreSQL replicas are pure read only, it cannot support temporary tables.

There is one way to overcome this with postgres_fdw - PostgreSQL Foreign Data Wrapper which are improved in PostgreSQL 9.3 which is now released.
  • Create a Master-Slave setup with PostgreSQL 9.3 (with synchronous or asynchronous replication as per your needs).
  • On the slave setup, setup another PostgreSQL 9.3 instance (with different port) with postgres_fdw and map all tables from slaves as foreign tables with same names as their remote counterparts.
  • Run reports which requires temporary tables using this new instance
Of course there are few caveats for this setup
  • Query plans: Currently they are still inefficient but as postgres_fdw improves, this will likely go away. Infact more usage of this use-case scenario will force it to be improved
  • Lot of data moving: Most DW reports do read lot of rows. However by setting it up on the same server most of it are loopback and dont go on the wire outside. 
  • More Resources: This will do require more memory/cpu on the server but it is still cheaper since the management of such a server is still more simpler compared to other complex designs to achieve the same goal
I would like to hear about your experiences on the same too so feel free to send me comments.

Tuesday, May 21, 2013

How to do Postgres Replication and Failover in VMware vFabric Data Director 2.7?


Last week VMware released vFabric Data Director 2.7. Among the many new features for various database, I wish to give a little more insight into my favorite ones which are regarding Postgres.

One of the big feature add from a broad perspective is support of Postgres 9.2 based managed database servers along with replication.  Lets look at how it is done in brief.
First vFabric Postgres 9.2 base DBVM needs to be uploaded into Data Director system resource pool and then converted into template. Note this template is different from vFabric Postgres 9.2 Virtual appliance and is available on the same location where Data Director 2.7 download exists.  Also this DBVM template is based on the latest PostgreSQL 9.2.4 core.
Next once the template is  loaded into the system resource pool it will show up in Base DBVMs section in System->Manage and Monor-> Templates-> BaseDBVMs

Then you would then  right click on the base DBVM and select “Convert to Base DB Template”.  Here you also have a new feature to add more disks (think Tablespaces in PostgreSQL) to the template. In Data Director 2.7, disks are added at the template level.


This process take some time and then it should show up in Base DB Templates. On the Base DB Template section right click on the newly created template and select “Validate”. This one creates a test database to see if it is successful or not before it can be rolled out to the Organizations and Database groups . Once the validation is a success, right click and select “Assign it to the resource bundle” and select the resource bundle that will have access to the template or create a new resource bundle to use the template.
Note in the Resource bundle creation steps lies yet another new  way to separate out IO on separate datastores since these may need different IO characteristics

The above image shows how the various types of datastores for your OS, Backup, Data and Logs (Data can be multiple location if you need multiple tablespaces)

Now all the Orgs/Database groups using the resource bundle will see the new vPostgres 9.2 template.
In a sample demo I created a database called MyDB using few wizard questions.
Now on the DB List we do a right click on the database to create more replicas


Couple of more quick wizard questions and a slave is ready. Similarly you can create more slaves of the same master. Now actually behind the scenes it does all the plumbing right from creating a new clone virtual machine from the master and setting it up using standard PostgreSQL replication on the newly created VM but from a user point of view it it is all point , click and done.

The database dashboard also has a new portlet to show bit more information about the replication.

To make a slave as a new master and it will give an option to  move all the other slaves to the new master select as follows:


After completion of the failover, the status shown will be similar to:

Where the original slave replica is now redirected to the new Master. The original master is kept as it is for Port-Mortem or other activities. This also gives a way to use it as new use cases like separating a slave for test purpose/DR purpose or separating “mirrors” etc. Best of all it uses all Standard PostgreSQL replication.
Of course one of the thing that it does not do is re-purpose the original Master as a slave of the new Master. For more details there is a perfect opportunity to find Heikki Linnakangas from VMware at PGCon in Ottawa this week and ask him the question after his session .. WHY??? :-)

Tuesday, May 14, 2013

How can PostgreSQL 9.3 beta1 help you?

PostgreSQL 9.3 beta1 is now available. Giving early access to software is always a good idea to test out evolutionary, revolutionary, radical ideas because unless it is field tested, it has not gone through its trial by fire to be proven gold.

There are many new changes introduced in PostgreSQL 9.3 beta1 and I do have few favorites in them.

For example Disk page checksums to detect filesystem failures. In fact this would allow VMware to use the now standard disk page checksum instead of a custom feature. This highly debated feature is required to identify silent bit corruptions (or deter malicious ones). I have been told in talks with database administrators (not just PostgreSQL DBAs) that typically in a year they would face one such incident atleast where one of the disk would show such a bit rot which goes unnoticed without any instrumentations to catch it.

Another change that goes in the right direction is how PostgreSQL maps the shared memory. This small change now allows no kernel changes to be done to start the database with a bigger shared buffer pool. This now allows one less cookbook step to be done to get the database working. Considering that in this cloud world where there are 100,000 VMs running databases one less step is a huge increase in productivity since this step actually required privileges higher than the database instance owner.

Yet another favorite feature is the custom background workers. This new mechanism is certainly a popular one in our team at VMware where are using it heavily to move some of the changes that we had done into custom background workers deployed as extensions and allowed us to align with core PostgreSQL and extra features enabled as extensions as needed.

Next I want to talk of three features : Writeable Foreign Tables and pgsql Foreign Data Wrapper and Automatically update VIEWs together. These features on its own itself are very useful and generic. However when used together it actually opens new possibilities using multiple federated PostgreSQL databases shards with single logical view of the whole database as one. Quite Powerful if you think about it. I hope to see people trying these fundamental features into new derived features now made possible.

Also new JSON functions help PostgreSQL on its evolution to be the Data Platform not just for relational data but also document data.


While I have barely scratched the surface of all the new features in PostgreSQL 9.3 beta1, I am already excited with this release and the possibilities I see going forward  in the world of data.

Wednesday, March 13, 2013

Thursday, November 15, 2012

A Step Forward

Recently I upgraded my  "lab" setup.
Now it currently looks as follows:

* 2x  Physical Hosts running vSphere 5.1
* Controlled by vCenter 5.1 Server Appliance backed with embedded Postgres Database instance
* Monitored by vCenter Operations which has two embedded Postgres Database instance in the vApp
*To monitor my VMs, I installed  vFabric Hyperic 5.0 also running embedded Postgres Database VM.
* DBaaS provider vFabric Data Director 2.5 also installed with its embedded Postgres database instance running too

Now for my VMs:
* vFabric Postgres 9.1.6.0 VM   integrated with vSphere HA
* My Linux Developer VM running PostgreSQL 9.2.1 

If you get everything you need with Postgres, why even QA for other databases anymore?
Well vFabric Hyperic took the first bold step forward





Wednesday, October 10, 2012

Wednesday, September 19, 2012

PGOpen 2012: DVDStore Benchmark and PostgreSQL

My slides to go with the Demo I did with DVDStore Benchmark and PostgreSQL session.
http://portal.sliderocket.com/vmware/PGOpen2012_DVDStore


Other session presentations are available at https://wiki.postgresql.org/wiki/Postgres_Open_2012




Friday, July 20, 2012

Postgres @ VMware

Recently VMware vCenter Server Appliance 5.0 u1a was released with embedded Postgres based distribution. Check out the release notes.

Quote from release notes:
 "vCenter Server Appliance Database Support: The DB2 express embedded database provided with the vCenter Server Appliance has been replaced with VMware vPostgres database. This decreases the appliance footprint and reduces the time to deploy vCenter Server further."

vCenter Server Appliance joins the growing list of VMware products embedding and/or supporting Postgres.

Tuesday, December 20, 2011

Using DVDStore with PostgreSQL

We now have support for PostgreSQL in the popular DVDStore Benchmark which stresses database using an emulated DVDStore e-Commerce website. DVDStore Benchmark is maintained by Dave Jaffe (Dell) and  Todd Muirhead (VMware).  It is an open source database test kit. The beauty of the benchmark kit is it allows the same web application being deployed either as
  1. Java/Tomcat  and connect to the database,
  2. Web Server/PHP and connect to the database,
  3. IIS/ASP.NET connect to the database or
  4. Direct connect to the database and invoking the business logic as stored procedures stored on the database itself.

Currently the PostgreSQL implementation details are as follows
  1. Java/Tomcat using PostgreSQL JDBC driver,
  2. Web Server/PHP  using  PHP-postgres modules which uses  libpq
  3. Currently there is noIIS/ASP.NET web app  implementation for PostgreSQL
  4.  Direct connect to PostgreSQL database and business logic implemented in stored procedures however the driver is implemented using .NET C# and requires Npgsql 2.0.11.0

Setup instructions for the database are relatively quite easy.
  1. Download ds21.tar.gz  and also ds21_postgresql.tar.gz from  http://linux.dell.com/dvdstore/
  2. Unzip them on the system running PostgreSQL
  3. The default data size is 10MB. If you want a different size execute 'perl Install_DVDStore.pl' in the ds2 directory. (Expects perl to be available on the system. I used the option 100, MB , PGSQL, LINUX respectively for the options.)
  4. Assuming you are logged on as the DB Owner and the database is on the localhost at port 5432, execute the script pgsql_create_all.sh in the ds2/pgsqlds2 directory. It will create a database "ds2", two users "ds2/ds2" and "web/web", create tables, load tables, create indexes, update sequences and finally run analyze. (The script needs to be modified slighly if the database is already hardened and you want to control the creation of database and the users.)

Setup for the actual load driver is probably easiest on  another Windows platform as follows as it was designed for .NET platform.
  1. Download and install Windows SDK v6.1 and .NET 3.5 framework  on a Windows Client machine.
  2. Once installed start the CMD prompt from Programs-> Windows SDKv6.1-> CMD Prompt.
  3. Verify the above CMD prompt has path setup for gacutil in windows (Try 'gacutil/l')
  4. Download Npgsql 2.0.11 for msnet35 and install the dlls using the gacutil.exe (Note other versions of Npgsql may have issues.)
    •  gacutil/i Npgsql.dll
    •  gacutil/i Mono.security.dll
    •  gacutil/i policy-2.0.Npgsql.dll


With the above setup you can use the ds2webdriver.exe in ds2/drivers or the direct ds2pgsqldriver.exe in ds2/pgsqlds2. More on running the benchmark driver itself  in another post.

Sunday, September 18, 2011

Next Stop: #PgWest 2011 - San Jose

Returned back from a great #pgopen in Chicago. It was nice to also see and meet again senior VMware executives, PostgreSQL community members and lot of folks (aka customers or potential customers ) who use PostgreSQL as key databases in their IT setup  in the conference.

Next stop now is #PgWest 2011 in San Jose starting on Septeber 27,2011. This year again #PgWest is in San Jose, California.

At this conference, I am presenting my first ever "Using vPostgres - A DB User perspective". The gist of this presentation is on how to use the vFabric Postgres client with the vFabric Postgres server deployed by vFabric Data Director. There are some small differences on how the client works compared to community PostgreSQL  and we will go over those in the session on exactly how it works, and see on how you use it for running and developing your own applications with it.

Also Alex Mirgorodskiy, VMware will be first time presenting in a PostgreSQL conference (that I am aware of at this point of time) on "vFabric Postgres Database Internals".  In this session Alex will go over the subtle tansparent changes which makes vFabric Postgres so easy to deploy, manage and perform in the vFabric Data Director world. Not to steal his thunder but this will cover the new Elastic Database Memory in detail.

Bill Hodak, VMware will present  about  vFabric Data Director itself on how it leverages the features of PostgreSQL and provide a management framework over it to provide enterprise framework and provide self-service features making it easy for even folks who donot know a lot about PostgreSQL database itself to deploy, tune, monitor, backup, restore, clone the database instances.

And to give an overview of VMware's commitment to PostgreSQL Community, Charles Fan, Sr.VP - VMware R&D will be presenting the keynote at #PgWest 2011.

Overall pretty excited about the conference and  to meet lot of new people in the bay area.