Sun x4170

Kerry Osborne’s Oracle Blog

Fun with Oracle Exadata V2

April 23, 2010, 10:43 pm

Well I’ve been holed up playing with an Exadata V2 machine for the past several weeks. Wow. Very interesting technology.
I must say that I believe the concept of offloading SQL processing to the storage layer is a game changer and I wouldn’t be at all surprised to see this as a standard feature a few years from now. What that means for other storage vendors is unclear at this point. So for this first post on the topic let me just describe the configuration (and some potential upgrades).
The basic architecture consists of a set of database severs and a set of storage servers.
Database Servers:

• Sun x4170 (1RU 64x server)
• 2 – Quad-core Intel Xeon E5540 2.53GHz processors
• 72G Ram (18x4G Dimms – max of 144G using 8G DIMMs)
• Dual-Port QDR InfiniBand Host Channel Adapter
• HBA with 512MB Battery Backed Write Cache (only for internal disks???)
• 4 – 146G internal drives (SAS 10,000 RPM)
• dual hot swappable power supplies
• no spare/empty slots!

Here’s what the Database Servers look like:

Storage Servers:

• Sun x4275 (2RU 64x server)
• 2 – Quad-core Intel Xeon E5540 (2.53GHz) processors
• 24G Ram
• Dual-Port QDR InfiniBand Host Channel Adapter
• HBA with 512MB Battery Backed Write Cache (only for internal disks???)
• dual hot swappable power supplies
• 4 – 96G Sun Flash PCIe Cards (total of 384 GB)
• 12 – 600 GB 15,000 RPM SAS or 2 TB 7,200 RPM SATA

Here’s what the Storage Servers look like:

In addition to these two components there is are 2 Infiniband switches (redundant) to tie the whole thing together. There is also a KVM and a couple of Cisco 4948 switches. And the whole thing comes in a pretty Rack with a fancy Exadata Logo. Here’s a couple of real pictures taken by a real crew member:

Oracle is big on the balanced configuration. That is to say, they have spent a lot of time trying to eliminate bottlenecks in any one component. That said, it seems to me that at some point they will start to allow more customizable configurations. For example, 72G on the database servers is not a huge amount of memory these days, especially for a consolidation platform that will house several instances. So here are some thoughts on potential ala-cart upgrades that may be possible at some point.

1. The database servers are capable of using any Intel 5500 series chips. They are shipped with 2.53GHz 5540′s which are middle of the road for the series. The 5540′s could potentially be replaced with 5570′s which are clocked at 2.93 GHz.
2. The database servers come with 18 4G Dimms for a total of 72G of RAM. 8G Dimms are available which would increase the RAM to 144G.
3. The database servers (Sun 4170) have no spare slots, making it difficult to attach external devices such as tape backup units and other fiber based storage. Swapping the 1U 4170′s for the 2U 4275′s would provide additional slots allowing more connection options.
4. Adding additional storage servers is possible regardless of which option is chosen (quarter rack, half rack, etc…)  Oracle has already agreed to this as long as they are not mounted in the Exadata rack enclosure.
5. External database servers can be configured to access the storage servers. This would require Infiniband cards on the external servers and of course you’d need to be running 11gR2 to take advantage of the storage servers “special sauce”.