Here are the current specs for my test servers:

Main storage server
Supermicro chassis with
2 x Intel Xeon X5650
12GB RAM
LSI 9265-8i RAID Controller
20 x 1TB SAS drives splitted in 2 RAID5 volumes of 9 disks each with 2 global hot spares
4  x 120 Intel 320 SSD with 2 of the drives are currently assigned to LSI CacheCade (SSD read cache)
Windows Server 8 Developer Preview with the following:
Hyper-V Role
File Services Role with Data Deduplication

Replication server
HP BL460C G6
1 X
18GB RAM
1 x 40GB volume on EMC CX3-40 SATA drives as boot volume
1 x 200GB volume on EMC CX3-40 SATA drives as data volume
Windows Server 8 Developer Preview with the following:
Hyper-V Role
File Services Role with Data Deduplication

Storage VM
1 VCPU
2GB RAM
1 IDE controller with the OS drive connected to it (VHDX)
1 SCSI controller with a data drive connected to it (VHDX)
Windows Server 8 Developer Preview with the following:
File Services Role with Data Deduplication and ISCSI Target

Here’s an idea of what the stack looks like:

1. Storage is hardware agnostic
2. Hyper-V Replica allows me to replicate the ISCSI volumes to another host for failover capabilities without clustering the physical hosts with a configurable number of snapshots
3. Storage can be expanded in multiple ways at different levels:
   1. Add more drives in the physical host
   2. Add storage enclosure to the physical host
   3. Mount volume from another host/storage array and assign to the storage VM (could be iSCSI, FC, SMB)
4. Can use standard tools to backup the storage VM or the physical host
5. I get IO caching through the LSI controller cache and through the CacheCade feature which uses SSDs for the read cache.

One thing I would like to see changed in upcoming releases of Windows 8 are the following:

• Currently, you cannot failover a Hyper-V Replica VM without downtime as the VM needs to be turned off before the failover.
• If I add a new VHDX to the VM, I need to drop the replication and reconfigure it. For large VHDX, that is a bit of a turn off!

Problems encountered:

When I was pushing a lot of write IO to my VM, replication became out of sync. Resynchronizing was an easy task but still. I’ve not enabled automatic synchronization on the VM and after a test I didn’t encounter the issue I experimented initially.

You may wonder what kind of performance I’m getting out of all this. The performance is actually pretty good. I’ll publish some stats in an upcoming post.

One thing I may experiment with in the future is the use of Windows Server 8 Spaces to potentially remove the need for the RAID controller. Unfortunately by doing this, I would lose the controller caching capabilities (SSD and controller cache).

I’ve done some testing with ESXi 5 but I haven’t found yet a good combination to expose the storage. I’m still investigating NexentaStor and other alternatives where I could do the replication from within the appliance. One thing that bugs me is that there’s no way to manage the RAID controller from within ESXi. I’m not sure also how the hardware errors are surfaced in ESXi (i.e. disk failures)

More on my tests with storage to come!

Another interesting fact from their research is it turns out that most of the spam business in the world uses about 4 different banks. Since it’s fairly difficult for them to change banks as not all banks will accept those high risk activities, it would be possible to effectively make it very difficult for criminals to process sales and move money around. To get there, it will take some time because of the politics involved as those bank are usually in emerging countries with “flexible laws” .

The morning concluded with interesting presentations about the use of GPUs. The first presentation was regarding more efficient scheduling of commands on the GPU using a custom driver and the second was regarding the virtualization of GPUs to make them available as a processing unit to virtual machines. This solution also involved a custom scheduler to ensure the GPU are used to their maximum capacity and to provide QoS if required.

During the afternoon there was an interesting presentation from a guy from HP Labs regarding a tool that proposes network topologies. Basically you feed the tool with certain properties that you want to have for your network (available bandwidth for instance), number of servers, etc. The tool will then generate a list of possible network topologies and weed out the ones that might not make sense using heuristics. For instance, it can show you the effect on cost of using 24 or 48 ports switches on cost and bandwidth. There are also plans to integrate more parameters in the automated analysis such as power and cooling for instance.

An other interesting presentation in the afternoon had to do with in place log data processing. Instead of moving large amount of log data to a central location for analysis, the software performs a large portion of the data manipulation and aggregation directly on the server where the log data is generated. There are also options to control the tolerance of freshness and completeness of the data while executing the data processing jobs. To ensure minimal impact on the primary services running on those hosts, throttling mechanisms are used with minimal overhead.

The last presentation of the day was regarding a research about the use of low cost accelerometers such as the ones found in smartphones in order to detect seismic activity. The high number of potential sensors can be used by first respondents to better identify the area with the most potential damage. It can also be used an early warning mechanism. Earthquakes typically have a primary wave that precedes the damaging seismic wave. There’s typically a 1 minute delay between those two that would allow basic actions to be taken prior to the damaging wave such as allowing people to get out of elevators, opening fire stations doors to minimize further delays of first responders. Using a large array of low cost sensors such as smart phones would also allow the capture of higher resolution data that would more accurately represent the seismic wave patterns.

The presenter concluded their presentation by showing an example of how tablets and smartphones could be used to perform basic medical diagnosis in emerging countries using low cost equipment.

When you look at the work being done in the area of static code analysis for the purposes of security and stability, I don’t think it would be far fetch to extract the performance attributes of the code through a similar process. Since most code is based on widely available frameworks, those performance attributes wouldn’t have to be re-evaluated for the core functions. The only thing that would need to be adjusted are the execution variances caused by the different hardware.

By extracting those attributes it would be fairly easy to simulate system behavior.  The following would then be easily predictable:

1) What happens when increase function x usage by 10%

2) Determine CPU, I/O and storage usage accurately

3) Determine application placement in the infrastructure

4) Scale the application more accurately

5) Determine run time costs in cloud infrastructure

6) Simulate application behavior changes after hardware/software change

If we push the concept a little bit more, embedding the performance attribute as part of the application binary or manifest would allow the platform to take the appropriate actions.

When the application statistics are collected, it also facilitates performance comparison of functions that achieve the same goal.

When you think about it, it’s really the next step in application performance profiling.

I then went on to ask to the panel if full hardware virtualization is still the way to go. I think what people are really looking for is a way to provide an isolated environment for their applications. Another requirement would be to have applications a little bit more mobile/highly available. Do we need full hardware virtualization to support such a scenario? I don’t think so. I think application virtualization is the way to go here. There are still something that would need to happen in that space before it would fully meet those major requirements. Right now application virtualization only provides an isolated environment in which the application executes. I think the next step would be to add that live migration functionality which would allow the application to be migrated from compute node to compute node while still maintaining their state. I think everyone would agree that moving an application state vs the whole VM state is much more efficient.

One use case scenario that would be interesting using that technology would be more efficient data processing. After listening to a lot of presentations on cloud computing today, it becomes clear that one of the major problem with processing large amount of data is moving this data. It was suggested by one of the person in the audience that we should look into moving processing closer to the data. If you take the application virtualization principle, you can move the application easily on the node where the data resides while maintaining isolation from the primary application that’s running on the node. You could maybe have cases where the application is moving from node to node while maintaining its state to get a final result. Something similar to what happens in graph traversal which would provide highly parallel tasks and high throughput potentially without over taxing your data center.

Food for thoughts!

The second presentation was about moving middleware code in the hypervisor space to avoid jumping through too many layers and therefore losing performance. In the research, a MySQL storage module was used to demonstrate the concept. While the concept presented slight performance improvements, it opens up a whole new can of worm on the security side. I personally think that would essentially be taking shortcuts in the name of performance improvements. I mean, it’s fairly intuitive to figure out that by removing execution layers and bringing the code closer to the metal, that performance would improve.

There was two presentations that  I found particularly interesting was regarding how to achieve effective QoS for storage and network I/O (Gatekeeper). The storage QoS solution involved the usage of multiple queues that are a bit more static in nature that would be linked to specific class of services to guarantee response time and IOPS. The classes are then nested to provide an ordering mechanism. For the network QoS solution, it essentially involves having agents on the hypervisor hosts to coordinate bandwidth allocation for each virtual machine. By choking data rates at the source, you can prevent situation where you can’t control how much data is sent to you. You can then more accurately predict demand and allocate capacity.

The day ended with a panel on the challenges of I/O virtualization. There was an interesting debate that came from a comment I made, which will be the subject of my next post.

The first step into efficiently enabling this functionnality would be to determine on what type of processor the instructions would execute most efficiently. Think of this as a loose guideline for the scheduler. In order to achieve this, the compiler or a code profiling tool could examine the instruction set and rank the preferred order of each processor type for each instructions. For instance, instruction x processor ranking could be GPU, CPU, APU. Once the scheduler has this information in hand, it can then optimize processor usage based on various set of conditions and policies. Here’s a few example of such controls:

1) Processor availability: Is the preferred processor currently free to execute?
2) Processor consumption: Is it efficient power wise to execute this instruction set on the preferred processor?
3) Instruction set priority/deadline: Can this instruction set be executed on a non-optimal processor because it’s not running at a high priority or because there’s no execution deadline?
4) Instruction set parallelization: Can this instruction set leverage the parallelization features of the processors available?
5) Instruction set execution location: Can this instruction set be executed on a remote processor without adverse effects?

In order to achieve this, the compiler would supply the binary of a executable with the specific instruction sets targeting each processor type. Only the processor scheduler can determine at runtime on which type of processor a particular bit of code would execute.

This concept could work in a way that is similar to execution plans in database engine. As more statistics are gathered by the execution of the various pieces of code on the system, the scheduler could take smarter decisions as to how the code should be executed. The ranking of the processor could then be adjusted in the processor ranking manifest of the executable.

I think this could greatly enhance processor usage in all class of computers by offering alternate execution paths that were not dynamically available  before. Right now, to take advantage of the available processor cycle of a specific type, one would have to rewrite the application to target it. The compilers and code profiling tools are good candidates at achieving those tasks as they already provide an abstraction layer from the targeted architecture. For example, the Intel compiler is able to target specific capabilities of the processor without having the developper to necessarily change a line of code. Programming language should stay what they are, a high level way to instruct the computer and all it’s resources to achieve a particular task. I think that targeting in code specific capabilities of an architecture is wandering away from the problem at hand to solve.

http://gamma.cs.unc.edu/DB/main.pdf

 Another use that I think might work is for indexing data. If you represent data using geometrical patterns, it would be thinkable to use a GPU to perform pattern matching in a very efficient manner due the highly parallel nature of those processing units. If you combine those patterns with set theory, you could define patterns that encompass the actual data. By combining geometrical pattern, those indices would be able to determine the data that is to be included in queries involving a wide range of aggregations and computations.

I’d be curious to see with the CLR integration in SQL Server, if one could call DirectX libraries to offload some work to a GPU.

Feel free to comment on the post, I’m learning and thinking out loud here! I’ll probably be posting more thoughts about this in the coming posts. I’m already thinking about applications for data minining and OLAP data…

First of all, the stability of their PC would most likely increase as each applications operates in a sandboxed environment, which can avoid a lot of headaches. No more complicated setup and configuration for end users as well. Just click on the icon and there it is. Installing an application of IT pros seems like no big deal, but for users at home, it’s a risky operation. Just deploy a stable base operating system image and the user is set.

Another upside to this model, is related to SaaS, as users could only pay for their actual usage of the software. For instance, when I’m at home, I’m not using Microsoft Office 100% of the time, I might only need it a few hours per week. I’m having a hard time paying a few hundred dollars for my copy Office for the usage I have for it at home. I’d much rather pay 100$ for a bank of 10 hours of use per month.

 Hopefully software vendor will get their act together as they did with full OS virtualization. As this article points out, we have a bit of way to go: http://www.news.com/Microsoft-Streaming-Office-infringes-license/2100-1012_3-6229776.html

When you have something like application virtualization, a published desktop or application via Citrix or Terminal Services, why whould you want to build web versions of application like what Google is trying to do with Google Docs?