Security In The Virtual World

Virtual Environment User Based Access Controls

Up until recently the network has largely been controlled by policies defined by IP Addresses, Subnets, Ports and sometimes content but we've all wanted to track and control user activity to no avail!

Traditional firewalls haven't been able to home in on a specific user or group of users due to its nature in controlling IP's.  With DHCP so largely deployed these days as a means to hand out IP's to users, how can one lock down a user based on IP?

NAC solutions have poped up everywhere to try and help lock down user access and many of these NAC solutions have made there way into switches or so called next generation replacement switches.

But what about these new things called Virtual Switches?  Can they control user activity also?  What if you've invested in LAN based solutions from Cisco, Consentry, Nevis and others for your NAC solution but are now thinking about moving to VDI (Virtual Desktop Infrastructure)?  Does that NAC solution do you any good?  Hmmm... sounds like more hardware to throw out!

Many have said that Citrix's move to aquire XenSource was to help them own the Virtual Desktop space and differentiate themselves from VMWare who currently owns the Virtual Server space.  But if companies rush to virtualize their desktops like many have done with servers, they will be in for a bigger security challenge than in the virtual server space .

Why do I say this?  Well, servers are not as interactive as desktops.  Servers serve... They distribute information whereas  Desktops  request information.   Desktops download  bad things into the environment, servers get compromised when there is some vulnerability.  Desktops are also User controled where as Servers are Administrative controlled.  Why does that matter? 

Imagine moving 100 desktops into the virtual environment and your users are now downloading viruses, spyware, etc. etc. into an environment that has no LAN based security controls.

Can't I just put Trend Micro Anti-Virus on each Virtual Desktop?  The answer is yes you can.  But, keep in mind that Virtualization means SHARED resource.  You would now have 100 Anti-Virus software products running on shared CPU's.  Just think if 100 virus scans turned on at the same time and started scanning 100 virtual hard drives.  Hmm.. It seems like that would eat up some CPU cycles?

So... One step is to have some identity based access controls that restrict what resources a user can access, when they can access them and how they access them. 

Virtual NAC???  No, I'm talking beyond just Access Controls.  I'm talking strict policy control embedded in a virtual switch that can control what users are allowed to do.

Has anyone seen a solution to this concern?

-JP

Isn't My Physical Firewall Good Enough?

When looking at how to secure Virtual Servers within your data center many people immediately think that if they have physical firewalls and such in their data center they are protecting those virtual servers.

The answer to the question on whether physical firewalls are good enough is no.  Physical Firewalls and some virtual security products sit inline between the virtual machines and the physical network.  These physical firewalls or virtual security products provide security between the physical world and the virtual world but provide no security within the virtual world.

If one VM is trying to access another VM how can you enforce security policy between those VM's.  The physical firewall isn't in the communication path as you can see from the diagram to the left.

So the only way to provide the VM to VM controls is to either run host based security solutions or a security solution that is in the VM to VM communication path (Virtual Security Switch).

-JP

Are enterprise customers concerned about virtual security?

Do you believe security in VMWare environments is a high concern?

Virtualization has become one of the hottest technologies of the decade and will revolutionize the way in which we do data center computing.

Do you believe security solutions that run within virtual environments should be a high priority, medium priority or low priority for enterprise customers given they already have physical security solutions such as firewalls and IPS devices already in place?

Please chime in by clicking the comments link bellow this posting.  I'd love to start a useful discussion on this topic to learn more on what people think.

Who do you trust?

I came up in the network / security industry with the concept of "trust no one" at the forefront of my brain.  Well, trust no one until you have been given assurance that you should trust someone or something.

So, do you trust "Virtual Disk Images" downloaded off the internet?  Would you download an image from VMWare's Virtual Market Place or a web site called ThoughtPolice.com?

Have no clue about what I am talking about?

Well, one of the cool things about virtualizaiton is that servers and desktops now have the ability to go mobile.  They can be copied from place to place and even be downloaded off the internet.  This capability makes it easy for you to get a server up and running. 

Remember the days when you had to install a Novell 3.11 server from 20-30 floppy disks?  It was painful wasnt it?  Worse than watching paint dry.  You had to stare at a screen and wait for the next prompt to change the floppy disk.  Then you would get to a question to enter some information that you didn't have a clue about and then have to rush to grab the manual.

Well, now with virtualization you or someone else can go through the installation process and once the server is  installed, you can replicate it without having to ever install it again.

The problem with the above sentence is "someone else".  Again, I trust no one else and I definitely don't trust someone I don't know installing a Linux server and publishing it on the internet for me to use.

But there are many people out there in the world that are ok with downloading "Virtual Disk Images" off the internet and placing them either in lab environments or production environments.  The problem with this is that anyone could create a Virtual Disk Image of the latest Fedora Linux operating system, purposely embed a trojan or virus in it and make it readily available on VMWare's Virtual Market Place or sites like ThoughtPolice.com

   Click Me                Click Me  

An unsuspecting, trusting individual could then download that "Virtual Disk Image", run it inside their VMWare environment and the next thing you hear is there data center or lab is attacked.

Downloading these virtual disk images are more dangerous than downloading a file off the internet or clicking on an attachment in an email from an unknown sender.  Why do I say this?  Because downloading a virtual disk image is a FULL ON operating system with many applications in it.  If a hacker has control of a full operating system they can do things like schedule attacks that happen in the middle of the night, port scan your network for information and email the results to a BotNet Master and even run a packet capture of traffic and FTP that to a BotNet master.  Imagine the possibilities and imagine being able to run any application not just a small file attachment.  An application buried in a directory somewhere on the Virtual Disk Image.

Did I just bum you out and paint another picture of doom and gloom?

Well, its not all doom and gloom.  Knowledge is power as they say and now with this knowledge you should think twice before downloading an image off the internet and use it without fully checking it out.  Fully checking it out means running anti-virus software INSIDE the image and making sure you have VM to VM aware firewalls within your virtual environment to isolate traffic flows between VM's.

Lastly, I think downloading these images is pretty cool and would love to be able to take advantage of someone else watching the paint dry during an installation however, I think there needs to be a "Verisign" of Virtual Disk Images.  This way someone who you trust can do the work of inspecting these images for me.

-JP

Every network has a firewall, shouldnt a virtual one have the same?

If you agree with the first part of the title to this blog, then logic would indicate that you agree with the second half of the title however the reality is that this isn't the practice that most companies are taking.

Why is this?  I believe this is because history proves itself time and time again and in this case history has proven that we are quick to take advantage of things that are cheaper and make our lives easier and put the "what if's" on hold. 

We do this all the time in every day life.  What if I die tomorrow? Well, I'll wait and get life insurance later, I'm still young and healthy.  What if someone breaks into my home, should I activate this burgler alarm system came with my new home?  Nah, I'll wait till later, my neighborhood is pretty safe.  Should  I buy the car with the dual air bags?  Nah, its useually just me driving in my car.

We tend to take the cheapest and easiest route and security is always difficult and sometimes costly.  It's the path of least resistance and security takes work, constant work.  Therefore we deploy virtual networks, know it needs to be secure but tell ourselves "we'll cross that bridge when we come to it".  Can we truely cross that bridge when we come to it?  By the time you come to it, your company is on the front page of the New York times indicating something like "TJ Max Just Hacked, Millions of Customer Credit Cards Stolen!".  I would think someone lost their job on that one for not thinking about security enough earlier on in the process.

The other flawed logic I hear from talking to people is:  My Virtual Environment is not in production yet so I'm ok.  Well, shouldn't you safeguard your non production environment also?  In most of the non production environments I've seen, customers are testing new software they are developing or something that is simulating what the production environment will look like.  Doesn't that data need to be protected also?  What if someone hacked into a lab environment and stole all of the source code for a new application your company was developing in the labs.  Wouldnt it be a pain if someone hacked your lab and caused a situation where you had to spend weeks to rebuild it again?

Enough said.. I think you get the point....

So, what is it that needs to be secured anyway?  What makes the virtual network a network that is unique and calls for even more security?

Well, the answer is simple.  Its a network, therefore it needs to have firewalls but what makes it more insecure is because the Virtual Switches are not equivalent to physical switches.  You can at least set up ACL's (Access Control Lists) on physical switches to isolate traffic but in the Virtual Switches you can not.
So without this ACL type of isolation you are even more insecure than your physical networks.  No Firewalls and no ACL capable switches.

What needs to be secured is communication between the machines within the virtual network.  Think about this for a moment:  If I put High Security virtual machines on the same network as Low Security Machines, are'nt those High Security Servers now in a Low Security environment?  Common sense answer right?  Of course they are now in a Low Security Environment!!  However, if you have isolation between those types of virtual machines, you've now isoloated, partitioned, segmented and split up your virtual network into High Security and Low Security segments. 

Listen to the bellow DEFCON video, it will also give you some GREAT technical visibility into what's flawed with how people are going about virtual networks.

-JP

Dense Computing = Less Security

In case you all haven't noticed, there is a trend taking place that is all about building "GREEN" and "VIRTUAL" data centers which take advantage of dense computing architectures.

This trend is taking off for a number of reasons:

Multi-Core processing = More processing power for more applications on a single server
Blade Server = More servers with more processors in a smaller amount of rack space
Virtualization = More operating systems in a physical server.

Multi-Core + Blade Server + Virtualization = Green, less cost, easier managed, less space, less cooling, less power, etc. etc. etc.

I think we all get it!  It has lots of advantages!

BUT... What we have created is "Dense Computing" which is putting a lot of security eggs into one basket.  Imagine having a Blade Server with 12 blades in it, each blade having 8 CPU cores fitting into about 15U of rack space.  You now have 96 CPU's to drive your operating systems and applications.  Wow!  In the old days that would have been a mainframe of sorts or some Cray Super Computer!  Or in more recent times that would have been 96 rack mountable servers in your data center.

Now, take this one blade server and replicate it until you fill up a rack and replicate it some more until you fill up a row in a data center.

Now you have lots of "Virtual Servers" and "Virtual Desktops" running in a very very small piece of real estate.  This is great news!  All delivered by the power of multi-core processing technology, blade based computing technology and virtualization technology.  Once again; Mutli-Core + Blade Computing + Virtualization = Green, less power, less rack space and uhhh..... LESS SECURE!

Why is this less secure?  Well in the past you had physical servers and in many cases you segmented off your data center by having physical firewalls between servers or server groups.  If all of these servers are now running in a virtual environment you no longer have the ability to physically isolate these servers and the problem  just got worse because you have more density of them in a place where you can't  secure them.

If you think about the example of one blade server environment with 96 CPU cores and virtualization layered on top of it, you can easily see an environment where one could get  960 virtual servers in a single blade server with 12 blades of dual quad core processors.  Wow!  Thats 960 virtual machines with no isolation between each other.  You could possibly get some isolation between the blades "IF" you turned on some ACL's in the "Integrated Blade Server Switch" but the traffic definitely isn't going to touch your physical NetScreen or Checkpoint firewall unless you start routing traffic out of the box and back in.

People are starting to talk about the security problems caused by virtualization but I thought I'd point out the fact that the problems gets even bigger when you virtualize on multi-core and blade server environments.

Think twice on your security design before you deploy!  Ask your security vendors to support virtualization! 
<---Click to view
Gartner has something called the Hype Cycle and I think this problem is more than "Hype" and is something that companies should take a serious look at right away.  The good news is that awareness and education in the market is taking place on this topic as indicated in this Graph showing Gartner now tracking "virtual security partitions".  Thanks Niel McDonald of Gartner for paying attention to this space!

JP

Virtual Security = Virtual Performance Challenge

Coming from NetScreen a performance leader in Firewall, Fortinet a performance leader in UTM and Reflex Security a performance leader in IPS many can see how performance is burned into my brain.

So, as I start thinking about security in the virtual environment I think not only about security but the performance impact security applications will have on the virtual environment.

People virtualize because CPU/Memory resources have been UNDER utilized.  People have traditionally bought a server to host an application and those applications are not always in use.  Many times they sit idle while other servers are maxed out and could use the help of those idle CPU's on the server in the next rack.  So, by sharing CPU/Memory resources virtualization allows for better use of resources and helps applications take advantage of CPU cycles when needed.  Ok, we get that.... Thats virtualization.

Security applications ARE typically utilized.  If there CPU's are idle then something is wrong.  We want those CPU's working 24/7 because we want to make sure we are secure.  Would you hire a security guard that slept on the job?  No, you want him attentive, walking around, checking for open windows, etc. etc.

So, now we have a challenge!  If we put security, something that is heavily utilized into an environment  that is intended for servers that were once under utilized we can cause a problem around why people virtualize in the first place.  Catch 22 eh? 

We need security but we don't want to pay for it.  Isn't that always the issue!

Well, not exactly.  The key thing to think about is the type of security that you need in the environment and then you need to asses whether or not that level of security is important enough for your business drivers.  Some things need to be protected more than others.

But, at a high level, think about this.  Security needs to be as close as possible to the things you are trying to protect.  The President has his security detail right beside him at all times.  This can be related to HOST based security.  The President also has Secret Service guys on the roof of the white house and on the front lawn.  This could be called Edge and Perimeter security respectively. 

Now, in the virtual environment HOST based security is VERY expensive from a resource perspective.  Imagine having Symantec Personal Firewall/AV on each virtual machine and lets say you have 20 virtual machines in an environment.  If all of those host based security tools kick off a virus scan at the same time, don't you think the CPU cycles will spike?

Once they spike, the CPU resources are not available anymore for the server applications which is what drove you to virtualize in the first place.

If I do some sort of network based security in the virtual switch then I'm as close as possible to the things I'm trying to protect without being on the things I'm trying to protect.  You now have one virtual security switch serving 20 VM's vs. 20 Symantec security applications.

Ok, so that makes sense.. straight forward right.  Its easier to manage 1 thing than 20 and you now have a shared security point in the network vs. distributed.  Got it.....

BUT, its not as simple as that.  The other question one needs to ask themselves is what type of security application is good enough for the assets I'm trying to protect.  Is it Firewall?  is it IPS?  is it Anti-Virus, etc. etc. etc.

Once you pick one you now need to think about the performance ramifications they individually have.

Firewall for example is less expensive than IPS.  It simply looks at less data.  IPS engines done in User space are more expensive than IPS engines done in Kernel space.   

I personally believe that IPS done in its traditional fashion is to expensive for the virtual environment.  Take Reflex Security's VSA product which I use to Product Manage at Reflex.  Its very expensive and depending on how its configured can consume 70% of the resources in the virtual environment.  Traditionally IPS has dedicated CPU's.  In fact, I designed a 10 gig IPS system that required 48 CPU cores.  It was great for the physical world but when you virtualize you don't want to dedicate that many CPU cores for IPS, otherwise you turn it into an IPS not a Virtual Environment.  You need those cycles for server applications.  In fact, if you go back and look at some of the press releases around the Reflex VSA product you'll see that Reflex multi-threaded their Virtual IPS product so that it could use more CPU's to deliver better performance in the virtual environment.  This doesn't actually make a whole lot of sense now that I think about it.  But, it was great marketing at the time!

See:  http://www.reflexsecurity.com/news/052207_reflexships.php

Firewall technology because its typically looking at headers and such take up far less CPU cycles to deliver the same level of performance as IPS.  But, their is a trade off with that to.  You don't get a view into the content.  So, it really comes down to the price/performance/risk assessment that companies need to make.

Soon you'll see vendors look for smarter ways to deliver Firewall + Content Inspection levels of performance without having to consume  as many CPU cycles.  This will then allow for a healthy balance of security and server virtualization.

John Peterson

What type of security do I need in my Virtual Network?

In the physical world we have all grown to understand that there are many types of security solutions that are needed to secure your environment.  We purchase products like Switches with ACL's, Firewalls, Intrusion Prevention Devices, Patch Management Appliances, Network Access Control appliances and many times we for go "best of breed" and go for the "all in one" approach and deploy UTM devices.

So what has changed for the virtual environment?  Nothing really.  Those same types of choices and things need to be looked at and considered.

But!  The Vendor community would lead you to believe that you don't need various types of security products in your virtual environment.  They would also lead you to believe that you only need their solution.  In fact, they all compete against each other to some extent. 

I'm sure if you were to ask Reflex who their competitors were, they would tell you Blue Lane and Catbird, or if you were to ask Catbird who their competitors were, they would say Blue Lane and Reflex.  I know this because I use to site these companies as competitors myself while serving as the Chief Product Officer at Reflex Security.

As a vendor we spend so much time trying to show our value that we loose sight of the real value of security solutions working together to provide a comprehensive and secure solution vs. a single point solution.

Think about this for a moment.  None of the following vendors really compete with each other, in fact they can complement each other:

Blue Lane - Provides Inline Patch Management
Reflex Security - Provides Intrusion Prevention
Montego Networks - Provides Secure Switching (Firewalling + Switching)

Still Secure - Provides IPS
Catbird - Provides IPS

Now, you can say Reflex, Catbird and Still Secure compete but the rest are very different.

The real question is how do you deploy a Firewall, Patch Management and Intrusion Prevention products in your virtual environment.  Well, one way is to deploy them in "series" and each product will require a dedicated virtual switch.  Take a look at the picture bellow and you will see how messy the design looks:

<-- Click to Enlarge

Each time a packet has to enter and leave a vSwitch you will experience some performance degradation; however this is a requirement by VMWare if you want to install "guest-based" security appliances. 

This  security product to vswitch, to security product, to vswitch is very much like an A/D (analog to digital) conversion that takes place on  digital networks.  Each time you make an A/D conversion you introduce  noise and noise introduces signal loss, which introduces poor performance or sound quality.

Not to mention its just really messy looking!

So, how does one deploy the security products one needs in the virtual environment without causing a performance challenge and how do we get the vendors to stop competing and start joining forces to deliver solutions that work together?

Well, one way of doing this is to put some intelligence in the switching architecture so that it can play "traffic cop" and send traffic to the needed security applications.  This type of design would be security in parallel vs. in series.  Take a look at the bellow graphic and it will be more clear:

<-- Click to Enlarge

You'll also note from this picture that the Security Switch in the center is already able to see VM to VM communication and by it playing traffic cop as well as switch and firewall it can also extend its VM to VM capabilities to security products that do not have that ability.

In the previous picture, products were deploed in series and there was no VM to VM Patch Management, or VM to VM Intrusion Prevention or VM to VM Network Access Control.  What you were able to get was VM to Physical Patch Management, Intrusion Prevention, etc.

With a product such as a Virtual Security Switch you get VM to VM everything hooked up to the Security Switch. 

What a concept!  Companies partnering to provide a comprehensive security solution.  No competing, each company focuses on their core competencies and works together to give customers what they really need.

Think about it, does McAfee compete with NetScreen?  Did Checkpoint compete against Tipping Point back in the early days?  No, we had Firewalls, Anti-Virus, Intrusion Prevention products all co-existing and many of these vendors partnered with Extreme and Foundry since they were the connectivity point of the network.

I think the virtual network is no different, so vendors, please stop confusing the market and telling customers they only need IPS, or only need Firewall, or only need Patch Management.  What customers need is choice and the ability to have the products they choose co-exist without causing major performance and management challenges.

-JP

VMSafe = A Safer More Secure VMWare Environment

New VMware VMsafe™ Technology Allows the Virtual Datacenter to Be More Secure Than Physical Environments

Twenty Industry-Leading Security Vendors, Including CheckPoint, McAfee and Symantec, Endorse VMsafe Technology and Announce Plans to Build Interoperable Security Solutions

Cannes, FRANCE, February 27, 2008 – VMware, Inc. (NYSE: VMW), the global leader in virtualization solutions from the desktop to the datacenter, today announced new security technology called VMware VMsafe™, http://www.vmware.com/go/vmsafe,  that protects applications running in virtual machines in ways previously not possible in physical environments.

To read more click here:  http://www.vmware.com/company/news/releases/vmsafe_vmworld.html

---------

Wow, what an announcement today for security vendors looking to sell their wares to a growing base of customers taking advantage of virtualization and a great way for VMWare to help its customers secure networks created by VMWare!

This announcement from  VMWare  does highlight that VMWare is serious about helping their customers address security challenges.  What is still to be determined however, is what this really means to customers.  There were 20 security companies announced in the partnership and little information about what security problem each company is solving.  I guess  we should expect to see 20 press releases from these individual security companies in the near future.

My educated guess though, is that most security vendors will just be offering their existing security products that are in many cases physical firewalls, anti-virus, UTM, etc. The real value will be from solutions that bring unique value to the virtual environment vs. network designs that dictate routing traffic out of the Virtual Environment to a physical security appliance and back in.  The other question is ; will the software vendors just be installing their software on the operating systems of Virtual Machines vs. Physical Machines?

Are there any real hooks being offered today that connect to VMWare and take advantage of these API's or are these things yet to come?  My educated guess is that these are still things yet to come from the majority of the vendors in the program.

I've had the privileged of reading the API documents as the CTO of Montego Networks which is also part of the VMSafe program that was just announced and am very excited about the future possibilities of the program. 

I'm excited to see the space finally get its due attention and am confident that the program will give birth to many new ideas and products that help solve the many security challenges introduced by virtualization.

There are so many vendors in this newly announced program.  I hope to see quality from the program vs. marketing quantity!