Groves Green IT

SVLG's 2nd Data Center Energy Efficiency Summit

The 2^nd Data Center Energy Summit taking place on October 15, 2009 at Netapp corporate offices in Silicon Valley was really a feat.  Silicon Valley was in such a state of chaos in the early months of this year as the recession hit small and medium size businesses, that it was difficult to know whether there would be sufficiently interesting and completed pilots.  Yet, the Valley rallied and the event was full (400 plus attendees), the speaker podium compelling, and the audience was in good spirits. 

 

Speaking with Zen Kishimoto, PhD Principal Analyst, Green IT at AltaTera Research Network  who edited the case studies, he brought his own insights as he completed the task  he faced this year of compiling the results of the case studies (Teresa Tung of Accenture had this role last year):  “Like last year, we had a lot of participation and many excellent results.  We have fewer completed case studies than last year but will have several additional ones coming soon. One of the studies that attracted my attention was done by Power Assure. It proved the reduction of power consumption by IT gear can be achieved via hardware refresh and dynamic power management… Another pilot is the one conducted at an Intel data center, which measured temperatures at each server and reflected them to the overall floor air conditioning via wireless sensors. This seems to be the first attempt to link the condition of IT gear to the building management system Overall the reported technologies and practices are of high quality and I am sure the audience will find the results highly applicable to their own data centers.”

 

KC Mares of Megawatt Consulting and co-chair of this event,  said: “Worldwide, we spend $23 B as an industry on data center industry annually. That is the equivalent of five major nuclear power plants. Last year’s pilots resulted in an average of 56% ROI for end users. Our goal is to chop away at those huge numbers with our case studies.”

 

After a keynote speech from NetApp’s CEO, the audience heard an interview between Jeff Byron, Commissioner on the California Energy Commission and KC Mares.  Jeff said that California rate payers will spend $38 billion over next 10 years to reduce GHG but we can’t afford not to spend it.  He said the electricity sector contributes 23% of Ca’s GHG and all industrial sectors will be asked to reduce their 1990 levels by 2020.

 

This year, I played a background role in the Summit, participating in the leadership team that has met since January 2009 to bring the case studies to completion, created the public relations strategy for the SVLG Data Center Energy Efficiency Summit. These are some of the pilots I found most interesting:

 

1.  In the new Research Triangle Park NetApp data center, Joe Miller, site operations manager there,  said his infrastructure supports 369 IT racks, 1800 engineering racks and 30 engineering network racks. The 360 IT racks run at 6 kw per rack, whereas the 1800 engineering racks run at 12 kw/rack.  Custom air handlers for the NetApp data enter enabled them to develop a patented process to enable a cold room and hot aisle . Equipment draws the air where it needs to go, so  they don’t need to confine it too tightly. Hot aisle is open return in their pressure-controlled cooling architecture with two sub-systems: one of modulated temperature, and the other is to manage the dampers to modulate the air flow with the controlled fan speed of air handling units dedicated to the  cold room. Netapp  calls this “cold aisle containment with pressure controlled cooling”. The chiller plant operates at 55 degrees. The air side economizer means the chiller runs fewer hours than conventionally.

 

Typically if you elevate the chiller temperature, it increases fan power. But as long as it is below 80 degrees going into the front of the equipment, you can allow elevated chiller temperatures.  (The most efficient chiller is one that is off, Joe joked) And Netapp tries to keep it down. Cold aisles are contained and hot aisle are open so dampers can fill it out. Cost savings features described above resulted in a $50K per rack build expense rather than $150K per rack, the industry average. 

 

In the event of a chiller failure, there is the opportunity to ventilate. The number of hours that they cannot ventilate is a small risk, he assured the audience

 

2. Mukesh Khattar gave an interesting presentation on the case study he did at Oracle.  He started with the assumption: Variable Frequency Drives (VFDs) are required for hot air containment with any type of enclosure. But do we need VFDs if we don’t have enclosures, he asked of his team.  Enclosures do avoid hot/cold spot issues but the cost savings really comes from the FVDs.  He said that in a 5K sq foot data center, they managed a retrofit. The estimated cost of the project was $130K. They estimated energy savings of $27k per year, and a utility incentive of $33k.  The fan speed went down 50% with dynamic load matching. Oracle concluded that they could avoid the cost of curtains which brought the cost of the project down by $50K roughly. It made the ROI very attractive. One of the unanticipated outcomes is that you could not reduce fan speed below 50% - the fans turn off.

 

3. Joyce Dickerson reported on a very complex project undertaken at Stanford University campus. They discovered 70 different server rooms with a wide variety of purpose and use. They selected five to meter with Synapsense products, and looked among their 70 installations for one of each of these: cooling types using DX, CRAH units, Rittal water cooled, APC Inrow, and fan coil & building air conditioning system hybrids.  Across the campus they found that 1-3% of the floor space but 9 – 30% of the energy in each building were taken up by these server “closets”.  Their first conclusion was to always isolate the satellite servers from the building air conditioning systems: that saved the most money because air conditioners generally had to run an entire building or floor while just a small area required the cool air.

 

4. Pixar did a cold aisle containment case study and with simple blanking panels, panels and curtains, and air-dense foam, and had a payack of 3.7 months on costs to obtain an annual energy savings of $30K.  Be sure to read that case study.

 

5. Ralph Renne of NetApp hosted a case study whose objective was to move their focus from temperature control to pressure control. To ensure that cold aisles will not be oversupplied or undersupplied,  Renne concluded that temperature sensor placement becomes crucial.   Temperature control does not guarantee that the volume of air is being delivered but pressure control does, and that pressure control also delivered more air into their cold aisle. 

 

Andrew Fanara of the EPA, who attended last year was here again this year because in his own words, “the activity around energy efficiency in data centers is more specific this year, moving away from rhetoric to action.  Companies making investments and gaining knowledge are better positioned as the economy turns around. SVLG has to be admired for pulling this event together in a time when most companies were pulling inward and cutting out any “extra projects”, yet as I’ve seen today, there is tremendous activity still going on.”

Paying tribute to MJB and acknowledging the power of peer to peer networks

First things first:  Mark Bramfitt of PG&E’s data center energy efficiency team, is leaving the company to pursue consulting opportunities.  he announced this last Thursday at the SVLG Data Center Energy Efficiency Summit 2009 at NetApp.  Mark has been my source of knowledge on energy efficiency, mechanical engineering and utility-programs related topics. He has been generous in educating me.  He has also been generous (with all of us) in his role as spokesperson for energy efficiency initiatives that utility companies can choose to offer.  His leadership across the industry undisputed.  There is no one to step into his shoes and he leaves a void.  Thanks so much, Mark!

I have more to say about the Summit but I'll include that in a future blog - just that it was a great event and my notes are on a different computer.  I wanted to take this opportunity to say that I had the honor of addressing the Site Uptime Network on Tuesday, October 20^th, in Atlanta Georgia.  I was there to present to the audience on research I am conducting, on the subject of “Awareness and attitudes toward water conservation in data centers.” I gave an overview of drought conditions in the USA and around the world, and showed a map that indicated that the predicted drought areas in north America overlap to a large extent with the areas where drought might occur. 

I became very interested in this topic when I realized at the Govenergy conference in August that energy efficiency initiatives across the federal sector in the USA included water conservation milestones.  (see earlier blog).  Although data centers were not expressly mentioned, 15% reduction in water usage from 2007 levels by 2015 in all GSA buildings indicated to me that the private sector data center operators should be implementing water strategies to coordinate with their energy strategies – and I didn’t know whether it was being done.

So I developed a 39 question survey together with Dr. Bob Sullivan and David Rosenberg and I am doing the research now. Results will be published in early 2010. Those who fill out the survey completely will be given the report at no charge.  The report will be for sale to other interested parties.  Contact me if you have questions about participating or sponsoring the research.  Your data center must draw at least 5MW of power to be included. Go to "email me" in the right hand column to reach me.

You can see some of the  North American Site Uptime Network here.  Although I had known about the group for a few years, I didn't know the following until a few days ago:

Network members average a 50% reduction in site failures during their first five years of membership.AND Network members average a 50% decrease in incident severity during their first five years of membership. These numbers must be quite comforting and well worth the $15K annual membership fees.

Ken Brill presented a slide that he titled Site Resilience = Tier plus sustainability.  He said Tiers certification is about concurrent maintenance and fault tolerance in contrast to operational sustainability which is about frequency of unexpected failures and duration of failure and ability to recover.   He also went over five tier topologies that are needed to achieve operational resilience. It was a masterful roadmap that was intuitive and helpful. 

Members are divided into North America and EMEA.  The North America group has about 100 members but some of them have asked to remain anonymous so a complete list is not readily available outside the group.  It has been going for 17 years. One of the earliest members, American Express, has been participating since 1993.  The members are divided into groups that generally don’t lump competitors in the marketplace together. So with five different groups going, there can be five corporate memberships in insurance, auto, banking, etc., without a breach of confidentiality. New members are voted into the group. 

 

Participation includes not just fees but active contribution. These include opening your data center to site visits of other members, and notification of abnormal incident reports, a minimum of four per year.I had the opportunity to listen in to a debriefing after a visit to a member facility and it was very illuminating and refreshing to see the peer to peer sharing that took place.

It seems to me that $15K is a small price to pay to share in such large risk mitigation.  Have a Green Day!

 

.

Intel invites the Data Center community to quasi debates

On June 11, Intel sponsored an event in Santa Clara titled the Intel Technology Great Debate.  I took notes then and again yesterday at the second event, held in San Francisco in conjunction with the Intel Developers' forum. This event yesterday brought in 30 or so new audience members whose interest in software development clearly dictated their perspective on the issues raised for data center energy efficiency.  Both events were well attended and oversubscribed.   

Here are some of my notes from June 11th:

Andrew Fanara spoke first. He cited a May 24 The Economist article. “Clearly there is frenzy around this. Good long term planning will get us out of it. Nothing less.” Lets avoid being bullied into bad decisions relevant to energy policy. World oil production is flat for 2005 – 2007. More costs will come down the pipeline to get it moving upwards again. Even with promising renewables, carbon based fuels (including 40% from oil) are still the largest share of our useable resources and energy efficiency is very important. To get the full range of technologies, CO2 might have to be $ 200 per ton, and it is currently less than 20% of that.

USA energy demand between 2008 and 2010 is equal to 10 new power plants, and data centers are currently using 5% of production and expected to surpass airlines by 2020. CO2 emissions are projected to quadruple from 170 MT (metric tons) to 670 MT. Carbon sequestration is in limbo until our Congress makes some decisions – it has lots of promise but won’t be realized anytime soon.

“The grid of today is not the grid of the future. We need a trillion dollars of new investment in the new grid to bring it up to the standards of the future. We’ve under-invested in the past. Efficiency is a way to mitigate some of these problems."

Customers, Boards, large investors, etc  all want to know: how much energy do you use, so we can understand the risks? Risk and cost containment require measuring what you use. When will we have zero energy consuming buildings? It is worth thinking about data centers as a big part of the entire building consumption.

I then took a lot of notes about the debate topic (AC v DC) that in hindsight make very little sense because some of the team members argued for their competitor's position.  So in retrospect the comments from Andrew are the most memorable part of the event.

At the August 18 debate there were two questions after an hour's summary of the various partners involved and their agendas.  The two questions were:

Which is better: high or low density data centers? moderated by Matt Eastman of IDC.
Verari, Dell, IDC Architects, Oracle, Microsoft were represented on the panels.

Which is better: data centers in concrete/brick buildings or storage containers?

First we discussed low density and high density attributes.

To begin with, how would the audience define low density and high density for the debate? We could use as examples:  6kw racks and under are low density, 7 – 12 kw are medium density and 13kw+ racks are high density. When density starts to drive design decisions it becomes high density.

Legacy cost allocations drive density: “he who goes to higher density first pays less” is a quote from the low density team’s slides. At some point in the future, 30k racks will be low density, so any definition is a moving target. We need to be looking at how we charge business units for power, said one panelist – that will drive the density decisions. Micosoft’s USA-based data center costs in the USA are 93% mechanical and electrical, with just 9% for the shell.

 The new advances mean that you can charge on the basis of consumption of electricity rather than on space utilization. This will result in different design. Server power has gone down at Microsoft and Oracle since chargebacks became power-consumption based. Provisioning strategies and security concerns mean that we won’t run all servers at 100%.

Cost per megawatt or kw per rack are better metrics than kw per square foot.

Shortest distance = less work to be done moving air, therefore

Shorter distance = more efficiency so density is GOOD!

But at the end of the voting, low density seemed more popular than high density because of the added costs to cool high density environments.The lifecycle for IT equipment is much shorter than that of the infrastructure so therefore, you have difficult meshing the two.

With respect to the 2nd debate, it was so congenial, with both sides admitting that the other had good points, and that new modular brick & mortar shell environments essentially offer many of the same benefits of the modularity in containers, that no clear consensus was reached.

Here is my recommendation to  Laurie Wigle of Intel, who created both events, for Debate #3:  Allow Henry Wong of  Intel and  Christian Beladi of Microsoft to moderate the panels so that the conversation between the contestants will be more evenly matched in personality.  That means David and Matt from IDC (currently the moderators) will have to take opposite sides on the Tupperware v BAM teams and we'll see different dynamics emerge.  Have a green day!

 

 

 

IDC has lots to say about greening the data center

Don't forget to register and use GROVE1595 for your code to attend the TUI Symposium at a discount. See my blog of February 23, 2008 for full explanation and the link.

I had the opportunity to hear Matthew Eastwood and Michelle Bailey of IDc speak at their Enterprise Data center Forum event in San Francisco on April 7th. IDC, you may know, is generally credited with maintaining the best industry-installed base and projections numbers among the analysts, and I like to hear their insights obtained from "following the numbers".  They did not disappoint.

To begin with, here is Matthew Eastwood’s list of attributes required when designing products to be economically, socially and ecologically sustainable. He gave examples from IT products

1. Durable (longer service life)
2. Energy efficient
3. Designed for reuse and/or recycling
4. Manufactured with local or renewable resources

He had a 5th attribute but I could not understand it in context and I forgot to ask him what it meant, so I will not include it here. Anyway, this is the measure by which we should judge appliances in the data center to be "green". 

Having said that, Eastman moved on to data centers.
Today, there are 365,000 full racks of servers representing US$15B in data center assets. And it takes $30 billion to cool those racks! Thirty million servers sit in those racks today, and that will grow to 45 million servers by 2011. IDC's research indicates that 90% of those racks are filled with X86 servers, and on average, where they are not virtualized, they are 90% underutilized. The good news is that 10% are virtualized, and within an organization that has started a virtualization initiative, 22% are virtualized.

Virtualization is ubiquitous in 100,000-plus employee companies up from 72% of all installed servers to 90% from 2006 – 07.  In medium companies  with 500 - 1000 employees, virtualized servers went from 38% – 71%.

Bailey added more numbers:  Today America has 7000 enterprise class data centers  of which 50% or more are In metropolitan/suburban areas.  They have 50,000 sq ft on average and support more than 1/3 of all systems within their respective companies.  Their average age is 12 years (built in 1996) and building updated facilities will cost an average of $25 – $50 million, with costs of construction exceeding $1000 per sq foot.  Today's "norm is designing with 125kws per sq foot.

Bailey provided this "solution" investment prioritization indicating how enterprises are dividing up their budgets for data center retrofits between today and when those new data centers can be occupied:

1. Room level changes: 30% hot cold aisles, reduced cabling, plenum clean up
2. System level changes: 29% low voltage chips, memory and power supplies
3. Rack level changes 24% blanking panels, in-row cooling, network sensors
4. Consulting services 17% design services & utility rebates, and thermal assessments

As a team, their conclusions were quite clear: Are we green yet? No, not close.  Have a Green Day!

Note: I am writing these notes from memory, not from the slides, so I may have blended the separate presentations by Eastman and Bailey, but I feel pretty sure that the notes are accurate...

 

 

 

 

Green IT: Yours, mine and everybody else

In Don Dodge's blog of April 2006 he says that SaaS was at 10% then and projected to grow to 25% of all software sales in 5 - 10 years. Is this projected growth in the SaaS channel poised to replace or augment the current ballooning in corporate data center demand? I don't know the answer... 

I am quoting my own blog from September 12, 2007 because I still don't have an answer but I can clarify the issue a little more today.  I see a huge disconnect between the cyber world of games and social networking and the online world of financial transactions. They are merging, of course, so some definitions need to be amended in order to understand the impact of thousands of data center supporting "the cloud". 

Few analysts, to my knowledge are looking at the whole picture.  One exception is AMR, a research company where analyst Bruce Richards keeps talking about social networking sites in his Supply Chain Management reports. He very clearly states that most traditional SCM software providers aren't providing the tools that young people know how to maximize. Now I assume that most companies are still hiring people younger than 25 years of age to download and upload data and create files to organize their relationships among partners and customers.  So they probably have two sets of data, one using traditional database tools, and the other using social  networking tools. It reminds me of having two sets of books in a black market economy: one for the tax paperwork and one for the family board meeting.

Few conference producers understand how to market to this convergence either. In a discussion with Tom Coughlin  a storage expert, he kind of, sort of, agreed.  On the front page of his website is written:

By 2010 we expect that close to 40% of total shipped hard disk drives will be used in consumer applications, up from about 14% in 2004 and by 2015 personal content will greatly exceed commercial content. 

I just don't think most people will be sure what is personal content and what is commercial content.  So why is this significant? Because rebates, incentives, taxes, warranties, etc. are all based on the categories of personal v business use. But IT purchases are a good example of how those products and their use are blurring. (Nobody carries around two phones anymore and the thumbdrive in my purse has my current workload as well as my holiday greeting card photo on it as well, doesn't yours?  Who takes the time to separate and sort their info neatly these days?) And who knows what a "corporate" data center will look like when one copy of your important data is on the cloud, one copy is on your thumbdrive, and one is resident in your company data center. Who is adding it all up so that we can really predict the power consumption of computer appliances in 2015? Write to me if you have the answer.  Have a green day!

GreenIT: IT Business Edge asks the questions

Well I got lucky this week for this blog entry because Carl Weinschenk interviewed me for his column about Green IT.  He asked, I answered, and he wrote.  Here are the results. 

Green IT:not solar powered anytime soon but good news anyway

Just a quick note to say there won't be any posts for the next four weeks as I will be travelling. I expect to meet with consulting colleagues in southern France, in Provence, who are working on sustainability strategies for their clients. I am expecting to learn alot about the French version of Green IT from them. You may know that France has sufficient electricity because they embrace nuclear power more than in the USA. So, the conventional wisdom is that French companies won't be moving toward Green IT anytime soon because they have cheap clean power.  Of course, if you have been reading this blog, you know that price and availability are only two of several driving forces for Green IT.  Others include space, connectivity density, and heat dissipation.  Few of us are intentionally trying to build rooms that are heated to 105 degrees Fahrenheit since they aren't very tolerable.

A little off topic, one of my favorite local professional associations, Silicon Valley Leadership Group, because it really does promote leadership, hosted Energy Summit 2007 on June 1st.  One of the speakers suggested renaming Silicon Valley to "Solar Valley" because there is so much innovation and investment in this alternative energy source.  By the end of the day, many of us in the audience felt less gloomy about Global Warming because so many initiatives are underway, metrics are being established, and possible solutions are in pilot phases.  Encouraging signs.  Have a green day!

Green IT:Simple measures to start

Continuing from last week’s blog with my education from the very erudite Bruce Taylor of Uptime Institute, every data center needs to look at the following five issues for both short and long-term energy savings.  These are:

1- Server consolidation, configuration, virtualization

2- Enabling “power-save” features on servers

3- Turning off “dead” servers (no longer in use but running)

4- Pruning “bloatware”

5- Improving the Site Infrastructure Energy Efficiency Ration

Typically, data centers can improve energy savings by 25% to 50% over a two-year period with just by tackling each of these in a cross-discipline way.The Insitute has develop a multi-functional team methodology known as ICE (for Intergrated Critical Environment) to provide both the business and technical rigor required. Not on this list above but a tried and true method is the email sent to all users announcing server shutdown for maintenance for a 24 hour period.  Active server users rebel instantly with reasons why their servers can’t be shut down.  If no angry responses result, shut them off indefinitely (35 days? More than an accounting period…) and then disconnect the servers.  This frees up space, energy, and manpower. 

Bruce was very clear that mitigating business risks are as important as energy savings in re-examining, retrofitting, and re-designing data centers. They are at the nerve center of every company, large or small.  So active participation and support are required from five key individuals across the organization: representatives from the offices of the CFO, CIO, and Real Estate and Facilities, as well as data center IT and facilities managers and technical teams.

Overwhelmed by the task? Outsourcing of non-strategic activities may be looking attractive.  Have a green day!

Green IT: if you’re big, you already know about UI , No, not ‘User Interface’ this time.

The Uptime Institute is a research-based academic group founded to serve data center owners/operators and senior facilities engineers across the USA, searching for leading practices for data center facilities and infrastructure systems design, engineering and operations.

The organization measures the energy waste in data centers resulting from power supply, distribution and cooling.  According to the Institute, gathered from the 85 large-scale corporate members in its network, data indicates that it takes 2.5 Watts at the building's electricity meter to deliver 1.0 Watt to the compute load.  So where is the other 1.5 Watts going?

According to an Institute White Paper, many solutions that can improve energy efficiency by 25% - 50% are technically feasible today with LITTLE or NO new capital expense.  The problem is that performance measures for data center staff are related to uptime and software integration/enhancement projects: there is no sizeable upside for their professional careers in data center energy efficiency - yet.  Why?  Because the largest stakeholders on energy efficiency, the CFO and the CIO, aren't yet part of the strategic conversation. 

For example, while fluid cooling rather than air condition will reduce cooling requirements by approximately 60% - 70%, most CIOs are not ready to spend their operations budget on the investment - it just doesn't seem as critical as other budget items, related to business applications growth.

On an enterprise-by-enterprise basis, the current attitude toward cost avoidance isn't staggering. But across the United States, it is. If the Uptime Institute's members fully adopted the recommendations on energy efficiency, it is a sure bet that several plans for building coal-fired power generation plants over the next decade could be postponed or shelved. Let's see, in CO2 emissions, that's how much? 

Have a green day!

Green IT = processing elsewhere?

Looking to the IT department for leadership about sustainability is really a splendid opportunity to look closely at the “spinal cord” of the corporation or organization. After all, the biggest product and business process innovations will have to be affected by or directly affect the IT infrastructure.  The urgency and the critical nature of looking at the energy spend today is directed by the utility industry’s move toward demand based pricing.

In Europe, for several years, consumers and businesses could earn more favorable rates if they used less energy during peak hours. This way of charging for energy is coming to the US as well.  So during peak air conditioning and lighting hours, prices go up.  The “Critical Peak Pricing (CPP) benefits you on weekdays in the summer season by reducing or shifting your energy usage away from the noon to 6 p.m. peak period during 12 or fewer CPP events. In exchange for this, you will receive a discount on all part and on-peak usage on all other days of the summer period that starts May 1 and ends October 31.” says the PG&E website.

With the ability to switch data processing over to geographical areas not affected by the peak pricing, data centers can avoid the higher priced energy – if they are prepared and if they receive advance notice – enough to send processing elsewhere.  Other options? Well, perhaps a container full of natural gas in your parking lot or on your building rooftop to offset demand, rather like an extra generator.

On the PG&E website, you can find PDFs presentations from the High Tech team. One of my favorite ones shows California holding steady while demand grows elsewhere across the USA– and that is before we had the software in place to shift demand outside the geographic area  with a button.  Now where will that processing go? I think we can move it anywhere your data center is - off shore as well as out of the region.  Something to think about.  Have a green day!