Data Center Grounding

[BillP]

I have a data center application where my client is asking me several questions about grounding (they state they have circulating ground currents and want an opinion about correct grounding methods).

They have a data center building fed by several 2400V-208V D-Y solidly grounded transformers (single-ended unit substations). The unit subs feed panelboards which feed CPCs (which have 208-208Y/120V k-factor shielded isolation transformers and 4 wire panelboards). The CPCs then go on to feed PDUs and racks (mostly single phase loads). There is a raised floor.

This is an old building so the client is unable to verify existing grounding. They have a copper pipe system underneath the floor (with soldered - yes soldered connections), which they use as a computer ground. They think the copper pipe system then has a conductor running out to a single test well where it bonds to the building's grounding electrode system (one spot only for bonding these two systems). They measure 50-60 amps flowing on the ground wires. There are probably ground connections all over the place inadvertently placed over the years.

They have a small amount of harmonics, as well as a small amount of voltage imbalance (due to strong load imbalance at the PDUs and CPCs), so I expect them to see triplens on the neutrals. But they still question the correctness of their grounding system.

I find lots of guides and NEC info that don't necessarily give me the answers I'm looking for.

Seems to me this system should be grounded/connected as follows:

1) Unit sub's neutral bonded to the grounding electrode system and frame of unit sub per NEC separately derived system (they aren't services).

2) Building steel and probably raised floor steel structure all bonded as part of this electrode system.

3) Equipment grounding conductor and neutral run to panelboard from unit sub, the ground bonds the panelboard frame and ground bar, the neutral runs to the neutral bus (isolated from frame and ground in panelboard).

4) Equipment grounding conductor runs to CPC from the panelboard (no neutral, since the CPC has a delta primary transformer).

5) CPC transformer neutral, electrostatic shield, and frame are all bonded together and connect to incoming equipment grounding conductor as well as building grounding electrode system (building steel and/or raised floor).

6) CPC panelboards have neutral and equipment grounding busses (with the neutral insulated from frame and ground bus) - with conductors from both of those busses to the transformer neutral connection point described above in 5.

7) CPC panelboard also has an isolated ground bus (or computer ground bus) with conductor running down to the copper pipe system under floor.

8) Copper pipe is grounded to building steel in multiple places to keep impedance between computer ground system and building ground system as low as possible. (SO BASICALLY ALL COMPUTER GROUNDS, SENSITIVE ELECTRONIC GROUNDS ARE KEPT ISOLATED ALL THE WAY TO THE ISOLATED GROUND BUS IN THE CPC, BUT THEN BONDED IN ONE LOCATION TO THE PIPE SYSTEM WHICH IS BONDED TO BUILDING STEEL AND THE BUILDING ELECTRODE SYSTEM)P

QUESTION: Does what I described sound like a proper and NEC-compliant description for a data center? What else should we be doing or what should be done differently?

QUESTION: Should the pipe (computer ground system) be bonded to the building steel in multiple places to keep the impedance low or should it be isolated from the building ground system as much as possible and eventually bonded together (as NEC requires) at only one location? (As my client believes it should? - and as is the old practice for grounding computer/DCS systems)?

QUESTION: Is there any reason for the feeder conduit coming to the CPC to have a non-metallic fitting?

QUESTION: The client expressed concern that the soldered copper pipe connections become high resistances when they get warm? This doesn?t make much sense to me. I know resistance is a function of heat, but solder itself doesn?t become a bad conductor when it?s warm, does it? Is this a data center thing, that maybe I?m missing? I want to suggest to my client that they just install copper jumpers around the joints that they think are bad connections?

Any assistance is greatly appreciated!

[cmarler_apc]

Hi JC,

I'll attach a document called "basics of grounding systems" which might help.
Grounding best practices for a data center are all over the map.

Your customer is talking about "ground loops" which is a common problem when grounding type power adapters are used, or there is a a nuetral to ground bond at or through the computing/network equipment.

You want to avoid this, and provide a clear current path from any equipment discharging onto ground.

1. Ensure your electrical panel ground is a true earth bond.
2. Design your data center power distribution so nuetrals and grounds run isolated back to the electrical panel from electrical outlet receptacles.
3. Connect frame grounds between racks.
4. Sink a new 10ft earth ground rod outside, take that to a copper bus bar mounted on the wall in the computer room, Tie one point of rack frames back to that earth ground.

This helps data communication connections between racks and service demarcs, which typically use the frame ground of shielded cable, by isolating currents from power ground currents wherever possible.

As far as code compliant judgement on your bid spec, and consulting on the copper pipes feasability of being used for ground reference, I'd have to do more research to give you a definitive answer.

Based on your description of their ground junctions on those pipes...., I'd guess the resistance change is not a function of a solder junction, but a symptom of a heat intolerance elsewhere in the current path. I wouldn't use pipes as ground. I'd check local code on the EMT conduit fittings to see if non metallic is acceptable..

[cmarler_apc]

Hi JC,

I'll attach a document called "basics of grounding systems" which might help.
Grounding best practices for a data center are all over the map.

Your customer is talking about "ground loops" which is a common problem when grounding type power adapters are used, or there is a a nuetral to ground bond at or through the computing/network equipment.

You want to avoid this, and provide a clear current path from any equipment discharging onto ground.

1. Ensure your electrical panel ground is a true earth bond.
2. Design your data center power distribution so nuetrals and grounds run isolated back to the electrical panel from electrical outlet receptacles.
3. Connect frame grounds between racks.
4. Sink a new 10ft earth ground rod outside, take that to a copper bus bar mounted on the wall in the computer room, Tie one point of rack frames back to that earth ground.

This helps data communication connections between racks and service demarcs, which typically use the frame ground of shielded cable, by isolating currents from power ground currents wherever possible.

As far as code compliant judgement on your bid spec, and consulting on the copper pipes feasability of being used for ground reference, I'd have to do more research to give you a definitive answer.

Based on your description of their ground junctions on those pipes though, I'd guess the resistance change is not a function of a solder junction, but a symptom of a heat intolerance elsewhere in the current path. I wouldn't use pipes as ground. I'd check local code on the EMT conduit fittings to see if non metallic is acceptable..

[BillP]

Your customer is talking about "ground loops" which is a common problem when grounding type power adapters are used, or there is a a nuetral to ground bond at or through the computing/network equipment.

You want to avoid this, and provide a clear current path from any equipment discharging onto ground.

It looks like a well written spec. You might want to add specific grounding instructions for the racks. I recommend a ground halo of 10/7 copper stranded elevated above the racks perimeter, as they do in communications switchgear offices. Connect that halo to earth like you described to the CPC from one spot, tail a frame ground down to each rack from the halo (and cut all those connections to copper pipes)

1. Ensure your electrical panel ground is a true earth bond.
2. Design your data center power distribution so nuetrals and grounds run isolated back to the electrical panel from electrical outlet receptacles.
3. Connect frame grounds between racks.
4. Sink a new 10ft earth ground rod outside, take that to a copper bus bar mounted on the wall in the computer room, Tie one point of rack frames back to that earth ground.

This helps data communication connections between racks and service demarcs, which typically use the frame ground of shielded cable, by isolating currents from power ground currents wherever possible.

As far as code compliant judgement on your bid spec, and consulting on the copper pipes feasability of being used for ground reference, I'd have to do more research to give you a definitive answer.

Based on your description of their ground junctions on those pipes though, I'd guess the resistance change is not a function of a solder junction, but a symptom of a heat intolerance elsewhere in the current path. I wouldn't use pipes as ground. I'd check local code on the EMT conduit fittings to see if non metallic is acceptable..

[cmarler_apc]

Raised floor is typically bonded to building steel and only necessary with grounded floor tiles for EMC compliance directives. There is no NEC requirement for it, and no signalling noise interference reason for it.

Your external ground rod needs to tie to building power grounding electrode. From the bonding point it should also go to the other ground references in the building, including the racks. Look at the building service and panel nuetral to bond connections.

[BillP]

here is an interesting read on the subject

[BillP]

isolate the halo from building steel and run oto it's own grounding electrode system outside which is then bonded to the building power grounding electrode system as the NEC requires, as is the building steel. Isolate all the neutrals from grounds in the power delivery path.



I found an interesting article by Michael Johnston, an expert in the field of electrical inspectors.

It goes into length to illustrate the NEC's take on your customer's environment, you can make a good case for improvements.



Below is an excerpt from the article.
There are two conductors of a grounded system? the grounded conductor and the equipment grounding conductor?that should be discussed, and a brief story related about each.



The grounded conductor (usually a neutral) of a system has been grounded once, at the service or at the source of a separately derived system. The term "grounded" is past tense, which means that the action has already happened. The grounding of the grounded (neutral) conductor of a system is accomplished by a connection to ground through a grounding electrode conductor either at the service or at a separately derived system.



The other conductor to look at is the equipment grounding conductor. The word "grounding" is present tense, which means the action is ongoing. In equipment grounding conductors, the action is ongoing through every electrical enclosure all the way to the last outlet on the branch circuit.

The equipment grounding conductor puts all metal enclosures at earth potential along the way, and also provides a low impedance path for fault current to flow on if a ground fault should occur in the system. So it is important that the equipment grounding conductor of the circuit make a complete and reliable circuit back to the source. At the source or service is where the grounded (neutral) conductor and the equipment grounding conductor are required to be connected together through a main bonding jumper.

The main bonding jumper is defined in the Code as the connection between the grounded conductor and the equipment grounding conductor at the service. In a separately derived system, this connection is made with a bonding jumper installed between the grounded conductor and the equipment grounding conductor. These bonding jumpers complete the fault current circuit back to the source.



The NEC, in recent cycles, has been revised to continue its migration away from the use of the grounded conductor downstream of the main bonding jumper in a service or downstream of the bonding connection at a separately derived system for grounding equipment. The reasons are elementary as stated earlier. Current, be it normal current or fault current, will take all the paths available to it to try to seek out its source.



If the grounded conductor (neutral) and equipment grounding conductors are connected at points downstream of the service or separately derived system connections, such as at sub panels, there will be multiple paths available for current to try to return to the source.
This can lead to current flowing on water piping systems, conduit, equipment grounding conductors, and any other electrically conductive path.



In the 1999 NEC, there was a revision to the rules covering the use of the grounded conductor for grounding purposes at a second building or structure.
Section 250-32 (b)(1) requires that if an equipment grounding conductor is installed with the feeder supplying the second building or structure, that isolation between the grounded (neutral) conductor is to be maintained.

There is an allowance in Section 250-32(b)(2) to utilize the grounded conductor of the feeder for grounding equipment under three specific and very restrictive conditions. First, an equipment grounding conductor is not included with the feeder supplying the building or structure. Second, there are no continuous metallic paths bonded to the grounding system in both buildings. Third, there is no ground-fault protection of equipment installed at the service. If all of these conditions are complied with, the grounded conductor must be used for grounding and be connected to the building or structure disconnecting means. The grounded conductor is also required to be connected to a grounding electrode at the building or structure and installed in accordance with Part C of Article 250. This will serve as the grounding means and as the path for normal current and also the path for fault current to clear overcurrent devices.



In Section 250-32(b)(2) the Code mentions a requirement of having no continuous metallic paths bonded to the grounding system in each structure. This is encompassing of all paths, not just wires or conduits. These paths could include items such as metal water pipes, other metal piping, steel members, and paths such as the shielding on a communications cable or a coaxial cable installed between the structures. It is important to remember that current will take all the paths to seek out the source. If this connection were made and there was a ground-fault protection device at the service in accordance with Section 230-95, these connections could desensitize the GFP device and it may not operate properly when called upon to do so in ground-fault conditions because of multiple paths for current.



In summary, it is important that the basic elements of current flow be understood and thought of carefully while applying the rules of the NEC.

Section 250-24(a)(5) states that a grounding connection to any grounded circuit conductor on the load side of the service disconnecting means shall not be made, unless otherwise permitted in the article. The FPN gives reference to three situations where this is acceptable, but is restrictive. Sections as reviewed in this writing are for separately derived systems in Section 250-30(b), for separate buildings or structures in Section 250-32, and for grounding equipment under the limitations of Section 250-142.



Installers and inspectors should be watchful to ensure there are no neutral to ground connections on the load side of the grounding connections at the service disconnecting means or on the load side of the grounding connections for a separately derived system. In other words, isolate the neutrals and equipment grounding conductor connections. Give current (be it fault current or normal current) the low impedance path anticipated by the requirements of the NEC.




....to read the full article with illustrations.

[BillP · Accepted solution]

I have a data center application where my client is asking me several questions about grounding (they state they have circulating ground currents and want an opinion about correct grounding methods).

They have a data center building fed by several 2400V-208V D-Y solidly grounded transformers (single-ended unit substations). The unit subs feed panelboards which feed CPCs (which have 208-208Y/120V k-factor shielded isolation transformers and 4 wire panelboards). The CPCs then go on to feed PDUs and racks (mostly single phase loads). There is a raised floor.

This is an old building so the client is unable to verify existing grounding. They have a copper pipe system underneath the floor (with soldered - yes soldered connections), which they use as a computer ground. They think the copper pipe system then has a conductor running out to a single test well where it bonds to the building's grounding electrode system (one spot only for bonding these two systems). They measure 50-60 amps flowing on the ground wires. There are probably ground connections all over the place inadvertently placed over the years.

They have a small amount of harmonics, as well as a small amount of voltage imbalance (due to strong load imbalance at the PDUs and CPCs), so I expect them to see triplens on the neutrals. But they still question the correctness of their grounding system.

I find lots of guides and NEC info that don't necessarily give me the answers I'm looking for.

Seems to me this system should be grounded/connected as follows:

1) Unit sub's neutral bonded to the grounding electrode system and frame of unit sub per NEC separately derived system (they aren't services).

2) Building steel and probably raised floor steel structure all bonded as part of this electrode system.

3) Equipment grounding conductor and neutral run to panelboard from unit sub, the ground bonds the panelboard frame and ground bar, the neutral runs to the neutral bus (isolated from frame and ground in panelboard).

4) Equipment grounding conductor runs to CPC from the panelboard (no neutral, since the CPC has a delta primary transformer).

5) CPC transformer neutral, electrostatic shield, and frame are all bonded together and connect to incoming equipment grounding conductor as well as building grounding electrode system (building steel and/or raised floor).

6) CPC panelboards have neutral and equipment grounding busses (with the neutral insulated from frame and ground bus) - with conductors from both of those busses to the transformer neutral connection point described above in 5.

7) CPC panelboard also has an isolated ground bus (or computer ground bus) with conductor running down to the copper pipe system under floor.

8) Copper pipe is grounded to building steel in multiple places to keep impedance between computer ground system and building ground system as low as possible. (SO BASICALLY ALL COMPUTER GROUNDS, SENSITIVE ELECTRONIC GROUNDS ARE KEPT ISOLATED ALL THE WAY TO THE ISOLATED GROUND BUS IN THE CPC, BUT THEN BONDED IN ONE LOCATION TO THE PIPE SYSTEM WHICH IS BONDED TO BUILDING STEEL AND THE BUILDING ELECTRODE SYSTEM)P

QUESTION: Does what I described sound like a proper and NEC-compliant description for a data center? What else should we be doing or what should be done differently?

QUESTION: Should the pipe (computer ground system) be bonded to the building steel in multiple places to keep the impedance low or should it be isolated from the building ground system as much as possible and eventually bonded together (as NEC requires) at only one location? (As my client believes it should? - and as is the old practice for grounding computer/DCS systems)?

QUESTION: Is there any reason for the feeder conduit coming to the CPC to have a non-metallic fitting?

QUESTION: The client expressed concern that the soldered copper pipe connections become high resistances when they get warm? This doesn?t make much sense to me. I know resistance is a function of heat, but solder itself doesn?t become a bad conductor when it?s warm, does it? Is this a data center thing, that maybe I?m missing? I want to suggest to my client that they just install copper jumpers around the joints that they think are bad connections?

Any assistance is greatly appreciated!