Lessons Learned: Metro-Ethernet/ Fiber Optics

Metro-Ethernet

There is a lot to like about metro-Ethernet offerings. It’s fast, it’s Layer 2, and comparably it is significantly cheaper than its DS-x counterparts for similar speed offerings with less complexity.  The caveats are lighting the fiber and any construction costs associated with building into your facility. If you are too far from access to the provider’s fiber ring, construction could be a deal breaker.

In the Metro-E space the most utilized offerings include a private solution (managed) and a standard offering that is fiber pairs handed off at the demarc. The key difference here is that in a private model the service provider will install switches in your premise and “light” the fiber from end to end so that they can manage and monitor it. You simply cross-connect into their switch. While this is a very nice option it comes at a steep price. The alternative via a standard type of offering is where the provider simply hands off pairs of fiber to your facility, they light the fiber to you, and you are responsible to light it back to them (Tx/Rx). The catch here is that you need to have your own optics capable of lighting the fiber possibly over a great distance so the required optics will be expensive (ZX).

In either case redundancies will be costly as ideally in a dark/lit fiber scenario you would employ multiple paths to the provider ring through at least two entrances to your facility, literally. Construction costs double as do MRCs but if you absolutely cannot afford for your internet or site-to-site connection to be jeopardized, at any cost, then this is the solution for you. Leading data center providers (Savvis, Rack Space, etc) all have multiple data providers each with multiple entrances to their facilities.

Fiber type and connecting modules are extremely important, pay attention to these details. It may seem obvious but when you’re buying, connecting, and configuring three dozen other pieces of infrastructure at the same time these details can be overlooked. The Metro-E provider will be using single-mode fiber cabling as is the standard for shooting light great distances. Your optics MUST match the fiber type. So if you will need to buy ZX modules for your own Cisco switches be sure to get the ones that match the handed off fiber type. These modules come in both single-mode and multi-mode flavors. Your cross-connect cables must match as well.

Whether standard or private, your service provider will demarc the connection in a location of your choosing, most likely a cabinet in your datacenter. The installation should include a fiber distribution box that they will install and terminate. Your provider will most likely hand off SC connectors on the other side of the distribution box. Your Cisco ZX module will require LC connectors so you will need to acquire a cable with different ends. Having just gone through this, my provider handed off duplexed SC connections so I had to remove the clips on the SC side of the cable to split the SC connections.

Questions to ask:

• What are the differences between a private managed service and a standard service?
• Do you need a point-to-point connection between 2 or more locations, or just internet access?
• What is the type of fiber being handed-off by your provider? Single-mode (SMF)? multi-mode (MMF)?
• Which standard SX? LX? ZX?
• What are the service SLAs?

Fiber Cabling

There are 2 primary types of fiber optic cabling in play in today's datacenters: single-mode (SMF) and multi-mode (MMF). SMF is capable of transmitting almost infinite bandwidth over hundreds of kilometers so is used mainly in backbone fiber rings. MMF can transmit light a shorter distance so is found in campus location use. SMF is also very expensive and less flexible than MMF. Of these 2 types there are a few different options. MMF is found in 50/125µm or 62.5/125µm and SMF is found in 8.3/125µm. These numbers, measured in microns (µm), indicate the diameter of the glass core that carries the light, and the outer cladding which confines the light to the core. 50 micron is the most common flavor of multi-mode fiber and is commonly used for both Ethernet and fiber-channel.

62.5 micron is the older standard, 50 micron is the newer. You can read more about both in the linked documents in the reference section below. A third specification groups the different transmission quality of the fiber types.  The performance codes for MMF are OM1, 2, and 3, for SMF they are OS1 and 2. These codes indicate the bandwidth capabilities for different light wavelengths. See the attachment “Understanding OM1, OM2…” below. For example, I just installed 6 strands of 50 micron OM3 MMF between my datacenter and user floor to run 3 x 10GigE links.

Just like CATx cabling, fiber cabling can be purchased from many “big name” companies like CablesToGo, or Belkin at steep premiums. I strongly recommend against buying from these companies as better quality can be found by smaller e-tailers that make and certify their own cables. My favorite vendors right now are fibercablesdirect.com and monoprice.com.

References:

Cabling and Test Considerations for 10Gige LANs [via fiberoptic.com]

Understanding OM1,OM2,OM3,OS1,OS2 Fiber for Data Centers [via Cisco Learning Network]

Multimode Fiber: 50 micron vs 62.5 micron [via netoptics.com]