UCS.txt

Baseboad Management Controller

POD

B-250: Full width, 1u, Cisco server, up to 2 Nehalem CPUs, 384GB memory w 8GB DIMMs.

• Monolithic, multiprocessor servers. To scale, add hardware to the server.
• Large failure domain (failure of a server impacts multiple apps and users)

• Commodity servers. Often one app per server. Usually x86.
• High power and cooling costs. Scale limited to one host. Usually apps don't scale "horizontally".

• Blade and other commodity hardware, things like GRID, VMware, etc.
• Implementation complicated. Limited scale.

• Software layer that abstracts physical hardware.
• Creates individual virtualized hardware for each VM.

• Mac address
• WWN
• Firmware version
• BIOS
• UUID

Server whose operating system and application personalities are not tied to the physical hardware. One way to make the server stateless is to use transportable, virtual, unique identifiers.

Server personality is the operating system configuration and application settings. This fully functional set of programs, files and settings is required to perform the given task of the server.

• Reduction in redundant equipment costs.
• Power and cooling savings.
• Shared network and storage access. May reduce cabling.
• Rapid hardware provisioning.
• Maximum use of physical space.

• Increased physical density may create power and cooling challenges
• Increased compute density could increase bandwidth and cabling requirements.
• Each chassis and local switching creates additional management points.

• Single core CPU - CPU is usually the bottleneck
• Without virtualization - multicore CPUs are often underutilized.
• With virtualization and multicore CPUs, memory is often bottleneck.

A virtualized set of hardware that operates like a physical server.

A virtual set of hardware (virtual machine) along with the OS, apps, and files associated with a "physical server".

A software layer that abstracts the physical hardware and creates the individual virtual hardware for the virtual machines (VM). I.e., VMware ESX, Hyper-V and Xen.

If a server fails, migrating to another server will require that unique identifiers like MAC address, WWN, BIOS and firmware settings be configured. This is a "stateful" server.

• Independent systems have to be managed including network, storage, host identity, etc.
• Often requires many different management and monitoring interfaces.

• Two Cisco UCS 6120/6140 Fabric Interconnects.
• - Manage up to 320 blades and 40 chassis
• - FCoE
• UCS 5108 blade serve chassis
• - 2 Cisco UCS 2104XP fabric extenders per chassis
• - Up to 8 blades per chassis



Yes.

• UCS Management and Switching - UCS 6100 series Fabric Interconnects
• UCS Blade Chassis - UCS 5100 blade chassis
• UCS Compute Nodes - Cisco UCS B series blade servers
• UCS Chassis Network Distribution - UCS 2100 series I/O modules
• UCS Server Network Distribution
• - Virtual Interface Card, Converged Network Adapter, 10Gb Ethernet Adapter

• CEE - Converged Enhanced Ethernet
• DCE - Date Center Ethernet
• They are the same thing.

• - Priority groups: virtualize links and allocates recources per traffic class.
• - Per priority flow control
• - End-to-end congestion management and notification
• - Shortest-path bridging: Layer 2 multipathing.

• - Lossless fabric: No drop storage links.
• - Provides tools for handling network congestion
• - Deterministic latency for HPC clusters
• - Enables a converged Ethernet fabric for reduced cost and complexity

In the case of FCoE and TCP/IP; seperate protocols operate as tunnels on the same physical media. Both FC and TCP/IP exist in seperate ethernet payloads.

Effectively, all ethernet fabrics today use a form of packet-loss flow control. Using IEEE 802.3X pause capability in standard ethernet, a lossless fabric can be configured. There is also a priority-based pause capability that can be used to allow multiple lossless fabrics to exist on a single physical infrastructure.

40% fewer connections and 66% fewer components.

• - Reduced # of NIC/HBA cards/ports.
• - Can be managed same as traditional FC.
• - Same FC model so easy for storage personnel to transition.
• - Remains stateless, don't have to map through stateful gateway. Performance benefits from this.
• - Power efficiency.
• - FC is bridged onto a lossless ethernet fabric.

A Network Bridge connects multiple network segments at the data link layer (Layer 2) of the OSI model. In Ethernet networks, the term Bridge formally means a device that behaves according to the IEEE 802.1D standard. A bridge and switch are very much alike; a switch being a bridge with numerous ports. Switch or Layer 2 switch is often used interchangeably with Bridge.

• 6U; Blade servers and power supplies install from the front, IOMs and fans from the rear.
• Up to 8 half-width blades, 4 full-width blades, 2 - full width, double height blades
• Physical divider for half height blades can be removed with no tools required.
• Max of 4 - 2500W hot-plug power supplies. 3 configs: non-redundant, N+1, N+N (grid). Single phase AC. They provide up to 550W per half-width server, 1100W per full width server. Power cabling in front (less chance of accident?)
• Chassis cools from front to rear. 8 hot-plug fan modules in rear with status lights per chassis.

IOM - I/O Module - extends the 10Gb CEE fabric from the fabric interconnects to the blade servers. No switching in the IOM. 2 - IOMs per chassis, technically only one required. 4 10Gb ports per IOM. If only one IOM installed, must be placed on the left bay. If both used (and cabled correctly?) they are hot-swappable and fully redundant from ethernet perspective. FC ??? Each blade server connects to both IOMs via mezzanine card. Supports up to 8:1 oversubscription.

CMC - chassis management controller. The CMC collects status data fromt he IOM using the IPMI (Intelligent Platform Management Interface) protocol over the integrated circuit serial bus. The CMC then communicates that information to the management node (UCS management server on the fabric interconnects?) by using the ethernet server link. The CMS controls fan speeds and power supplies, serves as a proxy for the UCS Manager to the blade servers for certain functions. Plays a part in the HA protocols.

• The midplane provides:
• - Redundant power to the IOM and blade servers.
• - Redundant data network (ethernet) connectivity between the IOM and blade servers.
• - Redundant 12C management paths to the blades
• - Dedicated management network (ethernet) connectivity
• - Supports autodiscovery of all components.

• - 20 (6120) and 40 (6140) port versions.
• - 1U and 2U high respectively.
• - 6120 has 20 SFP+ (small form-factor pluggable) ports for 10Gb ethernet and one expansion module 560Gb/s capacity
• - 6140 has 40 SFP + (small form-factor pluggable) ports for 10Gb ethernet and one expansion module 1120Gb/s capacity

• - FC only expansion module with 8 SFP ports that support 1,2,4Gb/s FC.
• - Combination expansion module: 4 SFP+ ports that support 10Gb/s ethernet, and 4 SFP ports that support 1,2,4Gb/s FC.
• - Ethernet only expansion module: 6 SFP+ ports that support 10Gb/s ethernet, no FC.

• B200-M1 half-width 12 DIMMs 1 Mezzanine Card 1or 2 Intel Xeon 5500 CPU
• B200-M2 half-width 12 DIMMs 1 Mezzanine Card 1 or 2 Intel Xeon 5600 CPU
• B250-M1 full-width 48 DIMMs 2 Mezzanine Cards 1 or 2 Intel Xeon 5500 CPU
• B250-M2 full-width 48 DIMMs 2 Mezzanine Cards 1 or 2 Intel Xeon 5600 CPU
• B440-M1 2u, full-width 32 DIMMs 2 Mezzanine Cards 2 or 4 Intel Xeon 7500 CPU
• 2 SAS hard drives for all models except B440-M1

BMC - Baseboard Management Controller on the UCS blades. The BMC uses the IPMI protocol over the 12C serial bus to manage devices on the baseboard. The BMC is responsible for providing remote keyboard, video, mouse (KVM) access to the end user.

Used on the B250. Custom ASICs virtualize 8 DIMMs per memory channel. With 3 memory channels per CPU, that adds up to: 8 DIMMS * 3 Channels * 2 CPUs = 48 DIMM slots.

• 73GB/15K SAS
• 146GB/10K SAS
• SDD soon.

• Cisco UCS VIC M81KR: Cisco virtual interface card (Palo?)
• Cisco UCS CNA M71KR-E: Emulex CNA
• Cisco UCS CNA M71KR-Q: QLogic CNA
• Cisco UCS 82598KR-CI: 10Gb ethernet adapter

• Supports multiple vNICs or vHBAs.
• 500,000 IOPs in both initiator and target mode.
• Supports kernel and hypervisro bypass.
• Dual 10Gb ethernet ports and dual FC ports to backplane.
• Provides failover between redundant links.
• No multipathing software is required on the host OS.

• 2 host side Gb ethernet ports and 2 FC ports to the backplane.
• Network ports can run either native ethernet or FCoE protocols
• Failover is performed by the FCoE asic Is this feature going to be deprecated?
• Both cards use Intel 10Gb ethernet chipsets.
• FCoE protocol encapsulation is offloaded from the host and peformed by the CNA.

• Made by Intel, only supports 10Gb ethernet.
• FCoE support must be handled by the OS on the CPU (not offloaded the the Mezzanine cards).

• Nexus 1000V bypasses the VMware vSwitch by leveraging the VMware vNetwork distributed virtual switch (DVS). Provides a single point of configuration for networking multiple ESX hosts. Also provides: policy-based connectivity for VMs, network security mobility, NDU software model.
• 

• VSM - Virtual Supervisor Module - a virtual or physical appliance runing Cisco OS. It performs management, monitoring and configuration tasks. Tight intergration with vCenter.
• VEM - Virtual Ethernet Module - Enables advanced networking capability on the hypervisor. Provides each VM witha virtual dedicated "switch port". The VEM sits on top of the VMware DVS (distributed virtual switch).

• - All administrative management activity occurs with the UCS Manager running on the fabric interconnect.
• - Management traffic flows from the fabric interconnect via a dedicated VLAN (4044) on the ethernet downlinks to the chassis management controller (CMC) on the IOMs.

• - Chassis related management tasks are passed via redundant serial links to the appropriate device.
• - IOM management tasks are passwd via integrated circuit (I2C) links to the appropriate IOM chipset.
• - Blade management tasks are passwd via dedicated redundant ethernet links from the BMC (Baseboard Management Controller).

• - UCS Manager runs on the fabric interconnects (UCS 6120 or 6140).
• - UCS Manager runs in the Nexus OS (NX-OS) on the fabric interconnects.
• - UCS Manager can be accessed via GUI, CLI or XML API.
• - UCS Manager provides KVM functionality to manage the blade servers.

CMC - Chassis Management Controller, collects the status of the IOM that it resides in using the IPMI (Intelligent Platform Management Interface) protocol over the I2C serial bus. The CMC communicates information back to the UCS Manager (management node) via ethernet server links. The CMC controls pwoer supply and fan speeds. It is also a proxy between the UCS Manager and the blade servers for some functionality. Also supports some UCS Manager HA functions.

The BMC communicates "lights-out" status information, KVM and other status information to the CMC.

• - Most existing system, LAN and SAN admin responsibilities remain unchanged.
• - RBAC allows current admins to manage appropriate domains with UCS Manager.

• - EHV - Ethernet host virtualizer mode
• - Traditional switch mode
• - For storage, the fabric interconnect operates in N_Port virtualization mode for SAN uplinks.
• This allows full compatibility with existing networks that contain switching infrastructure from other vendors.

When running is switch mode, the FIs are managed in the same manner as any other Layer 2 switch. The ethernet Network admin is responsible for LAN config including 802.1Q VLAN trunks and etherchannel aggregates.

When the FIs run in EHV mode, it does not participate in VLAN trunking protocol (VTP) or Spanning Tree Protocol (STP). The FIs are Layer 2 devices that do not participate in routing protocols. Uplink network configuration consists of link aggregation and trunk settings.

No.

• 1. Prevents it from participating in the FSPF (Fabric Short Path First) routing and domain schemes.
• 2. NPIV allows multiple N port IDs to share one physical N port.

No.

• - Coordination with the data center team.
• - Physical resource assignment
• - Server profile create
• - Creation and management of the UCS internal VSAN and VLAN objects.
• - BIOS and firmware management
• - UCS configuration backup

• - Physical and power status via LEDs
• - Logical status via UCS Manager GUI and CLI
• - Traditional troubleshooting techniques.



• - XML API (application programming interface)
• - Web-Based Enterprise Management (WBEM)
• - SNMP
• - IPMI
• - SDKs for full control of UCS are available

• - Create multitenant portals with your own presentation, authorization and monitoring mechanisms.
• - Integrating with custom or industry-standard orchestration tools (like HP Openview).
• - Populating CMDBs
• - Scripting and integrating with custom management solutions

The DME provides API services (all interfaces are ultimately translated into XML AP) and authentication servers. Custom scripting, CMDBs, etc. can be integrated using the XML API. Everything that si available in the user interface in the product is available through the XML API.

BladeLogic uses UCS features like policies and service profiles allowing administrators to manage and deploy servers, perform compliance operations ont he UCS platform. BladeLogic can be used by admins to perform day-to-day Cisco UCS system mangement without using UCS manager GUI or CLI. BladeLogic is a purpose built automation solution for UCS.

• - Can manage multiple UCS "fabrics" or "pods".
• - Can manage service profiles or "templates".
• - Can manage unique IDs like MAC addresses and WWNs for the entire server farm (multiple pods).
• BladeLogic uses custom objects to manage the Cisco UCS environment. Admins can provision and reprovision an entire technology stack.

• - Redundant fabric interconnects synchronize database and state data through dedicated, redundant ethernet links.
• - Architecture prevents split-brain scenarios.
• - Floating management IP addresses allows for seamless failover.
• - Management of redundant fabric interconnects occurs on active device only, changes are synchronized to standby.
• - Only management interface is active/standby; data traffic is active/active.

• - Recoveriong a full Cisco UCS Manager implementation after a disaster can use multiple backup types.
• - Backups can be automated to an external TFTP, FTP or SCP server.
• - Full UCS Manager database backups include physical hardware states such as service profile associations.
• - Configuration data can be backed up as XML.
• - XML-format backups do not include service profile associations. Can be edited before restoration.

• - All management occurs in the UCS Manager interfaces - no direct device or module management is necessary.
• - Network and system admin models are unchanged.
• - Cicso UCS Manager has many sources of diagnostic and status information to assist troubleshooting.
• - Broad management API support allows for 3rd party or custom integration.
• - Clustered fabric interconnects provide both management and network traffic HA.
• - Recovering the UCS environment after a disaster is possible using the UCS database and configuration backups.

• - Nonredundant - Only enough power to meet chassis requirements. Power supply failure results in chassis shutdown.
• - N+1 redundancy - Meets chassis requirements plus one power supply. No disruption if 1 power supply fails. All power supplies provide power and balance workload. Extra power supplies (N+2) are placed in standby mode.
• - Grid redundancy - Requires twice nonredundant configuration. Half supplies are wired to one power source, half to another power source.

• - Inbound power connections are not matrixed. Failure of an input causes loss of power to the connected power supply.
• - Outbound (to the blades, etc.) are matrixed. All power supplies pool power for all components.

• Out-of-band management is performed with these.
• 

• Two RJ45 ports for heartbeat. Use 802.3ad bond managed by NX-OS. Ports are configured LACP. Labeled L1 and L2. Must be cabled L1-L1 and L2-L2.
• 

No. Managed through the UCS Manager.

Blade dongle that includes a DB9 serial connection, USB connector and VGA connector.

No. Managed using UCS Manager.

• - Connect Layer 1 (L1) of Fabric Interconnect A to Layer 1 of Fabric Interconnect B.
• - Connect Layer 2 (L2) of Fabric Interconnect A to Layer 2 of Fabric Interconnect B.
• - Connect Fabric Interconnect A to IOM A of each chassis with 1 to 4 uplinks.
• - Connect Fabric Interconnect B to IOM B of each chassis with 1 to 4 uplinks.

Date Center Operating System on the fabric interconnects.

• Fabric A and B Data Center Operating System (DCOS) Networks. - these provide connectivity to the CMCs and management entities on the mezzanine cards.
• Fabric A and B Adapter Management (AM) Networks - provide connectivity between the DCOS instance in each FI and the management entities on the mezzanine cards for the purposes of allocating recources.
• Fabric A and B Adapter Management (AM) Infrastructure Networks - provide system wide connectivity between the AM instance in the UCS fabric interconnect, the fabric A and B CMCs, and the baseboard management controller (BMC) of each blade. UCS Manager bootstraps this process by passing out chassis numbers to each CMC. Other addresses are derived from that. The CMC and BMC interfaces are exposed as virtual interfaces on VLAN 4044.
• Fabric A and B Cisco UCS Manager PXE Networks - These isolated networks are used to boot host OS from images that are hosted by the UCS Manager. IP addresses are assigned by DHCP as part of the PXE process. VLAN 4047.

• VLANS: - configure globally; out-of-box VLAN is default (1); trunk all VLANs defined n profiles across uplinks.
• Port Channels - port channel uplinks created using UCS Manager; must match port channel config upward from Cisco UCS; Uses LACP 802.3ad/802.1ax.

For any VLAN to be supported on the blade servers, a VLAN object must be created in the GLOBAL UCS (both fabrics) configuration (under the LAN tab). VLANs associated with fabric A or fabric B can also be created, but not recommended.

Data Center Operating System

Adapter Manager networks (one each for fabric A and B). These networks provide connectivity between the DCOS instance of each FI (fabric interconnect) and the management entities on the mezzanine cards for the purposes of allocating recoures, defining identities, and monitoring adapter status. Layer 2 protocols are run by both Cisco UCS manager instances. Frames on this network use VLAN 4043.

• 4042: Connectivity between the Cisco UCS Manager and chassis management.
• 4043: Cisco UCS Manager to management entity on mezzanine adapters.
• 4044: Cisco UCS Manager to CMCs
• 4047: PXE boot for Cisco UCS Utility OS (UUOS)

As an example, port channel ID 1 can be created with uplink ports 19 and 20 on a Cisco UCS Fabric Interconnect. A matching configuration must be configured on the uplink switch. So the port channels are between the FIs and an external switch.

Can be configured globally or per fabric in UCS manager. Don't use default VSAN 1. Also have to configure a VLAN ID for each VSAN to carry FCoE traffic. Like any other VLAN/VSAN, must be a unique number. Probably best just to reserve a set of VLAN/VSAN IDs for storage.

• - Management of the 61xx FI's can be performed via serial console, out-of-band IP management, or Cisco UCS Manager GUI.
• - the UCS chassis and IOMs are managed by Cisco UCS Manager.
• - UCS 61xx FIs provides management channels for all Cisco UCS components and LAN/SAN data paths for the UCS server nodes.