ESX 3.x and VC 2.x Introduction Purpose of this guide This guide is designed for people who already know ESX 3.x and VC 2.x quite well. Although it starts as a beginners guide initially, it pretty rapidly starts to assume very good knowledge of the system. I would recommend you get to grips with the GUI first, and feel comfortable with Vi-3 before attempting this guide. That said – you might be wanting to carry out a discrete procedure from the command-line. So feel free to dip in – find what your looking for and then – dip out again! _ It is not a comprehensive guide to ALL the commands – just the primary ones. I hope to make this guide gradually more comprehensive, and cover all new commands that useful. I’ve deliberately not covered every single esxcfg command – because not all of them are terrifically useful… There are some big topics that I have yet to add to this guide – this includes setting NTP and Active Directory authentication for the Service Console Please email at the email address at the beginning of this document if spot any errors Where possible – I don’t use the VI Client. I only use the VI client if there is no other way – even if the VI client is easier. The reasoning behind this force the use of the command-line. Perhaps I should state why using the command-line might be useful… • Automation of tasks? • Er, because everything else is broken – and login in at the ESX host at the Service Console is your only option • Because sometimes its quicker (sometimes it’s quicker using the mouse too…!) • Because like Everest, it is there! But I prefer the mouse? • Then this guide isn’t for you… Choose File, and close… _ But seriously, what are commands – only words that carry out instructions Who in Windows hasn’t used the net use or net servicename stop/start command? I grew up on DOS/Windows3.x/Novell – a mix of GUI and commands. For which I am very grateful. I am happy in both environments. Although I do find myself swearing when commands refuse to work, but that said I also swear at Windows just as much – perhaps like Pete from Big Brother have “Tourettes Syndrome”. God knows when ever I watch Big Brother I feel uncontrollable the urge/need/desire to shout “wankers” at the top of my voice… Hopefully this guide will reduce the amount of swearing you do at computers. You must remember – they can’t hear or understand your profanities – and talking to inanimate objects is usually the first stage of madness… • • • Anyway, I digress – I’ve noticed as the GUI grew to dominate our environments – my command-line skills took a hit. If you don’t use it, you loose it – as the saying goes. I don’t intend my skills to be eroded by a little mouse. Of course they could be equally eroded by the keyboard. Hardware • I use a 192.168.2.x range on my network with 192.168.2.101 for esx1, 192.168.2.102 for esx2. My DNS server is 192.168.2.200. All IP’s here are just example from my network range – replace with your own • This guide assumes you have at least 2 servers with 2CPU’s at 1.4mgz each • 2GB of RAM • 2×36GB Hard Drives (SCSI Internal) • 4x Network Cards (production in a bond, vmotion and Service Console eth0) • This is the specification of my server an old Dell 1650 PowerEdge using PIII processors! • My hardware isn’t officially supported by VMware anymore. Anyway it still runs. But at some point I am going to buy two DL380, and re-use this hardware as my VC box and a NAS box… • I have VC set-up with a SQL Database. The layout of my VC looks like this before the upgrade: • Page 1 of 17 ESX 3.x and VC 2.x Two Servers, Two Virtual Machine Groups (for me and my mate, Trevor) The domain name is rtfmed.co.uk lavericm-admin – is the VC Administrator set at Server Farms instructor – is a Virtual Machine Admin from the RTFM Farm (used when I teach with my hardware) baverstockt – is a Virtual Machine User with rights only for Trevor’s VMs Group Software • In my case VC Server 2.x and SQL 2000 (SP3) all runs on Windows 2003 with Service Pack 1. This was the most current release of Windows at the time of writing this document • Warning: As ever my documents are released as is and without warranty Hardware & Software As we at the Service Console – you don’t really need to have exactly what I have but it would help to have what I have or better Conventions in this Guide I use PuTTy to get Service Console sessions I use nano rather than vi to edit and save files – mainly because I’m no Linux guru, and I find nano more user friendly. Vi text editor is popular in the Linux community because it pretty much standard amongst every Linux distribution. Don’t let anyone make you feel an idiot or small because you use nano. As long as you can successful manage the system that’s all that counts in my book. Linux people, do NOT flame me on the merits of using VI. I will absolute NOT respond! _ • Any Major titles marked with Red indicates this section is broken, released as is, and I intend to return to fix it. But got bored banging my head against a brick wall. _ • IMPORTANT: To have your command-line changes reflected in the VI Client you must restart the hostd service on the ESX host. You can do this by using service mgmt-vmware restart Change Log from 1.1 to 1.2 • None Module 1: Users and Rights Note: • In most case your rights will be governed by VirtualCenter and Active Directory • However, at the Service Console the account database that’s used is local to the Service Console • After an upgrade of ESX from 2.x to 3.x – root does have rights to connect remotely using SSH. • After a clean installation – the default is that root has no remote access via SSH. This to enforce traceability. User must logon as underprivileged users and elevate the rights to root. This change in rights is logged in /var/log/messages • The easiest way to create a normal user for use with PuTTy would be the VI Client. You can point it at the ESX host and use the “User & Groups” tab to create a new user Page 2 of 17 ESX 3.x and VC 2.x Creating a New User 1. Logon at the Service Console as ROOT (physical or ILO) 2. Type:| useradd lavericm-admin –p ******** Elevating yourself to ROOT 1. Logon at the Service Console as your user account 2. Type su Note: This Switches User and assumes root, unless otherwise specified. The – takes roots environmental settings (very important if you want to run any commands properly) 3. Type the password for root Note: You can also use a tool called sudo. This allows you to logon as an underprivileged user, and run commands with the ID of a privileged user Disabling Auditing on ROOT (Not Recommended) Note: • Some applications do not support levitation to a higher plain – for example WinSCP. Sure you could use WinSCP to gain access as an ordinary user, but then you might lack permission to copy the files you need. If you try to logon as root, WinSCP will give you access denied. • If you wish to disable the restriction on ROOT not being allowed direct access using SSH then carry out the following task. I wouldn’t recommend doing this as you will loose enforcement of your audit trail. 1. nano –w /etc/ssh/sshd_config 2. Locate: PermitRootLogin no 3. Place a # in front of PermitRootLogin no like so: #PermitRootLogin no 4. Exit Nano & Save the file 5. Restart sshd with service sshd restart Miscellaneous User Managements Tasks • • To change your password: passwd root List connected users w Note: This gives a result like so: Module 2: Getting around & Getting help Note: • • • • This module is for novices/newbies and people who have never used the Service Console before (sometimes referred to in the community as vmnix or the COS) The modules are NOT prescriptive – so if you would like skip and move to something more interesting please feel free to do so This section covers all the basics of using SSH and puTTy Oh, it occurs to me you might not know what these are… SSH is secure protocol (22) which allows you connect remote to an ESX host – giving a command-line interface. PuTTy is a free Page 3 of 17 ESX 3.x and VC 2.x SSH Client which is very popular in the community. This saves you using an ILO to get to the command-line environment The command-line environment is actually something called the BASH (Bourne Again Shell) • Module 3: File & Folder Management Page 4 of 17 ESX 3.x and VC 2.x Page 5 of 17 ESX 3.x and VC 2.x Module 4: Networking Note: • Networking involves the use of command esxcfg-vswitch, esxcfg-vswif0 and esxcfg-vmknic. These are quite involved commands – used in a particular order to achieve the results your are looking for. So I’ve decide to write more of a step-by-step guide than just a command-list with a brief explanation • esxcfg-vswitch is the main command – and it has a mix of parameter in lower and upper-case. Lower-case parameter manipulate the switch, where as upper-case switches manipulate the portgroup. So to add a switch its –a and add a upper-case –A adds a portgroup • In one way this is nice… but it’s incredibly easy to create a switch rather than portgroup attached to a switch… and I have done this a few times… _ Viewing your Switches & Service Console Networking 1. To view your switches type the command: esxcfg-vswitch –l Note: This shows me I have one vSwitch (vSwitch0) using one NIC (vmnic0) with one portgroup called “Service Console” which is not using VLAN. 2. To view your Service Console network settings esxcfg-vswif –l Note: Nothing to state here – but I think its interesting that it doesn’t show me my all important default gateway settings which would have been nice 3. To View your network card’s vmnic, pci (b:s:f), driver, link, speed, duplex and description: esxcfg-nics –l Creating a vSwitch (Internal) 1. To create a new switch type: esxcfg-vswitch –a vSwitch1 2. Then add a portgroup esxcfg-vswitch –A internal vSwitch1 Note: Lower-case -a for adding a switch, and upper-case -A for adding a portgroup 3. If you run the command esxcfg-vswitch –l you will see this information Page 6 of 17 ESX 3.x and VC 2.x 4. If you wish to see this reflected in the VI Client then type: service mgmt-vmware restart Note: This is how my switch configuration looks now Creating a vSwitch (Single NIC) 1. In this case we will create a new switch and port group in one line with: esxcfg-vswitch –a vSwitch2 –A production 2. Next patch a NIC to the vSwitch with esxcfg-vswitch -L vmnic1 vSwitch2 Note: Again this is a case-sensitive option. –l lists switches, whereas –L links nic’s to switches 3. If you run the command esxcfg-vswitch –l you will see this information 4. If you wish to see this reflected in the VI Client then type: service mgmt-vmware restart Page 7 of 17 ESX 3.x and VC 2.x Note: This is how my switch configuration looks now Creating a vSwitch (Multiple NIC’s) Note: It is very easy to create a NIC-Team just re-run the previous command, with a different NIC with 1. esxcfg-vswitch -L vmnic2 vSwtch1 2. If you run the command esxcfg-vswitch –l you will see this information 3. If you wish to see this reflected in the VI Client then type: service mgmt-vmware restart Note: This is how my switch configuration looks now Page 8 of 17 ESX 3.x and VC 2.x Deleting a Switch from vSwitch Note: I’m now running out NICS for the next part of this guide. So I am going to blow away my vSwitch2 to free up my NIC’s… 1. Type the command esxcfg-vswitch –d vswitch2 Note: Notice how it doesn’t ask – are you sure, but then again neither does the VI client. Creating PortGroups for VLAN Networking NIC Team 1. We have to create a switch, a portgroup for each VLAN, allocate our NIC’s and then set the VLAN ID esxcfg-vswitch –a vSwitch2 esxcfg-vswitch –A accounts vSwitch2 esxcfg-vswitch –A rnd vSwitch2 esxcfg-vswitch –A sales vSwitch2 esxcfg-vswitch –L vmnic1 vSwitch2 esxcfg-vswitch –L vmnic2 vSwitch2 2. The next part is to set the VLAN id for each network (account, rnd and sales) esxcfg-vswitch –v 10 –p accounts vSwitch2 esxcfg-vswitch –v 20 –p rnd vSwitch2 esxcfg-vswitch –v 30 –p sales vSwitch2 Page 9 of 17 ESX 3.x and VC 2.x 3. If you run the command esxcfg-vswitch –l you will see this information 4. If you wish to see this reflected in the VI Client then type: service mgmt-vmware restart Note: This is how my switch configuration looks now Creating Vmkernel Switches • • • • Note: This does seem a bit limited… I can’t see away of enabling the VMKernel switch for Vmotion Nor can I see how I would set the default gateway for the switch either You need a good name for the portgroup – because the esxcfg-vswitch command just shows as any ordinary switch… You need to use esxcfg-vmknic –l to see it • 1. First create the Switch, Portgroup and Assign a NIC esxcfg-vswitch –a vSwitch3 esxcfg-vswitch –A “VM Kernel” vSwitch3 esxcfg-vswitch –L vmnic3 vSwitch3 2. Next use the esxcfg-vmknic command to add in a VM Kernel NIC and set the IP and Subnet Mask esxcfg-vmknic –a “VM Kernel” –i 192.168.2.202 –n 255.255.255.0 3. Then set the vmkernel default gateway with esxcfg-route 192.168.2.1 4. If you run the command esxcfg-vswitch –l you will see this information Page 10 of 17 ESX 3.x and VC 2.x If you wish to see this reflected in the VI Client then type: service mgmt-vmware restart Note: This is how my switch configuration looks now Page 11 of 17 ESX 3.x and VC 2.x Changing your Service Console IP Settings Note: • You can do this through a putty session but clearly unless you have more than one vswif interface (with a different IP address) you will get disconnected • It is entirely possible – to have two vswif interface on two separate IP address – and use one connection to change the IP address of the other • You do run the risk of losing SSH connectivity if you screw up – so perhaps doing this through your ILO is a safer-bet and less hard work • You might also wish to include a new DNS entry for your ESX host before you make this change… 1. To view your current IP and Netmask type: esxcfg-vswif –l 2. To change your IP and Subnet Mask type: esxcfg-vswif –i 192.168.2.203 –n 255.255.255.0 vswf0 Note: You change your default gateway by editing nano –w /etc/sysconfig/network and then restart your networking with services network restart Your DNS settings are located in nano –w /etc/resolv.conf Setting the Speed and Duplex of NIC’s Note: Page 12 of 17 ESX 3.x and VC 2.x • • We have always been able to change the speed & duplex of the vmkernel NIC’s (ESX 2.x MUI) but in the past the only way to change the Service Console speed and duplex was by editing the /etc/modules.conf file You can now change both the Service Console and the vmkernel nics through the GUI There could be a chicken-egg/catch22 situation here though. If your Service Console NIC misneg’s its speed/duplex you maybe unable to connect with the VI client to change it. That’s when knowing the Service Console commands come in handy. _ • This can also be useful to reassign a 100Mps card to a Service Console and then fix its speed/duplex 1. Workout which NIC has been assigned to the portgroup “Service Console”with esxcfg-vswitch –l • 2. To view you current speed & duplex: esxcfg-nics –l 3. To set the speed/duplex of vmnic0 to 100Mps/Half-Duplex type: esxcfg-nics -s 100 -d half vmnic0 4. To reset to auto-neg esxcfg-nics -a vmnic0 Recreating your vswif0 Interface Note: OK, suppose something horrible goes wrong and you loose your vswif0 interface – this is how you would recreate it… 1. Logon locally to the ESX host or use your ILO card (do you have choice!) 2. Create a new switch esxcfg-vswitch -a vSwitch0 3. Create a new portgroup esxcfg-vswitch -p “Service Console” vSwitch0 4. Assign a NIC esxcfg-vswitch –L vmnic0 vSwitch0 5. Assign a vswif interface and set its ip/sn: esxcfg-vswif –a vswif0 -p “Service Console” i 192.168.2.102 –n 255.255.255.0 Removing a NIC from vSwitch • • esxcfg-vswitch –U vmnic vSwitch2 esxcfg-vswitch –D production vSwitch Deleting a PortGroup from vSwitch Managing the ESX Firewall Note: • Managing the firewall by the VI Client is real easy – it’s a tick of box style interface. If you use the GUI interface and then query with the commandline tool you get “friendly” information about what enabled. If you purely use the command-line tool you will just get TCP port numbers and directions (outgoing and incoming • They GUI tool also has some handy “built-in” friendly name for popular applications vendor specific agents like CommVault Dynamic/Static and many others Page 13 of 17 ESX 3.x and VC 2.x • By default 902 (VirtualCenter, uses TCP and UDP!), 80 (Web Access welcome page), 443 (Web Access Login page) and 22 (SSH) are enabled by default In additions there are some ports enabled for EMC AAM (Automated Availability Manager) and CIM (Common Information Model) developed by the DMTF, is a very broad approach to the management of systems and networks Other Ports worth knowing about: • • A bit like in ESX 2.x it is possible to set 3-level of security (high, medium and low) o High incoming/outgoing blocked o Medium incoming blocked, outgoing not blocked o Low firewall off, no blocking of incoming/outgoing traffic Viewing your Firewall Settings 1. Type the command esxcfg-firewall –q outgoing esxcfg-firewall –q incoming Note: You can also use esxcfg-firewall –q on its own. This gives you lots of stuff… most useful at the bottom: Changing Your Security Level Note: If you wanted to weaken your security to medium you could use 1. Type the command esxcfg-firewall –allowOutgoing –blockIncoming Note: You should get warnings like so: • Enabling a Single Service/Client/Agent Note: • If you want to SSH from an ESX host to another ESX host or SCP from an ESX host to another – you need to enable the SSH Client on port 22 • If you do this via the GUI and then do a esxcfg-firewall –q you will see friendly information like so: Page 14 of 17 ESX 3.x and VC 2.x Incoming and outgoing ports blocked by default. To do the same from the command-line you would type the command: esxcfg-firewall -e sshClient Note: To disable esxcfg-firewall -d sshClient Enabling non-Standard Ports Note: • Perhaps there is an application or service which is not listed for use with – e or –d • Or the port numbers have been changed to non-standard ports • It is possible to open a specific port by number, transport (udp/tcp) and direction (in/out) • To enable port 22 outbound from the server type the command: esxcfg-firewall -o 22,tcp,out,ssh Note: The ssh at the end is a friendly label. If I run esxcfg-firewall –q again at the bottom it states: Fixing ESX Networking after an Upgrade Note: • • If you have read my upgrade guide to ESX 3.x you will know that networking looks significantly different after an upgrade – rather than a clean install The following shows how to “clean up” networking using the Service Console commands Page 15 of 17 ESX 3.x and VC 2.x • This is how my upgraded ESX server networking looks like So what am I going to do with this? • What I am going to do is the following: o Remove the VMotion Switch0 altogether thus freeing up vmnic0 o Remove the “Legacy vmnet_0” portgroup from vSwitch1 o Remove the “Legacy vmnic2” portgroup from vSwitch2 o Remove vSwitch3 o Create a new vswif interface on a new switch called vSwitch0 using vmnic0 o Add vmnic1 to vSwitch2 o Patch all my VM’s to the Production on vSwitch2 o Create with a VM Kernel Port (vSwitch3) Removing VMotion Switch0 esxcfg-vswitch –d vSwitch0 Remove “Legacy vmnet_0” portgroup from vSwitch1 esxcfg-vswitch –D “Legacy vmnet_0” vSwitch1 Remove “Legacy vmnic2” portgroup from vSwitch2 esxcfg-vswitch -D “Legacy vmnic2″ vSwitch2 Page 16 of 17 ESX 3.x and VC 2.x Remove Old vswif Interface and Remove vSwitch3 • Disable vswif0 interface, and remove from the system You must disable the vswif interface before you delete the switch it is attached to… • Do this at the physical console via an ILO • esxcfg-vswif –d vswif0 • esxcfg-vswitch –d vSwitch3 Create a new vswif1 interface • esxcfg-vswitch -a vSwitch0 • esxcfg-vswitch -A “Service Console” vSwitch0 • esxcfg-vswitch –L vmnic0 vSwitch0 • • esxcfg-vswif –a vswif0 -p “Service Console” -i 192.168.2.x –n 255.255.255.0 Note: You will get this message Note: I found my new vswif0 interface was already enabled. But if doesn’t for you might like to try esxcfgvswif –d vswif0 Create a VM Kernel Port • esxcfg-vswitch -a vSwitch3 • esxcfg-vswitch -A “VM Kernel” vSwitch3 • esxcfg-vswitch -L vmnic3 vSwitch3 • esxcfg-vmknic -a “VM Kernel” -i 192.168.2.x -n 255.255.255.0 • esxcfg-route 192.168.2.1 Typical vSwitch Errors • Getting your case-sensitivity muddled up! o -a add vSwitch o -A add portgroup o -l list vSwitches o -L link vmnic o -d delete vSwitch o -D delete portgroup Trying to change things that are in use “Failed to remove vswitch: vSwitch3, Error: PortGroup “Legacy eth0″ on VirtualSwitch “vSwitch3″ is still in use: 1 active ports, vswif0” or “Legacy vmnic2, Error: Unable to delete portgroup “Legacy vmnic2″, for the following reasons: 1 active ports” • Page 17 of 17