For 3.5 follow these instructions.
You must be at the console of the server for this.
Hold down ALT and push F1.
Just type in the word ‘unsupported’, without the quotations of course, and hit enter.
You may see a warning about it being unsupported.
type in the root password that was set up when ESXi was installed and hit enter.
You will be at the prompt.
Type vi /etc/inetd.conf and hit enter. This will bring up the configuration file.
Look for where is shows #ssh and remove the # by placing the cursor in front of it and hitting the ‘i’ key. Then hit the backspace key.
Hit ESC and then type in :wq then hit enter.
That saves it and drops you back to the prompt.
Run this command: /sbin/services.sh restart
That should restart the managment services and start up SSH but not cause any issues with any currently running VMs.
Use whatever ssh client software to try and connect to it.

For 3.5u2 and above including 4.0 follow these instructions.
You must be at the console of the server for this.
Hold down ALT and push F1.
Just type in the word ‘unsupported’, without the quotations of course, and hit enter.
You may see a warning about it being unsupported.
type in the root password that was set up when ESXi was installed and hit enter.
You will be at the prompt.
Type vi /etc/inetd.conf and hit enter. This will bring up the configuration file.
Look for where is shows #ssh and remove the # by placing the cursor in front of it and hitting the ‘i’ key. Then hit the backspace key.
Hit ESC and then type in :wq then hit enter.
That saves it and drops you back to the prompt.
Type ‘ps aux|grep inetd’ and hit enter.
Make a note of the PID(number) that shows up for inetd.
Type in ‘kill -HUP PIDNumber’ with of course the PIDNumber being that number you saw from the previous command. This will restart the inetd process but should not cause any issues with the currently running VMs.
Use whatever ssh client software to try and connect to it. It should work just fine now.

Note: For copying files it only supports SCP, not SFTP. That is because it uses Dropbear for the SSH server instead of OpenSSH.

Below is a site I used for help on this:
http://chall32.blogspot.com/2009/10/vmware-esxi-ssh-sftp-access.html

It seems that I will be moving my VMware related VMs and all that to my IBM once I have things migrated to my Dell. I wanted ESXi to work on the Dell properly but the limits of SATA driver support and the underlying PITA it is to dig up logs to find out what exactly is happening has become more irritating than I am willing to deal with in a home environment.

I plan to back up the VMs and other data of course before I start. I will then make the Host OS Windows 2008 and maybe run a guest using the plain ol VMware Server 2.x to get a Linux shell server going. I would actually give Hyper-V a try but it requires a 64-bit processor and none of my server hardware has that. I know most, if not all, of my hardware should work with Windows.

I know some may think, why not try Linux. My IBM is currently running Linux as the host with VMware 1.09 on top of that with Windows 2003 as a guest for DC related things. Linux also runs as the file server using Samba authenticating off of the guest.

Overall using ESXi is not really a major problem if all your hardware is supported and you have the right license but of course I am using the free license and the server is no longer officially supported by VMware but still works. Once I am done, I will try to install ESXi on my IBM as it has almost identical hardware for the most part. The drives on the IBM are big enough for ESXi and the like but not big enough for any VMs so I will be trying to use an IDE drive or something like that to provide the non-fault tolerant space for the development and testbed VMs.

I think I have finally made some headway on my ESXi experiment. I have all the VMs and ESXi running on my RAID5 array. It all works great and the speed is not too bad but there is not enough space for data storage. I figured I could put a couple of 500GB drives and do a mirror. I had two possible options once I got ESXi seeing the SATA drives. One, partition them and make a huge virtual disk on each of the VMFS formatted disk. Then mirror using Windows 2008 for basic data redundancy. The other option was to set the drives up for ‘Raw Device Mapping’ (RDM) access. That involves mapping each drive to a placeholder VM disk. Then, all you do is add them to the VM you want to use them for. I am doing some major testing right now to make sure the controller and ESXi do not have any issues. The controller I am using has a SiI 3512 chip on it. It only supports SATA1 but for a file server you do not need as much on speed as you need it for reliability. Anyway, the two drives I am using for the RAID1(mirror) array are Western Digital Enterprise SATA2 drives.

To get the RDM thing going you first have to get to the local console. Hold down the ‘Alt’ key and push ‘F1’. It will show you what looks like a linux type command line interface. Once you have done that type in ‘unsupported’, without the quotations of course, and hit enter. Then it will ask for the password. Put in your root password and hit enter. It will drop you to the command line. It runs a mini-distribution of Linux called Busybox. When you are at the command line you need to run this command:

esxcfg-vmhbadevs

It will show you the list of physical drives names and what LUN ID they have. You need the physical ID, which is really long usually. Once you have that run this command.

cd /vmfs/volumes/datastore1

If you named your datastore something else then put that name in place. I would then create a directory in there and call it something meaningful as it will help keep the datastore structurally clean. Below is the command I ran, minus the really long number as I did not feel like typing the entire thing out.

vmkfstools -r /vmfs/devices/disks/vml.longnumberhere whateveryouwanttocallthedrive.vmdk

At that point you just need to go to the Remote Administration interface and then bring up the VM configuration and add the drive like you would when you wanted to add an existing virtual disk. Pay no mind to what size it says it is. The OS that you are binding the drive(s) to should know the correct size.

I have been off and on playing with VMware’s ESXi Server 3.5 u4 on my Dell Poweredge 2600 that is spec’ed similar to my IBM Server but has 6GB of RAM instead of 3GB. So far I have 4 VM’s running on it and it seems like a sweet spot for me as far as speed and responsiveness goes. I have a Development ‘box’ running Windows 2003 Server Standard w/ SQL Server 2005 and is given 2GB of RAM to play with. I also have a Windows 2008 ‘box’ that I gave 2GB of RAM to as I plan to make it my DC one day when it is slated to take over as my main Server and I also want it to one day do the TS Gateway thing. I have a third VM running Ubuntu JeOS Linux 9.04 and I gave it 512M of RAM. It will become my Shell server for now. As my last VM I have Windows XP Professional SP3 with 512M of RAM going as a ‘desktop’ machine of sorts. It is mainly for if my workstation is offline and I need to remote in and access workstation related things, it will be up for that as long as the server is on. The only thing I am working on still is the storage solution for my data. I am getting a couple of refurbished 500GB drives from Western Digital when they come back from repair and plan to do a software RAID1. I may be getting another 1 or 2 bad 500GB drives that are still under warranty to return and get replacements that will end up being my spares for when a drive fails. I already have some spare SCSI drives for the main array. Once I get those drives I will have to spend a fun Saturday configuring everything. Once I get the system mostly completed I will have to test it for a few weeks before making it live. Yes, I do this kind of thing for home use. I don’t like having the system fail on me right out of the gate. It will be nice to have one big system doing the job of 4 machines.