mrb's blog

Diskless Windows 7 iSCSI boot from OpenSolaris 2009.06 ZFS Server

By Marc Bevand

04 Dec 2010

Keywords: iscsi solaris sysadmin windows zfs

I wanted to boot Windows 7 from an iSCSI SAN, implemented with an OpenSolaris 2009.06 ZFS server. In theory, Windows 7, like Vista, 2008, 2008 R2, can be installed directly to an iSCSI target, but these instructions did not work for me. They require booting from LAN to register the iSCSI target as BIOS drive 0x80 with gPXE, and letting gPXE exit to rely on the BIOS then falling back gracefully to DVD-ROM boot in order to launch the Windows installer which would then detect BIOS drive 0x80 —the iSCSI target— and install on it. The first problem was that when gPXE exited, the BIOS would not fall back to DVD-ROM boot. Instead it forced me to press a key to reboot. So I wrote a custom COM32 program to, from gPXE, call into the BIOS via interrupt 19h to "load the OS". This worked. Interrupt 19h forced my BIOS to boot from DVD-ROM, launching the Windows installer. But then, even though the installer detected the iSCSI target, saw the correct size, was able to partition and format it, it refused to install on it. "This computer's hardware may not support booting to this disk. Ensure that the disk's controller is enabled in the computer's BIOS menu". All I saw, after pressing shift+F10 to get a shell and read setupact.log, was the same error message and that the drive did not have the "canInstall" capability. There was not a single clue of what was wrong. KB933925 was unhelpful.

Frustrated, and vividly reminded why I despise Windows (an open source OS would have let me inspect the code and fix the bug), I did not want to have to install Windows 7 on iSCSI via WinPE as it seemed too much work. So I resigned to install the OS on a regular physical disk first, then to transfer the image to an iSCSI target. Of course more hurdles came on the way, but I was able to resolve them. Hopefully my documented experience will save trouble to others who would like to enjoy the convenience of diskless booting Windows 7 via iSCSI:

  1. Install Windows 7 on a physical disk. When selecting a partition size (eg. 30000MB), if the disk is unpartitioned, the installer may forcefully create a small ~100MB system partition and a larger one of size 30000MB minus ~100MB. Be aware of this detail when transferring the image to the iSCSI target in later steps.
  2. Disable the LightWeight Filter (LWF) driver for the NIC that will be used for iSCSI boot. As explained in KB976042 this driver cannot start at boot time and would prevent the NIC from being available during iSCSI boot. One way to unbind the LWF driver from the NIC is to use bindview.exe which has to be compiled from the Windows Developer Kit(!) If you are like me and prefer easier solutions, do this by editing the registry (thanks to Takahiro Wagatsuma for this information):
    1. Open HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}
    2. Find and open the subkey for the NIC (eg. 0007)
    3. Open the subkey Linkage
    4. Edit the value FilterList. Delete the line that refers to the LWF driver UUID {B70D6460-3635-4D42-B866-B8AB1A24454C}. In my case I had to delete the second line. Before: 
      {158B0494-2576-4DE5-9E32-98DB9E177DD8}-{B5F4D659-7DAA-4565-8E41-BE220ED60542}-0000
{158B0494-2576-4DE5-9E32-98DB9E177DD8}-{B70D6460-3635-4D42-B866-B8AB1A24454C}-0000
      After: 
      {158B0494-2576-4DE5-9E32-98DB9E177DD8}-{B5F4D659-7DAA-4565-8E41-BE220ED60542}-0000
  3. On OpenSolaris: enable iSCSI, create a ZFS filesystem. I use OpenSolaris 2009.06, but any recent Solaris version should work the same. Enable the iSCSI target daemon and create a filesystem that will be dedicated to hosting the Windows image, so it can be independently snapshotted, cloned, etc: 
    $ svcadm enable iscsitgt
$ svcs iscsitgt
STATE          STIME    FMRI
online         Jun_26   svc:/system/iscsitgt:default
$ zfs create tank/win7
$ zfs list tank/win7
  4. Transfer the disk image to OpenSolaris. I boot the Windows machine into Linux, and use dd and ssh to transfer the image to OpenSolaris. Verify the partition sizes (remember that the installer may have created 2 partitions): 
    $ sfdisk -d /dev/sda
# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=     2048, size=   204800, Id= 7, bootable
/dev/sda2 : start=   206848, size= 61233152, Id= 7
/dev/sda3 : start=        0, size=        0, Id= 0
/dev/sda4 : start=        0, size=        0, Id= 0
    Copy the range of disk sectors that will include the MBR and these 2 partitions, that is the first 206848 + 61233152 = 61440000 sectors, or 61440000 * 512 = 31457280000 bytes. This number of bytes is exactly 30000*1024*1024 because I chose "30000MB" during installation. However always verify the sizes with sfdisk because sometimes I have noticed the installer adds 1 extra MB. I use dd and ssh to stream the image over the network to my OpenSolaris server: 
    $ dd if=/dev/sda bs=512 count=61440000 | ssh opensolaris "cat >/tank/win7/hda"
  5. On OpenSolaris, create the iSCSI target. First, snapshot the image (it will be useful): 
    $ zfs snapshot tank/win7@orig
    Then create the iSCSI target from it: 
    $ iscsitadm create target --type disk --backing-store /tank/win7/hda win7
$ iscsitadm list target
Target: win7
    iSCSI Name: iqn.1986-03.com.sun:...win7
    Connections: 0
    Immediately after iscsitadm create, OpenSolaris will start rewriting the entire image file over itself in order to force the allocation of all data blocks (thick image). This will take multiple minutes during which the iSCSI target is not really available (if you try to boot from it using gPXE, it will throw Input/output errors —it took me a while to find the cause of these errors). IMHO there should be an option to disable thick allocation. Anyway, monitor the progress of the rewrites with zfs list: 
    $ zfs list tank/win7
NAME        USED  AVAIL  REFER  MOUNTPOINT
tank/win7  29.2G  2.83T  29.2G  /tank/win7
    You will know they are finished when the "USED" space will have doubled (because thanks to the snapshot, the non-rewritten blocks have been preserved —I told you it would be useful!): 
    $ zfs list tank/win7
NAME        USED  AVAIL  REFER  MOUNTPOINT
tank/win7  58.4G  2.80T  29.2G  /tank/win7
    When the rewrites are done, roll back to the snapshot, if only to reduce the used space back to normal: 
    $ zfs rollback tank/win7@orig
$ zfs list tank/win7
NAME        USED  AVAIL  REFER  MOUNTPOINT
tank/win7  29.2G  2.83T  29.2G  /tank/win7
  6. Configure gPXE to offer booting Windows 7 from SAN. I will assume that you have a pre-existing DHCP setup serving gPXE configured to read a menu file that simply needs to be updated with the Windows 7 option. If not, follow one of the numerous gPXE HowTo guides. The gPXE menu file should be edited to add (replace x.x.x.x with the IP address of the OpenSolaris server, and the IQN with the one output earlier by iscsitadm list): 
    label win7
  kernel cmd.c32
  append sanboot iscsi:x.x.x.x::::iqn.1986-03.com.sun:...win7
    Where cmd.c32 should be extracted from the syslinux 3.86 tarball and placed in the TFTP directory. I have not tried the newer 4.x versions but they should work.
  7. This is it, test it! Remove the physical disk from the Windows machine, boot via PXE, select the win7 menu option, and enjoy your new Windows 7 environment.

Diskless machines booted from an iSCSI SAN implemented with an OpenSolaris ZFS server allow you to do powerful things. You can:

  • "zfs rollback" to revert a machine to snapshot.
  • "zfs clone && iscsitadm create target" to quickly provision a new machine (modulo the forced thick allocation).
  • "zfs destroy" to get rid of it.

Essentially, many of the features that are typically only available to virtualized environments are made available to physical machines. I have multiple diskless Linux machines that I use for my own GPGPU development projects. This setup gives me the option to run some tests under Windows 7 as well. Given that I frequently move GPUs from one machine to another, swap hardware components, upgrade or revert the OS/drivers, diskless booting and snapshotting certainly make my life much easier. If the above instructions to boot Windows 7 from iSCSI are useful to you, let me know!

Comments

Brian wrote: Could you explain more about how you setup your gpxe/dhcp server? Or perhaps just expand the post with the relevant parts of your config? I am trying to copy these steps - but I am having trouble with the network booting aspect?

I know you link to the etherboot page, but after some time it is still unclear to me the exacts steps to enable dhcp/gpx boot. 12 Jan 2011 03:16 UTC

mrb wrote: Well it's really simple. My DHCP server's dhcpd.conf file looks like:

subnet 10.110.0.0 netmask 255.255.255.0 {

...

next-server 10.110.0.254;

if exists user-class and option user-class = "gPXE" {
filename "/script";
} else {
filename "/undionly.kpxe.0";
}

}

That way the PXE BIOS's DHCP request will download and execute "undionly.kpxe.0" (gPXE image to be placed on the TFTP server). Then gPXE's DHCP request will download and interpret "script" which is another file served over TFTP which in my case looks like:

#!gpxe
chain menu.c32 http://${next-server}/gpxe.cgi?mac=${mac}

Feel free to replace the URL with something simpler for testing (http://x.x.x.x/gpxe.conf). In my case it is a CGI script returning a dynamically-generated gpxe config file depending on the MAC address of the booting machine. That config file is the one that needs to contain the section I documented:

lable win7
... 12 Jan 2011 08:24 UTC

Brian wrote: Thanks -
Another question that I can't seem to figure out. How do you derive the fully qualified name of he iscsi target from OSOL?

My list targets looks like this
@
sudo iscsiadm list target
Target: iqn.1986-03.com.sun:02:7cf41345-e30b-e38b-8922-b0b153266c25
Alias: -
TPGT: 1
ISID: 4000002a0000
Connections: 1@

The new LU I created looks like this:
@LU Name: 600144F05FDF080000004D2CFBC60005
Operational Status: Online
Provider Name : sbd
Alias : /dev/zvol/rdsk/tank/netboot/test1
View Entry Count : 1
Data File : /dev/zvol/rdsk/tank/netboot/test1
Meta File : not set
Size : 21474836480
Block Size : 512
Management URL : not set
Vendor ID : SUN
Product ID : COMSTAR
Serial Num : not set
Write Protect : Disabled
Writeback Cache : Enabled
Access State : Active@

But I am unsure how to put that together to create the fully qualified path of the isci device. I have only used iSCSI previously from ESXi - where I just gave it the path of the target and it discovered the devices. 13 Jan 2011 04:02 UTC

Brian wrote: By fully qualified path of the iscsi device, I am referring to the parameter you pass to:
append sanboot 13 Jan 2011 04:23 UTC

mrb wrote: I think my instructions are pretty clear :) Given your output it should be:

append sanboot iscsi:x.x.x.x::::iqn.1986-03.com.sun:02:7cf41345-e30b-e38b-8922-b0b153266c25

(replace x.x.x.x with the iSCSI server IP) 13 Jan 2011 08:01 UTC

Maurice wrote: I was following you're guide to create a SAN bootable windows 7 system. The only problem is that i'm using Solaris 11 Express so the command 'iscsitadm' isn't available. That means that i don't have a specific option with the 'iscsiadm' to mount a file as in a 'windows 7 image' as iscsi. Do you know how to achieve the same results as you have described om Solaris 11 Express? 17 Jul 2011 11:36 UTC

mrb wrote: Maurice: try installing the package SUNWiscsitgt. It provides the "Sun iSCSI Target" stack. 20 Jul 2011 11:06 UTC

Maurice wrote: The “Sun iSCSI Target" is installed by default but doesn't come with the "iscsitadm" command. 20 Jul 2011 16:34 UTC

mrb wrote: Run "pkg contents SUNWiscsitgt" to see the list of files installed by the package. Maybe iscsitadm is on the filesystem, but not in your PATH. 20 Jul 2011 20:51 UTC

Maurice wrote: I already did a find command to look for the command in every possible location, no result. The "pkg contents SUNWiscsitgt" gave no usefull results. After the "pkg install" command it says that there is no update required for the package. After the "pkg contents" command it says that there are no packages matching the name SUNWiscsitgt". Wierd... 20 Jul 2011 21:33 UTC

mrb wrote: It sounds like there is no package called SUNWiscsitgt in Solaris 11 Express. Perhaps its name changed. 21 Jul 2011 00:04 UTC

Maurice wrote: That was my conclusion as well, so do you have any tips how to achieve the same result with Solaris 11 Express? 21 Jul 2011 08:34 UTC

mrb wrote: Sorry, I don't know what they changed in the iSCSI stack in Solaris 11. 22 Jul 2011 03:37 UTC

Dan wrote: i deleted the lightweight filter line in the registry but when i reboot and check, the line returns back.

it also gives me an error after i deleted the line and clicked ok.
"Data of type REG_MULTI_SZ cannot contain empty strings. Registry editor will remove the empty string found. 05 Aug 2011 00:44 UTC

mrb wrote: Dan: this error means you are deleting both lines. You should only delete 1 of the 2.

Also, Windows may attempt to re-add the line if you restart it. Don't. Just power it off and take an image of the disk. 05 Aug 2011 03:38 UTC

Dan wrote: hi again, my win 7 booted from my iscsi target, thanks to your guide but it is so very slow to boot and load, im currently using ubuntu as target, do you think if i switch to opensolaris it would improve? what else could cause it to be super slow? 09 Aug 2011 04:42 UTC

mrb wrote: In my experience, the first diskless boot is slower than usual (Windows must be doing some reconfiguration). But subsequent boots should be faster.

If not, check disk i/o activity on the linux server with "iostat -x 1"; look at the last column (percent busy time) for each disk -- if close to 100% then the disks are madly seeking, ie your bottleneck is random IOPS. 10 Aug 2011 04:43 UTC

Dan wrote: i've found the solution to my problem, using the target machine's IP address as gateway in ipxe did it for me, it was my router that's slowing it down. 16 Aug 2011 06:08 UTC

jebro wrote: Any chance you could post the com32 program you wrote, mrb? 18 Oct 2011 15:13 UTC

mrb wrote: jebro: sure. Place the following file (mylocalboot.c) in the syslinux source tree under com32/samples:

#include <syslinux/boot.h>

#include <stdlib.h>

#include <com32.h>

int main(int argc, char *argv[])
{
uint8_t vector = 0x19;
static com32sys_t ireg;
__intcall(vector, &ireg, NULL);
return 0;
}

Then compile syslinux, it should produce a mylocalboot.c32 binary. From a gPXE config file you can invoke it with "kernel mylocalboot.c32".

PS: my blog software is incapable of correctly formatting code snippets in comments... *sighs* 19 Oct 2011 03:23 UTC

Clory wrote: Thank you so much for the guide, I was starting to think I was the only one who had trouble directly installing to the target.

If I made a script for ipxe that boot from the target omitting the com32 stuff should that work? 26 Oct 2011 03:11 UTC

mrb wrote: Clory: yeah, it should. 26 Oct 2011 06:11 UTC

hmos wrote: Last year I bookmarked this blog, thanks for your interesting comments. Your cooling pic was a little dejavu - I mounted an extra fan to my server using a pair of drywall anchors...

As this entry still gets eyes, may I suggest the iPXE as a gPXE replacement. iPXE has the exit command as well as conditionals. The iPXE site has instructions to iscsi boot other operating systems.

Also, a google will find bindview.exe - follow the KB without the Kit.

Your example uses thin files for the LU. Comstar loads the beginning of the file (64K maybe?) with LU information - so create the LU using the META FILE parameter if you want to dd without skipping the first few blocks.

Do you have knowledge on the performance difference files vs volumes? My simple testing favors volumes.

I have an old laptop with a 100mbs nic and no drive (died from the shakes - thanks for that post). Compressing the WinXP C: drive prior to copying to LU helps with that slower nic. 27 Oct 2011 09:57 UTC

mrb wrote: I have not looked too much at iSCSI performance. Volumes are in theory and in practice faster, but for my needs a file image is fast enough. In fact, I notice that running Windows 7 off my iSCSI server makes it more responsive during random I/O activity than running it off a local SATA drive, because the OpenSolaris server is more effective at caching data in its RAM than Windows itself. 28 Oct 2011 05:25 UTC

Clory wrote: Your blog made me want to try OpenSolaris but when I got to the party Oracle had already crashed it. So now I'm trying my hand at their OS.

The issue I am struggling with is that no matter what target I use I haven't been able to get Windows 7 to boot. I've tried Openfiler and FreeNAS both of which yielded the same results.

I'm originally a Debian person so I gave IETD a try but my wireshark analysis wasn't up to snuff but the one thing I did notice was that different sessions kept being initiated after iPXE released control over to Windows and a number of the old sessions were being reported by iPXE.

I'm hoping that whatever implementation of iSCSI that Oracle uses will remedy my issue but you seemed like a fairly knowledgeable fellow so I just wanted to get your input.

Thank you again. 28 Oct 2011 08:01 UTC

TooMeeK wrote: Hi there,
I've tried to install Win7 - without success :(
The problem is when I type all settings manually (ISC DHCP does not recognise for example "option ipxe.skip-san-boot") I have attached iSCSI drive (0x80) however when I type "exit" station just reboots not trying to boot from DVD. I have BOOT order: Network then DVD ROM. Simply it doesn't work on my motherboard.. 06 Nov 2011 23:21 UTC

TooMeeK wrote: Hi again after 2 days.
I had exacly same problem on my ASUS M3N-H HDMI: booting from PXE, after attaching iSCSI disk (with success) typing "exit" in iPXE menu caused "PRESS ENTER TO REBOOT" and no DVD booting. This is mobo specific problem.
There is no other way to workaround this than.. installing on iSCSI on another PC (or VM! as I did it on KVM). Then there will be new problem: the root device. It's attached to NIC vendor, MAC and other specific info. So booting on another NIC will cause BSOD and reboot after few minutes - here is KB article: http://support.microsoft.com/kb/976042 - and yes author, You have right. I'm testing this right now. 08 Nov 2011 22:46 UTC

TooMeeK wrote: However, this solution works when ALL NIC going to be used to boot from iSCSI are present in computer.
How to add NIC when it's different brand? I'm trying to workaround it since I have nForce 4 and in KVM NIC used is Intel PRO/1000...
booting from nFroce onboard causes BSOD not even trying to install new NIC. 08 Nov 2011 23:09 UTC

TooMeeK wrote: haha! I got You :D I did following trick:
download NIC drivers!
go to My computer -> Manage -> Device manager -> add older hardware
search for the NIC driver
add it, wait for install finished, will end with error 31 (software installed but hardware does not exist in the computer)
now DO NOT close the wizard
go to command line, issue nvspbind.exe -n and search for "Nvidia nForce Networking Controller", You will see that Lightweight driver for it is enabled. Disable it. DON'T FORGET to run cmd as Administrator!
However, after reboot still hangs at logo screen - "starting Windows" and reboots after few minutes. 08 Nov 2011 23:44 UTC

mrb wrote: TooMeeK: you should try installing windows on a HDD connected to the same machine that you plan to network-boot later, like I did in my post, to avoid all problems caused by incompatible hardware. 09 Nov 2011 06:16 UTC

sfinktah wrote: ""This computer's hardware may not support booting to this disk. Ensure that the disk's controller is enabled in the computer's BIOS menu""

You can ignore that message and just press next. It will install fine. 26 Nov 2011 16:33 UTC

mrb wrote: I don't remember I had an option to click "next". 29 Nov 2011 05:48 UTC

Greg wrote: Can you believe that caninstall capability is enabled by setting
"set keep-san 1" in gPXE? 08 Oct 2012 02:40 UTC

mrb wrote: I did try "set keep-san 1" as per http://www.etherboot.org/wiki/sanboot/win2k8_iscsi_install
but it did not help. 08 Oct 2012 07:43 UTC

nik wrote: TooMeeK: Did you find a way to boot win7 from iSCSI target with different NICs?
I'm facing similar problem. 28 May 2013 10:10 UTC

ymz wrote: I am using netbsd as my iscsi-target for windows 7 64bit. Did the LWF removal just before making disk image as your blog suggested. But, it reboots after the windows boot up screen shown. Any idea?

Thanks, 29 Mar 2014 14:53 UTC

Pazhani wrote: You are great genius. How do you come to know about this option. Your are really king of iscsi boot. I tried every option on possible forum about iscsi. Nothing worked. Met some issuses. But tried yours worked as charm. Even a man doesn't knows about iscsi can make it work using your forum. The willingness and kind heart to share the knowledge of yours shows your pure heart. You saved my job. 22 Mar 2015 04:17 UTC