Jun 14, 2013

Importing a FreeBSD ZFS Pool in illumos

Before we begin, please remember that the usual warnings about data loss apply. If you do follow these directions, I highly recommend that you at least have a zpool with redundancy (raidz, mirror) and try this only on a single disk at first, to make sure the zpool is still importable in FreeBSD/Linux before you alter every single disk.

My ZFS pool has been through a lot. I created it over two years ago on FreeBSD. Then when ZFS on Linux seemed to reach a usable state, I switched over to Gentoo Linux. And now, growing tired of managing the Gentoo install whose main purpose is just to launch LXC and KVM virtual machines, I decided to try out the illumos-based SmartOS.

I started out by trying an OpenIndiana LiveCD to make sure I wouldn’t have any problems importing my zpool. Little did I know…

root@openindiana:/jack# zpool import
   pool: tank
     id: 12050779155201421632
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

        tank          UNAVAIL  insufficient replicas
          raidz1-0    UNAVAIL  insufficient replicas
            c5t1d0p0  UNAVAIL  cannot open
            c5t2d0p0  UNAVAIL  cannot open
            c5t3d0p0  UNAVAIL  cannot open
            c5t4d0p0  UNAVAIL  cannot open
            c5t5d0p0  UNAVAIL  cannot open

It turns out that Solaris, and thus illumos, adds an EFI label to all disks in a zpool, even if you use the whole drive. Some searching suggested that it would be possible to format a drive in Solaris, add an EFI label, resilver the zpool, and repeat with with the other drives, one at a time. Since resilvering my array would probably take ~12+ hours, having to resilver five times was not something I was looking forward to. I decided to try it out on one drive for now anyway.

root@openindiana:/jack# format -e
Searching for disks...done

c5t1d0: configured with capacity of 2794.52GB
c5t2d0: configured with capacity of 2794.52GB
c5t3d0: configured with capacity of 2794.52GB
c5t4d0: configured with capacity of 2794.52GB


AVAILABLE DISK SELECTIONS:
       0. c4t0d0 <SanDisk-U3CruzerMicro-4.04 cyl 473 alt 2 hd 128 sec 32>
          /pci@0,0/pci1458,5006@1d/hub@1/storage@1/disk@0,0
       1. c5t1d0 <ATA-Hitachi HDS5C303-A580-2.73TB>
          /pci@0,0/pci1458,b005@1f,2/disk@1,0
       2. c5t2d0 <ATA-Hitachi HDS5C303-A580-2.73TB>
          /pci@0,0/pci1458,b005@1f,2/disk@2,0
       3. c5t3d0 <ATA-Hitachi HDS5C303-A580-2.73TB>
          /pci@0,0/pci1458,b005@1f,2/disk@3,0
       4. c5t4d0 <ATA-Hitachi HDS5C303-A580-2.73TB>
          /pci@0,0/pci1458,b005@1f,2/disk@4,0
       5. c5t5d0 <ATA-Hitachi HDS5C303-A580-2.73TB>
          /pci@0,0/pci1458,b005@1f,2/disk@5,0
Specify disk (enter its number): 5
selecting c5t5d0
[disk formatted]
No Solaris fdisk partition found.


FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        inquiry    - show vendor, product and revision
        scsi       - independent SCSI mode selects
        cache      - enable, disable or query SCSI disk cache
        volname    - set 8-character volume name
        !<cmd>     - execute <cmd>, then return
        quit
format> fdisk
No fdisk table exists. The default partition for the disk is:

  a 100% "SOLARIS System" partition

Type "y" to accept the default partition,  otherwise type "n" to edit the
 partition table.
WARNING: Disk is larger than 2TB. Solaris partition will be limited to 2 TB.
y
format> label
[0] SMI Label
[1] EFI Label
Specify Label type[1]:
Ready to label disk, continue? y

format> quit

Now I expected the disk I just formatted to show up as corrupt, but to my surprise…

root@openindiana:/jack# zpool import
   pool: tank
     id: 12050779155201421632
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

        tank          UNAVAIL  insufficient replicas
          raidz1-0    UNAVAIL  insufficient replicas
            c5t1d0p0  UNAVAIL  cannot open
            c5t2d0p0  UNAVAIL  cannot open
            c5t3d0p0  UNAVAIL  cannot open
            c5t4d0p0  UNAVAIL  cannot open
            c5t5d0    ONLINE

It seemed to work! Now, rebooting to Linux or FreeBSD and importing the zpool, it will resilver a very small part of the pool, lasting just seconds. Now just continue rebooting to Solaris, formatting a drive, rebooting to Linux or FreeBSD and importing the zpool, until you’ve formatted every disk.

root@openindiana:/jack# zpool import
   pool: tank
     id: 12050779155201421632
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        tank          ONLINE
          raidz1-0    ONLINE
            c5t1d0    ONLINE
            c5t2d0p0  ONLINE
            c5t3d0    ONLINE
            c5t4d0p0  ONLINE
            c5t5d0    ONLINE
root@openindiana:/jack# zpool import -f tank
root@openindiana:/jack# zpool status
  pool: tank
 state: ONLINE
  scan: resilvered 68K in 0h0m with 0 errors on Thu Jun 13 15:48:15 2013
config:

        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
          raidz1-0    ONLINE       0     0     0
            c5t1d0    ONLINE       0     0     0
            c5t2d0p0  ONLINE       0     0     0
            c5t3d0    ONLINE       0     0     0
            c5t4d0p0  ONLINE       0     0     0
            c5t5d0    ONLINE       0     0     0

errors: No known data errors

The only thing I’m still concerned about is why some disks in the zpool specify the partition (p0) and others just specify the whole disk (d0). However, the zpool can now be imported in Linux, FreeBSD, and illumos without any problems.

Now to start playing around with SmartOS…