Posts on Andrei Ostanin https://ostanin.org/posts/ Recent content in Posts on Andrei Ostanin Hugo -- gohugo.io en-us This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. Sat, 21 Jun 2014 13:10:56 +0000 Teensy++ 2.0 EPROM Reader https://ostanin.org/2014/06/21/teensy-2-0-eprom-reader/ Sat, 21 Jun 2014 13:10:56 +0000 https://ostanin.org/2014/06/21/teensy-2-0-eprom-reader/ Need to read a 16 bit parallel EPROM and have a Teensy++ 2.0 and some shift registers handy? Perfect! I bought a dead NAOMI game cartridge (Crackin’ DJ) and started on a quest to repair it. Sometimes it would start booting then lock up, sometimes it would reject the cartridge entirely, sometimes it would show up in the test menu and fail all ROM, and sometimes it wouldn’t show up at all. Need to read a 16 bit parallel EPROM and have a Teensy++ 2.0 and some shift registers handy? Perfect!

I bought a dead NAOMI game cartridge (Crackin’ DJ) and started on a quest to repair it. Sometimes it would start booting then lock up, sometimes it would reject the cartridge entirely, sometimes it would show up in the test menu and fail all ROM, and sometimes it wouldn’t show up at all. Since the game description, ROM CRCs and startup code all live in a 32 Mbit M27C322 EPROM (which is now 14 years old) that seemed like a likely place to start. Let’s dump it!

The setup in action

The address 20 address lines are output through three 74HC595 shift registers while the data bus and control lines are connected directly to the EPROM. A Python script dumps it in around 45 minutes.

Sadly, the EPROM dump matched the one from MAME perfectly. Now I have more troubleshooting to do. But, that means that the EPROM reader works perfectly! Get the code and schematics on GitHub.

]]> Playing with Mac OS X on KVM https://ostanin.org/2014/02/11/playing-with-mac-os-x-on-kvm/ Tue, 11 Feb 2014 15:06:32 +0000 https://ostanin.org/2014/02/11/playing-with-mac-os-x-on-kvm/ I always wanted to have an always-on Mac OS X machine which I could run Jenkins on to build and test iOS apps. While I have an old MacBook Pro I could use, I’d rather not have another computer running 24⁄7, especially one which would receive so little use. While OS X runs in VMWare (with some hacks) and VirtualBox, it can also run on KVM with a few patches thanks to Gabriel Somlo’s excellent work. I always wanted to have an always-on Mac OS X machine which I could run Jenkins on to build and test iOS apps. While I have an old MacBook Pro I could use, I’d rather not have another computer running 24⁄7, especially one which would receive so little use.

While OS X runs in VMWare (with some hacks) and VirtualBox, it can also run on KVM with a few patches thanks to Gabriel Somlo’s excellent work. He just updated his site around a week ago and mentions that Mountain Lion runs well. Seems like the perfect time to try it out!

Prerequisites

Patched QEMU + KVM + SeaBIOS…

Either follow the excellent instructions on Gabriel Somlo’s site to patch and compile the required components yourself, or if you happen to use Ubuntu 12.04, I’ve prepared some binaries which should speed the process along.

Using the Pre-Compiled Binaries

First, you’ll need libpixman-1-0 to run QEMU.

$ sudo apt-get install libpixman-1-0

Download and unpack the binaries.

$ tar xvJf osx-kvm.tar.xz
$ cd osx-kvm

Copy and load the patched KVM modules.

$ sudo modprobe -r kvm_intel
$ sudo cp kvm-kmod/kvm*.ko /lib/modules/`uname -r`/kernel/arch/x86/kvm/
$ dmesg | tail -n1
[11987.998234] loaded kvm module (for-linus)

Blank Drive Image

You can use qemu-img to create a blank drive image or use a zvol, actual device, whatever.

$ ./bin/qemu-img create -f qcow2 mac_hdd.img 20G
Formatting 'mac_hdd.img', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 lazy_refcounts=off

Mac OS X Installation Disc Image

I’m using Mac OS X Mountain Lion 10.8.5 for this. It’s important to have a clean disk image, not a hacked osx86 image because some of the kexts on the hacked images will cause a kernel panic on boot. It’s best to use the InstallESD.dmg located inside of the Install OS X Mountain Lion.app bundle.

The image needs to be an ISO file. If you have a DMG image then you can use Disk Utility’s “Convert” feature to make an ISO image (.cdr by default, but it’s the same thing).

VNC Client

Chicken is a great VNC client for OS X. It even supports creating an SSH tunnel automatically which is very handy.

OSK Key

The AppleSMC device needs a valid OSK key to function. This key is the same in all Macs and can be easily retrieved.

For OS X users, the Mac OS X Internals website has the source code to a program which can retrieve the key. I’ve included a binary of it in the above download.

For Linux on Mac users, Gabriel Somlo’s site has a C program which can retrieve this key.

Installing

sudo bin/qemu-system-x86_64 -enable-kvm \
	-m 2048 -cpu core2duo -machine q35 \
	-smp 2 \
	-usbdevice keyboard -usbdevice mouse \
	-vga std \
	-device isa-applesmc,osk="INSERT OSK KEY HERE" \
	-bios bios-mac.bin -kernel ./chameleon_svn2360_boot \
	-device ide-drive,bus=ide.2,drive=MacHDD \
	-drive id=MacHDD,if=none,cache=none,file=./mac_hdd.img \
	-device ide-drive,bus=ide.0,drive=MacDVD \
	-drive id=MacDVD,if=none,snapshot=on,file=./mountain-lion-10.8.5.iso -boot once=d

Now connect over VNC and you should see the Chameleon boot prompt.

Chameleon boot prompt

You can use -v for a verbose boot which can aid in troubleshooting. If you get a kernel panic here, make sure your OS X iso image is clean, not a hacked osx86 image.

If your terminal is filled with usb-kbd: warning: key event queue full and the keyboard doesn’t respond, just try again. While the keyboard works reliably once OS X boots, it sometimes has problems on the Chameleon boot prompt.

Hopefully the installer should just start right up.

Installer welcome screen

Continue with the installer and start up Disk Utility to partition your drive.

Disk Utility partitioning

Close Disk Utility and continue with the Reinstall OS X option. Install to the disk partition you just created.

Installer choose drive

The installer will reboot and put you back on the Chameleon prompt. Choose the (Installer) Macintosh HD partition to continue with the install.

Chameleon after installer reboot

The install will continue and reboot once more. Choose the Macintosh HD drive in Chameleon again to configure the install. I skipped setting up an Apple ID and registering.

Now you should have a mostly functional Mac OS X installation!

Successful Install

…though as you can see, it’s not exactly perfect.

Configuring

Energy Saver

Disable “Computer sleep” and “Display sleep” in the System Preferences. Resuming from sleep doesn’t work and will cause the virtual machine to lock up.

Energy Saver

Absolute Mouse Positioning

Most KVM guests can make use of the tablet device to synchronize the position of the mouse with it’s position over the VNC window. OS X, unfortunately, does not support this device.

If you enable OS X’s internal screen sharing and use it in place of QEMU’s VNC server then mouse positioning works perfectly. You can allow normal VNC viewers to connect by clicking on “Computer Settings…” in the Screen Sharing options.

Enabling VNC

Networking

The patches to QEMU include fixes to make the e1000 device work under OS X. To enable it for bridged networking, add a network bridge for KVM just add something like the following to the QEMU command:

-net nic,model=e1000,netdev=net0,macaddr=DE:AD:BE:EF:CA:FE -netdev tap,id=net0

Virtio

There is a virtio-net driver for Mac OS X which works great. Just install the driver, shutdown, and change the NIC model from e1000 to virtio in the QEMU command.

Virtio NIC

Display

QEMU emulates a Cirrus Logic GD5446 video card by default. Which this works fine over QEMU’s VNC server (though with a low color depth), the display glitches when using OS X’s Screen Sharing feature.

Cirrus Corrupt Display

QEMU also supports the vmware video card. There’s even a VMsvga2 driver for it on OS X. However, this driver caused a kernel panic on boot so this option too fails.

The best option that I have found is to use the std video card.

Chimera

Chimera is a bootloader which was forked from Chameleon. It’s easy to install and will allow us to boot to OS X automatically and remove the -kernel argument to QEMU. Download and install Chimera to get started.

org.chameleon.boot.plist

To automatically boot into OS X without needing to press enter at the prompt, edit (create) the org.chameleon.boot.plist file at /Extra/org.chameleon.boot.plist. A simple example to automatically boot would be:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Timeout</key>
	<string>5</string>
	<key>EthernetBuiltIn</key>
	<string>Yes</string>
	<key>PCIRootUID</key>
	<string>1</string>
</dict>

The EthernetBuiltIn and PCIRootUID keys fix the “Your device or computer could not be verified. Contact support for assistance.” error when logging in to the App Store.

smbios.plist

There’s another configuration file for Chimera, /Extra/smbios.plist. While I haven’t found a need to edit it personally (the App Store works fine without it), it can be used to change what model Mac the system identifies itself as as well as its serial number. There’s plenty of documentation as well as apps which help you edit the file.

Running

After the above steps, here is the command I use to boot the virtual machine:

sudo bin/qemu-system-x86_64 -enable-kvm \
	-m 2048 -cpu core2duo -machine q35 \
	-smp 2 \
	-usbdevice keyboard -usbdevice mouse \
	-vga std \
	-device isa-applesmc,osk="INSERT OSK KEY HERE" \
	-bios bios-mac.bin \
	-device ide-drive,bus=ide.2,drive=MacHDD \
	-drive id=MacHDD,if=none,cache=none,file=./mac_hdd.img \
	-net nic,model=virtio,netdev=net0,macaddr=DE:AD:BE:EF:CA:FE -netdev tap,id=net0

Conclusion

Overall, I’m very satisfied with running OS X on KVM. It performs more than well enough for my needs, seems to be stable, and doesn’t seem to waste excessive CPU cycles on the host.

]]> Routers and Their Lies https://ostanin.org/2014/02/11/on-routers-and-lies/ Tue, 11 Feb 2014 12:07:21 +0000 https://ostanin.org/2014/02/11/on-routers-and-lies/ I used to use a Buffalo WZR-HP-G302H router. It sucked. Wi-Fi would drop completely every 2-3 hours until I power cycled the thing. And the default Buffalo firmware is complete garbage. While many overseas models support the Buffalo-branded DD-WRT &ldquo;professional&rdquo; firmware, the Japanese models don&rsquo;t. Not only that, they also have locked bootloaders and can&rsquo;t be flashed over TFTP, don&rsquo;t provide a serial console during boot, and don&rsquo;t accept unencrypted firmware in their web interface. I used to use a Buffalo WZR-HP-G302H router. It sucked. Wi-Fi would drop completely every 2-3 hours until I power cycled the thing. And the default Buffalo firmware is complete garbage.

While many overseas models support the Buffalo-branded DD-WRT “professional” firmware, the Japanese models don’t. Not only that, they also have locked bootloaders and can’t be flashed over TFTP, don’t provide a serial console during boot, and don’t accept unencrypted firmware in their web interface.

This was a couple of years ago now, so there wasn’t the wealth of information on this router as there is now. I later found out that you can access a hidden maintenance page and unlock the bootloader. Then, OpenWRT’s page about the router says that OpenWRT should work if you recompile it with the erase size for the SPI flash changed from 128 KB to 64 KB. So I tried it. It didn’t boot anymore.

I wanted to open up the router and hook up a USB-serial adapter to see what exactly caused it to fail but the T10 S2 screws (those diamond shaped ones with the thing in the middle) required an expensive screwdriver and it just wasn’t worth it at the time. I tried reflashing it over TFTP but I couldn’t find any firmware for it which worked. It was still set to the Japanese region and the Japanese firmware is distributed in an encrypted form only. I gave up and bought a new router.

Now a few months ago, having acquired the necessary screwdriver, I decided to finally hook up the serial console and revive the router. There was no need, upon opening it up the router revealed its true nature…

WZR-HP-G302H internal

WZR-HP-G300NH!?

WZR-HP-G302H external

It definitely says WZR-HP-G302H on the case…

Flashing it with OpenWRT’s WZR-HP-G300NH firmware worked perfectly. Now I have a nice Linux device with Wi-Fi just waiting for a project.

]]> Plex Media Server in Docker https://ostanin.org/2013/09/14/plex-media-server-in-docker/ Sat, 14 Sep 2013 03:00:00 +0000 https://ostanin.org/2013/09/14/plex-media-server-in-docker/ Docker is the reason I switched from SmartOS back to Linux. The ability to manage applications with a workflow similar to git is very attractive. It solves most of the problems I had with SmartOS including complicated backups and needing KVM for some applications which don&rsquo;t run under Solaris zones. For me, the biggest problem with Docker right now is that the networking support is inflexible. While I managed to run applications like SABnzbd, Sick Beard, Couch Potato, and even rTorrent + ruTorrent among others with ease, Plex Media Server uses Avahi and it&rsquo;s own GDM network discovery protocols to broadcast its existence to clients. Docker is the reason I switched from SmartOS back to Linux. The ability to manage applications with a workflow similar to git is very attractive. It solves most of the problems I had with SmartOS including complicated backups and needing KVM for some applications which don’t run under Solaris zones.

For me, the biggest problem with Docker right now is that the networking support is inflexible. While I managed to run applications like SABnzbd, Sick Beard, Couch Potato, and even rTorrent + ruTorrent among others with ease, Plex Media Server uses Avahi and it’s own GDM network discovery protocols to broadcast its existence to clients. This requires the Plex server and clients to run on the same subnet which isn’t a configuration that Docker supports easily. After a few tries, I managed to get it to work with some ugly hacks.

Setting Up a Network Bridge on the Host

Ubuntu’s KVM networking guide comes in handy here. Following it, I created a br0 bridge with the following configuration added to the host’s /etc/network/interfaces:

auto br0
iface br0 inet dhcp
  bridge_ports eth0
  bridge_stp off
  bridge_fd 0
  bridge_maxwait 0

After restarting networking (service networking restart), the new bridge showed up in ifconfig:

host$ ifconfig
br0       Link encap:Ethernet  HWaddr 1c:6f:65:d8:af:fd
          inet addr:192.168.1.131  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::1e6f:65ff:fed8:affd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:710009 errors:0 dropped:0 overruns:0 frame:0
          TX packets:664838 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:643136133 (643.1 MB)  TX bytes:198627145 (198.6 MB)

Now we need some way to use this bridge in the Plex container.

Wrestling With Docker’s Networking Configuration

My first approach was to try using Pipeline for the network configuration. My approach was to use Pipework on the host as follows:

host$ docker run -d plex
bcb765ea1b73
host$ pipework br0 bcb765ea1b73 192.168.1.10

And from the point of view of the container, here is the network configuration before running Pipework:

root@bcb765ea1b73:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 82:48:50:71:6c:55
          inet addr:172.17.0.8  Bcast:172.17.255.255  Mask:255.255.0.0
          inet6 addr: fe80::8048:50ff:fe71:6c55/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:195 errors:0 dropped:0 overruns:0 frame:0
          TX packets:83 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:267817 (267.8 KB)  TX bytes:5723 (5.7 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)the point of view of the container:

And after running Pipework:

root@bcb765ea1b73:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 82:48:50:71:6c:55
          inet addr:172.17.0.8  Bcast:172.17.255.255  Mask:255.255.0.0
          inet6 addr: fe80::8048:50ff:fe71:6c55/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:195 errors:0 dropped:0 overruns:0 frame:0
          TX packets:83 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:267817 (267.8 KB)  TX bytes:5723 (5.7 KB)

eth1      Link encap:Ethernet  HWaddr 06:15:b2:28:fe:a5
          inet addr:192.168.1.10  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::415:b2ff:fe28:fea5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:238 (238.0 B)  TX bytes:238 (238.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

The eth1 interface was added and given an IP address in the same subnet as my LAN. However, since Plex doesn’t seem to have any configuration to specify which network interface to broadcast on. I used route in the container to make eth1 the default interface and set the gateway:

root@bcb765ea1b73:/# route add default gw 192.168.1.1 eth1

Now I could open up a web browser to http://192.168.1.10/web and see that Plex was indeed running. However, I couldn’t seem to publish the server to myPlex (UPnP not working?) and the server wouldn’t show up in Plex clients. There’s probably some proper way to get this to work, but since I’m not competent with networking I decided to do an ugly hack to make it all work.

Ugly Hack

Docker allows us to specify LXC configuration options directly with the -lxc-conf parameter to docker run. You can also disable Docker’s network configuration with the -n=false parameter. Combining these two I got the following docker run command:

docker run -d -n=false \
  -lxc-conf="lxc.network.type = veth" \
  -lxc-conf="lxc.network.flags = up" \
  -lxc-conf="lxc.network.link = br0" \
  -lxc-conf="lxc.network.ipv4 = 192.168.1.10" \
  -lxc-conf="lxc.network.ipv4.gateway=192.168.1.1" \
  plex

Now Plex would show up perfectly in my browser, Plex clients, and publishing to myPlex worked perfectly. The image I prepared stores the Plex configuration in /config which I bind mount into persistent storage on the host. After bind mounting everything, this is the command I use to start Plex:

docker run -d -n=false \
  -v /srv/media/videos:/videos \
  -v /srv/media/music:/music \
  -v /tank/virt/config/plex:/config \
  -lxc-conf="lxc.network.type = veth" \
  -lxc-conf="lxc.network.flags = up" \
  -lxc-conf="lxc.network.link = br0" \
  -lxc-conf="lxc.network.ipv4 = 192.168.1.10" \
  -lxc-conf="lxc.network.ipv4.gateway=192.168.1.1" \
  plex

You can find the Dockerfile to build the container on Github.

]]> Some Chef and Gentoo Stuff https://ostanin.org/2013/09/08/some-chef-and-gentoo-stuff/ Sun, 08 Sep 2013 03:00:00 +0000 https://ostanin.org/2013/09/08/some-chef-and-gentoo-stuff/ I experimented with using Puppet and later Chef to manage my home server and all the virtual machines on it. After automating the installation and management of most of the services I use (rutorrent, sabnzbd, sickbeard, couchpotato, gitlab, etc.), I decided that both Puppet and Chef were just too much trouble for my needs. They were too heavyweight for managing single instances of each service. However, I wrote quite a bit of code during that time which may be of some use. I experimented with using Puppet and later Chef to manage my home server and all the virtual machines on it. After automating the installation and management of most of the services I use (rutorrent, sabnzbd, sickbeard, couchpotato, gitlab, etc.), I decided that both Puppet and Chef were just too much trouble for my needs. They were too heavyweight for managing single instances of each service.

However, I wrote quite a bit of code during that time which may be of some use. Though most of it is very specific to my setup, I published some of it on Github.

chef-gentoo

Gentoo Linux is my Linux distribution of choice. However, Chef doesn’t have much support for Gentoo out of the box. In fact, portage support was outright broken when I first tried it (packages which have the same name in two different categories such as dev-db/mysql and virtual/mysql would always fail). I made a simple cookbook which fixed those issuses (based on a Chef bug report) and made it easy to add keywords, use flags, mask, and unmask for packages and manage eselect as well. You can check it out on Github.

chef-dotfiles

Chef’s Users cookbook is a great way to add users from a Chef data bag. I wanted each user to have their dotfiles git repository checked out, linked to the home directory, and updated during each Chef run. The code is again on Github.

Gentoo Install Script

When testing my Chef cookbooks, I’d find myself often reinstalling Gentoo from scratch in a virtual machine. To automate this process, I created a simple install script for Gentoo. While some things are hardcoded and it assumes you want an amd64 install, it may be of some use. The code is up on Github.

I still have a few other projects I’ve been meaning to upload for a while now. The biggest problem is writing meaningful README files for code I haven’t touched in months.

]]> Importing a FreeBSD ZFS Pool in illumos https://ostanin.org/2013/06/14/importing-a-freebsd-zfs-pool-in-illumos/ Fri, 14 Jun 2013 03:00:00 +0000 https://ostanin.org/2013/06/14/importing-a-freebsd-zfs-pool-in-illumos/ Before we begin, please remember that the usual warnings about data loss apply. If you do follow these directions, I highly recommend that you at least have a zpool with redundancy (raidz, mirror) and try this only on a single disk at first, to make sure the zpool is still importable in FreeBSD/Linux before you alter every single disk. My ZFS pool has been through a lot. I created it over two years ago on FreeBSD. Before we begin, please remember that the usual warnings about data loss apply. If you do follow these directions, I highly recommend that you at least have a zpool with redundancy (raidz, mirror) and try this only on a single disk at first, to make sure the zpool is still importable in FreeBSD/Linux before you alter every single disk.

My ZFS pool has been through a lot. I created it over two years ago on FreeBSD. Then when ZFS on Linux seemed to reach a usable state, I switched over to Gentoo Linux. And now, growing tired of managing the Gentoo install whose main purpose is just to launch LXC and KVM virtual machines, I decided to try out the illumos-based SmartOS.

I started out by trying an OpenIndiana LiveCD to make sure I wouldn’t have any problems importing my zpool. Little did I know…

root@openindiana:/jack# zpool import
   pool: tank
     id: 12050779155201421632
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

        tank          UNAVAIL  insufficient replicas
          raidz1-0    UNAVAIL  insufficient replicas
            c5t1d0p0  UNAVAIL  cannot open
            c5t2d0p0  UNAVAIL  cannot open
            c5t3d0p0  UNAVAIL  cannot open
            c5t4d0p0  UNAVAIL  cannot open
            c5t5d0p0  UNAVAIL  cannot open

It turns out that Solaris, and thus illumos, adds an EFI label to all disks in a zpool, even if you use the whole drive. Some searching suggested that it would be possible to format a drive in Solaris, add an EFI label, resilver the zpool, and repeat with with the other drives, one at a time. Since resilvering my array would probably take ~12+ hours, having to resilver five times was not something I was looking forward to. I decided to try it out on one drive for now anyway.

root@openindiana:/jack# format -e
Searching for disks...done

c5t1d0: configured with capacity of 2794.52GB
c5t2d0: configured with capacity of 2794.52GB
c5t3d0: configured with capacity of 2794.52GB
c5t4d0: configured with capacity of 2794.52GB


AVAILABLE DISK SELECTIONS:
       0. c4t0d0 <SanDisk-U3CruzerMicro-4.04 cyl 473 alt 2 hd 128 sec 32>
          /pci@0,0/pci1458,5006@1d/hub@1/storage@1/disk@0,0
       1. c5t1d0 <ATA-Hitachi HDS5C303-A580-2.73TB>
          /pci@0,0/pci1458,b005@1f,2/disk@1,0
       2. c5t2d0 <ATA-Hitachi HDS5C303-A580-2.73TB>
          /pci@0,0/pci1458,b005@1f,2/disk@2,0
       3. c5t3d0 <ATA-Hitachi HDS5C303-A580-2.73TB>
          /pci@0,0/pci1458,b005@1f,2/disk@3,0
       4. c5t4d0 <ATA-Hitachi HDS5C303-A580-2.73TB>
          /pci@0,0/pci1458,b005@1f,2/disk@4,0
       5. c5t5d0 <ATA-Hitachi HDS5C303-A580-2.73TB>
          /pci@0,0/pci1458,b005@1f,2/disk@5,0
Specify disk (enter its number): 5
selecting c5t5d0
[disk formatted]
No Solaris fdisk partition found.


FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        inquiry    - show vendor, product and revision
        scsi       - independent SCSI mode selects
        cache      - enable, disable or query SCSI disk cache
        volname    - set 8-character volume name
        !<cmd>     - execute <cmd>, then return
        quit
format> fdisk
No fdisk table exists. The default partition for the disk is:

  a 100% "SOLARIS System" partition

Type "y" to accept the default partition,  otherwise type "n" to edit the
 partition table.
WARNING: Disk is larger than 2TB. Solaris partition will be limited to 2 TB.
y
format> label
[0] SMI Label
[1] EFI Label
Specify Label type[1]:
Ready to label disk, continue? y

format> quit

Now I expected the disk I just formatted to show up as corrupt, but to my surprise…

root@openindiana:/jack# zpool import
   pool: tank
     id: 12050779155201421632
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

        tank          UNAVAIL  insufficient replicas
          raidz1-0    UNAVAIL  insufficient replicas
            c5t1d0p0  UNAVAIL  cannot open
            c5t2d0p0  UNAVAIL  cannot open
            c5t3d0p0  UNAVAIL  cannot open
            c5t4d0p0  UNAVAIL  cannot open
            c5t5d0    ONLINE

It seemed to work! Now, rebooting to Linux or FreeBSD and importing the zpool, it will resilver a very small part of the pool, lasting just seconds. Now just continue rebooting to Solaris, formatting a drive, rebooting to Linux or FreeBSD and importing the zpool, until you’ve formatted every disk.

root@openindiana:/jack# zpool import
   pool: tank
     id: 12050779155201421632
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        tank          ONLINE
          raidz1-0    ONLINE
            c5t1d0    ONLINE
            c5t2d0p0  ONLINE
            c5t3d0    ONLINE
            c5t4d0p0  ONLINE
            c5t5d0    ONLINE
root@openindiana:/jack# zpool import -f tank
root@openindiana:/jack# zpool status
  pool: tank
 state: ONLINE
  scan: resilvered 68K in 0h0m with 0 errors on Thu Jun 13 15:48:15 2013
config:

        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
          raidz1-0    ONLINE       0     0     0
            c5t1d0    ONLINE       0     0     0
            c5t2d0p0  ONLINE       0     0     0
            c5t3d0    ONLINE       0     0     0
            c5t4d0p0  ONLINE       0     0     0
            c5t5d0    ONLINE       0     0     0

errors: No known data errors

The only thing I’m still concerned about is why some disks in the zpool specify the partition (p0) and others just specify the whole disk (d0). However, the zpool can now be imported in Linux, FreeBSD, and illumos without any problems.

Now to start playing around with SmartOS…

]]> Freerunning Toshiba TMP68HC000P-16 on a Breadboard https://ostanin.org/2013/06/06/freerunning-toshiba-tmp68hc000p-16-on-a-breadboard/ Thu, 06 Jun 2013 03:00:00 +0000 https://ostanin.org/2013/06/06/freerunning-toshiba-tmp68hc000p-16-on-a-breadboard/ Building my own computer was always a dream of mine. I decided to take the first step by buying a Japanese Motorola 68000 clone, the Toshiba TMP68HC000P-16. Rated at 16 MHz and using CMOS technology, this seemed like one of the best versions of the 68000. To start with, I tried to freerun the chip. Freerunning is where you hardwire the data lines so that the CPU continually executes the same instruction over an over while the program counter counts up through memory. Building my own computer was always a dream of mine. I decided to take the first step by buying a Japanese Motorola 68000 clone, the Toshiba TMP68HC000P-16. Rated at 16 MHz and using CMOS technology, this seemed like one of the best versions of the 68000.

To start with, I tried to freerun the chip. Freerunning is where you hardwire the data lines so that the CPU continually executes the same instruction over an over while the program counter counts up through memory. For the 68000, connecting all the data lines to ground is a common way to freerun the CPU, making it execute OR instructions endlessly.

In the video, I connected all the data lines to ground and all the address lines to LEDs. The interrupt/error related signals are also connected so as not to disrupt the CPU. The clock is an 8 MHz crystal oscillator (should be easier to debug with my cheap logic analyzer at this speed) and there is a reset button connected as well. You can see the LEDs count up as the program counter is incremented.

I ordered some SRAM, EEPROMs, and a CPLD for the glue logic. Though I don’t have much free time to work on this project, hopefully I’ll have more updates in the future.

]]> Batch Convert Keynote Files to PDF with AppleScript https://ostanin.org/2013/05/31/batch-convert-keynote-files-to-pdf-with-applescript/ Fri, 31 May 2013 03:00:00 +0000 https://ostanin.org/2013/05/31/batch-convert-keynote-files-to-pdf-with-applescript/ I switched from Keynote to just writing presentations in Markdown and using Pandoc to convert them to PDF files with beamer. Markdown lends itself well to version control so I decided to store all my presentations in a git repository. I already had a couple of years worth of Keynote presentations which I would never touch again. Since Keynote file are almost 8x larger than their PDF counterparts, I wanted to convert all my old presentations to PDF before sticking them under version control as well. I switched from Keynote to just writing presentations in Markdown and using Pandoc to convert them to PDF files with beamer. Markdown lends itself well to version control so I decided to store all my presentations in a git repository.

I already had a couple of years worth of Keynote presentations which I would never touch again. Since Keynote file are almost 8x larger than their PDF counterparts, I wanted to convert all my old presentations to PDF before sticking them under version control as well. Here’s a script which finds every *.key and .pptx recursively under the current directory and converts it to a PDF.

#! /bin/sh

for FILE in `find "${PWD}" -name \*.pptx -or -name \*.key`; do
  DIR=`dirname $FILE`
  echo Converting $FILE to PDF
  osascript << EOF
tell application "System Events"
  tell application "Keynote"
    activate
    open "${FILE}"
  end tell
  tell process "Keynote"
    click menu item "PDF…" of menu of menu bar item "Share" of menu bar 1
    click button "Next…" of sheet 1 of window 1
    keystroke "G" using {command down, shift down}
    keystroke "${DIR}"
    click button "Go" of sheet 1 of sheet 1 of window 1
    click button "Export" of sheet 1 of window 1
  end tell
  do shell script "killall Keynote"
end tell
EOF
done

This is my first go at AppleScript (or a shell script/AppleScript hybrid of some sort) so there may be some better ways to approach the problem. For instance, telling Keynote to quite would result in the script hanging with Keynote asking me if I wanted to save or not. And since tell application "Keynote" to quit didn’t seem to return until Keynote quit, making it impossible to dismiss the save dialog on the next line. I use killall Keynote to get around this problem.

]]> Using OpenVPN for Select Sites with OpenWRT https://ostanin.org/2012/09/13/using-openvpn-for-select-sites-with-openwrt/ Thu, 13 Sep 2012 03:00:00 +0000 https://ostanin.org/2012/09/13/using-openvpn-for-select-sites-with-openwrt/ I love watching Hulu from time to time, but living in Japan means I need to use a proxy or VPN located in the US to view it. I used to use a PPTP VPN server on a VPS to view Hulu, routing all traffic through the VPN. This had some obvious drawbacks in that Hulu playback was slower, the VPN configuration had to be done on each computer, and I couldn&rsquo;t use things like BitTorrent at the same time as watching videos on Hulu. I love watching Hulu from time to time, but living in Japan means I need to use a proxy or VPN located in the US to view it. I used to use a PPTP VPN server on a VPS to view Hulu, routing all traffic through the VPN. This had some obvious drawbacks in that Hulu playback was slower, the VPN configuration had to be done on each computer, and I couldn’t use things like BitTorrent at the same time as watching videos on Hulu. While this was fine when I’d just watch videos on my laptop once every few weeks, I wanted a more transparent solution once an HTPC and an OpenWRT-capable router entered the equation.

The solution I came up with is simply connecting to an OpenVPN server from my OpenWRT router and selectively routing certain IP addresses through the VPN. The great thing about this setup is that it’s completely transparent for anyone on the LAN. Any computer can stream videos from Hulu without any configuration.

OpenVPN Server

I won’t cover setting up an OpenVPN server in any detail, as that’s already covered in countless other pages on the Internet. I will, however, offer a few hints from my experience setting up an OpenVPN server on FreeBSD.

The VPN doesn’t have to be fast. I used an old FreeBSD server of mine still running in my parents house on a slow cable connection.

I used a combination of these two guides to create a bridged VPN. Here’s my basic OpenVPN config file:

port 1194
proto udp

dev tun

ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/example.crt
key /usr/local/etc/openvpn/keys/example.key
dh /usr/local/etc/openvpn/keys/dh1024.pem

server 10.8.0.0 255.255.255.0

duplicate-cn

keepalive 10 120

comp-lzo

persist-key
persist-tun

status /var/log/openvpn/openvpn-status.log

verb 3

I then set up pf using the following pf.conf to NAT the VPN clients and give them access to the Internet:

ext_if="vr0"
vpn_if="tun0"
vpn_network="10.8.0.0/24"

nat on $ext_if from $vpn_network to any -> ($ext_if)

OpenVPN Client on OpenWRT

I mostly followed the guide here to set up the client. For reference, I listed all the changes to my config files on the router below.

/etc/config/network

config interface 'vpn'
        option ifname 'tun0'
        option defaultroute '0'
        option peerdns '0'
        option proto 'none'

/etc/config/firewall

config zone
        option input 'ACCEPT'
        option forward 'REJECT'
        option output 'ACCEPT'
        option name 'vpn'
        option masq '1'
        option network 'vpn'

config forwarding
        option dest 'lan'
        option src 'vpn'

config forwarding
        option dest 'vpn'
        option src 'lan'

/etc/config/openvpn

config openvpn 'example'
        option enabled '1'
        option client '1'
        option dev 'tun'
        option proto 'udp'
        option nobind '1'
        option persist_tun '1'
        option persist_key '1'
        option ca '/etc/openvpn/example.ca'
        option cert '/etc/openvpn/example.cert'
        option key '/etc/openvpn/example.key'
        option comp_lzo '1'
        option verb '3'
        option float '1'
        option remote 'example.us.ostanin.org'
        option client '1'

Selective Routing

Now, we can route IP addresses used by Hulu through the VPN. There are two ways to do this, either have the OpenWRT client call route to add the routes, or have the OpenVPN server push the routes to the client. I choose to push the routes from the server, as this would allow me to connect to the server from outside my LAN and still enjoy Hulu without any additional configuration.

OpenVPN allows you to add routes to the client using the push "route X.X.X.X 255.255.255.255" directive in it’s config file. So we just need to add all the necessary IP addresses for Hulu into the OpenVPN server’s config file and we’re good to go. I wrote a python script to automate the process:

#! /usr/bin/env python

SITES = {
    'hulu': [
        'hulu.com',
        'www.hulu.com',
        'static.hulu.com',
        'ads.hulu.com',
        'assets.hulu.com',
        't2.hulu.com',
        'urlcheck.hulu.com',
        'secure.hulu.com',
        'a.hulu.com',
        'b.hulu.com',
        'c.hulu.com',
        'd.hulu.com',
        'e.hulu.com',
        'f.hulu.com',
        'g.hulu.com',
        'h.hulu.com',
        'i.hulu.com',
        'j.hulu.com',
        'k.hulu.com',
        'l.hulu.com',
        'm.hulu.com',
        'n.hulu.com',
        'o.hulu.com',
        'p.hulu.com',
        'q.hulu.com',
        'r.hulu.com',
        's.hulu.com',
        't.hulu.com',
        'u.hulu.com',
        'v.hulu.com',
        'w.hulu.com',
        'x.hulu.com',
        'y.hulu.com',
        'z.hulu.com',
    ]  
}

import socket
import sys

sites = sys.argv[1:]

for site in sites:
    if site in SITES:
        print '###', site
        for host in SITES[site]:
            try:
                addrs = socket.gethostbyname_ex(host)[2]
                print '#', host
                for addr in addrs:
                    print 'push "route', addr, '255.255.255.255"'
            except:
                pass

When run with ./routemaker.py hulu, it will print out a list of all push directives needed to watch Hulu. Here is an example run:

$ ./routemaker.py hulu
### hulu
# hulu.com
push "route 60.254.153.10 255.255.255.255"
push "route 60.254.153.25 255.255.255.255"
# www.hulu.com
push "route 210.149.135.22 255.255.255.255"
push "route 210.149.135.45 255.255.255.255"
# static.hulu.com
push "route 23.3.104.33 255.255.255.255"
push "route 23.3.104.8 255.255.255.255"
# ads.hulu.com
push "route 23.3.104.65 255.255.255.255"
push "route 23.3.104.59 255.255.255.255"
# assets.hulu.com
push "route 23.3.104.42 255.255.255.255"
push "route 23.3.104.75 255.255.255.255"
# t2.hulu.com
push "route 23.3.104.32 255.255.255.255"
push "route 23.3.104.9 255.255.255.255"
# urlcheck.hulu.com
push "route 208.91.157.69 255.255.255.255"
# secure.hulu.com
push "route 118.215.182.31 255.255.255.255"
# a.hulu.com
push "route 23.3.104.59 255.255.255.255"
push "route 23.3.104.65 255.255.255.255"
# c.hulu.com
push "route 210.149.135.21 255.255.255.255"
push "route 210.149.135.14 255.255.255.255"
# g.hulu.com
push "route 208.91.157.28 255.255.255.255"
# i.hulu.com
push "route 208.91.157.16 255.255.255.255"
# m.hulu.com
push "route 23.3.104.66 255.255.255.255"
push "route 23.3.104.56 255.255.255.255"
# p.hulu.com
push "route 23.3.104.43 255.255.255.255"
push "route 23.3.104.81 255.255.255.255"
# r.hulu.com
push "route 210.149.135.29 255.255.255.255"
push "route 210.149.135.44 255.255.255.255"
# s.hulu.com
push "route 23.3.104.65 255.255.255.255"
push "route 23.3.104.33 255.255.255.255"
# t.hulu.com
push "route 208.91.157.68 255.255.255.255"
# u.hulu.com
push "route 208.91.157.20 255.255.255.255"

You can now copy and paste that to your OpenVPN server config and restart the server. You could also automate it to run once a day and restart OpenVPN automatically if you so wish.

If everything worked, the Hulu routes should have been added to your client:

root@router:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         182.236.18.15   0.0.0.0         UG    0      0        0 pppoe-wan
10.8.0.1        10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
10.8.0.5        *               255.255.255.255 UH    0      0        0 tun0
23.3.104.8      10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.9      10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.32     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.33     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.34     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.42     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.43     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.56     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.59     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.65     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.66     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.73     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.75     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.3.104.81     10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
23.48.134.31    10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
60.254.153.10   10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
60.254.153.25   10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
182.236.18.15   *               255.255.255.255 UH    0      0        0 pppoe-wan
192.168.1.0     *               255.255.255.0   U     0      0        0 br-lan
208.91.157.16   10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
208.91.157.20   10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
208.91.157.28   10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
208.91.157.68   10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
208.91.157.69   10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
210.149.135.30  10.8.0.5        255.255.255.255 UGH   0      0        0 tun0
210.149.135.53  10.8.0.5        255.255.255.255 UGH   0      0        0 tun0

And there you have it! Hulu should now work on all devices within the LAN. You can make other sites work too by adding their hostnames to the script as well. Keep in mind that ads from Hulu will be sent through the VPN while videos themselves will not, resulting in very laggy ads if your VPN is slow like mine. If your Hulu client allows choosing a quality setting for ads (like the XBMC client does), set them to the lowest quality.

]]> Backing Up Simple Websites with Git https://ostanin.org/2012/01/28/backing-up-simple-websites-with-git/ Sat, 28 Jan 2012 03:00:00 +0000 https://ostanin.org/2012/01/28/backing-up-simple-websites-with-git/ I&rsquo;m hosting three different WordPress blogs for different people (this one included) and recently decided to switch away from my home server to a VPS. My home server has a RAID-Z array and important data is backed up remotely on a regular basis giving me the peace of mind that my data is safe, but I can&rsquo;t be so sure about the VPS. Also, I might switch to a different VPS provider in the future and wanted to make deploying the blogs as easy as possible. I’m hosting three different WordPress blogs for different people (this one included) and recently decided to switch away from my home server to a VPS. My home server has a RAID-Z array and important data is backed up remotely on a regular basis giving me the peace of mind that my data is safe, but I can’t be so sure about the VPS. Also, I might switch to a different VPS provider in the future and wanted to make deploying the blogs as easy as possible. I came up with the following solution.

Goals

  • Daily backups usable for small websites with little traffic
  • Backups should be complete: all files, MySQL dump, logs
  • Flexibility

Why Git?

Using git has many inherent advantages over simply copying files to a remote server.

  • Daily backups will only store diffs, taking up little space
  • Using a VCS allows you to see history, merge fixes, branch, etc. if you need to (eg. for developing plugins/themes on one computer and easily merging them into your website)
  • Push backups to a remote server using http, https, ssh, or git protocols
  • Easily exclude files from backups using .gitignore
  • Setting up a new server is just a git clone away
  • Know your backup succeeded just by browsing your git repository
  • More!

So, why should we store the database backup and log files in git as well?

  • Complete history of your website – restore the files AND database to any state you want
  • The diffs in each commit give you the traffic for the day and which rows were added to your database
  • No need for separate backup methods!

Of course, git isn’t the perfect solution for all websites. High-traffic websites may find their git repositories quickly grow in size when storing database backups and log files.

Structure

To make things easier, I structured each of the sites in the following fashion:

/usr/local/www/
`- example.com/
   |- logs/
   |  `- example.com-access.log
   |  `- example.com-error.log
   |- root/ # WordPress installation
   `- sql/
      `- example.com.sql # MySQL dump

Also, the MySQL username and database name for each site is the same as the URL, so “example.com” for this case.

Prepare the Remote Repository on the Backup Server

Create an empty repository on the backup server.

$ cd /path/to/git/repositories
$ mkdir example.com.git
$ cd example.com.git
$ git init --bare --shared

Add data to the repository

We can clone the empty repository we just made and add the initial data to it. Do this on the server your site is currently hosted on.

$ git clone http://git.example.com/example.com
$ cd example.com
$ mkdir logs sql
$ touch logs/example.com-access.log logs/example.com-error.log
$ mysqldump -uexample.com -pPASSWORD --skip-extended-insert --skip-comments example.com > sql/example.com.sql
# Copy the WordPress install
$ cp -r /path/to/site/root root

I also made a simple Makefile to automate backing up. Edit it to suit your needs (especially the MYSQL_ variables) and place it in the root of your repository.

SITE_NAME=  $(notdir $(CURDIR))
MYSQL_USER= $(SITE_NAME)
MYSQL_PASS= PASSWORD
MYSQL_DB=   $(SITE_NAME)

.PHONY: backup
backup: sql_backup git_commit git_push
  @echo Backup of $(SITE_NAME) finished successfully!

.PHONY: sql_backup
sql_backup:
  @echo Backing up the database
  @mysqldump -u$(MYSQL_USER) -p$(MYSQL_PASS) --skip-extended-insert --skip-comments $(MYSQL_DB) > sql/$(SITE_NAME).sql

.PHONY: git_commit
git_commit: git_commit_sql git_commit_logs git_commit_root

.PHONY: git_commit_logs
git_commit_logs:
  @echo Committing log files
  @-git commit logs -m "Updating logs"

.PHONY: git_commit_root
git_commit_root:
  @echo Committing root
  @-git add root
  @-git commit root -m "Updating root"

.PHONY: git_commit_sql
git_commit_sql:
  @echo Committing sql
  @-git commit sql -m "Updating sql dump"

.PHONY: git_push
git_push:
  @echo Pushing commit
  @-git push

Now we can commit changes and push to our backup server. Remember that you can add a .gitignore to keep files or directories from being backed up.

$ git add .
$ git commit -m "initial commit"
$ git push origin master

Deploying

Now when you want to deploy your site, set up the MySQL user and database then checkout the site from git.

$ cd /usr/local/www
$ git clone --shared http://git.example.com/example.com
$ cd example.com
$ mysql -uexample.com -pPASSWORD example.com < sql/example.com.sql

As long as your web server points to /usr/local/www/example.com, you should now be able to access your site!

If you want to test backups, just generate some log activity or do something which would update the database and run make backup from /usr/local/www/example.com.

Automating Backups

You can automate backups by using cron.

Access your crontab

$ crontab -e

…and add the following line:

@daily cd /usr/local/www/example.com && make backup &> /dev/null

If all goes well, you should see daily commits popping up on your backup server! If you ever lose data, you can go back to the deploying section and go back to your last backup.

]]>