APC SmartUPS Online Serial over USB

 Wow, long time no see.

So, as always, another post to avoid this relevant xkcd:

Hardware involved:

• Dell R730XD (Arch Linux)
  
• APC SmartUPS Online (SRT1500RMXLA)
• Generic AliExpress minipc (router/pfSense)

Scenario:

The SmartUPS Online is managed by APCUPSD on the R730XD connected by the included RJ50 to Serial cable (940-1525A).  Other devices attached to the same SmartUPS are running APCUPSD and configured to query the R730XD over the network for UPS status.

Problem:

The R730XD is far and away the largest draw on the UPS, so it's the first thing that gets shutdown in a power outage.  However, now there is no automatic shutdown for other devices attached to the same UPS that are required for the network itself to function (DNS server, router, switch, cable modem, PoE injector).  The router device runs pfSense, but has no serial ports, and for some reason the USB connection to APCUPSD from this UPS provides limited information.

Solution:

A well-supported serial-to-usb converter (the one I used was FTDI based) with the serial/DB9 end of the UPS cable plugged into it.  Plugged in to a USB port on the pfSense minipc.  APCUPSD configuration:

UPS Cable: smart
UPS Type: modbus
Device: /dev/ttyU0

APCUPSD returns appropriate data for it being serial connected even though it is connected via USB. The other option on this particular minipc was to work up a rj45 console to db9 connection, which is where I was headed before I could use a serial to usb adapter.

Percona XtraDB Cluster with Pacemaker for HA

So, I've moved on from my last employer after a decade(ish).  I've landed in a shop that is heavily devops focused, but my first real task wasn't so much about that, but highly available MySQL servers.  The MySQL end of it ended up being not too bad, two nodes running Percona XtraDB Cluster, and a third system running garbd as an arbitrator.  Had some stumbles with encrypted tables (bad documentation I claim) but once running seemed to be pretty impervious to my ham-fisted attempts to break the cluster.  The hard part came when I was trying to figure out how to move the IP around.  The Percona blogs are a wealth of information, and I found this which would seem to be very straight forward.  Its not.  RedHat has changed the tools with which you configure the cluster since the writing of that blog post, so without further ado, here's my config so a) you don't have to spend the hours I did and b) so I don't have to re-learn this somewhere down the road.  The assumptions are a two node Percona Xtradb Cluster with the mysql_monitor from the linked Percona blog, and using PCS to configure the resources:


pcs resource create pxc_monitor ocf:percona:mysql_monitor user="clustercheckuser" password="password" pid="/var/run/mysqld/mysqld.pid"
  socket="/var/lib/mysql/mysql.sock" \
  cluster_type="pxc" op monitor interval="5s" timeout="30s" OCF_CHECK_LEVEL="1"
pcs resource clone pxc_monitor cl_pxc_monitor meta clone-max="2" clone-node-max="1"
pcs resource create vip ocf:heartbeat:IPaddr2 params ip="172.16.0.11" nic="eth0" op monitor interval="10s"
pcs constraint location vip rule id="require_read" readable eq 1

Using mbuffer to increase throughput of zfs send/recieve.

Using mbuffer to speed up zfs send/receive been talked about in many other places, I wanted to add my own experience to the mix.  I used mbuffer on each end, dumping 3.4TB of data with pools that had lz4 compression enabled.  I set the buffer to be 512MB, as the target system is tight on memory.  Without mbuffer, the zfs send averaged about 504Mbps.  Using mbuffer, I was seeing 972Mbps of network traffic coming across, and mbuffer showing 111MB/s passing though it, for 88% of theoretical maximum Gigabit Ethernet bandwidth (125MB/s).  With the caveat that you lose approximately 20% of your bandwidth to network overhead (framing, TCP headers, etc), I was absolutely ecstatic at the throughput.  Unfortunately, even at that speed it still took just over nine hours to complete the send of my snapshot.  At least you can do incrementals afterwards, as long as I keep the original snapshot around.

Just thought I'd share.

Using perlbrew to compile ec2-consisent-snapshot on CentOS

I was tasked with taking a snapshot of a CentOS 5.7 mysql db in EC2 before we upgraded the instance type.  The data files reside on a RAID0 consisting of two EBS volumes.  The unanimous recommendation was to use ec2-consisent-snapshot.  Well, the CentOS AMI I'm having to use is custom, and the version of Perl is OLD.  Perlbrew to the rescue!  Earlier in the day, I'd screwed up my system version of Perl (thank god this box does NOTHING other than run mysql) so I had to export the perllibs so perlbrew would install.  Instructions follow:

curl -L http://install.perlbrew.pl | bash
perlbrew install perl-5.20.0
perlbrew switch perl-5.20.0
perlbrew install-cpanm
yum install expat-devel.x86_64
yum install openssl-devel.x86_64
yum install mysql-devel.x86_64
yum install util-linux
cpanm Net::Amazon::EC2
cpanm File::Slurp
cpanm DBI
cpanm DBD::mysql
cpanm DateTime::Locale
cpanm DateTime
git clone git://github.com/alestic/ec2-consistent-snapshot
vi ec2-consistent-snapshot (update #! line to point to correct version)

Slow CIFS/SMB speeds on OS X Mavericks (10.9)

Lots of complaints about how slow accessing Windows/Samba shares via Mavericks.  I've tried a couple of different things, but only found one that seems to restore SMB to a semblance of its former speed.  Thankfully its really easy.

In 'Connect to Server' when you specify the server by: smb://servername/share, instead do this: smb://servername:139/share. 

While I've not tried to analyze a network trace, I suspect that forcing the port bypasses some sort of smb version negotiation process.  But that's just a guess.

No ProSet tabs after installing Intel drivers for PRO1000/PT Server Adapter

I rebuilt my Windows 7 Professional box this week.  It was time, and installing with SP1 slipstreamed allowed me not to have to load a third party AHCI driver for my 6Gb/s SATA controller.  Everything was fine until I realised my network speed was a little slow.  I hadn't installed Intel's driver and ProSet yet, so I downloaded the most recent.  After installation, the Advanced tab was gone, but no ProSet tabs.  Uninstall the driver, the card would get re detected, Advanced tab would return.  Reinstall Intel driver, Advanced tab gone, no ProSet tabs.  Updated to latest driver from Windows Update.  Same thing.  This went on for a couple of days with slight variations.  Here's what I did to ultimately get the Intel drivers and ProSet installed and working:

1. Uninstall the nic via Device Manager.  Check the "Delete driver" box.
2. Follow this to disable automatic driver installation. (That procedure only works on Win 7 Pro+)
3. Either re-scan for devices in Device Manager, or reboot your system. (I chose to reboot)
4. Once restarted, verify that the nic shows up under "Unknown Devices" as an "Ethernet Controler".
5. Revert the change you made in gpedit so you can install the correct driver now.
6. Install the appropriate version of Intel's drivers/utilities for your card.

You should now have all of the tabs for VLANs, Teaming, Link Speed, and Advanced.

Hope I save someone out there some time.

** ADDENDUM **

I upgraded the Intel drivers recently, and after that I was back to my original issue.  Turns out I also needed to uninstall VirtualBox to get things back to working.  

** ADDENDUM ** ADDENDUM **

Actually, if you do not install Host-Only networking in Virtualbox, the problem doesn't appear.

M.

DKMS for updated e1000e driver

So, since I updated the e1000e driver, I've gotten no more errors in my messages logs.  However, it does seem that every month or two, a kernel update gets pushed to Squeeze, which un-does my driver fix.  When I worked at Dell, and we needed driver to be recompiled for any kernel that got installed, DKMS was the solution.  I rarely have to use cutting edge hardware with, umm, mature operating systems these days (I use old hardware with old operating systems), I hadn't given it much thought.  While DKMS is awesome when someone else does all the upfront work for you, trying to sort it out on one's own is significantly less fun.  I spent several hours over the course of a couple of weeks reading various things, and all of them failing to do what I wanted.  Got back to this week and finally got it sorted out.

1. Dowload appropriate source for the e1000e driver: http://sourceforge.net/projects/e1000/files/e1000e%20stable/2.3.2/e1000e-2.3.2.tar.gz
2. Extract archive to /usr/src.
3. Create a dkms.conf in the /usr/src/e1000e-2.3.2 directory with the following contents:
   PACKAGE_NAME="e1000e"
   PACKAGE_VERSION="2.3.2"
   BUILT_MODULE_LOCATION[0]="src"
   BUILT_MODULE_NAME[0]="e1000e"
   DEST_MODULE_LOCATION[0]="/kernel/drivers/net/e1000e/"
   AUTOINSTALL="yes"
   MAKE[0]="BUILD_KERNEL=${kernelver} make -C src CFLAGS_EXTRA=-DDISABLE_PM"
   CLEAN[0]="make -C src clean"
   REMAKE_INITRD=yes
4. Now just add the module via dkms:
   dkms -m e1000e -v 2.3.2 add
   dkms -m e1000e -v 2.3.2 build
   dkms -m e1000e -v 2.3.2 install
   
   After completion, you should restart the system to verify that the initial ramdisk got rebuilt correctly, and that the correct version of the driver is being used (modinfo e1000e).

Updating e1000e driver in Debian Squeeze

I kept having issues with my onboard Intel NIC on my Debian Squeeze NAS, the all too common:

Detected Hardware Unit Hang:

The version of the driver is significatly old, so I downloaded the newest from Sourceforge and tried to compile it. It failed to compile do to some power management code that is not in the kernel source from Debian.  Eventually I found someone else who'd had the same problem with the igb driver and solved it just by adding a flag to make:  CFLAGS_EXTRA=-DDISABLE_PM

Did the same when building the e1000e driver, compiled smoothly, remade initramfs, rebooted, haven't seen an error since.

Ubuntu 12.04 LTS in Virtualbox SLOOOOOWWWWWW

I've been trying to install Ubuntu 12.04 LTS in Virtualbox 4.2.4 for about a week now.  It would install eventually, but it was slow to do so, and almost unbearable to actually try to use.  Someone mentioned that if you had multiple CPUs configured for the VM, change it to one.  I had only one CPU assigned, so I changed it to two, and restarted my install.  It completed in less than five minutes.  Moral, if your having performance issues installing or using Ubuntu 12.04 LTS, make sure you're using more than one CPU.

Linux Mint 14 - Suggestions in Search Bar from Google - Why doesn't this work?

So, I've been demoing various Linux OS' for a few days trying to figure out what to use to replace Ubuntu (12.10).  I didn't like Unity, I just didn't like the approach they were using.  I tried out three or four other distro/versions/respins before I tried out Linux Mint 14 (Cinnamon).  After a couple of hours of playing around with it, I was convinced that it was the right choice for my work flow/environment/etc.  I notice that Firefox doesn't even include Google for use in the Search Bar.  Fine, I follow their procedure and add it.  I look stuff up, I get a lot of Abstracts and papers, and not the results I was looking for.  Turns out I'd installed the Google Scholar search engine instead of the plain Google.  Fine.  Correct that, and now while I can search and they return "correct" results, I'm not getting any "suggestions" as I type my searches in to the Search Bar.  After a few hours (too many) I begin to suspect that its the search provider xml that is installed via the Linux Mint website,  I pull a copy of the search provider off a Windows box, drop it in the searchplugins directory, and magically it all work now.  The big tip off is that the Google icon for the version install by the Linux Mint website used a different icon.

Hope that helps someone.


*EDIT* - You can also download a fresh copy of Firefox and extract the searchplugin xml files from there and copy them over to your profile.

**EDIT** - You can also look through the Google entries at http://mycroft.mozdev.org/ to find a search provider that does what you want (I wanted Suggestions).

Zenoss - WMI Counters for Disks

I mentioned previously that I was having an issue with my step-child monitoring system, Zenoss, where the WMI disk counters were not behaving correctly.  The disks would be found when the system was modeled (or re-modeled) but they would always show up "bad counter for..", instead of you know, working.  My original train of thought was related to the zWinUser and zWinPassword properties that were used to collect WMI stats from all of our Windows boxen.  However, I wasn't entirely convinced because we use the same user/password for all of the Windows systems in that domain, and none of the others was exhibiting this behavior.  After some more research and a little bit of actively NOT thinking about the problem, I realized that the all of the other WMI events/stats were being pulled just not disks.  It turns out you can disable perf counters for disks via the registry in W2k3 server, via this key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Perfdisk\Performance\Disable Performance Counters

On my server that was acting up, this was set to 1.  I changed the value to 0,  and remodeled the device (its Zenoss, I do it everytime I change something) and all my events cleared themselves.

Pulling zWinUser and zWinPassword out of Zenoss ZopeDB

In my continuing saga of having to maintain a sparsely documented Zenoss instance.  I was trying to troubleshoot some WMI errors (that will be a later post), but I had no idea what the password was for the user we are using to pull this information from our various Windows servers.  And yes, I could have reset the password, but that may open a whole other can of worms that I didn't want to deal with.  So, using zendmd, which I mentioned previously, I used the following to pull zWinUser and zWinPassword:

$ zendmd
Welcome to the Zenoss dmd command shell!
'dmd' is bound to the DataRoot. 'zhelp()' to get a list of commands.
Use TAB-TAB to see a list of zendmd related commands.
Tab completion also works for objects -- hit tab after an object name and '.'
 (eg dmd. + tab-key).
>>> d = dmd.Devices.findDevice('192.168.1.11')
>>> d.zWinUser,d.zWinPassword
('DOMAIN\\user', 'password')
>>>

Good luck and godspeed...

Zenoss ZopeDB Access

So, I'm sort of tangentially in charge of our Zenoss monitoring system, and on and off over the past couple of years I've derided the fact that Zenoss decided to store the important information about objects in a Zope database, and I couldn't figure out how to query it without writing Python to interact with it programatically.  Today, I guess I stumbled across the right set of search terms for Google to spit out something genuinely useful: zendmd.  I'm not going to get in to how to use it, as there is some at least decent information out there, and once you get in, its very Python interpreter-y, it should be easy to get your bearings.

Permission Denied when excuting a script on OpenMediaVault

Couldn't execute my AV scan script for my downloaded files:

root@omv:/media/data/usenet/scripts# ./av_scan.sh
-bash: ./av_scan.sh: Permission denied
root@omv:/media/data/usenet/scripts# ls -lah av_scan.sh
-rwxr-xr-x 1 root root 71 Jul  9 23:20 av_scan.sh

Fought with it for a couple of hours on and off before I realised what might be going on:

 # mount

<snip>

/dev/mapper/data_vg-data_lv on /media/fa12c9af-de6a-4068-b55f-71d798ee8b6b type xfs (rw,noexec,usrquota,grpquota)

The OpenMediaVault developer made a decision that nothing on the "data" filesystem should be allowed to be executed.  I understand why it might be best for it to be that way, so I relocated the scripts to a directory on another filesystem, and everything works as it should.

Getting a SYBA Serial card working under Debian Squeeze

I have a SYBA SY-PCI15003 that I used previously to connect to a UPS under FreeBSD 7.3 (FreeNAS).  When I moved that system over to run Debian Linux Squeeze (OpenMediaVault), NUT would not detect my UPS.  I read a bunch of pages, went down a few blind alleys, and eventually just figured it out myself.  So, here's what I did to get the serial port to be usable again:

1) Is the card detected at all, yes it is:

lspci -v

04:05.0 Serial controller: NetMos Technology PCI 9865 Multi-I/O Controller (prog-if 02 [16550])
    Subsystem: Device a000:1000
    Flags: medium devsel, IRQ 17
    I/O ports at bce8 [size=8]
    Memory at df9fb000 (32-bit, non-prefetchable) [size=4K]
    Memory at df9fc000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: [48] Power Management version 2

04:05.1 Serial controller: Illegal Vendor ID Device 9865 (prog-if 02 [16550])
    Subsystem: Device a000:1000
    Flags: medium devsel, IRQ 18
    I/O ports at bcf0 [size=8]
    Memory at df9fd000 (32-bit, non-prefetchable) [size=4K]
    Memory at df9fe000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: [48] Power Management version 2

04:05.2 Serial controller: Illegal Vendor ID Device 9865 (prog-if 00 [8250])
    Subsystem: Device a000:0000
    Flags: medium devsel, IRQ 19
    I/O ports at bcf8 [size=8]
    I/O ports at <unassigned>
    Memory at df9ff000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: [48] Power Management version 2
    Kernel driver in use: serial

 2) What does the system think IRQ/IO for the device is:

# setserial -a /dev/ttyS0
/dev/ttyS0, Line 0, UART: unknown, Port: 0x03f8, IRQ: 19
    Baud_base: 115200, close_delay: 50, divisor: 0
    closing_wait: 3000
    Flags: spd_normal skip_test

3) Since this configuration doesn't work, let's change it to point to the parent device:

setserial /dev/ttyS0 uart 16550 irq 17 port 0xbce8


The documentation for the device states that the uart is a 16550, and lspci -v gave the IRQ and the I/O port (make sure that you include the 0x before the I/O port).  After the changes its looks like this:

# setserial -a /dev/ttyS0
/dev/ttyS0, Line 0, UART: 16550, Port: 0xbce8, IRQ: 17
    Baud_base: 115200, close_delay: 50, divisor: 0
    closing_wait: 3000
    Flags: spd_normal

4) Test. I already had NUT configured for my UPS, so my testing was as easy as starting up the service and it actually finding the UPS.  There's tons of info about how to test your serial ports, so I won't go into that here.

5) Make configuration persistent.  This will depend on what distribution you're using, for Debian Squeeze, I found the easiest thing to do was use /etc/serial.conf.  Mine didn't exist, but the contents of the file are very simple, its just the arguments to setserial, in my case I could create it thusly:

# echo "/dev/ttyS0 uart 16550 irq 17 port 0xbce8" >> /etc/serial.conf

6) Reboot if you'd like, to make sure it correctly persists.

C'est fini!