Tools on traviscj/blog

smart arguments

Thu, 21 Feb 2019 00:00:00 +0000

It’s easy to love the terseness one can get out of a command-line tool with a “pass by order” convention. For example, maybe I run something like

deploy myapp sea2 production

to deploy an application called myapp to the second production datacenter near Seattle. This works great, except now I need to remember the order of arguments is: deploy [app] [datacenter] [environment].

The typical solve for this problem is introducing named arguments, so you’d end up with

Naming is hard (so don't)

Sun, 07 Oct 2018 02:31:43 +0000

A lot of times I just want to record something. This should be one of those things computers are good at. Turns out this is a bit harder than it seems: many editors make you name a file to save it.

One easy-sounding way to record something is:

open up a text editor
mash the keyboard until the thought is out of your head and into the text editor
(hard part starts) what do we name that file?
where do we put that file?
do we remember to save that file, given the difficulty of (3) & (4)?

I think picking names too early is at best a minor annoyance, and at worst a pretty major distraction.

osquery

Fri, 12 Jan 2018 08:43:17 +0000

I’ve known about [osquery][] for a while, but recently spent some time digging around in it. [osquery]: https://osquery.io The basic idea is to provide a consistent SQL interface to a bunch of system data, instead of learning idiosyncrasies of individual commands (which themselves vary across operating systems). My hacker buddy Sharvil has worked on osquery a fair bit and explained to me a couple years ago that it actually uses [sqlite][]’s [virtual table functionality][vtab] to provide the interface – it’s a fascinating and brilliant project! [sqlite]: https://sqlite.org/ [vtab]: https://sqlite.org/vtab.html

relational java explorations with sqlite3 vtab

Fri, 05 May 2017 20:10:28 +0000

One idea i have been mulling over lately is exposing a java codebase relationally, with queries like:

SELECT *
FROM classes c
JOIN annotations a
  ON a.class_id = c.id
WHERE a.name = 'MyAnnotationClass';

The idea here is that you could search a codebase by pretending that you had tables for a whole bunch of things like:

classes
annotations
variables
literals
methods

This would be a useful thing because instead of relying on an IDE like IntelliJ to do “find usages” operations, you could actually script those interactions. Admittedly, of course, Java has some pretty sophisticated reflection APIs to find this sort of stuff too, but it seems like exploring the structure could be much easier writing Java code to traverse those trees.

.sqliterc

Fri, 31 Mar 2017 00:00:00 +0000

I came across Better output from sqlite3 command line. His .sqliterc file did not work for me, but the simpler

.mode "column"
.headers on

did work nicely.

I also found out that brew install sqlite3 does not install the sqlite3 binary to $PATH, which stinks.

The sqlite3 environment is still much less pleasant than the mysql cli. A few things on my wishlist:

tab completion of tables, fields, keywords, and functions.
nicer .schema output – I’ve just gotten really used to reading them in mysql output

It sounds like maybe apsw can be something useful? Or maybe Navicat? (but that’s expensive)

idea: transaction ordering in ledger-cli

Mon, 27 Mar 2017 00:00:00 +0000

I love ledger-cli but keeping it in sync with my bank statements drives me crazy. The problem is that the transactions can end up with an essentially arbitrary ordering, and the order they clear (and even the date they clear) is not necessarily under my direct control.

One answer to this is: stop caring about the ordering of your transactions! That’s a decent answer, except that not addressing the ordering issue means that you can only ever have “end-of-day” consistency. This means you need a different report to reconcile the transactions.

Useful homebrew formula

Thu, 01 Dec 2016 00:00:00 +0000

Last week I was telling Paul about things I have brew installed on my work laptop. I pulled a full list as I was doing some upgrades and stuff this morning.

Glancing over it a bit, here are some of my favorites/most usefuls:

autojump for a usage-adjusting cd – given a partial string it just goes to the most-used directory with that prefix. I have autojump aliased to j to shorten it up even more.
colordiff to make diffs/grep/etc nicer to look at in the terminal.
coreutils/dateutils is nice because it installs the GNU versions of a lot of standard linux commands like du and ls and date that work a bit strange/differently on OSX.
jq is amazing — lets you query JSON objects.
ledger to manage bank balances in plain text.
newsbeuter to subscribe to RSS feeds.
osquery lets you do SQL-like queries to find out about the system it’s running on (sharvil did a bunch of work on it, actually…)
terminal-notifier to pop up notifications from terminal/cron jobs.
tmux to have long-running terminal windows without having terminal windows open.
watch instead of “some command”+uparrow+enter+uparrow+enter+….
wget as shorthand for “curl http://xyz/tuv.json > tuv.json” (old habits…),
youtube-dl for downloading youtube videos.

build_json.sh

Wed, 19 Aug 2015 00:00:00 +0000

This might seem silly, but I’ve beeing playing with some json.sh scripts that build legitimate json bodies and are easily filled into a shell script variable as needed.

The basic driving idea was that there are lots of slick ways to pull data out of JSON(either by programming something with python’s json or running a command line tool like jq or whatever), but not as many friendly ways to build some JSON out of a given token or whatever. Often, you have a list of identifiers and you need to build a bunch of JSON blobs from that list.

Switching to jekyll

Mon, 01 Sep 2014 00:00:00 +0000

I spent a few hours this evening switching everything over from tcjblog to Jekyll. I really do miss a few parts of the old blog setup, but one of the main reasons I switched several years ago was LaTeX support. Back then, we didn’t have really sweet options like MathJax. And it just takes too long.

One of the other big things that was keeping me from updating the blog the last couple of months was… the stolen laptop. It turns out that I didn’t have very good backups for the last few website updates, which meant that I needed to manually restore blog posts from the website.

mnemonicode-0.73

Thu, 10 Jul 2014 00:00:00 +0000

I found a really cool tool called mnemonicode.

It basically encodes an arbitrary string into some easily-sayable and understandble phrases:

Traviss-MacBook-Pro% echo "hello world" | ./mnencode
 Wordlist ver 0.7
square angel stone. blitz pacific tango. zebra shave basic
Traviss-MacBook-Pro% echo square angel stone. blitz pacific tango. zebra shave basic | ./mndecode
hello world

some setup notes

Sun, 27 Oct 2013 00:00:00 +0000

zsh

Zsh is an amazing (mostly) drop-in replacement for bash.

On OSX, just run

> brew install zsh

You also want to install oh-my-zsh.

I went a little crazy with plugins. My list is:

plugins=(autojump brew git git-flow gnu-utils gpg2 osx textmate zsh-syntax-highlighting history-substring-search)

SSH

I ssh a lot. So I have a bunch of host entries in my .ssh/config file. Each looks like:

Host [shortname]
     Hostname [hostname]
     User     [username for hostname]
     IdentityFile [my home directory]/.ssh/[id_rsa file for this host]

You also need identity files for hosts you use. I tend to use a different one on each host I connect to; some might consider this overly paranoid. Probably it’d be better to use 1 key but only for a shorter period of time. In any case, you need to use

Numerical Development on OSX (in the Command Line)

Mon, 19 Aug 2013 00:00:00 +0000

I’ve been working on C implementations of my research projects, which can of course be a perilous project. I’ve found some tools that make it hugely, hugely better.

Homebrew

You can’t do a list like this without mentioning homebrew. You want homebrew instead of MacPorts or Fink or bailing twine and chewing gum or whatever else you were thinking about using. Just do it: You can find the homepage at brew.sh or just install with:

spring-mass visualization

Wed, 31 Jul 2013 00:00:00 +0000

I’m working on a paper about an algorithm for hotstarting nonlinear program solves; one application of this might be in the realm of nonlinear model predictive control. In these types of models, we first define the physical equations for the system under consideration. They are subject to some control parameters, which are just a mathematical representation of the input we could give the system. We also define an objective–something that we would like to minimize(usually something like time or energy).

better matlab email notifications

Tue, 25 Jun 2013 00:00:00 +0000

One of my officemates asked me how I did my MATLAB email notifications. I had originally done it here. Of course, the machine we work on most didn’t support a mail or msmtp command, so I found an SMTP script, which worked okay, but frustratingly also required Python. On a bit more sleep, it finally occured to me that MATLAB might have an email script.

Turns out, it does. I wrote a little wrapper script around this to abstract out some of the email details.

org-mode emacs to track mileage

Wed, 15 May 2013 00:00:00 +0000

I’ve been trying out emacs org-mode lately for keeping track of everything. One thing that seemed worth tracking is the mileage the new Focus gets. Turns out that org-mode supports a kind of spreadsheet, so I made a quick little video demo on how I use it:

Getting really fast at LaTeX

Wed, 10 Apr 2013 00:00:00 +0000

Taking LaTeX notes in real time is one of my favorite useless party tricks. I started developing my techniques during the last year or two of undergraduate work at University of Washington, and was extremely good at it by the end of coursework at Northwestern. There are several keys to taking LaTeX notes in real time:

Know LaTeX.
Shortcut appropriately.
Type keystrokes quickly.
One keystroke re-renders.
Type fewer keystrokes.
Type fewer “strange” keystrokes (backslashes, curly braces, etc)
Re-renders should scroll to your current location in the text.
Extremely rapid debugging of LaTeX expressions.

Before I get started, one other note: I mention TextMate a few places, but in principle there is no reason you couldn’t configure all the same things in Vim, Emacs, or Sublime Text. If I was starting over, I might use Sublime Text instead.

more stupid matlab tricks

Fri, 28 Sep 2012 00:00:00 +0000

I’ve been setting up long-ish runs on my MATLAB instance lately, which is nice because I can wanter off to do something else. The trouble is that I occasionally forget to come back or refocus on work again.

I’ve come up with two solutions to this:

The first is to use growlnotify to pop up a persistant growl message. This works well enough as long as I’m at the computer:

system('growlnotify -s -m "somefile is done"')

For more extended aways or if I’m going to take a nap while it runs, I might do something more like:

system('echo "matlab: uflX_aw_constr_log_5_15 done" | msmtp -a traviscj (mycell)@txt.att.net')

You need some setup before this works: I did a brew install msmtp and set up a .msmtprc file. After that, it should work goldenly, at least for AT&T customers. Others might need to look up their email/sms gateway number.

debugging matlab mex

Thu, 16 Aug 2012 00:00:00 +0000

One thing I’ve been doing a bit of lately is debugging MATLAB mex files; it seemed worth documenting. The broad strokes:

recompile all mex codes with ‘-g’
quit MATLAB
From a command prompt, run

$ /Applications/MATLAB_R2012a.app/bin/matlab -Dgdb
[snip]
(gdb) run -nojvm
>> run_your_mex_file

This gives debugging output. If there is an error in the code, you’re dropped back to a gdb prompt, where you can continue debugging in a semi-normal fashion. Typically, I end up doing a ‘where’ to get a stack trace, and debug from there more-or-less like usual.

Integrating Sparrow and Aperture/iPhoto

Fri, 13 Apr 2012 00:00:00 +0000

In my recent quest to take on digital photography, I’ve been using Apple’s Aperture to store download my pictures and do basic manipulation on them. It’s been working great, except that emailing is a pain because Apple hardcoded the email clients to:

Apple Mail.app
Eudora (really!?)
America Online(!?!)
Microsoft Entourage
Microsoft Outlook

while I’d prefer to use Sparrow, a lightweight client that Sharvil got me hooked on a while back. Rather than continue to use Mail.app to send pictures via email, I found out that Aperture actually uses some AppleScripts to do the actual sending of the email. That got me thinking: I just need to change the script to mail using Sparrow, instead of Mail.

cplex matlab interface

Wed, 01 Feb 2012 00:00:00 +0000

Just for my own reference, I’m documenting the interface to CPLEX.

CPLEX expects a problem in the form ( \begin{split} \min \qquad & g^Td + \frac12 d^TWd\ \text{subject to} \qquad & c_L \leq Ad \leq c_U\ & d_L \leq d \leq d_U \end{split} ) and is called by

cplex = Cplex('test');
cplex.Param.feasopt.tolerance.Cur = 1e-8;
if params.printLevel < 8
    cplex.DisplayFunc = [];
end
cplex.Model.sense = 'minimize';
cplex.Param.qpmethod.Cur = 1;
cplex.addCols(gk,[],bl-xk,bu-xk);
cplex.addRows(-ck, A0, -ck);
cplex.Model.Q = W;
cplex.Model.obj = g;
cplex.Model.lb = d_L;
cplex.Model.ub = d_U;
cplex.Model.lhs= c_L;
cplex.Model.rhs= c_U;
cplex.solve();

VirtualBox and HRD/JT65

Tue, 10 Jan 2012 00:00:00 +0000

One of the semi-frustrating aspects of the state of computing and ham radio is the unavailability of some software packages on OSX. In particular, I’ve been interested in experimenting with JT65 and also wanted to use my Yaesu FT-857d’s CAT cable to program memories and also try out HRD. (I should add–I’ve tried the JT65 stuff and some Yaesu memory programmers, but wasn’t satisfied with the Linux setup for those. I’ll probably look into getting them working more later, but for now, this article.)

Getting GnuPlot to compile on OSX

Thu, 27 May 2010 00:00:00 +0000

I misunderstood an officemate’s question yesterday and set out to compile gnuplot. Stock GNUPlot will fail to compile on OSX due to an incompatibility in the readline library. So when you compile, let GNUPlot know to use the one included:

> tar xzvf gnuplot*
> cd gnuplot-4.2.39
> ./configure --prefix=/usr/local --with-readline=builtin
> make
> sudo make install

Pylab PDF

Tue, 25 May 2010 00:00:00 +0000

I’m in the process of kicking the MATLAB habit and replacing it with Python. Pylab is a reasonable approximation to the 2D plotting capabilities of MATLAB, but by default Pylab doesn’t run on headless(ie, non-GUI) boxes. So on my macbook, where I’m often running things interactively, I have

backend : macosx

and on the faster core2quad machine I send off longer-running jobs to, I used

backend : PDF

in the .matplotlib/matplotrc file.

Spaces setup

Tue, 25 May 2010 00:00:00 +0000

My workflow is probably pretty specific, but I figured I’d document it for people looking for ideas on how to set it up. This one seems to work pretty well for me.

. Terminal: for running stuff on my and other computers. . Textmate: writing mathematics, programming code, etc . Mathematica, MATLAB, and Papers . Mail and Adium . Firefox, Chrome, Safari . iTunes and VLC

I have this arranged in a 3×2 grid, such that Textmate is top center. From there I can navigate to terminal, math programs, and Firefox directly(control-{left,right,down}). It takes two navigations to get to communication programs and entertainment, which is good for concentration while working.

ZSH and scp/rsync globbing

Tue, 25 May 2010 00:00:00 +0000

I finally got annoyed at typing

tjohnson> scp tcj:*txt .
zsh: no matches found: tcj:*txt

I’d always fixed this by the up-CTRL-A-over-over-over-over-quote-CTRL-E-over-over-quote method, ie

tjohnson> scp "tcj:*txt" .

but the more proper and less annoying version is

tjohnson> scp tcj:\*txt .

However, in the spirit of inspired overkill, there are a couple other options. You can define some new aliases with

alias scp='noglob scp'

but that hardly counts as overkill, so instead I edited my .zshrc to include

Wiki Problems

Tue, 21 Jul 2009 00:00:00 +0000

I really like the idea of having a wiki website. It’s great to be able to share documents really easily, be able to edit stuff without logging in, track changes you’ve made, and so on.

I have also tried quite a few wiki setups, including MediaWiki(mostly at a job a while back and a spectacular success), MoinMoin, and PmWiki. However, each of them fell flat in a certain annoying way.

Netlib

Wed, 27 May 2009 00:00:00 +0000

Wow. Just wow.

HMCP Set, TextMate, and RoR 2.0

Fri, 06 Feb 2009 00:00:00 +0000

I found a great little LaTeX homework layout for typesetting math homeworks a while back, and I’ve meant to write it up for quite a while. It’s from Harvey Mudd College’s Mathematics Department, engineered to automatically satisfy the department guidelines on how homeworks should be laid out. They also have thesis and poster templates, but the homework installation proceedures and sty files are located on the Homework Class page. Their website also has extensive information on their computer resources. While basically useless for anyone not at Harvey Mudd, it is a model for how this sort of website should look. Everything is clearly detailed at the perfect level, along with suggestions not just for how to do certain tasks, but also when you might want to do something.

Ubuntu Hacks -- Flash, Java, e17

Mon, 20 Oct 2008 00:00:00 +0000

This must be about the 1 millionth time someone’s written a blog post about this, because I waded through a ton of them to get ones that worked. Which means someday, someone will be wading through 10 million websites trying to get their Flash, Java, and e17 working.

For the browser issues, I found this thread on the Ubuntu forums. In short, you download the first file listed(the Browser Install Script), untar it, and run it. It will download and install the 32-bit libraries, 32-bit browser, and plugins from the set of {Flash, Java, VLC}. The only trick then is rewriting your scripts to use firefox32 instead of just firefox.

iPhone 2.0 - Impressions and Favorites

Tue, 05 Aug 2008 00:00:00 +0000

I know that about every other iPhone owner has posted to their blog about this and almost a month ago already, but I’ll throw my own take into the mix.

I originally planned on waiting a while before installing the second major revision of the iPhone software, but Sharvil tried it almost immediately and told me that I needed to give it a shot. So, I finally upgraded it, and have not been disappointed. Some highlights:

Johnson vs Acronis, Pt 2

Mon, 04 Aug 2008 00:00:00 +0000

Some more ways Acronis fails, just for the record:

Completely useless log files. Error messages are crucial in applications. Acronis says ‘Error creating file’. This could mean:
I couldn’t resolve the FTP server’s hostname
I could resolve the hostname, but could not connect to the FTP port
I could resolve the hostname and connect, but I could not authenticate.
I could resolve, connect, authenticate, but I could not create a new file.
Any number of other things. I could not create a temporary file(Why? No permissions in Acronis’s temporary folder? No hard drive space? It already exists? The folder that should be the temporary folder doesn’t exist?)

Another error we’ve gotten a few times is ‘Failed to process pair script (some hash).’ While it might be useful to someone, it’s not really any help. A lot of the same stuff applies: Is the file corrupt? What’s wrong with it? Improper syntax? The configuration files are XML, they should be editable as such. Point being: Acronis, get your stuff together and write some real, honest error checking code. It’s boring code to write, but it’s boilerplate stuff, and basic. This is what you hire interns for. Please do so. 2. The GUI stinks. Everything is wizard based. This in itself isn’t so much of a problem, but it becomes one with their absolute awful support of stuff like, oh, say, remembering passwords. To see how this is an issue: Say I realize that I want maximum encryption on my backup archives. So, I load up the tasks list, click the one I want to change, click the ‘changed settings’ button. It asks me what I want to backup(2-3 screens), then where I want to back it up to. I tell it the FTP site, and it asks me for my login for that site, then allows me to navigate(keep in mind this is already stored in the configuration file I’m trying to edit, in some encrypted form(hopefully, though I would not bet 2bits on it)). Then, it asks me if I want to sign in to the FTP anonymously or use a username and password. Upon selecting non-anonymously, it asks me for the same username and password. Every single time you try to change any part of the configuration. 3. The command line stinks. There are just a couple of commands, including trueimagecmd and trueimagemnt, but both of them are completely inconsistant on file argument parsing: trueimagemnt uses the more standard UNIX conventions of [-(shortopt)|–(longopt)] (arg) while trueimagecmd uses the completely zany –(longopt):(arg). The command trueimagecmd allows a –verify command, but doesn’t include it in the runtime help or the manual page. None of this seems like too big of a deal, but Acronis is basically the only program that we actual require any GUI at all for on 2 of our 4 main servers. It’d be very nice to have backups and not need a GUI.

Johnson vs Acronis

Sat, 02 Aug 2008 00:00:00 +0000

The story goes something like this: Originally, we used external hard drives plugged in via USB to our servers to back up our servers. This worked well, except that we had two external drives and 7 servers. About this time I started working this job and immediately set up a MediaWiki website for storing documentation and a Mantis Bug Tracker website for storing information about ongoing projects–I think I’ll try to write a post about that at some point as well. Anyways, these two websites originally ran on my personal webserver in Prosser, but we decided to run them on a company-owned server instead, and that this new machine could act as both a backup server and a webserver for those two websites.

Firefox PDF on OSX

Sun, 22 Jun 2008 00:00:00 +0000

June 3 was a fantastic day, though I didn’t know it at the time. The reason? My biggest bitch about Firefox has finally been fixed. The big problem is that Firefox’s OSX program has the default behavior of saving PDFs to Desktop and starting Preview with it. Which isn’t a huge problem, except that if you download the file several times, it just downloads a file with -N appended. This is sortof a pain because in college a lot of teachers call files hw5.pdf, which means that I pretty much have to go to the website each time, or look through about 5 files each time.

Vertigo

Wed, 12 Mar 2008 12:00:00 +0000

No, not that dizzy, spinning sensation. The Firefox extension.

Someone finally caught on to the fact that on a lot of modern computers, we have a lot more horizontal space than vertical. Whoever it was, they must have also been miffed that they never seemed to have room for all of their firefox tabs. So, they made a vertical tab menu, which is actually pretty cool. The standard stuff (ctrl-tab) still works, which is convenient.

Acronis TrueImage

Sat, 30 Jun 2007 14:00:00 +0000

TrueImage is… well… Quite the piece of software.

What I need it to do: Tell the check_backupdisk script that it’s actually running the backup (IE–don’t bitch on nagios) and handle all the mounting/umounting/fscking that it currently does. I think I’ll probably end up putting a wrapper script around it…. but we’ll see.

MRTG

Sat, 30 Jun 2007 13:00:00 +0000

So, another day, another (minor) problem. I had connected to a server with KDE instead of our usual GNOME interface. Which probably would have been fine except that KDE had a screensaver that sucked up the CPU. Which also would have been fine, if it hadn’t been a server that everyone else used. Oops. RRDtool and MRTG would have shown a CPU spike which I could have investigated.

Cacti

Sat, 30 Jun 2007 12:00:00 +0000

So, rrdtool is pretty sweet, but it’s somewhat of a pain to set up. So… we use Cacti. It’s amazing. Supposedly it’ll integrate with Nagios as well, and then all of this data will be at our fingertips. For now, I just set it up on my server that’s probably going to go down shortly. In any case, I feel like this project is somewhat coming together. Nagios is great for what’s going on right now, cacti needs a way to know what’s going on right now–A match made in heaven, or something.

RRDTool

Fri, 29 Jun 2007 14:00:00 +0000

I have apparently been missing out. There’s a really neat little unix tool called ‘rrdtool’ that uses a fixed-size database to look at how something changes over time. I think I’ll use it to monitor server load… and possibly network traffic.

Subversion Bite-sized Projects

Fri, 29 Jun 2007 13:00:00 +0000

Subversion added a tag in their bug-tracking database of projects that are ‘bite-sized’: self-contained, somewhat independent fixes that should happen. I’m not sure any look very interesting at the moment, but I’ll have to come back to it at some point. Bite-sized projects