Sql on traviscj/blog

feeds justification

Tue, 11 Jun 2019 00:00:00 +0000

I realized I’ve left out a major part from my sequence of previous feed-related posts: a justification for why we should bother with a separate feed_sync_id. So let’s give it a shot! The fundamental problem is:

AUTO_INCREMENT ids are assigned in insertion order, but become visible to other threads in commit order.

To see how this causes a problem, consider the interactions and visibilities between three transactions to the same database:

t0: TRX0: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k0", "v0"); COMMIT;
t1: TRX1: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k1", "v1");
t2: TRX2: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k2", "v2");
t3: TRX0: SELECT MAX(id) FROM kv;
t4: TRX2: COMMIT;
t5: TRX0: SELECT MAX(id) FROM kv;
t6: TRX1: COMMIT;
t7: TRX0: SELECT MAX(id) FROM kv;

Here, we have two transactions that both insert a new kv record. The database has to assign an id value to each of those records, because we might be creating other associations to those records in our application code. But other threads – TRX0 in this case – shouldn’t be able to see those records until we COMMIT, and so indeed the SELECT at t=t3 might return 1.

absolutely minimal OLTP to OLAP pipeline

Thu, 25 Apr 2019 00:00:00 +0000

Suppose we have some data in a production OLTP database, and we need to send it to some OLAP database. This post describes one of the simplest approaches, and how to make it productional enough to rely on.

For every table t, we need to:

introduce a new field, updated_at
introduce a new index on that field, so we can get the records that changed after a certain updated_at.

For example,

feed sequences

Tue, 08 Jan 2019 00:00:00 +0000

In the mysql feeds post, I mentioned that the publisher could do

SELECT MAX(feed_sync_id)+1 
FROM kv

to find the next feed_sync_id during the publishing process, but this is actually a really bad idea. (And I knew it at the time, so forgive me for selling lies…)

Republishing

Before we jump into the problematic scenario, I’d like to motivate it with a tiny bit of background.

The republish operation is extremely useful when consumers need to receive updates. It is also extremely simple! A query like

unreasonable wish -- an sql interface to websites

Sun, 06 Jan 2019 00:00:00 +0000

I was searching Amazon Prime for some Earl Grey tea for my wife. I got these results

This was just a basic search for earl grey tea in the Prime Pantry store.

I would love to

SELECT *
FROM products p
WHERE p.prime_pantry = 1
  AND p.keywords = 'earl grey tea'
  AND p.packaging = 'bulk'

I get that this’ll never happen, but man… it’d be nice.

watching progress in mysql

Fri, 04 Jan 2019 00:00:00 +0000

Pretty frequently at work, I end up polling a database with some command like

SELECT MAX(id) FROM my_tbl;
SELECT MAX(id) FROM my_tbl;
SELECT MAX(id) FROM my_tbl;
SELECT MAX(id) FROM my_tbl;
-- .... ad nauseam ....

I’ve eventually noticed a few patterns I use pretty consistently:

estimate ETA by including `NOW()`/`UNIX_TIMESTAMP()`

Generally, the point of hovering over the table is to get an estimate of when it will finish/catch up/whatever. For that, you generally want to include a timestamp in the query output, so when you come back a while later, you can know exactly how much time has elapsed. I might transform the above query into

sharded feeds

Mon, 29 Oct 2018 18:30:00 +0000

Suppose that:

our humble KV feed sees a lot of traffic.
someone needs to consume our KV feed with multiple threads.

data

The first step is to introduce a notion of “shards” into our data model:

ALTER TABLE `kv`
  ADD COLUMN `shard` INT(11) DEFAULT '0',
  ADD INDEX `k_fsi_s` (`feed_sync_id`, `shard`);

publishing

We don’t need to alter the publishing until the publishing itself is too slow to work with a single thread, but this introduces a lot of complications, so let’s just hold off for now.

history preserving data models

Mon, 02 Jul 2018 11:34:39 +0000

Start with a super simple data model:

CREATE TABLE kv (
  id BIGINT(22) NOT NULL AUTO_INCREMENT,
  k VARCHAR(255) NOT NULL,
  v LONGBLOB NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY u_k (`k`)
) Engine=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin

Suppose we want to audit “changes” to this data model.

Approach 1: `kv_log`

add data model like

CREATE TABLE `kv_log` (
  id BIGINT(22) NOT NULL AUTO_INCREMENT,
  changed_at TIMESTAMP NOT NULL,
  k VARCHAR(255) NOT NULL,
  old_v LONGBLOB NOT NULL,
  new_v LONGBLOB NOT NULL,
  
)

current value query: unchanged

mysql feeds

Fri, 29 Jun 2018 00:00:00 +0000

At work, we use a pattern called feeds that gets an incredible amount of work done. I’ve been wanting to describe it here for quite a while, and now seems as good of time as any.

The basic premise is: You have a service A with some data that other “consuming” services B, C, and D want to find out about. Maybe the data is payments, maybe it’s support cases, maybe it’s password changes… whatever. The other services might include your data warehouse, some event listeners, whatever.

osquery

Fri, 12 Jan 2018 08:43:17 +0000

I’ve known about [osquery][] for a while, but recently spent some time digging around in it. [osquery]: https://osquery.io The basic idea is to provide a consistent SQL interface to a bunch of system data, instead of learning idiosyncrasies of individual commands (which themselves vary across operating systems). My hacker buddy Sharvil has worked on osquery a fair bit and explained to me a couple years ago that it actually uses [sqlite][]’s [virtual table functionality][vtab] to provide the interface – it’s a fascinating and brilliant project! [sqlite]: https://sqlite.org/ [vtab]: https://sqlite.org/vtab.html

relational java explorations with sqlite3 vtab

Fri, 05 May 2017 20:10:28 +0000

One idea i have been mulling over lately is exposing a java codebase relationally, with queries like:

SELECT *
FROM classes c
JOIN annotations a
  ON a.class_id = c.id
WHERE a.name = 'MyAnnotationClass';

The idea here is that you could search a codebase by pretending that you had tables for a whole bunch of things like:

classes
annotations
variables
literals
methods

This would be a useful thing because instead of relying on an IDE like IntelliJ to do “find usages” operations, you could actually script those interactions. Admittedly, of course, Java has some pretty sophisticated reflection APIs to find this sort of stuff too, but it seems like exploring the structure could be much easier writing Java code to traverse those trees.

filter vs spec (draft)

Wed, 18 Jan 2017 00:00:00 +0000

Consider a silly data model to store data about cities like

message City {
  optional string city_name = 1;
  optional string state = 2;
  optional int32 population = 3;
  optional int32 year_founded = 4;
  // ... presumably others :-)
}

and some sample data like:

[
  {"city_name": "Portland", "state": "OR", "population": ...},
  {"city_name": "Portland", "state": "ME", "population": ...},
  {"city_name": "Springfield", "state": "FL", "population": ...},
  {"city_name": "Springfield", "state": "IL", "population": ...},
  {"city_name": "Springfield", "state": "CO", "population": ...}
]

There are some useful entities we can define: (DRAFT NB: don’t read too much into the matcher vs filter lingo.)

logging

Fri, 26 Sep 2014 00:00:00 +0000

In grad school, I spent a lot of time writing code that read output from nonlinear optimization solvers, and tried to do useful things with it. A much better way to do that is called “structured logging”, an idea I experimented with a bit during grad school. It has also been coming up in my working life, so I wanted to delve into it a bit deeper. For a quick introduction, check out Thoughts on Logging. For a lot longer introduction, see The Log: What every software engineer should know about real-time data unifying.

A Quick MySQL Reference

Wed, 27 May 2009 00:00:00 +0000

I got frustrated with not being able to write MySQL because I don’t do it often enough to be seeing it in my nightmares like MATLAB. But recently my datasets got annoyingly huge and it seemed like SQL might be a boon to me. So I set out to write a quick little program, and ended up writing this little reference along the way.

Also, Sharvil Shah contributed some comments to it, so thanks to him for that!