Distributed-Systems on traviscj/blog

Evolution of a Get Endpoint

Tue, 24 Jun 2025 19:47:52 -0500

My work project started pretty simple: there was a simple GetXyz endpoint that just looked up the xyz record in the database by a unique key & returned it. How complicated could it be?

It was a straightforward generalization of an old GetAbc functionality, it was really only used by oncall engineers through an admin console, it shouldn’t have been too big of a deal.

Ok, but then it outgrew its original datastore, so a separate service had to be created. We thought about migrating clients, but at the time it seemed faster to just implement once on our side & have clients continue calling us; essentially encapsulating the separate service as an implementation detail. But as part of the traffic swing, we figured it should support either the local or the remote read path.

Transacter and Intent/Result

Wed, 15 Jun 2022 08:07:47 -0500

At SQ, we had a family of Transacter interfaces:

public interface Transacter {
  void transaction(Function<Session, Void> session);
  
  public interface Session {
    DSLContext dsl();
  }
}

where DSLContext is a jOOQ concept & is the handle for doing work against the database. Then this would be used in some class like

public class KvDao {
  @Inject Transacter transacter;
  
  void put(String ns, String k, String v) {
    transacter.transaction(session -> {
        KvRecord record = new KvRecord(ns, k, v);
        session.dsl().insertInto(KV).set(record).execute();
    });
  }
}

The parameters ns, k, and v are shorthand for namespace, key, and value, respectively. We’ll show how these might be used together shortly!

feeds justification

Tue, 11 Jun 2019 00:00:00 +0000

I realized I’ve left out a major part from my sequence of previous feed-related posts: a justification for why we should bother with a separate feed_sync_id. So let’s give it a shot! The fundamental problem is:

AUTO_INCREMENT ids are assigned in insertion order, but become visible to other threads in commit order.

To see how this causes a problem, consider the interactions and visibilities between three transactions to the same database:

t0: TRX0: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k0", "v0"); COMMIT;
t1: TRX1: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k1", "v1");
t2: TRX2: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k2", "v2");
t3: TRX0: SELECT MAX(id) FROM kv;
t4: TRX2: COMMIT;
t5: TRX0: SELECT MAX(id) FROM kv;
t6: TRX1: COMMIT;
t7: TRX0: SELECT MAX(id) FROM kv;

Here, we have two transactions that both insert a new kv record. The database has to assign an id value to each of those records, because we might be creating other associations to those records in our application code. But other threads – TRX0 in this case – shouldn’t be able to see those records until we COMMIT, and so indeed the SELECT at t=t3 might return 1.

absolutely minimal OLTP to OLAP pipeline

Thu, 25 Apr 2019 00:00:00 +0000

Suppose we have some data in a production OLTP database, and we need to send it to some OLAP database. This post describes one of the simplest approaches, and how to make it productional enough to rely on.

For every table t, we need to:

introduce a new field, updated_at
introduce a new index on that field, so we can get the records that changed after a certain updated_at.

For example,

feed sequences

Tue, 08 Jan 2019 00:00:00 +0000

In the mysql feeds post, I mentioned that the publisher could do

SELECT MAX(feed_sync_id)+1 
FROM kv

to find the next feed_sync_id during the publishing process, but this is actually a really bad idea. (And I knew it at the time, so forgive me for selling lies…)

Republishing

Before we jump into the problematic scenario, I’d like to motivate it with a tiny bit of background.

The republish operation is extremely useful when consumers need to receive updates. It is also extremely simple! A query like

tradeoffs

Tue, 01 Jan 2019 00:00:00 +0000

Life is (just) a series of tradeoffs.

DawnAnn “Mom” Johnson

I heard this so many times growing up that I think I actually stopped hearing it. But recently, I realized that it’s never been more pertinent than to my current day-to-day life as a software engineer.

The fundamental tradeoff we make on the Risk Systems team is trading off between

how much fraud we catch (and symmetrically, miss!)
how many false positives we incur to catch that much fraud

Those false positives running rampant can inhibit growth (in terms of interested customers) of the product we’re trying to protect, but letting too much fraud through can make a product too expensive (and even liable to be shut down!)

sharded feeds

Mon, 29 Oct 2018 18:30:00 +0000

Suppose that:

our humble KV feed sees a lot of traffic.
someone needs to consume our KV feed with multiple threads.

data

The first step is to introduce a notion of “shards” into our data model:

ALTER TABLE `kv`
  ADD COLUMN `shard` INT(11) DEFAULT '0',
  ADD INDEX `k_fsi_s` (`feed_sync_id`, `shard`);

publishing

We don’t need to alter the publishing until the publishing itself is too slow to work with a single thread, but this introduces a lot of complications, so let’s just hold off for now.

Feeds as cache invalidation mechanism

Wed, 03 Oct 2018 02:31:43 +0000

One really cool use of feeds we’ve realized is that it gives a very efficient mechanism for application code to load the most recent versions of a table into memory. The basic idea is:

Set it up as a usual feed published table with an appropriate index on feed_sync_id.
Either alongside or within the cache, represent the latest loaded feed_sync_id.
Set up a cronjob/etc that reads the latest feed_sync_id and compares it to the cache’s feed_sync_id.
If they differ, reload the cache.
Ensure that all changes set feed_sync_id to null!

This works really well because the feed_sync_id in the database only gets updated on changes, so the reload cronjob mostly is a no-op. This means we can reload very frequently!

cross-dc sync with feed published KV

Tue, 10 Jul 2018 11:34:39 +0000

It’s been fun describing the feeds framework we use at Square. Today we’ll dive into a concrete problem:

We’ll stick with the feed-published kv table again.
We want two instances of some application code to bidirectionally synchronize the writes that happened on their instance to the other.
Eventually consistent is ok.

First, a bit of justification, though: I use this KV table to remember things I might have to look up without usual context, like my motorcycle’s license plate number, or that one weird python snippet I can never remember. I also have a whole slew of them at work – a bunch of random representative IDs for a bunch of things in our systems that I use from time to time. I also use a bunch of these as todo items at work, but that happens to work differently and is a topic for a future blog post :-)

mysql feeds

Fri, 29 Jun 2018 00:00:00 +0000

At work, we use a pattern called feeds that gets an incredible amount of work done. I’ve been wanting to describe it here for quite a while, and now seems as good of time as any.

The basic premise is: You have a service A with some data that other “consuming” services B, C, and D want to find out about. Maybe the data is payments, maybe it’s support cases, maybe it’s password changes… whatever. The other services might include your data warehouse, some event listeners, whatever.

CBCL: The Common Business Communication Language

Mon, 12 Dec 2016 23:05:28 +0000

I recently came across McCarthy’s CBCL paper on Hacker News. He presents a Lisp notation as a format for sharing requests between different software on different computers. He calls out

(PLEASE-RESERVE (EPSILON (x) 
  (AND
    (IS-FLIGHT x) 
    (DEPARTS MONDAY) 
    (ARRIVES (BEFORE WEDNESDAY)))))

where $\epsilon x$ is an operator maps a predicate to a value matching the predicate, and considers it an improvement on the “iota” definite description operator, which seems to be an operator that returns the unique value matching the predicate. It seems interesting to me that the unique match operator is not considered as useful as the arbitrary match operator – unique keys actually seem critical to a lot of my work with data models. Even more fascinating, though, is that this looks like an anonymous function!