Applied Knowledge

about

Fri, 23 Oct 2015 11:08:36 -0400

About

Melding computer science, economics, finance, and mathematics to create innovative solutions. ARD Capital, LLC.

email: alex [at] hoo dot la

github: astocko

twitter: @astocko

dynamic x86-64 generation & jit

Sun, 12 Jul 2015 11:54:19 -0400

My linear genetic programming engine relies on dynamic generation of x86-64 machine code to create thousands of functions which are evolved over generations. My initial version of the code generation backend was a small and simple C library which was capable of assembling 30 core instructions to a memory mapped region. Using the Intel manuals, the X86 Opcode and Instruction Reference, and Agner Fog’s tables, I patched together a usable library. Adding more features to the GP engine necessitated extending the machine code generator. When the complexities of implementing the later SSE opcodes and AVX extensions became a major timesink, I began looking around for replacements.

AsmJit is a stellar x86/x64 JIT and Remote Assembler implemented in C++. It has support for the entire Intel instruction set up to AVX2, features a high level cross-platform compiler which makes memory addressing, labels, and the stack painless. Moreover, AsmJit has a wonderful logging engine and error handling mechanism to enable debugging generated code.

Example

#include <asmjit/asmjit.h>

using namespace asmjit;

int main(int argc, char* argv[]) {
  JitRuntime runtime;
  X86Compiler c(&runtime);

  // Function returning 'int' accepting pointer and two indexes.
  c.addFunc(kFuncConvHost, FuncBuilder3<int, const int*, intptr_t, intptr_t>());

  X86GpVar p(c, kVarTypeIntPtr, "p");
  X86GpVar aIndex(c, kVarTypeIntPtr, "aIndex");
  X86GpVar bIndex(c, kVarTypeIntPtr, "bIndex");

  c.setArg(0, p);
  c.setArg(1, aIndex);
  c.setArg(2, bIndex);

  X86GpVar a(c, kVarTypeInt32, "a");
  X86GpVar b(c, kVarTypeInt32, "b");

  // Read 'a' by using a memory operand having base register, index register
  // and scale. Translates to 'mov a, dword ptr [p + aIndex << 2]'.
  c.mov(a, ptr(p, aIndex, 2));

  // Read 'b' by using a memory operand having base register only. Variables
  // 'p' and 'bIndex' are both modified.

  // Shift bIndex by 2 (exactly the same as multiplying by 4).
  // And add scaled 'bIndex' to 'p' resulting in 'p = p + bIndex * 4'.
  c.shl(bIndex, 2);
  c.add(p, bIndex);

  // Read 'b'.
  c.mov(b, ptr(p));

  // a = a + b;
  c.add(a, b);

  c.ret(a);
  c.endFunc();

  // The prototype of the generated function changed also here.
  typedef int (*FuncType)(const int*, intptr_t, intptr_t);
  FuncType func = asmjit_cast<FuncType>(c.make());

  // Array passed to 'func'
  const int array[] = { 1, 2, 3, 5, 8, 13 };

  int x = func(array, 1, 2);
  int y = func(array, 3, 5);

  printf("x=%d\n", x); // Outputs "x=5".
  printf("y=%d\n", y); // Outputs "y=18".

  runtime.release((void*)func);
  return 0;
}

References

column oriented databases & formats

Thu, 30 Apr 2015 11:51:37 -0400

Column oriented databases have become a staple in the big data analytics scene. This article will analyze the in-memory and on-disk formats of kdb+.

kdb+ is a high performance timeseries database implemented in the K language and widely deployed within the financial sector. Q is a wrapper around K to provide an easier interface to the database. kdb+ is implemented as a column store which makes operations like aggregations extremely fast.

###Architecture While kdb+ is both an in-memory and a persistent disk based column oriented data store, the in-memory representation and the on-disk file formats are quite simple. Tables in kdb+ are essentially ordered dictionaries of typed columns sorted on one or more columns. In python an in-memory table of stock symbols, dates, and their closing prices could be implemented as follows:

from collections import OrderedDict
daily = OrderedDict(
	[
    	('symbol', [1, 1, 1, 1, 2, 2, 2, 2]), 
        ('date', [0, 0, 0, 0, 0, 0, 0, 0]), 
        ('prices', [90.01, 91.02, 92.03, 93.04, 60.01, 61.02, 62.02, 63.03])
    ])

kdb+ makes tradeoffs in order to improve performance, two of which I have implemented in the table above and will explain below.

Strings

Strings stored in tables are symbolized and mapped to integers in a bidirectional dictionary. This tradeoff enables very fast selects on columns like a stock symbol as we only have to seek for a integer (8bit to 64bit depending on the size of the symbol dictionary). For our table the symbol map could be implemented as:

symbols = {
	1 : 'GOOG',
    2 : 'YHOO'
}
symbols2 = {
	'GOOG' : 1,
    'YHOO' : 2
}

Dates

Dates in kdb+ are also stored as integers and calculated as the distance from the epoch 01-01-2000. So, in our example implementation, 0 represents 01-01-2000.

Disk File Format

kdb+ can export tables in multiple formats. First, the in-memory table can be written to a binary file in row format:

uint8,uint32,float
1,0,90.01
1,0,91.02
1,0,92.03
1,0,93.04
2,0,60.01
2,0,61.02
2,0,62.03
2,0,63.03

Secondly, the table can be splayed and each column is written to its own binary file:

symbols.col (uint8): [1, 1, 1, 1, 2, 2, 2, 2]
dates.col (uint32): [0, 0, 0, 0, 0, 0, 0, 0]
prices.col (float): [90.01, 91.02, 92.03, 93.04, 60.01, 61.02, 62.02, 63.03]

Thirdly, the table can be partitioned by a column value and then splayed:

symbol_1 directory:
	dates.col (uint32): [0, 0, 0, 0]
    prices.col (float): [90.01, 91.02, 92.03, 93.04]
symbol_2 directory:
	dates.col (uint32): [1, 1, 1, 1]
    prices.col (float): [60.01, 61.02, 62.02, 63.03]

Querying

When kdb+ loads a table, it memory maps (mmap on linux) either the single table file, each column for a splayed table, or all columns for a partitioned table. Every splayed column is mapped as a continuous array of a data type and accessed accordingly. kdb+ also loads the symbol dictionary for symbol lookups. In our example, the table is sorted on the symbol and date columns.

Select price WHERE symbol = ‘GOOG’

Symbol is a symbol column, so GOOG is looked up and mapped to 1
In the single table file layout, we perform a binary search on the array of structures and find all rows with a symbol value of 1 and return them
In the splayed table layout, we perform a binary search and record the indices of the symbol array where symbol = 1. Then we index into the memory mapped column (array) for price and use our recorded indices to select the matching entries. i.e. price[0:3].
In the partitioned table, we perform the same steps as in the splayed table layout within the symbol_1 directory.

Splaying a table enables us to seek through less data, particularly when dealing with wide row data sets, in order to find the records we want to query. As data size increases, partitioning tables on a column like symbol or date further reduces I/O overhead. Another benefit of single type columns is that aggregations such as calculating the mean or sum become single vector operations.

Concluding Remarks

kdb+ supports many more features, including indexing and compression, but the basic architecture is as described. The primary contributing factors to its performance are in its column oriented structure and data representation format. In a real world deployment, real-time data is typically appended to an in-memory table and then archived to a splayed/partitioned table sorted on one or multiple columns. Often, the same table is stored in varying sorted orders to further improve the performance of specific queries. Similar to big data database systems like Cassandra, tables are modeled to fit commonly executed queries.

c macros & terse c

Sun, 18 Jan 2015 11:48:49 -0400

C macro magic is typically frowned upon in the developer community. Heavy preprocessor usage can make code hard to read, expansion difficult to understand, and the resulting code error prone. While all these things are true, macros do have nice uses cases such as in the implementation of high performance generic containers in C (see klib).

I have also encountered significant code bases in which macros where used as a semantic overlay easing the programmer’s transition from their preferred language into C. One such example is Kx System’s kdb+, Arthur Whitney’s high performance database system which is widely deployed in the financial industry.

Arthur Whitney designed the K and Q programming languages and his love for terse code has translated into their implementations. Following is a short code block from one of kdb+’s include files k.h:

#ifndef KX
#define KX
typedef char*S,C;typedef unsigned char G;typedef short H;typedef int I;typedef long long J;typedef float E;typedef double F;typedef void V;
#ifdef __cplusplus
extern"C"{
#endif
#ifndef KXVER
#error "Set KXVER=3 for kdb+3.0 or standalone c-api after 2011-04-20. Otherwise set KXVER=2"
#endif
#if KXVER>=3
typedef struct k0{signed char m,a,t;C u;I r;union{G g;H h;I i;J j;E e;F f;S s;struct k0*k;struct{J n;G G0[1];};};}*K;
typedef struct{G g[16];}U;
#define kU(x) ((U*)kG(x))
#define xU ((U*)xG)
extern K ku(U),ktn(I,J),kpn(S,J);
extern I setm(I);
#define DO(n,x)	{J i=0,_i=(n);for(;i<_i;++i){x;}}
#else
typedef struct k0{I r;H t,u;union{G g;H h;I i;J j;E e;F f;S s;struct k0*k;struct{I n;G G0[1];};};}*K;
extern K ktn(I,I),kpn(S,I);
#define DO(n,x)	{I i=0,_i=(n);for(;i<_i;++i){x;}}
#endif
#ifdef __cplusplus
}
#endif
//#include<string.h>
// vector accessors, e.g. kF(x)[i] for float&datetime
#define kG(x)	((x)->G0)
#define kC(x)	kG(x)
#define kH(x)	((H*)kG(x))
#define kI(x)	((I*)kG(x))
#define kJ(x)	((J*)kG(x))
#define kE(x)	((E*)kG(x))
#define kF(x)	((F*)kG(x))
#define kS(x)	((S*)kG(x))
#define kK(x)	((K*)kG(x))
.....

A cursory glance reveals that types are aliased with very short identifiers. Accessor macros are provided for different struct values and there is also a DO loop.

The source file c.c shows how these macros are used and hints at what the entire kdb+ code base probably looks like.

#define Q(e,s) {if(e)return printf("%s\n",s),-1;}      //error

int main(){K x,y;int c=khpu("localhost",5001,"usr:pwd");
 Q(c<0,"connect")Q(!c,"access")
 Q(!k(-c,"`trade insert(12:34:56.789;`xx;93.5;300)",(K)0),"err") // statement insert
 Q(!(x=k(c,"select sum size by sym from trade",0)),"err")    // statement select
 Q(!(x=ktd(x)),"type")                   // flip from keyedtable(dict)
  y=x->k;                                // dict from flip
  y=kK(y)[1],printf("%d cols:\n",y->n);  // data from dict
  y=kK(y)[0],printf("%d rows:\n",y->n);  // column 0
  printf("%s\n",kS(y)[0]);               // sym 0 
  r0(x);                                 // release memory
 x=knk(4,kt(1000*(time(0)%86400)),ks("xx"),kf(93.5),ki(300)); // data record
// DO(10000,Q(!k(-c,"insert",ks((S)"trade"),r1(x),(K)0),es)) // 10000 asynch inserts
// k(c,"",(K)0); // synch chase
// return 0;
 Q(!k(-c,"insert",ks("trade"),x,(K)0),"err")                           // parameter insert
 Q(!(x=k(c,"{[x]select from trade where size>x}",ki(100),(K)0)),"err") // parameter select
  r0(x);
 close(c);
 return 0;}
*/

The code is terse, but for someone with Arthur’s background, easy to read and write. Essentially he is mapping his stylistic preference for APL based languages onto C. While there is a certain elegeance to his approach, I do not recommend copying his style and becoming a macro addict in your large production applications which need to survive maintenance cycles.

If you’re interested in exploring the world of array programming languages, J is a powerful open source implementation.

c resources

Sat, 03 Jan 2015 11:44:44 -0400

For an updated list, please see https://github.com/astocko/c_compendium.

Books

Learn C The Hard Way - Zed Shaw’s book, a great resource to learn C
21st Century C: C Tips from the New School - A book to bring old school C programmers into the 21st Century

Articles

Why should I have written ZeroMQ in C, not C++ - A set of articles on the merits of C over C++ from the Martin Sústrik’s point of view; controversial but interesting

Libraries

Apophenia - Apophenia is an open statistical library for working with data sets and statistical or simulation models. It provides functions on the same level as those of the typical stats package (such as OLS, probit, or singular value decomposition) but gives the user more flexibility to be creative in model-building
GNU Scientific Library - “The library provides a wide range of mathematical routines such as random number generators, special functions and least-squares fitting. There are over 1000 functions in total with an extensive test suite.”
Klib: a Generic Library in C - A macro heavy C library providing containers and other utilities. Avoids the use of void* for performance.
Mongoose - An embeddable C http server
Nanomsg - An evolution of ZeroMQ, a high performance brokerless message queue
SFMT - SIMD version of the Mersenne Twister random number generator. Floating point and integer versions
TinyCThread - Small, portable implementation of the C11 threads API

Build Systems

CMake - CMake is a family of tools designed to build, test and package software
Ninja - Ninja is a small build system with a focus on speed. It differs from other build systems in two major respects: it is designed to have its input files generated by a higher-level build system, and it is designed to run builds as fast as possible
tup - Tup is a file-based build system for Linux, OSX, and Windows

Tools

Cog - Code generation tool
YouCompleteMe - VIM code completion for C-family and other languages
Warp - Alternative performance oriented C/C++ preprocessor implemented in D by Facebook

reboot

Tue, 23 Dec 2014 11:14:55 -0400

I’ve decided to reboot my blog and will be writing about my thoughts as they pertain to computer science, economics, and finance.