You're 100% right, eventually you're inundated with the marketing bullcrap. People will throw nearly meaningless words and phrases at you just to gain some added leverage. But, I wouldn't worry about it too much. You're a smart person. Try to look past that stuff. You will find what you're looking for but it won't be in the way you expect. Just look at this statement alone: The "conventional sense"... lol, as if there is an "HFT" authority that certifies developers?!!? On top of it the qualification has to be "firms" because there's no way a person can do >2% ADV alone. And, "any given market"... which markets are we talking about? Thinly traded pink sheets? So, with all due respect, after you pull back the curtains, @Databento is saying that 1/2 of their engineers previously had jobs in finance. Of course, maybe they got fired from their old jobs because they weren't very good. We can't be sure. The mathematical reality is, HFT firms "make up the low margins with incredibly high volumes"(1). So they trade quality for quantity. It's a reasonable tradeoff but it's not necessarily something to brag about. It's not much different than when discount brokers emerged and competed on commission. Technology has changed history but that history still rhymes. Not too long ago the big banks on wall st. used to clear transactions by exchanging physical certificates at the end of the day. Then, some smart person decided to use a bicycle to get from bank to bank faster and they became the first HFT trader! Lol Put things in perspective. When people have nothing of substance left to offer they're just going to try and bullshit you, gaslight you, whatever it takes to get what they want... a good employee at a cut-rate price. The reason they need the employee is because they can't actually do the job themselves. This is simply the dynamic nature of economic specialization and technological productivity gains. I could tell you I started my career at a bank founded by Alexander Hamilton when Modula-2 on VAX/VMS was "hot technology". I could say I consulted firms up and down Manhattan from the WTC (when it still existed) all the way to mid-town. I built software for the military, and I worked for years at the foremost telecom research lab in the world... because Ma Bell's got the ill communication. None of that amounts to a hill of beans unless I'm trying to convince you of something. Try not to fall for the superficial showboating, we all put our pants on one leg at a time. And, you seem to have a genuine curiosity... you're the exact type of person a giant vampire squid wants to latch on to. Keeping your humanity and soul is the challenge. As far as the $$ are concerned, at some point, more will always be printed. Money is literally made of linen, zinc and a little bit of copper... basically dirt. It is the world's ultimate marketing tool (with the possible exception of sex). Keep doing the rigorous analysis true science requires and trust your judgement. You'll have cafe society by the short-hairs before you know it
Of course, there may be. But that stuff alone doesn't dictate which OLAP or OLTP systems are any good or get implemented. Or even what an "insider" might consider "genuine" HFT (if there is such a thing). Sure... lawyers, legal, and compliance people need to make money too. Sit them in front of a computer though and they will write natural language Word(tm) documents, full of ambiguous language open to interpretation, which mostly matters to their political allies. Hell, those people are actually pulling down huge $$. I'm not even saying it's bad, just that it's way outside the scope of the OPs question and job description they referred to. Can you take the document you're referring to and actually discern which systems are "legitimate" HFT with 100% certainty? Esp. without a sovereign legislative body to back you? If so... you're a much better person than I am and I certainly commend you for that! Who's doing HFT and who isn't? What a joke, it's the same argument primary school children make to determine who they want to take to a dance. Anyone who trades on Robinhood with an iPhone could claim they do some kind of "HFT" because an A9 chip is clocked at nearly 2GHz. There's little point in splitting hairs as far as I'm concerned. Now I'm going away. I've had enough of this thread which, like many ET threads, has reached childish and cockamamie pissing contest levels. It was answered a long time ago: yes you can sometimes gain meaningful performance by eliminating TCP/IP in some circumstances.
What is OLAP, OLTP? The following diagrams are what I've currently mapped out. The mind map is too large as a single map, it's broken out with some gaps, and some nodes are not expanded but this is the general idea. Latency arbitrage is but one technique being implemented. HFT Agent Categories and how they are defined by trading volume and EOD position size. Top HFT firms that fall into the above categories, some are hybrid. Current Categories of HFT techniques The NYSE SLP Prop, SLMM and the HFT-A, HFT-B's haven't been completely mapped and verified yet. If anyone has info to add, criticisms or corrections; share some references.
I don't have any color on the SIPs' implementation. We've heard that they use a Solarflare kernel bypass stack for the latest iteration of it, but that's purely hearsay. (Edit: Last we checked, they were in the 18-30 mics median range, which is on the high side for this, but on the low side with generic kernel networking stack. So considering the scale of their distribution and everything, the hearsay seems accurate.)
@spy No one's disagreeing with you that there's no formal definition of "HFT". All I said is that my teammates worked at several of the firms named here, among others, all of which pass the 2% ADV threshold I mentioned: And yes we get any further cynicism: they could've been doing back office contract work for 3 months, the regulators misidentify what firms do, their LinkedIn profiles could be fake, who sets "convention"... But going that far is losing the context of what I originally said. Recall that I was just answering @earth_impersonator's question for any "official info" of "how kernel bypass is used in HFT". It's my tongue-in-cheek way of saying that no, there's no "official info" and no such "official" exists. But if he really wants some additional warranty to my post, that's all I could offer, a bunch of names of firms we've worked at. I have marketing posts but there's no marketing spin to this one; if anything I'm on your side in answering this with a tinge of sarcasm.
Well, VMA, Onload/ef_vi and Mellanox DPDK are the most common. Keep in mind userspace implies outside the kernel, so they're not technically "Linux", but "Linux-compatible".