Many companies discover their best products unintentionally. WhatsApp began as a simple status-update app, and only after a large portion of users started using status messages to communicate did the team realize messaging was the real value. Slack started as a multiplayer game called Glitch, but the founders soon noticed that the internal chat tool they had built for the team was far more useful than the game itself. Post-it Notes originated from a failed attempt to create a strong adhesive, which unexpectedly became a wildly successful product. You get the idea.

The TP-Link HS1xx line only rose to fame among home-lab enthusiasts and hackers after George Georgovassilis inspected its network traffic with Wireshark. Hidden in plain sight was exactly what people had been wishing for: a simple, easy-to-use API. That discovery transformed the device from an ordinary smart plug into a cult favorite.

Here's a tiny Java implementation you might find useful. If you decide to try it, let us know in the comments what you’re controlling. At wasteofserver, we use it to manage several electric heaters. A bunch of ESP32 continuously monitors room temperature through thermal sensors, based on that data we control HS100 plugs to switch the heaters on or off as needed. This gives us far more consistent temperature control than the heaters' built-in thermostats can provide.

We'll share that code eventually, but for now, go grab a few HS110 plugs, recently rebranded Kasa Smart Plug, and start automating your home! 😉

package org.frankie.util;

import java.io.DataInputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.Socket;
import java.util.Base64;

public class TpLinkClient {

    private static final String PAYLOAD_ON = "AAAAKtDygfiL/5r31e+UtsWg1Iv5nPCR6LfEsNGlwOLYo4HyhueT9tTu36Lfog==";
    private static final String PAYLOAD_OFF = "AAAAKtDygfiL/5r31e+UtsWg1Iv5nPCR6LfEsNGlwOLYo4HyhueT9tTu3qPeow==";
    private static final String PAYLOAD_SYSINFO = "AAAAI9Dw0qHYq9+61/XPtJS20bTAn+yV5o/hh+jK8J7rh+vLtpbr";
    private static final String PAYLOAD_EMETER = "AAAAJNDw0rfav8uu3P7Ev5+92r/LlOaD4o76k/6buYPtmPSYuMXlmA==";
    private static final int DEFAULT_HS100_PORT = 9999;


    public static String on(String ip) throws IOException {
        return sendToPlug(ip, DEFAULT_HS100_PORT, PAYLOAD_ON);
    }

    public static String off(String ip) throws IOException {
        return sendToPlug(ip, DEFAULT_HS100_PORT, PAYLOAD_OFF);
    }

    public static String sys_info(String ip) throws IOException {
        return sendToPlug(ip, DEFAULT_HS100_PORT, PAYLOAD_SYSINFO);
    }

    public static String emeter(String ip) throws IOException {
        return sendToPlug(ip, DEFAULT_HS100_PORT, PAYLOAD_EMETER);
    }


    /**
     * Sends a Base64 encoded command to the plug and returns the raw response bytes.
     * <p>
     * Method is synchronized as IoT hardware is generally low-spec, and there's really not
     * a valid case for concurrent access
     *
     * @param ip            The IP address of the plug
     * @param port          The port (usually 9999)
     * @param base64Payload The Base64 encoded command string
     * @return The response from the plug
     */
    public static synchronized String sendToPlug(String ip, int port, String base64Payload) throws IOException {
        byte[] commandBytes = Base64.getDecoder().decode(base64Payload);

        try (Socket socket = new Socket(ip, port)) {
            socket.setSoTimeout(5000); // Timeout to avoid hanging forever

            try (OutputStream out = socket.getOutputStream();
                 DataInputStream in = new DataInputStream(socket.getInputStream())) {

                out.write(commandBytes);
                out.flush();

                // The TP-Link protocol response starts with a 4-byte big-endian length header.
                // Reading this allows us to know exactly how much data to expect.
                int length = in.readInt();

                byte[] response = new byte[length];
                in.readFully(response);

                return decodeResponse(response);
            }
        }
    }


    private static String decodeResponse(byte[] data) {
        int key = 171;
        StringBuilder sb = new StringBuilder();
        for (byte b : data) {
            // MASKING WITH 0xFF IS CRITICAL, treats the byte as unsigned (0 to 255) preventing negative numbers
            int nextKey = b & 0xFF;
            int decrypted = key ^ nextKey;
            key = nextKey;
            sb.append((char) decrypted);
        }
        return sb.toString();
    }

}

If you're here, you probably already know what ANSI escape sequences are.

You output something like the code below, and it prints in color:

System.out.println("world is black and white");
System.out.println("\u001B[34m" + "world has color" + "\u001B[0m");

We don’t often manually write System.out directly to the console. More likely, you’re using something like Logback to manage your output and have configured logback.xml for color output.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyymmdd HH:mm:ss.SSS} %highlight(%-5level) [%blue(%t)] %yellow(%C): %msg%n%throwable</pattern>
        </encoder>
    </appender>

    <root level="trace">
        <appender-ref ref="CONSOLE"/>
    </root>
</configuration>

To get color to work on Windows, you need to install Jansi:

<dependency>
    <groupId>org.fusesource.jansi</groupId>
    <artifactId>jansi</artifactId>
    <version>2.4.1</version>
</dependency>

If you only install AnsiConsole without enabling pass-through, it will stop working in IntelliJ. To get it working on both Windows and IntelliJ, you need to do both; install AnsiConsole on startup and allow codes to pass through.

    public static void main(String[] args) {
        // for colors to work in Windows, linux and IntelliJ
        System.setProperty("jansi.passthrough", "true");
        AnsiConsole.systemInstall();

        log.trace("trace");
        log.debug("debug");
        log.info("info");
        log.warn("warn");
        log.error("error");
    }

Hope it helps. I, for sure, know that'll I'll be here in a couple of months, rechecking how to do this once again! ;)

While I'm fortunate enough to develop software for state-of-the-art companies, I've always been the "printer guy", a badge I wear with pride.

And maybe that's the reason why, for more than a quarter of a century, people have turned up with storage medium to salvage. I never did it professionally, it’s just something I enjoy and, more often than not, I’m able to help.

The process

My main approach is two-fold:

• capture what you can from the dying medium
• try to recreate the data

Sometimes, just by adjusting read tolerance, you're able to capture everything from a disk Windows or Mac reject, as the OS times out faster than the dying medium can respond. I've had a disk hooked to a box for 7 months; 100% recovery success!

Sometimes you'll have chunks of data missing, but one of the File Allocation Table / Master File Table / Container superblock / root B-tree are still there and so most data is salvageable with both filenames and tree location.

At the extreme, you’re left with nothing but some of the raw data. All you can do is carve it out - essentially reading every byte from the medium, identifying known headers like JPG (FF D8) or MKV (1A 45 DF A3), and proceed to capture all sequential data until the end of the file. If for any reason, the file is fragmented, carving will obviously fail.

The call from Pierre Zago

Frankie, this hasn't happened before, I'm generally cautious... I don't know what I was thinking - I've accidentally formatted an SD card with audio for an entire session. Worse, I've saved some files onto it!

Pierre is an incredibly talented comedian and an extraordinary actor. While his work is multifaceted, he became widely known for street sketches, some of them absolutely universal. Just check the one below where he simply says "excuse-me".

With that simple action, Pierre had joined the club. Everyone messes up. Even Pixar. Don't worry, I said, while taking the card, the overwritten contents are gone, but given that it's a microphone formatted card, even without FAT tables we should be able to carve out most of whatever you've recorded as data will probably be sequential.

Little did I know, this would turn out to be one of the most interesting toy projects I’ve worked on in a while. The last time I had this much fun was rebuilding a hardware RAID-0 that contained the masters for an album by 'Os Azeitonas'.

The Echo (echo, echo)...

As expected, the only way to get data back was by carving it out of the image dump. There's a multitude of tools to choose from, Photorec (open source), Recuva (watch out for bundleware), ReclaiMe (paid), etc... though I'm partial to R-Studio (paid); their results consistently outperform the competition.

In this instance, nonetheless, things wouldn't be so simple. Every software tried was able to extract the wav files, but there was something rather wrong with the data as they all had this repetition, an echo of sorts.

I'm a bit dull of hearing, so opened them in Audacity to check what was going on. You can clearly see a pattern here:

While the chunks have a pretty similar waveform - there isn't any sort of binary correlation. Besides, there was also something murky, every wav file had what appeared to be 2 consecutive headers.

Mind you, at this point, my knowledge of wav files was that the header is 52 49 46 46. Aside from using a mic, I didn't query Pierre on how he had actually recorded the data. However, when I saw "ZOOM M3" tag in the header, I called the authority on everything sound.

Perfect Pitch, a sort of wizardry

Ed's been on speed dial for the better part of this lifetime. I'm lucky like that. Beyond being an unbelievable composer - just listen to the breathtaking Best Youth - he's also a pitch-perfect, living and breathing, encyclopedia on sound.

Zoom? Yes, yes. I have one. They record both wav and raw. Data is mangled? Ah. Sure. Recover? Twisted files? That's going to be an impossible task and, even if it's not, it'll be easier to just re-record.

And there he had me. It would definitely be easier to just re-record, even Pierre at one point suggested doing a voice-over, but where would the fun be in that?

The Magic Number

I didn't even appreciate the concept of a raw wave until Ed explained it, but now I knew the mic saves two simultaneous files, everything fell into place.

It's common for digital cameras to store both formats, but a microphone is a different beast. There's no way to determine upfront how long the user will record, which wouldn't be an issue if this were a single stream. As it's storing both tracks, there is quite a bit more play involved.

You see, as soon as you hit record, the mic creates two files and continuously flushes the captured data from both onto the card. That, is done in chunks of data. One for the RAW, another for the WAV, repeat.

Since we need to isolate those slices of data in order to untangle it, we must know their exact size. And there lies our magic number!

My first thought was that the chunks might be aligned with exFAT allocation unit size. In this case, 128 KBytes. Let's test it.

The header, clearly states that this is a stereo file (2 channels), recorded at 32 bits per channel, sampled 48k times per second. If you remember from the image above, the chunks repeat at approximately 0.7 seconds.

Let's get a rough approximation on the chunk bytes so we know the ballpark.

1 second of data = 2 channels * 32 bits * 48000 samples
1 second of data = 384000 bytes
0.7 seconds ~ 268800 bytes

And just like that, the idea that data may be chunked to exFAT AUS of 128 Kbytes is instantly disproved.

The next obvious step would be to travel upwards on base 2. Given that 4096 is a good balance for buffers, let's evaluate from there:

4096 * 32 = 131072 (falls short by about 1/2)
4096 * 64 = 262144 (is in the ballpark of what we're expecting)

262144/384000 ~ 0.682 seconds of data

0.682 seconds matched our estimate of 0.7 seconds so perfectly that I immediately knew 262144 was the constant we were after.

The reconstruction

Conceptually, the problem was solved. Now it was just a matter of building the tool. For that it would be necessary to:

• Untangle the files directly from the image dump. Since pieces recovered by other carving software would be half the expected size (the data chunk contains two files, so it’s actually double the size reported).
• Learn how to create a RIFF header.
• Create a RIFF header with a BEXT chunk to make the recovered files compatible with the "M3 ZOOM Edit & Play" software.

And I'm pretty sure you'll feel more at home there.

However, for the sake of Google indexing, I’m leaving the methods that create both the RIFF and BEXT headers here, something I couldn’t find, which unfortunately made the process take longer than I’d like to admit.

public class RiffFile {

    /**
     * Creates a RIFF header with BEXT and fmt chunks
     *
     * @param sampleRate    the sample rate of the audio (8000Hz, 44100Hz, 48000Hz, etc) times per second the audio is sampled
     * @param bitsPerSample the bits per sample (8bits, 16bits, 32bits, etc)
     * @param channels      the number of channels (1 mono, 2 stereo, etc)
     * @param audioDataSize the size of the audio data in bytes
     * @return the RIFF header
     * @throws IOException if an I/O error occurs
     */
    public static byte[] createRiffHeader(int sampleRate, short bitsPerSample, short channels, int audioDataSize) throws IOException {
        // calculate the byte rate, block align and file size
        int byteRate = sampleRate * channels * bitsPerSample / 8;
        short blockAlign = (short) (bitsPerSample * channels / 8);

        // stream that will carry the new RIFF file
        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        DataOutputStream out = new DataOutputStream(byteArrayOutputStream);

        // riff header
        out.writeBytes("RIFF");
        out.writeInt(Integer.reverseBytes(0));
        out.writeBytes("WAVE");                             // 9-12 Format always WAVE

        // bext chunk
        writeBextChunk(out);

        // fmt chunk
        out.writeBytes("fmt ");                             // 13-16 chunkID is "fmt " with trailing whitespace
        out.writeInt(Integer.reverseBytes(16));             // 17-20 size of this chunk, is 16 byts
        out.writeShort(Short.reverseBytes((short) 3));      // 21-22 (2 bytes) audioFormat (1 PCM integer, 3 IEEE 754 float)
        out.writeShort(Short.reverseBytes(channels));       // 23-24 (2 bytes) numChannels (1 mono, 2 stereo, 4, etc)
        out.writeInt(Integer.reverseBytes(sampleRate));     // 25-28 (4 bytes) sampleRate (8000, 44100, 48000, etc)
        out.writeInt(Integer.reverseBytes(byteRate));       // 29-32 (4 bytes) byteRate (sampleRate * numChannels * bitsPerSample/8)
        out.writeShort(Short.reverseBytes(blockAlign));     // 33-34 (2 bytes) blockAlign (numChannels * bitsPerSample/8)
        out.writeShort(Short.reverseBytes(bitsPerSample));  // 35-36 (2 bytes) bitsPerSample (8bits, 16bits, 32bits, etc)

        // data chunk
        out.writeBytes("data");                             // 37-40 chunkID ID is "data"
        out.writeInt(Integer.reverseBytes(audioDataSize));  // 41-44 size of this chunk varies
        out.close();

        // write the full size of the file on the 4-8 bytes
        byte[] outArr = byteArrayOutputStream.toByteArray();
        int size = outArr.length - 8;
        ByteBuffer.wrap(outArr, 4, 4).order(ByteOrder.LITTLE_ENDIAN).putInt(size);
        return outArr;
    }

    private static void writeBextChunk(DataOutputStream out) throws IOException {
        // bext chunk
        out.writeBytes("bext");
        out.writeInt(Integer.reverseBytes(256 + 32 + 32 + 10 + 8 + 8 + 8 + 2 + 180 + 4 + 4 + 4 + 4 + 4 + 180)); // bext chunk size (fixed size for BWF)

        // description 256 bytes
        writeToArray(out, 256, "");                     // 256 bytes description
        writeToArray(out, 32, "ZOOM M3");               // 32 bytes originator
        writeToArray(out, 32, "");                      // 32 bytes originator reference
        writeToArray(out, 10, "2023-10-01");            // 10 bytes origination date
        writeToArray(out, 8, "12:00:00");               // 8 bytes origination time
        writeToArray(out, 8, "12:00:00");               // 8 bytes time reference

        out.writeLong(Long.reverseBytes(0L));           // 8 bytes time reference
        out.writeShort(Short.reverseBytes((short) 0));    // 2 bytes version
        out.write(new byte[180]);                         // 180 bytes UMID
        out.writeFloat(0.0f);                          // 4 bytes loudness value
        out.writeFloat(0.0f);                          // 4 bytes loudness range
        out.writeFloat(0.0f);                          // 4 bytes max true peak level
        out.writeFloat(0.0f);                          // 4 bytes max momentary loudness
        out.writeFloat(0.0f);                          // 4 bytes max short term loudness

        // zoom m3 needs this bit to allow file to be read from "zoom m3 edit & play"
        writeToArray(out, 180, "A=PCM,F=48000,W=32,M=stereo,T=M3;VERSION=1.00;MSRAW=ON ;");
    }
    
}

As you may see, there wasn't much effort put into the BEXT chunk; I simply created it to ensure "Zoom M3 Edit & Play" would be compatible.

The Tool

I hope you had an interesting read. I tried to lift the curtain on the thought process while keeping this engaging. The actual code is hopefully self-explanatory, and you will find it here:

GitHub - wasteofserver/zoom_m3_mic_wav_data_recover: A data recovery tool for Zoom M3 MicTrak Stereo Shotgun Microphone/Recorder

A data recovery tool for Zoom M3 MicTrak Stereo Shotgun Microphone/Recorder - wasteofserver/zoom_m3_mic_wav_data_recover

GitHubwasteofserver

The challenge wasn't just about recovering lost recordings - it was about understanding why traditional tools failed and developing a method that worked.

While it might have been easier to re-record, the thrill of solving the puzzle made the effort worthwhile. In the end we got a custom-built solution that successfully restores Zoom M3 MicTrak recordings.

If you ever find yourself in a similar situation, hopefully, this breakdown helps you out. And if not, well, at least you got to enjoy a little adventure into the world of data recovery.

This post is a bit different from my usual content. While it includes a snippet of code, it's designed for a less technical audience, thus the flood of screenshots.

I received an atypical cry for help. A couple is splitting and one of the partners is spamming the other with abusive emails. The receiver created a Gmail filter to trash the messages, but Gmail trash retention policy is 30 days and, in moments of weakness, can't resist clicking in the trash and reading them, which understandably exacerbates the issue.

Given the violent nature of the emails, I didn’t want to risk deleting potential evidence, so my suggestion was to create a process that would:

• Auto-forward the emails from that sender to a lawyer
• Delete those emails and purge them from trash, bypassing Gmail's 30-day retention policy

Create a forward address

On this particular case, the forward address will be the lawyer. Go to Gmail settings, select "Forwarding and POP/IMAP" and click on "Add a forwarding address".

After adding a forward address, Gmail will send a confirmation to that email - your lawyer - asking for permission. As soon as that's granted, you may proceed to the next step. Just remember to keep Forwarding disabled!

Create a new Filter

Here we will be specifying that all emails that come from someone@example.com will be forwarded to laywer@example.com and then sent to the trash.

Go to "Filters and Blocked Addresses" and then click on "Create a new filter".

The form for adding filters will open. You want all messages from that specific address to be filtered, so just add "someone@example.com" to the filter and choose "Create filter".

Now you'll have to select exactly what you want the filter to do. On our specific case, we'll forward the email to the lawyer, so tick that box. We also want to delete the email, so also tick that one.

This solves half of the issue and is the most straightforward part of the process. The tricky bit comes next, how to instantly purge those emails from the trash so that we're not tempted to read them. Google Apps Script to the rescue!

Create a Google Apps Script

Google Drive offers a feature that allows you to host and run scripts. While most developers are familiar with this, power users may not be aware of it. For the task at hand, this feature is absolutely perfect.

Head into https://script.google.com/, follow the authentication procedures, if needed, and then click on "New Project".

You're now on your project screen. You'll need to interact with Gmail so let's add that service. Click on the big + next to "Services", look for Gmail API and add it.

Now, replace the myFunction with this piece of code. Remember, YOU MUST CHANGE someone@example.com to the actual address you want to remove from trash!

function deleteMailsFromTrash() {
  var gmailSearchString = `in:trash from:someone@example.com`

  var threads = GmailApp.search(gmailSearchString);
  const n = threads.length;
  if (n <= 0) {
    Logger.log("No threads matching search string \"%s\"", gmailSearchString);
    return
  } else {
    Logger.log("%s threads matching action **%s**", n, gmailSearchString);
  }

  for (var i = 0; i < threads.length; i++) {
    var thread = threads[i];
    Logger.log(`\t Thread# ${i} [ID: ${thread.getId()}]: [message : ${thread.getFirstMessageSubject()}] deleted`);
    Gmail.Users.Threads.remove('me', thread.getId());
  }
}

Your screen should now resemble this. Go ahead and rename "Untitled project" to something more meaningful, like "Purge Specific Mails from Trash". Also change myFunction to deleteMailsFromTrash and then hit Run.

You'll be asked to give permissions to access your Google account.

Now you'll get an error! Google hasn't verified this app. While the developer hasn't verified the app, you shouldn't use. In this particular case, you are the developer! That's why I didn't release this solution as a pre-made script. It's safer to have the code running on your side.

Click on that "Go to Purge Specific Mails from Trash (unsafe)" link and then continue. On your Apps Script window, you'll see the first execution of your script.

On my case, since I don't have email from someone@example.com on the "trash" the program simply prints a "No threads matching search string". On your specific case, you may see that a couple of emails have been deleted! Well done.

Now everything is working, but we still need to set up a trigger to run the script automatically, ensuring unwanted emails are deleted in a timely manner.

Configure a trigger!

You'll want a time driven trigger. Something that runs on a schedule making sure that the emails that were placed in the trash by the filter you created above are purged before you can get to them.

Click on the clock on the left sidebar, then on the big blue button on the bottom right that says "Add Trigger" and configure the filter as per the image below.

I’ve set this script to run every 5 minutes, but you can adjust the interval to as low as 1 minute if you prefer. Tweak as needed. Setting a longer interval is simply a way to considerate of Google's infrastructure.

Now, to make sure things are working as expected, on the left sidebar, click on executions. You'll see a table with all the times the script has run. As you've just implemented it, there should probably be two to three runs. One manual Type: Editor from when you manually executed it and then a couple more from the time-driven trigger labeled with Type: Time-Driven.

As challenging as it may be to navigate these situations, it’s important to remember that protecting your health is a priority.

While technology can help us minimize harmful distractions, healing takes time and self-compassion. Stay strong, take care of yourself, and don’t hesitate to seek support.

You deserve peace and healing through this process.

I don’t know if you’ve tried rucking before, but it’s something that liberates my mind.

It’s simply walking while carrying weight. Originally a military exercise, rucking is gaining popularity for its benefits to physical health, stability, and mental well-being. Give it a try!

Your logs are filled with 404 hits on /.DS_Store, /backup.sql, /.vscode/sftp.json and a multitude of other URLs. While these requests are mostly harmless, unless of course, your server actually does have something to offer at those locations, you should placate the bots.

Why?

Hitting a server is a resource intensive task and, given that those bots have an extensive list of different URLs, there's no caching mechanism that can help you. Besides, stopping bots is always a safety measure.

We've previously used HAProxy to mitigate attacks on Wordpress login page, the idea is to extend that approach to also cover 404 errors.

I've taken inspiration from Sasa Tekovic, particularly in ensuring that legitimate search engine crawlers aren’t blocked and allowing 404 errors on static resources. This prevents genuine missing resources - an error that's bound to happen - from inadvertently blocking legitimate users.

Before implementing, it's always good to spin up a local test environment. Let's start HAProxy and Apache using Docker. We do need an actual backend server to give us those 404.

version : '3'

services:
    haproxy:
        image: haproxy:3.1.3-alpine
        ports:
            - "8100:80"
        volumes:
            - "./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg"
        networks:
            - webnet
    apache:
        image: httpd:latest
        container_name: apache1
        ports:
            - "8080:80"
        volumes:
            - ./html:/usr/local/apache2/htdocs/
        networks:
            - webnet
            
networks:
    webnet:

The haproxy.cfg file is pretty much self-explanatory:

global
    log stdout format raw daemon debug
	
defaults
    log     global
	mode    http

frontend main
    bind *:80
	
	acl static_file path_end .css .js .jpg .jpeg .gif .ico .png .bmp .webp .csv .ttf .woff .svg .svgz
    acl excluded_user_agent hdr_reg(user-agent) -i (yahoo|yandex|kagi|(google|bing)bot)

    # tracks IPs but exclude hits on static files and search engine crawlers
    http-request track-sc0 src table mock_404_tracking if !static_file !excluded_user_agent
    # increment gpc0 if response code was 404
    http-response sc-inc-gpc0(0) if { status 404 }
   	# checks if the 404 error rate limit was exceeded
    http-request deny deny_status 403 content-type text/html lf-string "404 abuse" if { sc0_gpc0_rate(mock_404_tracking) ge 5 }

    # whatever backend you're using
    use_backend apache_servers

backend apache_servers
    server apache1 apache1:80 maxconn 32

# mock backend to hold a stick table
backend mock_404_tracking
    stick-table type ip size 100k expire 10m store gpc0,gpc0_rate(1m)

As it stands, this setup effectively rate-limits bots generating excessive 404s. However, we also want to integrate it with our previous example, where we used HAProxy to block attacks on WordPress.

global
    log stdout format raw daemon debug
	
defaults
    log     global
	mode    http

frontend main
    bind *:80
	
	# We may, or may not, be running this with Cloudflare acting as a CDN.
    # If Cloudflare is in front of our servers, user/bot IP will be in 
    # 'CF-Connecting-IP', otherwise user IP with be in 'src'. So we make
    # sure to set a variable 'txn.actual_ip' that has the IP, no matter what
    http-request set-var(txn.actual_ip) hdr_ip(CF-Connecting-IP) if { hdr(CF-Connecting-IP) -m found }
    http-request set-var(txn.actual_ip) src if !{ hdr(CF-Connecting-IP) -m found }

    # gets the actual IP on logs
    log-format "%ci\ %hr\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %r\ %ST\ %Tr IP:%{+Q}[var(txn.actual_ip)]"

    # common static files where we may get 404 errors and also common search engine
    # crawlers that we don't want blocked
	acl static_file path_end .css .js .jpg .jpeg .gif .ico .png .bmp .webp .csv .ttf .woff .svg .svgz
    acl excluded_user_agent hdr_reg(user-agent) -i (yahoo|yandex|kagi|google|bing)

    # paths where we will rate limit users to prevent Wordpress abuse
	acl is_wp_login path_end -i /wp-login.php /xmlrpc.php /xmrlpc.php
    acl is_post method POST

	# 404 abuse blocker
    # track IPs but exclude hits on static files and search engine crawlers
    # increment gpc0 counter if response status was 404 and deny if rate exceeded
    http-request track-sc0 var(txn.actual_ip) table mock_404_track if !static_file !excluded_user_agent
    http-response sc-inc-gpc0(0) if { status 404 }
    http-request deny deny_status 403 content-type text/html lf-string "404 abuse" if { sc0_gpc0_rate(mock_404_track) ge 5 }

    # wordpress abuse blocker
    # track IPs if the request hits one of the monitored paths with a POST request
    # increment gpc1 counter if path was hit and deny if rate exceeded
    http-request track-sc1 var(txn.actual_ip) table mock_wplogin_track if is_wp_login is_post
    http-request sc-inc-gpc1(1) if is_wp_login is_post 
    http-request deny deny_status 403 content-type text/html lf-string "login abuse" if { sc1_gpc1_rate(mock_wplogin_track) ge 5 }
	
    # your backend, here using apache for demonstration purposes
    use_backend apache_servers

backend apache_servers
    server apache1 apache1:80 maxconn 32

# mock backends for storing sticky tables
backend mock_404_track
    stick-table type ip size 100k expire 10m store gpc0,gpc0_rate(1m)
backend mock_wplogin_track
    stick-table type ip size 100k expire 10m store gpc1,gpc1_rate(1m)

And there you have it. HAProxy once again used for much more than as a simple reverse proxy. It's a little Swiss Knife!

This headlamp has been a game-changer when working on repairs.

I had one, but when it broke, hesitated to replace it, resorting to the phone’s flashlight. And sure, it works - but once you experience the convenience of having both hands free again - there’s no going back. If you need reliable, hands-free lighting, this is a must-have!

If you have some Linux servers and want them to dispatch your emails (system, logging, monitoring, etc) to an external email address, you can use a proper SMTP service, like Gmail, for that.

Notice that this will not work for mass emails. The idea is to send a few emails per day to yourself. If you're relaying emails for actual users, you must set up a dedicated mail server or use an external service like Mailgun or Amazon Simple Email Service.

I'm assuming:

1) you want your Linux box to send emails from an existing Gmail account
2) you've created an "app specific password" for this service

Remove possible previous installations of sendmail, postfix, exim and qmail:

sudo apt-get remove packagename sendmail postfix exim qmail

Install postfix:

sudo apt-get install postfix

Configure postfix with the values below:

sudo vim /etc/postfix/main.cf

# your preference may vary
inet_protocols = ipv4
inet_interfaces = loopback-only
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination

# your box hostname
myhostname = example-mail-box.wasteofserver.com
mydestination = $myhostname, localhost

# use Gmail with TLS as destination
relayhost = [smtp.gmail.com]:465
smtp_sasl_auth_enable = yes
smtp_use_tls = yes
smtp_tls_wrappermode = yes
smtp_tls_security_level = encrypt
smtp_sasl_security_options = noanonymous
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_password

# if something goes wrong you may want to uncomment these to check for logs
# on /var/log/mail.log and /var/log/mail.error
#debug_peer_list=smtp.gmail.com
#debug_peer_level=3

Create a file with your Gmail account and access credentials:

sudo vim /etc/postfix/sasl/sasl_password

With the following content:

smtp.gmail.com example@gmail.com:your-app-password-here

And then convert it into a database file that postfix can read:

sudo postmap /etc/postfix/sasl/sasl_password

Restart the service:

sudo /etc/init.d/postfix restart

And now send an email using the command line to check if everything's working as it should:

echo "test email" | mailx -a "From: wasteofserver <no-reply@wasteofserver.com>" -s "this is a test email from wasteofserver" youremail@youraccount.com

The logs should show something like this:

Jun 25 17:14:56 AEC1E9C0043: uid=0 from=<wasteofserver>
Jun 25 17:14:58 AEC1E9C0043: to=<you@gmail.com>, relay=smtp.gmail.com[173.194.76.109]:587, delay=2.2, delays=0.06/0.01/1.4/0.73, dsn=2.0.0, status=sent (250 2.0.0 OK  1624637698 v5sm11709310wml.26 - gsmt
Jun 25 17:14:58 AEC1E9C0043: removed

Alternatively, you can also check your Gmail "sent folder". As the email is sent using your Gmail account (using SMTP), the email will also be there.

Edit: Thanks to u/kevdogger for pointing that in Arch Linux, the default is no longer B-tree maps but lmdb (Lightning Memory-Mapped Database) and, as such, you will need to change main.cf to point to the correct file.

The easiest way is to prefix the map with the actual type, something like this:

# different types of maps created by postmap, change your config file accordingly
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_password
smtp_sasl_password_maps = btree:/etc/postfix/sasl/sasl_password
smtp_sasl_password_maps = lmdb:/etc/postfix/sasl/sasl_password

Last Christmas, my wife asked for walkie-talkies. I thought it would be one of those gifts that end up collecting dust, but decided to get her a high-quality set anyway. We now use them regularly—at least once a month—for just about everything. From road trips and hikes to beach outings, these walkie-talkies have become an essential part of our adventures.

When a product genuinely delivers on its promise, it makes all the difference. I can’t recommend these enough.

The most effective way to prevent bots from spamming your server is to drop them at the firewall. This is generally achieved using tools like Denyhosts or fail2ban, which monitor your logs, identify suspicious activity, and block the offending IP addresses before they cause harm.

Denyhosts works at the application level by adding entries to /etc/hosts.deny, whereas fail2ban operates at the firewall level using iptables, which makes it far more efficient.

However, on resource-constrained machines, fail2ban can still be taxing. A few years ago, we shared a demo of a lightweight log parser called banbylog, tailored specifically for our needs (SSH and WordPress activity monitoring) at a much lower resource cost. If that sounds like a good fit, feel free to check it out, but keep in mind that it’s not production-ready!

While the consensus is to parse logs and block hostile IPs at the firewall, it will not work under Cloudflare umbrella!

Why can't we flag offending IPs with Cloudflare?

Because the IPs that are "attacking" your server are not the actual offending IPs, but Clouflare machines that are proxying the request to your server. Take a look at the diagram below:

Cloudflare acts as a middleman between your server and the users. The only IP addresses visible to your firewall are from Cloudflare, not from the original user.

In fact, you should explicitly whitelist Cloudflare IP range. If you happen to block an IP that belongs to Cloudflare, legitimate users will see your site as down.

If the IP we see doesn't belong to the user making the request, what can we look for?

Every request Cloudflare sends to your server has an attached header that carries the user original IP under CF-Connecting-IP. And that's what we can leverage to get them! Unfortunately, reading http headers is too upstream for iptables, thus HAProxy to the rescue!

While iptables operates at Layer 4, HAProxy can operate at OSI Layer 7.

The easiest way to test HAProxy configurations is to boot a Docker instance of HAProxy:

docker run --name haproxy -p 888:80 -v ${pwd}:/usr/local/etc/haproxy --sysctl net.ipv4.ip_unprivileged_port_start=0 haproxy:2.3

The above assumes you're using Powershell and are in the directory that contains haproxy.cfg. Change ${pwd} accordingly if not.

docker kill -s HUP haproxy

Below is a straightforward haproxy.cfg that will put HAProxy listening on port 80, log to stdout so you can get an instant glimpse on what's going on, and also print the CF-Connecting-IP header.

global
    # output logs straight to stdout
    log stdout format raw daemon debug

defaults
    # put HAProxy in Level 7 mode allowing to inspect http protocol
    log     global
    mode    http

frontend  main
    bind *:80
	
    # capture original IP from header and put it in variable txn.cf_conn_ip
    http-request set-var(txn.cf_conn_ip) hdr(CF-Connecting-IP)

    # get the original IP on logs (just to check if things are working)
    log-format "%ci\ %hr\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %r\ %ST\ %Tr CF-IP:%{+Q}[var(txn.cf_conn_ip)]"

    # use backend ok, which always returns a 200, for debug
    use_backend ok

backend ok
	http-request return status 200 content-type "text/plain" lf-string "ok"

Let's make a test request to HAProxy.

I'm partial to Bruno, a portable and offline alternative to Postman. Download it, add the CF-Connecting-IP header and make a POST request to http://localhost:888/wp-login.php to test if everything's working.

If things went as planned, your HAProxy should have printed this:

172.17.0.1 main ok/<NOSRV> -1/-1/0 78 LR POST /wp-login.php HTTP/1.1 200 -1 CF-IP:"222.222.222.222"

On our particular case, bots are hitting wp-login.php, xmlrpc.php and xmrlpc.php (last one is a typo, but we've had more than 100k hits in the last 24h!). We also know that they're flooding the server with POST requests trying to brute-force passwords.

frontend main
    # requests that will be monitored and blocked if abused
    acl is_wp_login path_end -i /wp-login.php /xmlrpc.php /xmrlpc.php
    acl is_post method POST

We now have a couple of options:

a) Now that we have the offending IPs in the log, we could change banbylog to write them to a file, and have HAProxy deny those requests. While this would work, HAProxy would need to be constantly reloaded.

b) Or we may simply leverage HAProxy stick tables and do a rate-limiting on offending requests.

While "a" would allow us to ban the offending IP for an indefinite amount of time, "b" has less moving pieces.

frontend main
    # requests that will be monitored and blocked if abused
    acl is_wp_login path_end -i /wp-login.php /xmlrpc.php /xmrlpc.php
    acl is_post method POST

    # table than can store 100k IPs, entries expire after 1 minute
	stick-table type ip size 100k expire 1m  store http_req_rate(1m)

    # we'll track (save to table) the original IP only if the request hits
    # one of the monitored paths with a POST request
    http-request track-sc0 hdr(CF-Connecting-IP) if is_wp_login is_post

    # we now query the stick-table and if the IP has made more than 
    # 5 requests of the offending type in the last minute, 
    # current request is denied
    http-request deny if is_wp_login is_post { sc_http_req_rate(0) gt 5 }

HAProxy has multiple deny options, tarpit, silent drop, reject or shadowban. A tarpit deny would be something like this:

# make request hang for 20 seconds before replying back with a 403
timeout tarpit 20s
http-request tarpit deny_status 403 if is_wp_login is_post { sc_http_req_rate(0) gt 2 }

Just for reference, here's the full (overly simplified) HAProxy config file that blocks requests if they hit one of the monitored URL's with more than 5 hits in less than a minute:

global
    log stdout format raw daemon debug
	
defaults
    log     global
	mode    http

frontend main
    bind *:80
	
    # capture original IP from header and put it in variable txn.cf_conn_ip
    http-request set-var(txn.cf_conn_ip) hdr(CF-Connecting-IP)

    # get the original IP on logs (just to check if things are working)
    log-format "%ci\ %hr\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %r\ %ST\ %Tr CF-IP:%{+Q}[var(txn.cf_conn_ip)]"

    acl is_wp_login path_end -i /wp-login.php /xmlrpc.php /xmrlpc.php
    acl is_post method POST
	stick-table type ip size 100k expire 1m  store http_req_rate(1m)
	http-request track-sc0 var(txn.cf_conn_ip) if is_wp_login is_post
	http-request deny if is_wp_login is_post { sc_http_req_rate(0) gt 5 }

    # obviously change this to whatever backend you're using
    use_backend ok

backend ok
	http-request return status 200 content-type "text/plain" lf-string "ok"

Re-Captcha, obviously!

While this article focus on the server side of things, the first thing you should obviously do, is to foolproof forms with Re-Captcha, which you can easily do with a plugin such as Advanced Google reCAPTCHA by WebFactory.

EDIT: A user queried: «I have some WordPress installations under Cloudflare and some exposing the server directly. Can it work on both?»

You can. It's as simple as doing something like this:

# match IP against Cloudflare list
acl from_cf src -f /usr/local/etc/haproxy/cloudflare_ips.lst

# if IP originates from cloudflare check CF-Connecting-IP header, otherwise use src
http-request set-var(txn.client_ip) hdr(CF-Connecting-IP) if from_cf
http-request set-var(txn.client_ip) src if !{ var(txn.client_ip) -m found }

This week, I assisted a friend in upgrading to professional TP-Link access points. I'm a strong advocate for devices that excel at a single task, and these EAP610 access points do just that. Highly recommended!

Before going heads on into Java and the ways to tackle interference, either from the garbage collector or from context switching, let's first glance over the fundamentals of writing code for your future self.

Premature optimization is the root of all evil.

You've heard it before; premature optimization is the root of all evil. Well, sometimes. When writing software, I'm a firm believer of being:

1) as descriptive as possible; you should try to narrate intentions as if you were writing a story.

2) as optimal as possible; which means that you should know the fundamentals of the language and apply them accordingly.

As descriptive as possible

Your code should speak intention, and a lot of it pertains to the way you name methods and variables.

int[10] array1;        // bad
int[10] numItems;      // better
int[10] backPackItems; // great

While numItems is abstract, backPackItems tells you a lot about expected behaviour.

Or say you have this method:

List<Countries> visitedCountries() {
    if(noCountryVisitedYet)
        return new ArrayList<>(0);
    }
    // (...)
    return listOfVisitedCountries;
}

Can we do better? We definitely can!

List<Countries> visitedCountries() {
    if(noCountryVisitedYet)
        return Collections.emptyList();
    }
    // (...)
    return listOfVisitedCountries;
}

Imagine you're reading the above code for the first time and stumble on the guard clause that checks if the user has actually visited countries. Also imagine this is buried in a lengthy class, reading Collections.emptyList() is definitely more descriptive than new ArrayList<>(0), you're also making sure it's immutable making sure client code can't modify it.

As optimal as possible

Know your language and use it accordingly. If you need a double there's no need to wrap it in a Double object. The same goes to using a List if all you actually need is an Array.

Know that you should concatenate Strings using StringBuilder or StringBuffer if you're sharing state between threads.

// don't do this
String votesByCounty = "";
for (County county : counties) {
    votesByCounty += county.toString();
}

// do this instead
StringBuilder votesByCounty = new StringBuilder();
for (County county : counties) {
    votesByCounty.append(county.toString());
}

Know how to index your database. Anticipate bottlenecks and cache accordingly. All the above are optimizations. They are the kind of optimizations that you should be aware and implement as first citizens.

How do you kill it first?

I'll never forget about a hack I read a couple of years ago. Truth be said, the author backtracked quickly, but it goes to show how a lot of evil can spur from good intention.

// do not do this, ever!
int i = 0;
while (i<10000000) {
    // business logic
    
    if (i % 3000 == 0) { //prevent long gc
        try {
            Thread.sleep(0);
        } catch (Ignored e) { }
    }
}

You can read more on why and how the above code works in the original article and, while the exploit is definitely interesting, this is one of those things you should never ever do.

• Works by side effects, Thread.sleep(0) has no purpose in this block
• Works by exploiting a deficiency of code downstream
• For anyone inheriting this code, it's obscure and magical

Only start forging something a bit more involved if, after writing with all the default optimizations the language provides, you've hit a bottleneck. But steer away from concoctions as the above.

How to tackle that Garbage Collector?

If after all's done, the Garbage Collector is still the piece that's offering resistance, these are some of the things you may try:

• If your service is so latency sensitive that you can't allow for GC, run with "Epsilon GC" and avoid GC altogether.
  -XX:+UnlockExperimentalVMOptions -XX:+UseEpsilonGC
  This will obviously grow your memory until you get an OOM exception, so either it's a short-lived scenario or your program is optimized not to create objects
  
  
• If your service is somewhat latency sensitive, but the allowed tolerance permits some leeway, run GC1 and feed it something like -XX:MaxGCPauseTimeMillis=100 (default is 250ms)
  
  
• If the issue spurs from external libraries, say one of them calls System.gc() or Runtime.getRuntime().gc() which are stop-the-world garbage collectors, you can override offending behaviour by running with -XX:+DisableExplicitGC
  
  
• If you're running on a JVM above 11, do try the Z Garbage Collector (ZGC), performance improvements are monumental! -XX:+UnlockExperimentalVMOptions -XX:+UseZGC. You may also want to check this JDK 21 GC benchmark.

Note 1: since Java 15, ZGC is production ready, but you still have to explicitly activate it with -XX:+UseZGC.

Note 2: The VM considers machines as server-class if the VM detects more than two processors and a heap size larger or equal to 1792 MB. If not server-class, it will default to the Serial GC.

In essence, opt for GC tuning when it's clear that the application's performance constraints are directly tied to garbage collection behavior and you have the necessary expertise to make informed adjustments. Otherwise, trust the JVM's default settings and focus on optimizing application-level code.

u/shiphe - you'll want to read the full comment

Other relevant libraries you may want to explore:

Java Microbenchmark Harness (JMH)

If you're optimizing out of feeling without any real benchmarking, you're doing yourself a disservice. JMH is the de facto Java library to test your algorithms' performance. Use it.

Java-Thread-Affinity

Pinning a process to a specific core may improve cache hits. It will depend on the underlying hardware and how your routine is dealing with data. Nonetheless, this library makes it so easy to implement that, if a CPU intensive method is dragging you, you'll want to test it.

LMAX Disruptor

This is one of those libraries that, even if you don't need, you'll want to study. The idea is to allow for ultra low latency concurrency. But the way it's implemented, from mechanical sympathy to the ring buffer, brings a lot of new concepts. I still remember when I first discovered it, seven years ago, pulling an all-nighter to digest it.

Netflix jvmquake

The premiss of jvmquake is that when things go sideways with the JVM, you want it to die and not hang. A couple of years ago, I was running simulations on an HTCondor cluster that was on tight memory constraints and sometimes jobs would get stuck due to "out of memory" errors. This library forces the JVM to die, allowing you to deal with the actual error. On this specific case, HTCondor would auto re-schedule the job.

Final thoughts

The code that made me write this post? I've written way worse. I still do. The best we can hope for is to continuously mess up less.

I'm expecting to be disgruntled looking at my own code a few years down the road.

And that's a good sign.

Given the nature of this post, I found it appropriate to promote a product I've had for quite some time, a home shredder!

This is the 3rd different model/brand I've had and can definitely attest to Amazon Basics sturdiness.

While I do prefer signed and encrypted PDFs, there are a lot of financial institutions that still share data via paper. I shred those. Then I shred some other trivial stuff just to make it safe by obscurity.

In all honesty, I find it soothing.

Edits & Thank You:

• to FlorianSchaetz for catching the mutability error in visitedCountries() and the detailed explanation.
• to u/brunocborges and u/BikingSquirrel for explaining that on lower end machines, you get Serial GC
• to u/shiphe for taking the time to better explain when you should mess with the GC and when you shouldn't
• to u/tomwhoiscontrary to put me in the right track regarding what should be considered standard practice
• to u/BikingSquirrel (again) for providing the link to JDK 21 garbage collectors benchmark

Calculating profits and losses from a mathematical perspective is trivial, just subtract all things sold to all things bought. The remainder is your profit. That's pretty much how mom-and-pop retail works.

In future markets tough, people may get confused as you'll have open positions, which in retail would translate to inventory, that may not have been bought yet (if you're short selling).

Say you're trading in Mini S&P Futures from CME.

The contract lot size is 50 units and it's currently trading at around 5140. Let's see how that works out in a simple BUY/SELL trade:

// final value is num_contracts * lot_size * price
BUY  2 ESH4 @ 5140.00 > 514_000 USD
SELL 2 ESH4 @ 5141.00 > 514_100 USD
                           +100 USD (PNL)

Let's add a third order to the mix to see how it works:

BUY  2 ESH4 @ 5140.00 > 514_000 USD
SELL 2 ESH4 @ 5141.00 > 514_100 USD
SELL 2 ESH4 @ 5142.00 > 514_200 USD

If you think it through, you have an open position of -2 ES contracts that you'll eventually have to close. To calculate the instant P&L you must assume current market price.

BUY  2 ESH4 @ 5140.00 > 514_000 USD
SELL 2 ESH4 @ 5141.00 > 514_100 USD
SELL 2 ESH4 @ 5142.00 > 514_200 USD
// current market price is 5145.00
// you have -2 ES contracts to close
BUY  2 ESH4 @ 5145.00 > 514_500 USD
                           -200 USD (PNL)              

Simple arithmetic. Buy orders are treated as negative values, sell orders are treated as positive values. That's really all you need to think about.

If you're not flat, you'll have a position that you must pretend to close (at current market value) to calculate your current P&L.

BUY  3 ESH4 @ 5100 = -3 * 50 * 5100
BUY  2 ESH4 @ 5150 = -2 * 50 * 5150
BUY  1 ESH4 @ 5200 = -1 * 50 * 5200
SELL 4 ESH4 @ 5300 = +4 * 50 * 5300
// contracts open (-3) - 2 - 1 + 4 = -2
                   = -2 * 50 * current_market_price

So now that we've covered the basics, you really only need to iterate through all the orders, count buys as negative, sells as positive, and, in the end, calculate whatever is open at the current market price.

What about performance?

It's pretty evident that given a large amount of orders, doing this calculation for every single tick will become troublesome very fast. But as you've probably guessed by now, the system only needs to do that calculation on fills.

As soon as an order gets filled, you calculate the P&L for all filled positions and cache it. You also cache the number of contracts open. Then you just need to do a single calculation on price change:

pnl_realized + contracts_open * lot_size * current_market_price

If your system is not doing any sort of risk control, you can even push that calculation to the user GUI.

Commission fees?

Once you get the mechanics, they are very simple to add. Even if your current broker offers a flat fee, you'll want to delegate such calculation to its own class, as it will make code more extensible to future changes. I'm thinking about quantity rebates, tiers, etc.

I've recently been gifted this Amazon Basics phone holder, and I'm converted.

I use it to browse online newspappers during breakfast and it also serves as a recipe holder in the kitchen. It has proven its worth daily. Highly recommended!

A friend asked if I could help as he was having a hard time integrating IKBR TWS API. Apparently, he's not alone:

Even after all the above pain in my day job, I'm still yet to come across an offering as crap as the TWS API. - DanWritesCode

I have been developing code for about 15 years, professionally, and have never seen a piece of garbage like the TWS Api. - puzzled_orc

I put IBKR TWS not just at the bottom of the list, it's twice as bad as the most crappy system out there. - Causal-Capital

After reviewing TWS offer, it's definitely legacy code.

The API grew to support a multitude of requests in a non-standard and non-organized fashion. But legacy code, is also code that has been running for ages, most likely providing mission-critical functionality, and there's certainly value in that.

When you publish an API you're creating a contract with the world. As soon as it's out there, you can't really change it without creating ripples across the entire ecosystem. If requirements take an unexpected curve, as they generally do, it's easy to find yourself cornered and forced to provide a less than ideal interface.

What you can definitely do, is improve the documentation and provide clear code samples with what should be best practices. While I can't really point a finger at the API as I have no idea of the challenges involved at the time, the documentation could improve.

TWS Connection

Let's see how you're supposed to integrate the API connection. Notice that per the API instructions «it is important that the main EReader object is not created until after a connection has been established», which reads like misplaced behaviour.

Erapper wrapper = new EWrapperImpl();
EReaderSignal signal = new EJavaSignal();
EClientSocket client = new EClientSocket(wrapper, signal);

client.eConnect("host", PORT, CONNECTION_ID);

We can see that:

• wrapper implements the interface where you'll get callbacks (i.e. receive things from the API, market data, fills, etc)
• client allows you to interact with TWS (i.e. send things to the API, orders, etc)

And then we have both signal and reader classes.

• signal is used to signal the reader thread that there's data to process
• reader processes incoming messages

According to IBKR documentation, you should then start reader and a launch a new thread to process the queue:

final EReader reader = new EReader(client, signal);   
reader.start();

// IBKR thread created solely to empty the messaging queue
new Thread(() -> {
    while (client.isConnected()) {
        m_signal.waitForSignal();
        try {
            reader.processMsgs();
        } catch (Exception e) {
            // ...
        }
    }
}).start();

I didn't dig up TWS code to see how it behaves internally, but from a birds-eye perspective, both signal and reader are misplaced and would be better encapsulated within the client.

The user just needs a way to pass data to TWS (client) and a way to get data from TWS (wrapper). All other low level implementation details would better belong inside the implementation.

Broken API socket connection

One of the places where my friend was having issues was in the re-connecting code. You should design to accommodate failure, much more so if you're developing an automated trading strategy.

The socket connection between your code and TWS will eventually break, so that's a good place to start. Glancing over TWS API we have that:

// creates socket connection to TWS/IBG.
client.eConnect("host", PORT, CONNECTION_ID);

// closes the socket connection and terminates its thread.
eDisconnect (bool resetState=true)

If we break the socket at the network level, like IKBR states, we get a callback in the wrapper implementation:

@Override
public void error(int id, int errorCode, String errorMsg, String advancedOrderRejectJson) {
    log.warn("error: id={}, errorCode={}, errorMsg={}, advancedOrderRejectJson={}", 
        id, errorCode, errorMsg, advancedOrderRejectJson);
}

On a broken socket (network disconnect), the above will log something like:

error: id=-1
errorCode=502
errorMsg=Couldn't connect to TWS. Confirm that "Enable ActiveX and Socket
  Clients" is enabled and connection port is the same as "Socket Port" on the
  TWS "Edit->Global Configuration...->API->Settings" menu. Live Trading ports:
  TWS: 7496; IB Gateway: 4001. Simulated Trading ports for new installations
  of version 954.1 or newer: TWS: 7497; IB Gateway: 4002, 
  advancedOrderRejectJson=null

Notice that at this point we're not connected to TWS, but client.isConnected() still returns true. While not ideal, it's documented. You must issue a eDisconnect().

// pseudo code for broken socket behaviour

client.isConnected() -> returns true
// here we break the socket
client.isConnected() -> returns true
client.eDisconnect();
wrapper.connectionClosed(); // we get a callback here
client.isConnected() -> returns false

Given the above, it would look as though we could issue a connect() to re-establish connection.

client.eConnect("host", PORT, CONNECTION_ID);
wrapper.connectAck(); // we get a callback here
wrapper.connectionClosed(); // and then we get a callback here

What's happening here?

Remember the thread that we created to empty the queue? Let's revisit that:

new Thread(() -> {
    while (client.isConnected()) { // when client disconnets it should exit
        signal.waitForSignal(); // but it's waiting on signal!
        try {
            reader.processMsgs();
        } catch (Exception e) {
            // ...
        }
    }
}).start();

Even though client.isConnected() returns false, we're stopped on waitForSignal() so the while loop won't exit!

As soon as the above thread gets a new signal, client is already connected and the above thread will not stop! This was creating havoc in my friend's code! As previously stated, all this behaviour could (should?) have been encapsulated upstream but, given that it's not, we can do it ourselves.

A proper start, stop & reconnect sequence

Per everything above, there is a meticulous sequence that you must follow for TWS to properly connect. You must create some objects, but some can only be created after specific callbacks have been received, etc.

When things start getting involved and into a spaghetti-like sequence, I recommend that you stick with the cleanest possible approach. In this case, it's a scenario where you create all required variables and another one where you reset them.

Properly encapsulate Reader

Making sure reader is properly initialized and terminated is a big part of the original issue, so we're isolating it to reduce clutter.

/**
 * Responsible for reading messages from the TWS socket.
 * <p>
 * Should be stopped when the connection to TWS is lost.
 */
public class Reader implements Runnable {
    private final AtomicBoolean running = new AtomicBoolean(true);

    private final EReaderSignal readerSignal;
    private final EReader reader;

    public Reader(EClientSocket clientSocket, EReaderSignal readerSignal) {
        this.readerSignal = readerSignal;
        reader = new EReader(clientSocket, readerSignal);
        reader.start();
    }

    @Override
    public void run() {
        while (running.get()) {
            readerSignal.waitForSignal();
            try {
                reader.processMsgs();
            } catch (Exception e) {
                // dispatch to error mechanism
            }
        }
    }

    public void stop() {
        running.set(false);
        readerSignal.issueSignal();
        reader.interrupt();
    }
}

And then the calling class may look something like this:

public class StartTwsApi {
    private static final Logger log = LoggerFactory.getLogger(Start.class);
    EWrapperImpl wrapper;
    EJavaSignal readerSignal;
    EClientSocket clientSocket;
    Executor executor = Executors.newSingleThreadExecutor();
    Reader reader;
    
    StartTwsApi() {
        initializeWrapperAndSocket();
        connectToTWS();
    }
    
    private void initializeWrapperAndSocket() {
        wrapper = new EWrapperImpl(this, this);
        readerSignal = new EJavaSignal();
        clientSocket = new EClientSocket(wrapper, readerSignal);
    }
    
    private void connectToTWS() {
        log.info("Connecting to TWS...");
        clientSocket.eConnect("localhost", 4005, 2);

        // safeguard to make sure we do connect / keep trying
        ScheduledExecutorService executorService = Executors.newScheduledThreadPool(1);
        executorService.schedule(() -> {
            if (!clientSocket.isConnected()) {
                log.warn("Still not connected (10s). Retrying connect...");
                onTwsConnectionFailure();
            }
        }, 10, TimeUnit.SECONDS);
    }
    
    private void startReader() {
        executor.execute(reader);
    }
    
    private void stopReader() {
        if (reader != null) {
            reader.stop();
        }
    }
    
    /**
     * You'll call this method when you receive a callback from TWS 
     * stating that the connection is lost.
     * <p>
     * It stops the reader thread and re-connects to TWS.
     */
    public void onTwsConnectionFailure() {
        stopReader();
        log.info("Stopped reader. Will re-connect in 5 seconds...");
        Uninterruptibles.sleepUninterruptibly(5, TimeUnit.SECONDS);
        connectToTWS();
    }
    
    /**
     * Call this method when you get a connectAck() on TWS 
     * wrapper callback implementation
     */
    public void connectAck() {
        log.info("connectAck received. Safe to start reader thread...");
        reader = new Reader(clientSocket, readerSignal);
        startReader();
    }
    
}

My recommendation is that you abstract all this behaviour under a specific TWS module, and then pass a cleaner interface back to your actual code.

There is much more that we've delved on. Historical data, for instance, is an area where there were pretty interesting inconsistencies:

• public void historicalTicksBidAsk(int reqId, List<HistoricalTickBidAsk> ticks, boolean done)
  tickData is received in a single callback as a List of Objects (max 1000)
• public void historicalData(int reqId, Bar bar)
  barData is received as a series of multiple callbacks with a single Bar object; when all bars arrive, you get a callback on historicalDataEnd(int reqId, String startDateStr, String endDateStr).

After a brief chat on Reddit with u/fit-interaction4450 I remembered another place where dragons lurk. After connecting you'll get a callback on connectAck() which is a signal to start the reader thread, but you must wait until another callback on nextValidId(int orderId) to actually start interacting with TWS.

And while IBKR has it documented (that's all we may ask for), it still speaks volumes to the way the API grew.

Important: The IBApi.EWrapper.nextValidID callback is commonly used to indicate that the connection is completed and other messages can be sent from the API client to TWS. There is the possibility that function calls made prior to this time could be dropped by TWS.

My friend, not having read the above, solved it by adding a sleep() after the connection. When I asked why that was there, he said that sometimes messages were not being received by TWS and pausing for a bit solved it. It sure did (most of the time, anyway), but caught my attention as a sleep() without a clear intention is, by definition, a code smell.

Last thoughts:

Designing an API is hard. Designing an API to stand the test of time is even harder. While I can identify antipatterns and a lack of behaviour guidelines, I may only assume that business needs and decisions dictated the curved approach we now see. The very same decisions that made IBKR into a 45 billion company.

I don't know if there's an audience for TWS API articles. If you want more, let me know in the comments below.

My latest acquisition was the Arctis Nova 7 headphones. Sound is so crispy you can immediately tell the difference between Spotify native app vs Spotify web!

Microphone quality is also superb and has been praised in latest conference calls.

If you're in the market for a new headset, I highly recommend checking out the Nova 7.

In one way or another, this post has been written thousands of times across the web. I'm rewriting it for two reasons:

• Command reference
• mosh under Cygwinerror.

Mosh (mobile shell)

Mosh uses UDP to create an unbreakable ssh session. If you're on the road with flaky connections, it's a lifesaver.

On Windows, you can use mosh either in WSL or Cygwin. If using via Cygwin, you may stumble on the error «Did not find remote IP address (is SSH ProxyCommand disabled?)».

$ ./mosh frankie@192.168.5.2
CreateProcessW failed error:2
posix_spawnp: No such file or directory
./mosh: Did not find remote IP address (is SSH ProxyCommand disabled?).

To fix it, it's a simple as passing a few extra parameters:

$ ./mosh --predict=always --experimental-remote-ip=remote frankie@192.168.5.2

--predict=controls use of speculative local echo
--experimental-remote-ip= used to discover IP address mosh connects to

Notice that if you're having errors connecting, you should check if you're actually allowed to UDP into the server. For that, use netcat.

nc -u <host> <port>

tmux (terminal multiplexer)

tmux puts your terminal connection into a proper session. Say you're working on a remote server from your home. You can disconnect that session. Go to work and resume the session exactly as it was.

On top of that, it allows you to easily split the screen.

tmux new - to start
tmux new -s session-name - to name the session
tmux -t session-name - to attach to the named session

During the session, your go-to command is Ctrl-b.

Ctrl+b % - splits the screen vertically
Ctrl+b " - splits the screen horizontally
Ctrl+b x - closes current session

I've never had one of these iPhone cables fail.

Even after being chewed up by Roomba. It's funny how Amazon does a better iPhone cable than Apple, but there you have it. Can't recommend them enough.

Some APIs change because the world evolves, some because they are fundamentally broken from the start; looking at you Thread.stop!

What makes API design hard is that it must endure time. As soon as the contract is out there, you can't just take it back without a world of pain. Stadia is a perfectly good example of lost trust.

What's with a recipe?

Both Coca-Cola and Pepsi have iterated their formulas over the years. It's a known fact that one of the major ingredients in Coke is now missing; coca-leave (cocaine) and cola-nut (caffeine) gave Coca-Cola its name.

The 70s saw the commodities boom and, with it, the last formula change.

In the aftermath of the six-days war, Egypt closed the Suez Canal, which stirred the oil crises. There was also a massive spike in Coffee price given that Brazil lost 2/3rds of its coffee plants and both Angola and Guatemala saw a civil war and a major earthquake.

Amidst the instability, the largest sugar producer at the time, the Soviet Union, started hoarding the carbohydrate to fend off further uncertainty.

That was the last change Pepsi and Coca-Cola made: high-fructose corn syrup. Until now.

What's up with Pepsi API, then?

Given that for the last 50 years there was an implicit contract, it came as a surprise to learn that PepsiCo was sneakily messing with the original drink's recipe.

The idea is pretty smart, but for it to work, Pepsi can't make any fuss around it. The splashy campaigns of the past gave way to a covertness operation.

During the pandemic, Pepsi Original, the blue can, saw a remake. On the outside, everything stayed surreptitious the same. On the inside, though, a tiny bit of sugar was replaced by AcesulfameK.

The idea is to keep replacing sugar by AcesulfameK in tiny amounts and, eventually, Pepsi Original will be sugar-free. Or not. Depends on market reaction.

On the few occasions PepsiCo talked about the change, it did so under the ruse of promoting a healthier lifestyle and without conceding a timeline. My best guess is that they'll be monitoring sales to see how far can they go.

But let's be honest, the end goal is profit and sugar-tax, which is not yet a thing in the US, eats a lot of profit in Europe.

In 2017 Pepsi sold 127 million litres of Pepsi original in the UK alone. In sugar-tax only, this amounts to over 10 million pounds. Coca-cola will pay more than 22 million of sugar tax just in the UK!

Sugar is bad, so this should be good, right?

Let's get back to the API. When you implement a public contract, your consumers, fittingly, have a certain degree of expectation. You will most definitely have an EULA that gives you free reigns over the system, but having people use your product is the end goal. As such, backward incompatible change will be frowned upon and, if it must be so, has to be explicit and perfectly documented.

Pepsi's strategy to bypass the sugar-tax is precisely the opposite.

You may argue that the majority of users will not notice and, if you're being pedantic, that it wasn't an API change, just a data change. While most people will definitely not notice, there's a minority whose systems will break.

People who are allergic to AcesulfameK and Aspartame!

Embrace change, but engage users!

A good API is designed taking into consideration the consumer's needs. It should be explicit and allow for seamless integration. It should be complete. Concise and hard to misuse.

Be a good API designer.

Don't be like Pepsi.

Out of spite, get yourself some Coca-Cola.

Edit: There's a Reddit thread where users say that in the US, Pepsi is still using the original formula. On top of that, in Peru (a sugar-tax country), Coca-Cola is following Pepsi's lead by feeling the markets' reaction to changing the formula without warning users.

Edit 2025: A user shared this chart. While it may not align perfectly with the specific focus of this post given PepsiCo’s broader scope, the timeline somewhat corresponds.

As one of Java paradigms is "write once, run anywhere", and calling external software makes that much more complicated, you'll want to steer away from it as much as possible. However, sometimes there's really no other viable option.

The example below shows how to ping a host and capture both the standard and the error output streams.

String cmd = "ping -c 4 -W 1 8.8.8.8"

try {
    ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
    pb.redirectErrorStream(true);
    Process process = pb.start();
    process.waitFor(10, TimeUnit.SECONDS);
    try (BufferedReader br = new BufferedReader(
        new InputStreamReader(process.getInputStream())
    )) {
        StringBuilder stringBuilder = new StringBuilder();
        String line;
        while ((line = br.readLine()) != null) {
            stringBuilder.append(line);
        }
        if (process.exitValue() != 0) {
            // mail sending failed, error is on stringBuilder
        }
    }
} catch (IOException | InterruptedException e) {
	// recover / etc
}

On my email sending class, I generally have a fallback that relies on the machine sendmail/mail program. It kicks in if the SMTP server is not responding or is badly configured.

You may be interested in checking how to use sendmail (or mail) for sending emails as a one-liner.

While this website is not about reviews, I try to command it in the same spirit I used to navigate Usenet, if it may be useful, it should be shared.

I've been lucky enough to be gifted both these earphones, as disappointed previous owners, disposed them my way.

The AirPods belonged to my wife that unfortunately could not get a proper fit. The WF-1000XM4 were offered by a brother that's an audiophile and upgraded to some very expensive, custom-tip made, ones.

After one month of commutes, extensive hours in the tube, 8 aeroplane rides, multiple jogs, 6 hours Zwifting and many more typing away, I have a pretty solid opinion on what works. For concrete details on fidelity and ample scrutiny, you should read what the soundguys wrote, the critic below is just my biased analysis of having using them both for a considerable amount of time.

Sound Quality

Just by looking at the specs, given the difference in driver's size, I'd thought Apple would have the upper hand. The SoundGuys also prove that, objectively, the AirPods are more accurate. My experience though is mixed. I found the WF-1000XM4 extremely pleasant to listen to music. Bass had punch and trebles were detailed. Personally, I'd say it's a close match. Perhaps leaning towards Sony.

Noise-cancelling

Sony is king in aeroplanes. Noise-cancelling is just on another level. With a closed back and memory foam tips, you get incredible passive noise-cancelling. When you add active suppression to that, it just blows the AirPods away.

Comfort

The 35% increase in weight isn't noticeable, but I can't wear them for more than a couple of hours at a time because I get tired. For extended use, nothing beats over-ear headphones.

I noticed that the Sony headphones took longer to fit because of the memory foam. When I needed to quickly take a phone call, I often chose the AirPods since they were faster to put on.

Sports

None of these is an actual sports earphone. I'm a fan of Bose's SoundSport which has a cable around the neck and large controls you can easily manage while running. Unfortunately, most brands discontinued cable earphones.

I'm yet to try the new Bose Sport Earbuds, so I can't comment on those, but one thing's for sure, the AirPods Pro and the WF-1000XM4 fall short of proper sport headphones.

There's a major difference though, while I could (and did!) take the AirPods for a jog, I could not even pretend to try and run with the WF-1000XM4. Having both a closed back and memory foam, Sony's earbuds create an airtight compartment in your ear canal. That seal, the reason why they are fantastic at noise-cancelling, is also their undoing for sports.

When you run (at least when I run), there is a slight movement in your ears as you hit the asphalt. That motion also wiggles the earphones. Regrettably, as the WF-1000XM4 are airtight, that shift creates a difference in pressure that directly hits your tympanums. Painful.

Aerodynamics (added August 2024)

Lately, I’ve been cycling a lot and noticed that using the WF-1000XM4s against the wind significantly increases the noise. Even active noise-canceling can’t damp a small breeze. The buzz of the wind rushing past the oddly shaped Sony earbuds is deafening. In this regard, the AirPods Pro are the clear winner.

Verdict

After one month, I found myself leaving home exclusively with the AirPods Pro. They're more versatile, faster to engage and, letting some outside noise in, makes them safer on the streets.

I'd say Sony designed the WF-1000XM4 with a specific purpose, whereas Apple made an all-round product.

Sony's WF-1000XM4 - 170 USD
Apple's AirPods Pro - 190 USD

Hope this helps steer your choice.

Edit: Noise Cancelling Chart

After a question from u/CorvusCascadia I've superimposed both charts from the soundguys.com so that we may better visualize the surreal difference in noise attenuation.

How to list all known implementations of an interface is something that I'm asked a lot. I'm biased towards two ways of doing it.

Java 6 brought us the Service Provider Interface (SPI) which has at its core the ServiceLoader class, responsible for loading implementations.

Java itself uses the Service Provider Interface (SPI) in many ways:

• CurrencyNameProvider
• NumberFormatProvider
• TimeZoneNameProvider

Using it is straightforward. You just have to declare all providers in a provider configuration file. In order to do so, put them in a META-INF/services/com.example.providers.CustomProvider and the content of this file will list the existing providers:

com.example.providers.ProviderOfFood
com.example.providers.ProviderOfHappiness

I don't really like the above solution, it's brittle and prone to errors. I've seen programs where providers were not available as there was a typo, and I've also seen lost hours because developers refactored classes forgetting to update the respective META-INF files.

Both libraries below achieve the same.

Say you have the following code:

public interface Provider {
    void sayHello();
}

@AutoService(Provider.class)
public class ProviderOfFood implements Provider {
    ...
}

@AutoService(Provider.class)
public class ProviderOfHappiness implements Provider {
    ...
}

Google AutoService

At compile time, Google AutoService searches your code for annotations of type @AutoService and automatically writes the META-INF file for you. This implementation, having the search at compile time, is the fastest one. You would call it like this:

ServiceLoader<Provider> providers = ServiceLoader.load(Provider.class);
for (Provider p : providers) {
    p.sayHello();
}

Ronmano Reflections

Finds your classes at runtime using reflections. To make faster, you must narrow down the search scope by providing some package info to the library. This implementation has the benefit of working better with hot-reloaded code. Though you'll pay the price in speed. You would call it like this:

Reflections reflections = new Reflections("com.example.providers");
Set<Class<?>> subTypes = reflections.get(SubTypes.of(Provider.class).asClass());

for (Class<?> aClass : subTypes) {
    Constructor<?> ctor = aClass.getConstructor();
    Provider p = (Provider) ctor.newInstance();
    p.sayHello();
}

Which library should we use?

Depends.

ServiceLoader is 10x faster than Ronmano Reflections, but even on very modest hardware, running Ronmano Reflections 10_000 times will only take you back an extra ~8 seconds.

When you use ServiceLoader you initialize the classes. If you just want to list them, you can read the META-INF file.

Using Ronmano Reflections tough, you get the list of the classes, and you have to initialize them by hand. That, depending on the situation, may be a plus.

I'm a fan of mechanical keyboards, but this season gifted some LogicTech K120 to family and friends who I knew had outrageously worn out keyboards. Huge success! If I can't offer a perishable item, I try to make it something that wears out. I've learnt my lesson on decorative articles a long time ago! ;)