Say what?

A recent work project utilised the power of Amazon’s MemoryDB to build a distributed caching layer (if you wish to find out more about Amazon MemoryDB see here). Under the hood, MemoryDB effectively provides a Redis Cluster running in the cloud.

The project in question is written primarily in Java and executes as a series of Lambda functions running within a couple of Step Functions. As part of some improvement work carried out on the project we abstracted any execution of Redis commands into a custom repository layer, built on top of the excellent Lettuce library. All was running smoothly until we wanted to test our changes…

This is where the fun began.

Current State

The TLDR; is that the current state of testing libraries for Redis in Java is pretty poor. The initial thought was to make use of an in-memory mock library (which is common practices for many DB’s e.g. via h2). A quick Google turns up two popular libraries:

GitHub - fppt/jedis-mock: A simple redis java mock for unit testing
- Actively maintained but not suitable for our project as we use Lettuce
GitHub - kstyrc/embedded-redis: Redis embedded server for Java integration testing
- Not actively maintained and didn’t support the version of Redis we use (version 6)

What’s more, digging though the documentation/issues for both libraries highlight that clustering is not a feature that either library supports: jedis-mock/supported_operations and Add support a real Redis Cluster · Issue #55 · kstyrc/embedded-redis.

Exploring the many forks available for embedded-redis did turn up a few versions capable of creating a Redis cluster e.g. GitHub - aarondwi/embedded-redis-cluster: Embedded Redis Cluster Server, to be used in local/dev settings and/or integration test and GitHub - tarossi/embedded-redis: Redis embedded server for Java integration testing respectively. However, both versions utilised old versions of Redis and didn’t implement the full suite of required operations.

Ultimately the idea of using an in memory mock was abandoned!

Testcontainers

The next thought was to use a real instance of Redis during testing rather than an in memory mock. To accomplish this we looked to use Docker and the pretty awesome Testcontainers library. This is again a fairly standard pattern when wanting to perform integration tests.

To make things easier, or so was believed, the container image used to create the test Redis Cluster was GitHub - Grokzen/docker-redis-cluster: Dockerfile for Redis Cluster (redis 3.0+) which creates a Redis cluster within a single container.

Networking

Now our story takes yet another turn. To understand the problem a little more context is required. The project in question makes use of Jenkins for CI. The Jenkins build slaves run on AWS as containers within EKS.

Clustering within Redis (see this doc for detailed information) works by creating a mesh network (effectively a TCP connection) between all nodes in a cluster. This is achieved via a cluster meet command that looks like the following:

CLUSTER MEET ip port

It’s important to note that the meet command uses an IP address and not a hostname.

Now back to our project - the Jenkins build slave is a docker container that is responsible for creating a sibling docker container via a shared docker socket. The Redis cluster is created in a new container, each node within the cluster forms a mesh using something like the following command:

CLUSTER MEET 127.0.0.1 7000

From the above we can see that the cluster nodes register themselves under 127.0.0.1.

When an integration test is executed it first creates a connection to the master node within the cluster. We can do so by making use of the exposed IP & port provided by Testcontainers. However, one of the features of connecting to a Redis cluster with Lettuce is that the clusters topology (i.e. the nodes that make up the cluster) is created/refreshed as part of the instantiation of a StatefulRedisClusterConnection (to determine what slots are assigned to which node in the cluster). This process attempts to communicate with each node in the cluster using the IP address and port registered during the meet phase. So essentially the connection code in our Java project, running on our Jenkins CI docker image, attempts to create a TCP connection to 127.0.0.1:7000 - which nothing is listening on and therefore, once again causes our test’s to fail :(

I’m not going to lie - I did consider giving up at this point as I couldn’t initially figure out what was needed. However, I decided to take a break and come back with a fresh pair of eyes.

Solution

Finally, the thing that most people will be interested in - how did I manage to resolve this issue. I’ve shared some code snippets and accompanying explanations below.

BaseTestClass

The following base class was created to instantiate our Redis Cluster via Testcontainers.

@Testcontainers
public class BaseTestClass {

    private static final DockerImageName REDIS_IMAGE = DockerImageName.parse("bitnami/redis-cluster:6.2");
    private static final MockedStatic<EnvUtil> mocked = Mockito.mockStatic(EnvUtil.class);

    public static GenericContainer<?> CLUSTER = new GenericContainer<>(REDIS_IMAGE)
            .withEnv("ALLOW_EMPTY_PASSWORD", "yes")
            .withEnv("REDIS_CLUSTER_REPLICAS", "0")
            .withEnv("REDIS_NODES", "127.0.0.1 127.0.0.1 127.0.0.1")
            .withEnv("REDIS_CLUSTER_CREATOR", "yes")
            .withEnv("REDIS_CLUSTER_DYNAMIC_IPS", "no")
            .withEnv("REDIS_CLUSTER_ANNOUNCE_IP", "127.0.0.1")
            .withExposedPorts(6379);

    static {
        CLUSTER.waitingFor(Wait.forLogMessage(".*Cluster state changed: ok*\\n", 1))
                .start();
    }

    @BeforeAll
    static void setup() {
        mocked.when(() -> EnvUtil.getInteger("REDIS_MEMORY_DB_PORT_NAME")).thenReturn(CLUSTER.getMappedPort(6379));
        mocked.when(() -> EnvUtil.getString("REDIS_MEMORY_DB_HOST_NAME")).thenReturn(CLUSTER.getHost());
    }
}

This code:

Uses the Redis cluster image provided by bitnami
- This allows us to create a single node cluster which makes the subsequent connection code easier. It still is possible to use the Redis cluster image provided by Grokzen but this requires us to work with at least 3 exposed ports (as the minimum size for the cluster is 3).
Starts the Redis cluster docker container and waits for the message Cluster state changed: ok to appear in the logged messages to ascertain that the cluster created successfully.
Mocks calls to a utility function that looks up environment variables (used in the connection code) to use the values provided by Testcontainers

Connection code

Next, we need to update our connection to the Redis Cluster to make use of the container.

final RedisURI uri = RedisURI.create(host, port);
final RedisClusterClient client = RedisClusterClient.create(uri);

// Test code added for illustration
final RedisClusterNode node = new RedisClusterNode();
node.setUri(uri);
node.setSlots(IntStream.range(1, 16384).boxed().collect(Collectors.toList()));

final Partitions partitions = new Partitions();
partitions.add(node);
client.setPartitions(partitions);
// End test code

client.setOptions(ClusterClientOptions.builder()
        .pingBeforeActivateConnection(true)
        .build());

return client.connect();

The above shows how the connection to the cluster needs to be updated. How you implement this in your code will likely differ from the above. You would obviously only do this in a way that is used by your tests! Here’s an explanation of what is happening:

Create a Redis Uri that will use the host and port values we mocked in the base class (exact implementation for getting host and port omitted - but essentially look up from some env variables)
Manually create a cluster node specifying that its URI is the same as the URI used by the cluster client (which it is as its the only node in the cluster) and that all available slots are on that node (which is again true as we have a single node cluster)
Update the clustered client specifying that it contains a single partition - our single clustered node

Tests

Last but not least, we need to write a test that makes use of these changes. The test below is representative.

public class RepositoryTest extends BaseTestClass {

    private Repository repository;

    @BeforeEach
    public void startUp() {
        final StatefulRedisClusterConnection<String, String> connection = RedisConnector.getConnection();
        this.repository = new Repository(connection);
    }

    @Test
    public void shouldSaveData() {
        final String key = "key";
        final String field = "field";
        final String value = "{\"test\": \"data\"}";

        this.repository.save(key, field, value);

        final Optional<String> actual = repository.getFromHash(key, field, String.class);

        assertThat(actual.isPresent(), is(true));
        assertThat(actual.get(), is(expected));
    }
}

The above test code:

Extends the BaseTestClass, ensuring that the cluster is created by Testcontainers before any test is executed
Before each test is executed, creates a connection to cluster using the modified connection code
Create our repository using the connection to the Redis cluster running in docker.
The rest is normal test code - essentially save something to the DB, get it back out and check that the two match

And that’s it - we can now continue to implement integration tests for our Redis cluster. I hope you found this guide useful.

Say what?#

Current State#

Testcontainers#

Networking#

Solution#

BaseTestClass#

Connection code#

Tests#