Golang Redis Pipelines, WATCH, and Transactions
Redis pipelines allow to improve performance by executing multiple commands using a single client-server-client round trip. Instead of executing 100 commands one by one, you can queue the commands in a pipeline and then execute the queued commands using a single write + read operation as if it is a single command.
Pipelines
To execute multiple commands with a single write + read operation:
pipe := rdb.Pipeline()
incr := pipe.Incr(ctx, "pipeline_counter")
pipe.Expire(ctx, "pipeline_counter", time.Hour)
cmds, err := pipe.Exec(ctx)
if err != nil {
panic(err)
}
// The value is available only after Exec is called.
fmt.Println(incr.Val())
Alternatively, you can use Pipelined
which calls Exec
when the function exits:
var incr *redis.IntCmd
cmds, err := rdb.Pipelined(ctx, func(pipe redis.Pipeliner) error {
incr = pipe.Incr(ctx, "pipelined_counter")
pipe.Expire(ctx, "pipelined_counter", time.Hour)
return nil
})
if err != nil {
panic(err)
}
// The value is available only after the pipeline is executed.
fmt.Println(incr.Val())
Pipelines also return the executed commands so can iterate over them to retrieve results:
cmds, err := rdb.Pipelined(ctx, func(pipe redis.Pipeliner) error {
for i := 0; i < 100; i++ {
pipe.Get(ctx, fmt.Sprintf("key%d", i))
}
return nil
})
if err != nil {
panic(err)
}
for _, cmd := range cmds {
fmt.Println(cmd.(*redis.StringCmd).Val())
}
Watch
Using Redis transactions, you can watch for changes in keys and execute the pipeline only if the watched keys have not changed by another client. Such conflict resolution method is also known as optimistic locking.
WATCH mykey
val = GET mykey
val = val + 1
MULTI
SET mykey $val
EXEC
Transactions
You can wrap a pipeline with MULTI and EXEC commands using TxPipelined
and TxPipeline
, but it is not very useful on its own:
cmds, err := rdb.TxPipelined(ctx, func(pipe redis.Pipeliner) error {
for i := 0; i < 100; i++ {
pipe.Get(ctx, fmt.Sprintf("key%d", i))
}
return nil
})
if err != nil {
panic(err)
}
// MULTI
// GET key0
// GET key1
// ...
// GET key99
// EXEC
Instead, you should transactional pipelines with Watch, for example, we can correctly implement INCR command using GET
, SET
, and WATCH
. Note how we use redis.TxFailedErr
to check if the transaction has failed or not.
// Redis transactions use optimistic locking.
const maxRetries = 1000
// Increment transactionally increments the key using GET and SET commands.
func increment(key string) error {
// Transactional function.
txf := func(tx *redis.Tx) error {
// Get the current value or zero.
n, err := tx.Get(ctx, key).Int()
if err != nil && err != redis.Nil {
return err
}
// Actual operation (local in optimistic lock).
n++
// Operation is commited only if the watched keys remain unchanged.
_, err = tx.TxPipelined(ctx, func(pipe redis.Pipeliner) error {
pipe.Set(ctx, key, n, 0)
return nil
})
return err
}
// Retry if the key has been changed.
for i := 0; i < maxRetries; i++ {
err := rdb.Watch(ctx, txf, key)
if err == nil {
// Success.
return nil
}
if err == redis.TxFailedErr {
// Optimistic lock lost. Retry.
continue
}
// Return any other error.
return err
}
return errors.New("increment reached maximum number of retries")
}
Monitoring Performance
Monitoring the performance of a Redis database is crucial for maintaining the overall health, efficiency, and reliability of your system. Proper performance monitoring helps identify and resolve potential issues before they lead to service disruptions or performance degradation.
Uptrace is a OpenTelemetry APM that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.
Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.
Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.
In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.