Redis is awesome! Can be used for different purposes and has clients available in various languages.

Last week we had an issue where the keys were saving with a larger expiration and the cpu usage of our redis instance started to grow reaching the limit…. So, after digging the cause we found that we had millions of keys with the expire set to a larger number that we want and decided to re-set the expiration of those keys.

So, the first approach we take was run a bash script using the redis-cli

1
2
3
4

#!/bin/bash
redis-cli --scan --pattern <prefix>:* | \
awk -v cmd_template="expire __key__ <expire_time>" '/^/ {cmd=cmd_template; gsub(/__key__/, $1, cmd); split(cmd, cmdargs," "); printf "*" length(cmdargs) "\r\n"; for(i=1; i<=length(cmdargs); i++) {printf "$" length(cmdargs[i]) "\r\n" cmdargs[i] "\r\n" }};' | \
redis-cli --pipe

The script was making his job…but it was slow, so we move to a different approach, running a lua script and looking for a solution to bulk operations using lua I found this post describing a similar scenario.

Next step was making small changes to the script in the post, so we end running with a cursor of 1000 keys with this two script.

The first one is a bash script used to load/call the lua one.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#!/bin/bash

if [ $# -ne 3 ]
then
  echo "Expire keys from Redis matching a pattern using SCAN & EXPIRE"
  echo "Usage: $0 <host> <port> <pattern>"
  exit 1
fi

cursor=-1
keys=""

while [[ $cursor -ne 0 ]]; do
  if [[ $cursor -eq -1 ]]
  then
    cursor=0
  fi

  reply=$(redis-cli -h $1 -p $2 SCAN $cursor MATCH $3 COUNT 1000)
  cursor=$(expr "$reply" : '\([0-9]*[0-9 ]\)')
  echo "Cursor: $cursor"

  keys=$(echo $reply | awk '{for (i=2; i<=NF; i++) print $i}')
  [ -z "$keys" ] && continue

  keya=( $keys )
  count=$(echo ${#keya[@]})
  redis-cli -h $1 -p $2 EVAL "$(cat expire.lua)" $count $keys
done

and the lua

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

local modified={};

for i,k in ipairs(KEYS) do
    local ttl=redis.call('ttl', k);
    if ttl >= 86400 then
        redis.call('EXPIRE', k, 60)
        modified[#modified + 1] = k;
    end
end

return modified;

After run this script, we saw how the keys start expiring and the cpu came back to its normal metric. in this case not only redis was awesome, but lua also!!

You can check the original code in github, I saved this gem to the next I need to make bulk operations in redis.

Also posted in collected notes.