Skip to content

Instantly share code, notes, and snippets.

@KFoxder

KFoxder/readme.md

Last active June 16, 2020 21:52
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save KFoxder/d5b44aff3fe066290be5c96c4072df57 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save KFoxder/d5b44aff3fe066290be5c96c4072df57 to your computer and use it in GitHub Desktop.
Download ZIP
Celery / Redis / Kombu Upgrade Notes
Raw
readme.md

Celery / Redis / Kombu

Django Autocompleter Changes

  1. Revert this commit: https://github.com/ycharts/django-autocompleter/commit/ca9881e24da522302c449f36be0a574d36c5332e
  2. Revert this commit: https://github.com/ycharts/django-autocompleter/commit/73326273d175424a0af2202134e9dfa1a21de724
  3. Bump version to latest dot release ( at the time of writing this would be 0.8.4)

YCharts Repo Changes

  1. hiredis (hiredis-py) is upgrade to 1.0.0 from 0.2.0
  2. Redis (redis-py) upgrade to 3.4.1 from 2.10.5
    1. Read the migration guide for reference: https://github.com/andymccurdy/redis-py#upgrading-from-redis-py-2x-to-30
      • We use the Redis client instead of StrictRedis which is now no longer supported since StrictRedis. So we need to make sure that the differing implementaitons that Redis client had for zadd and pipeline do not require changes in our code. We use both zadd and pipeline in our code.
    2. No changes for django-redis-sessions
    3. No changes for redlock
  3. Upgrade Celery to 4.4.x from 4.2.0
  4. Upgrade Kombu to the latest version (4.6.8 from 4.3.0)
  5. Revert this commit: https://github.com/ycharts/ycharts/pull/6599/files#diff-9eb78032baccc38f3f987c4fc851e30e
    • Specifically change that code related to catching the exception since it shouldn't happen in the latest version.
  6. Add back these reverted changes in this PR: https://github.com/ycharts/ycharts/pull/5782/files
    1. Make sure you get the django-autocompleter version correct though after you update the version in Django Autocompleter
    2. Specifically audit all uses of zadd since they need to be updated.
  7. Determine if we need to change the health_check_interval in BROKER_TRANSPORT_OPTIONS
@KFoxder
Copy link
Author

KFoxder commented Jun 16, 2020

@jls83 @kdabiedeen couple things to point out post these changes:

  1. Should we be using ConnectionPool? https://github.com/andymccurdy/redis-py#connection-pools
    • I feel like this would be something we should probably do especially when it comes to our web machines. Can you take a deeper look and propose a way forward or why we shouldn't?
  2. If use a connection pool is that when we would really need to use health_check_interval config option?
  3. Can we update all uses of .pipeline() to be used a context manager?
  4. Should we be setting retry_on_timeout? Why not?
  5. Can you read through all changes and make sure there isn't anything else to consider while we are deep in redis land?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment