Redis Integration

Initially, the status of background analysis tasks was tracked using simple in-memory Python dictionaries. However, this approach proved unsuitable for the production deployment using Gunicorn with multiple workers. Since each worker process has its own isolated memory space, the in-memory storage could not be shared between the worker handling the analysis updates and the worker responding to status requests from the frontend.

To resolve this, Redis was introduced to provide a centralized, shared store for task state information accessible by all Gunicorn workers[cite: 656].

Implementation Details:

• Setup: Redis was configured to run locally on the VPS.
• Data Handling: Helper functions were implemented within the FastAPI application to manage interactions with Redis. Task state dictionaries are serialized to JSON before being stored in Redis and deserialized upon retrieval.
• Expiration (TTL): To ensure efficient resource usage and prevent Redis from filling up with old task data, a short Time-To-Live (TTL) of 15 seconds was set for task entries after they reach a completed state. This allows enough time for the frontend to fetch the final status while ensuring automatic cleanup.