When writing Python scripts, it’s common to want to display information messages in the user terminal and possibly save them to a file like output.log. Therefore, it’s essential to understand the levels of logging in Python, the types of outputs in Unix-like systems, and how to direct the appropriate type of logging to the right output.

Understanding Output Streams in Unix

In Unix systems, there are two primary streams for output: standard output (stdout) and standard error (stderr). These streams serve different purposes:

  • Standard Output (stdout): This stream is used by programs to display normal output to the user. It typically includes informative messages, data, and command results. By default, stdout is directed to the terminal, but you can redirect it to files or other processes.
  • Standard Error (stderr): Programs use this stream to display error output to the user, containing error messages and diagnostic information. It provides a separate channel for communicating problems. Like stdout, stderr is directed to the terminal by default, but it can also be redirected.

Understanding Python Logging Levels

Python’s logging module tracks events that occur during program execution, aiding in debugging errors, monitoring performance, and analysing usage patterns. The module includes different levels of severity for logging messages:

  • DEBUG: Detailed information, primarily useful for diagnosing problems.
  • INFO: Confirmation that things are working as expected.
  • WARNING: An indication that something unexpected happened, or a problem may arise in the near future.
  • ERROR: Indicates a more serious problem where the software couldn’t perform a function.
  • CRITICAL: A severe error, suggesting that the program may be unable to continue running.

By default, only messages with a level of WARNING or higher are recorded, but this can be configured. Additionally, logging output can be directed to various destinations, such as files, consoles, or other processes.

Example Implementation of Logging in Python

Let’s look at a minimal example of implementing logging in a Python script:

 1#!/usr/bin/env python
 2
 3import logging
 4
 5logger = logging.getLogger("example")
 6logger.setLevel(logging.WARNING)
 7
 8file_handler = logging.FileHandler("example.log")
 9logger.addHandler(file_handler)
10
11logger.debug("This is a debug message")
12logger.info("This is an info message")
13logger.warning("This is a warning message")
14logger.error("This is an error message")
15logger.critical("This is a critical message")

By running this script from the command line, a file named example.log is created with the logged messages from the warning level upwards:

1This is a warning message
2This is an error message
3This is a critical message

To also display output in the terminal, we need to add a console handler:

1console_handler = logging.StreamHandler()
2logger.addHandler(console_handler)

Now, logging output will appear in both the terminal and the example.log file.

Handling Output Redirection in Shell Scripts

Suppose we want the script to default to terminal output and then redirect it to example.log using the Unix output redirection operator > operator. In that case, we can structure the script like this:

 1import logging
 2
 3logger = logging.getLogger("example")
 4logger.setLevel(logging.WARNING)
 5
 6console_handler = logging.StreamHandler()
 7
 8logger.debug("This is a debug message")
 9logger.info("This is an info message")
10logger.warning("This is a warning message")
11logger.error("This is an error message")
12logger.critical("This is a critical message")

Running this script from the command line will output to the terminal, but not to any file. To redirect output to an example.log file, we expect that we do so by using the Unix > operator:

1python log_example.py > example.log

However, looking into the content of example.log, we see that the file is empty! So, what is going on?

Dealing with Python Logging Output

The confusion mentioned above arises because Python’s logging module directs output to stderr by default instead of stdout, whereas the > operator in Unix is employed to redirect stdout rather than stderr. To address this issue, we have several potential solutions.

  • Redirecting Both stdout and stderr: Use the &> operator in the shell to redirect both streams to one file.
  • Separating stderr: Use the 2> operator to capture stderr in a separate file while keeping stdout in the terminal.
  • Changing Python Logging: Adjust the Python script to direct logging output to stdout instead of stderr, making it easier to redirect using shell commands. This can be done as follows:
1import sys
2import logging
3
4# Configure logging to direct output to stdout
5console_handler = logging.StreamHandler(sys.stdout)

Using verbose for Logging Levels

Another useful tip is to pass a verbose argument to determine the level of logging:

1import sys
2import logging
3
4# Configure logging to direct output to stdout
5console_handler = logging.StreamHandler(sys.stdout)
6console_handler.setLevel(logging.DEBUG if verbose else logging.INFO)

This allows flexibility in setting the logging level based on the verbosity of the script’s execution.

Conclusion

Understanding how to direct output in both Python and shell scripting can streamline the debugging process. Whether redirecting output in the shell or adjusting Python scripts, mastering these techniques improves our log management skills, making development and troubleshooting more efficient.