Debugging a piece of code is an activity as important as writing it. The importance of debugging is much more noticeable on large projects like OpenStack, which require both developers and common users contribute and report when something is not going well.

Here are some pointers to collect useful logs and debugging processes.

Logging

Looking through the logs can be tricky... not if you know how to use regular expressions!

The first thing to do when we see a strange behavior that is repeated with the same input data, is to look at logs.

Logs are files, usually plain text and arranged chronologically, that contain information about the code that is executed to perform an action.

Analyzing the information found there, you may be able to detect what the problem is or, at least, where the error occurs.

The location of the logs in the system may vary depending on the implementation, but in GNU/Linux they usually are stored in /var/log.

In OpenStack the location of these logs varies according how have you deployed it: using DevStack or by hand, whether with the distribution repository packages you are using or from source.

If you're using DevStack keep in mind that by default, DevStack doesn’t log. To enable this option you will need to add the following lines to the localrc file.

LOGFILE=$DEST/stack.sh.log

– Name and location of the stack.sh script logs. Without this line the output of this script will appear in the terminal where you run it, without being stored in any location.

SCREEN_LOGDIR=$DEST/logs/screen

– Location of the screen logs. DevStack runs OpenStack services under GNU Screen, thus providing the facility to view its current execution status. The logs simply store the output of these screens..

LOGDAYS=1

– Number of days to be logged. The old logs are replaced with the latest runs.

$DEST is the location in which the logs will be located. You can set this value as you wish.

Screen logs are just a record of the screens initiated by DevStack but you can check them anytime during the execution. The information provided is very useful to debug any OpenStack service.

GNU Screen provides the same functions as the DEC VT100 terminal, plus other aggregates. If you have no experience with this terminal, make sure to write down the following shortcuts.

 Attach to a screen      screen -r stack
 Switch screens          ctrl+a <screen no.>  (e.g. ctrl+a 1) 
 Scroll in a screen      ctrl+a esc and up/down arrows to scroll
 List screens        ctrl+a " and select the screen with up/down arrows
 Next screen         ctrl+a n 
 Detach from the screen  ctrl+a d

Manual deployment

Here you will find most services logs in /var/log/service, where service is the name of the desired service. For example, in /var/log/keystone you will find Keystone logs.

In my experience, the only service that differs from this default location is Horizon. You can find ... (more)