Collecting Logs for Debugging Purposes

Owned by Tarun Krishna T
Last updated: 30 Dec 2022 by Manikanta Y
4 min read

The architecture of an end-to-end LoRaWAN solution include entities such as Devices, Gateways, Network Server, Custom application/integrations. Hence, with some many entities involved, at times, it becomes important to have simultaneous end-to-end chain of logs to know what might be causing an issue. For example, a downlink failure can occur at Network Server level or at physical Gateway etc. So, to pinpoint the issue, logs at each level would be very helpful.

Collecting Debug logs from a Physical Device:

The process varies based on device model. Please refer to device guide/manuals or contact manufacturer.

Collecting Packet Forwarder logs from a Physical Gateway:

The process varies based on gateway model. Please refer to gateway guide/manuals or contact manufacturer.

Collecting MAC State of the end device from The Things Stack (Network Server):

MAC State describes the current state of the end device in the current session. It contains the current parameters, desired parameters, as well as the last 20 uplinks and downlinks of the end device. To download the device's MAC state, follow the below steps.

Step 1: Navigate to the Device Overview page tab in TTS Console

Step 2: Click on the Download MAC data button from the MAC data section at the bottom of The Things Stack Console.

Collecting Events from The Things Stack (Network Server):

The Things Stack generates lots of events that allow you to get insight in what is going on when you face an issue. You can subscribe to the relevant application, gateway, and end device events as and when necessary using any of the two methods mentioned below based on the scenario.

Method 1: From TTS Console

Step 1: Navigate to Device &/ Gateway Live data tab in TTS Console

Step 2: Enable Verbose Stream

Step 3: Perform the tests or Reproduce the scenario/issue

Step 4: Once your tests are completed, you can simply click on the Export as JSON button.

Note:
1. Follow the same process for collecting Device & Gateway event logs.
2. If you want to run tests for a longer span and require logs of the whole test duration, this method might not be suitable. You can use Method 2 for this purpose, which is collecting logs using CLI.

Method 2: Using CLI

Prerequisites:

  1. Install CLI. Ref: https://www.thethingsindustries.com/docs/getting-started/cli/

  2. Create a User API Key:

    • Login to The Things Stack Console.

    • Click on Username from the top right corner of the Console.

    • Click on Personal API keys from the dropdown

    • Click +Add API key button

    • You can create a User API Key in The Things Stack Console by assigning the relevant rights. Please consider the below rights as minimum rights.

      • Link as Application to a Network Server for traffic exchange, i.e. read uplink and write downlink

      • Read application traffic (uplink and downlink)

      • View gateway status

      • Read gateway traffic

Step 1:

Login to CLI using the below ttn-lw-cli command.

$ ttn-lw-cli login --api-key <API KEY>

Step 2:

  • Command to subscribe to the event stream of an end-device and save them to a text file:

ttn-lw-cli events subscribe --application-id APPLICATION_ID --device-id DEVICE_ID >> DEVICE_ID-events.log

  • Command to subscribe to the event stream of a gateway and save them to a text file:

ttn-lw-cli events subscribe --gateway-id GATEWAY_ID >> GATEWAY_ID-events.log

Now, the GATEWAY-events.log and DEVICE_ID-events.log files will keep logging the events till you terminate the streams. These can be used for further analysis.
Ref: https://thethingsindustries.com/docs/getting-started/events/

Tips:

  1. You can use multiple CMD/terminal windows to run the above commands simultaneously for devices and gateways of a particular test case.

  2. Make sure to let the command run in the CMD over a period in which you observe the issue and then terminate.

  3. Login to the CLI using the OAuth Access token will allow subscribing to the events, but only for a shorter-duration as the OAuth Access token expire after an hour. Hence, login using User API Key is suggested in this guide.

 

Additionally, if you are on self-hosted/enterprise deployments, you can also access more logs to know what is happening at the stack level. Refer below.

Enabling DEBUG log level in different distributions of The Things Stack

In The Things Stack DEBUG level logs are not enabled by default. Based on the distribution type, below are the instructions to enable DEBUG level logs.

After collecting the relevant logs for debugging, you can change the log level to info. Please note that the DEBUG level generates more logs and would incur cost if deployments are hosted on Cloud Services.

Enterprise/Open Source Deployment:

  1. Edit The Things Stack configuration ttn-lw-stack-docker.yml file and add the below lines.

  2. Restart the stack using the below command.

  3. Collect the logs using the below command

Note: Make sure you are in the right directory that has the docker-compose.yml to restart the stack.

AWS AMI Deployment:

  1. SSH into the EC2 instance. Please see How can I SSH into my machine reference for information to SSH into the EC2 instance.

  2. Add the following lines to the file /tti/lorawan-stack/config.yml.

3. Restart the service for it to come into effect.

4. The debug logs can be read using the journalctl

AWS ECS Deployment:

  1. Go to the 4-2a-configuration CloudFormation stack.

  2. Navigate to the Resources section and find GlobalConfiguration

  3. Go to its Physical ID and edit the configuration

  4. Add the below lines to the configuration file.

  5. Restart the ECS tasks to apply the configuration.

  6. Check the CloudWatch log events for DEBUG logs.