The architecture of an end-to-end LoRaWAN solution include entities such as Devices, Gateways, Network Server, Custom application/integrations. Hence, with some many entities involved, at times, it becomes important to have simultaneous end-to-end chain of logs to know what might be causing an issue. For example, a downlink failure can occur at Network Server level or at physical Gateway etc. So, to pinpoint the issue, logs at each level would be very helpful.
Collecting Debug logs from a Physical Device:
The process varies based on device model. Please refer to device guide/manuals or contact manufacturer.
Collecting Packet Forwarder logs from a Physical Gateway:
The process varies based on gateway model. Please refer to gateway guide/manuals or contact manufacturer.
Collecting Events from The Things Stack (Network Server):
The Things Stack generates lots of events that allow you to get insight in what is going on when you face an issue. You can subscribe to relevant application, gateway, end device events as and when necessary.
Prerequisites:
Install CLI. Ref: https://www.thethingsindustries.com/docs/getting-started/cli/
Create a User API Key:
Login to The Things Stack Console.
Click on
Username
from the top right corner of the Console.Click on
Personal API keys
from the dropdownClick
+Add API
key buttonYou can create a
User API Key
in The Things Stack Console by assigning the relevant rights. Please consider the below rights as minimum rights.Link as Application to a Network Server for traffic exchange, i.e. read uplink and write downlink
Read application traffic (uplink and downlink)
View gateway status
Read gateway traffic
Step 1:
Login to CLI using the below ttn-lw-cli
command.
$ ttn-lw-cli login --api-key <API KEY>
Step 2:
Command to subscribe to the event stream of an end-device and save them to a text file:
ttn-lw-cli events subscribe --application-id APPLICATION_ID --device-id DEVICE_ID >> DEVICE_ID-events.log
Command to subscribe to the event stream of a gateway and save them to a text file:
ttn-lw-cli events subscribe --gateway-id GATEWAY_ID >> GATEWAY_ID-events.log
Now, the GATEWAY-events.log
and DEVICE_ID-events.log
files will keep logging the events till you terminate the streams. These can be used for further analysis.
Ref: https://thethingsindustries.com/docs/getting-started/events/
Tips:
You can use multiple CMD/terminal windows to run the above commands simultaneously for devices and gateways of a particular test case.
Make sure to let the command run in the CMD over a period in which you observe the issue and then terminate.
Login to the CLI using the
OAuth Access
token will allow subscribing to the events, but only for a shorter-duration as theOAuth Access
token expire after an hour. Hence, login usingUser API Key
is suggested in this guide.
Additionally, if you are on self-hosted/enterprise deployments, you can also access more logs to know what is happening at the stack level. Refer below.
Enabling DEBUG log level in different distributions of The Things Stack
In The Things Stack DEBUG
level logs are not enabled by default. Based on the distribution type, below are the instructions to enable DEBUG
level logs.
After collecting the relevant logs for debugging, you can change the log level to info
. Please note that the DEBUG
level generates more logs and would incur cost if deployments are hosted on Cloud Services.
Enterprise/Open Source Deployment:
Edit The Things Stack configuration
ttn-lw-stack-docker.yml
file and add the below lines.log: level: debug
Restart the stack using the below command.
docker-compose restart
Collect the logs using the below command
docker logs -t <stack_container_name>
Note: Make sure you are in the right directory that has the docker-compose.yml
to restart the stack.
AWS AMI Deployment:
SSH into the EC2 instance. Please see How can I SSH into my machine reference for information to SSH into the EC2 instance.
Add the following lines to the file
/tti/lorawan-stack/config.yml
.
log: level: debug
3. Restart the service for it to come into effect.
$ sudo systemctl restart lorawan-stack.service
4. The debug logs can be read using the journalctl
$ sudo journalctl -f -u lorawan-stack.service
AWS ECS Deployment:
Go to the
4-2a-configuration
CloudFormation stack.Navigate to the
Resources
section and findGlobalConfiguration
Go to its Physical ID and edit the configuration
Add the below lines to the configuration file.
log: level: debug
Restart the ECS tasks to apply the configuration.
Check the CloudWatch log events for DEBUG logs.