Debugging Data
The following commands are used for requesting the TAC debug information.
tac debug pkg create and delete
New command
Old commandtac-debug-pkg create last<time_to_now> tac-debug-pkg delete last<time_to_now>
tac-debug-pkg create {from start_time | to end_time} {logs-filter namespace namespace | pod_name pod_name} {cores-filter { process process_name }} {{ cfg | cores | logs | metrics | stats } {false | true}} tac-debug-pkg delete tac-id tac_id
Previously, the command syntax required a user to specify a time period by entering from and to criteria.
The new syntax for tac-debug-pkg create and tac-debug-pkg delete commands now allows users to specify the duration relative to the current time using the last keyword:
tac-debug-pkg create last<time_to_now> tac-debug-pkg delete last<time_to_now>
<time_to_now> specifies the time to now in terms of the number of:
Days - Expressed as "D", “d”, or “day”; for example “5D”
Hours - Expressed as “H”, “h”, or “hour”; for example “3h”
Minutes - Expressed as “M”, “m”, “min”, or “minute”; for example “18minute”
Seconds - Expressed as “S”, “s”, “sec”, or “second”; for example “3600sec”
Additionally, omitting the to keyword from the from parameter instructs the system to collect the TAC package from the specified time until now:
tac-debug-pkg create from <time_to_now>
The from keyword no longer requires the use of the to keyword if you are creating the TAC package from a specific time until now.
tac-debug-pkg usage examples User Intention
Command
collect tac-debug-package for last 50 seconds
tac-debug-pkg create last 50S
collect tac-debug-package for last 10 minutes
tac-debug-pkg create last 10min
collect tac-debug-package for last 3 hours
tac-debug-pkg create last 3H
collect tac-debug-package for last 7 days
tac-debug-pkg create last 7day
delete all collected tac-debug-package for the past 2 days
tac-debug-pkg delete last 2D
collect tac-debug-package from 2019-08-09_01:00:00 to now
tac-debug-pkg create from 2019-08-09_01:00:00
Other tac debug pkg commands
tac-debug-pkg merge tac-id tac_id tac-debug-pkg status tac-debug-pkg list
-
Access the Apache server through the Ops Center.
URL: https://show-tac-manager.smi-show-tac.{IP address}.<domain_name>
-
A directory is created based on the tac-id: /tac/[tac-id]/
-
A manifest file is created for each of the tac-debug-pkg to the store metadata. A sample manifest.json file is shown below:
{ tac-id:"1554868784", from:"2019-04-08_00:00:00", to:"2019-04-10_00:00:00", cores:[{node:"node-01", file:"/cores/node-01/core.test.0.2f4afbe0dc494e879d3f42429fed1c38.20130.1554770483000000.xz"}, {node:"node-01", file:"/cores/node-01/core.test.0.2f4afbe0dc494e879d3f42429fed1c38.18448.1554770577000000.xz"}], config:[{node:"node-01", file:"/tac/1554868784/config/<ipv4address>_configuration.tar.gz.base64"}], stats:[{node:"node-01", file:"/tac/1554868784/stats/Stats_2019-04-08_00-00-00_2019-04-10_00-00-00.tar.gz"}], logs:[{node:"node-01", file:"/tac/1554868784/logs/Logs_2019-04-10_04-00-17.tar.gz"}], metrics:[{node:"node-01", file:"/tac/1554868784/metrics/Metrics_2019-04-08_00-00-00_2019-04-10_00-00-00.tar.gz"}]} }
Important | Authentication to the Apache server is enabled by default. |
-
The following services collect and retrieve logs, data chunks, and bulk statistics.
-
Core retrieving service - This service retrieves the list of coredump based on the time duration. The systemd coredump service compresses the core files. The configuration parameters in the core files determines the name of the core file. Due to the core file large size, they are not copied on the disk. You can access it through the proxy from its original location.
file location: ./cores/{node name}/core.xxx... Sample file location on server: cores/node-1/core.test.0.99775297099c489ea08052d533206b66.10213.1554504010000000.xz
-
Logs retrieving service - This service collects Kernel, System, Containers level logs using JournalD service. In return, the sender receives a tar file which contains logs files based on the time duration. The files are created with following naming convention:
./tmp/logs/{random string}/{namespace}/{pod}/{container.log}
-
A sample file (Tar) format with the timestamp embedded in the file name is shown below:
./tmp/logs/{random string/Logs_{yyyy-mm-dd_hh-mm-ss}.tar.gz
-
Prometheus data retrieving service - This service retrieves snapshot of data chunks saved by the Prometheus service. You can specify the time duration for saving a snapshot. A sample file and directory structure for the data snapshot is shown below:
directory: data/snapshots/20190405T175611Z-7ee562389bd9ab66/01D7N0QVNBRXF5MRVFQB5MQQCW files: ./chunk/0001 ./index ./meta.json ./tombstones
-
Bulk statistics retrieving service - This service retrieves statistics saved by the Prometheus service. You can specify the time duration for saving the statistics. A tar file is stored onto the Apache server for review. A sample file location on the server is shown below:
tac/0123456789/stats/Stats_2019-4-04_00-00-00_2019-04-04_18-00-00.tar.gz
The following example collect logs for pods in cdl-global namespace for CDL application.
cee# tac-debug-pkg create from 2019-12-18_00:00:00 to 2019-12-18_20:00:00 logs-filter { namespace cdl-global }
response : Tue Dec 18 18:40:55 UTC 2019 tac-debug pkg ID : 157660805