Palo Alto Firewalls: CLI Commands for Troubleshooting
CLI Commands for Troubleshooting Palo Alto Firewalls
When troubleshooting network and security issues on many different devices I always miss some command options to do exactly what I want to do on the device I am currently working with. Therefore, I list a few commands for the Palo Alto Networks firewalls to have a short reference for myself. Maybe some other network professionals will find it useful.
However, since I am almost always using the GUI this short reference only lists commands that are useful for the console while not present in the GUI.
This blog post will be a living document. Whenever I use some “new” commands for troubleshooting issues, I will update it. If there are any useful commands missing, please send me a comment!
For a complete list of all CLI commands, use the CLI Reference Guides from PAN. Or use the official Quick Reference Guide: Helpful Commands PDF.
Standard Show Commands
The following commands are really the basics and need no further description. I list them just as a reference:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | show system info //shows the uptime of the device show session info //packet rate, # of sessions, fastpath active, etc. show session id show interface { all | name-of-the-interface } show routing route show routing protocol show arp all show mac all show jobs all show jobs id show running resource-monitor //Resource Statistics show system resource follow //=="top", CPU usage and processes show system disk-space debug software restart |
Find
Since PAN-OS 6.0, the “find” command helps searching for the needed command in case you do not fully know the whole set of commands. With “find command”, all possible commands are displayed. With “find command keyword xyz”, all commands containing “xyz” are shown.
1 2 | find command find command keyword |
Ping, Traceroute, and DNS
A standard ping command looks like that:
1 | ping host 8.8.8.8 |
Note that this ping request is issued from the management interface! To use a data interface as the source, the option source
1 | ping inet6 yes source 2003:51:6012:120::1 host 2a00:1450:4008:800::1017 |
A traceroute command looks like that:
1 | traceroute host 8.8.8.8 |
The source
To resolve DNS names, e.g., to test the DNS server that is configured on the management interface, simply ping a name:
1 | ping host ip.webernetz.net |
Routing
Debugging dynamic routing protocols functions like this:
1 2 3 4 5 | debug routing pcap debug routing pcap show debug routing pcap debug routing pcap debug routing pcap |
Or follow the routed.log:
1 | tail follow yes mp-log routed.log |
Test
The Palo offers some great test commands, e.g., for testing a route-lookup, a VPN connection, or a security policy match. Use the question mark to find out more about the test commands. Here are some useful examples:
1 2 3 4 | test routing fib-lookup virtual-router default ip test vpn ipsec-sa tunnel test security-policy-match ? test security-policy-match from trans-internet to pa-trust-server source 192.168.86.5 destination 192.168.120.2 protocol 6 application ssl destination-port 443 |
Viewing Management-Plane Logs
In order to view the debug log files, “less” or “tail” can be used. The keyword “mp-log” links to the management-plane logs (similar to “dp-log” for the dataplane-logs). The tail command can be used with “follow yes” to have a live view of all logged messages. And as always: Use the question mark in order to display all possibilities.
Examples:
1 2 3 4 | less mp-log ? less mp-log dnsproxyd.log tail follow yes mp-log dhcpd.log tail follow yes mp-log routed.log |
Capturing Management Packets
To view the traffic from the management port at least two console connections are needed. The first one executes the tcpdump command (with “snaplen 0” for capturing the whole packet, and a filter, if desired),
1 | tcpdump snaplen 0 filter "port 53" |
while the second console follows the live capture:
1 | view-pcap follow yes mgmt-pcap mgmt.pcap |
Test traffic can be generated with a third console session, e.g.:
1 | ping host webernetz.net |
Later on, the pcap file can be moved to another computer with the following command:
1 | scp export mgmt-pcap from mgmt.pcap to |
Alternatively, tftp can be used:
1 | tftp export mgmt-pcap from mgmt.pcap to |
Live Viewing of Packet Captures
When using the Packet Capture feature on the Palo Alto, the filter settings can easily be made from the GUI (Monitor -> Packet Capture). These settings as well as the current size of the running packet capture files can be examined with:
1 | debug dataplane packet-diag show setting |
Now, the current capturing in follow mode can be viewed with:
1 | view-pcap follow yes filter-pcap |
And for a really detailed analysis, the counters for these filtered packets can be viewed. This exactly reveals how many packets traversed which way, and so on. With the “delta yes” option, only the counter values since the last execution of this command are shown. The “packet-filter yes” option uses the packet filter from the GUI (Monitor -> Packet Capture) to filter the counters:
1 | show counter global filter packet-filter yes delta yes |
For example, here are the delta counters after a few DNS lookups:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | weberjoh@fd-wv-fw02> show counter global filter packet-filter yes delta yes Global counters: Elapsed time since last sampling: 44.689 seconds name value rate severity category aspect description -------------------------------------------------------------------------------- pkt_sent 24 0 info packet pktproc Packets transmitted pkt_outstanding 24 0 info packet pktproc Outstanding packet to be transmitted pkt_alloc 120 2 info packet resource Packets allocated session_allocated 19 0 info session resource Sessions allocated session_installed 19 0 info session resource Sessions installed flow_host_pkt_xmt 144 3 info flow mgmt Packets transmitted to control plane flow_host_service_allow 24 0 info flow mgmt Device management session allowed appid_ident_by_dport_first 19 0 info appid pktproc Application identified by L4 dport first dfa_sw 48 1 info dfa pktproc The total number of dfa match using software ctd_sml_vm_check_domain 24 0 info ctd pktproc sml vm check domain ctd_bloom_filter_nohit 24 0 info ctd pktproc The number of no match for virus bloom filter aho_sw 48 1 info aho pktproc The total usage of software for AHO ctd_pkt_slowpath 48 1 info ctd pktproc Packets processed by slowpath -------------------------------------------------------------------------------- Total counters shown: 13 -------------------------------------------------------------------------------- |
Or, even more interesting, filtered on “drop” severity. (Note the reasons on the right-hand side):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | weberjoh@fd-wv-fw02> show counter global filter delta yes severity drop Global counters: Elapsed time since last sampling: 166.755 seconds name value rate severity category aspect description -------------------------------------------------------------------------------- flow_rcv_dot1q_tag_err 726 4 drop flow parse Packets dropped: 802.1q tag not configured flow_no_interface 726 4 drop flow parse Packets dropped: invalid interface flow_ipv6_disabled 1 0 drop flow parse Packets dropped: IPv6 disabled on interface flow_tcp_non_syn_drop 50 0 drop flow session Packets dropped: non-SYN TCP without session match flow_fwd_l3_mcast_drop 50 0 drop flow forward Packets dropped: no route for IP multicast flow_fwd_l3_ttl_zero 9 0 drop flow forward Packets dropped: IP TTL reaches zero flow_fwd_zonechange 8 0 drop flow forward Packets dropped: forwarded to different zone flow_dos_pf_ipspoof 17 0 drop flow dos Packets dropped: Zone protection option 'discard-ip-spoof' flow_dos_pf_noreplyttl 6 0 drop flow dos Packets dropped: Zone protection option 'suppress-icmp-timeexceeded' -------------------------------------------------------------------------------- Total counters shown: 9 -------------------------------------------------------------------------------- |
Examining the Session Table
If a network connection failure is not found in the traffic log, the session table can be asked for sessions in DISCARD state, filtered based on its source, or whatever. This is useful at the console because the session browser in the GUI does not store the filter options and is therefore a bit unhandy. All commands start with “show session all filter …”, e.g.:
1 2 3 | show session all filter state discard show session all filter application dns destination 8.8.8.8 show session all filter from trust to untrust application ssl state active |
To have an overview of the number of sessions, configured timeouts, etc.:
1 | show session info |
For investigating a single session in more detail, use:
1 | show session id |
Watch out for the: “Hardware session offloading” line. If it is “true” you might want to disable the fastpath during troubleshooting (inside the config mode):
1 2 | set session offload no set deviceconfig setting session offload no //= persistent, even after reboot. CAUTION! |
To see whether there are some “predict” sessions in which the Palo Alto uses a ALG (appliation layer gateway) to predict dynamic ports (e.g., SIP, active FTP), use this command:
1 | show session all filter type predict |
A specific session can then be cleared with:
1 | clear session id |
Reason for Session Close
You cannot see the reason for a closed session in the traffic log in the GUI. For this purpose, find out the session id in the traffic log and type in the following command in the CLI (Named the “Session Tracker“). Note the last line in the output, e.g. “tracker stage firewall : Aged out” or “tracker stage firewall : TCP FIN”. This shows what reason the firewall sees when it ends a session:
1 | show session id |
Alternatively, the traffic log on the CLI can display the session tracker when used with the option “show-tracker equal yes” such as:
1 2 3 | show log traffic show-tracker equal yes show log traffic show-tracker equal yes direction equal backward show log traffic show-tracker equal yes direction equal backward app equal ipv6-icmp from equal pa-ripe-atlas |
VPN Issues
The general show commands for VPN sessions are:
1 2 | show vpn gateway show vpn ike-sa |
(Palo Alto: How to Troubleshoot VPN Connectivity Issues). Though you can find many reasons for not working site-to-site VPNs in the system log in the GUI, some more CLI commands might be useful. To reveal whether packets traverse through a VPN connection, use this: (it shows the number of encap/decap packets and bytes, i.e., the actual traffic flow)
1 | show vpn flow name |
Or use the counter values for ipsec issues:
1 | show counter global filter delta yes | match ipsec |
And for a detailled debugging of IKE, enable the debug (without any more options)
1 | debug ike pcap on |
then follow the pcap with
1 | view-pcap follow yes debug-pcap ikemgr.pcap |
and do NOT forget to set the debugging off!
1 | debug ike pcap off |
The complete ikemgr.pcap can be downloaded from the Palo with scp or tftp, e.g.:
1 | scp export debug-pcap from ikemgr.pcap to |
GlobalProtect
Current users and flow:
1 2 | show global-protect-gateway current-user show global-protect-gateway flow |
Displaying the Config in Set Mode
The XML output of the “show config running” command might be unpractical when troubleshooting at the console. That’s why the output format can be set to “set” mode:
1 | set cli config-output-format set |
Now, enter the configure mode and type show . This reveals the complete configuration with “set …” commands. (Click here for more information.) Here is a sample output of a particular show command:
1 2 3 4 5 6 7 | weberjoh@fd-wv-fw02# show network interface ethernet ethernet1/1 set network interface ethernet ethernet1/1 layer3 ip 172.16.1.2/24 set network interface ethernet ethernet1/1 layer3 untagged-sub-interface no set network interface ethernet ethernet1/1 layer3 interface-management-profile ping set network interface ethernet ethernet1/1 link-speed auto set network interface ethernet ethernet1/1 link-duplex auto set network interface ethernet ethernet1/1 link-state auto |
The pipe (|) can be used to grep certain values with the “match” keyword, such as:
1 2 3 | weberjoh@fd-wv-fw02# show | match 192.168.120.2 set deviceconfig system ip-address 192.168.120.2 set address h_fd-wv-fw02_mgmt ip-netmask 192.168.120.2 |
To show the complete config without breaks (which is “terminal length 0” on Cisco devices), the following command can be used (BEFORE the configure mode is entered):
1 | set cli pager off |
To omit line breaks (carriage returns), use this one:
1 | set cli terminal width 500 |
High Availability
Some show and request commands (even though present in the GUI):
1 2 3 4 5 6 7 | show high-availability ? show high-availability all show high-availability state show high-availability link-monitoring show high-availability path-monitoring request high-availability state suspend request high-availability state functional |
Export/Import Files
To copy files from or to the Palo Alto firewall, scp or tftp can be used. The commands have both the same structure with “export … to” or “import … from”, e.g.:
1 2 3 4 | scp export log system to scp import software from tftp export configuration from running-config.xml to tftp import url-block-page from |
User-IDs and Groups
State of the LDAP server connections:
1 | show user group-mapping state all |
List the groups that are stored in the Palo Alto:
1 | show user group list |
Manual group mapping refresh:
1 | debug user-id refresh group-mapping all |
Show the group memberships for a particular user:
1 | show user user-IDs match-user |
Show the members of a particular group:
1 | show user group name "AD\name-of-the-group" |
IP to User mapping:
1 | show user ip-user-mapping all |
User-ID cache clearance:
1 | clear user-cache all |
IP Addresses of FQDN Objects
When using objects with FQDNs, the current IP addresses are not shown in the GUI. The following command displays respectively refreshes them:
1 | request system fqdn { show | refresh } |
IP Addresses of Dynamic Block Lists
Similar, the entries in a dynamic block list can be viewed with:
1 | request system external-list show name |
DNS Proxy
To verify the functionality of DNS proxy objects, at least two commands are useful. Both outputs should speak for themselves:
1 2 | show dns-proxy statistics all show dns-proxy cache all |
Active URL Vendor/Database
I had some issues with the two different URL databases “brightcloud” and “PAN-DB”. This is the command to show unambiguously which vendor is active on the PA (independent of the licenses):
1 | show system setting url-database |
The output is either “brightcloud” or “paloaltonetworks”. The standard URL DB up to PAN-OS 5.0 is brightcloud. Beginning with PAN-OS 6.0, the default is PAN-DB (refer to the release notes, section “Changes to Default Behavior”). To change the vendor (of course only if it is licensed), click the “Activate” link under licenses in the GUI.
PAN-DB URL Test & Cache
To show the category of a specific URL, use one of the following commands:
1 2 3 | test url test url-info-cloud test url-info-host |
To display the current URL cache from the PAN-DB, two steps are required. The first one is the creation of a logfile which contains all entries and the second one is to display this logfile:
1 2 | show system setting url-cache all less dp-log dp_url_DB.log |
Fan Speed
Ok, this is not a troubleshooting command, but nevertheless very useful. It sets the fan speed to “auto” which immediately drops the noise of the fan, e.g. on a PA-200:
1 | set system setting fan-mode auto |
https://blog.webernetz.net/2013/11/21/cli-commands-for-troubleshooting-palo-alto-firewalls/
Defaults
Just for reference:
- Default Management Interface IP: 192.168.1.1
- Login: admin
- Password: admin
To change the IP settings of the management inferface via the console:
1 2 3 | configure set deviceconfig system ip-address 192.168.1.5 netmask 255.255.255.0 default-gateway 192.168.1.1 dns-setting servers primary 8.8.8.8 commit |
The change in the CLI is as follows:PAN-OS 7.0 and earlier releases:
set external-list recurring hourly at
PAN-OS 7.1 and later releases:
set external-list recurring hourly
With the new ability to specify intelligence sources through a list on an external domain, you must now specify the list. Example changes in the CLI follow:PAN-OS 7.0 and earlier releases:
show profiles spyware botnet-domains action
show profiles spyware botnet-domains action alert
show profiles spyware botnet-domains action allow
show profiles spyware botnet-domains action block
show profiles spyware botnet-domains action sinkhole
PAN-OS 7.1 and later releases:
show profiles spyware botnet-domains lists action
show profiles spyware botnet-domains lists action alert
show profiles spyware botnet-domains lists action allow
show profiles spyware botnet-domains lists action block
show profiles spyware botnet-domains lists action sinkhole
With the introduction of two-factor authentication in GlobalProtect, a number of API requests have been changed. Use the CLI with the command debug cli on to see changes in the corresponding XML requests. Affected commands are within the following command hierarchy:
set global-protect global-protect-portal satellite-config
set global-protect global-protect-portal client-config
set global-protect global-protect-portal portal-config
To view the results of a query, the request format has been updated to be uniform between firewalls and Panorama:PAN-OS 7.0 and earlier releases:
show query id <1-4294967295>
PAN-OS 7.1 and later releases:
show query result id <1-4294967295> skip <0-4294967295>
With the introduction of configurable maximum segment size (MSS) adjustment sizes, the request format to enable MSS adjustment has changed:PAN-OS 7.0 and earlier releases:
set network interface ethernet layer3 adjust-tcp-mss
set network interface ethernet layer3 units adjust-tcp-mss
set network interface vlan adjust-tcp-mss
set network interface vlan units adjust-tcp-mss
set network interface loopback adjust-tcp-mss
set network interface loopback units adjust-tcp-mss
PAN-OS 7.1 and later releases:
set network interface ethernet layer3 adjust-tcp-mss enable
set network interface ethernet layer3 units adjust-tcp-mss enable
set network interface vlan adjust-tcp-mss enable
set network interface vlan units adjust-tcp-mss enable
set network interface loopback adjust-tcp-mss enable
set network interface loopback units adjust-tcp-mss enable
The maintenance mode menu for selecting the mode of operation changed:Firewall platforms —The Set CCEAL4 mode menu is renamed to Set FIPS-CC mode. Additionally, the Set FIPS mode menu is removedPanorama virtual appliances, M-Series appliances, and WF-500 appliances —The Set CCEAL4 mode menu is renamed to Set FIPS-CC mode.
If your firewall is set to FIPS mode, you must change the mode of operation to CCEAL4 mode (using Set CCEAL4 mode menu option in maintenance mode) before you upgrade to a PAN-OS 7.0. or later release. See upgrade considerations for more details on upgrading a firewall that is set to FIPS mode.
Hardware-based and software-based decompression is supported on all Palo Alto Networks platforms (excluding VM-Series firewalls). Starting in PAN-OS 7.1, a hybrid mode (enabled by default) allows firewalls to dynamically switch from hardware-based decompression to software-based decompression when the hardware decompression engine is under a heavy load and then switch back when the load decreases. Prior to PAN-OS 7.1, you could manually switch between decompression modes but you could choose only one mode at a time: hardware (default) or software.
You can modify this new setting ( zip mode auto ) so that the firewall performs only hardware-based decompression or software-based decompression as needed.
PAN-OS 7.0 and earlier releases:
set deviceconfig setting zip sw [yes|no]
PAN-OS 7.1 and later releases:
set deviceconfig setting zip mode [sw | hw | auto]
New counters are also introduced to the show system setting zip command output to monitor the number of times that the firewall switches from hardware-based decompression to software-based decompression:
Number of SW Forced Switchovers —The number of times that the firewall forces a switchover to software-based decompression. A forced switchover can occur when the firewall is in hardware zip mode if the hardware decompression engine becomes unresponsive.Number of SW Automatic Switchovers —The number of times the firewall has dynamically switched from hardware-based to software-based decompression when in automatic zip mode.The following command now shows asterisks (*) instead of zeroes (0) when a corresponding CPU core load percentage is not currently being measured or cannot be measured:
show running resource-monitor
An asterisk may indicate potential issues such as a malfunction that causes packet processing to pause. When issues like this occur, the response repeatedly shows an asterisk instead of a number. It is normal for core 0 to always show an asterisk.
The netstat command has moved from the root level to within the request command hierarchy:PAN-OS 7.0 and earlier releases
netstat programs yes interface yes
PAN-OS 7.1 and later releases
request netstat programs yes interface yes
Additionally, use of the request netstat programs command option now requires superuser or superreader permissions.
The following User-ID configuration commands, used to retrieve the list of groups and the corresponding list of members from an LDAP server, now require you to specify the virtual system to which the LDAP server profile belongs:PAN-OS 7.0 and earlier releases:
show user group-mapping naming-context server | server-port <1-65535> use-ssl is-active-directory proxy-agent | proxy-agent-port <1-65535>
show user group-selection use-ssl base bind-dn bind-password name-attribute group-object container-object filter search-scope proxy-agent | proxy-agent-port <1-65535> force server [ ... ]
show user group-selection use-ssl base bind-dn bind-password name-attribute group-object container-object filter search-scope proxy-agent | proxy-agent-port <1-65535> force server-port [ ... ]
PAN-OS 7.1 and later releases:
show user group-mapping naming-context server | sp_vsys_id server-port <1-65535> use-ssl is-active-directory proxy-agent | proxy-agent-port <1-65535>
show user group-selection sp_vsys_id use-ssl base bind-dn bind-password name-attribute group-object container-object filter search-scope proxy-agent | proxy-agent-port <1-65535> force server [ ... ]
show user group-selection sp_vsys_id use-ssl base bind-dn bind-password name-attribute group-object container-object filter search-scope proxy-agent | proxy-agent-port <1-65535> force server-port [ ... ]
The CLI command to set the maximum number of multicast packets queued per session has changed. The new command updates the configuration instead of running an operational command. This change, which persists even if the firewall is reset, now requires you to commit your configuration changes:PAN-OS 7.0 and earlier releases:
set session max-pending-mcast-pkts-per-session <0-2000>
PAN-OS 7.1 and later releases:
set deviceconfig setting session max-pending-mcast-pkts-per-session <1-2000>
With the role-based access control enhancements, on firewalls enabled for multiple virtual systems, you must specify the target virtual system before you can view or set application status. The following commands have changed:PAN-OS 7.0 and earlier releases:
request get-disabled-applications vsys
request get-application-status vsys application
request set-application-status-recursive vsys enable-dependent-apps application status
PAN-OS 7.1 and later releases:
First set the target vsys.
set system setting target-vsys
Then enter the command to retrieve or set the application status.
request get-disabled-applications
request get-application-status application
request set-application-status-recursive enable-dependent-apps application status
PAN-OS 7.1.3 adds a configuration that allows checking for application and threat content updates hourly. Use the following commands to enable hourly checking and to verify the configuration:PAN-OS 7.1.3 and later releases:
debug management-server content hourly-check set enable
debug management-server content hourly-check show
PAN-OS 7.1.5 adds commands that allow you to restart firewall processes (bfd, cryptod, dhcpd, ikemgr, keymgr, and pppoed) that previously required root access to restart. Use the following commands to restart these processes:PAN-OS 7.1.5 and later releases:
debug software restart process bfd
debug software restart process crypto
debug software restart process dhcp
debug software restart process ikemgr
debug software restart process keymgr
debug software restart process pppoe
PAN-OS 7.1.5 adds a command that allows you to configure how the firewall handles the Priority Code Point (PCP) value in the VLAN tag field when forwarding the frame between different VLANs. By default, the firewall automatically unsets the PCP value when forwarding between VLANs for greater security. To address a requirement in a particular customer environment, you can configure the firewall to pass through the PCP value so that it is preserved on frame forwarding. Use the following command to configure this behavior, where the default value is
no to disable PCP pass-through:PAN-OS 7.1.5 and later releases:set session pass-through-1q-pcp
To view the PCP configuration, use the existing command to display VLANs:
show vlan all
The command output has the following updates associated with the PCP pass-through configuration:
pvst+ tag rewrite: enabled
pvst+ native vlan id: 1
drop stp: disabled
802.1Q PCP pass through: disabled
0 Response to "Palo Alto Firewalls: CLI Commands for Troubleshooting "
Post a Comment