SSH login delay|slow

SSH Delay Login

Many times we observe there is some delay in SSH login and the delay might be from 2 seconds to 2 minutes. There are a few troubleshooting steps we need to perform on the SSHD server to identify the root cause. Refer to the solution section for resolution.

Topic

  • How to debug sshd delay login?
  • SSH login slow
  • Too much delay in ssh login
  • Troubleshoot SSH login
  • How to troubleshoot SSHD daemon?

apt

  • Linux
  • Centos
  • RHEL
  • Ubuntu
  • Debian
  • Suse
  • Arch Linux

Solution


Mainly there are 6 possibilities for the delay in ssh login which we’ll check them step by step.

  1. SSHD server performs DNS check for each login
  2. Syslog/rsyslog service issue on the SSHD server
  3. systemd-logind timeout
  4. PAM configuration issue
  5. Debug SSHD daemon
  6. AD/LDAP delay login

Troubleshoot – 1 SSHD DNS Check

  • It is a default behavior of sshd application to perform DNS(Forward and reverse) check for every connecting client.
  • If the recurssive DNS server configured in /etc/resolv.conf file doesn’t respond to the DNS queries sent by SSHD daemon or if the DNS server takes a few minutes or seconds of time to respond to the SSHD daemon, we’ll expect delay in ssh login.

  • In the following example, ssh client machine 192.168.200.1 connects to the SSHD server 192.168.200.11 and the total delay is 2 minutes 1.8 seconds.
        $ time ssh root@192.168.200.11
        root@192.168.200.11's password: 
        [root@centos-testsrv ~]# exit
        logout
        Connection to 192.168.200.11 closed.

        real    2m1.851s    <<<<<<<<<<<<<<<<< Here
        user    0m0.008s
        sys 0m0.006s

  • Following network trace using tcpdump command shows that SSHD server attempts to perform DNS forward and reverse query by 192.168.100.4 DNS server for the connecting client 192.168.200.1. After spending two minutes of time there is no response from the DNS server 192.168.100.4.

  • Here the reason is DNS server is down or the DNS server can’t be reached from the SSHD server due to the network firewall blocks the DNS TCP/UDP port 53 from SSHD server to DNS server.

  • If the DNS server is down start the DNS server and if it is a network issue, allow the DNS ports in the firewall to allow DNS communication between SSHD server and DNS server.

# tcpdump -i any port 53 -nnn
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
12:44:46.455515 IP 192.168.200.11.58667 > 192.168.100.4.53: 32767+ PTR? 1.200.168.192.in-addr.arpa. (44)  <<<<< start time
12:44:51.461678 IP 192.168.200.11.58667 > 192.168.100.4.53: 32767+ PTR? 1.200.168.192.in-addr.arpa. (44)
12:44:56.474254 IP 192.168.200.11.34274 > 192.168.100.4.53: 54645+ A? gateway. (25)   
12:45:01.479909 IP 192.168.200.11.34274 > 192.168.100.4.53: 54645+ A? gateway. (25)    
12:45:22.248081 IP 192.168.200.11.32861 > 192.168.100.4.53: 30714+ A? gateway. (25)
12:45:27.255240 IP 192.168.200.11.32861 > 192.168.100.4.53: 30714+ A? gateway. (25)
12:45:32.263670 IP 192.168.200.11.34624 > 192.168.100.4.53: 53176+ A? gateway. (25)
12:45:37.270370 IP 192.168.200.11.34624 > 192.168.100.4.53: 53176+ A? gateway. (25)
12:45:42.279373 IP 192.168.200.11.49958 > 192.168.100.4.53: 28525+ A? gateway. (25)
12:45:47.285643 IP 192.168.200.11.49958 > 192.168.100.4.53: 28525+ A? gateway. (25)
12:45:52.412127 IP 192.168.200.11.52656 > 192.168.100.4.53: 27014+ A? gateway. (25)
12:45:57.417799 IP 192.168.200.11.52656 > 192.168.100.4.53: 27014+ A? gateway. (25)
12:46:02.444674 IP 192.168.200.11.39194 > 192.168.100.4.53: 13244+ A? gateway. (25)
12:46:07.450301 IP 192.168.200.11.39194 > 192.168.100.4.53: 13244+ A? gateway. (25)
12:46:12.503904 IP 192.168.200.11.53855 > 192.168.100.4.53: 62443+ A? gateway. (25)
12:46:17.510057 IP 192.168.200.11.53855 > 192.168.100.4.53: 62443+ A? gateway. (25)
12:46:22.517226 IP 192.168.200.11.33567 > 192.168.100.4.53: 13082+ A? gateway. (25)
12:46:27.522890 IP 192.168.200.11.33567 > 192.168.100.4.53: 13082+ A? gateway. (25)
12:46:32.525808 IP 192.168.200.11.43315 > 192.168.100.4.53: 16378+ A? gateway. (25)
12:46:37.531994 IP 192.168.200.11.43315 > 192.168.100.4.53: 16378+ A? gateway. (25)   <<<<< end time

  • Another reason could be DNS server is up and running but the DNS server takes time to respond to the DNS query sent by the SSHD server. If that is the case, you will clearly see that in the TCPDUMP analysis.

Resolution

  • To resolve DNS issue, either resolve the DNS issue on the DNS server which should quickly respond to the DNS query sent from SSHD server or add the following configuration line to /etc/ssh/sshd_config configuration file and then restart sshd service.
# add the following line to /etc/ssh/sshd_config configuration file. 
UseDNS no

# Then restart sshd service 
# service sshd restart 
or
# systemctl restart sshd

Troubleshoot – 2 Syslog/Rsyslog daemon issue

  • When a user logins to the SSHD server, SSHD application generates authpriv syslog events.

  • Rsyslog daemon on the system is configured to forward system events to the remote syslog server over tcp 514 or any other tcp port.

  • If the remote tcp syslog server has some issue, that delays accepting logs from the SSHD server or outging socket buffer is full on the SSHD server due to the remote syslog server doesn’t accept logs quickly, we’ll expect a slight delay in login.

  • You can use the following tcpdump command on the SSHD server to track the issue.

------------------------------------------
SSHD server: 192.168.200.11
Remote Syslog Server: 192.168.100.55
------------------------------------------

# tcpdump -i any port 514 -nnn 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
14:25:55.194190 IP 192.168.200.11.58119 > 192.168.100.55.514: SYSLOG authpriv.info, length: 108
14:25:55.316950 IP 192.168.200.11.58119 > 192.168.100.55.514: SYSLOG daemon.info, length: 76
14:25:55.322582 IP 192.168.200.11.58119 > 192.168.100.55.514: SYSLOG authpriv.info, length: 110
14:25:55.322664 IP 192.168.200.11.58119 > 192.168.100.55.514: SYSLOG auth.info, length: 79

  • In the following command, observe Send-Q field and if you see any bigger value other than zero (0), that confirms syslog server has some issue.
# netstat -ntaup|grep -e "Send-Q" -e "514"  ## Replace 514 with the remote syslog server port 
Proto Recv-Q   Send-Q         Local Address              Foreign Address                State                   PID/Program name    
tcp        0          100005673  192.168.200.1:51284    192.168.100.55:514       ESTABLISHED        5033/rsyslog

Resolution

  • If syslog issue is found, login to the syslog server and fix the issue.
  • Restarting rsyslog service on the SSHD server will provide temporary relief.

Troubleshoot – 3 systemd-logind timeout

  • If systemd-logind program has any issue that delays ssh login.
  • Check /var/log/messages and /var/log/secure files, search for systemd-logind timeout errors.

Resolution

  • If this is the case, upgrading systemd application to the latest version would resolve the issue.
  • If after upgrading systemd program to the latest version, the issue doesn’t get resolved, report the issue to the OS vendor.

Troubleshoot – 4 incorrect PAM configuration

  • If there are incorrect PAM configuration found in /etc/pam.d directory, that will create some some issue with ssh login.
  • Check each PAM configuration file in /etc/pam.d directory, mostly you need to see the following PAM configuration files those are related to system login.
    • /etc/pam.d/password-auth
    • /etc/pam.d/system-auth
    • /etc/pam.d/sshd

Troubleshoot – 5 DEBUG SSHD program

  • If all above 4 troubleshooting methods don’t find the RCA and resolution, the final thing is to debug SSHD daemon by adding the "LogLevel DEBUG3" configuration in /etc/ssh/sshd_config file and then restart sshd service.

  • On the client system try login attempt to the SSHD server and on the SSHD server system check /var/log/secure file to monitor SSHD debug events.

  • On the client system you can execute the following command to see ssh verbose login.

$ ssh -vvv user@server_ip_or_name

Troubleshoot – 6 AD/LDAP delay login

  • If the sshd server is configured for AD/LDAP authentication and the remote LDAP or Active Directory server has an issue providing remote login, we’ll expect some delay or login won’t happen.

  • If this is the case, troubleshoot the login issue on both SSHD and LDAP/AD server.


Related articles

If you have enjoyed the above article, refer to following related articles:

You May Also Like

avatar

About the Author: TekFik

TekFik is a technical blogging site helps techies and engineers to solve their day to day issues and also allows everyone to share knowledge and feedback. Please feel free to contact us at tekfik.rd@gmail.com if there is anything.

Leave a Reply

Your email address will not be published. Required fields are marked *