Troubleshooting

Checking your service

An excellent tool was created to upload a test pod to your host, this tool is aimed to test and troubleshoot a codius host. Do not setup scripted runs of the tool for constant monitoring or any other reasons. You can find it on https://upload.host1.codius.live/ and just enter your hostname and wait (it might take a while to start, especially the first time you use it.

If the upload tool says its fine, then its fine. If its not … continue on reading.

Restarting everything

A lot of logs, files, mounts are not cleaned up correctly … next to the fact your service might be stuck. If you tested your server with the upload tool and its failing to upload the pod, the first step would be restarting everything. To cleanup everything we made a nice script you can use:

#!/bin/bash
systemctl daemon-reload
systemctl stop codiusd.service
systemctl stop moneyd-xrp
systemctl stop hyperd
systemctl stop nginx
for mount in `cat /proc/mounts | grep /run/hyper | awk '{ print $2 }'`; do umount $mount; done
for mount in `cat /proc/mounts | grep /var/lib/hyper/hosts | awk '{ print $2 }'`; do umount $mount; done
rm -rf /var/lib/hyper/hosts/*
rm -rf /var/lib/hyper/containers/*
rm -rf /run/hyper/*
rm -rf /var/log/hyper/qemu/*.log
systemctl start nginx
systemctl start hyperd
systemctl start moneyd-xrp
systemctl start codiusd.service
502 Bad Gateway

If you try to access https://codius.example.com and are getting this error, it means that your nginx is unable to connect to your codiusd. Check for the error in /var/log/nginx/error.log.

connect() to 127.0.0.1:3000 failed (13: Permission denied) while connecting to upstream

If you see a13: Permission denied like shown above, the likely culprit is SElinux. Run the following command:

setsebool -P httpd_can_network_connect 1

Another common issue is that Codiusd is not listening:

connect() failed (111: Connection refused) while connecting to upstream

In this case, check which processes are listening with netstat -tulpn:

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      33155/nginx: master
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2093/sshd
tcp        0      0 127.0.0.1:3000          0.0.0.0:*               LISTEN      47079/node
tcp        0      0 169.254.77.68:7768      0.0.0.0:*               LISTEN      47079/node
tcp        0      0 127.0.0.1:7768          0.0.0.0:*               LISTEN      19651/node
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2017/master
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      33155/nginx: master
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd
tcp6       0      0 :::80                   :::*                    LISTEN      33155/nginx: master
tcp6       0      0 :::22                   :::*                    LISTEN      2093/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      2017/master
udp        0      0 0.0.0.0:111             0.0.0.0:*                           20975/rpcbind
udp        0      0 127.0.0.1:323           0.0.0.0:*                           1344/chronyd
udp        0      0 0.0.0.0:798             0.0.0.0:*                           20975/rpcbind
udp6       0      0 :::111                  :::*                                20975/rpcbind
udp6       0      0 ::1:323                 :::*                                1344/chronyd
udp6       0      0 :::798                  :::*                                20975/rpcbind

If you don’t see 127.0.0.1:7768, that means that Moneyd is not listening. Go to the “Moneyd not listening” section below.

If you see 127.0.0.1:3000, but you don’t see 127.0.0.1:3000, that means that Moneyd is listening but Codiusd is not. Go to the “Codiusd not listening” below.

If you see 127.0.0.1:3000 but are still getting 111: Connection refused, then it is likely a firewall issue. Make sure you run these commands to properly configure your firewall:

firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-cmd --zone=public --add-port=7768/tcp --permanent
firewall-cmd --reload
Moneyd not listening

Check if your Moneyd is running:

systemctl status moneyd-xrp

If it is running, try stopping it and running it in debug mode:

systemctl stop moneyd-xrp
DEBUG=* moneyd xrp:start

If you see errors and need help understanding what’s going on, feel free to ask for help on Gitter. The most common issue is insufficient funds in your XRP wallet. You need 16 XRP in addition to the reserve. The reserve depends on the account. For a brand-new account, it’s 20 XRP but it increases as you use different XRP Ledger features. To be sure, you can send 16 XRP more to the wallet, restart Moneyd and see if that fixes the problem.

Once you’ve got Moneyd working on the command line, stop it (Ctrl+C) and run it as a service again:

systemctl start moneyd-xrp
Codiusd not listening

In this case, check if Codiusd is running:

systemctl status codiusd

You can try restarting it:

systemctl restart codiusd

Or check the logs for any useful information:

journalctl -t codiusd -n 200

If you see errors and need help understanding what’s going on, feel free to ask for help on Gitter.