It is assumed here you already have a DETER account and password, created following these instructions. Please log in to DETER and "Begin an Experiment," naming it "tunnels-xxx" (where xxx is some suffix like your initials likely to be unique to avoid name conflict with other students' experiments) and loading in this network specification (.ns) file. When you upload the file, click the checkbox "Do Not Swap In." The file produces this internetwork:

Examine the file, compare it with the diagram, and note the correspondence. This is a 4-subnet internetwork. (Each adjacent pair comprises one of the subnets. Nodes 1, 2, and 3 serve as routers that join subnet pairs.)

Swap your new experiment in.

To interact with your nodes you need to gain an interface to each. You gain a terminal window interface to any of the nodes by "double ssh," connecting to users.isi.deterlab.net and from there to the node. For example, to connect to node1:

ssh  <your account name>@users.isi.deterlab.net
ssh  node1.tunnels.USCCSci530.isi.deterlab.net

(This documentation is written as if the experiment name is "tunnels". The real name of your experiment being the variation tunnels-xxx, you should substitute the real name for your particular experiment wherever you see this document using "tunnels".)

Set up your monitor screen to contain 5 terminal windows, one to each of the experiment nodes arranged as in the graphic below. To do so first use ssh to get 5 terminal window connections to users.isi.deterlab.net, then from them again to the 5 nodes respectively as above. Once you are logged into a node, become root there, by executing (5 times, in the different terminal windows):

sudo su -   (be sure to include the hyphen)

Your screen will look something like the following, with the windows stacked in the same order as in the above figure, and the title bars accessibly arranged for single-click selection of the window of any desired node.

Working servers for experimentation

To use tunnels we need a tunnel destination, a server to which to send data. First as a reference case we can interact with that server normally, untunneled. Then we can run the same interaction through the various tunnel alternatives we will create, optionally obstructing the untunneled data path to verify the tunnel's functionality.

Let's use a couple of servers, and let's choose node0 as their placement location. We'll interact with them from node4 at the opposite end of the internetwork. node0 will make available a udp echo server running on udp port 7, a tcp echo server running on tcp port 7, and the tcp web server apache/httpd running on tcp port 80. These servers are in place. Verify their presence. On node0:

netstat -pantu | grep -E "xinetd|httpd"

You should see:

[root@node0 dbm]# netstat -pantu | grep -E "xinetd|httpd"
tcp        0      0 :::7                        :::*                        LISTEN      3283/xinetd
tcp        0      0 :::80                       :::*                        LISTEN      3310/httpd
udp        0      0 :::7                        :::*                                    3283/xinetd

What about suitable clients for these servers? A suitable node4 echo client to run against the echo servers is netcat (nc). A suitable web client to run against apache is lynx. Let's use these clients to test the servers. On node4:

echo "The cow jumped over the moon." | nc -u node0 7    (talks to node0's udp echo server)
echo "The cow jumped over the moon." | nc -t node0 7    (talks to node0's tcp echo server)

You must terminate the first command with a ctrl-C keystroke. (The convenience of using "node0" here instead of "10.2.2.2" works because DETER maps them together for you in each local /etc/hosts file. Use the IPs themselves if you prefer but the names are more recognizable.) "The cow jumped over the moon." will appear on your screen, having been bounced back to you (echoed) by the server at the other end. Let's sniff this en route, at node2. On node2:

nicaddressing    (identify which interface is 100.1.1.254's and substitute it below for "ethX")
tcpdump  -xXnnti  ethX

Now go back to node4 and repeat the above client "echo..." commands. When they run, note the activity on node2's screen showing the passage of traffic back and forth. Scrutinize it till you locate within it on node2's screen the "...cow..." phrase. It's fully legible, not encrypted.

Similarly let's test the web server. It will serve a default web page named index.html if there is one. There isn't, so let's create one and make it distinguishing. On node0:

echo '<h1>Hi, you have reached node0.</h1>'   >   /var/www/html/index.html

Test it by browsing this page from node4 using the lynx character-mode web browser. (Browsers have two halves-- the server interaction half, and the local display half. Lynx is no different from any other browser in terms of server interaction, which we care about; it lacks the local-display half of familiar GUI browsers, but we don't care about that. Minimalist, non-GUI display is good enough for our network diagnostic purposes.) On node4 browse node0 (10.2.2.2):

lynx http://node0:80

Make sure the "Hi, you have reached node0." from node0's default web page reaches your screen. (lynx tutorial: q followed by y will quit; ctrl-r will refresh.)

These interactions are working normally, the old fashioned way. That is, by using standard routing. DETER places routes (in the machines' routing tables) to enable all our nodes up and down the line to reach each other. In particular there are no tunnels involved.

We're going to build some. We'll test them by running these interactions through them, instead of through standard routing as just seen. So we will want to defeat the operation of standard routing for these purposes and the tunnels' acid test. node2 in the middle is a good place to put an obstructive firewall rule in the iptables FORWARDing chain. We could make a rule that blocks the traffic by its port numbers or by its source and/or destination IP addresses. Let's do the latter. On node2:

iptables -A FORWARD -s node4 -d node0 -j DROP

It says, "Disallow IP from node4 to node0." That's sufficient. Go back to node4 and repeat the two client echo|nc commands and the one lynx command. Now, no data comes back. The original data never reach node0 because it can't get through node2. If you wanted to remove the obstruction-- but for purposes of our exercise please leave it in place-- you would run the above iptables commands with -A replaced by -D. (Examining/verifying the firewall ruleset in effect on node2 can be done with "iptables -nL". It will show the obstruction, referencing the nodes in IP terms as "10.1.1.2" and "10.2.2.2".)

IP "over" (or "in") IP takes advantage of the fact that what an IP packet carries is data, data has many varieties and, among them, IP packets are themselves data. So there is nothing to stop an IP packet from carrying another IP packet. That's what IP-over-IP does. It is formalized as a standard protocol, assigned protocol number 4. An IP packet that's carrying another IP packet as its data declares so by bearing the number 4 in the "protocol" field of its header.

We will put IP-over-IP tunnel endpoints on nodes 1 and 3. That is, we'll create new tunnel interfaces on them (with the "ifconfig" command). Then with IP routing, which corresponds destinations with interfaces, we will tell node1 that the remote 10.1.1.0/24 network is reachable as a destination through its tunnel interface (with the "route" command). Similarly we'll tell node 3 that the remote 10.2.2.0/24 network is reachable through its tunnel interface. For each of the 2 nodes a script that does this is provided.

In your node1 terminal window, obtain and execute its script (examine it briefly if you wish):

cd
cp /proj/USCCSci530/exp/tunl-ipip-node1 .  (observe the final dot, shorthand for "current directory," an important part of this syntax)
chmod +x tunl-ipip-node1
./tunl-ipip-node1   (when it displays a routing table press enter to see a second one; they are "before" and "after" tunnel construction)

and similarly in your node3 terminal window:

cd
cp /proj/USCCSci530/exp/tunl-ipip-node3 .
chmod +x tunl-ipip-node3
./tunl-ipip-node3   (when it displays a routing table press enter to see a second one; they are "before" and "after" tunnel construction)

The tunnels are now in place. View them. On both node1 and node3:

ifconfig  tunl0

Go back to node4 and repeat the two client echo|nc commands and the one lynx command. It's working again, despite the node2 firewall. That's because the data comes to node2 inside the tunnel, to which its firewall does not apply. Try this too, from node4:

ping  -c 1  node0

To see explicit evidence of the tunnel let's use tcpdump to sniff  some of the interfaces the data must traverse to travel over-and-back. There are 8 of them (look at the diagram, where each address belongs to an interface). We'll selectively look at several of them. As an example, in your node3 terminal window start the tcpdump command on the interior interface, the one addressed as 10.1.1.1. But first you'll have to figure out which one that is, in terms of eth1, eth2, eth3... nomenclature. DETER nodes generally have 5 ethernet interfaces, and it may vary which one gains a particular IP address used by an experiment. It might be eth2 today, but when you repeat the experiment tomorrow maybe eth5. You can find out which interface has IP 10.1.1.1 in the node3 terminal window with this script:

nicaddressing

You can do this for any of the 8 interfaces, in its machine's terminal window. Sniff the 10.1.1.1 interface on node3. Once you know which interface it is (I'll represent it here as "ethX"), in the node3 terminal window, initiate tcpdump:

nicaddressing    (identify which interface is 10.1.1.1's and substitute it below for "ethX")
tcpdump  -nnti  ethX not stp   ("not stp" suppresses display of the active and distracting spanning tree protocol)

Now, back in the node4 terminal window again ping:

ping  -c 1  node0

A packet passes through, addressed between the node4 and node0 ping endpoints. Another passes back, addressed in reverse. And, importantly, they are both ping (ICMP echo) packets (i.e., IP packets that carry ping packets). The dump looks like this:

IP 10.1.1.2 > 10.2.2.2: ICMP echo request, id 44314, seq 1, length 64
IP 10.2.2.2 > 10.1.1.2: ICMP echo reply, id 44314, seq 1, length 64

Now do the same thing but watch the data passage at node3's exterior interface. That's the one with IP 100.1.1.1. Find out which it is as above. I'll represent it as "ethY". Then in the node3 terminal window initiate tcpdump:

tcpdump  -nnti  ethY not stp

and back in the node4 terminal window again ping:

ping  -c 1  node4

A packet passes through, and another passes back reverse-addressed. But they are addressed between nodes 3 and 1, the tunnel endpoints; not between nodes 4 and 0, the ping endpoints. And importantly, they are not ping  packets. The dump looks like this:

IP 100.1.1.1 > 200.2.2.1: IP 10.1.1.2 > 10.2.2.1: ICMP echo request, id 38682, seq 1, length 64 (ipip-proto-4)
IP 200.2.2.1 > 100.1.1.1: IP 10.2.2.1 > 10.1.1.2: ICMP echo reply, id 38682, seq 1, length 64 (ipip-proto-4)

In terms of carried data, rather than ping packets they are IP packets; that is, instead of IP packets that carry ping packets, we have IP packets that carry other IP packets (the essence of IP-over-IP). And the addressing of the carried ones, differing from the carrying ones, is between ping endpoints not tunnel endpoints.

Repeat this for yourself on a few of the other interfaces. Note that north of node3 and south of node1 all is normal ping. But at any of the in-between interfaces the traffic is tunnel traffic, IP-over-IP. If you add a -v to your tcpdump command to make it verbose, it will reveal the IP header's protocol field, showing on the interior interface something like:

IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 10.1.1.2 > 10.2.2.2: ICMP echo request, id 47130, seq 1, length 64

and on the exterior interface:

IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto: IPIP (4), length: 104) 100.1.1.1 > 200.2.2.1: IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto: ICMP (1), length: 84) 10.1.1.2 > 10.2.2.2: ICMP echo request, id 47642, seq 1, length 64

Note the "proto: ICMP (1)" in the interior case, and the "proto: IPIP (4)" in the exterior. Also in the exterior case notice that the carried data its represented as the IP packet it is, complete with its own "proto: ICMP (1)".

Let's neutralize the tunl0 interfaces. On node1 and nod3:

ifconfig  tunl0  down
ifconfig  tunl0  0.0.0.0

The most common, baseline use of ssh is to establish a character-mode login session on another machine, much like telnet. The machine where you wish to log in must run an ssh server, and you must have an account there. The advantage of ssh is that your client and the remote server automatically encrypt and decrypt everything they send to each other. What if you're interested in communication not with the ssh server, but another computer to which it is attached in a local network? ssh has a secondary feature that accomplishes this, called ssh port forwarding.

Let's restate this in terms of our experiment setting. If you are at node4 and have an account on node1 then you can gain a login shell on node1 provided it runs an ssh server and you run an ssh client. Everything going back and forth will be encrypted. But what if you're interested in communication not with node1, but node0 to which it is attached? ssh port forwarding can help. You're not limited just to logging in to node0, but can communicate with any service it runs with the help of ssh to get your packets through. (This could encompass login in to node0 if node0 itself runs an ssh server, but can instead be browsing node0 should it run a  web server or sending files to node0 should it run an ftp server, or using any service node0 offers.) The communication traffic between node4 and node0, whatever it is, travels in encrypted form as far as node1; it is unencrypted there and goes the rest of the way unencrypted.

Establish a forwarding tunnel to the target ports 7 and 80 on the target machine node0

First, from node4 use ssh to log in to node1. But supply the extra syntax for the port forwarding feature. On node4:

ssh   -fN    -L 1007:node0:7    -L 1080:node0:80    <your account name>@node1
   (local ports 1007 and 1080 become surrogates for remote 7 and 80)

When prompted for login, give your user username and password (the same as you use for DETER iteself). You become logged in to node1. However the -fN options provide a little trick, putting this logged-in ssh client session into the background. Consequently you don't get node1's shell prompt as usual, but revert back immediately to your original shell on node4. Nevertheless, your login to node1 is sustained and the port forwarding for those two ports is in place. (If you want to see it, the "ps ax" command will show the ssh process.) Check that ssh on your machine is indeed listening to ports 1007 and 1080. On node4:

netstat -pant | grep -E "1007|1080"

Connect to the target port of target machine with a client that matches the service running on that port

Test that both echo and http traffic gets though. On node4:

lynx  127.0.0.1:1080
 and
echo hellooooooooooooooooo  |   nc  -t  127.0.0.1  1007

Note that node4 is talking to itself (127.0.0.1)! Yet the the conversation is carried to node0 and that's where the responses come from. If you care to sniff the echo interaction, you'll see that the data is encrypted (you can't see hellooooooooooooooooo) north of node1 but visible between there and node0.

We're done, but don't forget that ssh client is running in the background. Let's clean it up. On node4:

killall  ssh

In the standard routed scenario above, the lynx client on node4 connects and talks directly to the apache server on node0, port 80. We want to reorganize this a bit by making the following changes:

1) disconnect node4 lynx from node0 apache, have him talk to node3 stunnel instead.
2) place stunnel on node3 and node1 and have them talk to each other.
3) disconnect node0 apache from node4 lynx, have him hear from node1 stunnel instead.

Configure stunnel on node 3 as a client and node1 as a server, for which the config files are provided. First on node3:

cp /proj/USCCSci530/exp/stunnel-client.conf  /etc/stunnel/

Then on node1:

cp /proj/USCCSci530/exp/stunnel-server.conf  /etc/stunnel/

Here are those two configuration files you just copied:

We want to preserve the association of apache with port 80. By contrast, we don't care what ports the other 2 conversations utilize. So let node4 lynx reach node3 stunnel using node3's port 2000, let node3 stunnel reach node1 stunnel using node1's port 30000, and let node1 stunnel reach node0 apache using node0's port 80. That's what these files configure. The choices of ports 2000 and 30000 are arbitrary.

The server copy of stunnel needs a certificate for this to work, and we can create one. Do the following on the stunnel server machine, node1:

cd /etc/stunnel
openssl req -new -x509 -days 3650 -nodes -out stunnel.pem -keyout stunnel.pem      (accept all the defaults)
chmod 600 stunnel.pem

Finally, run the 2 copies of stunnel giving their respective config files on the command lines. On node3:

stunnel  /etc/stunnel/stunnel-client.conf

And on node1:

stunnel  /etc/stunnel/stunnel-server.conf

Nothing happened. But stunnels are running on both boxes, willing to listen to the browser and talk to the server (respectively). Check for stunnel listening on the node3's port 2000 and on node1's 30000 by running this command on both:

netstat -pant | grep stunnel

It lists the various ports being listened to and picks out the one(s) with stunnel as listener. You should see something like:

[root@node3 dbm]# netstat -pant | grep stunnel
tcp        0      0 0.0.0.0:2000                0.0.0.0:*                   LISTEN      13467/stunnel

and:

[root@node1 stunnel]# netstat -pant | grep stunnel
tcp        0      0 0.0.0.0:30000               0.0.0.0:*                   LISTEN      13365/stunnel

Now use your stunnels. On node4 bring up the lynx web browser again. But this time, point it to node3 and also specify the port to talk to. That port is 2000. On node4:

lynx  http://node3:2000

You should see the server's default web page appear. Now, however, the traffic is passing between the pair of stunnels imposed between browser and server. And they are encrypting it. Watch tcpdump on node2 while doing this and note the targeting of node1's port 30000 as specified in the stunnels' configurations. Verify the conversation's dependency on stunnel by killing one of the stunnels. On node3:

killall  stunnel

then on node4 try to refresh the browser. It won't. It's cut off from stunnel so the traffic route is disrupted.

We will run 3 OpenVPN scenarios.

 - a routed tunnel, unencrypted
 - a routed tunnel, encrypted using static preshared keys
 - a bridged tunnel, encrypted using SSL

The tunnel endpoints will be node4 and node1 (see the network diagram). node4 plays the role of the road warrior in a hotel, while node1 plays that of the gateway in an office network, that network being 10.2.2.0/24 in our diagram. Warrior node4 wants access to machines in the office, such as node0.

OpenVPN is installed on node4 and node1. Producing the 3 scenarios is a matter of putting the corresponding configuration files on these nodes. OpenVPN uses directory /etc/openvpn to keep its config files. I've prepared appropriate files for you to put there. Do so, separately on each of the 2 nodes, as follows:

On node4:
cd  /etc/openvpn
tar  -xvf  /proj/USCCSci530/etcopenvpn-node4.tar

On node1:
cd  /etc/openvpn
tar  -xvf  /proj/USCCSci530/etcopenvpn-node1.tar

Scenario 1: routed tunnel, unencrypted

On node1:
openvpn  /etc/openvpn/server-unencrypted.conf  &

Then on node4:
openvpn  /etc/openvpn/client-unencrypted.conf  &

Let's look at the resulting screenshots and try to interpret in light of the config files. The screens will look something like this:

(At this point press enter on both to see a fresh command prompt; OpenVPN continues to run in the background leaving you free to issue other commands.) The config files you invoked, responsible for this, are:

In these files
 line 1 - each copy of OpenVPN is pointed to the machine where the other one is
 line 2 - they are told to construct new local interfaces, both named tun0, to become tunnel endpoints
 line 3 - they are told what addresses to give their interface, and what address the other will have (note the specs are reciprocal)
 line 4 - the client is told to give its routing table a route to the network behind the server, gatewayed through the server's tunnel endpoint address

Read the screen messages in this light. Note the chosen tunnel endpoint addresses (arbitrarily 10.20.30...) are outside any existing subnet. The endpoint machines can still refer to each other with their original addresses but can now also do so by these new ones. Traffic to the new one will be encapsulated through a tunnel that uses UDP to carry its encapsulated traffic. Note also the warning in the screen messages that this tunnel doesn't encrypt what it passes.

Investigate some of this. On both machines:

ifconfig

Note the presence of the two tun0 interfaces. Note their type and addresses. They are point-to-point (good only to reach a single machine-- the other). Next, on both machines:

route  -n

Note that there is a new route on the server, to the client (10.20.30.2), through the tun0 interface. And that there are two new routes on the client. A reciprocal one to the server (10.20.30.1) through the tun0 interface. Plus another to the remote network (10.2.2.0/24), through the server as gateway (owing to line4 in the config file).

Now let's send 2 pings from client to server, one addressed to the server's physical interface address and the other to its new tunnel interface address. While doing it, sniff the passing traffic at node2. On node2:

tcpdump  -nnti  ethX  icmp  or  udp

Then on node4: 

ping  -c1  -p48656c6c6f 200.2.2.1; ping  -c1  -p48656c6c6f  10.20.30.1

This compound command sends two pings, one to each address and carrying the word "Hello" (in ASCII, 48656c6c6f).

Look at both pings on node2. A ping is a paired ICMP "echo request" and answering "echo reply". In the node2 dump do you see such a pair? That's a ping. node4 pinged twice, so do you see two such pairs? You see 2 exchanges, but only one of them is ICMP echo. That's the ping to 200.2.2.1. But what about the ping to 10.20.30.1, do you see it? Do you even see "10.20.30.1" anywhere? It is more revealing to ask tcpdump to show the entire content of the traffic. So modify the node2 tcpdump command slightly and repeat. On node2:

tcpdump  -xXnnti  ethX  icmp  or  udp

Then on node4 again: 

ping  -c1  -s30  -p48656c6c6f  200.2.2.1; ping  -c1  -s30  -p48656c6c6f  10.20.30.1

node2 looks something like this:

Of these two packet-pair exchanges, which one's packets are fatter? What do they have that the skinny ones don't? Are they encrypted? Are they tunneled?

What if node4 wants to ping through node1 all the way in to node0? Let's do it and see which kind of exchange is chosen for that. On node2:

tcpdump  -xXnnti  ethX  icmp  or  udp

Then on node4 again: 

ping  -c1  -p48656c6c6f  10.2.2.2

Does this use the skinny- or fat-packet exchange method? ICMP echo or UDP? tunneled or not? Encrypted? The dump you're looking at on node2 tells you.

We ran OpenVPN in the background so to stop it we must manually kill it. On both node1 and node4:

killall  openvpn

Scenario 2:  routed tunnel, encrypted using static preshared keys

We will do much the same thing now with the addition of a static key, a copy of which each node shares. Below we now create it, distribute it, modify the config files to use it, and re-run OpenVPN to do so.

On node1:

cd  /etc/openvpn
openvpn  --genkey  --secret  static.key
scp   static.key   <your account name>@node4:/tmp     (give your password when prompted)

On node4:

mv /tmp/static.key /etc/openvpn/

Run OpenVPN on the endpoint nodes. On node1:
openvpn  /etc/openvpn/server-statickey.conf  &

Then on node4:
openvpn  /etc/openvpn/client-statickey.conf  &

The config files you invoked are just a little different:

We added the last line to each file telling it to use the key, that is, to encrypt. The key files on the two nodes are identical.

Note that the "encryption features disabled" warning message doesn't appear on screen this time.

On node2:    

tcpdump  -xXnnti  ethX  icmp  or  udp

Then on node4, press enter to gain a shell prompt, and again: 

ping  -c1  -p48656c6c6f  10.2.2.2

What's different this time? Does this use the skinny- or fat-packet exchange method? ICMP echo or UDP? tunneled or not? Encrypted? The dump you're looking at on node2 tells you.

On both node1 and node4:

killall  openvpn

Scenario 3:  bridged tunnel, encrypted using SSL

We will do it yet again, this time changing the encryption method. We will also use bridging instead of routing, meaning that an IP address from the office network 10.2.2.0/24 will be extended and applied to warrior node4. As if a cable from the office switch were extended cross-country to him. That is, as if he were sitting in the office.

OpenVPN won't do the bridging, but will use it. For bridging there are several commands (e.g., brctl), installed on node1 already, that do the job. We will run them, from a script named bridge-start, before we run OpenVPN.

Instead of a shared, static key we will place key and certificate files generated by OpenSSL. Creating and placing them is a process sufficiently complex that OpenVPN provides scripts to streamline it, in a directory named "easy-rsa" (if interested see /usr/share/openvpn/easy-rsa/2.0/). For this exercise I pre-ran the scripts, produced the files, and put them in the tar file you used. In effect you distributed them when you unpacked the tar file, above. They are already in place. (Distributing this way defeats security, but here the purpose is tutorial.)

The new config files are substantially different:

The biggest differences are 1) the references to the cryptographic files in the "ca", "cert", "key", and "dh" directives, 2)  the use of "dev tap0" instead of "dev tun0" as name of the virtual interface device to be constructed, and 3) the "server-bridge 10.2.2.0 255.255.255.0 10.2.2.50 10.2.2.59" directive on the server side. The crypto directives make OpenVPN use SSL. tap devices as opposed to tun devices encapsulate data-link layer ethernet as opposed to network layer IP. That is, they bridge traffic through instead of routing it through. And the server-bridge directive engages OpenVPN with the bridge produced by brctl and allocates addresses to clients out of the given subrange (10.2.2.50-59) of the given network (10.2.2.0/255.255.255.0). In practice, to maintain address uniqueness, a system administrator would avoid allocating the subrange IPs to office computers and leave them for dynamic allocation to connecting warriors.

Run OpenVPN on the endpoint nodes with this new configuration:

First on node1:
/etc/openvpn/bridge-start
openvpn /etc/openvpn/server-bridged.conf &

Then on node4:
openvpn /etc/openvpn/client-bridged.conf &

Let's see what we have done. On node4 (press enter to regain shell prompt):

ifconfig

Note the presence of the tap0 interface. Note its address and type. The address, probably 10.2.2.50, comes from the range in the "server-bridge" directive on node1. It is of the regular shared-medium subnet type just like a regular ethernet NIC would be (as opposed to point-to-point) bearing an address from the same office subnet as office machines. It's on the same footing as office machines. It is to an office machine like any other office machine. It might as well actually be in the office instead of the hotel. Look at its routing table. On node4:

route  -n

There is a route like this:

telling node4 that its tap0 interface is the avenue to the 10.2.2.0/255.255.255.0 network. Note it is not gatewayed. Putting a frame out the tap0 interface is putting it at the doorstep of the computers in the 10.2.2.0 network, with no intermediating help from any stepping-stone router. This is standard intra-LAN, local delivery. Warrior node4 is  LAN-attached, effectively.

We can demonstrate this by showing that ethernet broadcast traffic from node4 reaches node1. To generate ethernet broadcast let's use arp. It emits its discover requests in the form of ethernet broadcast frames, addressed to the broadcast address FF:FF:FF:FF:FF:FF. Trying to ping a machine that's considered local will generate an arp broadcast looking for it. First on node0:

tcpdump  -ennti  ethX  arp

tcpdump's "e" option will reveal ethernet frame headers. Then on node4:

ping  -c1  10.2.2.3

10.2.2.3 doesn't exist so won't reply, but node4's appeals seeking it, which are frame broadcasts, will be seen at node0. Surprise number 1 is that node4 evidently thinks node0 is local, and surprise number 2 is that this thinking is not misplaced because the broadcasts get where they need to go.

You should see something like this. It's well-known that ethernet can't natively cross a router. Yet here it is:

This is not IP. Routing routes IP only, not ethernet. There is no IP in these frames so they did not get routed here, rather their source node4 is "here" in the first place. Moreover, between node4 and the gateway node1 the traffic was encrypted.