I'm sure you know how to make Linux program output be the input to another program via the pipes that make it possible to, for example, quickly find an isolated file in a very large directory using something like this:

ls | grep missingfilename

When analyzing problems I'll often ask for the output of some program to be stored in a file then sent to me for diagnosis and the familiar form for doing that is like this:

problemprogram > filefordiagnosis

This sends the output of the program named problemprogram to the file named filefordiagnosis, replacing the contents of any existing file with that name. The file is then sent to me by e-mail or web upload and I retrieve it and perform the diagnosis.

Occasionally I see a problem where I am trying to diagnose the cause of a host crash and there is some command where the last line of output before an automated reboot is the crucial one but is never seen. If it is written to screen then it doesn't stay on long enough to be read before the reboot and if it is written to disk then it isn't flushed from disk cache buffers before the reboot and doesn't appear in the file uploaded or sent to me.

I don't have a magical way to guarantee seeing all output that precedes an automated reboot but I do have a cool way of making the host responsible for committing the output to persistent storage a host other than the one that will crash. Imagine you were working on the host at ip address 192.168.1.2 and running ls was going to crash, now imagine how cool it would be if you could do this:

ls | 192.168.1.100:8080

Assuming the hosts are on the same cabling and the one at 192.168.1.100 has a program listening on port 8080 that will output anything it received and assuming the host at 192.168.1.100 will not crash then you can gather the data up to the point of the crash without losing any of it in unflushed cache buffers and it will stay on screen as long as you are looking at the output of the listening program.

Well, a simple program to do this exists and is part of the SUSE Linux distributions, it's called netcat and I think it's very cool. Starting from the host that won't crash, all you want to do is to get netcat to listen for a connection, it will automatically output everything sent to it by a program that connects to it:

netcat -l -p 8080

Then, at the host that will crash you pipe the output of the problem program to netcat giving the IP address and port for the listening instance:

ls | netcat 192.168.1.100 8080

On the 192.168.1.100 host you will see as much ls output as the crashing host can send before it crashes.

For permanent storage of the received information on the listening host just use the normal I/O re-direction semantics in the netcat command line:

netcat -l -p 8080 > /path/to/logfile

Or, do some other manipulation of the displayed or saved output, e.g.:

netcat -l -p 8080 | hexdump -C
netcat -l -p 8080 | hexdump -C > /path/to/hexlogfile

Use these (or with your preferred binary translation tool) for seeing or saving binary data sent over the connection respectively. Actually netcat can do its own hex dump of the data it is outputting by using -o and giving a filename as a parameter, so perhaps this is a better example of pipelining netcat output:

netcat -l -p 8080 | grep something

I only need netcat at one end to examine the traffic from another program

An instance of netcat doesn't require that the program it's exchanging data with be another instance of netcat. For example, if you wanted to capture the data in an initial negotiation step of a http client then you could do this:

netcat -l -p 8080

In another shell on the same host, do this:

wget http://localhost:8080/path/to/file

You won't receive a file and will have to exit the wget client with a break or kill signal but the listening instance of netcat will output what was in the wget programs initial request:

GET /path/to/file HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: localhost:8080
Connection: Keep-Alive

Sending from listener to connector

If you had some case where you needed to see the exchange when the connecting program receives some data (i.e. transfer from 192.168.1.100 to 192.168.1.2 in the examples above) then all you have to do is to run a program that outputs the data or create a file that contains the necessary data and pipe it as the input to the listening netcat:

echo “HTTP:/1.1 404 Not Found\r\n\r\n” | netcat -l -p 8080

That's only the start of a fake web server that has no content, but you'd need a few more lines of http header to convince wget or any other client that netcat was an empty web server. You can actually make netcat appear to be a one-shot web server that responds to GET requests by sending a proper http header and the contents of a file.

Encrypting the transfer

Now let's say you needed to have the data transferred to and from an instance of netcat encrypted. Assuming there's also a ssh server on the host where netcat is listening then encrypting the exchange is as simple as ensuring traffic to the port the listening instance of netcat is using is filtered on public interfaces and then create a simple ssh tunnel from the connecting host to the netcat listen port on the localhost address:

ssh -L 8080:127.0.0.1:8080 user@192.168.1.100

Login to 192.168.1.100 and start another shell on 192.168.1.2 and in that shell on 192.168.1.2 use netcat as before but make the connection point localhost:

ls | netcat localhost 8080

The traffic will go through the ssh tunnel between 192.168.1.2 port 8080 and 192.168.1.100 port 8080.

Quirks

• In all the listen examples I use port 8080 but it can be any port number the user is entitled to create a listening socket on. That often means that for non-root users ports 1 through 1023 are forbidden as are all that are already in-use leaving all others free for use.

• The listening instance of netcat will exit when the connecting instance closes the connection so you might have to use it in a while [ 1 ] loop or a shell script if you want it to be tidy.

• If you want to use UDP instead of TCP for the traffic then you can use the -u option on the netcat command lines.

Almost off-topic...but not quite

Going back to the case above of the host that crashes when you run a program and not being able to see the program's output, I'd like to talk about something more specific. Imagine you do know that the crash is a kernel BUG or similar and the stack backtrace you see suggests a crash in a specific syscall. The instant reaction is probably going to be to get a bug created and assigned to a developer but you could probably do something more to help the developer deal with your problem quickly. If you could say which syscall was being made by the program and what the parameters were it might allow quick selection of the most appropriate developer and let the developer focus on a narrow range of possibilities. That really could save days of developer time and would only require you to spend a few minutes running the problem program in strace and capturing the trace output.

If you remember, the reason for using netcat was to allow the output to be seen up to the point of failure and for it to not be lost because of the failure. Well, a simple case of using strace might be like this:

strace problemprogram

The problem is that the output will be both the output from proplemprogram intermingled with the output of strace so this will capture more than the strace data the developer is interested in:

strace problemprogram | netcat 192.168.1.100 8080

If the system wasn't going to crash you could just use the -o option of strace to save the strace output only to a file:

strace -o logfile problemprogram

The content of logfile will be the strace output generated while problemprogram was running. The file probably won't be complete if there was any cached output not committed to disk when the host crashed but strace has a cool syntax I'd like to point out as a way of getting as much strace output as possible sent to a host that won't crash:

strace -o “|netcat 192.168.1.100 8080” problemprogram

Note the placement of the argument to the -o switch is in quotes to define the scope of the argument even though it has spaces.
Note the use of the | as the first character in the -o argument (! could also be used). This means that the strace output should be piped alone into the command line in the remainder of the argument, i.e. netcat 192.168.1.100 8080.

As shown, the strace output alone would be displayed or saved by any netcat -l -p 8080 command started on the 192.168.1.100 host, the one that won't crash in the examples I've been using.