World Library  
Flag as Inappropriate
Email this Article

Pipe (Unix)

Article Id: WHEBN0000560915
Reproduction Date:

Title: Pipe (Unix)  
Author: World Heritage Encyclopedia
Language: English
Subject: Gnuplot, BSD Daemon, Find, Rich Rosen
Collection:
Publisher: World Heritage Encyclopedia
Publication
Date:
 

Pipe (Unix)

In Unix-like computer operating systems (and, to some extent, Microsoft Windows), a pipeline is the original software pipeline: a set of processes chained by their standard streams, so that the output of each process (stdout) feeds directly as input (stdin) to the next one. Each connection is implemented by an anonymous pipe. Filter programs are often used in this configuration.

The concept was invented by Douglas McIlroy for Unix shells and it was named by analogy to a physical pipeline.[1] Abstract (illustred) and concrete examples with the shell syntax:

 % program1 | program2 | program3 
 % ls -l | grep key | more

Unix pipeline can be thought of as left associative infix operation whose operands are programs with parameters. Programatically all programs in pipeline run at the same time (in parallel), but, looking at syntax, it can be thought that one runs after another. It is a functional composition. One can be reminded of functional programming, where data is passed from one function to another (as their input or output).

Pipelines in command line interfaces

All widely used Unix and Windows shells have a special syntax construct for the creation of pipelines. In all usage one writes the filter commands in sequence, separated by the ASCII vertical bar character "|" (which, for this reason, is often called "pipe character"). The shell starts the processes and arranges for the necessary connections between their standard streams (including some amount of buffer storage).

Error stream

By default, the standard error streams ("stderr") of the processes in a pipeline are not passed on through the pipe; instead, they are merged and directed to the console. However, many shells have additional syntax for changing this behaviour. In the csh shell, for instance, using "|&" instead of "|" signifies that the standard error stream too should be merged with the standard output and fed to the next process. The Bourne Shell can also merge standard error, using 2>&1, as well as redirect it to a different file.

Pipemill

In the most commonly used simple pipelines the shell connects a series of sub-processes via pipes, and executes external commands within each sub-process. Thus the shell itself is doing no direct processing of the data flowing through the pipeline.

However, it's possible for the shell to perform processing directly, using a "pipemill" (since a while command is used to "mill" over the results from the initial command). This construct generally looks something like:

command | while read var1 var2 ...; do
   # process each line, using variables as parsed into $var1, $var2, etc
   # (note that this is a subshell: var1, var2 etc will not be available
   # after the while loop terminates)
   done

When using programs such as ssh, stdin is being passed to the remote command. As such the default standard input handling of ssh drains the remaining hosts from the while loop.[2] To prevent this, stdin might be redirected from /dev/null (by < /dev/null). In the case of ssh, -n might be used to prevent ssh reading from stdin.

Creating pipelines programmatically

Pipelines can be created under program control. The Unix pipe() system call asks the operating system to construct a new anonymous pipe object. This results in two new, opened file descriptors in the process: the read-only end of the pipe, and the write-only end. The pipe ends appear to be normal, anonymous file descriptors, except that they have no ability to seek.

To avoid deadlock and exploit parallelism, the Unix process with one or more new pipes will then, generally, call fork() to create new processes. Each process will then close the end(s) of the pipe that it will not be using before producing or consuming any data. Alternatively, a process might create a new thread and use the pipe to communicate between them.

Named pipes may also be created using mkfifo() or mknod() and then presented as the input or output file to programs as they are invoked. They allow multi-path pipes to be created, and are especially effective when combined with standard error redirection, or with tee.

Implementation

In most Unix-like systems, all processes of a pipeline are started at the same time, with their streams appropriately connected, and managed by the bfr is available to provide larger buffers if required.

Network pipes

Tools like netcat and socat can connect pipes to TCP/IP sockets.

History

The pipeline concept and the vertical-bar notation were invented by Douglas McIlroy, one of the authors of the early command shells, after he noticed that much of the time they were processing the output of one program as the input to another. His ideas were implemented in 1973 when Ken Thompson added pipes to the UNIX operating system.[3] The idea was eventually ported to other operating systems, such as DOS, OS/2, Microsoft Windows, and BeOS, often with the same notation.

Although developed independently, Unix pipes are similar to, and were preceded by, the 'communication files' developed by Ken Lochner [4] in the 1960s for the Dartmouth Time Sharing System.[5]

The robot in the icon for Apple's Automator, which also uses a pipeline concept to chain repetitive commands together, holds a pipe in homage to the original Unix concept.

Other operating systems

Main article: Pipeline (software)

This feature of Unix was borrowed by other operating systems, such as Taos and MS-DOS, and eventually became the pipes and filters design pattern of software engineering.

See also

References

External links

  • History of Unix pipe notation
    • Doug McIlroy’s original 1964 memo, proposing the concept of a pipe for the first time
  • The Open Group
  • Pipes: A Brief Introduction by The Linux Information Project (LINFO)
  • Unix Pipes – powerful and elegant programming paradigm (Softpanorama)
  • Ad Hoc Data Analysis From The Unix Command Line at – Shows how to use pipelines composed of simple filters to do complex data analysis.
  • Use And Abuse Of Pipes With Audio Data – Gives an introduction to using and abusing pipes with netcat, nettee and fifos to play audio across a network.
  • stackoverflow.com – A Q&A about bash pipeline handling.sv:Vertikalstreck#Datavetenskap
This article was sourced from Creative Commons Attribution-ShareAlike License; additional terms may apply. World Heritage Encyclopedia content is assembled from numerous content providers, Open Access Publishing, and in compliance with The Fair Access to Science and Technology Research Act (FASTR), Wikimedia Foundation, Inc., Public Library of Science, The Encyclopedia of Life, Open Book Publishers (OBP), PubMed, U.S. National Library of Medicine, National Center for Biotechnology Information, U.S. National Library of Medicine, National Institutes of Health (NIH), U.S. Department of Health & Human Services, and USA.gov, which sources content from all federal, state, local, tribal, and territorial government publication portals (.gov, .mil, .edu). Funding for USA.gov and content contributors is made possible from the U.S. Congress, E-Government Act of 2002.
 
Crowd sourced content that is contributed to World Heritage Encyclopedia is peer reviewed and edited by our editorial staff to ensure quality scholarly research articles.
 
By using this site, you agree to the Terms of Use and Privacy Policy. World Heritage Encyclopedia™ is a registered trademark of the World Public Library Association, a non-profit organization.
 


Copyright © World Library Foundation. All rights reserved. eBooks from Project Gutenberg are sponsored by the World Library Foundation,
a 501c(4) Member's Support Non-Profit Organization, and is NOT affiliated with any governmental agency or department.