Sh (from English shell) is a compulsory command interpreter for UNIX-compatible programs in response to the POSIX commonplace. Nonetheless, its capabilities are restricted, so extra feature-rich command interpreters similar to Bash or Ksh are sometimes used as a substitute. Ksh is often utilized in BSD-family working programs, whereas Bash is utilized in Linux-family working programs. Command interpreters simplify fixing small duties associated to working with processes and recordsdata. This text will give attention to Linux working programs, so the dialogue will revolve round Bash.
Python, however, is a full-fledged interpreted programming language, typically used for writing scripts or fixing small utility duties. It’s onerous to think about a contemporary UNIX-like system with out each sh and Python, except it’s a gadget with a minimalist OS like a router. For instance, in Ubuntu Oracular, the python3 bundle can’t be eliminated as a result of the grub-common bundle is determined by it, which in flip is determined by grub2-common and, consequently, grub-pc, the precise working system bootloader. Thus, Python 3 can confidently be used as a substitute for Bash when crucial.
When fixing numerous duties on the OS or file system degree, the query could come up: which language, Bash or Python, is extra advantageous to make use of in a selected case? The reply is determined by the duty at hand. Bash is advantageous when it’s good to rapidly resolve a easy process associated to course of administration, file search, or modification. Nonetheless, because the logic turns into extra complicated, Bash code can develop into cumbersome and tough to learn (though readability primarily is determined by the programmer). After all, you’ll be able to break the code into scripts and capabilities, create sh-libraries, and join them by way of the supply command, however masking them with modular assessments turns into difficult.
Preface
Who’s this text for?
This text is for many who are inquisitive about system administration, are accustomed to one of many two languages, and need to perceive the opposite. Or for many who need to find out about some options of Bash and Python that they may not have recognized earlier than. Fundamental command-line abilities and familiarity with programming fundamentals are required to know the fabric.
For an entire image, together with code readability, the article will evaluate debugging capabilities, syntax, and numerous use circumstances. Comparable examples in each languages can be supplied. In Python code, chances are you’ll often see commas on the finish of enumerations—this isn’t an error. Such a method is taken into account good observe as a result of it avoids marking the final component as modified when including new parts to the enumeration.
The article will contemplate Bash model 3.0 or increased and Python model 3.7 or increased.
Debugging Scripts
Each languages are interpreted, which means that in script execution, the interpreter is aware of rather a lot in regards to the present execution state.
Debugging in Bash
Debugging by way of xtrace
Bash helps the xtrace possibility (-x), which could be set both within the command line when beginning the interpreter or inside the script itself:
#!/bin/bash
# Specify the place to write down logs, open the file for writing:
exec 3>/path/to/log/file
BASH_XTRACEFD=3 # which file descriptor to output debug data to
set -x # allow debugging
# ... code to debug ...
set +x # disable debugging
Such logs will also be written to the systemd journal if implementing a easy service:
#!/bin/bash
# Specify the place to write down logs:
exec 3> >(systemd-cat --priority=debug)
BASH_XTRACEFD=3 # which stream to output debug data to
set -x # allow debugging
# ... code to debug ...
set +x # disable debugging
Debugging in Bash will present which instructions are being executed and with which arguments. If it’s good to get the present values of variables or the code of executed capabilities, you are able to do this with the set command with out arguments. Nonetheless, for the reason that output of the command could be fairly massive, set is extra appropriate for handbook debugging than for occasion logging.
Debugging by way of lure
One other debugging technique is setting handlers for command execution utilizing the lure command on the particular “lure” DEBUG. The instructions being executed could be obtained by way of the built-in variable BASH_COMMAND. Nonetheless, you can not get the return code from this handler as a result of it’s executed earlier than the command itself is named.
lure 'echo "+ ${BASH_COMMAND}"' DEBUG
However it is going to be extra helpful to intercept errors and output the command and line quantity the place the error occurred. To inherit this interception by capabilities, you additionally have to set the functrace possibility:
set -o functrace
lure 'echo "+ line ${LINENO}: ${BASH_COMMAND} -> $?"' ERR
# Check:
ls "${PWD}"
ls unknown_file
Debugging in Python
Debugging by way of pdb
Python has wealthy debugging and logging instruments. For debugging, Python has the pdb module. You’ll be able to run a script with debugging enabled from the console, through which case the debug mode can be activated in distinctive conditions:
python3 -m pdb my_script.py
Instantly within the code, you’ll be able to set breakpoints utilizing the built-in breakpoint() operate.
#!/usr/bin/python3
import os
breakpoint()
# Now you'll be able to attempt, for instance, the supply os command:
# (Pdb) supply os
The language is object-oriented, and all the pieces in it’s an object. You’ll be able to see the strategies out there for an object utilizing the dir() command. For instance, dir(1) will present the strategies out there for the article 1. Instance of calling one in all these strategies: (1).bit_length(). In lots of circumstances, this helps to know arising questions with out the necessity to learn the documentation. In debug mode, you too can use the dir() command to get details about objects and print() to get variable values.
Logging by way of the logging module
Python supplies the logging module, which lets you log debug data with specified logging ranges and log sources. On the whole, logging seems to be one thing like this:
import logging
logging.basicConfig(
filename="myscript.log",
degree = logging.DEBUG, # output DEBUG, INFO, WARNING, ERROR, and CRITICAL ranges
)
logger = logging.getLogger('MyApp')
logger.debug('Some debug data')
logger.error('Some error')
Comparability of Bash and Python Semantics
Variables and Knowledge Varieties
Primitive Knowledge Varieties
In Bash, all variables are strings, however string variables will also be used as numbers. For arithmetic calculations, the syntax $(( expression )) is used.
str_var="some_value" # string, array of characters
int_var=1234 # string "1234", however can be utilized in calculations
int_var=$(( 1 + (int_var - 44) / 111 - 77 )) # string: "-66"
In Python:
str_var="some_value" # class str
int_var = 1234 # class int
int_var = 1 + (int_var - 44) // 111 - 77 # -66, class int
Floating-point numbers aren’t supported in Bash. And that is logical, as a result of if it’s good to use floating-point numbers in command-line scripts, you’re clearly doing one thing on the mistaken degree or within the mistaken programming language. Nonetheless, floating-point numbers are supported in Ksh.
String Formatting
Each Bash and Python assist variable substitution in formatted strings. In Bash, formatted strings are strings enclosed in quotes, whereas in Python, they’re strings with the f prefix.
Each languages additionally assist C-like fashion output of formatted strings. In Bash, this fashion you’ll be able to even format floating-point numbers, though the language itself doesn’t assist them (the decimal separator is set by the locale).
var1='Some string'
var2=0,5
echo "Variable 1: ${var1}, variable 2: ${var2}"
# Variable 1: Some string, variable 2: 0,5
# With out the present locale
LANG=C
printf 'String: %s, quantity: %d, floating-point quantity: %f.n'
'str' '1234' '0.1'
# With the present locale
printf 'String: %s, quantity: %d, floating-point quantity: %f.n'
'str' '1234' '0,1'
# String: str, quantity: 1234, floating-point quantity: 0,100000.
In Python:
var1 = 'Some string'
var2 = 0.5
print(f"Variable 1: {var1}, variable 2: {var2}")
# Variable 1: Some string, variable 2: 0.5
# With out the present locale:
print('String: %s, quantity: %d, floating-point quantity: %f.'
% ('str', 1234, 0.1))
# String: str, quantity: 1234, floating-point quantity: 0.100000.
# With the present locale:
import locale
locale.setlocale('') # apply the present locale
print(locale.format_string('String: %s, quantity: %d, floating-point quantity: %f.',
('str', 1234, 0.1)))
# String: str, quantity: 1234, floating-point quantity: 0,100000.
You’ll be able to discover a distinction concerning the locale—in Python, the print() operate ignores the locale. If it’s good to output values contemplating the locale, you have to use the locale.format_string() operate.
Arrays
In Bash, arrays are primarily textual content separated by areas (by default). The syntax could be very particular; for instance, to repeat an array (by way of @), you have to enclose all its parts in quotes; in any other case, any areas within the parts themselves will trigger the component to be break up into elements. However typically, working with arrays is comparable in easy circumstances:
arr=( 'First merchandise' 'Second merchandise' 'Third merchandise' )
echo "${arr[0]}" "${arr[1]}" "${arr[2]}"
arr_copy="${arr[@]}" # copying the array, quotes are obligatory
arr[0]=1
arr[1]=2
arr[2]=3
echo "${arr[@]}"
echo "${arr_copy[0]}" "${arr_copy[1]}" "${arr_copy[2]}"
In Python:
arr = [ 'First', 'Second', 'Third' ]
print(arr[0], arr[1], arr[2])
arr_copy = arr.copy() # however you too can do it like in Bash: [ *arr ]
arr[0] = 1
arr[1] = 2
arr[2] = 3
print(*arr)
print(arr_copy[0], arr_copy[1], arr_copy[2])
The * operator in Python performs unpacking of lists, dictionaries, iterators, and so on. That’s, the weather of the array are listed as in the event that they had been separated by commas as arguments.
Associative Arrays
Bash additionally helps associative arrays (not like Sh), however the capabilities for working with them are restricted. In Python, associative arrays are known as dictionaries, and the language supplies very wealthy capabilities for working with them.
declare -A assoc_array=(
[name1]='Worth 1'
[name2]='Worth 2'
[name3]='Worth 3'
)
# Assigning a price by key:
assoc_array['name4']='Worth 4' # assigning a price
# Factor-wise entry:
echo "${assoc_array['name1']}"
"${assoc_array['name2']}"
"${assoc_array['name3']}"
"${assoc_array['name4']}"
echo "${!assoc_array[@]}" # output all keys
echo "${assoc_array[@]}" # output all values
# Iterate over all parts
for key in "${!assoc_array[@]}"; do
echo "Key: ${key}"
echo "Worth: ${assoc_array[$key]}"
carried out
In Python:
assoc_array = {
'name1': 'Worth 1',
'name2': 'Worth 2',
'name3': 'Worth 3',
}
# Assigning a price by key:
assoc_array['name4'] = 'Worth 4'
# Factor-wise entry
print(
assoc_array['name1'],
assoc_array['name2'],
assoc_array['name3'],
assoc_array['name4']
)
print(*assoc_array) # output all keys
print(*assoc_array.values()) # output all values
for key, worth in assoc_array.gadgets():
print(f"Key: {key}")
print(f"Worth: {worth}")
Module Importing
In Bash, there are not any modules as such. However you’ll be able to execute a script within the present interpreter utilizing the supply command. Primarily, that is analogous to importing modules, since all capabilities of the included script develop into out there within the present interpreter’s namespace. In Python, there’s full assist for modules with the flexibility to import them. Furthermore, the Python commonplace library accommodates numerous modules for all kinds of use circumstances. Primarily, what’s applied in Bash by third-party command-line utilities could also be out there in Python as commonplace library modules (and if not, you’ll be able to set up extra libraries).
In Bash:
# Embrace the file mylib.sh with some capabilities:
supply mylib.sh
# Let's examine the listing of obtainable capabilities (all of them):
declare -F
In Python:
# Import the module mylib.py or mylib.pyc:
import mylib
# Let's examine the listing of obtainable objects within the mylib module:
print(dir(mylib))
Conditionals and Loops
Conditional Operator
In Bash, circumstances work on two ideas: both a command is supplied as a situation, and its return code is checked, or built-in Bash double sq. or double spherical brackets are used. Within the case of a return code, 0 is true (all the pieces is ok), whereas within the case of double spherical brackets, it is the other—the results of an arithmetic expression is checked, the place 0 is fake.
In Python, the usual strategy for programming languages is used: False, 0, '', [], set(), {}—all of those are equated to False. Non-empty, non-zero values are equated to True.
In Bash:
if [[ "${PWD}" == "${HOME}" ]]; then
echo 'Present listing: ~'
elif [[ "${PWD}" == "${HOME}"* ]]; then
echo "Present listing: ~${PWD#${HOME}}"
else
echo "Present listing: ${PWD}"
fi
if (( UID < 1000 )); then
echo "You might be logged in as a system person. Please log in as your self."
fi
In Python:
import os
curr_dir = os.environ['PWD']
home_dir = os.environ['HOME']
if curr_dir == home_dir:
print('Present listing: ~')
elif curr_dir.startswith(home_dir):
print('Present listing: ~' + curr_dir[len(home_dir):])
else:
print(f"Present listing: {curr_dir}")
if os.environ['UID'] < 1000:
print('You might be logged in as a system person. Please log in as your self.')
Loops
Each languages assist for and whereas loops.
Loop with Factor Iteration
In each languages, the for loop helps component iteration by way of the in operator. In Bash, parts of an array or parts of a string separated by separators recorded within the IFS variable (default: area, tab, and newline) are iterated. In Python, the in operator permits iterating over any iterable objects, similar to lists, units, tuples, and dictionaries, and is safer to work with.
In Bash:
# Recoding textual content recordsdata from CP1251 to UTF-8
for filename in *.txt; do
tmp_file=`mktemp`
iconv -f CP1251 -t UTF-8 "${filename}" -o "${tmp_file}"
mv "${tmp_file}" "${filename}"
carried out
In Python:
import glob
from pathlib import Path
# Recoding textual content recordsdata from CP1251 to UTF-8
for filename in glob.glob('*.txt'):
file = Path(filename)
textual content = file.read_text(encoding='cp1251')
file.write_text(textual content, encoding='utf8')
Loop with Counter
A loop with a counter in Bash seems to be uncommon; the shape for arithmetic calculations is used (((initialization; circumstances; actions after iteration))).
In Bash:
# Get a listing of all regionally registered hosts:
mapfile -t strains < <(grep -P -v '(^s*$|^s*#)' /and so on/hosts)
# Output the listing with numbering:
for ((i = 0; i < "${#strains[@]}"; i += 1)); do
echo "$((i + 1)). ${strains[$i]}"
carried out
In Python:
from pathlib import Path
import re
def is_host_line(s):
return not re.match(r'(^s*$|^s*#)', s)
strains = listing(filter(is_host_line, Path('/and so on/hosts').read_text().splitlines()))
for i in vary(0, len(strains)):
print(f"{i + 1}. {strains[i]}")
Features
As in common languages, Bash helps capabilities. In essence, capabilities in Bash are just like separate scripts—they will additionally settle for arguments like common scripts, they usually return a return code. However, not like Python, they can’t return a outcome aside from a return code. Nonetheless, you’ll be able to return textual content by way of the output stream.
In Bash:
some_function()
{
echo "Script: $0."
echo "Operate: ${FUNCNAME}."
echo "Operate arguments:"
for arg in "$@"; do
echo "${arg}"
carried out
return 0
}
some_function One Two Three 4 5
echo $? # Return code
In Python:
import examine
def some_function_is_ok(*args):
attempt: # If all of the sudden run from the interpreter
script_name = __file__
besides:
script_name=""
print('Script: ' + script_name)
print('Operate: ' + examine.getframeinfo(examine.currentframe()).operate)
print('Operate arguments:')
print(*args, sep='n')
return True
outcome = some_function_is_ok('One', 'Two', 'Three', '4', '5')
print(outcome) # True
Enter, Output, and Error Streams
The enter stream is used to obtain data by a course of, whereas the output stream outputs data. Why streams and never common variables? As a result of in streams, data could be processed because it arrives. Since data from the output stream can endure additional processing, error messages can break this data. Due to this fact, errors are output to a separate error stream. Nonetheless, when operating a command in interactive mode, these streams are combined. Since these are streams, they are often redirected, for instance, to a file. Or vice versa, learn a file into the enter stream. In Bash, the enter stream has the quantity 0, the output stream—1, and the error stream—2. If the stream quantity shouldn’t be specified within the redirection operator to a file, the output stream is redirected.
Writing to a File
Writing to a file in Bash is completed utilizing the > operator, which redirects the command’s output to the desired file. In Python, you’ll be able to write textual content recordsdata utilizing the pathlib module or commonplace means—by opening a file by way of the open() operate. The latter possibility is extra complicated however is well-known to programmers.
In Bash:
# Clear a textual content file by redirecting an empty string to it:
echo -n > some_text_file.txt
# Write a line to a file, overwriting it:
echo 'Line 1' > some_other_text_file.txt
# Append a line to a file:
echo 'Line 2' >> some_other_text_file.txt
In Python:
from pathlib import Path
# Overwrite the file with an empty string (make it empty):
Path('some_text_file.txt').write_text('')
# Overwrite the file with a line:
Path('some_other_text_file.txt').write_text('Line 1')
# Open the file for appending (a):
with open('some_other_text_file.txt', 'a') as fd:
print('Line 2', file=fd)
Writing Multi-line Textual content to a File
For multi-line textual content in Bash, there’s a particular heredoc format (an arbitrary label after <<<, the repetition of which on a brand new line will imply the tip of the textual content), which permits redirecting arbitrary textual content to the command’s enter stream, and from the command, it may be redirected to a file (and right here you’ll be able to’t do with out the exterior cat command). Redirecting file contents to a course of is way easier.
In Bash:
# Redirect multi-line textual content to a file for appending:
cat <<> some_other_text_file.txt
Line 3
Line 4
Line 5
EOF
# Redirect file contents to the cat command:
cat < some_other_text_file.txt
In Python:
# Open the file for appending (w+):
with open('some_other_text_file.txt', 'w+') as fd:
print("""Line 3
Line 4
Line 5""", file=fd)
# Open the file for studying (r):
with open('some_other_text_file.txt', 'r') as fd:
# Output the file contents line by line:
for line in fd:
print(line)
# Or fd.learn(), however then the complete file can be learn into reminiscence.
Studying from a File
In Bash, studying from a file is completed by way of the < signal. In Python, you’ll be able to learn in the usual approach by way of open(), or just by way of Path(...).read_text():
In Bash:
cat < some_other_text_file.txt
In Python:
import pathlib
print(Path('some_other_text_file.txt').read_text())
Stream Redirection
Streams could be redirected not solely to a file or course of but in addition to a different stream.
In Bash:
error()
{
# Redirect the output stream and error stream to the error stream (2).
>&2 echo "$@"
}
error 'An error occurred.'
In Python:
print('An error occurred.', file=sys.stderr)
In easy circumstances, redirection to a file or from a file in Bash seems to be a lot clearer and easier than writing to a file or studying from it in Python. Nonetheless, in complicated circumstances, Bash code can be much less comprehensible and tougher to research.
Executing Exterior Instructions
Working exterior instructions in Python is extra cumbersome than in Bash. Though, in fact, there are easy capabilities subprocess.getoutput() and subprocess.getstatusoutput(), however they lose the benefit of Python by way of passing every particular person argument as a listing component.
Getting Command Output
In the event you merely have to get textual content from a command and you’re certain that it’ll at all times work, you are able to do it as follows:
In Bash:
cmd_path="`which ls`" # backticks execute the command and return its output
echo "${cmd_path}" # output the command path
In Python:
import subprocess
cmd_path = subprocess.getoutput("which ls").rstrip('n')
print(cmd_path) # output the trail to the ls command
However getting command output by way of backticks in Bash can be incorrect if it’s good to get an array of strains. In Python, subprocess.getoutput() accepts a command line, not an array of arguments, which carries some dangers when substituting values. And each choices don’t ignore the return code of the executed command.
Working a utility in Python to get some listing right into a variable will take way more code than in Bash, though the code in Python can be a lot clearer and easier:
In Bash:
mapfile -t root_files < <(ls /) # put the listing of recordsdata from / into root_files
echo "${root_files[@]}" # Output the listing of recordsdata
In Python:
import subprocess
outcome = subprocess.run(
['ls', " # we are sure that such a command exists
capture_output = True, # get the command output
text = True, # interpret input and output as text
)
root_files = result.stdout.splitlines() # get lines from the output
print(*root_files, sep='n') # output one file per line
Getting and Processing Return Codes
With full error handling, it becomes even more complicated, adding checks that complicate the code:
In Bash:
root_files="`ls /some/path`" # Run the command in backticks
if [[ $? != 0 ]]; then
exit $?
fi
echo "${root_files[@]}" # Output the listing of recordsdata
In Python:
import subprocess
import sys
outcome = subprocess.run(
['ls', '/some/path'],
capture_stdout = True, # get the command output
textual content = True, # interpret enter and output as textual content
shell = True, # to get the return code, not an exception, if the command doesn't exist
)
if outcome.returncode != 0:
sys.exit(outcome.returncode)
root_files = outcome.stdout.break up('n') # get strains from the output
del root_files[-1] # the final line can be empty as a result of n on the finish, delete it
print(*root_files, sep='n') # output one file per line
Executing a Command with Solely Getting the Return Code
Executing a command with solely getting the return code is barely easier:
In Bash:
any_command any_arg1 any_arg2
exit_code=$? # get the return code of the earlier command
if [[ $exit_code != 0 ]]; then
exit 1
fi
In Python:
import subprocess
import sys
outcome = subprocess.run(
[
'any_command',
'any_arg1',
'any_arg2',
],
shell = True, # to get the error code of a non-existent course of, not an exception
)
if outcome.returncode != 0:
sys.exit(1)
Exceptions As an alternative of Dealing with Return Codes
However all the pieces turns into even easier if the script exit mode on any error is enabled. In Python, this strategy is utilized by default; errors don’t have to be checked manually; a operate can throw an exception and crash the method.
In Bash:
set -o errexit # crash on command errors
set -o pipefail # the complete pipeline fails if there's an error contained in the pipeline
critical_command any_arg1 any_arg2
In Python:
import subprocess
subprocess.run(
[
'critical_command',
'any_arg1',
'any_arg2',
],
verify = True, # throw an exception on a non-zero return code
)
In some circumstances, exceptions could be caught and dealt with. In Python, that is carried out by way of the attempt operator. In Bash, such catches are carried out by way of the same old if operator.
In Bash:
set -o errexit # crash on command errors
set -o pipefail # the complete pipeline fails if there's an error contained in the pipeline
if any_command any_arg1 any_arg2; then
do_something_else any_arg1 any_arg2
fi
In Python:
import subprocess
attempt:
subprocess.run(
[
'critical_command',
'any_arg1',
'any_arg2',
],
verify = True, # throw an exception on a non-zero return code
)
besides:
subprocess.run(
[
'do_something_else',
'any_arg1',
'any_arg2',
],
verify = True, # throw an exception on a non-zero return code
)
In high-level languages, error dealing with by way of exceptions is most well-liked. The code turns into easier and clearer, which means there’s much less probability of creating a mistake, and code assessment turns into cheaper. Though generally such checks look extra cumbersome than a easy return code verify. Whether or not to make use of this fashion of error dealing with largely is determined by whether or not such exception checks can be frequent or can be in distinctive circumstances.
Constructing Pipelines
In Bash, pipelines are frequent observe, and the language itself has syntax for creating pipelines. Since Python shouldn’t be a command interpreter, it’s carried out a bit extra cumbersome by way of the subprocess module.
In Bash:
ls | grep -v '.txt$' | grep 'construct'
In Python:
import subprocess
p1 = subprocess.Popen(
['ls'],
stdout = subprocess.PIPE, # to cross output to the following command
textual content = True,
)
p2 = subprocess.Popen(
[
'grep',
'-v',
'.txt$'
],
stdin = p1.stdout, # create a pipeline
stdout = subprocess.PIPE, # to cross output to the following command
textual content = True,
)
p3 = subprocess.Popen(
[
'grep',
'build',
],
stdin = p2.stdout, # create a pipeline
stdout = subprocess.PIPE, # already for studying from the present course of
textual content = True,
)
for line in p3.stdout: # learn line by line as knowledge arrives
print(line, finish='') # every line already ends with n
Pipelines with Parallel Knowledge Processing
In Bash, pipelines could be created each between instructions and between instructions and interpreter blocks. For instance, you’ll be able to redirect a pipeline to a line-by-line studying loop. In Python, processing knowledge from a parallel course of can be carried out by easy line-by-line studying from the method’s output stream.
In Bash:
# Get a listing of recordsdata containing some textual content:
discover . -name '*.txt'
| whereas learn line; do # sequentially get file paths
if [[ "${line}" == *'text'* ]]; then # substring in string
echo "${line}"
fi
carried out
In Python:
import subprocess
p = subprocess.Popen(
[
'find',
'.',
'-name',
'*.txt'
],
stdout=subprocess.PIPE,
textual content=True,
)
whereas True:
line = p.stdout.readline().rstrip('n') # there's at all times n on the finish
if not line:
break
if 'textual content' in line: # substring in string
print(line)
Parallel Course of Execution with Ready for Completion
In Bash, operating a course of within the background is supported on the language syntax degree (the & operator), and you’ll run each particular person instructions within the background and elements of the interpreter (for instance, capabilities or loops). However at this degree of complexity, the code will typically be easier and clearer whether it is written in Python, particularly since the usual library supplies capabilities that on the command interpreter degree are applied by third-party utilities that have to be thought of as dependencies.
In Bash:
unalias -a # in case somebody copies straight into the terminal
get_size_by_url()
{
url="$1"
# Get the file dimension from the Content material-Size subject of the response headers to a HEAD request
curl --head --silent --location "${url}"
| whereas learn -r line; do
# Discover the dimensions within the headers utilizing a daily expression
if [[ "${line}" =~ ^Content-Length:[[:space:]]*(.+)[[:space:]]+$ ]]; then
echo -n "${BASH_REMATCH[1]}" ## 1 corresponds to the primary opening bracket
return 0
fi
carried out
}
download_range()
{
url="$1"
begin=$2
finish=$3
output_file="$4"
((curr_size = finish - begin + 1))
curl
--silent
--show-error
--range "${begin}-${finish}"
"${url}"
--output -
| dd
of="${output_file}"
oflag=seek_bytes
search="${begin}"
conv=notrunc
}
download_url()
{
url="$1"
output_file="$2"
((file_size = $(get_size "${url}")))
# Allocate disk area for the file upfront:
fallocate -l "${file_size}" "${output_file}"
range_size=10485760 # 10 MiB
# Divide into elements of as much as 100 MiB:
((ranges_count = (file_size + range_size - 1) / range_size))
declare -a pids ## We are going to save all course of identifiers
for ((i = 0; i < ranges_count; i += 1)); do
((begin = i * range_size))
((finish = (i + 1) * range_size - 1))
if ((finish >= file_size)); then
((finish = file_size - 1))
fi
# Begin downloading within the background:
download_range "${url}" $begin $finish "${output_file}" &
pids[$i]=$! # keep in mind the PID of the background course of
carried out
wait "${pids[@]}" # look ahead to the processes to finish
}
In Python:
import requests
from multiprocessing import Course of
import os
def get_size_by_url(url):
response = requests.head(url)
return int(response.headers['Content-Length'])
def download_range(url, begin, finish, output_file):
req = requests.get(
url,
headers = { 'Vary': 'bytes=" + str(begin) + "-' + str(finish) },
stream = True,
)
req.raise_for_status()
with open(output_file, 'r+b') as fd:
fd.search(begin)
for block in req.iter_content(4096):
fd.write(block)
def download_url(url, output_file):
file_size = get_size_by_url(url)
range_size = 10485760 # 10 MiB
ranges_count = (file_size + range_size - 1) // range_size
with open(output_file, 'wb') as fd:
# Allocate area for the file upfront:
os.posix_fallocate(fd.fileno(), 0, file_size)
processes = []
for i in vary(ranges_count):
begin = i * range_size
finish = begin + range_size - 1
if finish >= file_size:
finish = file_size - 1
# Put together the method and run it within the background:
course of = Course of(
goal = download_range, # this operate will work within the background
args = (url, begin, finish, output_file),
)
course of.begin()
processes.append(course of)
for course of in processes:
course of.be a part of() # look ahead to every course of to finish
Course of Substitution
A separate subject value mentioning is course of substitution in Bash by way of the <(...) assemble, since not everybody is aware of about it, but it surely makes life a lot simpler. Generally it’s good to cross streams of knowledge from different processes to instructions, however the instructions themselves can solely settle for file paths as enter. You may redirect the output of processes to non permanent recordsdata, however such code can be cumbersome. Due to this fact, Bash has assist for course of substitution. Primarily, a digital file is created within the /dev/fd/ area, by way of which data is transmitted by passing the title of this file to the mandatory command as a daily argument.
In Bash:
# Discover frequent processes on two hosts:
comm
<(ssh user1@host1 'ps -x --format cmd' | type)
<(ssh user2@host2 'ps -x --format cmd' | type)
In Python:
from subprocess import check_output
def get_common_lines(lines1, lines2):
i, j = 0, 0
frequent = []
whereas i < len(lines1) and j < len(lines2):
whereas lines2[j] < lines1[i]:
j += 1
if j >= len(lines2):
return frequent
whereas lines2[j] > lines1[i]:
i += 1
if i >= len(lines1):
return frequent
frequent.append(lines1[i])
i += 1
j += 1
return frequent
lines1 = check_output(
['ssh', 'user1@host1', 'ps -x --format cmd'],
textual content = True,
).splitlines()
lines1.type()
lines2 = check_output(
['ssh', 'user2@host2', 'ps -x --format cmd'],
textual content = True,
).splitlines()
lines2.type()
print(*get_common_lines(lines1, lines2), sep='n')
Setting Variables
Working with Setting Variables
Setting variables permit passing data from father or mother processes to little one processes. Bash has built-in assist for surroundings variables on the language degree, however there is no such thing as a associative array of all surroundings variables. Details about them can solely be obtained by way of the exterior env command.
In Bash:
# Assigning a price to an surroundings variable:
export SOME_ENV_VAR='Some worth'
echo "${SOME_ENV_VAR}" # getting the worth
env # output the listing of surroundings variables utilizing an exterior command
In Python:
import os
# Assigning a price to an surroundings variable:
os.environ['SOME_ENV_VAR'] = 'Some worth'
print(os.environ['SOME_ENV_VAR']) # getting the worth
print(os.environ) # output the array of surroundings variables
Setting Values for Particular person Processes
Setting variables are handed from the father or mother course of to little one processes. Generally chances are you’ll want to vary just one surroundings variable. Since Python is positioned as an utility programming language, it is going to be considerably extra difficult to do that in Python, whereas in Bash, assist for such variable setting is built-in:
In Bash:
# Set Russian localization for launched purposes
export LANG='ru_RU.UTF-8'
LANG='C' ls --help # however run this command with English localization
echo "LANG=${LANG}" # be sure the surroundings variables aren't affected
In Python:
import os
import subprocess
# Assigning a price to an surroundings variable:
os.environ['LANG'] = 'ru_RU.UTF-8'
new_env = os.environ.copy()
new_env['LANG'] = 'C'# Assigning a price to an surroundings variable:
export SOME_ENV_VAR='Some worth'
echo "${SOME_ENV_VAR}" # getting the worth
subprocess.run(
['ls', '--help'],
env = new_env,
)
print('LANG=' + os.environ['LANG']) # be sure the surroundings variables aren't affected
Executing Arbitrary Code
Executing arbitrary code shouldn’t be required in on a regular basis conditions, however each languages have this functionality. In Bash, this can be helpful, for instance, to return variables modified by the method or to return named execution outcomes. In Python, there are two operators: eval() and exec(). The Bash eval analog on this case is the exec() operator, because it permits executing a listing of instructions, not simply evaluating expressions. Utilizing eval() and exec() could be very unhealthy observe in Python, and these operators can at all times get replaced with one thing extra appropriate, except it’s good to write your personal command interpreter primarily based on Python.
In Bash:
get_user_info()
{
echo "person=`whoami`"
echo "curr_dir=`pwd`"
}
eval $(get_user_info) # execute the command output
echo "${person}"
echo "${curr_dir}"
In Python:
import getpass
import os
def user_info_code():
return f"""
person="{getpass.getuser()}" # very unhealthy observe
curr_dir="{os.getcwd()}" # please do not do that
"""
exec(user_info_code())
print(person)
print(curr_dir)
# However returning named values typically
# is best by way of courses, namedtuple, or dictionaries
from collections import namedtuple
import getpass
import os
UserInfo=namedtuple('UserInfo', ['user', 'curr_dir'])
def get_user_info():
return UserInfo(getpass.getuser(), os.getcwd())
data = get_user_info()
print(data.person)
print(data.curr_dir)
Working with the File System and Processes
Getting and Altering the Present Listing
Altering the present listing within the command line is often required when doing one thing manually. However getting the present listing could also be wanted in scripts, for instance, if the script or this system being launched does one thing with recordsdata within the present listing. For a similar cause, chances are you’ll want to vary the present listing if it’s good to run one other program that does one thing in it.
In Bash:
current_dir=`pwd` # get the present listing
echo "${current_dir}"
cd /some/path # change to a listing
In Python:
import os
current_dir = os.getcwd() # get the present listing
print(current_dir)
os.chdir('/some/path') # change to a listing
Working with Alerts
In Bash, the kill command is built-in, which is why man kill will show assist for a very totally different command with totally different arguments. By the way in which, sudo kill will already name the kill utility. However Python code remains to be barely clearer.
In Bash:
usr1_handler()
{
echo "Obtained USR1 sign"
}
# Set a handler for the SIGUSR1 sign:
lure 'usr1_handler' USR1
# Ship a sign to the present interpreter:
kill -USR1 $$ # $$ — PID of the father or mother interpreter
Compilation Functionality
Bash by definition doesn’t assist compiling its scripts, which is maybe why all the pieces in it strives for minimalism in names. Python, though interpreted, could be compiled into platform-independent bytecode executed by the Python Digital Machine (PVM). Executing such code can enhance script efficiency. Normally, bytecode recordsdata have the .pyc extension.
Selecting a Language Relying on the Job
As a abstract of the article, the primary postulates could be shaped about which language is best to make use of through which circumstances.
Bash is extra advantageous to make use of in circumstances:
- fixing easy duties that may be solved quicker with good information of the language;
- easy command-line scripts the place work is completed with processes, recordsdata, directories, and even onerous drives and the file system;
- if wrappers are created over different instructions (beginning a command interpreter could be quicker than beginning the Python interpreter);
- if Python shouldn’t be out there within the system for some cause.
Python is extra appropriate for circumstances:
- fixing duties associated to textual content processing, mathematical calculations, or implementing non-trivial algorithms;
- if Bash code can be tough to learn and perceive;
- if it’s good to cowl the code with unit assessments (the
unittestmodule); - if it’s good to parse a big set of command-line parameters with a hierarchy of choices between instructions;
- if it’s good to show graphical dialog containers;
- if script efficiency is important (beginning in Python could also be slower, however executing code could be quicker);
- for creating continuously operating companies (systemd companies).
In case you could have discovered a mistake within the textual content, please ship a message to the writer by choosing the error and urgent Ctrl-Enter.
