My favorite Linux distribution:
Useful resources:
GNU isn’t really an OS. It’s more of a set of rules or philosophies that govern free software, that at the same time gave birth to a bunch of tools while trying to create an OS. So GNU tools are basically open versions of tools that already existed, but were reimplemented to conform to principals of open software. GNU/Linux is a mesh of those tools and the Linux kernel to form a complete OS, but there are other GNUs, e.g. GNU/Hurd.
Unix and BSD are “older” implementations of POSIX that are various levels of “closed source”. Unix is usually totally closed source, but there are as many flavors of Unix as there are Linux (if not more). BSD is not usually considered “open”, but it was considered to be very open when it was released. Its licensing also allowed for commercial use with far fewer restrictions than the more “open” licenses of the time allowed.
Linux is the newest of the four. Strictly speaking, it’s “just a kernel”; however, in general, it’s thought of as a full OS when combined with GNU Tools and several other core components.
The main governing differences between these are their ideals. Unix, Linux, and BSD have different ideals that they implement. They are all POSIX, and are all basically interchangeable. They do solve some of the same problems in different ways. So other then ideals and how they choose to implement POSIX standards, there is little difference.
For more info I suggest your read a brief article on the creation of GNU, OSS, Linux, BSD, and UNIX. They will be slanted towards their individual ideas, but those articles should give you a better idea of the differences.
Useful resources:
CLI is an acronym for Command Line Interface or Command Language Interpreter. The command line is one of the most powerful ways to control your system/computer.
In Unix like systems, CLI is the interface by which a user can type commands for the system to execute. The CLI is very powerful, but is not very error-tolerant.
The CLI allows you to do manipulations with your system’s internals and with code in a much more fine-tuned way. It offers greater flexibility and control than a GUI regardless of what OS is used. Many programs that you might want to use in your software that are hosted on say Github also require running some commands on the CLI in order to get them running.
My favorite tools
screen - free terminal multiplexer, I can start a session and My terminals will be saved even when you connection is lost, so you can resume later or from homessh - the most valuable over-all command to learn, I can use it to do some amazing things:
sshfsrsync server with no rsync deamon by starting one itself via sshvi/vim - is the most popular and powerful text editor, it’s universal, it’s work very fast, even on large filesbash-completion - contains a number of predefined completion rules for shellTips & Hacks
CTRL + Rpopd/pushd and other shell builtins which allow you manipulate the directory stackCTRL + U, CTRL + E!* - all arguments of last command!! - the whole of last command!ssh - last command starting with sshUseful resources:
BASH is my favorite. It’s really a preferential kind of thing, where I love the syntax and it just “clicks” for me. The input/output redirection syntax (>>, << 2>&1, 2>, 1>, etc) is similar to C++ which makes it easier for me to recognize.
I also like the ZSH shell, because is much more customizable than BASH. It has the Oh-My-Zsh framework, powerful context based tab completion, pattern matching/globbing on steroids, loadable modules and more.
Useful resources:
man [commandname] can be used to see a description of a command (ex.: man less, man cat)
-h or --help some programs will implement printing instructions when passed this parameter (ex.: python -h and python --help)
w - a lot of great information in there with the server uptimetop - you can see all running processes, then order them by CPU, memory utilization and morenetstat - to know on what port and IP your server is listening on and what processes are using thosedf - reports the amount of available disk space being used by file systemshistory - tell you what was previously run by the user you are currently connected toUseful resources:
ls -al output mean?In the order of output:
-rwxrw-r-- 1 root root 2048 Jan 13 07:11 db.dump
File permissions is displayed as following:
- or l or d, d indicates a directory, a - represents a file, l is a symlink (or soft link) - special type of filer = readablew = writablex = executableIn your example -rwxrw-r--, this means the line displayed is:
-)rwx)rw-)r--)Useful resources:
For a summary of logged-in users, including each login of a username, the terminal users are attached to, the date/time they logged in, and possibly the computer from which they are making the connection, enter:
# It uses /var/run/utmp and /var/log/wtmp files to get the details.
who
For extensive information, including username, terminal, IP number of the source computer, the time the login began, any idle time, process CPU cycles, job CPU cycles, and the currently running command, enter:
# It uses /var/run/utmp, and their processes /proc.
w
Also important for displays a list of last logged in users, enter:
# It uses /var/log/wtmp.
last
Useful resources:
The most significant advantage of executing the running process in the background is that you can do any other task simultaneously while other processes are running in the background. So, more processes can be completed in the background while you are working on different processes. It can be achieved by adding a special character & at the end of the command.
Generally applications that take too long to execute and doesn’t require user interaction are sent to background so that we can continue our work in terminal.
For example if you want to download something in background, you can:
wget https://url-to-download.com/download.tar.gz &
When you run the above command you get the following output:
[1] 2203
Here 1 is the serial number of job and 2203 is PID of the job.
You can see the jobs running in background using the following command:
jobs
When you execute job in background it give you a PID of job, you can kill the job running in background using the following command:
kill PID
Replace the PID with the PID of the job. If you have only one job running you can bring it to foreground using:
fg
If you have multiple jobs running in background you can bring any job in foreground using:
fg %#
Replace the # with serial number of the job.
Useful resources:
To be completed.
Running (everything) as root is bad because:
Stupidity: nothing prevents you from making a careless mistake. If you try to change the system in any potentially harmful way, you need to use sudo, which ensures a pause (while you’re entering the password) to ensure that you aren’t about to make a mistake.
Security: harder to hack if you don’t know the admin user’s login account. root means you already have one half of the working set of admin credentials.
You don’t really need it: if you need to run several commands as root, and you’re annoyed by having to enter your password several times when sudo has expired, all you need to do is sudo -i and you are now root. Want to run some commands using pipes? Then use sudo sh -c "command1 | command2".
You can always use it in the recovery console: the recovery console allows you to recover from a major mistake, or fix a problem caused by an app (which you still had to run as sudo). Ubuntu doesn’t have a password for the root account in this case, but you can search online for changing that - this will make it harder for anyone that has physical access to your box to be able to do harm.
Useful resources:
You’d use top/htop for both. Using free and vmstat command we can display the physical and virtual memory statistics respectively. With the help of sar command we see the CPU utilization & other stats (but sar isn’t even installed in most systems).
Useful resources:
Linux load averages are system load averages that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing. Most tools show three averages, for 1, 5, and 15 minutes.
These 3 numbers are not the numbers for the different CPUs. These numbers are mean values of the load number for a given period of time (of the last 1, 5 and 15 minutes).
Load average is usually described as “average length of run queue”. So few CPU-consuming processes or threads can raise load average above 1. There is no problem if load average is less than total number of CPU cores. But if it gets higher than number of CPUs, this means some threads/processes will stay in queue, ready to run, but waiting for free CPU.
It is meant to give you an idea of the state of the system, averaged over several periods of time. Since it is averaged, it takes time for it to go back to 0 after a heavy load was placed on the system.
Some interpretations:
Useful resources:
The passwords are not stored anywhere on the system at all. What is stored in /etc/shadow are so called hashes of the passwords.
A hash of some text is created by performing a so called one way function on the text (password), thus creating a string to check against. By design it is “impossible” (computationally infeasible) to reverse that process.
Older Unix variants stored the encrypted passwords in /etc/passwd along with other information about each account.
Newer ones simply have a * in the relevant field in /etc/passwd and use /etc/shadow to store the password, in part to ensure nobody gets read access to the passwords when they only need the other stuff (shadow is usually protected more strongly than passwd).
For more info consult man crypt, man shadow, man passwd.
Useful resources:
To change all the directories e.g. to 755 (drwxr-xr-x):
find /opt/data -type d -exec chmod 755 {} \;
To change all the files e.g. to 644 (-rw-r--r--):
find /opt/data -type f -exec chmod 644 {} \;
Useful resources:
command not found. How to trace the source of the error and resolve it?It looks that at one point or another are overwriting the default PATH environment variable. The type of errors you have, indicates that PATH does not contain e.g. /bin, where the commands (including bash) reside.
One way to begin debugging your bash script or command would be to start a subshell with the -x option:
bash --login -x
This will show you every command, and its arguments, which is executed when starting that shell.
Also very helpful is show PATH variable values:
echo $PATH
If you run this:
PATH=/bin:/sbin:/usr/bin:/usr/sbin
most commands should start working - and then you can edit ~/.bash_profile instead of ~/.bashrc and fix whatever is resetting PATH there. Default PATH variable values for root and other users is in /etc/profile file.
Useful resource:
CTRL + C but your script still running. How do you stop it? ***To be completed.
Useful resources:
grep command? How to match multiple strings in the same line?The grep utilities are a family of Unix tools, including egrep and fgrep.
grep searches file patterns. If you are looking for a specific pattern in the output of another command, grep highlights the relevant lines. Use this grep command for searching log files, specific processes, and more.
For match multiple strings:
grep -E "string1|string2" filename
or
grep -e "string1" -e "string2" filename
Useful resources:
head: to check the starting of a file.tail: to check the ending of the file. It is the reverse of head command.cat: used to view, create, concatenate the files.more: used to display the text in the terminal window in pager form.less: used to view the text in the backward direction and also provides single line movement.Useful resources:
Ctrl+C, but on some systems, the “delete” character or “break” key can be used.Useful resources:
kill command do?In Unix and Unix-like operating systems, kill is a command used to send a signal to a process. By default, the message sent is the termination signal, which requests that the process exit. But kill is something of a misnomer; the signal sent may have nothing to do with process killing.
Useful resources:
rm and rm -rf?rm only deletes the named files (and not directories). With -rf as you say:
-r, -R, --recursive recursively deletes content of a directory, including hidden files and sub directories-f, --force ignore nonexistent files, never promptUseful resources:
grep recursively? Explain on several examples. ***To be completed.
archive.tgz has ~30 GB. How do you list content of it and extract only one file?# list of content
tar tf archive.tgz
# extract file
tar xf archive.tgz filename
Useful resources:
If you want to execute each command only if the previous one succeeded, then combine them using the && operator:
cd /my_folder && rm *.jar && svn co path to repo && mvn compile package install
If one of the commands fails, then all other commands following it won’t be executed.
If you want to execute all commands regardless of whether the previous ones failed or not, separate them with semicolons:
cd /my_folder; rm *.jar; svn co path to repo; mvn compile package install
In your case, I think you want the first case where execution of the next command depends on the success of the previous one.
You can also put all commands in a script and execute that instead:
#! /bin/sh
cd /my_folder \
&& rm *.jar \
&& svn co path to repo \
&& mvn compile package install
Useful resources:
chmod to give all users execute access to a file without affecting other permissions?chmod a+x /path/to/file
a - for all usersx - for execution permissionr - for read permissionw - for write permissionUseful resources:
To sync the contents of dir1 to dir2 on the same system, type:
rsync -av --progress --delete dir1/ dir2
-a, --archive - archive mode--delete - delete extraneous files from dest dirs-v, --verbose - verbose mode (increase verbosity)--progress - show progress during transferUseful resources:
cp filename{,.orig})cp, rsync or tar)git (or any other version control) to keep track of configuration files (e.g. etckeeper for /etc directory)Useful resources:
find / -type f -size +20M
Useful resources:
sudo su - and not just sudo su?sudo is in most modern Linux distributions where (but not always) the root user is disabled and has no password set. Therefore you cannot switch to the root user with su (you can try). You have to call sudo with root privileges: sudo su.
su just switches the user, providing a normal shell with an environment nearly the same as with the old user.
su - invokes a login shell after switching the user. A login shell resets most environment variables, providing a clean base.
Useful resources:
find / -mmin -60 -type f
Useful resources:
They are essential to investigate issues on the system. Log management is absolutely critical for IT security.
Servers, firewalls, and other IT equipment keep log files that record important events and transactions. This information can provide important clues about hostile activity affecting your network from within and without. Log data can also provide information for identifying and troubleshooting equipment problems including configuration problems and hardware failure.
It’s your server’s record of who’s come to your site, when, and exactly what they looked at. It’s incredibly detailed, showing:
Factors to consider:
By collecting and analyzing logs, you can understand what transpires within your network. Each log file contains many pieces of information that can be invaluable, especially if you know how to read them and analyze them.
Useful resources:
An incremental backup is a type of backup that only copies files that have changed since the previous backup.
Useful resources:
A RAID (Redundant Array of Inexpensive Disks) is a technology that is used to increase the performance and/or reliability of data storage.
Useful resources:
useradd -m -g initial_group username
-g/--gid: defines the group name or number of the user’s initial login group. If specified, the group name must exist; if a group number is provided, it must refer to an already existing group.
If not specified, the behaviour of useradd will depend on the USERGROUPS_ENAB variable contained in /etc/login.defs. The default behaviour (USERGROUPS_ENAB yes) is to create a group with the same name as the username, with GID equal to UID.
Useful resources:
To be completed.
Useful resources:
The chmod command is used to change permissions of files and directories. It supports two modes of operation: symbolic mode and octal mode.
Syntax: chmod [options] [permissions] file(s)
Examples:
chmod u+rw file.txtchmod g-x script.shchmod o+rx programchmod u=rw,go=r file.txtSyntax: chmod [options] [mode] file(s)
Examples:
chmod 755 script.shchmod 700 private.txtchmod 777 public_dirThe chown command is used to change the ownership of files and directories.
Syntax: chown [options] owner:group file(s)
Examples:
chown john:users file.txtchown -R alice:staff project_dirThe chgrp command is used to change the group ownership of files and directories.
Syntax: chgrp [options] group file(s)
Examples:
chgrp developers script.shchgrp -R team project_dirThe folder structure in Ubuntu Linux follows the Filesystem Hierarchy Standard (FHS), which is a standard for organizing the files and directories on a Unix-like operating system. Here is an overview of the main directories you will typically find in Ubuntu:
/bin: Contains essential command-line executable files (binaries) that are available to all users.
/boot: Contains files related to the boot process, including the Linux kernel, initial ramdisk (initrd), and bootloader configuration.
/dev: Contains device files that represent and allow access to various hardware devices on the system.
/etc: Contains system-wide configuration files for various applications and services.
/home: The home directories for individual users. Each user typically has a subdirectory here to store their personal files and settings.
/lib and /lib64: These directories contain shared libraries needed by the system and applications. The “lib64” directory is present on 64-bit systems.
/media: Mount point for removable media devices such as USB drives or optical discs.
/mnt: A general-purpose mount point for temporarily mounting filesystems.
/opt: Contains optional software packages installed on the system. Applications installed here are often self-contained in their own directories.
/proc: A virtual filesystem that provides information about processes and system status. It is used by many system utilities to obtain runtime information.
/root: The home directory for the root user, the administrative superuser.
/run: A temporary filesystem that contains runtime data for various system services. It is cleared on each reboot.
/sbin: Contains system binaries (executable files) that are primarily used by the root user for system administration tasks.
/srv: Contains data for services provided by the system.
/sys: A virtual filesystem that exposes kernel-related information and configuration.
/tmp: A directory for temporary files created by applications and users. Its contents are typically cleared on each reboot.
/usr: Contains user-related programs, libraries, documentation, and shared resources. It has subdirectories such as /usr/bin for user binaries, /usr/lib for libraries, and /usr/share for shared data.
/var: Contains variable data that changes during the system’s operation, such as logs, databases, and spool files.
This is a high-level overview of the Ubuntu Linux folder structure. Each directory serves a specific purpose in organizing the system’s files and resources.