Disable docker from mocking with iptables:

echo "{
\"iptables\": false
}" > /etc/docker/daemon.json

(did not get correct config in /etc/default/docker)

allow forwarding to the container:

cp -p /etc/default/ufw /etc/default/ufw.bak
sed -i -e 's/DEFAULT_FORWARD_POLICY="DROP"/DEFAULT_FORWARD_POLICY="ACCEPT"/g' /etc/default/ufw

systemctl restart docker
systemctl restart ufw

allow container to go out to internet.
exact ip from ifconfig docker0
(haven't done this yet, but from inside the container was able to go out.  docker-compose port maps to specific ip , but that should not matter for outbound traffic 
the other thing is that the host, bofh, had a rule that allow full access (meant for cueball-bofh traffic))


iptables -t nat -A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE

File System Isolation

1. File system inside container is isolated from the host
2. No Disk I/O throtle, no disk quota.
3. Union FS: DeviceMapper in Fedora. AUFS in Ubuntu.

Docker vs LXC

1. LXC is lightweight, but heavily embeded into Gentoo Linux, not portable.
2. Docker started in Ubuntu. application portability was key goal. Gentoo was providing a container for security/isolation.
3. Both rely on Cgroup, union file system.

docker run

Stock httpd conainer

# run a stock container (apache httpd) on amazon linux
# no changes to the content of the container

docker run -p 80:80   -i -t --rm --name ApacheA -v ~/htdocs4docker:/usr/local/apache2/htdocs/ httpd 
docker run -p 8000:80            --name ApacheB -v          "$PWD":/usr/local/apache2/htdocs/ httpd &
	# run a process (httpd)  in a new container
	# each process has its own FS, network, isolated process tree.
	# most of the default param are defined in the IMAGE, but cli arg will overwrite the IMAGE conf.
	# 
	# -p = map host port 8000 to port 80 on the container 
	# 	if not specified, default maps NOTHING, 
	# 	so outside, even on the host, can't get into the inside of the docker app!
	# -i = interactive, def=false.         ^P ^Q ^Z bg can suspend and background this.  cannot use with &
	# -t = allocate pseudo-tty, def=false
	# --name foo = assign a name to the container.  if not specified, a random 2_words name will be assigned
	# --rm = remove container when exiting, def=false.  
	# 	if don't delete container, it linger around.  
	# 	will see them with docker ps -a.  remove with docker rm ContainerName
	# 	Note that each time a container is started, the name must not match existing or stopped container.
	#	--rm is good especially for testing, to avoid having to do lot of clean up work.
	# -v "$PWD":/usr/local/apache2/htdocs/  bind mounts the host's current dir into the container's apache htdocs dir.
	# -v or --volume=... format is host : container


docker run -it httpd  /bin/bash
	# this run (a new) apache httpd container, and start the bash shell running *inside* the container env.
	# /usr/local/apache2/htdocs inside this container is where the web pages are served up from (if not mapped with -v)
	# changes to this will persist inside this instance, till container is removed with docker rm ContainerName 


docker exec -it ApacheB /bin/bash	
	# this run the bash command in an already running container called ApacheB
	# -it will make it interactive with tty, thus leaving one with a shell inside the container
	# ^D or exit will terminate this exec.  the container will continue to run. it is like logout.

docker exec ApacheB ps -ef
	# will run "ps -ef" inside the container, print output, and terminate the exec
	# think of "ssh remotehost ps -ef" to run command w/o interactive shell in a remote environment

# docker exec depends on a running container
# to start a container with a specific command is done by specifying a different entrypoint. eg
docker run --entrypoint "/bin/ls" ubuntu
# but there seems to be some kernel issue, -v does not work
docker run -v $HOME:/myhome --entrypoint "/bin/ls /myhome" ubuntu
# fails miserably (on Zorin 12/Xenial or CentOS 7)

Container of OSes in Amazon Linux

docker pull ubuntu
docker run -it ubuntu /bin/bash
	# start a bash shell that run inside the docker process tree env
	# the run command by default attach stdin, stdout and stderr to the console,
	# so all keystrokes will pass thru (which is why hitting ^Z) does not put docker to background.
	# this was done on backbox
	# apt-get works, no aptitude (even though host does have this command)
	# ps -ef shows only the bash process!

 
docker pull rhel7
docker run -it rhel7 /bin/bash
	# this was done on backbox
	# uname shows ubuntu
	# /etc/redhat-release exist in the FS inside the container
	# yum works
	# df -hl is very different than the ubuntu container
	# mount shows the same list of mounts as in the ubuntu container

CoreOS

docker pull httpd
docker run -p 80:80 httpd

docker pull pdevine/elinks2     
docker run -it --rm pdevine/elinks2 elinks http://www.yahoo.com		# will start interactive browser 
docker run -it --rm pdevine/elinks2 elinks http://172.31.28.159 	# use the host's eth0 IP to access the web server running on it

alternate entry points, run as user, etc

docker run  -it --entrypoint /usr/bin/Rscript -v $HOME:/tmp/home  --user=$(id -u):$(id -g) tin6150/r4eta

docker run  -it --entrypoint /bin/bash  --user=$(id -u):$(id -g) tin6150/r4eta

Building an image using dockerfile

mkdir  myapp
cd     myapp
vi     dockerfile
docker build -t tin6150/myapp:v3 .		# . will search for ./dockerfile  -t is for tag

# optional upload to docker hub, assuming user tin6150 has been registered
# the "push path" depends on the tag used to do the build, not the relative file path of where dockerfile is located.
docker push     tin6150/myapp 

docker rmi      myapp... 		# remove an image from the lost host.

FROM ubuntu:14.04			# use ubuntu as a base image to build this docker container/app
MAINTAINER user <user@example.com>
RUN apt-get update && apt-get install -y ruby ruby-dev		# 
RUN gem install sinatra
# always run the apt-get update and install command in the same line using &&, 
# or it would result in a db problem on the resulting container.
# on flip side, it is kinda "bad form" needing to run update, should just base it on a newer image.

• Each RUN line adds a layer, better to chain unix commands using &&
• ADD command will copy files from the build host to the container. build is then no longer "build anywhere"

Example Docker HTTPD image + psg static web content

docker run    -p 80:80   -i -t --name ApachePsg -v ~/htdocs4docker:/mnt/htdocs httpd /bin/bash
		# /mnt/htdocs will be created inside the container by the startup process

		# note inside this container, it has very few commands.  no scp, no wget.
		# run this inside the container's bash :
		# cp -pR /mnt/htdocs/psg/* /usr/local/apache2/htdocs
		# httpd 						# this starts process and return to bash

# run the following on the host (ie, from a different window)
docker commit ApachePsg		# create a new image from changes done to exisiting container
docker kill   ApachePsg

docker run    -p 80:80   -i -t --name ApachePsg2 -v ~/htdocs4docker:/mnt/htdocs ApachePsg /bin/bash
## above didn't work.


docker start  ApachePsg		# re-start a container using its name
docker attach ApachePsg		# attach to the container (last start with bash, so get back to a prompt)

docker export 		# Export the contents of a container's filesystem as a tar archive
docker save		# Save an image(s) to a tar archive (streamed to STDOUT by default)
docker commit 		# Create a new image from a container's changes
docker commit  -m "commitMsg" RunningContainerName  	
			# but how is it don't see the new item in docker images?
			# nor was i able to do docker run with the new image...
			# certainly, after commit, psg content remained inside the container.
			# No way to give a name to the image!  and no way to rename it!
			# -m commitMsg is very important for identifying the image using history
			# Even after commit is done, docker ps -a don't show the new container id
			# because that was saved to disk and isn't the running instance!
			# so can only see the info by carefully checking with docker images -a
			# Managment in Docker is said to be none existing!  Kubernetes??

docker images -a		# see that 8e300072616a is newest image created
docker history 8e300072616a	# 8e300072616a is the IMAGENAME from above docker commit.  
				# this cmd will show the commitMsg
				# and some info on how image was created, and the 

docker run    -p 80:80   -i -t --name ApachePsgII 8e300072616a  /usr/local/apache2/bin/httpd
				# confirm that the new image can start


docker rename		# rename a container
? rename image... no way to do this??   but when upload to dockerhub, give it a new name, and pull it again...

docker login			# login to docker hub.  credentials will be saved as base64 enconding in ~/.docker/config.json (ie, NOT encrypted!)

docker commit -m "apache psgIII" edb6d60c97da tin6150/apache_psg3
				# edb6d60c97da is a container id (from docker ps -a)
				# tin6150/apache_psg3 is username/imagename

docker push tin6150/apache_psg3	# push the image to hub.docker.com
				# at this point, image is uploaded to the web and ready for use by others :)


docker tag e4718e38a3b1  tin6150/apache_psg_3a:dev2	# was able to tag and push, no commit...  
	## but this push all images on the chain to create the current image??  
	## certainly pushing lot of stuff...  but prev pull end up with lot of stuff too...
docker push              tin6150/apache_psg_3a:dev2	# in hub.docker.com, see 81MB image.

# docker tag ref https://docs.docker.com/mac/step_six/




podman build -t tin6150/r4envids -f Dockerfile .  | tee Dockerfile.monolithic.LOG
podman login docker.io
podman push tin6150/r4envids           # had pre-created the repo in hub.docker.com
podman push tin6150/r4envids:newtag





## on a different machine, eg centos7, can pull the image to test it out

docker run -p 80:80 -i -t --name ApachePsg_C tin6150/apache_psg3 /bin/bash
     # docker run will pull the image, and create a container to run this service.
     # dropped into bash prompt, can check the htdocs dir.  
     # httpd will start the web server


docker run -p 80:80 -i -t --name ApachePsg_C tin6150/apache_psg_3a:dev2 httpd	# when no latest tag exist, must specify which one to get

Other more advance topics

X11 apps

echo $DISPLAY # :0.0
xhost +
docker run -it -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix  tin6150/metabolic /usr/bin/xpdf

Docker Data Volume

Performance

docker run --cpu-shares=n
	# assign a weight of how much cpu this run command get
	# by default all container share cpu equally.
	# weight are 0 to 1024.  All containers weight are added and then ratio calculated based on requested weight.
	
docker run --cpu-quota=n
	# wonder if they are enforced using cgroup

Dependencies

docker run will pull dependencies needed (think of yum auto download dependencies).

multi-container apps... one way is to view them as different pieces running on multiple host and rely on network communication.

Linking Containers

Multi-container app

docker run --name wordpressdb -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=wordpress -d mysql:5.7	
# starts MySQL container

docker run -e WORDPRESS_DB_PASSWORD=password --name wordpress --link wordpressdb:mysql -p 0.0.0.0:80:80 -d wordpress   
# starts wordpress container
# --link name:alias is to create a private network connection to the named container
#   (--link also copy ENV variables, if conflict, duplicated to HOST_ENV_abc and HOST_PORT_nnn)
# if don't specify to bind to 0.0.0.0:80, wordpress default to some docker bridge IP.
# use "-i -t" instead of "-d" to see web page access log in the console.  
# Can then detach from session using ^P ^Q (it will NOT put container in paused state)
# alt, ^C on process.  docker start wordpress will largely resume where it was left off 
# (wordpress saved most state info to disk).


# (tested to work on amazon linux.  centos7 didn't work, maybe SELinux... maybe cuz ran docker as root.)

Docker Compose

docker-compose up		# read docker-compose.yml, and bring up container(s) as specified in the declarative yml file

probably compete with kubernetes.

Security

1. docker create iptables in the PREROUTING level, thereby often bypassing host's FirewallD or UFW rules. see firewall#docker
2. Docker Daemon typically listen to unix socket. But it could be configured to listen on network port, in which case TLS certs (with key/passphrase) is recommended. see Docker security doc

Quay.io Registry

ghcr - GitHub Container Registry

head -1 ~/.ssh/token | docker login ghcr.io  -u tin6150 --password-stdin
docker tag 869b44021e4a ghcr.io/tin6150/metabolic:4.0
docker push             ghcr.io/tin6150/metabolic:4.0

# but failed, maybe due to size 

	Error: Error copying image to the remote destination: 
	Error writing blob: Error uploading layer to 
	https://ghcr.io/v2/tin6150/metabolic/blobs/upload/8a4c3f7f-1628-43e6-b9aa-0655fe6c1faa?digest=sha256%3A138bf1d88427d73721eb2b23281827e551e1fe7ec42ac63bd35da9859ff8d3ab: 
	received unexpected HTTP status: 504 Gateway Time-out

docker tag b982a301de32 ghcr.io/tin6150/r4envids
docker push             ghcr.io/tin6150/r4envids

github package registry, older way to support container, avoid use:

head -1 ~/.ssh/token | docker login https://docker.pkg.github.com -u tin6150 --password-stdin

docker tag IMAGE_ID docker.pkg.github.com/OWNER/REPOSITORY/IMAGE_NAME:VERSION
docker build     -t docker.pkg.github.com/OWNER/REPOSITORY/IMAGE_NAME:VERSION PATH
docker push         docker.pkg.github.com/OWNER/REPOSITORY/IMAGE_NAME:VERSION

Podman, Docker-compose

# trying on greyhound 


Other than Podman and its dependencies, be sure the podman-docker and docker-compose packages are installed.
(In addition to... )

sudo systemctl start podman.socket

test the socket:
sudo curl -H "Content-Type: application/json" --unix-socket /var/run/docker.sock http://localhost/_ping

# these should work with podman
sudo docker-compose up

sudo podman ps
sudo podman network ls
sudo podman volume ls

SDK

-e  argument of docker run command is likely under the client.containers.run() environment option, need a dict.

PYTHONUNBUFFERED=1   # set to any value to tell python not to use buffered output
buffered=no          # is for Rstudio, but seems to affect cli R in linux as well.

import docker

    atlas = client.containers.run(
        atlas_image,
        volumes=atlas_docker_vols,
        command=atlas_cmd,
        #stdout=docker_stdout,  ## this read settings from yaml
        auto_remove=True,				## likely the --rm option of docker cli
        stdout=True,
        stderr=True,
        #environment='PYTHONUNBUFFERED=1',
        environment=["PYTHONUNBUFFERED=1","buffered=no"],
        detach=True)
        ##xxdetach=False)
    for log in atlas.logs(
            #stream=True, stderr=True, stdout=docker_stdout):
            #stream=True, stderr=True, stdout=True):
            #stream=True, stderr=True, stdout=docker_stdout, timestamps=True):
            stream=True, stderr=True, stdout=True, timestamps=True):
        print(log)

    # 4. CLEAN UP
    atlas.remove()

Reference

• Docker basic from AWS
• Docker Getting Started
• Docker (Taos)slide 50 demo cgroup on memory
• redis key-value db
• nginx (engine x) load balancer
• Open Container Initiative (OCI) - Plumbing: RunC, Notary (content signing)
• man docker # docker daemon man page
• man docker run # man page for the run command of docker
• Container SlideShare
• tba
• tba

Container Runtime

• Docker runtime engine (LXC?)
• containerd
• rkt
• singularity

Container Orchaestration

• Kubernetes
• Docker Swarm
• Rancher
• Mesos Marathon

Container Landscape

Players in the container world, circa 2016. (Univa slide)

Container in HPC

LBNL/NERSC Shifter

• NeRSC R&D + collaboration w/ Cray, in production use at NeRSC (Edison, Cori), open source release soon. Cray may develop commercial support for this.
• Paper in SC 2015.11
• https://www.nersc.gov/news-publications/nersc-news/nersc-center-news/2015/shifter-makes-container-based-hpc-a-breeze/
• Flexibility: user-defined images (UDI) sw stack, placed as container image.
• Reproducible research: UDI can be validated and workflow will run exactly every time it is started, even if environment may have aged.
• SW stack isolation. Different users in diff realm maintain their own UDI, HPC just run them as jobs, they are completely independent of one another. Accomplished by using linux VFS namespaces to run multiple Shifter containers.
• Utilize public image repos like DockerHub, user can start application stack without sys admin priviledges.
• Not using Docker daemon, as that assume local disk (aufs, btrfs)
• native app perf, fast job startup.
• FS are re-mounted, no setuid.
• Containers are read-only to user (write to network location?) (Dockers allow changes to container after admin issue save command.)
• Shifter essentially adopt docker images, but have to redo the backend to provide docker daemon-like features executed by the HPC batch manager system.

LBNL/HPCS Singularity v 1.0

• For feature of the newer Singularity 2.2, see Docker vs Singularity vs Shifter
• VM does not address portability. Singularity address portability.
• It is like getting the portability of Docker and make it run in an HPC batch system. However, Singularity does not use any piece of Docker.
• SAPP (Singularity App) = Combination of RPM and Container.
• ldd shows what library a program need. Singularity automatically build dependency tree and add all the required dependency into container it builds.
• all dependencies, down to C libraries, are included. Thus, SAPP is very portable and reproducible.
• SAPP is kinda like a static-link binary, but its container also bundle additional scripts, config files etc that make up a workflow.
• Developer can provide SAPP of complex workflow and user won't have to worry about any dependencies!!
• file system mounted inside the container without using bind mounts.
• container run as user-space process.
• By and large have no kernel dependencies, other than IB and MPI.
• SAPP runs as user process, thus easy to get it to run by HPC scheduler. There is probably little the HPC scheduler need to run, other than treat SAPP as an app and run it. This is a HUGE diff compared to Shifter.
• OpenMPI support is already included. http://singularity.lbl.gov/docs_hpc.html
• On the high level, there are a lot of ideas similarity with Shifter. After all, both address similar concerns of wanting reproducible workflow. Implementations of the two are drastically different. see faq
• Developed by Gregory Kurtzer, of Warewulf fame et al. Ver 1 was released April 14, 2016. Ver 2 in beta now. Avail at gmkurtzer.github.io/. G.K. is from Berkeley Lab High Performance Computing Services (HPCS).

Container in UGE

• Univa promises to allow qsub to run job involving docker Containers.
• qsub will have an -xdv option to map file system of the host to process running inside container
• docker will be defined as a boolean resrouce, and requested via -l option of qsub
• Additional management and orchestration by new products: Navops Launch, Command.
• Prioritization for different containers

Copyright info about this work

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike2.5 License. Pocket Sys Admin Survival Guide: for content that I wrote, (CC) some rights reserved. 2005,2012 Tin Ho [ tin6150 (at) gmail.com ]

Some contents are "cached" here for easy reference. Sources include man pages, vendor documents, online references, discussion groups, etc. Copyright of those are obviously those of the vendor and original authors. I am merely caching them here for quick reference and avoid broken URL problems.

Where is PSG hosted these days?