Let's walk through how to build and push Docker images programmatically using Go. To do this, we need to talk to the Docker daemon via the Docker Engine API. This is similar to how the Docker CLI works, but instead of entering commands through a CLI, we'll be writing code with Docker's Go SDK.
\nAt the time of writing, the official Docker Go SDK docs provide great examples of running basic Docker commands with Go. However, it's missing examples on building and pushing Docker images, so we'll go over those in this blog.
\nBefore we begin, this blog assumes you have a working knowledge of Docker and Go. We'll go over the following:
\nFirst, we need to set up the environment. Create a project and include the app we want to containerize:
\nmkdir docker-go-tutorial && cd docker-go-tutorial && mkdir node-helloWe'll add a simple Node.js app:
\n// node-hello/app.js\nconsole.log("Hello From LoginRadius");with the Dockerfile:
\n// node-hello/Dockerfile\nFROM node:12\nWORKDIR /src\nCOPY . .\nCMD [ "node", "app.js" ]Next, install the Go SDK. These are the Docker related imports we will be using:
\n"github.com/docker/docker/api/types"\n"github.com/docker/docker/client"\n"github.com/docker/docker/pkg/archive"One way to build a Docker image from our local files is to compress those files into a tar archive first.
\nWe use the archive package provided by Docker:
\n"github.com/docker/docker/pkg/archive"tar, err := archive.TarWithOptions("node-hello/", &archive.TarOptions{})\nif err != nil {\n return err\n}Now, we can call the ImageBuild function using the Go SDK:
\nNote that the image tag includes our Docker registry user ID, so we can push this image to our registry later.
\nopts := types.ImageBuildOptions{\nDockerfile: "Dockerfile",\nTags: []string{dockerRegistryUserID + "/node-hello"},\nRemove: true,\n}\nres, err := dockerClient.ImageBuild(ctx, tar, opts)\nif err != nil {\nreturn err\n}To print the response, we use a scanner to go through line by line:
\nscanner := bufio.NewScanner(res.Body)\nfor scanner.Scan() {\n lastLine = scanner.Text()\n fmt.Println(scanner.Text())\n}This prints the following:
\n{"stream":"Step 1/4 : FROM node:12"}\n{"stream":"\\n"}\n{"stream":" ---\\u003e e4f1e16b3633\\n"}\n{"stream":"Step 2/4 : WORKDIR /src"}\n{"stream":"\\n"}\n{"stream":" ---\\u003e Using cache\\n"}\n{"stream":" ---\\u003e b298b8519669\\n"}\n{"stream":"Step 3/4 : COPY . ."}\n{"stream":"\\n"}\n{"stream":" ---\\u003e Using cache\\n"}\n{"stream":" ---\\u003e 1ff6a87e79d9\\n"}\n{"stream":"Step 4/4 : CMD [ \\"node\\", \\"app.js\\" ]"}\n{"stream":"\\n"}\n{"stream":" ---\\u003e Using cache\\n"}\n{"stream":" ---\\u003e 6ca44f72b68d\\n"}\n{"aux":{"ID":"sha256:238a923459uf28h80103eb089804a2ff2c1f68f8c"}}\n{"stream":"Successfully built 6ca44f72b68d\\n"}\n{"stream":"Successfully tagged lrblake/node-hello:latest\\n"}The last step would be checking the response for errors, so if something went wrong during the build, we could handle it.
\nerrLine := &ErrorLine{}\njson.Unmarshal([]byte(lastLine), errLine)\nif errLine.Error != "" {\n return errors.New(errLine.Error)\n}For example, the following error can occur during build:
\n{"errorDetail":{"message":"COPY failed: stat /var/lib/docker/tmp/docker-builder887191115/z: no such file or directory"},"error":"COPY failed: stat /var/lib/docker/tmp/docker-builder887191115/z: no such file or directory"}All together, the file looks like this:
\npackage main\n\nimport (\n\t"bufio"\n\t"context"\n\t"encoding/json"\n\t"errors"\n\t"fmt"\n\t"io"\n\t"time"\n\n\t"github.com/docker/docker/api/types"\n\t"github.com/docker/docker/client"\n\t"github.com/docker/docker/pkg/archive"\n)\n\nvar dockerRegistryUserID = ""\n\ntype ErrorLine struct {\n\tError string `json:"error"`\n\tErrorDetail ErrorDetail `json:"errorDetail"`\n}\n\ntype ErrorDetail struct {\n\tMessage string `json:"message"`\n}\n\nfunc main() {\n\tcli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())\n\tif err != nil {\n\t\tfmt.Println(err.Error())\n\t\treturn\n\t}\n\n\terr = imageBuild(cli)\n\tif err != nil {\n\t\tfmt.Println(err.Error())\n\t\treturn\n\t}\n}\n\nfunc imageBuild(dockerClient *client.Client) error {\n\tctx, cancel := context.WithTimeout(context.Background(), time.Second*120)\n\tdefer cancel()\n\n\ttar, err := archive.TarWithOptions("node-hello/", &archive.TarOptions{})\n\tif err != nil {\n\t\treturn err\n\t}\n\n\topts := types.ImageBuildOptions{\n\t\tDockerfile: "Dockerfile",\n\t\tTags: []string{dockerRegistryUserID + "/node-hello"},\n\t\tRemove: true,\n\t}\n\tres, err := dockerClient.ImageBuild(ctx, tar, opts)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tdefer res.Body.Close()\n\n\terr = print(res.Body)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\nfunc print(rd io.Reader) error {\n\tvar lastLine string\n\n\tscanner := bufio.NewScanner(rd)\n\tfor scanner.Scan() {\n\t\tlastLine = scanner.Text()\n\t\tfmt.Println(scanner.Text())\n\t}\n\n\terrLine := &ErrorLine{}\n\tjson.Unmarshal([]byte(lastLine), errLine)\n\tif errLine.Error != "" {\n\t\treturn errors.New(errLine.Error)\n\t}\n\n\tif err := scanner.Err(); err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}The equivalent Docker CLI command would be:
\ndocker build -t <dockerRegistryUserID>/node-hello .We'll push the Docker image we created to Docker Hub. But, we need to authenticate with Docker Hub by providing credentials encoded in base64.
\nIf you don't want to use your Docker Hub password, you can set up an access token and provide that in the Password field instead.
\nvar authConfig = types.AuthConfig{\nUsername: "Your Docker Hub Username",\nPassword: "Your Docker Hub Password or Access Token",\nServerAddress: "https://index.docker.io/v1/",\n}\nauthConfigBytes, _ := json.Marshal(authConfig)\nauthConfigEncoded := base64.URLEncoding.EncodeToString(authConfigBytes)Now, call the ImagePush function in the Go SDK, along with your encoded credentials:
\nopts := types.ImagePushOptions{RegistryAuth: authConfigEncoded}\nrd, err := dockerClient.ImagePush(ctx, dockerRegistryUserID + "/node-hello", opts)Together, this looks like:
\nfunc imagePush(dockerClient *client.Client) error {\n\tctx, cancel := context.WithTimeout(context.Background(), time.Second*120)\n\tdefer cancel()\n\n\tauthConfigBytes, _ := json.Marshal(authConfig)\n\tauthConfigEncoded := base64.URLEncoding.EncodeToString(authConfigBytes)\n\n\ttag := dockerRegistryUserID + "/node-hello"\n\topts := types.ImagePushOptions{RegistryAuth: authConfigEncoded}\n\trd, err := dockerClient.ImagePush(ctx, tag, opts)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tdefer rd.Close()\n\n\terr = print(rd)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}The equivalent Docker CLI command (after docker login) would be:
\ndocker push <dockerRegistryUserID>/node-hello\n\nAssuming that the reader of this blog has a fair idea of the docker ecosystem!
\n
Docker has changed the world of software development! Since the last few years, docker has helped the developer community, enterprises, and startups to create solutions quickly and effectively. Also, deployment is relatively hassle-free with docker (conditions apply). And worth mentioning is, it resolves the “Works fine on my system” problem.
\nWell, just not docker, there are many container ecosystems available as an alternative to Docker. For example, Mesos by Apache and Vagrant by Hashicorp. Docker is more loved for what it is not, and that is Bulky! Docker is otherwise lightweight, works mostly with Linux and eventually was identified by Kubernetes in 2015. But, that story... some other time!!
\nStraight to the context, Docker Compose! Docker compose is a powerful utility that is bundled with Docker installation. Docker-Compose can be used for production and development, to make things virtually seamless.
\nDocker is doing a great job when it comes to describing and developing microservices.
\nWhile working on microservices where quite a few ( being reasonably complicated, in our example 6 ) containers talk to each other to complete a task and since it is an end to end task, the whole thing looks very complicated. To rectify :
\nthe developer has two choices, create a script in a scripting language of choice, or choose docker-compose. Well, docker-compose is an easier option.
\nHere in this kind of practical example, we are going to demonstrate a similar situation, where multiple containers talk to each other. We are going to use docker-compose as our primary tool to build and deploy images.
\nExactly! What is the problem? As such, there is no problem, but it may arise if we do not program or develop it well in an integrated manner. Programming a component and managing it well reduces the time-to-production to a greater degree.
\n\n\nI am not a great programmer; I am just a good programmer with great habits.
\n\n
\n- Martin Fowler
\n
While designing and developing complicated systems where microservices are involved, integration and debugging become cumbersome.
\nTo ease it up docker-compose acts a friend.
\nFor the example above we are going to write a docker-compose. Docker-Compose has got powers to
\nFirst things first, the Dockerfile.
\nStarting from the API. The API needs a docker image to build. The following is the Dockerfile for the same.
\nFROM node:alpine\nWORKDIR '/app'\nCOPY ./package.json ./\nRUN npm install \nCOPY . .\nCMD ["npm", "run", "start"]Since the other two components, listener and frontend are also written in JavaScript the Dockerfile does not change at all for them too, for this particular example.
\nThe following is a two-liner Dockerfile for gateway. The conf file targets apiserver and frontend. Please have a look at the configuration file in the repository itself.
\nFROM nginx\nCOPY ./nginx.conf /etc/nginx/conf.d/default.confThe other 2 components :
\nFirst things first, file version. Since it is all yaml out there, the file version tells docker-compose about the data structure. The data structure which is expected by docker \"compose\".
so we start the docker-compose as
\nversion: '3'All the components in the docker-compose are known as services. Next, we are going to write some services. Yes, the services we have written for our business.
Message Queue :
\nThe message queue shall use softwaremill/elasticmq image out of the box and the applications will access that on port 9324. The service description in docker-compose shall look like this.
mqserver:\n image: softwaremill/elasticmq\n ports:\n - '9324:9324'That is analogous to docker run -it -p '9324:9324' softwaremill/elasticmq.
A bit more complicated service next, mongodb.
\nmongodbdb: \n depends_on: \n - mqserver\n image: mongo\n container_name: mongo\n volumes:\n - /data/db:/mongodata\n ports:\n - '27017:27017'There are a couple of things one may notice more here.
\ndepends_on shall wait for the container of mqserver service to come up, before mongodbdb service goes on air.
\n\nDocker is only liable to check the containers when we use depends_on, if you want to check whether your service (which you have embedded) is ready to use or not, use
\nhealthcheckandlink.
Apart from this, we are mounting a volume and exposing ports for an image mongo which gets pulled directly from the docker container registry. container_name will give the service a name, so that it can be discovered *in the network. *Kind of a domain name in the internal docker-compose network.
Now the developer's proprietary API service:
\napiserver:\n restart: always\n depends_on: \n - mqserver\n - mongodbdb\n\n build:\n dockerfile: Dockerfile\n context: ../fs-express-api/.\n volumes:\n - /app/node_modules\n environment: \n - NODE_ENV=production\n - AWS_SQS_URL=http://mqserver:9324/\n - AWS_SQS_FQ_URL=http://mqserver:9324/queue/fsqueue\n - AWS_ACCESS_KEY=na\n - AWS_SECRET_ACCESS_KEY=na\n - AWS_REGION=REGION\n - QUEUE_NAME=fsqueue\n - MONGODB_CONNECTION=mongodb://mongo:27017/fsorder\n - API_APP_PORT=5000\n ports:\n - '5000:5000'The buildtag will be used by the docker-compose build command to build an image. The build is defined as Dockerfile and the context is docker context, the directory which you want to use as context. Equivalent to: docker build -f filename.
The more you are seeing here is environment and restart options. Environment variables are used to set environment variables while running the image. This is another part where docker-compose comes handy. Think about setting all nine environment variables with docker run.
restart is a bit interesting. One may specify a restart policy if the application crashes, which leads to stopping the docker image itself. always specifies to restart the container if it exit.
In the environment section, one can see mqserver and mongo, instead of the regular localhost and loopback address. This is because when we run docker-compose, it creates a DNS recordset of all the services mentioned. The services can be referred to by their names. A slight deviation here may happen when the container is explicitly named. Here, container_name = mongo is used for service mongodbdb. Hence, the other containers in the environment shall refer to it by mongo and not mongodbdb.
The pollingmod and frontend services are quite the same.
\n pollingmod:\n restart: always\n depends_on: \n - mqserver\n - mongodbdb\n build:\n dockerfile: Dockerfile\n context: ../polling-mod/\n volumes:\n - /app/node_modules\n environment: \n - AWS_QUEUE_URL=http://mqserver:9324/queue/fsqueue\n - AWS_ACCESS_KEY=na\n - AWS_SECRET_ACCESS_KEY=na\n - AWS_REGION=REGION\n - MONGODB_CONNECTION=mongodb://mongo:27017/fsorder \n\n frontend:\n build:\n dockerfile: Dockerfile\n context: ../fs-frontend\n volumes:\n - /app/node_modules\n - ../fs-frontend:/app\n stdin_open: true\n ports:\n - '3000:3000'Finally, the gateway. So, a gateway is created on the front using Nginx to make the development production-grade. If you go through the gateway code you will find that the services are accessed by their name frontend and apiserver.
upstream frontend{\n server frontend:3000;\n}\n\nupstream apiserver{\n server apiserver:5000;\n}The Nginx container opens port 80 to an external port 5500. The Nginx container keeps configuration where the user can see just one port 5500 from out of the system, everything else is gray.
\nI am here, implementing a Clair tool in gitlab-ci.yml to get secure docker images.
\nimage_scanning:\n stage: scan\n image: docker:stable\n tags:\n - gitlab-org-docker\n services:\n - docker:19.03.8-dind \n variables:\n DOCKER_DRIVER: overlay2\n allow_failure: true\n before_script:\n - echo $CI_BUILD_TOKEN | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY\n script:\n - docker run -d --name db arminc/clair-db:latest\n - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.1\n - apk add -U wget ca-certificates\n - docker pull $CI_REGISTRY_IMAGE:$PROJECT_NAME-latest || true\n - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64\n - mv clair-scanner_linux_amd64 clair-scanner\n - chmod +x clair-scanner\n - touch clair-whitelist.yml\n - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done\n - retries=0\n - echo "Waiting for clair daemon to start"\n - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done\n - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml $CI_REGISTRY_IMAGE:$PROJECT_NAME-latest || true\n - cat gl-container-scanning-report.json\n artifacts:\n paths: [gl-container-scanning-report.json]\n rules:\n - if: '$CI_COMMIT_BRANCH == "staging"'\n when: alwaysgitlab-org-docker is a GitLab shared-runner to run this analysis (an agent on which the above-described job will run), it will fetch the latest ms image and will run it against the CVE database, at last record the report in the JSON file which we can store as artifacts. These artifacts can be further used by the developer to see and resolve the vulnerabilities.